Bronchoscope path planning device

Abstract

This application relates to the field of medical information processing technology, specifically to a path planning device for bronchoscopy. First, a construction module collects multimodal medical image data of the patient across multiple respiratory phases and generates a 3D model including the lesion, bronchi, and accompanying blood vessels. Then, the 3D model is loaded onto an AR display device, and a virtual anatomy module performs AR virtual anatomy on the 3D model based on target operation commands until the lesion's transparency reaches a set level, resulting in an anatomical model corresponding to the 3D model. Next, an initial path is generated in the anatomical model based on preset safety constraints. Finally, a dynamic planning module dynamically updates the initial path in conjunction with real-time respiratory phases to obtain the target path. This application overcomes the blind spot caused by lesion obstruction, intuitively presenting the target small bronchial segment obscured by the lesion, thus improving the operational accuracy and safety of bronchoscopy.

Description

Technical Field

[0001] This application relates to the field of medical information processing technology, specifically to a bronchoscope path planning device. Background Technology

[0002] Bronchoscopy is one of the core methods for the diagnosis and treatment of lung tumors, infectious diseases, and diffuse lung diseases. It involves inserting a slender bronchoscope through the airway into the lung to achieve visual observation and precise manipulation of the bronchial tree at all levels. In bronchoscopic procedures, such as peripheral lung nodule biopsy and subsegmental lesion resection, small bronchi of grades 4 and 5 (usually <3mm in diameter) often become "blind spots" due to obstruction by lesions (e.g., ground-glass nodules, peripheral lung cancer). This makes it difficult for surgeons to accurately determine the direction of the bronchi during preoperative planning, leading to misjudgments of the insertion path, such as deviating from the bronchial lumen or accidentally touching accompanying blood vessels. Summary of the Invention

[0003] The purpose of this application is to provide a bronchoscope path planning device to solve the problem of low accuracy in the path planning of traditional bronchoscopes.

[0004] To achieve the above objectives, a first aspect of this application provides a bronchoscope path planning device, comprising:

[0005] A construction module is used to collect multimodal medical image data of multiple respiratory phases of patients and generate a three-dimensional model. The three-dimensional model is loaded onto an AR display device. The three-dimensional model includes lesions, bronchi and accompanying blood vessels matching the bronchi. The bronchi include target small bronchial segments that are obscured by the lesions. The accompanying blood vessels include target accompanying blood vessels that match the target small bronchial segments.

[0006] The virtual anatomy module is used to perform AR virtual anatomy on the three-dimensional model based on the target operation command until the transparency of the lesion reaches the set transparency, thereby obtaining the anatomical model corresponding to the three-dimensional model.

[0007] The path generation module is used to take the path from the inlet node of the bronchus to the opening node of the target small bronchial segment as the starting segment path and the target operating area as the end point, and generate an initial path in the anatomical model based on preset safety constraints.

[0008] The dynamic planning module is used to dynamically update the initial path in conjunction with real-time respiratory phases to obtain the target path, and to provide risk warnings for the target path on the display interface of the AR display device.

[0009] The beneficial effects of this application are:

[0010] This application provides a bronchoscope path planning device. First, it acquires multimodal medical image data of multiple respiratory phases of the patient through a construction module and generates a three-dimensional model containing lesions, bronchi and accompanying blood vessels. The three-dimensional model includes a target small bronchial segment obscured by the lesion and a target accompanying blood vessel matching the target small bronchial segment.

[0011] Then, the 3D model is loaded onto the AR display device. The virtual anatomy module performs AR virtual dissection on the 3D model based on target operation commands until the lesion's transparency reaches the set level, resulting in the corresponding anatomical model. By dynamically adjusting the lesion's transparency, doctors can penetrate the lesion to observe the spatial relationship between the target small bronchial segment and the lesion and accompanying blood vessels.

[0012] Next, the path generation module takes the path from the inlet node of the bronchus to the opening node of the target small bronchial segment as the starting path and the target operating area as the endpoint, and generates the initial path in the anatomical model based on preset safety constraints, thereby balancing the distance and safety of path planning.

[0013] Finally, the initial path is dynamically updated by combining the dynamic programming module with the real-time respiratory phase to obtain the target path, which reduces the impact of bronchial displacement caused by respiratory motion on path accuracy. The target path is also displayed on the AR display device's interface to provide risk warnings, enabling the operator to dynamically adjust the timing of the operation.

[0014] In summary, this application utilizes multimodal medical imaging data and AR display devices to achieve virtual anatomy overcoming lesion obstruction blind spots, intuitively presenting the target small bronchial segments obscured by lesions. Through safety constraints and real-time dynamic path planning, a closed loop of preoperative planning and intraoperative navigation is formed, transforming subjective experience into quantifiable technical parameters, reducing reliance on the operator's subjective experience, and improving the operational accuracy and safety of bronchoscopy.

[0015] Other features and advantages of this application will be described in detail in the following detailed description section. Attached Figure Description

[0016] Figure 1 This is a schematic diagram illustrating the application scenario of the bronchoscope path planning device in one embodiment of this application.

[0017] Figure 2 This is a schematic diagram of the structure of a bronchoscope path planning device provided in one embodiment of this application;

[0018] Figure 3 This is a schematic diagram of the structure of a three-dimensional model provided in an embodiment of this application;

[0019] Figure 4This is a side view of a three-dimensional model provided in an embodiment of this application;

[0020] Figure 5 This is a side view of an anatomical model provided in an embodiment of this application;

[0021] Figure 6 This is a schematic diagram illustrating the application scenario of the bronchoscope path planning device in another embodiment of this application.

[0022] Figure 7 This is a schematic diagram of a bronchoscope path planning device provided in another embodiment of this application.

[0023] Explanation of reference numerals in the attached figures

[0024] 1. Controller; 2. Bronchoscope; 3. AR display device; 4. Collaborative system; 5. Terminal. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified. In this application, the term "exemplary" is used to mean "used as an example, illustration, or description." Any embodiment described as "exemplary" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to implement and use this application. In the following description, details are set forth for illustrative purposes. It should be understood that those skilled in the art will recognize that this application can be implemented without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid unnecessary detail that would obscure the description of this application. Therefore, this application is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0027] Figure 1This is a schematic diagram illustrating an application scenario of the bronchoscope path planning device according to an embodiment of this application. The application scenario may include a controller 1, a bronchoscope 2, and an AR display device 3. The controller 1 communicates with both the bronchoscope 2 and the AR display device 3.

[0028] Controller 1 integrates a path planning device for bronchoscope 2, which may include a memory and a processor. The memory is configured to store instructions and data, and the processor is configured to retrieve instructions and data from the memory and execute the steps of the path planning device for bronchoscope 2 when running instructions. Controller 1 can perform core tasks such as data processing, model building, path planning, and dynamic adjustment. In one example, controller 1 can receive and fuse multimodal medical imaging data, such as the patient's four-dimensional computed tomography (4D-CT) images, enhanced computed tomography (CT) images, and endobronchial ultrasound (EBUS) images. The 4D-CT and enhanced CT images can be acquired based on different scanning modes of the same spiral CT scanner and then transmitted to the memory of controller 1 for storage. 4D-CT can capture the spatial relationship between the bronchi and lesions under the dynamic breathing of the patient. Enhanced CT can capture clearly visualized accompanying vessels, assisting in the constraint of bronchial orientation.

[0029] The bronchoscope 2 combines a data acquisition terminal and a path execution carrier. EBUS images can be acquired through the EBUS integrated on the bronchoscope 2. Therefore, a miniature ultrasound probe, such as an EBUS probe, can be integrated into the front end of the bronchoscope 2. This miniature ultrasound probe matches the working channel of the bronchoscope 2 and can utilize the lesion-penetrating properties of ultrasound waves to capture the wall structure of small bronchi obscured by lesions, compensating for the limitation of CT in penetrating lesions. The controller 1, through data interaction with the bronchoscope 2, can acquire detailed data of the lesion-obscured area and provide real-time feedback on its position and orientation. Based on the acquired data, it can plan the path for the bronchoscope 2 and guide the execution of the path.

[0030] Traditional bronchoscopy path planning typically requires physicians to operate based on a plan displayed on a monitor, necessitating frequent head-down checks and re-head-up movements. This makes it difficult to accurately correlate the virtual model with the actual anatomical location, especially for small bronchi, where switching can easily lead to positioning errors of 2-3 mm, far exceeding clinical safety thresholds. Furthermore, for spatial blind spots obscured by lesions, physicians must rely on subjective imagination to piece together 3D relationships, which can easily result in misjudgments of bronchial direction due to spatial positioning deviations, leading to operational risks. Therefore, this embodiment of the application establishes communication between the controller 1 and the augmented reality (AR) display device 3, providing a visual interaction and navigation terminal.

[0031] AR display device 3 is an intelligent reality terminal that uses optical perspective technology to overlay virtual information onto the real environment in real time. It retains the user's intuitive perception of the real scene while also supplementing the visual environment with computer-generated virtual content, enabling interaction between the displayed world and digital information. For example, in the bronchoscopy scenario of this application embodiment, AR display device 3 may include a head-mounted AR display device. The head-mounted AR display device can display the field of view of the bronchoscope 2, which refers to the area of ​​the bronchus and surrounding tissues captured in real time by the optical imaging sensor at the front end of the bronchoscope and transmitted to the AR display device 3. Then, real-time anatomical information, navigation paths, operation instructions, risk warnings, and deviation alerts can be overlaid onto the field of view of the bronchoscope 2 using AR markers.

[0032] Understandable, Figure 1 The electronic devices in the application scenario of the bronchoscope 2 path planning device shown do not constitute a limitation on the embodiments of this application. That is, the number and type of devices included in the application scenario of the bronchoscope 2 path planning device, or the number and type of devices included in each electronic device, do not affect the overall implementation of the technical solution in the embodiments of this application, and can all be considered as equivalent substitutions or derivatives of the technical solutions claimed in the embodiments of this application.

[0033] In this application embodiment, controller 1 can be an independent device, or a device network or device cluster composed of devices. For example, controller 1 described in this application embodiment includes, but is not limited to, a computer, a network host, a single network device, a set of multiple network devices, or a cloud device composed of multiple devices. Among them, the cloud device is composed of a large number of computers or network devices based on cloud computing.

[0034] Those skilled in the art will understand that Figure 1The application scenarios shown are merely one application scenario corresponding to the technical solution of this application, and do not constitute a limitation on the application scenarios of the technical solution of this application. Other application scenarios may include more than one application scenario. Figure 1 The number of more or fewer electronic devices shown, or the network connections of electronic devices, for example Figure 1 Only one electronic device is shown in the image. It is understood that the path planning device of the bronchoscope 2 may also include one or more other electronic devices, which are not limited here.

[0035] It should be noted that, Figure 1 The application scenario of the bronchoscope 2 path planning device shown is merely an example. The application scenario of the bronchoscope 2 path planning device described in this application embodiment is to more clearly illustrate the technical solution of this application embodiment and does not constitute a limitation on the technical solution provided in this application embodiment.

[0036] Based on the application scenario of the bronchoscope 2 path planning device described above, an embodiment of the bronchoscope 2 path planning device is proposed. The modules and units in this embodiment can communicate with each other. A detailed description is provided below with reference to the accompanying drawings.

[0037] Figure 2 This is a schematic diagram of the path planning device for a bronchoscope 2 provided in one embodiment of this application. Figure 2 As shown, the path planning device 200 of the bronchoscope 2 may include a construction module 201, a virtual anatomy module 202, a path generation module 203, and a dynamic planning module 204.

[0038] The construction module 201 is used to collect multimodal medical image data of multiple respiratory phases of the patient and generate a three-dimensional model, and load the three-dimensional model onto the AR display device 3.

[0039] Multimodal medical imaging data refers to medical imaging data acquired through multiple methods, such as 4D-CT images, enhanced CT images, and EBUS images. A single CT image is difficult to penetrate lesions, and a single EBUS image has a limited field of view. Therefore, multimodal medical imaging data can compensate for the deficiencies of single-modal imaging. Multiple respiratory phases refer to key nodes in the respiratory cycle where the bronchial position is relatively stable. For example, a respiratory cycle from end of inspiration to end of expiration can include five key respiratory phases: end of inspiration, mid-inspiration, quiescent phase, mid-expiration, and end-expiration. This allows for the capture of the bronchial displacement patterns during respiration.

[0040] After acquiring multimodal medical image data, spatial coordinates of the data can be aligned using a registration algorithm to generate a three-dimensional model in the same coordinate system. Figure 3This is a schematic diagram of the structure of a three-dimensional model provided in an embodiment of this application. See also... Figure 3 The 3D model can include the lesion, bronchi, and accompanying vessels matching the bronchi. The 3D model can include multiple levels of bronchial branches, for example, from level 1 to 5, where levels 4 and 5 are small bronchi with a diameter of less than 3 mm. The bronchi can include target small bronchial segments obscured by the lesion, and the accompanying vessels can include target accompanying vessels matching the target small bronchial segments. The target small bronchial segment is a level 4-5 small bronchus obscured by the lesion; for example, a bronchus with a diameter of less than 3 mm is considered a small bronchus. The target small bronchial segment is the last small passage for the bronchoscope 2 to reach the peripheral lesion, and its anatomical characteristics directly affect the feasibility of the route. The target accompanying vessel is a pulmonary artery branch accompanying the target small bronchial segment, with a direct distance of approximately 1-2 mm, and the distance between the two is usually less than 2 mm. This is a key safety risk source that needs to be avoided in route planning. Figure 3 In the middle, the target bronchial segment and the target accompanying blood vessel are obscured by the lesion.

[0041] Finally, the 3D model is converted to a format compatible with the AR display device 3, such as GLB (GL Transmission Format Binary). Then, it is transmitted to the AR display device 3 to achieve spatial anchoring between the model and the patient's actual lung region, providing a foundation for subsequent virtual anatomy and pathway planning. For example, consider a side view of the 3D model. Figure 4 As shown, Figure 4 This is a side view of a three-dimensional model provided in the embodiments of this application. The lesion is a gray elliptical area, and the target small bronchus and the target accompanying blood vessel are obscured by the lesion.

[0042] The virtual anatomy module 202 is used to perform AR virtual anatomy on the three-dimensional model based on the target operation command until the transparency of the lesion reaches the set transparency, thereby obtaining the anatomical model corresponding to the three-dimensional model.

[0043] Multimodal medical imaging data can collect location data of target small bronchial segments that are obscured by lesions. In order to make the obscured small bronchial segments more intuitive, AR virtual anatomy can be performed through AR display device 3 to make the obscured target small bronchial segments intuitively visible.

[0044] Specifically, the interactive sensor of the AR display device 3 can capture the operator's target operation command and perform AR virtual dissection based on the target operation command. The target operation command is the dissection operation request issued by the operator (such as a doctor) through natural interaction (such as through gestures or voice). AR virtual dissection uses AR technology to perform non-invasive digital dissection and transparency operations on a 3D model, simulating the observation effect of physical dissection, but without destroying the integrity of the model, thus obtaining an anatomical model corresponding to the 3D model. The anatomical model is the 3D model after dissection. While preserving the spatial anchoring of the lesion, it makes the obscured target small bronchial segments visible, thereby transforming spatial imagination into intuitive observation of the 3D AR model, reducing the time required for the operator to understand the relationship between the target small bronchus, lesion, and accompanying blood vessels. Figure 5 This is a side view of an anatomical model provided in an embodiment of this application. Figure 5 As shown, the lesion is a light gray oval area, and the target small bronchus and the target accompanying blood vessels are visible through the lesion without visual obstruction, and the outline of the lesion can still be seen.

[0045] The path generation module 203 is used to generate an initial path in the anatomical model based on preset safety constraints, with the path between the inlet node of the bronchus and the opening node of the target small bronchial segment as the starting path and the target operating area as the endpoint.

[0046] The bronchial inlet node is the accessible starting point of the bronchus, such as the opening of a third-order or higher bronchus. The inlet node has a fixed location, a large lumen, and is clearly visible on CT images, facilitating stable bronchoscopy entry. The opening node of the target small bronchial segment is the connection point with the superior bronchus, i.e., the boundary between the unobstructed segment and the segment obstructed by the lesion. See [link to relevant documentation]. Figure 5 The proximal superior bronchus of the opening node is visible, while the distal target small bronchial segment is obscured by the lesion. Using the path from the bronchial inlet node to the opening node of the target small bronchial segment as the starting path allows for precise connection between the known, visible safe lumen and the obscured lumen requiring virtual navigation. Furthermore, the operating principle of bronchoscope 2 is to enter along the natural lumen, prohibiting penetration of normal lung tissue or lesions. The starting path can be entirely along the centerline of the natural lumen of bronchus 3 or higher, adhering to the physical constraints of bronchoscope 2. The target operating area is the endpoint of bronchoscope 2's operation and can be dynamically defined according to the specific clinical scenario. For example, in a biopsy scenario, the biopsy sampling point is the target operating area. In a resection scenario, the resection boundary is the target operating area.

[0047] The path from the starting point to the endpoint contains a small bronchial segment obscured by the lesion. These small bronchial segments are less than 3 mm in diameter and surrounded by dense blood vessels, making them easily compressed by the lesion, resulting in high operational risk and low tolerance for error. Therefore, pre-set safety constraints can be introduced to limit the safe range of the path design, reducing serious complications such as vascular injury and bronchial wall tearing. Pre-set safety constraints are quantitative indicators based on clinical operating procedures and bronchial anatomy, ensuring a balance between operability and safety in the bronchoscopy path. In one example, safety constraints may include the spatial distance between the path and the accompanying target blood vessel, the morphological fit with the target small bronchial segment, and the fit with the centerline of the target bronchial lumen. Based on these pre-set safety constraints, an initial path can be generated in the anatomical model. This initial path, the pre-planned baseline path, serves as a safety template for subsequent dynamic adjustments.

[0048] The dynamic programming module 204 is used to dynamically update the initial path in conjunction with the real-time respiratory phase to obtain the target path, and to provide risk warnings for the target path on the display interface of the AR display device 3.

[0049] Real-time respiratory phase refers to the patient's current respiratory cycle position and needs to be matched with the pre-acquired phase of the 4D-CT image to accurately predict bronchial displacement. For example, real-time respiratory signals from the patient are acquired using a chest wall motion sensor and combined with the respiratory phase model from the 4D-CT image. A dynamic time warping algorithm is then used to match the current phase to determine the real-time displacement of the target small bronchial segment. The deviation between the initial path and the bronchial position at the current phase is then calculated, and the initial path is updated based on this deviation to obtain the target path. The target path is a dynamically updated intraoperative navigation path that adapts in real-time to changes in bronchial position caused by respiration.

[0050] Simultaneously, risk warnings can be displayed on the AR display device 3's interface. Risk warnings transform abstract safety parameters into intuitive visual signals to assist operators in quickly assessing operational risks and improving decision-making efficiency. For example, risk warnings can be color-coded and / or flashing indicators on a risk heatmap, annotations of dynamic parameters, and operational suggestions. Through real-time phase matching and path correction, the accuracy of respiratory dynamic interference paths can be resolved, reducing the deviation between the path and the actual bronchial position. Risk warnings can provide early risk alerts and are presented intuitively, reducing operator decision-making time and improving operational accuracy and efficiency.

[0051] In summary, this embodiment of the application, based on multimodal medical imaging data and combined with the virtual anatomy of the AR display device 3, breaks through the blind spot obscured by the lesion, intuitively presenting the target small bronchial segment obscured by the lesion. Through safety constraints and real-time dynamic path planning, a closed loop of preoperative planning and intraoperative navigation is formed, transforming subjective experience into quantifiable technical parameters, reducing reliance on the operator's subjective experience, and improving the operational accuracy and safety of the bronchoscope 2.

[0052] In this embodiment, the multimodal medical imaging data may include 4D-CT images, EBUS images, and enhanced CT images. The construction module 201 may include a first acquisition unit, a second acquisition unit, a third acquisition unit, and a preprocessing unit.

[0053] The first acquisition unit is used to acquire first image data from 4D-CT images. 4D-CT images are dynamic images that add a time dimension to 3D CT, and can capture lung anatomy at different respiratory phases using respiratory gating technology to demonstrate the impact of respiratory motion on lung structure. The first image data is a dataset of the global spatial relationship between bronchi and lesions extracted from 4D-CT images. Therefore, the first image data can include the global spatial relationship between bronchi and lesions. For example, the first image data can include the relative position, distance, and morphological changes of bronchi and lesions at multiple respiratory phases. Recording the positional changes of bronchi and lesions along the respiratory cycle dimension provides raw motion data for subsequent respiratory adaptation in path planning, addressing the motion blind spots of static CT.

[0054] The second acquisition unit is used to acquire second image data through EBUS images. EBUS images are ultrasound images of the airway walls and surrounding tissues acquired through the EBUS probe. Ultrasound waves can penetrate soft tissues, such as lesions, thus solving the problem of insufficient imaging of obscured areas in CT images. The EBUS probe is inserted into the bronchial passage, and a 360° rotating scan is initiated, focusing on scanning the area adjacent to the lesion at the distal end of the segmental bronchus. The unobstructed bronchial wall appears as a three-layered hyperechoic structure, while the target small bronchial segment obscured by the lesion appears as a hyperechoic wall surrounded by a hypoechoic lesion. Therefore, the boundary of the bronchial wall and the lesion tissue can be clearly distinguished. Recording the ultrasound image sequence and the real-time position of the probe allows for the extraction of second image data. The second image data includes the wall characteristics of the target small bronchial segment obscured by the lesion. The wall characteristics refer to the ultrasound imaging features of the small bronchial wall, such as wall thickness, continuity, and distance from the lesion boundary, thereby determining whether the lumen is patent and whether it has been invaded by the lesion. EBUS images can penetrate lesions such as glass nodules and solid tumors, enabling visualization of the walls of obscured target small bronchial segments.

[0055] The third acquisition unit is used to acquire third image data through enhanced CT images. Enhanced CT images are CT images acquired after intravenous injection of contrast agent, which enhances vascular visualization and solves the problem of blurred visualization of small blood vessels in ordinary CT. The scanning range of the third image data can be consistent with that of the first image data. When the CT value reaches a set value, a scan can be initiated to obtain the pulmonary artery image with optimal vascular visualization, thereby obtaining the third image data. The third image data is a dataset of target accompanying blood vessel features extracted by enhanced CT; therefore, the third image data can include the course, diameter, and location of the target accompanying blood vessel.

[0056] The preprocessing unit preprocesses the first, second, and third image data to obtain a spatially aligned multimodal data matrix. Preprocessing involves denoising, registration, and fusion of the original image data to eliminate data errors and improve data quality. For example, Gaussian filtering can be used to remove respiratory motion artifacts from the first image data. Wavelet transform can be used to suppress ultrasound noise in the second image data. Median filtering can be used to remove artifacts caused by contrast agent concentration fluctuations in the third image data.

[0057] Spatial alignment maps image data acquired at different modalities and times to the same three-dimensional coordinate system using anatomical landmarks or sensor coordinates, ensuring the accuracy and reliability of spatial relationships between bronchi, lesions, and related structures. For example, EBUS data and 4D-CT data, as well as enhanced CT data and 4D-CT data, can be registered using an iterative nearest-point algorithm based on bronchoscopic landmarks (BLT) of 4D-CT. Specifically, the bronchial wall features of ultrasound images are mapped to the bronchial coordinate system of 4D-CT based on the coordinates of the positioning sensor of the bronchoscope 2. Alignment with anatomical landmarks such as the pulmonary artery trunk and hilum ensures that the spatial positions of vascular data are consistent with those of bronchi and lesions.

[0058] The multimodal data matrix is ​​a standardized three-dimensional voxel matrix that integrates first, second, and third image data. Each voxel includes grayscale values ​​from 4D-CT to reflect tissue density, echo intensity from EBUS to reflect vessel wall features, and vascular markers from enhanced CT to indicate whether it is a vascular region. The final output is standardized data used for three-dimensional model generation.

[0059] The first acquisition unit provides a global dynamic anatomical framework, the second acquisition unit supplements the details of obscured small bronchial segments, the third acquisition unit clarifies vascular safety constraints, and the preprocessing unit achieves spatial alignment of the data. This approach not only overcomes the inherent limitations of single-modal imaging but also constructs a precise data foundation that is global, local, and secure through multimodal fusion, providing high-quality raw data for subsequent virtual anatomy and pathway planning.

[0060] In this embodiment of the application, the construction module 201 may further include a segmentation unit, a first attention unit, a second attention unit, and a modeling unit.

[0061] The segmentation unit is used to input the multimodal data matrix into the 3D U-Net segmentation network to obtain the initial segmentation results. 3D U-Net is a U-shaped three-dimensional medical image segmentation model that can achieve multi-scale feature fusion through an encoder-decoder architecture. Inputting the preprocessed multimodal data matrix into the 3D U-Net segmentation network outputs a semantic segmentation map of the same size as the multimodal data matrix. Initial segmentation results can be obtained through threshold segmentation. The initial segmentation results can include initial bronchi, lesions, and accompanying vessels. Initial bronchi can include the three-dimensional contours of bronchi of grades 1-5, with portions obscured by lesions marked as initial small bronchial segments. Lesions are complete three-dimensional morphologies and can include the contact boundaries of bronchi. Accompanying vessels are the initial contours of the entire pulmonary vascular network and can include branches accompanying the bronchi.

[0062] The first attention unit is used to input the initial segmentation result into the first attention layer, extract the lesion mask from the initial segmentation result, and perform weighted optimization on the initial segmentation result based on the lesion mask to obtain the bronchial feature map. The first attention layer is a spatial attention matrix based on the lesion location. It strengthens the bronchial features in the lesion-occluded area through weight allocation and suppresses non-bronchial features within the lesion, thus solving the problem that the features of the occluded segment are overwhelmed by the lesion signal in the initial segmentation.

[0063] A lesion mask is a binary matrix that marks the lesion space. A three-dimensional binary mask of the lesion is extracted from the initial segmentation result, where the lesion area is 1 and the non-lesion area is 0. The lesion-affected area mask is obtained through morphological dilation, which can include the lesion and its surrounding 1mm area to reduce the loss of edge features. Attention weight rules can be designed based on the lesion mask. Specifically, in the bronchial feature map, the first weight of the initial small bronchial segments obscured by the lesion is greater than a first predetermined weight; the second weight of the non-bronchial area inside the lesion is less than a second predetermined weight; and the first predetermined weight is greater than the second predetermined weight.

[0064] The bronchial feature map is a set of bronchial features after attention weighting. The first set of weights is a threshold to ensure that the features of obscured segments are not ignored, and the second set of weights is a threshold to ensure that lesion tissue is not misidentified as a bronchus. The difference between the first and second set of weights ensures the effectiveness of the attention mechanism and can be set based on actual needs and experience. For example, for bronchial regions, those located within the lesion's influence area can be assigned a higher weight, with the weight increasing closer to the lesion center to enhance the features of the obscured segments. For non-bronchial regions within the lesion, such as pure lesion regions, a lower weight can be assigned to suppress non-target features. Bronchial features in non-lesion regions can be assigned a moderate weight to maintain the initial features. Multiplying the initial segmentation result by the attention weight matrix voxel-by-voxel yields the bronchial feature map. The features of the initial small bronchial segments obscured by the lesion are significantly enhanced, while noise features within the lesion are suppressed, thereby improving the continuity and clarity of obscured segments in the bronchial feature map.

[0065] The second attention unit inputs the bronchial feature map into the second attention layer, compares the first orientation of the initial small bronchial segment with the second orientation of the target accompanying blood vessel, and corrects the first orientation based on anatomical patterns to obtain the target small bronchial segment. The second attention layer is a relational attention mechanism based on the anatomical patterns of the accompanying bronchi and blood vessels, which can correct the orientation of the obscured bronchus by associating the blood vessel orientation. The first orientation refers to the three-dimensional spatial trajectory features of the initial small bronchial segment, and the second orientation is the three-dimensional spatial trajectory features of the target accompanying blood vessel. The three-dimensional spatial trajectory features can include curvature, direction, bifurcation, etc.

[0066] The first orientation can be extracted from bronchial feature maps, and the second orientation can be extracted from enhanced CT scans. Anatomical patterns, based on the inherent spatial relationship between bronchi and blood vessels summarized from extensive clinical data, serve as the basis for orientation correction, ensuring the correction results conform to anatomical characteristics. In one example, a pre-defined library of bronchial and vascular anatomical patterns can be used to compare the spatial accompaniment, bifurcation correspondence, and diameter correlation between the first and second orientations. When the matching degree between the first and second orientations is lower than the set matching degree, an error is determined in the initial orientation, and the first orientation can be corrected. For example, the curvature of the initial orientation can be fine-tuned using the second orientation as a reference. Another example is the correction of bronchial bifurcation points based on bifurcation correspondence. This ensures that the final output matching degree between the target small bronchial segment and the target accompanying blood vessel meets the set matching degree, for example, 90%. This solves the problem of orientation errors in obscured segments.

[0067] The modeling unit is used to spatially align the target small bronchial segment with the lesion and the target accompanying blood vessel to form a 3D model. The 3D model is a digital 3D structure integrating the target small bronchial segment, lesion, and target accompanying blood vessel. Spatial alignment is achieved by adjusting the 3D coordinates of different structures to a unified coordinate system using anatomical landmarks such as bronchial openings, lesion boundaries, or vascular bifurcation points, ensuring that the relative positions of the bronchi, lesions, and blood vessels are consistent with the patient's actual anatomy. For example, using the 3D coordinates of the target small bronchial segment as a reference, the contact boundary between the lesion and the target bronchial segment, the distance between the target accompanying blood vessel and the target small bronchial segment, and the connection point with the opening node of the unobstructed segment of the trachea are determined.

[0068] Then, the model details can be optimized. For example, the walls of the target small bronchial tubes can be smoothed, semi-transparent textures can be added to the lesions, and accompanying blood vessels can be color-coded, such as highlighted in red, to distinguish them from the bronchial tubes and lesions. Bronchial tubes can be marked in blue, and lesions in gray, thus achieving visual differentiation. Finally, the target small bronchial segment, lesion, accompanying blood vessels, and unobstructed bronchial segments are integrated into a complete 3D model and output in a format compatible with AR display devices. The 3D model can include the 3D coordinates, morphological parameters, and semantic labels of each structure.

[0069] By enhancing the features of the occluded segment with dual attention units and correcting the orientation based on anatomical rules, spatial fusion of multiple structures can be achieved, ultimately solving the problems of inaccurate segmentation and unreliable orientation of the target small bronchial segment obscured by lesions, and generating a more accurate three-dimensional model.

[0070] In this embodiment, the target operation instructions may include sectional operation instructions and magnification operation instructions. Sectional operation instructions refer to instructions for performing virtual sectional operations, such as selecting a sectional plane and adjusting transparency, and are the core operations for non-invasive observation of the obscured segment. Magnification operation instructions refer to instructions for performing magnification operations on the target small bronchial segment to solve the problem of difficulty in distinguishing it with the naked eye. The virtual anatomy module 202 may include a marking unit, a sectional unit, a highlighting unit, and a magnification unit.

[0071] The marking unit is used to add a first mark to the target small bronchial segment and a second mark to the target accompanying blood vessel in the 3D model, and to display the sectional parameters for the lesion. The first and second marks are visual identifiers used to distinguish the target small bronchial segment from the target accompanying blood vessel. They can be quickly identified through color and morphological differences, resolving confusion caused by overlapping structures in the 3D model. For example, the target small bronchial segment can be marked with a blue dashed line, the lumen centerline with a blue solid line, and the opening node highlighted with a blue fluorescent dot. The target accompanying blood vessel can be marked with a red dashed box, the vessel centerline with a red solid line, and the closest point to the bronchus with a red warning dot. The preset sectional parameters can be displayed on the AR display device 3's interface. These sectional parameters are quantitative indicators describing virtual anatomical operations, such as transparency and sectional planes, providing operational guidance for the operator. This reduces the operator's recognition time and improves decision-making efficiency. Furthermore, visual differentiation allows for intuitive presentation of the target, reducing the likelihood of operation deviating from the target area.

[0072] The sectioning unit responds to received sectioning operation commands by parsing the commands, determining the command type, and then performing virtual dissection on the 3D model based on the parsed commands, while simultaneously acquiring the lesion's transparency in real time. For example, if the command is a plane selection command, a corresponding sectioning plane is generated in the 3D model, displaying the structures on both sides of the plane in real time. If the command is a transparency adjustment command, the lesion's transparency is dynamically adjusted in increments, with the model rendering updated after each adjustment. The model rendering engine acquires the lesion's transparency in real time and feeds it back to the parameter display area of ​​the AR display device 3. This allows for penetrating observation without real-time cutting of the lesion, reducing irreversible damage to the model and supporting repeated adjustments.

[0073] The highlighting unit is used to switch the color of the first marker from the initial color to the target color if the detected lesion transparency reaches the set transparency level. The set transparency is the optimal observation threshold verified in clinical trials. When the set transparency is reached, the target small bronchial segment that was obscured by the lesion is clearly visible, and the lesion outline can still be used as a location reference.

[0074] Specifically, the system compares the current transparency of the lesion with the set transparency in real time. When the current transparency reaches the set transparency, a highlighting mechanism is triggered to switch the color of the first marker from the initial color to the target color. The initial color and the target color represent two visual states. The initial color is used for normal display, while the target color is used for enhanced cues. Therefore, the brightness of the target color is greater than that of the initial color, and this difference in brightness conveys a signal that the operation has met the requirements. In this way, the operation status can be intuitively fed back, enhancing the recognition of the target structure.

[0075] The magnification unit responds to received magnification operation commands by magnifying the target small bronchial segment by a set factor to obtain a corresponding anatomical model in three dimensions. The set factor is determined based on the limits of human visual perception and AR display resolution. Parsing the magnification operation command yields the magnification center and the set factor. Using the magnification center as the origin, the target small bronchial segment is magnified locally in three dimensions. The edges of the magnified area can be marked with a semi-transparent gray frame, while maintaining the spatial proportions before and after magnification to reduce distortion. The magnified 3D model retains the details of the bronchial wall, the boundary with the lesion, and the distance to accompanying blood vessels, and is output as an anatomical model. The anatomical model is a 3D model obtained after virtual dissection, highlighting the local details of the target small bronchial segment while preserving the global anatomical relationships.

[0076] In this embodiment, the virtual anatomy module 202 may further include a database creation unit, a motion sensing unit, a matching unit, and a determination unit.

[0077] The library building unit is used to construct a mapping library that includes the mapping relationship between action features and operation commands, so as to transform the operator's natural actions into system-executable commands. Action features are quantitative parameters describing gestures or visual actions, such as trajectory coordinates, speed, and eye focus, which are the basis for distinguishing different operation intentions. For example, operation commands can include slicing operation commands and magnification operation commands. Action features can include gesture features, such as sliding two fingers up and down corresponding to transparency adjustment, and pinching or opening two fingers corresponding to magnification or shrinking. Gesture features can be determined based on the finger movement trajectory, angle, and speed. Action features can also include visual features, such as looking at a target area and nodding, which can correspond to magnification. Visual features can be determined based on eye focus, head posture, etc.

[0078] The motion sensing unit is used to collect the operator's gesture information and / or visual information through multimodal sensors to obtain sensor data. A multimodal sensor refers to a hardware combination that can simultaneously collect signals from multiple modalities such as gestures and vision. For example, the AR display device 3 can integrate an infrared depth camera, eye tracker, and voice sensor, which can capture operational intentions more comprehensively than a single sensor. The sensor data consists of pre-processed gesture and visual information. Pre-processing may include noise reduction and timestamp synchronization to reduce data interference.

[0079] The matching unit extracts features from sensor data to obtain target action features, and then matches these features with a mapping library. Target action features are feature vectors extracted from sensor data that characterize the intended action. By extracting features from sensor data, target action features can be obtained. Then, the target action features are matched with the mapping library based on similarity; for example, the Euclidean distance can be calculated using the K-nearest neighbor algorithm. The results are then sorted by confidence level to obtain the final matching result.

[0080] The determination unit identifies operation instructions in the mapping library that match the target action features as target operation instructions. These target operation instructions are system-executable virtual dissection instructions determined after matching verification, representing the final expression of the operation intent. For example, instructions in the matching structure with a confidence level higher than a set threshold can be filtered, and their validity verified individually. Then, semantic parsing is performed on valid instructions to distinguish different types, such as dissection and magnification instructions, and their types and parameters are labeled accordingly. Finally, the labeled target operation instructions are distributed to the corresponding execution units, triggering the corresponding dissection operation. This reduces the possibility of erroneous operations and improves the accuracy of the instructions.

[0081] In this embodiment, the path generation module 203 may include a constraint unit, a starting unit, a search unit, and a filtering unit.

[0082] The constraint unit generates preset safety constraints based on spatial safety constraints, morphological adaptation constraints, and lumen fit constraints. Spatial safety constraints ensure a safe distance between the path and blood vessels or lesions, reducing the probability of accidents such as blood vessel rupture or lesion penetration caused by direct instrument contact with risk sources. Morphological adaptation constraints ensure the path conforms to the physical limits of bronchoscopy and the physiological curvature of the bronchi, ensuring the path can be actually executed by the instrument. Lumen fit constraints ensure the path follows the natural lumen of the bronchus, reducing the probability of the path detaching from the lumen or penetrating lung tissue. The preset safety constraints, a combination of spatial safety constraints, morphological adaptation constraints, and lumen fit constraints, serve as the criteria for judging the safety and feasibility of the path.

[0083] In one example, in an embodiment of this application, the constraint unit may be specifically used to perform the following steps.

[0084] The first distance range between the initial path and the accompanying blood vessel, and the second distance range between the initial path and the lesion boundary, are used as spatial safety constraints. The first distance range is the minimum safe distance between the initial path and the accompanying blood vessel, used to reduce the risk of vascular injury. The second distance range is the minimum safe distance between the initial path and the lesion boundary, used to reduce the possibility of instrument penetration into the lesion. For example, the first and second distance ranges can be set based on third image data obtained from enhanced CT images, combined with lesion characteristics, to form spatial safety constraints between the path, the target accompanying blood vessel, and the lesion boundary.

[0085] The curvature range of the initial path is determined based on the bending angle range of the target small bronchial segment, and this curvature range serves as a morphological adaptation constraint. The bending angle range is the complete angular range of the target small bronchial segment itself, used to ensure that the path does not exceed the natural tolerance range of the bronchus. The curvature range of the initial path is a curvature index that converts the bending angles into path curvature indicators, correlated with the physical maneuverability of the bronchoscope 2, and used to quantify path feasibility. For example, the natural bending angle of the target small bronchial segment can be measured from a bronchial feature map, which may include the maximum bending angle of a single segment and the minimum bending angle at the bifurcation. Then, the bending angles are converted into the curvature range of the initial path to obtain the morphological adaptation constraint.

[0086] The centerline deviation range of the initial path is determined based on the centerline of the target small bronchial segment, and this deviation range is used as a constraint for luminal fit. The luminal centerline is the geometric center trajectory of the target small bronchial segment's lumen, serving as the baseline for path planning to ensure the path follows the lumen rather than penetrating lung tissue. The centerline deviation range is the preset maximum allowable deviation between the path centerline and the luminal centerline, ensuring the path does not deviate from the lumen. For example, the luminal centerline of the target small bronchial segment can be extracted from a 3D model to obtain a 3D coordinate sequence. Then, a deviation threshold is set based on the luminal diameter to output the luminal fit constraint.

[0087] Finally, based on spatial safety constraints, morphological adaptation constraints, and luminal fit constraints, preset safety constraints are generated. Each constraint includes constraint type, threshold range, violation judgment criteria, and priority. All constraints are formulated based on the actual anatomical features, accompanying vascular parameters, and lesion characteristics of the target small bronchial segment, reducing the problem of general constraints not fitting the small bronchial structure. Transforming fuzzy experience into concrete numerical values ​​allows path search to determine constraint satisfaction through numerical calculations, reducing subjective bias and improving path safety, accuracy, and feasibility.

[0088] The starting unit is used to generate the initial segment path by taking the bronchial inlet node as the starting point and the target small bronchial segment's opening node as the ending point, following the natural lumen of the bronchus in the 3D model. The natural lumen's orientation is the bronchus's own physiological trajectory, formed by the support of the bronchial wall, ensuring that the path conforms to anatomical rules. Specifically, the bronchial inlet node and the target small bronchial segment's opening node can be extracted from the 3D model. Then, along the bronchial lumen centerline from the inlet node to the opening node, the 3D trajectory is extracted, such as by sampling at 0.5mm intervals, to obtain the coordinates, orientation vector, and bifurcation position of each node. Based on the natural lumen's orientation, a smooth path is generated by B-spline curve fitting, resulting in the initial segment path, and the 3D coordinate sequence and key parameters of the initial segment path are output. This not only ensures the safety of the initial segment but also conforms to clinical operating habits, reducing the adaptation cost to navigation.

[0089] The search unit is used to search for candidate paths in the anatomical model, starting from the opening node of the target small bronchial segment and ending at the target operating area. The improved A* algorithm can include a penalty term for distance from accompanying blood vessels and a curvature penalty term.

[0090] The improved A* algorithm is a path search algorithm that optimizes the cost function based on the traditional A* algorithm. By introducing vessel distance penalty terms and curvature penalty terms, it is better suited to the safety and operational requirements of small bronchi. The cost function is a quantitative indicator that measures the quality of a path, comprehensively considering length, vessel distance, and curvature to ensure a balance between efficiency, safety, and operational feasibility. For example, total cost = path length + vessel distance penalty term + curvature penalty term. When the distance between the path and the target accompanying vessel is less than a set distance, such as 1.5mm, the vessel distance penalty term = 5 × (1.5 - d), where the smaller d is, the greater the penalty. When the path curvature is greater than a set curvature, such as 20°, the curvature penalty term = 2 × (c - 20), where the larger c is, the greater the penalty.

[0091] An improved A* algorithm is initiated, starting from the opening node of the target small bronchial segment and ending at the target operating area, to search for multiple candidate paths within the anatomical model. These candidate paths are potential paths obtained through the improved A* algorithm, and can include different directions, lengths, and costs, providing a selection space for subsequent filtering. For example, in the 3D anatomical model, the search expands by 8 neighborhood grids each time, selecting the node with the lowest total cost until the target operating area is reached. The search stops after generating a predetermined number of candidate paths to reduce local optima. Each candidate path can contain a complete coordinate sequence and cost parameters. This reduces the possibility of local optima where one path is shortest but closer to a blood vessel, while another path is slightly longer but safer.

[0092] The filtering unit selects paths from candidate paths that meet preset safety constraints to obtain the initial path. The initial path is the optimal path selected from all candidate paths that satisfy the preset safety constraints, serving as the baseline path for preoperative planning. In one example, the searched candidate paths can be checked item by item according to the preset safety constraints, retaining those that satisfy all preset safety constraints as valid paths. If there are 0 valid paths, the results are fed back to the search unit, which readjusts the parameters of the cost function to search for candidate paths. Valid paths are scored, for example, by weighted summation based on the weight ratio of each preset safety constraint, and then the path with the highest score is selected as the initial path. The weight ratio can be set based on the actual scenario. This balances the length, safety, and ease of operation of the initial path, improving the clinical execution success rate and providing a baseline path for subsequent dynamic planning.

[0093] In this embodiment, the dynamic programming module 204 may include an acquisition unit, a deviation calculation unit, a path update unit, and a risk warning unit.

[0094] The acquisition unit collects the patient's real-time respiratory signals and, combined with multimodal medical image data of multiple respiratory phases, identifies the patient's real-time respiratory phase. In one example, a piezoelectric sensor attached to the chest wall can be used to collect the patient's respiratory motion signals, simultaneously recording the peak, trough, and intermediate nodes of the respiratory waveform to generate real-time respiratory data. Then, it calls upon pre-stored multimodal image data of five respiratory phases from a 4D-CT image to extract feature points and the relative positions of target small bronchi in each phase. A dynamic time warping algorithm is then used to match the waveform features of the real-time respiratory signal with the five phase templates of the 4D-CT, calculate the similarity, and output the current patient's real-time respiratory phase, thereby achieving accurate tracking of the respiratory state. The dynamic time warping algorithm has low latency; low recognition latency ensures timely data synchronization and allows sufficient time for real-time path updates.

[0095] The deviation calculation unit is used to calculate the three-dimensional spatial deviation between the initial path and the target small bronchial segment under the real-time respiratory phase, based on the displacement model of the respiratory phase and the target small bronchial segment. The displacement model of the respiratory phase and the target small bronchial segment is a mathematical model describing the three-dimensional displacement of the target small bronchial segment under the respiratory phase, converting respiratory motion into calculable spatial displacement parameters. This displacement model can be constructed based on five phases of 4D-CT image data, recording the three-dimensional displacement of the target small bronchial segment under different phases, and obtaining the displacement function for continuous phases through polynomial fitting. Substituting the real-time respiratory phase into the displacement model, the three-dimensional displacement vector of the target small bronchial segment under the current phase is output. Then, the three-dimensional spatial deviation between the initial path and the lumen centerline of the current target small bronchial segment is calculated. By accurately quantifying respiratory displacement, prediction accuracy can be improved, and the numericalization of deviation provides operators with intuitive data references, improving decision-making efficiency.

[0096] The path update unit corrects the initial path to obtain the target path when the 3D spatial deviation exceeds a set deviation. The set deviation is a threshold for determining whether a path update is necessary; it can be set based on clinical operational precision and lumen diameter to ensure that the deviation is within an acceptable safety range and is not frequently adjusted. When the 3D spatial deviation exceeds this set deviation, the path correction process is triggered. For example, the initial path can be specifically corrected based on the real-time displacement vector to output the target path. The target path is a corrected path that precisely matches the target small bronchial segment at the current respiratory phase, and may include a real-time coordinate sequence, the current deviation value, and safety parameters.

[0097] The risk alert unit overlays risk heatmaps, dynamic parameters, and operational suggestions onto the display interface of AR display device 3. The risk heatmap is a visualization tool that uses color to visually represent the risks around the path, converting distance values ​​into visual signals and alleviating the burden of interpreting abstract data. For example, high risk is displayed in red, medium risk in yellow, and low risk in green. The risk heatmap dynamically updates as the path changes. Dynamic parameters are quantitative indicators reflecting real-time respiratory status, path deviation, and safe distance, providing comprehensive data information. For example, respiratory phase, three-dimensional spatial deviation, safe distance, and path curvature can be displayed in the sidebar of the AR display device 3's interface. Operational suggestions are clinical operational guidelines generated based on risk constraints and dynamic parameters, providing referable action instructions and reducing decision-making difficulty. Real-time display of operational prompts and dynamic data parameters through the AR display device 3's interface improves operational safety and decision-making efficiency.

[0098] Figure 6 This is a schematic diagram illustrating an application scenario of the bronchoscope path planning device according to another embodiment of this application. For example... Figure 6As shown, in another embodiment of this application, the path planning device 200 of the bronchoscope 2 can also communicate with the collaborative system 4, and the collaborative system 4 communicates with multiple terminals 5. The collaborative system 4 is a data processing platform that supports real-time collaboration among multiple disciplines, and connects the path planning device 200 of the bronchoscope 2 to terminals of multiple departments, such as respiratory medicine, thoracic surgery, and pathology, to break down information barriers between departments and achieve multi-terminal data synchronization.

[0099] Figure 7 This is a schematic diagram of the path planning device for a bronchoscope 2 provided in another embodiment of this application. Figure 7 As shown, the path planning device 200 for the bronchoscopy 2 in this embodiment may further include a collaborative planning module 205. The collaborative planning module 205 adjusts the target path based on decision information from multiple terminals 5 to obtain a collaboratively planned path. Specifically, data from the AR display interface is pushed to the collaborative system 4 in real time via a synchronization mechanism, and then distributed by the collaborative system 4 to multiple terminals 5, ensuring consistency in anatomical information and path status among the participating terminals, providing a unified visual benchmark for multidisciplinary discussions. Then, decision information from each terminal based on its professional scenario is received, and data processing is performed based on this decision information to obtain the collaboratively planned path. The collaboratively planned path is the optimal path that integrates multidisciplinary opinions and satisfies all safety and operational constraints. In this way, it ensures that the bronchoscopy 2 successfully reaches the target area while also considering the safety and effectiveness of diagnosis and treatment across multiple disciplines.

[0100] In this embodiment, the collaborative planning module 205 may include a synchronization unit, a collaboration unit, an adjustment unit, and a prompting unit.

[0101] The synchronization unit is used to synchronize the display interface of the AR display device 3 to the collaborative system 4. For example, a real-time communication link between the bronchoscope 2's path planning device and the collaborative system 4 can be established through an encrypted transmission protocol to verify the identities of both parties and ensure data transmission security. Then, the content of the AR display device 3's display interface that needs to be synchronized to the collaborative system 4 is defined. When the AR display device 3's display interface is updated, instant synchronization is triggered, achieving bidirectional synchronization and breaking the spatial limitations of the operation process.

[0102] The collaborative unit acquires the first annotation information of the bronchoscope insertion path from the respiratory terminal, the second annotation information of the incision avoidance area from the thoracic surgery terminal, and the third annotation information of the operation range of the target operation area from the pathology terminal, and synchronizes the first, second, and third annotation information to the display interface of the AR display device 3. The first annotation information is the respiratory terminal's optimization suggestion for the insertion path based on the bronchoscopy 2 operation, focusing on the operability of the path, such as reducing friction and jamming. The second annotation information is the area to be avoided marked by the thoracic surgeon from a surgical perspective, focusing on the surgical safety of the path, such as avoiding pleural damage and reducing postoperative complications. The third annotation information is the pathologist's definition of the target operation area (such as biopsy or ablation), focusing on the pathological effectiveness of the operation, such as ensuring that the sample covers the core of the lesion. In one example, the annotation information can be in a unified data format, including annotation type, three-dimensional coordinate range, department identifier, marker identifier, annotation time, and priority. Then, the parsed annotation information is overlaid onto the corresponding position on the display interface of AR display device 3. For example, a blue arrow is associated with the target path, and text annotations float next to the arrow. Red annotation areas are semi-transparently overlaid on the corresponding areas to indicate risks. In this way, multidisciplinary perspectives can be integrated to improve decision-making efficiency, while also retaining traceability information to provide a data source for subsequent traceability decisions.

[0103] The adjustment unit performs conflict checking based on the first, second, and third annotation information, as well as preset safety constraints. Conflict checking is the process of verifying whether there are contradictions between multidisciplinary annotation constraints and preset safety constraints to ensure that multiple objectives can be achieved simultaneously. For example, it involves traversing inflection points, avoiding restricted areas, and meeting safety distance requirements. If the conflict check passes, the target path is adjusted based on the first, second, and third annotation information to obtain a collaboratively planned path. The collaboratively planned path is the final path that integrates multidisciplinary annotation constraints and satisfies all safety rules. For example, the inflection point position can be adjusted according to the first annotation information to refit the path curve. The path can be translated overall based on the second annotation information. The path endpoint can be fine-tuned based on the highlighted area center of the third annotation information. The final output collaboratively planned path can include a comparison of parameters before and after adjustment.

[0104] The prompting unit is used to display a conflict prompt box on the AR display device 3's interface when there are conflicts between the first, second, and third annotation information and preset safety constraints, and provides adjustment suggestions for the target path. The conflict prompt box is an interactive interface element that intuitively displays the conflict content, image, and source, transforming abstract constraint contradictions into understandable problem descriptions. The adjustment suggestions provide specific solutions for the conflict scenario, and can provide decision-making references for multidisciplinary writing based on the system's analysis of constraint weights and spatial relationships. Specifically, when a conflict occurs, the specific content and spatial location of the conflict can be located. Then, multiple feasible adjustment suggestions are generated based on the conflict scenario, along with parameter descriptions. This reduces the difficulty of decision-making, improves the efficiency of conflict resolution, and ensures the safety of the decision.

[0105] This application's embodiments can improve the accuracy of preoperative identification of small bronchi obscured by lesions, reducing endoscopy failures due to misjudgment of their path. Furthermore, combined with the AR display device 3, it can improve the adaptation of dynamic pathways to breathing, significantly reducing errors compared to static pathways and lowering the risk of accidental vascular contact. In addition, multidisciplinary AR collaboration can significantly shorten decision-making time, reducing the total time for preoperative planning and intraoperative decision-making, allowing novice operators to quickly grasp the anatomy and pathway logic of small bronchi through the AR display device 3.

[0106] The above examples illustrate this application only to aid understanding and are not intended to limit its scope. Those skilled in the art to which this application pertains can make various simple deductions, modifications, or substitutions based on the ideas presented.

Claims

1. A bronchoscope path planning device, characterized in that, include: A construction module is used to collect multimodal medical image data of multiple respiratory phases of patients and generate a three-dimensional model. The three-dimensional model is loaded onto an AR display device. The three-dimensional model includes lesions, bronchi and accompanying blood vessels matching the bronchi. The bronchi include target small bronchial segments that are obscured by the lesions. The accompanying blood vessels include target accompanying blood vessels that match the target small bronchial segments. The virtual anatomy module is used to perform AR virtual anatomy on the three-dimensional model based on the target operation command until the transparency of the lesion reaches the set transparency, thereby obtaining the anatomical model corresponding to the three-dimensional model. The path generation module is used to take the path between the inlet node of the bronchus and the opening node of the target small bronchial segment as the starting segment path and the target operating area as the end point, and generate an initial path in the anatomical model based on preset safety constraints. The dynamic planning module is used to dynamically update the initial path by combining real-time respiratory phases to obtain the target path, and to provide risk warnings for the target path on the display interface of the AR display device. The building module includes: A segmentation unit is used to input a multimodal data matrix into a 3D U-Net segmentation network to obtain an initial segmentation result. The multimodal data matrix is ​​obtained based on the multimodal medical image data. The initial segmentation result includes an initial bronchus, the lesion, and the accompanying blood vessel. The initial bronchus includes an initial small bronchial segment that is obscured by the lesion. The first attention unit is used to input the initial segmentation result into the first attention layer, extract the lesion mask in the initial segmentation result, and perform weighted optimization on the initial segmentation result based on the lesion mask to obtain a bronchial feature map. In the bronchial feature map, the first weight of the initial small bronchial segment that is covered by the lesion is greater than the first set weight, the second weight of the non-bronchial region inside the lesion is less than the second set weight, and the first set weight is greater than the second set weight. The second attention unit is used to input the bronchial feature map into the second attention layer, compare the first orientation of the initial small bronchial segment with the second orientation of the target accompanying blood vessel, and correct the first orientation based on anatomical rules to obtain the target small bronchial segment. The modeling unit is used to spatially align the target small bronchial segment with the lesion and the target accompanying blood vessel to form the three-dimensional model.

2. The path planning device according to claim 1, characterized in that, The multimodal medical imaging data includes 4D-CT images, EBUS images, and enhanced CT images. The construction module includes: The first acquisition unit is used to acquire first image data through 4D-CT images, the first image data including the global spatial relationship between the bronchus and the lesion; The second acquisition unit is used to acquire second image data through EBUS images. The second image data includes the wall features of the target small bronchial segment obscured by the lesion. The third acquisition unit is used to acquire third image data through enhanced CT images. The third image data includes the course, diameter and location of the target accompanying blood vessel. The preprocessing unit is used to preprocess the first image data, the second image data, and the third image data to obtain a spatially aligned multimodal data matrix.

3. The path planning device according to claim 1, characterized in that, The target operation commands include dissection operation commands and magnification operation commands, and the virtual dissection module includes: A marking unit is used to add a first mark to the target small bronchial segment, add a second mark to the target accompanying blood vessel in the three-dimensional model, and display the cutting parameters for the lesion; The slicing unit is used to perform virtual dissection on the three-dimensional model in response to receiving a slicing operation command, and to obtain the transparency of the lesion in real time; A highlighting unit is used to switch the color of the first marker from the initial color to the target color if the transparency of the lesion is detected to reach the set transparency, wherein the brightness of the target color is greater than that of the initial color; The magnification unit is used to respond to a magnification operation command received to magnify the target small bronchial segment by a set factor to obtain the anatomical model corresponding to the three-dimensional model.

4. The path planning device according to claim 3, characterized in that, The virtual anatomy module also includes: The library building unit is used to construct a mapping library that includes the mapping relationship between action features and operation instructions; The motion sensing unit is used to collect the operator's gesture information and / or visual information through a multimodal sensor to obtain sensor data; A matching unit is used to extract features from the sensor data to obtain target action features, and to match the target action features with the mapping library. The determining unit is used to determine the operation instruction in the mapping library that matches the target action feature as the target operation instruction.

5. The path planning device according to claim 1, characterized in that, The path generation module includes: The constraint unit is used to generate preset safety constraints based on spatial safety constraints, morphological adaptation constraints, and lumen fitting constraints. The starting unit is used to generate a starting segment path by taking the inlet node of the bronchus as the starting point and the opening node of the target small bronchial segment as the ending point, along the natural lumen of the bronchus in the three-dimensional model. The search unit is used to search for candidate paths in the anatomical model with the opening node of the target small bronchial segment as the starting point and the target operating area as the ending point. The cost function of the improved A* algorithm includes a distance penalty term with the accompanying blood vessel and a curvature penalty term. The filtering unit is used to filter paths that meet the preset security constraints from the candidate paths to obtain the initial path.

6. The path planning device according to claim 5, characterized in that, The constraint unit is also used for: The first distance range between the initial path and the accompanying blood vessel and the second distance range between the initial path and the boundary of the lesion are used as spatial safety constraints; The curvature range of the initial path is determined based on the bending angle range of the target small bronchial segment, and the curvature range is used as a morphological adaptation constraint. The centerline deviation range of the initial path is determined based on the lumen centerline of the target small bronchial segment, and the centerline deviation range is used as the lumen fitting constraint. Based on the spatial safety constraints, the morphological adaptation constraints, and the lumen fitting constraints, the preset safety constraints are generated.

7. The path planning device according to claim 1, characterized in that, The dynamic programming module includes: The acquisition unit is used to acquire the patient's real-time respiratory signal and, in combination with the multimodal medical image data of multiple respiratory phases, identify the patient's real-time respiratory phase. The deviation calculation unit is used to calculate the three-dimensional spatial deviation between the initial path and the target small bronchial segment under the real-time respiratory phase based on the displacement model of the respiratory phase and the target small bronchial segment. The path update unit is used to correct the initial path to obtain the target path when the three-dimensional spatial deviation is greater than a set deviation. The risk warning unit is used to overlay a risk heat map, dynamic parameters, and operation suggestions on the display interface of the AR display device.

8. The path planning device according to claim 1, characterized in that, The route planning device also communicates with a collaborative system, which communicates with multiple terminals. The route planning device further includes: The collaborative planning module is used to adjust the target path based on the decision information of multiple terminals to obtain a collaboratively planned path.

9. The path planning device according to claim 8, characterized in that, The collaborative planning module includes: The synchronization unit is used to synchronize the display interface of the AR display device to the collaborative system; The collaborative unit is used to acquire the first annotation information of the respiratory terminal for the endoscope path, the second annotation information of the thoracic surgery terminal for the incision avoidance area, and the third annotation information of the pathology terminal for the operation range of the target operation area, and synchronize the first annotation information, the second annotation information and the third annotation information to the display interface of the AR display device; The adjustment unit is used to perform conflict verification based on the first annotation information, the second annotation information, the third annotation information and the preset security constraints. If the conflict verification passes, the target path is adjusted based on the first annotation information, the second annotation information and the third annotation information to obtain the collaborative planning path. The prompting unit is used to pop up a conflict prompt box on the display interface of the AR display device and provide adjustment suggestions for the target path when there is a conflict between the first annotation information, the second annotation information, the third annotation information and the preset safety constraint.

Images