Real-time airway digital twinning system based on bronchofiberscope video and use method of real-time airway digital twinning system

Abstract

The invention discloses a bronchofiberscope video-based real-time airway digital twinning system and a use method thereof, and the system comprises a bronchofiberscope which is internally provided with a CMOS image sensor and a motion sensing unit so as to collect airway condition data; the work station is electrically connected with the fiber bronchoscope so as to process the collected airway condition data and construct a three-dimensional airway model; the structured light scanner is electrically connected with the work station so as to carry out registration on the scanned head and neck body surface data and the three-dimensional airway model; the motion monitoring unit is electrically connected with the work station and used for monitoring respiratory motion and providing respiratory phase compensation for the three-dimensional airway model; and the information interaction unit is in communication connection with the work station and is used for capturing dynamic real views of the head, the neck and the airway and carrying out dynamic superposition registration on the three-dimensional airway model so as to obtain an airway digital twinborn model in the work station. The problems of three-dimensional visualization, dynamic modeling, real-time navigation and poor risk assessment in difficult airway management can be solved.

Description

Technical Field

[0001] The present invention relates to the intersection of biomedical engineering, medical image processing and augmented reality technology, and in particular to a real-time airway digital twin system based on bronchoscopic video and a method for using the system. Background Art

[0002] In modern medicine, especially in anesthesiology and respiratory medicine, the accuracy of airway assessment and intubation navigation directly impacts surgical safety and rescue success rates. In recent years, with the advancement of technologies such as computer vision, augmented reality (AR), and artificial intelligence (AI), virtual airway reconstruction and visual navigation systems have become a research hotspot. However, in practical applications, several key technical bottlenecks remain that require breakthroughs.

[0003] First, the current mainstream virtual airway three-dimensional reconstruction methods generally rely on thin-slice CT (computed tomography) images as a data source. This imaging method not only exposes patients to higher radiation doses, but also has a long data acquisition and processing cycle, making it difficult to meet the immediate diagnosis and treatment needs in emergency or resource-constrained environments. In addition, static CT images cannot reflect dynamic changes such as soft tissue collapse, secretion obstruction, or glottic edema that may occur during surgery, resulting in significant differences between preoperative modeling results and the actual airway status, thereby increasing the risk of intubation failure or mucosal damage.

[0004] Secondly, the technology for real-time 3D reconstruction of monocular fiberoptic bronchoscopic video streams is still immature. Traditional SLAM (Simultaneous Localization and Mapping) algorithms based on feature point matching are prone to feature point loss in environments with uniform airway wall texture and uneven lighting, resulting in positioning drift and model fragmentation. While deep learning-based methods improve reconstruction accuracy, their high computational complexity makes it difficult to achieve high frame rates on bedside devices, limiting their practicality in scenarios where modeling is performed while the airway is inspected.

[0005] Third, most existing AR navigation systems use a one-time registration strategy, ignoring micro-neck movements and airway deformation caused by chest movement. When the patient takes a deep breath or changes position, the virtual model and the actual anatomy can easily become misaligned, causing the navigation path to shift, compromising the accuracy and safety of intubation. Furthermore, the optical overlay effect of the AR system is often disrupted by factors such as strong light from surgical lights and reflections from wet laryngeal surfaces, further reducing the system's stability and reliability.

[0006] Fourth, most current 3D visualization platforms only provide image display capabilities and lack the ability to automatically identify and quantify key anatomical parameters. For example, indicators such as the narrowest cross-sectional area, bend radius, and intubation path collision probability still rely on the physician's experience and judgment. The lack of objective, verifiable data support hinders the standardization of clinical decision-making and restricts the development of teaching, training, and quality control systems. Furthermore, existing teaching and quality control platforms are mostly limited to post-recording playback and lack linkage with 3D reconstruction data, making it difficult to form an effective feedback mechanism and a closed-loop AI model training system.

[0007] In summary, existing technologies still have obvious shortcomings in dealing with three-dimensional visualization, dynamic modeling, real-time navigation and risk assessment in difficult airway management. There is an urgent need for a new generation of intubation assistance system that can achieve high-quality three-dimensional airway reconstruction without CT conditions and combines lightweight algorithms, dynamic alignment mechanisms and intelligent risk assessment to improve the safety, efficiency and traceability of clinical operations.

[0008] The statements herein merely provide background information related to the present invention and do not necessarily constitute prior art. Summary of the Invention

[0009] The purpose of the present invention is to provide a real-time airway digital twin system based on bronchoscopic video and its use method, which can solve the problems of poor three-dimensional visualization, dynamic modeling, real-time navigation and risk assessment capabilities in difficult airway management.

[0010] To achieve the above objectives, the present invention provides a real-time airway digital twin system based on bronchoscopic video and a method for using the same, comprising:

[0011] A fiberoptic bronchoscope with a built-in CMOS image sensor and motion sensing unit to collect data on the patient's airway condition;

[0012] a workstation comprising a central processing unit, the workstation being electrically connected to the fiberoptic bronchoscope to process the collected airway condition data and construct a three-dimensional airway model;

[0013] a structured light scanner electrically connected to the workstation to register scanned surface data of the patient's head and neck with the three-dimensional airway model;

[0014] a motion monitoring unit, electrically connected to the workstation, for monitoring the patient's respiratory motion and providing respiratory phase compensation for the three-dimensional airway model;

[0015] The information interaction unit is communicatively connected to the workstation and is used to capture dynamic real views of the head, neck and airway, and dynamically overlay and register the three-dimensional airway model to obtain a digital twin model of the airway in the workstation.

[0016] Optionally, the CMOS image sensor is electrically connected to a workstation, and the CMOS image sensor transmits the collected airway video image to the workstation and extracts initial visual posture information.

[0017] Optionally, the motion sensing unit is electrically connected to a workstation, and the motion sensing unit measures and collects motion state data of the fiber bronchoscope in the airway and transmits the data to the workstation.

[0018] Optionally, the workstation includes an embedded platform, which includes: a lightweight convolutional network module, an extended Kalman filter, a voxel distance field fusion module and a risk analysis module;

[0019] The lightweight convolutional network module is used to process the video image and motion state data and perform dense depth estimation;

[0020] The extended Kalman filter is used to combine the initial visual posture information and the motion state data of the fiber bronchoscope in the airway to obtain a six-degree-of-freedom posture estimation result;

[0021] The voxel distance field fusion module is used to fuse dense depth and six-degree-of-freedom absolute pose to construct a three-dimensional airway model;

[0022] The risk analysis module is communicatively connected to the information interaction unit and is used to perform risk assessment and surgical decision-making on the airway digital twin model.

[0023] Optionally, the motion sensing unit is an inertial measurement unit or a magneto-inertial navigation module; the motion monitoring unit is a breathing belt or a depth camera; and the information interaction unit is an augmented reality head-mounted display device or a boom-type display or a touch tablet.

[0024] The present invention also provides a method for using a real-time airway digital twin system based on bronchoscopic video, which comprises the following steps:

[0025] Step S1, electrically connecting the fiber bronchoscope to the workstation to collect the patient's airway condition data;

[0026] Step S2, constructing a real-time three-dimensional airway model by fusing the airway condition data using a lightweight convolutional network module, an extended Kalman filter, and a voxel distance field fusion module;

[0027] Step S3, registering and respiratory phase compensating the three-dimensional airway model, and dynamically superimposing the registered three-dimensional airway model through an augmented reality head display device to obtain an airway digital twin model;

[0028] Step S4: Perform real-time risk assessment and visualization prompts on the airway digital twin model, and make surgical decisions.

[0029] Optionally, step S1 includes:

[0030] S1.1, CMOS image sensor collects image information in the airway and converts it into fiberoptic bronchoscopic video image;

[0031] S1.2, the central processing unit in the workstation receives the video image and extracts the initial visual pose information.

[0032] Optionally, step S2 includes:

[0033] S2.1, data input and sliding window processing;

[0034] S2.2, uses a lightweight convolutional network module to perform dense depth estimation on fiberoptic bronchoscopy video images;

[0035] S2.3, using an extended Kalman filter to fuse the initial visual pose information and the motion state data of the fiberoptic bronchoscope in the airway to obtain a six-degree-of-freedom absolute pose;

[0036] S2.4. Use the voxel distance field fusion module to fuse dense depth and six-degree-of-freedom absolute pose to construct a three-dimensional airway model.

[0037] Optionally, step S2.3 includes:

[0038] Extracting initial visual pose information: The central processing unit extracts directional rotation feature points between two adjacent frames and performs bidirectional optical flow correction. Ultimately, it obtains the relative visual pose information between the two frames by minimizing the photometric error.

[0039] An extended Kalman filter is used to combine the relative visual posture information and the motion state data of the fiber bronchoscope in the airway to obtain a six-degree-of-freedom absolute posture.

[0040] Optionally, step S3 includes:

[0041] S3.1, acquiring surface data of the patient's head and neck using a structured light scanner, and normalizing the surface data and the three-dimensional airway model using a workstation to align the three-dimensional airway model;

[0042] S3.2, detecting and calculating the current respiratory phase angle through the respiratory belt, and performing respiratory phase compensation on the registered three-dimensional airway model through the workstation;

[0043] S3.3, mapping the phase-complemented three-dimensional airway model onto the real view captured by the augmented reality head-mounted display device, and transmitting the dynamically superimposed and registered three-dimensional airway model to the workstation to finally obtain the airway digital twin model.

[0044] Optionally, step S4 includes:

[0045] The risk analysis module calculates three key risk indicators: the narrowest airway cross-sectional area, airway tortuosity, and intubation friction risk, and visualizes the key risk indicator results;

[0046] Based on the results of key risk indicators, airway surgery is decided: when the narrowest cross-sectional area of ​​the airway is less than 55 mm 2 When the airway curvature radius is less than 10 mm, a thin-diameter fiber bronchoscope should be used instead; when the airway curvature radius is less than 10 mm, the catheter should be pre-bent; when the probability of hitting the wall is greater than 0.25, lubricant should be added and the advancement speed should be slowed down.

[0047] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0048] 1. The present invention provides a real-time airway digital twin system based on bronchoscopic video and its use method. By optimizing the image acquisition and processing process, it can output a three-dimensional airway model with millimeter-level accuracy within 1 minute without relying on thin-layer CT scans with high radiation doses, greatly shortening the preoperative preparation time and avoiding the radiation risk to patients due to imaging examinations. It is particularly suitable for clinical scenarios with high timeliness requirements such as emergency rescue and ICU bedside intubation.

[0049] 2. The present invention provides a real-time airway digital twin system based on bronchoscopic video and its usage method, which achieves high-quality three-dimensional reconstruction based on monocular vision. It does not require additional binocular lenses or other special hardware equipment. It is compatible with the current mainstream bronchoscopic system, effectively reducing equipment procurement and maintenance costs, and can be seamlessly embedded in existing clinical operation processes, thereby improving the practicality and promotion feasibility of the system.

[0050] 3. The present invention provides a real-time airway digital twin system based on bronchoscopic video and its usage method, which introduces a respiratory phase compensation mechanism and a dynamic registration algorithm. It can track the anatomical changes caused by the patient's chest movement and neck micro-movement in real time, maintain the spatial consistency between the virtual model and the real airway structure, and ensure that functions such as stenosis prompts and sharp bend warnings are always accurately aligned during intubation, significantly improving the stability and clinical reliability of AR navigation.

[0051] 4. The present invention provides a real-time airway digital twin system based on bronchoscopic video and its use method, which has the ability to automatically identify and quantify key airway parameters, such as the narrowest cross-sectional area, bend radius, and catheter wall strike probability, and generate interpretable risk assessment indicators based on these parameters. This function not only provides doctors with intuitive and objective operational guidance, but also can be linked to electronic medical record systems to provide standardized records and decision support for difficult airway management, while also providing a solid data foundation for teaching and training, quality control, and the continuous optimization of AI-assisted diagnostic models. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is an operating block diagram of the real-time airway digital twin system of the present invention. DETAILED DESCRIPTION

[0053] The present invention will be further described in detail below through preferred embodiments in conjunction with the accompanying drawings. According to the following description, the advantages and features of the present invention will be clearer. It should be noted that the drawings are in a very simplified form and use non-precise proportions, which are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention. In order to make the purposes, features and advantages of the present invention more obvious and easy to understand, please refer to the accompanying drawings. It should be noted that the structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the limiting conditions for the implementation of the present invention, so they have no technical significance. Any modification of the structure, change in the proportional relationship or adjustment of the size should still fall within the scope of the technical content disclosed by the present invention without affecting the efficacy and purpose that can be achieved by the present invention.

[0054] The present invention provides a real-time airway digital twin system based on fiberbronchoscope video, which includes: a fiberbronchoscope with a built-in CMOS (complementary metal oxide semiconductor) image sensor and a motion sensing unit, which is used to extend into the patient's airway to collect airway condition data; a workstation, electrically connected to the fiberbronchoscope, for processing the collected airway condition data and constructing a three-dimensional airway model; a structured light scanner, electrically connected to the workstation, for aligning the scanned surface data of the patient's head and neck with the three-dimensional airway model; a motion monitoring unit, electrically connected to the workstation, for monitoring the patient's respiratory movement and providing respiratory phase compensation for the three-dimensional airway model; an information interaction unit, which is communicatively connected to the workstation, and after capturing the real view of the head and neck area and airway under dynamic physiological motion conditions, the information interaction unit dynamically superimposes and aligns the three-dimensional airway model after alignment and respiratory phase compensation to obtain an airway digital twin model on the workstation.

[0055] Among them, the motion perception unit is an inertial measurement unit or a magneto-inertial navigation module; the motion monitoring unit is a breathing belt or a depth camera; and the information interaction unit is an augmented reality head-mounted display device, a boom-type display, or a touch tablet.

[0056] In the following, a specific embodiment of the present invention is described by taking the motion sensing unit as an inertial measurement unit (IMU), the motion monitoring unit as a breathing belt, and the information interaction unit as an augmented reality head display device as an example.

[0057] Among them, the CMOS image sensor is electrically connected to the workstation, and the CMOS image sensor collects image information in the airway and converts the collected image information into a video image in real time, and extracts the video image into initial visual posture information through the central processing unit in the workstation; the inertial measurement unit is electrically connected to the workstation, and the inertial measurement unit measures and collects the motion state data of the fiber bronchoscope in the airway (i.e., three-axis angular velocity and three-axis linear acceleration), and transmits it to the workstation.

[0058] Furthermore, the workstation includes an embedded platform that deploys multiple algorithm modules, including: a lightweight convolutional network module, an extended Kalman filter (EKF), a voxel distance field fusion module, and a risk analysis module.

[0059] Optionally, the voxel distance field fusion module may be replaced by an octree structure based on a signed distance function or an implicit multi-layer perceptron.

[0060] The lightweight convolutional network module is used to process the video image and motion state data frame by frame and perform dense depth estimation.

[0061] In a specific embodiment of the present invention, the lightweight convolutional network module uses a lightweight convolutional neural network (DeepLite-Mono) to perform dense depth estimation on each frame of video image. The lightweight convolutional neural network includes a 6-layer downsampling encoder and a 6-layer symmetric decoder, with a feature pyramid branch added in the middle to preserve multi-scale semantic information.

[0062] Furthermore, the input video image resolution is 960×540 pixels. After processing it with a lightweight convolutional neural network, the output is an airway depth map with the same resolution as the input. To adapt to edge computing environments, this lightweight convolutional neural network undergoes tensor pruning and 8-bit integer quantization before being deployed on an embedded platform, ensuring inference efficiency while maintaining depth estimation accuracy.

[0063] The central processing unit in the workstation calls an extended Kalman filter (EKF), which combines the initial visual posture information and the motion state data of the fiber bronchoscope in the airway to obtain a six-degree-of-freedom absolute posture.

[0064] Specifically, by fusing the relative visual pose of adjacent image frames with the synchronously acquired three-axis angular velocity and three-axis linear acceleration data using an extended Kalman filter, the fiberoptic bronchoscope's absolute six-degree-of-freedom pose in a unified coordinate system is determined in real time. This not only suppresses the drift that can easily occur with feature matching alone, but also provides a continuous and accurate pose prior for voxel distance field accumulation and augmented reality registration.

[0065] Among them, the central processing unit in the workstation calls the voxel distance field fusion module, and the dense depth estimation results and the six-degree-of-freedom absolute pose results are transmitted to the voxel distance field fusion module, thereby constructing a continuous and seamless three-dimensional airway model.

[0066] The risk analysis module is used to perform risk assessment and surgical decision-making on the generated airway digital twin model. The risk analysis module uses a risk analysis algorithm.

[0067] In the present invention, the fiberoptic bronchoscope is a flexible fiberoptic bronchoscope. Preferably, the outer diameter of the flexible fiberoptic bronchoscope is 4.8 mm; the resolution of the CMOS image sensor is 1920×1080 pixels and the frame rate is 30 frames per second (fps); the structured light scanner is a handheld structured light scanner; the breathing belt is a resistive strain breathing belt; and the augmented reality head-mounted display device is a Magic Leap 2 head-mounted display device.

[0068] The present invention also provides a method for operating the real-time airway digital twin system, such as Figure 1 As shown, it includes the following steps:

[0069] Step S1: electrically connect the flexible fiber bronchoscope to the workstation to preliminarily collect the patient's airway condition data.

[0070] Specifically, the CMOS image sensor collects image information in the airway and converts the collected information into a fiberoptic bronchoscopic video image with a resolution of 1920×1080 pixels and a frame rate of 30 frames per second in real time. The workstation then extracts initial visual posture information from the video image through a central processing unit therein.

[0071] The video image extraction includes the following processes: the video image is subjected to median filtering of a 3*3 window to remove salt noise, subjected to brightness stretching according to a 5%-95% interval of a gray histogram and automatically corrected for white balance, so that the contrast of tracheal wall texture is fully sufficient, then subjected to gamma mapping with a gamma value of 0.9 (i.e. gamma=0.9) for slight brightening, and finally the edge image information is calculated.

[0072] In step S2, the airway condition data is fused by a lightweight convolutional network module, an extended Kalman filter and a voxel distance field fusion module of the workstation to construct a real-time three-dimensional airway model.

[0073] Specifically, step S2 includes the following steps:

[0074] S2.1, data input and sliding window processing;

[0075] The workstation reads 4 frames of video images from the flexible bronchoscope in succession each time, and synchronously collects 4 groups of motion state data of the flexible bronchoscope in the airway measured by the inertial measurement unit in the same time period. The video images and the data measured by the inertial measurement unit jointly constitute a sliding window, the window step is 1 frame, and the adjacent windows are highly overlapped to maintain the time continuity of motion estimation, while avoiding the loading pressure caused by loading too many frames at one time.

[0076] S2.2, a lightweight convolutional network module is used to perform dense depth estimation on the flexible bronchoscope video images frame by frame;

[0077] Specifically, the input video image has a resolution of 960*540 pixels, and after being processed by the lightweight convolutional neural network, the output is an airway depth map with the same resolution as the input. To adapt to the edge computing environment, the lightweight convolutional neural network model is deployed on an embedded platform after tensor pruning and 8-bit integer quantization, which ensures the inference efficiency while maintaining the depth estimation accuracy.

[0078] S2.3, an extended Kalman filter is used to fuse the initial visual pose information and the motion state data of the flexible bronchoscope in the airway to obtain a six-degree-of-freedom absolute pose;

[0079] Specifically, step S2.3 includes the following steps:

[0080] S2.3.1, extracting initial visual pose information: the central processing unit extracts directional rotation feature points between adjacent two frames, and performs bidirectional optical flow correction, and finally obtains the relative visual pose information between the two frames by minimizing the photometric error;

[0081] S2.3.2, using an extended Kalman filter to combine the relative visual posture information and the motion state data of the flexible fiber bronchoscope in the airway to obtain a six-degree-of-freedom absolute posture.

[0082] S2.4. Use the voxel distance field fusion module to fuse dense depth and six-degree-of-freedom absolute pose to construct a three-dimensional airway model.

[0083] Through step S2.4, a real-time three-dimensional airway model with precise semantic labels was established without relying on any tomography or binocular hardware, and the full-process processing delay was kept <40ms, which could be updated synchronously with clinical bronchoscopy operations.

[0084] The fusion process described in step S2.4 is as follows:

[0085] S2.4.1, Voxel field update: A cubic grid space with a side length of 180mm and a voxel resolution of 1.5mm is initialized through the workstation, and a truncated signed distance field (TSDF) is established in it to represent local geometric information. Whenever a new set of airway depth maps and corresponding poses are obtained, the point cloud corresponding to the frame is projected into the voxel grid, its distance to the surface is calculated, and the corresponding voxel node is updated according to the weight. If the cumulative weight of a voxel node reaches the set threshold, it is marked as "reliable" and waits for subsequent surface extraction. After about 20 frames of fusion, the airway segment of about 60mm in front of the camera can be completely filled. To improve memory efficiency, the voxel structure is implemented using a sparse hash table, and storage space is allocated only for the observed nodes. The overall storage capacity is controlled within 400MB.

[0086] S2.4.2, Mesh extraction and semantic coloring: After the voxel field update is completed, the marching cubes algorithm is executed to extract the isosurface. In order to reduce the complexity of the triangular mesh, coplanar triangles are merged according to the adjacent normal angle threshold (for example, 15°), and the double mesh simplification algorithm is further used to compress the total number of triangles to 42% of the original. At the same time, the three-dimensional convolutional semantic classifier is called to assign red, yellow, and blue labels to key anatomical structures (such as the glottis, cricoid cartilage, and main tracheal bifurcation), respectively. The unidentified areas remain in the default transparent gray to achieve visual auxiliary positioning. The final generated triangular mesh is streamed out via glTF 2.0 format, with an inter-frame update granularity of 200μs, which can be seamlessly connected to the downstream augmented reality navigation module.

[0087] Step S3: align and compensate for the respiratory phase of the three-dimensional airway model, and dynamically superimpose the aligned three-dimensional airway model through an augmented reality head display device to obtain an airway digital twin model.

[0088] Specifically, step S3 includes the following steps:

[0089] S3.1, acquiring surface point cloud data of the patient's head and neck by the structured light scanner, and normalizing the point cloud data and the three-dimensional airway model by the workstation to register the three-dimensional airway model;

[0090] The process of acquiring surface point cloud data of the head and neck includes: the patient is in a supine position, and the operator uses a handheld structured light scanner to collect surface point cloud data of the face to the clavicle region, and places optical marker stickers at key positions.

[0091] More specifically, before starting the bronchoscope intubation, the patient needs to be adjusted to a supine position suitable for intubation, and the operator places small optical marker stickers with high-reflective two-dimensional codes at three key positions (manubrium, zygomatic arch, and above the thyroid cartilage). Then the operator uses a handheld structured light scanner to slowly circle around the patient's head and neck area for about 5 seconds to acquire surface point cloud data from the face to the clavicle region.

[0092] Next, the workstation performs scale normalization on the outer contour of the semantic airway mesh generated in step S2.4.2 and the surface point cloud data obtained in the above steps. Based on the positional relationship of the optical marker stickers, an initial transformation matrix is established to make the two sets of data roughly overlap. Then, the Iterative Closest Point (ICP) algorithm is called, and the nearest neighbor distance is iterated 30 times until the root mean square error is reduced to below 2 mm. Finally, the three-dimensional airway model is registered with the patient's surface data to within millimeters in the resting breathing state.

[0093] S3.2, detecting and calculating the current respiratory phase angle by the respiratory belt, and compensating the registered three-dimensional airway model by the workstation;

[0094] Specifically, the operator will put a reusable resistance strain respiratory belt on the patient's chest. The respiratory belt outputs a chest expansion displacement signal at a frequency of 50 Hz. The resistance strain respiratory belt transmits the respiratory displacement signal to the workstation through a zero-delay serial port. An independent thread in the workstation detects the peak and valley changes of the respiratory waveform in real time and calculates the current respiratory phase angle.

[0095] Then, when it is detected that the three-dimensional airway model is offset from the current respiratory phase, the workstation performs real-time translation compensation of the three-dimensional airway model along the sagittal plane direction according to the displacement amount of the respiratory belt multiplied by an empirical coefficient of 0.8. This process is updated every 20 ms, so that the three-dimensional airway model moves synchronously with the chest.

[0096] Experiments show that without respiratory phase compensation, a model misalignment of approximately 5-10 mm may occur at the end of a deep inhalation. After respiratory phase compensation, the model registration error is always controlled within 2 mm throughout the entire respiratory cycle, and the stenosis prompt, sharp bend warning, and risk heat map always maintain precise correspondence with the actual airway position.

[0097] S3.3, mapping the phase-complemented three-dimensional airway model onto a real view captured by an augmented reality head-mounted display device, and transmitting the dynamically superimposed and registered three-dimensional airway model to a workstation to obtain an airway digital twin model.

[0098] Specifically, the operator wears a Magic Leap 2 headset to enter the AR mode of the augmented reality headset. The depth camera built into the augmented reality headset runs spatial mapping at a frequency of 30Hz, capturing the patient's facial and neck shape (especially narrow segments, curved segments or other high-risk structures) in real time and comparing it with the initial point cloud. If it is found that the anchor point is offset due to slight movement of the patient's head or changes in ambient lighting, the system will immediately trigger the fine-tuning mechanism, recalculate the six-degree-of-freedom absolute pose, and translate or rotate the three-dimensional airway model as a whole to ensure that the superimposed three-dimensional airway model does not drift with changes in viewing angle.

[0099] Step S4: Perform real-time risk assessment and visualization prompts on the airway digital twin model, and make surgical decisions.

[0100] Specifically, step S4 includes the following steps:

[0101] S4.1. The workstation's risk analysis module calculates three key risk indicators (KRIs): the narrowest airway cross-sectional area, airway tortuosity, and intubation friction risk, and visualizes the KRI results.

[0102] Specifically, the risk analysis algorithm generates vertical sections every 0.5 mm along the center line of the airway digital twin model and calculates its cross-sectional area in real time.

[0103] The risk analysis algorithm calculates the curvature radius of all sampling points in the centerline of the airway digital twin model and draws the curvature distribution curve.

[0104] The risk analysis algorithm calculates the closest distance from the tip of the fiber bronchoscope lens to the tracheal wall, accumulates the pixels with a distance less than 3 mm, and superimposes a red hot spot in the corresponding area. The color depth of the red hot spot reflects the risk probability.

[0105] In addition, all information is refreshed every 100 ms, and risk information can be directly obtained through the head-mounted device; further, when the triggering condition occurs, the head-mounted device center will also pop up a text prompt and a buzzer reminder, and be synchronized to the workstation; if all indicators are normal, a green check mark is displayed to indicate that the path is safe; the decision-making process is seamlessly embedded in the standard operation of intubation, without additional time consumption.

[0106] S4.2, according to the results of the key risk indicators, decision-making airway operation;

[0107] For the narrowest cross-sectional area of the airway <55mm 2 The result triggers the decision to use a thin-diameter fiber bronchoscope; for the case where the radius of curvature is <10mm, a pre-bent catheter prompt is triggered; when the probability of hitting the wall is >0.25, the red spot presents a deep red color, and a prompt to increase the lubricant and slow down the pushing speed is triggered.

[0108] S4,3, bedside verification and clinical workflow integration;

[0109] The operator pastes the optical mark and connects the respiratory monitoring device according to the results of step S4.2, and then performs a 30-second fiber bronchoscope exploration, and the workstation generates a risk report within 40 seconds and imports it into the electronic medical record of the hospital network, and generates a searchable process record for department quality control review and teaching training.

[0110] In summary, the present application provides a real-time airway digital twin system based on fiber bronchoscope video and a method for using the same, which realizes a second-level closed loop in a narrow and dynamic contraction airway through monocular fiber bronchoscope depth inertia synchronous positioning, millimeter-level truncated signed distance field incremental fusion, and respiratory phase coupling registration. The airway digital twin system first uses a lightweight network to infer a dense depth, then obtains a drift-free six-degree-of-freedom absolute pose by extending the Kalman filter to the same frame IMU angular velocity and acceleration, and uses it as the basis for the geometry and coordinates of the continuous point cloud. And every 40 ms, the 1.5 mm sparse hash truncated signed distance field is accumulated and updated to ensure the steady growth of the three-dimensional airway model, and then the phase difference calculated from the 50Hz chest displacement is converted into a sagittal plane microshift by a coefficient of 0.8, and finally the AR superposition error is compressed from about 6mm to 2mm. The entire process does not require CT or binocular fiber bronchoscope hardware, and can complete three-dimensional reconstruction, visualization and navigation in real time at the bedside. The present application has significant advantages in improving intubation safety, operation efficiency and diagnosis and treatment standardization, and has good clinical application prospect and promotion value.

[0111] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0112] In the description of the present invention, it should be understood that the terms "center," "height," "thickness," "up," "down," "vertical," "horizontal," "top," "bottom," "inside," "outside," "axial," "radial," "circumferential," and the like, indicating positions or location relationships, are based on the positions or location relationships shown in the accompanying drawings and are intended only to facilitate the description of the present invention and simplify the description. They do not indicate or imply that the devices or elements referred to must have a specific orientation, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limiting the present invention. In the description of the present invention, unless otherwise specified, "plurality" means two or more.

[0113] In the description of the present invention, unless otherwise expressly specified or limited, the terms "mounted," "connected," "connected," and "fixed" should be understood in a broad sense. For example, they may refer to fixed or detachable connections, or integration; mechanical or electrical connections; direct or indirect connections through an intermediate medium; and internal communication between two components or interaction between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0114] In the present invention, unless otherwise expressly specified or limited, a first feature being "above" or "below" a second feature may include the first and second features being in direct contact, or may include the first and second features being in contact not directly but through another feature between them. Furthermore, a first feature being "above," "above," and "above" a second feature may include the first feature being directly above or obliquely above the second feature, or may simply mean that the first feature is higher in level than the second feature. A first feature being "below," "below," and "below" a second feature may include the first feature being directly below or obliquely below the second feature, or may simply mean that the first feature is lower in level than the second feature.

[0115] Although the present invention has been described in detail through the above preferred embodiments, it should be understood that the above description is not intended to limit the present invention. After reading the above description, various modifications and substitutions of the present invention will become apparent to those skilled in the art. Therefore, the scope of protection of the present invention should be defined by the appended claims.

Claims

1. A real-time airway digital twin system based on bronchoscopic video, characterized by: include: A fiberoptic bronchoscope with a built-in CMOS image sensor and motion sensing unit to collect data on the patient's airway condition; a workstation comprising a central processing unit, the workstation being electrically connected to the fiberoptic bronchoscope to process the collected airway condition data and construct a three-dimensional airway model; a structured light scanner electrically connected to the workstation to register scanned surface data of the patient's head and neck with the three-dimensional airway model; a motion monitoring unit, electrically connected to the workstation, for monitoring the patient's respiratory motion and providing respiratory phase compensation for the three-dimensional airway model; The information interaction unit is communicatively connected to the workstation and is used to capture dynamic real views of the head, neck and airway, and dynamically overlay and register the three-dimensional airway model to obtain a digital twin model of the airway in the workstation.

2. The airway digital twin system according to claim 1, characterized in that: The CMOS image sensor is electrically connected to the workstation, and the CMOS image sensor transmits the collected airway video image to the workstation and extracts initial visual posture information.

3. The airway digital twin system according to claim 2, characterized in that: The motion sensing unit is electrically connected to the workstation, and the motion sensing unit measures and collects the motion state data of the fiber bronchoscope in the airway and transmits the data to the workstation.

4. The airway digital twin system according to claim 3, characterized in that: The workstation includes an embedded platform, which includes: a lightweight convolutional network module, an extended Kalman filter, a voxel distance field fusion module and a risk analysis module; The lightweight convolutional network module is used to process the video image and motion state data and perform dense depth estimation; The extended Kalman filter is used to combine the initial visual posture information and the motion state data of the fiber bronchoscope in the airway to obtain a six-degree-of-freedom posture estimation result; The voxel distance field fusion module is used to fuse dense depth and six-degree-of-freedom absolute pose to construct a three-dimensional airway model; The risk analysis module is communicatively connected to the information interaction unit and is used to perform risk assessment and surgical decision-making on the airway digital twin model.

5. The airway digital twin system according to claim 1, characterized in that: The motion sensing unit is an inertial measurement unit or a magneto-inertial navigation module; the motion monitoring unit is a breathing belt or a depth camera; and the information interaction unit is an augmented reality head display device, a boom-type display, or a touch tablet.

6. A method for using a real-time airway digital twin system based on bronchoscopic video, implemented based on the airway digital twin system according to any one of claims 1 to 5, characterized in that: The usage method includes the following steps: Step S1, electrically connecting the fiber bronchoscope to the workstation to collect the patient's airway condition data; Step S2, constructing a real-time three-dimensional airway model by fusing the airway condition data using a lightweight convolutional network module, an extended Kalman filter, and a voxel distance field fusion module; Step S3, registering and respiratory phase compensating the three-dimensional airway model, and dynamically superimposing the registered three-dimensional airway model through an augmented reality head display device to obtain an airway digital twin model; Step S4: Perform real-time risk assessment and visualization prompts on the airway digital twin model, and make surgical decisions.

7. The method for using the airway digital twin system according to claim 6, wherein: The step S1 comprises: S1.1, CMOS image sensor collects image information in the airway and converts it into fiberoptic bronchoscopic video image; S1.2, the central processing unit in the workstation receives the video image and extracts the initial visual pose information.

8. The method for using the airway digital twin system according to claim 7, wherein: The step S2 comprises: S2.1, data input and sliding window processing; S2.2, uses a lightweight convolutional network module to perform dense depth estimation on fiberoptic bronchoscopy video images; S2.3, using an extended Kalman filter to fuse the initial visual pose information and the motion state data of the fiberoptic bronchoscope in the airway to obtain a six-degree-of-freedom absolute pose; S2.

4. Use the voxel distance field fusion module to fuse dense depth and six-degree-of-freedom absolute pose to construct a three-dimensional airway model.

9. The method for using the airway digital twin system according to claim 8, wherein: The step S2.3 comprises: Extracting initial visual pose information: The central processing unit extracts directional rotation feature points between two adjacent frames and performs bidirectional optical flow correction. Ultimately, it obtains the relative visual pose information between the two frames by minimizing the photometric error. An extended Kalman filter is used to combine the relative visual posture information and the motion state data of the fiber bronchoscope in the airway to obtain a six-degree-of-freedom absolute posture.

10. The method for using the airway digital twin system according to claim 9, wherein: The step S3 comprises: S3.1, acquiring surface data of the patient's head and neck using a structured light scanner, and normalizing the surface data and the three-dimensional airway model using a workstation to align the three-dimensional airway model; S3.2, detecting and calculating the current respiratory phase angle through the respiratory belt, and performing respiratory phase compensation on the registered three-dimensional airway model through the workstation; S3.3, mapping the phase-complemented three-dimensional airway model onto the real view captured by the augmented reality head-mounted display device, and transmitting the dynamically superimposed and registered three-dimensional airway model to the workstation to finally obtain the airway digital twin model.

11. The method for using the airway digital twin system according to claim 10, wherein: The step S4 comprises: The risk analysis module calculates three key risk indicators: the narrowest airway cross-sectional area, airway tortuosity, and intubation friction risk, and visualizes the key risk indicator results; Based on the results of key risk indicators, airway surgery is decided: when the narrowest cross-sectional area of ​​the airway is less than 55 mm 2 When the airway curvature radius is less than 10 mm, a thin-diameter fiber bronchoscope should be used instead; when the airway curvature radius is less than 10 mm, the catheter should be pre-bent; when the probability of hitting the wall is greater than 0.25, lubricant should be added and the advancement speed should be slowed down.

Images