An augmented reality-based bronchoscope path planning system, device, and storage medium

Abstract

The application provides a bronchoscope path planning system and device based on augmented reality and a storage medium, and the system comprises the following steps: acquiring a bronchial three-dimensional model; acquiring a bronchial real-time image; outputting an abnormal marking image through a YOLO algorithm; outputting a recommended route image through a Markov chain and a dynamic programming algorithm; superimposing and processing the bronchial three-dimensional model, the bronchial real-time image, the abnormal marking image and the recommended route image and displaying them in the user's field of view; and superimposing and displaying the bronchial three-dimensional model and the recommended route image, and superimposing and displaying the bronchial real-time image and the abnormal marking image. The application makes a dynamic response to the abnormality in the operation through the YOLO algorithm and the Markov chain, and provides instant information feedback for navigation. The application automatically detects the abnormal area through the YOLO algorithm, and improves the reliability of the marking. The application realizes stereoscopic navigation display through the augmented reality technology, and improves the operation efficiency in the complex environment.

Description

Technical Field

[0001] This invention relates to the field of medical device technology, specifically to an augmented reality-based bronchoscopy path planning system, device, and storage medium. Background Technology

[0002] A bronchoscope is a medical device used for the diagnosis and treatment of respiratory diseases. It is primarily applicable to various diagnostic and treatment scenarios, including observation of lesions in the lobar, segmental, and subsegmental bronchi, biopsy sampling, and bacteriological and cytological examinations. Through the attached biopsy sampling attachment, the bronchoscope can assist in the detection of early lesions and simultaneously perform procedures such as tissue biopsy, secretion aspiration, and foreign body removal. With its flexible or rigid endoscope design, combined with real-time image transmission technology, this device provides clinicians with diagnostic evidence and treatment options for respiratory diseases, effectively improving the accuracy of diagnosis and the targeted nature of treatment, making it an important diagnostic and treatment tool in the field of modern respiratory medicine.

[0003] Due to the complex structure and numerous bifurcations of the bronchi (such as... Figure 5 As shown in the image, before the bronchoscope is inserted, the doctor needs to plan the path based on the pre-acquired CT images and manually mark abnormal points to provide auxiliary guidance for the operation during the procedure, reduce the possibility of getting lost in the complex bronchial channels, and ensure that the bronchoscope reaches the target area smoothly.

[0004] However, this method has the following significant drawbacks: 1) This method has obvious lag and cannot be adjusted in real time according to the actual situation during the operation; 2) Manually marked abnormal points are prone to human error and mistakes, which increases the risk of surgery; 3) Traditional path planning is based on two-dimensional navigation design using CT tomographic images. Doctors need to construct the three-dimensional structure of the airway in their minds by stitching together multi-layer images. The understanding of spatial relationships is not intuitive enough, which can easily lead to misjudgment of anatomical structures and operational errors.

[0005] Therefore, how to design an augmented reality-based bronchoscopy path planning system, device, and storage medium that can automatically mark anomalies and update path planning in real time is an unsolved technical problem in the existing technology. Summary of the Invention

[0006] Therefore, the technical problem to be solved by the present invention is to overcome the technical defects of the prior art, which relies on pre-acquired CT images for manual marking and path planning, resulting in a lack of real-time performance and easy path deviation. The present invention provides an augmented reality-based bronchoscopic path planning system, device and storage medium that can automatically mark and update path planning in real time.

[0007] This application mainly includes the following aspects:

[0008] In a first aspect, embodiments of this application provide a bronchoscopy path planning method based on augmented reality, the method comprising:

[0009] A 3D model of the bronchus is obtained; a real-time image of the bronchus is obtained by taking a picture with a bronchoscope; anomaly identification is performed on the real-time bronchus image using the YOLO algorithm, and an anomaly marker image is output; navigation calculation is performed using Markov chain and dynamic programming algorithms, and a recommended route image is output; the 3D model of the bronchus and the recommended route image are superimposed to obtain a first superimposed image; the real-time bronchus image and the anomaly marker image are superimposed to obtain a second superimposed image.

[0010] According to one embodiment of the present invention, the step of performing navigation calculation using Markov chains and dynamic programming algorithms includes: establishing a transition probability matrix based on the Markov chain; and using dynamic programming algorithms to establish a cost function by combining the transition probability matrix, anomaly risk function, physiological characteristic risk function, and physical distance, and solving for the optimal path.

[0011] According to one embodiment of the present invention, the step of establishing the transition probability matrix based on the Markov chain includes: establishing a set of bronchial states. , The set of bronchial states Each element in the bronchial tree represents a different discrete location point in the bronchial tree, and n is the total number of discrete location points in the bronchial tree.

[0012] For the set of bronchial states Any two elements in and Calculate from arrive transition probability , ;in, Let i be the i-th discrete location point in the bronchial tree. Let i be the j-th discrete location point in the bronchial tree, where i and j both belong to the range 1 to n. for arrive Number of transfers, From The total number of transitions from the starting point; construct the transition probability matrix based on the calculation results. ,in:

[0013] ;

[0014] Each element satisfies the following condition:

[0015] .

[0016] According to one embodiment of the present invention, the step of solving for the optimal path by using a dynamic programming algorithm, combining the transition probability matrix, the outlier risk function, the physiological feature risk function, and the physical distance to establish a cost function includes: establishing an outlier risk function based on outliers identified by YOLO. , ;in, For path Anomalies on The risk coefficient of the anomaly is used to quantify the severity of the anomaly. The anomaly point to the path Distance; establishing a physiological feature risk function based on the real-time bronchial images. , ;in, For path segment The actual average diameter of the airway; ;in, and These are obtained through the real-time images of the bronchi. and The actual diameter of the airway; For path segment The average diameter of a normal airway;

[0017] ;in and They are and The diameter of the normal airway; based on the aforementioned transition probability matrix. The transition probability in outlier risk function Physiological characteristic risk function and physical distance Establish a single transfer cost function :

[0018] in, for arrive physical distance, for arrive The transition probability, for arrive outlier risk function, for arrive Physiological characteristic risk function, , and These are weighting coefficients used to balance the transition probabilities. The anomaly risk function and the physiological characteristic risk function For the single transition cost function Impact; Based on single-transfer cost function Establish the minimum cumulative cost function Thus, the optimal path is found, where... , ;in, This is the current state. for The next state, Initial state to state The minimum cumulative cost, Initial state To the current state The optimal cost is the cost of calculating the optimal solution to the subproblem. Current state arrive The cost of a single transfer.

[0019] According to one embodiment of the present invention, the bronchial state set any element in Includes the following information: Coordinates in three-dimensional space From the starting point to The Euclidean distance is used to calculate from arrive physical distance ;from Total number of departures Used to calculate from arrive transition probability ; arrive Risk coefficient of outliers on the path Used to calculate from arrive Outlier risk function on the path ; Actual average airway diameter and the average diameter of a normal airway Used to calculate from arrive Physiological characteristic risk function .

[0020] According to one embodiment of the present invention, the step of anomaly identification of the real-time bronchial image using the YOLO algorithm includes: pre-annotating and training the YOLO model using multiple bronchoscopic videos and CT images, so that the YOLO model can be used to identify abnormal points and the location of key tissues in the bronchus, wherein the abnormal points include any one or more of the following: tumors, lesions, airway obstruction; the key tissues include any one or more of the following: bifurcation points, cartilage, main bronchus; inputting the real-time bronchial image into the YOLO model for identification, and inferring the abnormal points and key tissues in the current image.

[0021] According to one embodiment of the present invention, the method further includes: after obtaining the first superimposed image and the second superimposed image, performing data parsing on the first superimposed image and the second superimposed image, outputting them to a head-mounted augmented reality device, and displaying them in the user's field of vision.

[0022] Secondly, embodiments of this application also provide an augmented reality-based bronchoscopy navigation system, the system comprising: a first acquisition module for acquiring a three-dimensional model of the bronchus; a second acquisition module for acquiring real-time images of the bronchus; an anomaly recognition module for performing anomaly recognition on the real-time images of the bronchus and outputting anomaly marker images; a navigation calculation module for performing navigation calculations and outputting recommended route images; a data processing module for overlaying the three-dimensional model of the bronchus and the recommended route images to obtain a first overlay image and overlaying the real-time images of the bronchus and the anomaly marker images to obtain a second overlay image; and an output module for parsing the first overlay image and the second overlay image and outputting them to a head-mounted augmented reality device.

[0023] Thirdly, embodiments of this application also provide a computer device, the computer device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform the steps of the method described above when executing the instructions.

[0024] Thirdly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method described above.

[0025] The technical solution provided by this invention has the following advantages:

[0026] 1. This invention utilizes real-time images acquired by a bronchoscope and, through the synergistic effect of the YOLO algorithm and Markov chains, dynamically responds to airway abnormalities during surgery, provides immediate information feedback for navigation, and completes path adjustments in a short time, ensuring that navigation information remains consistent with the operational scenario.

[0027] 2. This invention uses the YOLO algorithm to replace manual identification and labeling, automatically detecting abnormal areas such as tumors, lesions, and airway obstruction, which can avoid labeling errors caused by human factors and improve the reliability of labeling.

[0028] 3. This invention achieves three-dimensional navigation display through augmented reality technology, overlaying the 3D model of the bronchus with the recommended route image in the user's field of vision, enabling doctors to more accurately establish spatial cognition of anatomical structures during operation and improving surgical efficiency in complex bronchial branching environments.

[0029] 4. This invention adds a bronchial feature enhancement submodule to the YOLO backbone network, which improves the network's ability to distinguish complex bronchial structures and low-contrast lesion areas. This module effectively improves the average detection accuracy of the YOLO model on bronchoscopic image datasets and enhances its recognition performance of important structures such as the main bronchus, bifurcation points, and airway obstructions.

[0030] 5. This invention breaks through the limitations of traditional Markov chain single geometric modeling by constructing a bronchial state set containing multidimensional anatomical and physiological information. It upgrades the state definition of Markov chain from simple spatial coordinates to a composite model that integrates structural features, functional attributes and risk features, providing a more comprehensive decision basis for navigation systems and improving the rationality and safety of path planning in complex bronchial environments.

[0031] 6. This invention incorporates real-time detection of anomaly risk functions and physiological characteristic risk functions into navigation calculations, enabling the transition matrix to reflect real-time changes in the intraoperative airway environment. This achieves automatic suppression of high-risk paths and priority selection of safe paths, endowing the navigation system with dynamic decision-making capabilities to adapt to complex lesion scenarios and improving the reliability of surgical path planning.

[0032] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0033] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is an overall flowchart of the bronchoscopic path planning method based on augmented reality of the present invention.

[0035] Figure 2 This is a flowchart of the process for obtaining a three-dimensional model of the bronchus according to the present invention.

[0036] Figure 3 This is a flowchart illustrating the anomaly identification of real-time bronchial images using the YOLO algorithm, as described in this invention.

[0037] Figure 4 This is an architecture diagram of the bronchoscopy navigation system based on augmented reality according to the present invention.

[0038] Figure 5 This is a schematic diagram of the structure of the bronchi in the lungs.

[0039] Figure 6 This is a schematic diagram of the first superimposed image of the present invention.

[0040] Figure 7 This is a schematic diagram of the second superimposed image of the present invention.

[0041] Figure labels: 1. Edge of real-time bronchial image; 2. Key tissue identification box; 3. Key tissue label; 4. Anomaly identification box; 5. Anomaly label; 6. Recommended route image; 7. 3D model of bronchus. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0043] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0044] The present application will now be described in detail with reference to the accompanying drawings and embodiments. Example

[0045] like Figure 1 As shown, this embodiment provides a bronchoscopy path planning method based on augmented reality, including the following steps:

[0046] S101: Obtain a 3D model of the bronchus;

[0047] S102: Acquire real-time images of the bronchi; the real-time images of the bronchi are acquired by taking pictures with a bronchoscope;

[0048] S103: The real-time bronchial image is subjected to anomaly identification using the YOLO algorithm, and an anomaly-marked image is output.

[0049] S104: Navigation calculations are performed using Markov chains and dynamic programming algorithms, and a recommended route image is output.

[0050] S105: Overlay the bronchial 3D model and the recommended route image to obtain a first overlay image; overlay the bronchial real-time image and the anomaly marker image to obtain a second overlay image.

[0051] The following details each step of the augmented reality-based bronchoscopy route planning method:

[0052] S101: Obtain a 3D model of the bronchus;

[0053] like Figure 2 As shown, the step of obtaining the 3D model of the bronchus in step S101 includes the following steps:

[0054] Step 1011: Scan the QR code using a head-mounted augmented reality device;

[0055] Step 1012: Obtain the bronchial 3D model link by parsing the online database using the QR code;

[0056] Step 1013: Based on the bronchial 3D model link, download and store the bronchial 3D model in the built-in sandbox environment of the head-mounted augmented reality device;

[0057] Step 1014: In the built-in sandbox environment, convert the 3D model of the bronchus into a renderable mesh that can be displayed normally, and reduce the number of polygons to improve the mesh reconstruction speed.

[0058] For example, the head-mounted augmented reality device used is the HoloLens device developed by Microsoft, which supports augmented reality (AR) and can suspend a virtual 3D model of the bronchus next to the real patient. Doctors can directly perform remote operations through gestures or eye gaze, such as rotating, zooming, and dragging, to further observe the details of the model.

[0059] Specifically, the hospital has a medical record database website containing each patient's personal information, medical records, 3D models, and other information. Doctors can use their HoloLens devices to scan QR codes containing patient information and 3D model data URLs (Uniform Resource Locators). The HoloLens device can then automatically retrieve a JSON file containing patient-related data from the hospital's medical record database website, extract the download links for the required 3D models (such as STL or OBJ formats), and then download and store these model files.

[0060] Next, the 3D model is converted into a rendering mesh suitable for display in a virtual environment. Specific steps include:

[0061] After obtaining the 3D model file, since STL models have a specific format, the Assimp library is usually used to parse them. Specifically, the Assimp library can convert models in specific formats like STL into general geometric data, which includes information such as vertices, faces, and materials.

[0062] After obtaining the general geometric data, the QEM (Quadric Error Metrics) simplification algorithm is used to process it. This algorithm merges similar vertices by calculating the error matrix, reducing the number of redundant polygons while maintaining the overall shape of the model. This optimizes the general geometric data, generating simpler geometric data that retains key features, and significantly improves the display effect and rendering speed of the model in the virtual environment.

[0063] Then, Unity's built-in Mesh processing library is used to further process the geometric data optimized by the QEM algorithm. The Mesh processing library adapts this geometric data to Unity's Mesh data structure. For example, it converts vertex coordinates to coordinates in the Unity coordinate system and fills the vertex and triangle index data into the corresponding fields of Unity Mesh. After such processing, the 3D model is transformed into mesh data suitable for rendering and display in the Unity virtual environment, so that it can be used in HoloLens devices.

[0064] It should be noted that the bronchial 3D model obtained in step S101 is "one model per person", that is, it is modeled using a segmentation algorithm based on the patient's CT image before the operation, and then uploaded to the medical record database website set up by the hospital. This is a prior art well known to those skilled in the art.

[0065] S102: Acquire real-time images of the bronchi; real-time images of the bronchi are acquired by taking pictures with a bronchoscope;

[0066] Specifically, real-time bronchial images are acquired using a medical electronic bronchoscope, which integrates a high-definition camera (resolution ≥1920×1080, frame rate ≥30fps) and supports multiple imaging modes such as white light and narrow-band imaging (NBI), which can clearly capture bronchial mucosal texture, lumen morphology and abnormal lesion features.

[0067] S103: Anomaly identification is performed on real-time bronchial images using the YOLO algorithm, and anomaly-marked images are output.

[0068] like Figure 3 As shown, in one possible implementation, step S103, which involves identifying anomalies in real-time bronchial images using the YOLO algorithm and outputting anomaly-marked images, includes the following steps:

[0069] Step 1031: The YOLO model is labeled and trained in advance using multiple bronchoscopic videos and CT images so that the YOLO model can be used to identify abnormal points and the location of key tissues in the bronchi. Abnormal points include any one or more of the following: tumors, lesions, airway obstruction; key tissues include any one or more of the following: bifurcation points, cartilage, main bronchus.

[0070] Step 1032: Convert the YOLO model to an ONNX model for inference in other frameworks and platforms;

[0071] Step 1033: Deploy the ONNX model on the head-mounted augmented reality device and select the appropriate inference library in Unity;

[0072] Step 1034: Input the real-time image of the bronchus into the ONNX model for identification, and infer the abnormal points and key tissues in the current image.

[0073] It should be noted that in order to enable cross-platform inference of the YOLO model on head-mounted augmented reality devices, the trained YOLO model needs to be converted into the ONNX (Open Neural Network Exchange) format, which supports migration between different frameworks and hardware platforms.

[0074] To improve the target recognition performance of the YOLO algorithm in bronchoscopic images, this embodiment makes targeted adaptive improvements to the YOLO neural network structure. Specifically, since common anatomical features in bronchoscopic images, such as multi-level bifurcation, annular airways, and weak mucosal texture, are difficult to accurately capture by conventional convolutional layers, this embodiment adds a bronchial feature enhancement submodule dedicated to bronchial feature extraction after the C3 structure module in the latter part of the YOLO backbone network.

[0075] This bronchial feature enhancement submodule comprises the following two-layer structure:

[0076] The first layer consists of 3×3 convolutional kernels, used to extract fine-grained spatial features such as bronchial wall edges, bifurcation point contours, and mucosal surface textures; specifically, upon receiving feature maps from the YOLO backbone network C3 module... Then, a convolution operation is performed on it using a 3×3 convolution kernel, that is, the receptive field of the 3×3 pixel convolution kernel is used to cover the local region, and the feature map is output. Then the feature map The Batch Normalization layer normalizes the features of each channel, stabilizing the training process and accelerating convergence. Then, a ReLU activation function is used to introduce non-linearity, preserving positive response features and suppressing irrelevant background information, thereby outputting the feature map. .

[0077] The second layer uses a 1×1 convolutional kernel to compress feature channels while preserving spatial distribution information. This reduces computational cost without altering the spatial dimensions of the feature map, compressing only the channel dimension of each pixel. This preserves the spatial distribution information of the bronchial structure, such as the relative positions of bifurcation points and the direction of the lumen. Specifically, upon receiving the feature map output from the first layer... Then, channel compression is performed using a 1×1 convolution kernel to reduce the number of channels from... Down to ,For example Output feature map Then the feature map The Batch Normalization layer normalizes the features of each channel, stabilizing the training process and accelerating convergence. Then, a ReLU activation function is used to introduce non-linearity, preserving positive response features and suppressing irrelevant background information, thereby outputting the feature map. .

[0078] After that, The enhanced features are fused with feature maps from other layers of the backbone network (such as shallow high-resolution features and deep semantic features) and then fed into the YOLO detection head. Based on the enhanced features, the detection head locates and classifies targets such as abnormal points (such as tumors, lesions, and airway obstruction) and key tissues (such as bifurcation points, cartilage, and main bronchus) in the bronchoscopy image, and outputs an image containing labels such as target coordinates and categories. For example, the target can be selected using bounding boxes and labeled with corresponding tags on one side of the recognition bounding box.

[0079] In this embodiment, the bronchial feature enhancement submodule is used to improve the network's ability to discriminate complex bronchial structures and low-contrast lesion areas. This module effectively improves the average detection accuracy of the YOLO model on the bronchoscopic image dataset and enhances its recognition performance of important structures such as the main bronchus, bifurcation points, and airway obstructions.

[0080] S104: Navigation calculations are performed using Markov chains and dynamic programming algorithms, and a recommended route image is output.

[0081] In one possible implementation, the navigation calculation in step S104 includes the following steps:

[0082] Step 1041: Construct the transition probability matrix based on the Markov chain;

[0083] Step 1042: Using a dynamic programming algorithm, a cost function is established by combining the transition probability matrix, outlier risk function, physiological characteristic risk function, and physical distance to solve for the optimal path.

[0084] The step 1041, which involves establishing the transition probability matrix based on the Markov chain, includes:

[0085] Step 1041a: Establish a set of bronchial states , Among them, the set of bronchial states Each element in the bronchial tree represents a different discrete location point in the bronchial tree, and n is the total number of discrete location points in the bronchial tree.

[0086] Step 1041b: For the bronchial state set Any two elements in and Calculate from arrive transition probability ,

[0087] ;in, Let i be the i-th discrete location point in the bronchial tree. Let i be the j-th discrete location point in the bronchial tree, and both i and j belong to the range 1 to n. for arrive Number of transfers, From Total number of transfers from the starting point;

[0088] Step 1041c: Establish the transition probability matrix based on the calculation results. ,in:

[0089] ;

[0090] Each element satisfies the following condition:

[0091] .

[0092] Here, the bronchial tree is discretized into several states. Each state Each corresponds to a location in the bronchial tree; specifically, the state. It can be divided according to anatomical structure and adjusted according to the complexity of the bronchial model, for example:

[0093] .

[0094] In one possible implementation, the bronchial state set any element in Includes the following information: Coordinates in three-dimensional space From the starting point to The Euclidean distance is used to calculate from arrive physical distance ;from Total number of departures Used to calculate from arrive transition probability ; arrive Risk coefficient of outliers on the path Used to calculate from arrive Outlier risk function on the path ; Actual average airway diameter and the average diameter of a normal airway Used to calculate from arrive Physiological characteristic risk function .

[0095] In this embodiment of the application, a bronchial state set containing multidimensional anatomical-physiological information is constructed. This breakthrough overcomes the limitations of traditional Markov chain single geometric modeling, upgrading the state definition of Markov chains from simple spatial coordinates to a composite model that integrates structural features, functional attributes, and risk characteristics. This provides a more comprehensive decision-making basis for navigation systems and improves the rationality and safety of path planning in complex bronchial environments.

[0096] The step 1042, which involves using a dynamic programming algorithm to establish a cost function by combining the transition probability matrix, outlier risk function, physiological characteristic risk function, and physical distance to solve for the optimal path, includes:

[0097] Step 1042a: Establish an outlier risk function based on outliers identified by YOLO. ,in, , For path Anomalies on The risk coefficient for outliers is used to quantify the severity of outliers. outlier to path The distance;

[0098] Step 1042b: Establish a physiological feature risk function based on real-time bronchial images , , For path segment The actual average diameter of the airway ,in and These were obtained through real-time images of the bronchi. and The actual diameter of the airway; For path segment The normal average diameter of the airway ,in and They are and The diameter of the tube in a normal airway;

[0099] Step 1042c: Based on the transition probability matrix The transition probability in outlier risk function and physiological characteristic risk function Establish a single transfer cost function ,in,

[0100] in, for arrive physical distance, for arrive The transition probability, for arrive outlier risk function, for arrive Physiological characteristic risk function, , and These are weighting coefficients used to balance the transition probabilities. outlier risk function and physiological characteristic risk function For a single transition cost function The impact;

[0101] This application incorporates anomaly risk functions and physiological feature risk functions with real-time detection into the navigation calculation, enabling the transition matrix to reflect real-time changes in the intraoperative airway environment. This achieves automatic suppression of high-risk paths and priority selection of safe paths, giving the navigation system dynamic decision-making capabilities to adapt to complex lesion scenarios and improving the reliability of surgical path planning.

[0102] Step 1042d: Based on the single-transition cost function Establish the minimum cumulative cost function Thus, the optimal path is found, where...

[0103] , ;

[0104] in, This is the current state. for The next state, Initial state to state The minimum cumulative cost, Initial state To the current state The optimal cost is the cost of calculating the optimal solution to the subproblem. Current state arrive The cost of a single transfer.

[0105] In this embodiment, the navigation calculation process incorporates real-time detection information and physiological feature assessment into the dynamic programming solution by constructing a multi-level cost function system. The anomaly risk function, based on real-time YOLO identification results, quantifies the impact of anomaly severity and spatial distance on the path, ensuring that the transfer cost reflects the real-time distribution of lesions during surgery. The physiological feature risk function, using bronchial diameter data extracted from real-time bronchial images, assesses the physiological passage cost caused by differences in airway patency, ensuring that path selection conforms to airway functional characteristics. These two functions, along with transfer probability and physical distance, constitute a single transfer cost function, which is then optimized globally through a minimum cumulative cost function. This multi-factor fusion modeling approach enables the navigation system to dynamically integrate anatomical structures, physiological states, and real-time risk information, automatically suppressing high-risk transfer paths and prioritizing safe paths that conform to physiological rationality during path planning, effectively improving the comprehensiveness and reliability of navigation decisions in complex lesion environments.

[0106] S105: Overlay the bronchial 3D model and the recommended route image to obtain a first overlay image; overlay the bronchial real-time image and the anomaly marker image to obtain a second overlay image.

[0107] In one possible implementation, the bronchoscopic path planning method further includes: after obtaining the first superimposed image and the second superimposed image, performing data parsing on the first superimposed image and the second superimposed image, outputting them to a head-mounted augmented reality device, and displaying them in the user's field of vision.

[0108] Specifically, users of the head-mounted augmented reality device can simultaneously see a first overlay image and a second overlay image. In the first overlay image, the 3D model 7 of the bronchus is presented in a semi-transparent, three-dimensional form, while the recommended route image 6 is overlaid as highlighted lines within the bronchial lumen of the 3D model. This clearly displays the navigation path from the current location to the target area (e.g., ...). Figure 6 As shown); in the second overlay image, the real-time bronchial image floats on one side of the field of view in the form of a rectangular window. The key tissue identification box 2, key tissue label 3, abnormal point identification box 4, and abnormal point label 5 within the edge 1 of the real-time bronchial image are overlaid in real-time on the corresponding area of ​​the real-time bronchial image, intuitively marking the location and type of abnormal points such as tumors and bifurcation points, as well as key tissues (e.g., Figure 7 (As shown).

[0109] Here, the user's field of vision also includes a semi-transparent control panel, which floats near the 3D model of the bronchus. The user can select functions and operate the system by looking at the virtual buttons on the control panel. The virtual buttons include: a first virtual button, displayed at the top of the control panel, for starting the QR code scanning operation; a second virtual button, displayed below the first virtual button, for importing the 3D model operation; a third virtual button, displayed below the second virtual button, for starting the anomaly identification operation; and a fourth virtual button, displayed below the third virtual button, for exiting the system operation.

[0110] Users interact with the system via virtual buttons on the eye-tracking gaze control panel. This hands-free interaction method aligns with aseptic surgical procedures, reducing the risk of intraoperative contamination. The virtual buttons are arranged vertically according to their functional logic. The first virtual button triggers QR code scanning to obtain preoperative patient data; the second virtual button supports dynamic import of intraoperative 3D models; the third virtual button initiates real-time anomaly detection; and the fourth virtual button is used to safely exit the system. The hierarchical layout of the function keys conforms to clinical operating procedures, allowing surgeons to quickly control the system while maintaining surgical posture. This achieves an organic integration of image navigation and interactive operation, providing a smooth and interference-free visualization and control interface for bronchoscopy.

[0111] like Figure 4 As shown, based on the same application concept, this application embodiment also provides an augmented reality-based bronchoscopy navigation system, including a first acquisition module, a second acquisition module, an anomaly recognition module, a navigation calculation module, a data processing module, and an output module; wherein the first acquisition module is used to acquire a three-dimensional model of the bronchus; the second acquisition module is used to acquire real-time images of the bronchus; the anomaly recognition module is used to perform anomaly recognition on the real-time images of the bronchus and output anomaly marker images; the navigation calculation module is used to perform navigation calculations and output recommended route images; the data processing module is used to overlay the three-dimensional model of the bronchus and the recommended route images to obtain a first overlay image and to overlay the real-time images of the bronchus and the anomaly marker images to obtain a second overlay image; the output module is used to parse the first overlay image and the second overlay image and output them to a head-mounted augmented reality device.

[0112] Based on the same concept, embodiments of this application also provide a computer device, including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by a processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the above-described augmented reality-based bronchoscopy path planning method when executing the instructions.

[0113] Based on the same concept, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described augmented reality-based bronchoscopy path planning method.

[0114] In this embodiment, the computer program, when run by the processor, can also execute other machine-readable instructions to perform other methods as described in the embodiments. For details on the specific execution steps and principles, please refer to the description of the embodiments, which will not be repeated here.

[0115] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0116] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0117] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0118] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0119] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0120] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. An augmented reality-based bronchoscopy path planning system, comprising: The system comprises: a first acquisition module for acquiring a bronchial three-dimensional model; a second acquisition module for acquiring a bronchial real-time image; an anomaly identification module for identifying anomalies in the bronchial real-time image by a YOLO algorithm and outputting an anomaly marked image; the anomaly identification module identifies anomalies in the bronchial real-time image by the YOLO algorithm, including: pre-labeling and training a YOLO model using a plurality of bronchoscope videos and CT images, so that the YOLO model can be used to identify abnormal points and the positions of key tissues in the bronchus, wherein the abnormal points include any one or more of the following: tumors, lesions, airway obstruction; the key tissues include any one or more of the following: bifurcation points, cartilage, main bronchus; inputting the bronchial real-time image into the YOLO model for identification, and inferring the abnormal points and the key tissues in the current frame; a navigation calculation module for navigation calculation by a Markov chain and dynamic programming algorithm and outputting a recommended route image; a data processing module for superimposing the bronchial three-dimensional model and the recommended route image to obtain a first superimposed image, and superimposing the bronchial real-time image and the anomaly marked image to obtain a second superimposed image; an output module for data analysis of the first superimposed image and the second superimposed image and outputting to a head-mounted augmented reality device.

2. The augmented reality-based bronchoscope path planning system of claim 1, wherein, The navigation calculation module calculates the navigation by the Markov chain and the dynamic programming algorithm, including: establishing a transition probability matrix according to the Markov chain; establishing a cost function by a dynamic programming algorithm, combining the transition probability matrix, an abnormal point risk function, a physiological feature risk function, and a physical distance to solve an optimal path.

3. The augmented reality-based bronchoscope path planning system of claim 2, wherein, The step of establishing the transition probability matrix according to the Markov chain includes: establishing a bronchial state set , wherein each element in the bronchial state set is a different discrete location point in a bronchial tree, and n is a total number of discrete location points in the bronchial tree. For any two elements of the bronchial state set and , a transition probability from to is computed,​ ； wherein, is the i-th discrete location point in the bronchial tree, is the j-th discrete location point in the bronchial tree, and i and j both belong to 1 to n, is to the number of transitions, is the total number of transitions from outset; The transition probability matrix is established according to the calculation results wherein: ； wherein each element satisfies the following conditions: 。 4. The augmented reality-based bronchoscope path planning system of claim 3, wherein, The step of establishing the cost function by the dynamic programming algorithm, combining the transition probability matrix, the abnormal point risk function, the physiological feature risk function, and the physical distance to solve the optimal path includes: Abnormal point risk function based on yolo recognition , ; wherein, is an abnormal point on a path , is a risk coefficient of the abnormal point, for quantifying the severity of the abnormal point, is a distance from the abnormal point to the path ; establishing a physiological characteristic risk function based on the real-time bronchial image , ; wherein is the actual airway mean tube diameter for the path segment ​ ； wherein, and are the actual airway diameters at and respectively, as obtained from the real-time image of the bronchus; is the normal airway average diameter for the path segment . ； wherein and are respectively and the tube diameter of a normal airway; based on the transition probabilities in the transition probability matrix establishing a single transition cost function :​​​​ ； wherein, is the physical distance, to is the transition probability, to is the outlier risk function, to is the physiological feature risk function, , , and are weight coefficients for balancing the influence of the transition probability , the outlier risk function and the physiological feature risk function on the single transition cost function . Based on a single transfer cost function Establishing a minimum cumulative cost function solving the optimal path, wherein, ， ； wherein, is the current state, is the next state, is the start state to state the minimum cumulative cost, is the initial state to the current state the optimal cost, i.e. the optimal solution of the subproblem has been computed, is the current state to the one-step transition cost.

5. The augmented reality-based bronchoscope path planning system of claim 4, wherein, any of the bronchial states includes the following information:​ Coordinates in three-dimensional space ; From the starting point to The Euclidean distance is used to calculate from arrive physical distance ; The total number of transitions from outset , for calculating the transition probabilities from to ;​ to risk factor for an abnormal point on a path , for calculating a risk function for an abnormal point on a path from to an abnormal point on a path ; Actual airway mean tube diameter and normal airway mean tube diameter , for calculating a physiological characteristic risk function from to .​ 6. The augmented reality-based bronchoscope path planning system of claim 1, wherein: After the output module obtains the first superimposed image and the second superimposed image, the output module performs data analysis on the first superimposed image and the second superimposed image, outputs them to a head-mounted augmented reality device, and displays them in the user's field of view.

7. A computer device, comprising: The system comprises: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the one processor, and the instructions are executed by the at least one processor to cause the at least one processor to execute the instructions to implement the system of any one of claims 1 to 6.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the system of any one of claims 1 to 6.

Images