Bronchoscope path planning method, system and device based on augmented reality and storage medium

Abstract

The invention provides a bronchoscope path planning method, system and device based on augmented reality and a storage medium. The method comprises the steps of obtaining a bronchus three-dimensional model; acquiring a bronchus real-time image; outputting an abnormal mark image through a YOLO algorithm; outputting a recommended route image through a Markov chain and a dynamic programming algorithm; superposing the bronchial three-dimensional model, the bronchial real-time image, the abnormal mark image and the recommended route image, and displaying in the visual field of the user; the bronchus three-dimensional model and the recommended route image are displayed in an overlapped mode, and the bronchus real-time image and the abnormal mark image are displayed in an overlapped mode. According to the method, through the YOLO algorithm and the Markov chain, dynamic response is made to abnormal conditions in the operation, and instant information feedback is provided for navigation; according to the method, the abnormal region is automatically detected through the YOLO algorithm, so that the marking reliability is improved; according to the invention, three-dimensional navigation display is realized through the augmented reality technology, and the operation efficiency in a complex environment is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical equipment, and in particular to a bronchoscopic path planning method, system, device and storage medium based on augmented reality. Background Art

[0002] A bronchoscope is a medical device used for the diagnosis and treatment of respiratory diseases. It is primarily suitable for a variety of diagnostic and treatment scenarios, including observation of lobar, segmental, and subsegmental bronchial lesions, biopsy sampling, bacteriological and cytological examinations. Through the attached biopsy sampling accessories, the bronchoscope can assist in the detection of early lesions and simultaneously perform procedures such as tissue biopsy, secretion aspiration, and foreign body removal. This device, through a flexible or rigid scope design and combined with real-time image transmission technology, provides clinicians with diagnostic evidence and treatment options for respiratory diseases, effectively improving the diagnostic accuracy and targeted treatment of respiratory diseases, and has become a key diagnostic and treatment tool in the field of modern respiratory medicine.

[0003] Because the bronchial tubes are complex and have many branches (such as Figure 5 Therefore, before the bronchoscope enters, the doctor needs to plan the path based on the pre-acquired CT images and manually mark the abnormal points to provide auxiliary guidance for the operation during the operation, reduce the possibility of getting lost in the complex bronchial passages, and ensure that the bronchoscope reaches the target area smoothly.

[0004] However, this method has the following significant flaws: 1) This method has obvious lags and cannot be adjusted in real time according to the actual situation during surgery; 2) Manually marked abnormal points are prone to human errors and mistakes, resulting in increased surgical risks; 3) Traditional path planning is based on two-dimensional plane navigation design through CT tomographic images. Doctors need to construct the three-dimensional structure of the airway in their minds by stitching together multi-level images. The understanding of spatial relationships is not intuitive enough, which can easily lead to anatomical structure misjudgment and operational errors.

[0005] Therefore, how to design an augmented reality-based bronchoscopic path planning method, system, device and storage medium that can automatically mark abnormal points and update path planning in real time is a technical problem that has not yet been solved in the existing technology. Summary of the Invention

[0006] Therefore, the technical problem to be solved by the present invention is to overcome the technical defects of the existing technology due to reliance on pre-acquired CT images for manual marking and path planning, which leads to lack of real-time performance and easy occurrence of path deviation, thereby providing an augmented reality-based bronchoscopic path planning method, system, device and storage medium that can automatically mark and update path planning in real time.

[0007] This application mainly includes the following aspects: In a first aspect, an embodiment of the present application provides a bronchoscopic path planning method based on augmented reality, the method comprising: Acquire a three-dimensional bronchial model; acquire a real-time bronchial image; the real-time bronchial image is obtained by bronchoscope photography; perform abnormality identification on the real-time bronchial image using the YOLO algorithm and output an abnormality-marked image; perform navigation calculation using a Markov chain and a dynamic programming algorithm and output a recommended route image; superimpose the three-dimensional bronchial model and the recommended route image to obtain a first superimposed image; and superimpose the real-time bronchial image and the abnormality-marked image to obtain a second superimposed image.

[0008] According to one embodiment of the present invention, the step of performing navigation calculation through Markov chain and dynamic programming algorithm includes: establishing a transition probability matrix based on the Markov chain; establishing a cost function through the dynamic programming algorithm, combining the transition probability matrix, outlier risk function, physiological characteristic risk function and physical distance to solve the optimal path.

[0009] According to one embodiment of the present invention, the step of establishing a transition probability matrix according to the Markov chain includes: establishing a bronchial state set , , wherein the bronchial state set Each element in is a different discrete location point in the bronchial tree. is the total number of discrete locations in the bronchial tree; For the bronchial state set Any two elements in and , calculated from arrive The transition probability , ;in, The first discrete locations, The first discrete locations, and and All belong to 1 to , for arrive The number of transfers, For The total number of transfers started; establish the transfer probability matrix based on the calculation results ,in: ; Each element satisfies the following conditions: , .

[0010] According to one embodiment of the present invention, the step of establishing a cost function by combining the transfer probability matrix, the outlier risk function, the physiological characteristic risk function and the physical distance through a dynamic programming algorithm to solve the optimal path includes: establishing an outlier risk function based on the outliers identified by YOLO , ;in, For path The abnormal point on is the risk coefficient of the abnormal point, which is used to quantify the severity of the abnormal point. The abnormal point To the path Establishing a physiological characteristic risk function based on the real-time bronchial image , ;in, For path segments The actual average airway diameter; ;in, and The real-time images of the bronchus are obtained respectively. and The actual diameter of the airway; For path segments The average diameter of normal airways; ;in and They are and The diameter of the normal airway; Based on the transition probability matrix The transition probability in , outlier risk function , physiological characteristic risk function and physical distance Establish a single transfer cost function : ;in, for arrive physical distance, for arrive The transition probability, for arrive The outlier risk function, for arrive The physiological characteristic risk function, 、 and is the weight coefficient used to balance the transition probability , the outlier risk function and the physiological characteristic risk function For the single transfer cost function The impact of; based on the single transfer cost function Establishing the minimum cumulative cost function , thus solving the optimal path, where , ;in, is the current state, for The next state of is the starting state To status The minimum cumulative cost, Initial state To the current state The optimal cost of the subproblem, that is, the optimal solution of the subproblem has been calculated, Current status arrive The cost of a single transfer.

[0011] According to one embodiment of the present invention, the bronchial state set Any element in Include the following information: Coordinates in three-dimensional space From the starting point to The Euclidean distance from arrive Physical distance ;from Total number of transfers departing , used to calculate from arrive The transition probability ; arrive Risk factor of abnormal points on the path , used to calculate from arrive Outlier risk function on the path ; Actual average airway diameter and the average diameter of normal airways , used to calculate from arrive Physiological characteristic risk function .

[0012] According to one embodiment of the present invention, the step of identifying abnormalities in the real-time bronchial image using the YOLO algorithm includes: pre-labeling and training the YOLO model using multiple bronchoscopic videos and CT images so that the YOLO model can be used to identify abnormal points and the locations of key tissues in the bronchi, wherein the abnormal points include any one or more of the following: tumors, lesions, and airway obstruction; the key tissues include any one or more of the following: bifurcation points, cartilage, and main bronchus; the real-time bronchial image is input into the YOLO model for identification, and the abnormal points and key tissues in the current image are inferred.

[0013] According to one embodiment of the present invention, it also includes: after obtaining the first overlay image and the second overlay image, performing data analysis on the first overlay image and the second overlay image, outputting them to a head-mounted augmented reality device, and displaying them in the user's field of view.

[0014] In the second aspect, an embodiment of the present application also provides a bronchoscopic navigation system based on augmented reality, the system comprising: a first acquisition module: for acquiring a three-dimensional model of the bronchus; a second acquisition module: for acquiring a real-time image of the bronchus; an abnormality recognition module: for performing abnormality recognition on the real-time image of the bronchus and outputting an abnormality mark image; a navigation calculation module: for performing navigation calculation and outputting a recommended route image; a data processing module: for superimposing the three-dimensional model of the bronchus and the recommended route image to obtain a first superimposed image and superimposing the real-time image of the bronchus and the abnormality mark image to obtain a second superimposed image; an output module: for performing data analysis on the first superimposed image and the second superimposed image, and outputting the data to a head-mounted augmented reality device.

[0015] In a third aspect, an embodiment of the present application further provides a computer device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the one processor, and the instructions are executed by the at least one processor so that the at least one processor implements the steps of the method described above when executing the instructions.

[0016] In a third aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described above are implemented.

[0017] The technical solution provided by the present invention has the following advantages: 1. The present invention utilizes real-time images collected by bronchoscopes and, through the synergistic effect of the YOLO algorithm and Markov chain, dynamically responds to airway abnormalities during surgery, provides immediate information feedback for navigation, and completes path adjustment in a short time, so that navigation information remains consistent with the operation scenario.

[0018] 2. The present invention uses the YOLO algorithm to replace manual recognition and labeling, automatically detecting abnormal areas such as tumors, lesions, and airway obstruction, which can avoid labeling errors caused by human factors and improve the reliability of labeling.

[0019] 3. The present invention realizes three-dimensional navigation display through augmented reality technology, superimposing the three-dimensional model of the bronchus and the recommended route image in the user's field of view, enabling doctors to establish spatial cognition of the anatomical structure more accurately during the operation, thereby improving the efficiency of surgery in complex bronchial branching environments.

[0020] 4. This paper adds a bronchial feature enhancement module to the YOLO backbone network to improve the network's ability to discriminate complex bronchial structures and low-contrast lesions. This module effectively improves the average detection accuracy of the YOLO model on bronchoscopic image datasets and enhances its recognition of important structures such as main bronchi, bifurcations, and airway obstructions.

[0021] 5. The present invention breaks through the limitations of the traditional Markov chain's single geometric modeling by constructing a bronchial state set containing multi-dimensional anatomical and physiological information. The state definition of the Markov chain is upgraded from simple spatial coordinates to a composite model that integrates structural characteristics, functional attributes and risk characteristics, providing a more comprehensive decision-making basis for the navigation system and improving the rationality and safety of path planning in complex bronchial environments.

[0022] 6. The present invention adds real-time detection of abnormal point risk functions and physiological characteristic risk functions to the navigation calculation, so that the transfer matrix can reflect the real-time changes of the airway environment during surgery, realize the automatic suppression of high-risk paths and the priority selection of safe paths, and give the navigation system dynamic decision-making capabilities to adapt to complex pathology scenarios, thereby improving the reliability of surgical path planning.

[0023] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 This is an overall flow chart of the bronchoscopic path planning method based on augmented reality of the present invention.

[0026] Figure 2 This is a flow chart of the present invention for obtaining a three-dimensional bronchial model.

[0027] Figure 3 This is a flowchart of the present invention using the YOLO algorithm to perform abnormality recognition on real-time bronchial images.

[0028] Figure 4 This is an architectural diagram of the bronchoscopic navigation system based on augmented reality of the present invention.

[0029] Figure 5 This is a diagram of the structure of the bronchi in the lungs.

[0030] Figure 6 is a schematic diagram of a first superimposed image of the present invention.

[0031] Figure 7 is a schematic diagram of the second superimposed image of the present invention.

[0032] Figure 1: 1. Real-time bronchial image edge; 2. Key tissue identification frame; 3. Key tissue label; 4. Abnormal point identification frame; 5. Abnormal point label; 6. Recommended route image; 7. 3D bronchial model. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of illustration and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowcharts can be implemented out of sequence, and steps without logical context can be reversed or implemented simultaneously. In addition, those skilled in the art, under the guidance of the contents of this application, can add one or more other operations to the flowchart, or remove one or more operations from the flowchart.

[0034] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0035] The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments. Example

[0036] like Figure 1 As shown, this embodiment provides a bronchoscopic path planning method based on augmented reality, comprising the following steps: S101: Obtain a 3D bronchial model; S102: Acquire a real-time bronchial image; the real-time bronchial image is obtained by taking a picture with a bronchoscope; S103: performing abnormality recognition on the real-time bronchial image using a YOLO algorithm, and outputting an abnormality marked image; S104: Perform navigation calculations using a Markov chain and dynamic programming algorithm, and output a recommended route image; S105: superimposing the bronchial three-dimensional model and the recommended route image to obtain a first superimposed image; superimposing the bronchial real-time image and the abnormality mark image to obtain a second superimposed image.

[0037] The following is a detailed description of the steps of the bronchoscopic path planning method based on augmented reality: S101: Obtain a 3D bronchial model; like Figure 2 As shown, the step of obtaining the bronchial three-dimensional model in step S101 includes the following steps: Step 1011: Scan the QR code using a head-mounted augmented reality device; Step 1012: Parsing the online database based on the QR code to obtain a link to the bronchial 3D model; Step 1013: Downloading and storing the bronchial 3D model in a built-in sandbox environment of the head-mounted augmented reality device based on the bronchial 3D model link; Step 1014: In the built-in sandbox environment, the bronchial 3D model is converted into a rendering mesh that can be displayed normally, and the number of polygons is reduced to increase the mesh reconstruction speed.

[0038] For example, the head-mounted augmented reality device uses the HoloLens device developed by Microsoft, which supports augmented reality (AR) and can suspend a virtual 3D model of the bronchus next to a real patient. Doctors can perform air operations directly through gestures or eye gaze, such as rotating, zooming, and dragging, to further observe the details of the model.

[0039] Specifically, the hospital has a medical records database website that contains each patient's personal information, medical records, 3D models, and other information. Doctors can use the HoloLens device to scan a QR code containing the patient's information and the URL (Uniform Resource Locator) of the 3D model data. The HoloLens device then automatically retrieves the JSON file containing the patient's relevant data from the hospital's medical records database website, extracts the download link for the required 3D model (such as STL or OBJ format), and then downloads and stores these model files. The 3D model is then converted into a rendering mesh suitable for display in a virtual environment. The specific steps include: After obtaining the 3D model file, the Assimp library is usually used to parse it because the STL model has a specific format. Specifically, the Assimp library can convert models in specific formats such as STL into general geometric data, which contains information such as vertices, faces, and materials. After obtaining the general geometry data, the QEM (Quadric Error Metrics) simplification algorithm is used to process it. This algorithm calculates an error matrix to merge similar vertices. While maintaining the overall shape of the model, it reduces the number of redundant polygons and optimizes the general geometry data, generating more concise geometry data that retains key features. This significantly improves the display quality and rendering speed of the model in the virtual environment. Then, Unity's built-in Mesh processing library is used to further process the geometric data optimized by the QEM algorithm; the Mesh processing library will adapt these geometric data to Unity's Mesh data structure, such as converting vertex coordinates into coordinates in the Unity coordinate system, and filling the vertex and triangle index data into the corresponding fields of Unity Mesh. After such processing, the three-dimensional model is converted into mesh data suitable for rendering and display in the Unity virtual environment, so that it can be used in the HoloLens device.

[0040] It should be noted that the bronchial three-dimensional model obtained in step S101 is a "one-person-one-model" model, that is, it is modeled using a segmentation algorithm based on the patient's CT image before surgery, and then uploaded to the medical record database website set up by the hospital. It is an existing technology well known to technicians in this field.

[0041] S102: Acquire real-time bronchial images. Real-time bronchial images are obtained by bronchoscope. Specifically, real-time bronchi images are collected through a medical electronic bronchoscope. The bronchoscope is integrated with a high-definition camera (resolution ≥1920×1080, frame rate ≥30fps), supports multiple imaging modes such as white light and narrow-band imaging (NBI), and can clearly capture the bronchial mucosal texture, lumen morphology and abnormal lesion characteristics.

[0042] S103: using the YOLO algorithm to identify abnormalities in the real-time bronchial image and output an abnormality marked image; like Figure 3 As shown, in a possible implementation, performing abnormality recognition on the real-time bronchial image using the YOLO algorithm and outputting the abnormality marked image in step S103 includes the following steps: Step 1031: Pre-labeling and training a YOLO model using multiple bronchoscopic videos and CT images enables the YOLO model to identify abnormal points and key tissue locations in the bronchi, wherein abnormal points include any one or more of the following: tumors, lesions, and airway obstruction; and key tissues include any one or more of the following: bifurcations, cartilage, and main bronchus. Step 1032: Convert the YOLO model to an ONNX model for inference in other frameworks and platforms. Step 1033: Deploy the ONNX model on the head-mounted augmented reality device and select the corresponding inference library in Unity; Step 1034: Input the real-time bronchial image into the ONNX model for recognition, and infer abnormal points and key tissues in the current image.

[0043] It should be noted that in order to achieve cross-platform inference of the YOLO model on head-mounted augmented reality devices, the trained YOLO model needs to be converted into a model in ONNX (Open Neural Network Exchange) format, which supports migration between different frameworks and hardware platforms.

[0044] To improve the YOLO algorithm's object recognition performance in bronchoscopic images, this embodiment makes targeted adaptive improvements to YOLO's neural network architecture. Specifically, because common anatomical features in bronchoscopic images, such as multi-level bifurcations, circular airways, and weak mucosal texture, are difficult to accurately capture with conventional convolutional layers, this embodiment adds a bronchial feature enhancement submodule dedicated to bronchial feature extraction to the latter part of YOLO's backbone network, namely, after the C3 structural module.

[0045] The bronchial feature enhancer module contains the following two layers: The first layer is a 3×3 convolution kernel, which is used to extract fine-grained spatial features such as bronchial wall edges, bifurcation point contours, and mucosal surface textures. Specifically, after receiving the feature map from the C3 module of the YOLO backbone network, Then, a convolution operation is performed on it through a 3×3 convolution kernel, that is, the receptive field of 3×3 pixels of the convolution kernel is used to cover the local area and output the feature map ; Then the feature map The Batch Normalization layer normalizes the features of each channel to stabilize the training process and accelerate convergence; then the ReLU activation function is used to introduce nonlinearity, retain positive response features, suppress irrelevant background information, and output feature maps. .

[0046] The second layer is a 1×1 convolution kernel, which is used to compress the feature channel and retain the spatial distribution information. While reducing the amount of calculation, it does not change the spatial size of the feature map, but only compresses the channel dimension of each pixel, thereby retaining the spatial distribution information of the bronchial structure, such as the relative position of the bifurcation point, the direction of the lumen, etc. Specifically, after receiving the feature map output by the first layer, After that, the channel is compressed by 1×1 convolution kernel, and the number of channels is reduced from down to ,For example , output feature map ; Then the feature map The Batch Normalization layer normalizes the features of each channel to stabilize the training process and accelerate convergence; then the ReLU activation function is used to introduce nonlinearity, retain positive response features, suppress irrelevant background information, and output feature maps. .

[0047] Afterwards, It is fused with feature maps of other layers of the backbone network (such as shallow high-resolution features and deep semantic features), and then enters the detection head of YOLO; based on the enhanced features, the detection head locates and classifies abnormal points (such as tumors, lesions, airway obstruction) and key tissues (such as bifurcation points, cartilage, main bronchi) in the bronchoscopic image and other targets, and outputs an image containing labeled results such as target coordinates and categories. For example, the target can be selected using a bounding box and the corresponding label can be marked on one side of the recognition box.

[0048] In this application example, the bronchial feature enhancement submodule is used to improve the network's ability to discriminate complex bronchial structures and low-contrast lesions. This module effectively improves the average detection accuracy of the YOLO model on a bronchoscopic image dataset, enhancing its recognition of important structures such as main bronchi, bifurcations, and airway obstructions.

[0049] S104: Perform navigation calculations using a Markov chain and dynamic programming algorithm, and output a recommended route image; In a possible implementation, the navigation calculation in step S104 includes the following steps: Step 1041: Establish a transition probability matrix based on the Markov chain; Step 1042: Using a dynamic programming algorithm, a cost function is established by combining the transition probability matrix, the outlier risk function, the physiological characteristic risk function, and the physical distance to solve the optimal path.

[0050] The step of establishing a transition probability matrix according to the Markov chain in step 1041 includes: Step 1041a: Create a bronchial state set , , wherein the bronchial state set Each element in is a different discrete location point in the bronchial tree. is the total number of discrete locations in the bronchial tree; Step 1041b: For the bronchial state set Any two elements in and , calculated from arrive The transition probability , ;in, The first discrete locations, The first discrete locations, and and All belong to 1 to , for arrive The number of transfers, For Total number of transfers departing; Step 1041c: Establish a transition probability matrix based on the calculation results ,in: ; Each element satisfies the following conditions: , .

[0051] Here, the bronchial tree is discretized into several states , each state Each corresponds to a location in the bronchial tree; specifically, the status It can be divided according to the anatomical structure and adjusted according to the complexity of the bronchial model, for example: .

[0052] In one possible embodiment, the bronchial state set Any element in Include the following information: Coordinates in three-dimensional space From the starting point to The Euclidean distance from arrive Physical distance ;from Total number of transfers departing , used to calculate from arrive The transition probability ; arrive Risk factor of abnormal points on the path , used to calculate from arrive Outlier risk function on the path ; Actual average airway diameter and the average diameter of normal airways , used to calculate from arrive Physiological characteristic risk function .

[0053] In the embodiment of the present application, a bronchial state set containing multi-dimensional anatomical-physiological information is constructed. It breaks through the limitations of the traditional single geometric modeling of the Markov chain and upgrades the state definition of the Markov chain from simple spatial coordinates to a composite model that integrates structural characteristics, functional attributes and risk characteristics, providing a more comprehensive decision-making basis for the navigation system and improving the rationality and safety of path planning in complex bronchial environments.

[0054] The step of establishing a cost function by combining the transition probability matrix, the outlier risk function, the physiological characteristic risk function, and the physical distance through a dynamic programming algorithm in step 1042 to solve the optimal path includes: Step 1042a: Establish an outlier risk function based on the outliers identified by YOLO ,in , For path The abnormal point on is the risk coefficient of the abnormal point, which is used to quantify the severity of the abnormal point. The abnormal point To the path distance; Step 1042b: Establishing a physiological characteristic risk function based on the real-time bronchial image , , For path segments The actual average airway diameter, ;in and The real-time images of the bronchus are obtained respectively. and The actual diameter of the airway; For path segments The average diameter of the normal airway, ,in and They are and The diameter of the normal airway; Step 1042c: Based on the transition probability matrix The transition probability in , outlier risk function , physiological characteristic risk function and physical distance Establish a single transfer cost function ,in, ,in, for arrive physical distance, for arrive The transition probability, for arrive The outlier risk function, for arrive The physiological characteristic risk function, 、 and is the weight coefficient used to balance the transition probability , the outlier risk function and the physiological characteristic risk function For the single transfer cost function the impact of; The embodiment of the present application adds an abnormal point risk function and a physiological characteristic risk function including real-time detection to the navigation calculation, so that the transfer matrix can reflect the real-time changes of the airway environment during surgery, realize the automatic suppression of high-risk paths and the priority selection of safe paths, and give the navigation system dynamic decision-making capabilities to adapt to complex lesion scenarios, thereby improving the reliability of surgical path planning.

[0055] Step 1042d: Based on a single transfer cost function Establishing the minimum cumulative cost function , thus solving the optimal path, where , ; in, is the current state, for The next state of is the starting state To status The minimum cumulative cost, Initial state To the current state The optimal cost of the subproblem, that is, the optimal solution of the subproblem has been calculated, Current status arrive The cost of a single transfer.

[0056] In the navigation calculation process of the embodiment of the present application, a multi-level cost function system is constructed to integrate real-time detection information and physiological characteristic evaluation into the dynamic programming solution process. The outlier risk function is based on the real-time recognition results of YOLO, quantifying the impact of the severity of the outliers and the spatial distance on the path, so that the transfer cost can reflect the real-time distribution of intraoperative lesions; the physiological characteristic risk function uses the diameter data extracted from the real-time bronchial image to evaluate the physiological passage cost caused by the difference in airway patency to ensure that the path selection conforms to the functional characteristics of the airway. The two together with the transfer probability and physical distance constitute a single transfer cost function, and then global optimization is achieved through the minimum cumulative cost function. This multi-factor fusion modeling method enables the navigation system to dynamically integrate anatomical structure, physiological state and real-time risk information, automatically suppress high-risk transfer paths in path planning, and give priority to safe paths that are in line with physiological rationality, effectively improving the comprehensiveness and reliability of navigation decisions in complex lesion environments.

[0057] S105: superimposing the bronchial three-dimensional model and the recommended route image to obtain a first superimposed image; superimposing the bronchial real-time image and the abnormality mark image to obtain a second superimposed image.

[0058] In a possible embodiment, the bronchoscopic path planning method further includes: after obtaining the first overlay image and the second overlay image, performing data analysis on the first overlay image and the second overlay image, outputting them to a head-mounted augmented reality device, and displaying them in the user's field of view.

[0059] Specifically, a user using a head-mounted augmented reality device can simultaneously see the first superimposed image and the second superimposed image, wherein the bronchial three-dimensional model 7 in the first superimposed image is presented in a semi-transparent three-dimensional form, and the recommended route image 6 is superimposed and displayed in the bronchial lumen of the three-dimensional model in the form of highlighted lines, which can clearly show the navigation path from the current position to the target area (such as Figure 6 The real-time bronchial image in the second superimposed image is displayed as a rectangular window on one side of the visual field. The key tissue identification frame 2, key tissue label 3, abnormal point identification frame 4 and abnormal point label 5 within the edge 1 of the real-time bronchial image are superimposed and displayed in real time on the corresponding area of ​​the real-time bronchial image, and the location and type of abnormal points such as tumors and bifurcation points and key tissues are intuitively marked (such as Figure 7 shown).

[0060] Here, the user's field of view also includes a translucent operation panel, which is suspended near the bronchial three-dimensional model; the user can stare at the virtual buttons on the operation panel to select functions and operate the system; the virtual buttons include: a first virtual button, displayed at the top of the operation panel, for starting to scan the QR code; a second virtual button, displayed below the first virtual button, for importing the three-dimensional model; a third virtual button, displayed below the second virtual button, for starting abnormality identification; a fourth virtual button, displayed below the third virtual button, for exiting the system.

[0061] Users interact with functions by gazing at virtual buttons on the operation panel. This interaction method requires no hand contact, conforms to aseptic surgical procedures, and reduces the risk of intraoperative contamination. The virtual buttons are arranged vertically according to their functional logic. The first virtual button triggers QR code scanning to obtain preoperative patient data, the second virtual button supports dynamic import of 3D models during surgery, the third virtual button initiates real-time anomaly recognition, and the fourth virtual button is used to safely exit the system. The hierarchical layout of the function keys aligns with clinical operational procedures, allowing doctors to quickly control the system while maintaining surgical posture. This seamless integration of image navigation and interactive operations provides a smooth, non-interference visualization and control interface for bronchoscopic surgery.

[0062] like Figure 4 As shown, based on the same application concept, the embodiment of the present application also provides a bronchoscopic navigation system based on augmented reality, including a first acquisition module, a second acquisition module, an abnormality recognition module, a navigation calculation module, a data processing module and an output module; wherein the first acquisition module is used to acquire a three-dimensional model of the bronchus; the second acquisition module is used to acquire a real-time image of the bronchus; the abnormality recognition module is used to perform abnormality recognition on the real-time image of the bronchus and output an abnormality mark image; the navigation calculation module is used to perform navigation calculation and output a recommended route image; the data processing module is used to superimpose the three-dimensional model of the bronchus and the recommended route image to obtain a first superimposed image and to superimpose the real-time image of the bronchus and the abnormality mark image to obtain a second superimposed image; the output module is used to perform data analysis on the first superimposed image and the second superimposed image, and output them to a head-mounted augmented reality device.

[0063] Based on the same application concept, an embodiment of the present application also provides a computer device, including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by a processor, and the instructions are executed by the at least one processor so that the at least one processor implements the above-mentioned augmented reality-based bronchoscopic path planning method when executing the instructions.

[0064] Based on the same application concept, an embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned bronchoscopic path planning method based on augmented reality is implemented.

[0065] In the embodiment of the present application, the computer program can also execute other machine-readable instructions when run by the processor to execute other methods described in the embodiment. For the specific execution method steps and principles, please refer to the description of the embodiment and will not be repeated here.

[0066] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interface, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0067] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0068] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0069] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0070] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0071] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The scope of protection of the present application is not limited thereto. Although the present application has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed in the present application, or make equivalent replacements for some of the technical features thereof. However, these modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A bronchoscopic path planning method based on augmented reality, characterized in that: include: Obtain a three-dimensional model of the bronchus; Obtaining a real-time bronchial image; the real-time bronchial image is obtained by taking a bronchoscope; Perform abnormality recognition on the real-time bronchial image using the YOLO algorithm and output an abnormality marked image; Perform navigation calculations using Markov chains and dynamic programming algorithms, and output recommended route images; superimposing the bronchial three-dimensional model and the recommended route image to obtain a first superimposed image; The bronchial real-time image and the abnormality mark image are superimposed to obtain a second superimposed image.

2. The bronchoscopic path planning method based on augmented reality according to claim 1, characterized in that: The steps of performing navigation calculation by using a Markov chain and a dynamic programming algorithm include: Establish a transition probability matrix based on the Markov chain; A cost function is established by combining the transfer probability matrix, the outlier risk function, the physiological characteristic risk function and the physical distance through a dynamic programming algorithm to solve the optimal path.

3. The bronchoscopic path planning method based on augmented reality according to claim 2, characterized in that: The step of establishing a transition probability matrix according to the Markov chain comprises: Establishing bronchial state collection , , wherein the bronchial state set Each element in is a different discrete location point in the bronchial tree. is the total number of discrete locations in the bronchial tree; For the bronchial state set Any two elements in and , calculated from arrive The transition probability , ； in, The first discrete locations, The first discrete locations, and and All belong to 1 to , for arrive The number of transfers, For Total number of transfers departing; Establish a transition probability matrix based on the calculation results ,in: ； Each element satisfies the following conditions: ， 。 4. The bronchoscopic path planning method based on augmented reality according to claim 3, characterized in that: The step of establishing a cost function by combining the transition probability matrix, the outlier risk function, the physiological characteristic risk function, and the physical distance through a dynamic programming algorithm to solve the optimal path includes: Establishing anomaly risk function based on outliers identified by YOLO , ; in, For path The abnormal point on is the risk coefficient of the abnormal point, which is used to quantify the severity of the abnormal point. The abnormal point To the path distance; Establishing a physiological characteristic risk function based on the real-time bronchial image , ; in, For path segments The actual average airway diameter; ； in, and The real-time images of the bronchus are obtained respectively. and The actual diameter of the airway; For path segments The average diameter of normal airways; ； in and They are and The diameter of the normal airway; Based on the transition probability matrix The transition probability in , outlier risk function , physiological characteristic risk function and physical distance Establish a single transfer cost function : ； in, for arrive physical distance, for arrive The transition probability, for arrive The outlier risk function, for arrive The physiological characteristic risk function, 、 and is the weight coefficient used to balance the transition probability , the outlier risk function and the physiological characteristic risk function For the single transfer cost function the impact of; Based on the single transfer cost function Establishing the minimum cumulative cost function , thus solving the optimal path, where ， ； in, is the current state, for The next state of is the starting state To status The minimum cumulative cost, Initial state To the current state The optimal cost of the subproblem, that is, the optimal solution of the subproblem has been calculated, Current status arrive The cost of a single transfer.

5. The bronchoscopic path planning method based on augmented reality according to claim 4, characterized in that: The bronchial state set Any element in Include the following information: Coordinates in three-dimensional space ; From the starting point to The Euclidean distance from arrive Physical distance ; from Total number of transfers departing , used to calculate from arrive The transition probability ; arrive Risk factor of abnormal points on the path , used to calculate from arrive Outlier risk function on the path ; Actual average airway diameter and the average diameter of normal airways , used to calculate from arrive Physiological characteristic risk function .

6. The bronchoscopic path planning method based on augmented reality according to claim 1, characterized in that: The step of identifying abnormalities in the real-time bronchial image using the YOLO algorithm includes: The YOLO model is pre-labeled and trained using multiple bronchoscopic videos and CT images so that the YOLO model can be used to identify abnormal points and key tissue locations in the bronchi, wherein the abnormal points include any one or more of the following: tumors, lesions, and airway obstructions; and the key tissues include any one or more of the following: bifurcations, cartilage, and main bronchus. The real-time bronchial image is input into the YOLO model for recognition, and the abnormal points and the key tissues in the current image are inferred.

7. The bronchoscopic path planning method based on augmented reality according to claim 1, characterized in that: Also includes: After obtaining the first superimposed image and the second superimposed image, data analysis is performed on the first superimposed image and the second superimposed image, and the images are output to a head-mounted augmented reality device and displayed in a user's field of view.

8. A bronchoscopic navigation system based on augmented reality, characterized in that: include: The first acquisition module is used to obtain a 3D model of the bronchus; The second acquisition module is used to obtain real-time images of the bronchi; Abnormality recognition module: used for identifying abnormalities in the real-time bronchial image and outputting an abnormality marked image; Navigation calculation module: used to perform navigation calculations and output recommended route images; Data processing module: used for superimposing the bronchial three-dimensional model and the recommended route image to obtain a first superimposed image, and superimposing the bronchial real-time image and the abnormality mark image to obtain a second superimposed image; Output module: used to perform data analysis on the first overlay image and the second overlay image, and output the data to a head-mounted augmented reality device.

9. A computer device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the one processor, and the instructions are executed by the at least one processor so that the at least one processor implements the steps of the method according to any one of claims 1 to 7 when executing the instructions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images