AR navigation bronchoscope system capable of monitoring blind area in real time based on artificial intelligence

Abstract

The invention discloses an AR (Augmented Reality) navigation bronchoscope system capable of monitoring a blind area in real time based on artificial intelligence, relates to the technical field of bronchoscope navigation, and solves the problems that a navigation route is difficult to adjust in time according to various endoscope images acquired in real time in different bronchus, and meanwhile, the blind area can be monitored in real time. The interference of breathing factors on the movement of the bronchus endoscope is not considered, so that the operation is easy to carry out in improper time, and the operation risk is improved. By combining bronchial characteristic data extracted by the DCNN image processing module, the DRL navigation module can optimize path planning in real time and adjust a navigation strategy according to the current bronchial condition, and by analyzing and optimizing the bronchial characteristic data, the DRL navigation module can generate personalized path planning to adapt to bronchial structures and lesion conditions of different patients; interference on the respiratory system of the patient can be reduced by monitoring the respiratory state and adjusting the navigation strategy, and the operation risk is reduced.

Description

Technical Field

[0001] The present invention belongs to the field of bronchoscopic navigation, and specifically is an artificial intelligence-based AR navigation bronchoscope system capable of real-time monitoring of blind spots. Background Art

[0002] Endoscopic surgery plays an important role in the diagnosis and treatment of respiratory diseases. However, due to the complex, narrow and curved structure of the bronchus, there are often blind spots during the operation, making it difficult for doctors to directly observe the target area. Traditional endoscopic operations rely on the doctor's experience and feel, and are prone to problems such as accidental stabs. Therefore, the development of an AR navigation bronchoscope system based on artificial intelligence technology can effectively solve these problems. The AR navigation bronchoscope system based on artificial intelligence that can monitor blind spots in real time will bring a more intelligent, precise and safe operating experience to endoscopic surgery, and is expected to become an important technological innovation in the diagnosis and treatment of respiratory diseases in the future.

[0003] Most existing navigation bronchoscope systems only determine the position through endoscopic images. It is difficult to adjust the navigation route in time according to the various endoscopic images collected in real time inside different bronchi. At the same time, they do not take into account the interference of respiratory factors on the movement of bronchoscopes, making it easy to operate at inappropriate times, increasing operational risks. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art; to this end, the present invention proposes an AR navigation bronchoscope system based on artificial intelligence that can monitor blind spots in real time, which is used to solve the problem that it is difficult to adjust the navigation route in time according to a variety of endoscopic images collected in real time inside different bronchi. At the same time, the interference of respiratory factors on the movement of the bronchoscope is not taken into account, and it is easy to operate at an inappropriate time, which increases the technical problem of operational risks.

[0005] In order to solve the above problems, the first aspect of the present invention provides an AR navigation bronchoscope system based on artificial intelligence that can monitor blind spots in real time, comprising:

[0006] Model building module: used to obtain 3D scanning images of the bronchi and to build a 3D virtual model of the bronchi through the 3D scanning images;

[0007] Numbering module: classifying the bronchi in the three-dimensional virtual model of bronchus according to the established three-dimensional virtual model of bronchus, and numbering the bronchi in the three-dimensional virtual model of bronchus according to the classification of the bronchi;

[0008] Navigation path generation module: Based on the established bronchial 3D virtual model and bronchial number, the shortest path planning to reach the target area is generated through the path planning algorithm;

[0009] Video stream acquisition module: used to obtain video stream data collected by the endoscope;

[0010] DCNN image processing module: recognizes the internal image of the current bronchus and the image of the bifurcation of the bronchial opening connected to the current bronchus by training the DCNN deep convolutional neural network model, detects the target image, and extracts the characteristic data of the current bronchus from the recognized image;

[0011] DRL navigation module: used to navigate the endoscope according to the shortest path planning generated by the navigation path generation module, and optimize the shortest path planning according to the characteristic data of the current bronchus extracted by the DCNN image processing module;

[0012] Respiratory interference monitoring module: used to monitor the respiratory status of the subject in real time, evaluate the respiratory interference according to the patient's respiratory status, and adjust the navigation strategy of the DRL navigation module according to the respiratory interference.

[0013] As a further solution of the present invention: classifying the bronchi in the three-dimensional virtual model of the bronchi, and numbering the bronchi in the three-dimensional virtual model of the bronchi according to the classification of the bronchi, comprising the following steps:

[0014] Classify the bronchi in the 3D virtual model of the bronchi, including trachea, main bronchi, lobar bronchus, segmental bronchus, small bronchus, bronchioles, terminal bronchioles and respiratory bronchus;

[0015] According to the classification of bronchi, an identifier is set for each classification, and the bronchi under each classification are numbered;

[0016] The bronchi in the bronchial three-dimensional virtual model are numbered according to the identifier of the bronchial classification and the bronchial number under the classification.

[0017] As a further solution of the present invention: the navigation path generation module generates the shortest path planning to reach the target area according to the established bronchial three-dimensional virtual model and the bronchial number through the path planning algorithm, including the following steps:

[0018] Acquire the established bronchial three-dimensional virtual model and the bronchial number, locate the position of the target area in the bronchial three-dimensional virtual model, and use the located position as the end position and the position of the trachea in the bronchial classification as the starting position;

[0019] The Dijkstra shortest path algorithm is used to generate the shortest path planning from the end point to the starting point, and the bronchial number array corresponding to the passing order is generated.

[0020] As a further solution of the present invention: the DCNN image processing module recognizes the internal image of the current bronchus and the image of the bifurcation of the bronchial opening connected to the current bronchus by training the DCNN deep convolutional neural network model, and detects the target image, and extracts characteristic data of the current bronchus from the recognized image, including the following steps:

[0021] Acquire the internal image data of each numbered bronchus, the image of the bifurcation of the orifice of each numbered bronchus, and the detection target image in the historical record in the bronchoendoscopic historical image data;

[0022] In the internal image data of each numbered bronchus, the bronchial inner wall image is annotated, and the corresponding bronchial number is added as a label;

[0023] In the image of the bifurcation of the orifice of each numbered bronchus, the image of the bronchial orifice connected to the bifurcation is annotated, and an identifier of the current bronchial classification and the bronchial number corresponding to the annotated bronchial orifice image are added as labels;

[0024] Annotate the target images and target areas detected in the historical records, and add the target type as a label;

[0025] The labeled images are divided into a test set and a validation set, and the DCNN deep convolutional neural network model is trained using the test set and the validation set;

[0026] After the video stream data collected by the endoscope is processed in real time by the video stream acquisition module, it is sent to the trained DCNN deep convolutional neural network model to annotate the bronchial inner wall image, the bronchial orifice image connected to the bifurcation, and the target area, and the characteristic data of the current bronchus is extracted based on the annotated image.

[0027] As a further solution of the present invention: after the video stream data collected by the endoscope is processed in real time by the video stream acquisition module, it is sent to the trained DCNN deep convolutional neural network model to annotate the bronchial inner wall image, the bronchial orifice image connected to the bifurcation and the target area, and the characteristic data of the current bronchus is extracted according to the annotated image, including the following arrangement:

[0028] The video stream acquisition module acquires the video stream data acquired by the endoscope in real time, and extracts one frame of video frame data every five video frames;

[0029] Preprocessing the extracted video frame data, including denoising and contrast enhancement;

[0030] Input the preprocessed video frames into the trained DCNN deep convolutional neural network model;

[0031] The trained DCNN deep convolutional neural network model is used to annotate the bronchial wall images in the video frames and identify the corresponding bronchial numbers; the images of the bronchial orifices connected at the bifurcation are annotated, and the identifier of the current bronchial classification and the bronchial number corresponding to the annotated bronchial orifice images are identified; the detected target areas are annotated and the target types are identified;

[0032] Extract the current bronchial inner diameter data through the marked bronchial inner wall image; extract the inner diameter data of the bronchus connected at the bifurcation through the marked bronchial tube orifice image; extract the target area area data through the marked target area image;

[0033] The extracted current bronchial inner diameter data, the inner diameter data of the bronchus connected at the bifurcation, and the target region area data are used as characteristic data of the current bronchus.

[0034] As a further solution of the present invention: the DRL navigation module navigates the endoscope according to the shortest path planning generated by the navigation path generation module, and optimizes the shortest path planning according to the characteristic data of the current bronchus extracted by the DCNN image processing module, including the following steps:

[0035] The bronchial 3D virtual model is input into the DRL reinforcement learning algorithm model, and the state space is defined as the bronchial number corresponding to the current bronchial number and the bronchial orifice connected to the bifurcation, and the action space is defined as upward rotation, downward rotation, left rotation, right rotation, forward and backward;

[0036] Set the reward function to give positive rewards for successfully reaching the target area or finding a blind spot, and negative rewards for collisions;

[0037] The DRL navigation model is trained by using the bronchial 3D virtual model extracted from the historical image data of bronchoscopy, the state space and action space of the historical data, and the set reward function;

[0038] The trained DRL navigation model navigates the endoscope according to the shortest path planning generated by the navigation path generation module, detects the current moving speed of the bronchoscope, and adjusts the current moving speed of the bronchoscope according to the optimization scheme of the respiratory interference monitoring module;

[0039] According to the characteristic data of the current bronchus extracted by the DCNN image processing module, it is determined whether to perform navigation according to the shortest path planning generated by the navigation path generation model;

[0040] If navigation is not performed according to the shortest path planning, the current bronchial number is reported, and an error message is sent to the navigation path generation module to regenerate the shortest path planning;

[0041] If navigation is performed according to the shortest path planning, navigation will continue.

[0042] As a further solution of the present invention: judging whether to perform navigation according to the shortest path planning generated by the navigation path generation model based on the characteristic data of the current bronchus extracted by the DCNN image processing module, comprises the following steps:

[0043] Determine whether the target area image marked by the DCNN image processing module is an occluder;

[0044] If the target area image marked by the DCNN image processing module is an occluder, the allowable passing diameter is obtained by subtracting the diameter of the occluder from the current bronchial inner diameter data;

[0045] If the target area image marked by the DCNN image processing module is not an occluder, the current bronchial inner diameter data and the inner diameter data of the bronchus connected to the bifurcation are compared, and the smallest inner diameter data is selected as the allowable passing diameter;

[0046] When the allowable diameter is larger than the endoscope diameter, navigation is performed according to the shortest path planning generated by the navigation path generation model; otherwise, navigation is not performed according to the shortest path planning.

[0047] As a further solution of the present invention: the respiratory disturbance monitoring module monitors the respiratory state of the subject in real time and evaluates the respiratory disturbance according to the patient's respiratory state, including the following steps:

[0048] Real-time monitoring of the subject's respiratory status data, including the subject's respiratory rate, blood oxygen saturation and pulse rate;

[0049] Set a testing time period and divide the testing time period into several testing time intervals, and randomly select the breathing rate, blood oxygen saturation and pulse rate of the tested person at any time point in each testing time interval;

[0050] The volatility of respiratory status was assessed using the following formula:

[0051]

[0052] Wherein, R is the respiratory state volatility evaluation value, σ1 is the variance of the screened respiratory rate data in the detection period, μ1 is the average value of the screened respiratory rate data in the detection period, σ2 is the variance of the screened blood oxygen saturation data in the detection period, μ2 is the average value of the screened blood oxygen saturation data in the detection period, σ3 is the variance of the screened pulse rate data in the detection period, μ3 is the average value of the screened pulse rate data in the detection period;

[0053] Obtain the bronchial inner wall image and the bronchial orifice image connected to the bifurcation marked by the DCNN image processing module, and obtain the current bronchial inner diameter data and the inner diameter data of the bronchus connected to the bifurcation;

[0054] The influence coefficient of the inner diameter of the bronchus was evaluated using the following formula:

[0055]

[0056] Among them, E is the influence coefficient of the inner diameter of the bronchus, R1 is the inner diameter data of the current bronchus when the current bronchus is dilated, R2 is the inner diameter data of the current bronchus when the current bronchus is contracted, R0 is the average value of the inner diameter data of the current bronchus in the historical image data of bronchial endoscopy, Rao is the average value of the inner diameter data of the i-th bronchus connected at the bifurcation in the historical image data of bronchial endoscopy, Ra i is the inner diameter data of the i-th bronchus connected to the bifurcation when the current bronchus contracts, i∈(1,2,…,n), n is the total number of bronchi connected to the bifurcation;

[0057] According to the respiratory state volatility evaluation value and the bronchial inner diameter influence coefficient, the respiratory interference is evaluated by the following formula:

[0058]

[0059] Wherein, K is the respiratory disturbance assessment value.

[0060] As a further solution of the present invention: adjusting the navigation strategy of the DRL navigation module according to the respiratory disturbance includes the following steps:

[0061] Obtaining a respiratory disturbance evaluation result, obtaining a current moving speed of the bronchoscope, and taking one tenth of the current moving speed of the bronchoscope as a minimum speed adjustment unit;

[0062] If the respiratory disturbance assessment value K>50, the current moving speed of the bronchoscope is reduced by 5 minimum speed adjustment units;

[0063] If 30≤K<50, the current moving speed of the bronchoscope is reduced by 3 minimum speed adjustment units;

[0064] If 20≤K<30, the current moving speed of the bronchoscope is reduced by 2 minimum speed adjustment units;

[0065] If 10≤K<20, the current moving speed of the bronchoscope is reduced by 1 minimum speed adjustment unit;

[0066] If K<10, the current moving speed of the bronchoscope remains unchanged;

[0067] If K<0, the current moving speed of the bronchoscope is increased by 2 minimum speed adjustment units.

[0068] Compared with the prior art, the present invention has the following beneficial effects:

[0069] The present invention has excellent performance in image recognition through the DCNN deep convolutional neural network, and can efficiently and accurately identify the internal images of the bronchus and the bifurcation images at the connection; at the same time, the DCNN model can accurately locate the current internal structure of the bronchus and the bifurcation position of the connection, thereby realizing automated analysis of the internal structure of the bronchus.

[0070] The present invention can intelligently guide the endoscope to navigate according to the shortest path planning through DRL deep reinforcement learning technology, helping doctors reach the target area more quickly and accurately; combined with the bronchial characteristic data extracted by the DCNN image processing module, the DRL navigation module can optimize the path planning in real time and adjust the navigation strategy according to the current bronchial condition. At the same time, through the analysis and optimization of the bronchial characteristic data, the DRL navigation module can generate personalized path planning, adapt to the bronchial structure and pathological conditions of different patients, and provide more accurate navigation services.

[0071] The present invention facilitates providing personalized navigation strategies for each patient by real-time monitoring of the patient's respiratory status and evaluating respiratory disturbance, so as to better adapt to their physiological characteristics and conditions; timely monitoring of the respiratory status and adjusting the navigation strategy can reduce the interference to the patient's respiratory system and reduce the risk of operation. Adjusting the navigation strategy according to the patient's respiratory condition can avoid operating at inappropriate times, improve operational efficiency and reduce operation time; at the same time, by adjusting the navigation strategy according to the respiratory status, unnecessary interference and discomfort to the patient can be reduced, and the patient's comfort during the operation can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0073] Figure 1 Schematic diagram of the system framework of the present invention. DETAILED DESCRIPTION

[0074] The technical scheme of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0075] See also Figure 1 The first aspect of the present invention provides an AR navigation bronchoscope system based on artificial intelligence that can monitor blind spots in real time, including:

[0076] Model building module: used to obtain 3D scanning images of the bronchi and to build a 3D virtual model of the bronchi through the 3D scanning images;

[0077] Numbering module: classifying the bronchi in the three-dimensional virtual model of bronchus according to the established three-dimensional virtual model of bronchus, and numbering the bronchi in the three-dimensional virtual model of bronchus according to the classification of the bronchi;

[0078] Navigation path generation module: Based on the established bronchial 3D virtual model and bronchial number, the shortest path planning to reach the target area is generated through the path planning algorithm;

[0079] Video stream acquisition module: used to obtain video stream data collected by the endoscope;

[0080] DCNN image processing module: recognizes the internal image of the current bronchus and the image of the bifurcation of the bronchial opening connected to the current bronchus by training the DCNN deep convolutional neural network model, detects the target image, and extracts the characteristic data of the current bronchus from the recognized image;

[0081] DRL navigation module: used to navigate the endoscope according to the shortest path planning generated by the navigation path generation module, and optimize the shortest path planning according to the characteristic data of the current bronchus extracted by the DCNN image processing module;

[0082] Respiratory interference monitoring module: used to monitor the respiratory status of the subject in real time, evaluate the respiratory interference according to the patient's respiratory status, and adjust the navigation strategy of the DRL navigation module according to the respiratory interference.

[0083] Specifically, the bronchi in the three-dimensional virtual model of the bronchus are classified according to the established three-dimensional virtual model of the bronchus through the numbering module, and the bronchi in the three-dimensional virtual model of the bronchus are numbered according to the classification of the bronchi; the navigation path generation module generates the shortest path planning to reach the target area through the path planning algorithm according to the established three-dimensional virtual model of the bronchus and the numbering of the bronchi; by establishing a three-dimensional virtual model of the bronchus and classifying and numbering the bronchi, the system can more accurately identify and locate the bronchi in different parts, thereby generating a more accurate navigation path; at the same time, the path planning algorithm is used to quickly generate the shortest path planning to reach the target area in the three-dimensional virtual model of the bronchus.

[0084] The CNN image processing module is set to identify the internal image of the current bronchus and the image of the bifurcation of the bronchial orifice connected to the current bronchus by training the DCNN deep convolutional neural network model, as well as detect the target image, and extract the characteristic data of the current bronchus from the identified image; the DCNN deep convolutional neural network has excellent performance in image recognition, and can efficiently and accurately identify the internal image of the bronchus and the bifurcation image of the connection; at the same time, the DCNN model can accurately locate the current bronchial internal structure and the bifurcation position of the connection, thereby realizing the automated analysis of the internal structure of the bronchus.

[0085] The DRL navigation module navigates the endoscope according to the shortest path planning generated by the navigation path generation module, and optimizes the shortest path planning according to the current bronchial characteristic data extracted by the DCNN image processing module; the DRL deep reinforcement learning technology can intelligently guide the endoscope to navigate according to the shortest path planning, helping doctors reach the target area more quickly and accurately; combined with the bronchial characteristic data extracted by the DCNN image processing module, the DRL navigation module can optimize the path planning in real time and adjust the navigation strategy according to the current bronchial condition. At the same time, through the analysis and optimization of the bronchial characteristic data, the DRL navigation module can generate personalized path planning to adapt to the bronchial structure and lesion conditions of different patients, and provide more accurate navigation services.

[0086] The respiratory interference monitoring module monitors the respiratory status of the patient in real time, evaluates the respiratory interference according to the patient's respiratory status, and adjusts the navigation strategy of the DRL navigation module according to the respiratory interference; by real-time monitoring of the patient's respiratory status and evaluating respiratory interference, it is convenient to provide each patient with a personalized navigation strategy to better adapt to their physiological characteristics and conditions; timely monitoring of the respiratory status and adjustment of the navigation strategy can reduce interference with the patient's respiratory system and reduce operational risks. Adjusting the navigation strategy according to the patient's respiratory condition can avoid operating at inappropriate times, improve operational efficiency and reduce operating time; at the same time, by adjusting the navigation strategy according to the respiratory status, it can reduce unnecessary interference and discomfort to the patient and improve the patient's comfort during surgery.

[0087] In one embodiment of the present invention, the bronchi in the three-dimensional virtual model of the bronchi are classified, and the bronchi in the three-dimensional virtual model of the bronchi are numbered according to the classification of the bronchi, including the following steps:

[0088] Classify the bronchi in the 3D virtual model of the bronchi, including trachea, main bronchi, lobar bronchus, segmental bronchus, small bronchus, bronchioles, terminal bronchioles and respiratory bronchus;

[0089] According to the classification of bronchi, an identifier is set for each classification, and the bronchi under each classification are numbered;

[0090] The bronchi in the bronchial three-dimensional virtual model are numbered according to the identifier of the bronchial classification and the bronchial number under the classification.

[0091] Specifically, in this embodiment, the trachea identifier is A, the main bronchus class identifier is B, the lobar bronchus class identifier is C, the segmental bronchus class identifier is D, the small bronchus class identifier is E, the bronchioles class identifier is F, the terminal bronchioles class identifier is G, and the respiratory bronchus class identifier is H;

[0092] Main bronchial tubes, including:

[0093] Right mainstem bronchus: Branches off the right side of the trachea and enters the right lung.

[0094] Left mainstem bronchus: Branches off from the left side of the trachea and enters the left lung.

[0095] Lobar bronchial type, including:

[0096] Right upper lobe bronchus: enters the right upper lobe of the lung.

[0097] Right middle lobe bronchus: enters the right middle lobe of the lung.

[0098] Right lower lobe bronchus: enters the right lower lobe of the lung.

[0099] Left upper lobe bronchus: enters the left upper lobe of the lung.

[0100] Left lower lobe bronchus: enters the left lower lobe of the lung.

[0101] Segmental bronchi: Lobar bronchi further branch into segmental bronchi. These bronchi are numerous and vary according to the anatomical structure of the lung lobes. For example, the upper lobe bronchus of the right lung can be further divided into upper, middle and lower segments; the upper lobe bronchus of the left lung can also be divided into upper and lower segments, etc. In terms of specific number, there are about 8 segmental bronchi in the left lung and about 10 in the right lung.

[0102] Small bronchi: Segmental bronchi further branch into small bronchi.

[0103] Bronchioles: Small bronchi branch into bronchioles.

[0104] Terminal bronchioles: The bronchioles further branch into terminal bronchioles.

[0105] Respiratory bronchial tubes: the branches below the terminal bronchioles are the respiratory part of the lungs;

[0106] In this embodiment, the right main bronchus is numbered B-001, and the left main bronchus is numbered B-002.

[0107] In one embodiment of the present invention, the navigation path generation module generates the shortest path planning to reach the target area according to the established bronchial three-dimensional virtual model and the bronchial number through a path planning algorithm, including the following steps:

[0108] Acquire the established bronchial three-dimensional virtual model and the bronchial number, locate the position of the target area in the bronchial three-dimensional virtual model, and use the located position as the end position and the position of the trachea in the bronchial classification as the starting position;

[0109] The Dijkstra shortest path algorithm is used to generate the shortest path planning from the end point to the starting point, and the bronchial number array corresponding to the passing order is generated.

[0110] In one embodiment of the present invention, the DCNN image processing module recognizes the internal image of the current bronchus and the image of the bifurcation of the bronchial opening connected to the current bronchus by training the DCNN deep convolutional neural network model, and detects the target image, and extracts characteristic data of the current bronchus from the recognized image, including the following steps:

[0111] Acquire the internal image data of each numbered bronchus, the image of the bifurcation of the orifice of each numbered bronchus, and the detection target image in the historical record in the bronchoendoscopic historical image data;

[0112] In the internal image data of each numbered bronchus, the bronchial inner wall image is annotated, and the corresponding bronchial number is added as a label;

[0113] In the image of the bifurcation of the orifice of each numbered bronchus, the image of the bronchial orifice connected to the bifurcation is annotated, and an identifier of the current bronchial classification and the bronchial number corresponding to the annotated bronchial orifice image are added as labels;

[0114] Annotate the target images and target areas detected in the historical records, and add the target type as a label;

[0115] The labeled images are divided into a test set and a validation set, and the DCNN deep convolutional neural network model is trained using the test set and the validation set;

[0116] After the video stream data collected by the endoscope is processed in real time by the video stream acquisition module, it is sent to the trained DCNN deep convolutional neural network model to annotate the bronchial inner wall image, the bronchial orifice image connected to the bifurcation, and the target area, and the characteristic data of the current bronchus is extracted based on the annotated image.

[0117] In one embodiment of the present invention, after the video stream data collected by the endoscope is processed in real time by the video stream acquisition module, it is sent to the trained DCNN deep convolutional neural network model to annotate the bronchial inner wall image, the bronchial orifice image connected to the bifurcation, and the target area, and the characteristic data of the current bronchus is extracted according to the annotated image, including the following arrangement:

[0118] The video stream acquisition module acquires the video stream data acquired by the endoscope in real time, and extracts one frame of video frame data every five video frames;

[0119] Preprocessing the extracted video frame data, including denoising and contrast enhancement;

[0120] Input the preprocessed video frames into the trained DCNN deep convolutional neural network model;

[0121] The trained DCNN deep convolutional neural network model is used to annotate the bronchial wall images in the video frames and identify the corresponding bronchial numbers; the images of the bronchial orifices connected at the bifurcation are annotated, and the identifier of the current bronchial classification and the bronchial number corresponding to the annotated bronchial orifice images are identified; the detected target areas are annotated and the target types are identified;

[0122] Extract the current bronchial inner diameter data through the marked bronchial inner wall image; extract the inner diameter data of the bronchus connected at the bifurcation through the marked bronchial tube orifice image; extract the target area area data through the marked target area image;

[0123] The extracted current bronchial inner diameter data, the inner diameter data of the bronchus connected at the bifurcation, and the target region area data are used as characteristic data of the current bronchus.

[0124] In one embodiment of the present invention, the DRL navigation module navigates the endoscope according to the shortest path planning generated by the navigation path generation module, and optimizes the shortest path planning according to the characteristic data of the current bronchus extracted by the DCNN image processing module, including the following steps:

[0125] The bronchial 3D virtual model is input into the DRL reinforcement learning algorithm model, and the state space is defined as the bronchial number corresponding to the current bronchial number and the bronchial orifice connected to the bifurcation, and the action space is defined as upward rotation, downward rotation, left rotation, right rotation, forward and backward;

[0126] Set the reward function to give positive rewards for successfully reaching the target area or finding a blind spot, and negative rewards for collisions;

[0127] The DRL navigation model is trained by using the bronchial 3D virtual model extracted from the historical image data of bronchoscopy, the state space and action space of the historical data, and the set reward function;

[0128] The trained DRL navigation model navigates the endoscope according to the shortest path planning generated by the navigation path generation module, detects the current moving speed of the bronchoscope, and adjusts the current moving speed of the bronchoscope according to the optimization scheme of the respiratory interference monitoring module;

[0129] According to the characteristic data of the current bronchus extracted by the DCNN image processing module, it is determined whether to perform navigation according to the shortest path planning generated by the navigation path generation model;

[0130] If navigation is not performed according to the shortest path planning, the current bronchial number is reported, and an error message is sent to the navigation path generation module to regenerate the shortest path planning;

[0131] If navigation is performed according to the shortest path planning, navigation will continue.

[0132] Specifically, in this embodiment, a reward function is set to give positive rewards for successfully reaching the target area or finding a blind spot, and negative rewards for collisions, which is implemented by the following code:

[0133] def reward_funct i on(state,act i on,next_state):

[0134] #Define the target location

[0135]

[0136] In one embodiment of the present invention, judging whether to perform navigation according to the shortest path planning generated by the navigation path generation model based on the characteristic data of the current bronchus extracted by the DCNN image processing module includes the following steps:

[0137] Determine whether the target area image marked by the DCNN image processing module is an occluder;

[0138] If the target area image marked by the DCNN image processing module is an occluder, the allowable passing diameter is obtained by subtracting the diameter of the occluder from the current bronchial inner diameter data;

[0139] If the target area image marked by the DCNN image processing module is not an occluder, the current bronchial inner diameter data and the inner diameter data of the bronchus connected to the bifurcation are compared, and the smallest inner diameter data is selected as the allowable passing diameter;

[0140] When the allowable diameter is larger than the endoscope diameter, navigation is performed according to the shortest path planning generated by the navigation path generation model; otherwise, navigation is not performed according to the shortest path planning.

[0141] In one embodiment of the present invention, the respiratory disturbance monitoring module monitors the respiratory state of the subject in real time and evaluates the respiratory disturbance according to the patient's respiratory state, including the following steps:

[0142] Real-time monitoring of the subject's respiratory status data, including the subject's respiratory rate, blood oxygen saturation and pulse rate;

[0143] Set a testing time period and divide the testing time period into several testing time intervals, and randomly select the breathing rate, blood oxygen saturation and pulse rate of the tested person at any time point in each testing time interval;

[0144] The volatility of respiratory status was assessed using the following formula:

[0145]

[0146] Wherein, R is the respiratory state volatility evaluation value, σ1 is the variance of the screened respiratory rate data in the detection period, μ1 is the average value of the screened respiratory rate data in the detection period, σ2 is the variance of the screened blood oxygen saturation data in the detection period, μ2 is the average value of the screened blood oxygen saturation data in the detection period, σ3 is the variance of the screened pulse rate data in the detection period, μ3 is the average value of the screened pulse rate data in the detection period;

[0147] Obtain the bronchial inner wall image and the bronchial orifice image connected to the bifurcation marked by the DCNN image processing module, and obtain the current bronchial inner diameter data and the inner diameter data of the bronchus connected to the bifurcation;

[0148] The influence coefficient of the inner diameter of the bronchus was evaluated using the following formula:

[0149]

[0150] Among them, E is the influence coefficient of the inner diameter of the bronchus, R1 is the inner diameter data of the current bronchus when the current bronchus is dilated, R2 is the inner diameter data of the current bronchus when the current bronchus is contracted, R0 is the average value of the inner diameter data of the current bronchus in the historical image data of bronchial endoscopy, Rao is the average value of the inner diameter data of the i-th bronchus connected at the bifurcation in the historical image data of bronchial endoscopy, Ra i is the inner diameter data of the i-th bronchus connected to the bifurcation when the current bronchus contracts, i∈(1,2,…,n), n is the total number of bronchi connected to the bifurcation;

[0151] According to the respiratory state volatility evaluation value and the bronchial inner diameter influence coefficient, the respiratory interference is evaluated by the following formula:

[0152]

[0153] Wherein, K is the respiratory disturbance assessment value.

[0154] In one embodiment of the present invention, adjusting the navigation strategy of the DRL navigation module according to the respiratory disturbance includes the following steps:

[0155] Obtaining a respiratory disturbance evaluation result, obtaining a current moving speed of the bronchoscope, and taking one tenth of the current moving speed of the bronchoscope as a minimum speed adjustment unit;

[0156] If the respiratory disturbance assessment value K>50, the current moving speed of the bronchoscope is reduced by 5 minimum speed adjustment units;

[0157] If 30≤K<50, the current moving speed of the bronchoscope is reduced by 3 minimum speed adjustment units;

[0158] If 20≤K<30, the current moving speed of the bronchoscope is reduced by 2 minimum speed adjustment units;

[0159] If 10≤K<20, the current moving speed of the bronchoscope is reduced by 1 minimum speed adjustment unit;

[0160] If K<10, the current moving speed of the bronchoscope remains unchanged;

[0161] If K<0, the current moving speed of the bronchoscope increases by 2 minimum speed adjustment units.

[0162] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. An AR navigation bronchoscope system based on artificial intelligence that can monitor blind spots in real time, characterized in that: include: Model building module: used to obtain 3D scanning images of the bronchi and to build a 3D virtual model of the bronchi through the 3D scanning images; Numbering module: classifying the bronchi in the three-dimensional virtual model of bronchus according to the established three-dimensional virtual model of bronchus, and numbering the bronchi in the three-dimensional virtual model of bronchus according to the classification of the bronchi; Navigation path generation module: Based on the established bronchial 3D virtual model and bronchial number, the shortest path planning to reach the target area is generated through the path planning algorithm; Video stream acquisition module: used to obtain video stream data collected by the endoscope; DCNN image processing module: recognizes the internal image of the current bronchus and the image of the bifurcation of the bronchial opening connected to the current bronchus by training the DCNN deep convolutional neural network model, detects the target image, and extracts the characteristic data of the current bronchus from the recognized image; DRL navigation module: used to navigate the endoscope according to the shortest path planning generated by the navigation path generation module, and optimize the shortest path planning according to the characteristic data of the current bronchus extracted by the DCNN image processing module; Respiratory interference monitoring module: used to monitor the respiratory status of the subject in real time, evaluate the respiratory interference according to the patient's respiratory status, and adjust the navigation strategy of the DRL navigation module according to the respiratory interference.

2. According to claim 1, an artificial intelligence-based AR navigation bronchoscope system capable of real-time monitoring of blind spots is characterized in that: Classifying the bronchi in the three-dimensional virtual model of the bronchi and numbering the bronchi in the three-dimensional virtual model of the bronchi according to the classification of the bronchi includes the following steps: Classify the bronchi in the 3D virtual model of the bronchi, including trachea, main bronchi, lobar bronchus, segmental bronchus, small bronchus, bronchioles, terminal bronchioles and respiratory bronchus; According to the classification of bronchi, an identifier is set for each classification, and the bronchi under each classification are numbered; The bronchi in the bronchial three-dimensional virtual model are numbered according to the identifier of the bronchial classification and the bronchial number under the classification.

3. According to claim 1, an artificial intelligence-based AR navigation bronchoscope system capable of real-time monitoring of blind spots is characterized in that: The navigation path generation module generates the shortest path planning to reach the target area according to the established bronchial three-dimensional virtual model and the bronchial number through a path planning algorithm, including the following steps: Acquire the established bronchial three-dimensional virtual model and the bronchial number, locate the position of the target area in the bronchial three-dimensional virtual model, and use the located position as the end position and the position of the trachea in the bronchial classification as the starting position; The Dijkstra shortest path algorithm is used to generate the shortest path planning from the end point to the starting point, and the bronchial number array corresponding to the passing order is generated.

4. According to claim 1, an artificial intelligence-based AR navigation bronchoscope system capable of real-time monitoring of blind spots is characterized in that: The DCNN image processing module recognizes the internal image of the current bronchus and the image of the bifurcation of the bronchial opening connected to the current bronchus by training the DCNN deep convolutional neural network model, detects the target image, and extracts characteristic data of the current bronchus from the recognized image, including the following steps: Acquire the internal image data of each numbered bronchus, the image of the bifurcation of the orifice of each numbered bronchus, and the detection target image in the historical record in the bronchoendoscopic historical image data; In the internal image data of each numbered bronchus, the bronchial inner wall image is annotated, and the corresponding bronchial number is added as a label; In the image of the bifurcation of the orifice of each numbered bronchus, the image of the bronchial orifice connected to the bifurcation is annotated, and an identifier of the current bronchial classification and the bronchial number corresponding to the annotated bronchial orifice image are added as labels; Annotate the target images and target areas detected in the historical records, and add the target type as a label; The labeled images are divided into a test set and a validation set, and the DCNN deep convolutional neural network model is trained using the test set and the validation set; After the video stream data collected by the endoscope is processed in real time by the video stream acquisition module, it is sent to the trained DCNN deep convolutional neural network model to annotate the bronchial inner wall image, the bronchial orifice image connected to the bifurcation, and the target area, and the characteristic data of the current bronchus is extracted based on the annotated image.

5. According to claim 4, an artificial intelligence-based AR navigation bronchoscope system capable of real-time monitoring of blind spots is characterized in that: After the video stream data collected by the endoscope is processed in real time by the video stream acquisition module, it is sent to the trained DCNN deep convolutional neural network model to annotate the bronchial wall image, the bronchial orifice image connected to the bifurcation, and the target area, and the characteristic data of the current bronchus is extracted based on the annotated image, including the following arrangements: The video stream acquisition module acquires the video stream data acquired by the endoscope in real time, and extracts one frame of video frame data every five video frames; Preprocessing the extracted video frame data, including denoising and contrast enhancement; Input the preprocessed video frames into the trained DCNN deep convolutional neural network model; The trained DCNN deep convolutional neural network model is used to annotate the bronchial wall images in the video frames and identify the corresponding bronchial numbers; the images of the bronchial orifices connected at the bifurcation are annotated, and the identifier of the current bronchial classification and the bronchial number corresponding to the annotated bronchial orifice images are identified; the detected target areas are annotated and the target types are identified; Extract the current bronchial inner diameter data through the marked bronchial inner wall image; extract the inner diameter data of the bronchus connected at the bifurcation through the marked bronchial tube opening image; Extracting target area data through the marked target area image; The extracted current bronchial inner diameter data, the inner diameter data of the bronchus connected at the bifurcation, and the target region area data are used as characteristic data of the current bronchus.

6. The AR navigation bronchoscope system based on artificial intelligence and capable of real-time monitoring of blind spots according to claim 5, characterized in that: The DRL navigation module navigates the endoscope according to the shortest path planning generated by the navigation path generation module, and optimizes the shortest path planning according to the characteristic data of the current bronchus extracted by the DCNN image processing module, including the following steps: The bronchial 3D virtual model is input into the DRL reinforcement learning algorithm model, and the state space is defined as the bronchial number corresponding to the current bronchial number and the bronchial orifice connected to the bifurcation, and the action space is defined as upward rotation, downward rotation, left rotation, right rotation, forward and backward; Set the reward function to give positive rewards for successfully reaching the target area or finding a blind spot, and negative rewards for collisions; The DRL navigation model is trained by using the bronchial 3D virtual model extracted from the historical image data of bronchoscopy, the state space and action space of the historical data, and the set reward function; The trained DRL navigation model navigates the endoscope according to the shortest path planning generated by the navigation path generation module, detects the current moving speed of the bronchoscope, and adjusts the current moving speed of the bronchoscope according to the optimization scheme of the respiratory interference monitoring module; According to the characteristic data of the current bronchus extracted by the DCNN image processing module, it is determined whether to perform navigation according to the shortest path planning generated by the navigation path generation model; If navigation is not performed according to the shortest path planning, the current bronchial number is reported, and an error message is sent to the navigation path generation module to regenerate the shortest path planning; If navigation is performed according to the shortest path planning, navigation will continue.

7. The AR navigation bronchoscope system based on artificial intelligence and capable of real-time monitoring of blind spots according to claim 6, characterized in that: According to the characteristic data of the current bronchus extracted by the DCNN image processing module, it is determined whether to perform navigation according to the shortest path planning generated by the navigation path generation model, including the following steps: Determine whether the target area image marked by the DCNN image processing module is an occluder; If the target area image marked by the DCNN image processing module is an occluder, the allowable passing diameter is obtained by subtracting the diameter of the occluder from the current bronchial inner diameter data; If the target area image marked by the DCNN image processing module is not an occluder, the current bronchial inner diameter data and the inner diameter data of the bronchus connected to the bifurcation are compared, and the smallest inner diameter data is selected as the allowable passing diameter; When the allowable diameter is larger than the endoscope diameter, navigation is performed according to the shortest path planning generated by the navigation path generation model; otherwise, navigation is not performed according to the shortest path planning.

8. The AR navigation bronchoscope system based on artificial intelligence and capable of real-time monitoring of blind spots according to claim 5, characterized in that: The respiratory disturbance monitoring module monitors the respiratory state of the subject in real time and evaluates the respiratory disturbance according to the patient's respiratory state, including the following steps: Real-time monitoring of the subject's respiratory status data, including the subject's respiratory rate, blood oxygen saturation and pulse rate; Set a testing time period and divide the testing time period into several testing time intervals, and randomly select the breathing rate, blood oxygen saturation and pulse rate of the tested person at any time point in each testing time interval; The volatility of respiratory status was assessed using the following formula: Wherein, R is the respiratory state volatility evaluation value, σ1 is the variance of the screened respiratory rate data in the detection period, μ1 is the average value of the screened respiratory rate data in the detection period, σ2 is the variance of the screened blood oxygen saturation data in the detection period, μ2 is the average value of the screened blood oxygen saturation data in the detection period, σ3 is the variance of the screened pulse rate data in the detection period, μ3 is the average value of the screened pulse rate data in the detection period; Obtain the bronchial inner wall image and the bronchial orifice image connected to the bifurcation marked by the DCNN image processing module, and obtain the current bronchial inner diameter data and the inner diameter data of the bronchus connected to the bifurcation; The influence coefficient of the inner diameter of the bronchus was evaluated using the following formula: Among them, E is the influence coefficient of the inner diameter of the bronchus, R1 is the inner diameter data of the current bronchus when the current bronchus is dilated, R2 is the inner diameter data of the current bronchus when the current bronchus is contracted, R0 is the average value of the inner diameter data of the current bronchus in the historical image data of bronchial endoscopy, Rao is the average value of the inner diameter data of the i-th bronchus connected at the bifurcation in the historical image data of bronchial endoscopy, Ra i is the inner diameter data of the i-th bronchus connected to the bifurcation when the current bronchus contracts, i∈(1,2,…,n), n is the total number of bronchi connected to the bifurcation; According to the respiratory state fluctuation evaluation value and the bronchial inner diameter influence coefficient, the respiratory interference is evaluated by the following formula: Wherein, K is the respiratory disturbance assessment value.

9. The AR navigation bronchoscope system based on artificial intelligence and capable of real-time monitoring of blind spots according to claim 8, characterized in that: The navigation strategy of the DRL navigation module is adjusted according to the respiratory disturbance, including the following steps: Obtaining a respiratory disturbance evaluation result, obtaining a current moving speed of the bronchoscope, and taking one tenth of the current moving speed of the bronchoscope as a minimum speed adjustment unit; If the respiratory disturbance assessment value K>50, the current moving speed of the bronchoscope is reduced by 5 minimum speed adjustment units; If 30≤K<50, the current moving speed of the bronchoscope is reduced by 3 minimum speed adjustment units; If 20≤K<30, the current moving speed of the bronchoscope is reduced by 2 minimum speed adjustment units; If 10≤K<20, the current moving speed of the bronchoscope is reduced by 1 minimum speed adjustment unit; If K<10, the current moving speed of the bronchoscope remains unchanged; If K<0, the current moving speed of the bronchoscope increases by 2 minimum speed adjustment units.

Images