A bronchoscope-assisted positioning method and system for tuberculosis care

Abstract

The present invention relates to the technical field of bronchial image analysis, and particularly relates to a bronchoscope-assisted positioning method and system for tuberculosis care. The method obtains an area correction amount based on the branch angle between the corresponding position of each frame of the internal gray-scale image of the bronchus in the virtual bronchial tree and its next bifurcation, and the area difference between the internal gray-scale image of the bronchus and the target hole area of the previous internal gray-scale image of the bronchus, and combines the distance between the corresponding position of the image in the virtual bronchial tree and its next bifurcation to correct the area of the target hole area to obtain the true area; according to the change of the true area of the target hole area of the internal gray-scale image of the bronchus between the corresponding position of the internal gray-scale image of the bronchus in the virtual bronchial tree and its previous bifurcation, it assists in positioning the bronchoscope. The present invention reduces the error in the recognition of abnormal structures in the bronchial airway and improves the accuracy of assisting in positioning the bronchoscope.

Description

Technical Field

[0001] The present invention relates to the technical field of bronchial image analysis, and in particular to a bronchoscope-assisted positioning method and system for tuberculosis care. Background Art

[0002] A bronchoscope is a medical instrument used to observe the internal structures of the bronchi and lungs. When there are abnormal structures such as tumors, foreign bodies, or tuberculosis in the bronchi, it may cause airway stenosis or blockage, increasing the difficulty of the bronchoscope passing through the bronchi. Therefore, it is necessary to assist doctors in adjusting the treatment path of the bronchoscope through early warning.

[0003] During the movement of the bronchoscope in the bronchi, the area of the region representing the hole formed by the bronchi in the images collected by the bronchoscope will change regularly due to the perspective phenomenon. The prior art determines whether there are abnormal structures in the bronchi by analyzing the degree of change rule of the hole area. However, because the bronchi have a tree-like structure, when the bronchoscope is about to reach the bifurcation of the bronchial tree, the angle between the bronchus where the bronchoscope is located and the next bronchus to be entered will block the hole, resulting in a sudden change in the area of the hole region. Moreover, abnormal structures in the bronchial passages will also cause a sudden change in the hole area, thus causing errors in the judgment of abnormal structures in the bronchial passages and reducing the accuracy of assisting in positioning the bronchoscope. Summary of the Invention

[0004] In order to solve the technical problem that the branch angle at the bifurcation of the bronchial tree and abnormal structures cause a sudden change in the area of the hole region formed by the bronchi, resulting in a low accuracy rate of identifying abnormal structures in the bronchial passages, the purpose of the present invention is to provide a bronchoscope-assisted positioning method and system for tuberculosis care, and the specific technical solutions adopted are as follows:

[0005] In a first aspect, an embodiment of the present invention provides a bronchoscope-assisted positioning method for tuberculosis care, and the method includes:

[0006] Real-time collecting video images of the bronchi of a user to be measured through a bronchoscope, and obtaining a virtual bronchial tree of the user to be measured;

[0007] Obtaining a target hole region of each frame of the internal grayscale image of the bronchi in the video image; according to the corresponding position of each frame of the internal grayscale image of the bronchi in the virtual bronchial tree, the branch angle of the next bifurcation at this position, and the area difference between the target hole region of each frame of the internal grayscale image of the bronchi and the target hole region of the internal grayscale image of the bronchi before it, obtaining the area correction amount of the target hole region of each frame of the internal grayscale image of the bronchi.

[0008] According to the distance between the corresponding position of each frame of the internal bronchial gray-scale image in the virtual bronchial tree and its next bifurcation, and the area correction amount, correct the area of the target hole region to obtain the true area of the target hole region of each frame of the internal bronchial gray-scale image;

[0009] According to the distance between the corresponding position of the last frame of the internal bronchial gray-scale image in the virtual bronchial tree and its next bifurcation, and the change of the true area of the target hole region of the internal bronchial gray-scale image between the corresponding position of the last frame of the internal bronchial gray-scale image in the virtual bronchial tree and its previous bifurcation, assist the bronchoscope in positioning.

[0010] Further, the obtaining of the target hole region of each frame of the internal bronchial gray-scale image in the video image includes:

[0011] For each frame of the internal bronchial gray-scale image, use the maximum inter-class variance method to obtain the segmentation threshold for the gray-scale values of the pixel points in the internal bronchial gray-scale image, and use the connected domain composed of the pixel points with gray-scale values less than the segmentation threshold as the original hole region;

[0012] Denote the original hole region that the bronchoscope will pass through in the internal bronchial gray-scale image in the video image as the target hole region of each frame of the internal bronchial gray-scale image.

[0013] Further, the obtaining of the area correction amount of the target hole region of each frame of the internal bronchial gray-scale image includes:

[0014] According to the corresponding position of each frame of the internal bronchial gray-scale image in the virtual bronchial tree, determine the branch level of the corresponding image; denote the branch angle of the next bifurcation of the corresponding position of each frame of the internal bronchial gray-scale image in the virtual bronchial tree as the target branch angle of the corresponding image;

[0015] Denote the internal bronchial gray-scale image at each bifurcation position of the virtual bronchial tree as the bifurcation position image; in the video image, denote the sequence of the internal bronchial gray-scale images between each bifurcation position image and its previous frame of bifurcation position image as the same-branch image sequence; obtain the final area change amount function of each same-branch image sequence;

[0016] Substitute the target branch angle and the branch level of each frame of the internal bronchial gray-scale image into the final area change amount function of the same-branch image sequence where the internal bronchial gray-scale image is located to obtain the area correction amount of the target hole region of each frame of the internal bronchial image.

[0017] Further, the obtaining of the final area change amount function of each same-branch image sequence includes:

[0018] Perform a non - linear analysis on the number of branch levels, the target branch angle, and the change amount of area, construct the initial area change function for each sequence of images of the same branch, and the initial area change function contains several regression coefficients;

[0019] Obtain the historical branch image sequence for each sequence of images of the same branch, and take the difference between the area of the target hole region of each frame of image in the historical branch image sequence and the area of the target hole region of its previous frame of image as the change amount of the area of the target hole region of each frame of image in the historical branch image sequence;

[0020] Substitute the number of branch levels and the target branch angle of each frame of image in the historical branch image sequence into the initial area change function to obtain the area change fitting value of each frame of image in the historical branch image sequence;

[0021] According to the difference between the area change fitting value and the area change amount of each image in the historical branch image sequence, obtain the error function of the historical branch image sequence; take the value of the regression coefficient corresponding to the minimum value of the error function as the optimal value of each regression coefficient in the initial area change function; substitute the optimal value of the regression coefficient into the initial area change function to obtain the final area change function for each sequence of images of the same branch.

[0022] Furthermore, the obtaining of the true area of the target hole region of each frame of gray - scale image of the bronchus includes:

[0023] Judge whether the number of the original hole regions in each frame of gray - scale image of the bronchus is greater than the constant 1. If not, take the area of the original hole region as the true area of the target hole region of the corresponding image;

[0024] If so, according to the distance between the corresponding position of each frame of gray - scale image of the bronchus in the virtual bronchial tree and its next bifurcation, and the area correction amount, obtain the area correction coefficient of the target hole region of the corresponding image;

[0025] Use the area correction coefficient to weight the area of the target hole region of each frame of gray - scale image of the bronchus to obtain the true area of the target hole region of each frame of gray - scale image of the bronchus.

[0026] Furthermore, the assisting the bronchoscope for positioning according to the distance between the corresponding position of the last frame of gray - scale image of the bronchus in the video image and its next bifurcation, and the change situation of the true area of the target hole region of the gray - scale image of the bronchus between its position and its previous bifurcation includes:

[0027] Record the last frame of the gray-scale image of the bronchial interior in the video image as the current image, and record the sequence of gray-scale images of the bronchial interior formed by the gray-scale images of the bronchial interior between the corresponding position of the current image in the virtual bronchial tree and its previous bifurcation as the analysis image sequence; obtain the area change rule value of the current image according to the change of the true area of the target hole region in the gray-scale images of the bronchial interior in the analysis image sequence;

[0028] Obtain the abnormal possibility of the current image according to the distance between the corresponding position of the current image in the virtual bronchial tree and its next bifurcation and the area change rule value; assist the bronchoscope in positioning according to the abnormal possibility.

[0029] Further, the obtaining the area change rule value of the current image includes:

[0030] Record the sequence of the true areas of the target hole regions in the gray-scale images of the bronchial interior in the analysis image sequence as the area sequence; respectively obtain the first-order difference sequence and the second-order difference sequence of the area sequence, and record the mean value of the elements in the second-order difference sequence as the area change rate of the current image; obtain the area discrete value of the current image according to the degree of dispersion of the elements in the first-order difference sequence; take the number of changes in the signs of two adjacent elements in the first-order difference sequence as the area trend value of the current image;

[0031] Obtain the area change rule value of the current image according to the area change rate, the area discrete value and the area trend value.

[0032] Further, the assisting the bronchoscope in positioning according to the abnormal possibility includes:

[0033] When the abnormal possibility is greater than the preset normal threshold, there is an abnormal structure in front of the corresponding position of the current image in the virtual bronchial tree; when the abnormal possibility is less than or equal to the preset normal threshold, there is no abnormal structure in front of the corresponding position of the current image in the virtual bronchial tree.

[0034] Further, the total number of pixel points in the target hole region is the area of the target hole region.

[0035] In a second aspect, another embodiment of the present invention provides a bronchoscope-assisted positioning system for tuberculosis care, and the system includes:

[0036] A data acquisition module, configured to collect video images of the bronchus of a user to be measured in real time through a bronchoscope and obtain the virtual bronchial tree of the user to be measured;

[0037] A hole correction analysis module is used to obtain the target hole area of the grayscale image inside the bronchus in each frame of the video image; according to the corresponding position of the grayscale image inside the bronchus in each frame in the virtual bronchial tree, the branch angle of the next bifurcation at this position, and the area difference between the target hole area of the grayscale image inside the bronchus in each frame and the target hole area of the grayscale image inside the bronchus in the previous frame, obtain the area correction amount of the target hole area of the grayscale image inside the bronchus in each frame.

[0038] A hole area correction module is used to correct the area of the target hole area according to the distance between the corresponding position of the grayscale image inside the bronchus in each frame and its next bifurcation and the area correction amount, and obtain the true area of the target hole area of the grayscale image inside the bronchus in each frame.

[0039] An auxiliary positioning module is used to assist the bronchoscope in positioning according to the distance between the corresponding position of the last frame of the grayscale image inside the bronchus in the video image and its next bifurcation, and the change of the true area of the target hole area of the grayscale image inside the bronchus between the corresponding position of the last frame of the grayscale image inside the bronchus in the virtual bronchial tree and its previous bifurcation.

[0040] The present invention has the following beneficial effects:

[0041] In the embodiment of the present invention, the corresponding position of the grayscale image inside the bronchus in the virtual bronchial tree reflects the moving speed of the bronchoscope, and further presents the degree of change in the area of the target hole area of the grayscale image inside the bronchus; the branch angle of the next bifurcation at the above position reflects the degree of sudden change in the area of the target hole area, and is analyzed in combination with the area difference of the target hole area of the grayscale image inside the bronchus in adjacent frames to improve the accuracy of the area correction amount of the target hole area; the distance between the corresponding position of the grayscale image inside the bronchus in the virtual bronchial tree and its next bifurcation reflects the degree of influence of the perspective phenomenon on the grayscale image inside the bronchus, and adjusts the target hole area in combination with the area correction amount, so that the corrected true area effectively reduces the influence degree of the branch angle of the bifurcation on the area of the target hole area. By presenting the degree of change law of the area of the target hole area through the change of the true area of the target hole area of the grayscale image inside the bronchus between the corresponding position of the grayscale image inside the bronchus in the virtual bronchial tree and its previous bifurcation; the distance between the corresponding position of the grayscale image inside the bronchus in the virtual bronchial tree and its next bifurcation presents the possibility of sudden change in the area of the target hole area, and the foreign object structure in the bronchus will also cause sudden change in the area of the target hole area. At the same time, by combining the above two factors to analyze the degree of change law of the area of the target hole area, the error of identifying abnormal structures in the bronchus is reduced, and the accuracy of the bronchoscope for auxiliary positioning is improved. Brief Description of the Drawings

[0042] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0043] Figure 1 The flowchart of the steps of a bronchoscope-assisted positioning method for tuberculosis care provided by an embodiment of the present invention;

[0044] Figure 2 The partial schematic diagram of the centerline of the virtual bronchial tree provided by an embodiment of the present invention;

[0045] Figure 3 The flowchart of the steps of a method for obtaining the area correction amount of the target hole area of a bronchial internal grayscale image provided by an embodiment of the present invention;

[0046] Figure 4 The structural diagram of a bronchoscope-assisted positioning system for tuberculosis care provided by an embodiment of the present invention;

[0047] Figure 5 The schematic diagram of a computer device of a bronchoscope-assisted positioning device for tuberculosis care provided by an embodiment of the present invention. Detailed Embodiments

[0048] In order to further elaborate on the technical means and effects adopted by the present invention to achieve the intended invention purpose, the following, in conjunction with the drawings and preferred embodiments, will detail the specific embodiments, structures, features and effects of a bronchoscope-assisted positioning method and system for tuberculosis care proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures or characteristics in one or more embodiments can be combined in any suitable form.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.

[0050] The following will specifically describe the specific solutions of a bronchoscope-assisted positioning method and system for tuberculosis care provided by the present invention with reference to the drawings.

[0051] Embodiment 1:

[0052] The present invention proposes a bronchoscope-assisted positioning method for tuberculosis care. Please refer to Figure 1 , which shows a flowchart of the steps of a bronchoscope-assisted positioning method for tuberculosis care provided by an embodiment of the present invention. The method includes:

[0053] Step S1: Real-time collect video images of the bronchus of the user to be tested through a bronchoscope, and obtain the virtual bronchial tree of the user to be tested.

[0054] Specifically, in this solution, a bronchus detection device is used to examine the bronchus of the user to be tested. The bronchus detection device includes a detection tube, a bronchoscope, and a main control device. Among them, the bronchoscope is an electromagnetic navigation bronchoscope, which is arranged at the end of the detection tube and is used to collect images of the bronchus of the user to be tested. The images collected by the bronchoscope can be transmitted to the main control device through the detection tube. A virtual bronchial tree, that is, a three-dimensional virtual model of the bronchus of the user to be tested, is set in the main control device; before bronchoscopy, the standard virtual bronchial tree corresponding to the user to be tested can be obtained by CT image reconstruction.

[0055] After obtaining the virtual bronchial tree of the user to be tested and before performing bronchoscopy, determine the position to be detected in the virtual bronchial tree, and adopt depth-first traversal planning to plan the path from the root of the virtual bronchial tree to the position to be detected, and record it as the detection path. The bronchoscope travels through the bronchial tree of the user to be tested according to the detection path. During the travel, the bronchoscope is used to collect internal bronchial images in real time, and the internal bronchial images are grayscale processed and denoised to obtain internal bronchial grayscale images. The video images of the bronchus of the user to be tested are composed of the internal bronchial grayscale images during the time period from the moment when the bronchoscope enters the root position of the virtual bronchial tree to the current moment; the last frame of the internal bronchial grayscale image in the video image is the image collected by the bronchoscope at the current moment.

[0056] It should be noted that the electromagnetic navigation bronchoscope is equipped with an electromagnetic navigation system. The electromagnetic navigation system tracks the bronchoscope in real time through an electromagnetic field and displays the position of the bronchoscope in the virtual bronchial tree on the display screen of the main control device in real time; depth-first traversal of the bronchial tree is performed to ensure that the bronchoscope will not pass through the same bronchus multiple times when running according to the detection path. Therefore, each frame of the internal bronchial grayscale image in the video image corresponds to only one position in the virtual bronchial tree. The bifurcation of the bronchial tree is equivalent to the node of the tree. In this embodiment, a weighted average grayscale algorithm is selected for grayscale processing, and Gaussian filtering is used for denoising. The specific methods are not introduced here and are all well-known technical means to those skilled in the art.

[0057] The display of the virtual bronchial tree on the display screen depends on a three-dimensional coordinate system. The three-dimensional coordinate system is constructed. In this embodiment, the origin of the three-dimensional coordinate system is the root of the virtual bronchial tree of the user to be measured. The X-axis points to the right side of the user to be measured, the Y-axis points to the front of the user to be measured, that is, the front position of the chest, and the Z-axis points to the head of the user to be measured.

[0058] The patent document with the publication number CN113450364A and the name "A method for extracting the centerline of a tree-like structure based on a three-dimensional flux model" discloses a method for extracting the centerline of a tree-like three-dimensional model. According to the above method, the centerline of the virtual bronchial tree on the display screen of the main control device is extracted. This centerline also has a tree-like structure, and the bifurcation points of the centerline are equivalent to the nodes of the tree. Based on the bifurcation points of the centerline of the virtual bronchial tree, the centerline is divided into different bronchial centerlines, and the coordinates of the two endpoints of each bronchial centerline are obtained in the three-dimensional coordinate system. It should be noted that the bifurcation openings in the virtual bronchial tree correspond one-to-one with the bifurcation points of the centerline of the virtual bronchial tree, and the connection positions of two connected bronchial centerlines share a bifurcation point.

[0059] The bronchoscope travels through the virtual bronchial tree along the detection path, arranges the bifurcation points corresponding to the bifurcation openings passed by the bronchoscope in the virtual bronchial tree in chronological order to obtain a bifurcation point sequence; for each bronchial centerline, find the bifurcation points corresponding to the two endpoints of the bronchial centerline in the bifurcation point sequence, and subtract the coordinates of the bifurcation point with the smaller subscript from the coordinates of the bifurcation point with the larger subscript to obtain the direction vector of the bronchial centerline. For each frame of the internal gray-scale image of the bronchus, the bronchus where the internal gray-scale image of the bronchus is located in the virtual bronchial tree is recorded as the current bronchus, and the target bronchus is the bronchus that the bronchoscope is about to pass through, that is, the next bronchus to pass through, and the target bronchus is connected to the current bronchus at the next bifurcation opening at the above position; the included angle between the direction vectors of the bronchial centerlines corresponding to the current bronchus and the target bronchus is used as the branching angle of the next bifurcation opening at the corresponding position of the internal gray-scale image of the bronchus collected by the bronchoscope in the virtual bronchial tree. Among them, the method for obtaining the included angle between two vectors is a well-known technology and will not be elaborated here; the value range of the branching angle is 。

[0060] As an example, Figure 2 is a partial schematic diagram of the centerline of the virtual bronchial tree provided by an embodiment of the present invention. As Figure 2 shown, Figure 2 the solid line in Figure 2There are 3 bronchial centerlines, including: the curve between point A and point B, the curve between point B and point C, and the curve between point B and point F. It is known that the bronchoscope advances along the detection path and passes through points A, B, and C in sequence. Point F is not on the detection path. The direction vector of the bronchial centerline between point A and point B is , that is, the coordinates of point B minus the coordinates of point A; the direction vector of the bronchial centerline between point B and point C is , that is, the coordinates of point C minus the coordinates of point B. If the bronchoscope is in the bronchus corresponding to the bronchial centerline between point A and point B, then the next bifurcation point of the bronchoscope at the current position is point B; the internal gray-scale image of the bronchus at the corresponding position of the bronchoscope in the virtual bronchial tree is recorded as the analysis image, then the branch angle of the next bifurcation of the analysis image at the corresponding position in the virtual bronchial tree, that is, the branch angle of bifurcation point B at the corresponding position in the virtual bronchial tree, is equal to the angle G between ray E1 and ray E2; among them, ray E1 has the same direction as the direction vector of the bronchial centerline between point A and point B, and ray E2 has the same direction as the direction vector of the bronchial centerline between point B and point C.

[0061] Step S2: Obtain the target hole area of each frame of the internal gray-scale image of the bronchus in the video image; according to the corresponding position of each frame of the internal gray-scale image of the bronchus in the virtual bronchial tree, the branch angle of the next bifurcation of the position, and the area difference between the target hole area of each frame of the internal gray-scale image of the bronchus and the target hole area of its previous internal gray-scale image of the bronchus, obtain the area correction amount of the target hole area of each frame of the internal gray-scale image of the bronchus.

[0062] If the bronchoscope is close to the bifurcation of the virtual bronchial tree, there are multiple holes in the internal gray-scale image of the bronchus collected by the bronchoscope. The holes represent the entrances of the next bronchi that the bronchoscope may enter. To analyze whether there are abnormal structures in the bronchi on the detection path, it is necessary to determine the target hole area.

[0063] Preferably, in some possible implementation manners of the embodiments of the present invention, the method for obtaining the target hole area includes: for each frame of the internal gray-scale image of the bronchus, use the maximum inter-class variance method to obtain the segmentation threshold for the gray-scale values of the pixel points in the internal gray-scale image of the bronchus, and use the connected domain composed of pixel points with gray-scale values less than the segmentation threshold as the original hole area; record the original hole area that the bronchoscope in the internal gray-scale image of the bronchus in the video image is about to pass through as the target hole area of each frame of the internal gray-scale image of the bronchus.

[0064] There is a current position area and a front area of the bronchoscope in the grayscale image of the bronchus interior; since the tip of the bronchoscope is equipped with a light source, compared with the current position area, the front area is irradiated with less light, so the grayscale value of the front area is lower. Therefore, the pixel points with grayscale values less than the segmentation threshold form the original hole area, representing the front area of the bronchoscope, that is, the bronchial area that the bronchoscope may pass through.

[0065] If the bronchoscope is far from the next bifurcation of the bronchial tree, there is only one bronchus that the bronchoscope can pass through, that is, there is only one original hole area in the grayscale image of the bronchus interior, then the original hole area of the grayscale image of the bronchus interior is used as the target hole area. If the bronchoscope is close to the next bifurcation of the bronchial tree, there may be multiple bronchi that the bronchoscope needs to pass through, that is, there are multiple original hole areas in the grayscale image of the bronchus interior. The original hole area corresponding to the bronchus that the bronchus corresponding to the position of the grayscale image of the bronchus interior in the virtual bronchial tree needs to penetrate next in the detection path is used as the target hole area.

[0066] During the process of the bronchial tree extending into the lungs, that is, from the main bronchus to the smallest respiratory bronchioles and alveoli, the airway diameter of the bronchi gradually decreases and the number of branches increases. To avoid the bronchoscope damaging the bronchial wall or straying into other branches, the closer the bronchoscope is to the end of the bronchial tree, the slower its moving speed needs to be, resulting in a smaller degree of change in the area of the target hole area in the grayscale image of the bronchus interior.

[0067] When the bronchoscope is close to the bifurcation of the bronchial tree, this bifurcation is a position that the bronchoscope has not reached yet, so the above bifurcation is the next bifurcation at the corresponding position of the grayscale image of the bronchus interior in the virtual bronchial tree; because there is a certain angle between the bronchus where the bronchoscope is located and the bronchus to be penetrated at the bifurcation position, causing partial areas of the bronchus to be blocked, the area of the target hole area in the grayscale image of the bronchus interior suddenly becomes smaller, thus resulting in a larger degree of change in the area of the target hole area in the grayscale image of the bronchus interior.

[0068] In summary, the branch angle between the corresponding position of the grayscale image of the bronchus interior in the virtual bronchial tree and the next bifurcation at this position is related to the degree of change in the area of the target hole area. The area correction amount of the target hole area is obtained by integrating the above factors.

[0069] Please refer to Figure 3 , which shows a flowchart of the steps of a method for obtaining the area correction amount of the target hole area of a grayscale image of the bronchus interior provided by an embodiment of the present invention. The method includes:

[0070] Step S210: Determine the branch level of each frame of the gray-scale image inside the bronchus according to its corresponding position in the virtual bronchial tree; record the branch angle at the next bifurcation of the corresponding position of each frame of the gray-scale image inside the bronchus in the virtual bronchial tree as the target branch angle of the corresponding image.

[0071] Obtain the layer number of the bifurcation point of the center line of the virtual bronchial tree; in this embodiment, arbitrarily select a bifurcation point as the target point, and set the branch level of the gray-scale image inside the bronchus corresponding to the bronchus between the target point and the next bifurcation point on the path from the target point to the root node as the layer number of the target point. This solution reflects the corresponding position of the gray-scale image inside the bronchus in the virtual bronchial tree through the branch level; if the branch level is larger, the corresponding position of the gray-scale image inside the bronchus in the virtual bronchial tree is closer to the end of the bronchial tree, and the moving speed of the bronchoscope is slower, then the degree of change in the area of the target hole region of adjacent frames of the gray-scale image inside the bronchus is smaller.

[0072] It should be noted that the above-mentioned bronchus includes the target point, but does not include the next bifurcation point on the path from the target point to the root node. Among them, the method for obtaining the layer number of the node of the tree is a well-known technology and will not be elaborated here.

[0073] In the case where there are multiple original hole regions in the gray-scale image inside the bronchus, if the target branch angle is larger, the occlusion of the above-mentioned second bronchial region by the included angle between the bronchus where the bronchoscope is located and the bronchus to be penetrated at the bifurcation position is more serious, resulting in a larger degree of change in the area of the target hole region of adjacent frames of the gray-scale image inside the bronchus.

[0074] It should be noted that the branch levels of the gray-scale images inside the bronchus collected by the bronchoscope passing through the same bronchus are all equal, and the target branch angles are all equal. Among them, the airway between two adjacent bifurcation points in the virtual bronchial tree is a bronchus. Since the gray-scale image inside the bronchus corresponding to the bifurcation point of the virtual bronchial tree and its previous frame of the gray-scale image inside the bronchus are in two bronchi, in order to reduce the analysis error, the area of the target hole region of the gray-scale image inside the bronchus corresponding to the bifurcation point of the virtual bronchial tree is set to zero.

[0075] Step S220: Denote the gray-scale image inside the bronchus at each bifurcation position of the virtual bronchial tree as the bifurcation position image; in the video image, denote the sequence of the gray-scale images inside the bronchus formed between each bifurcation position image and its previous frame of the bifurcation position image as the same-branch image sequence; obtain the final area change function of each same-branch image sequence.

[0076] If the number of branch levels is larger, the degree of change in the area of the target hole region in the internal gray-scale images of adjacent-frame bronchi is smaller; if the target branch angle is larger, the occlusion of the second bronchial region by the included angle between the bronchus where the bronchoscope is located and the bronchus to be penetrated at the bifurcation position is more serious, resulting in a larger degree of change in the area of the target hole region in the internal gray-scale images of adjacent frames; and the degree of change in the area of the target hole region is measured by the change amount of the area. Therefore, the number of branch levels and the target branch angle are respectively related to the change amount of the area. An initial area change amount function is constructed using the number of branch levels, the target branch angle, and the change amount of the area, so as to predict the actual area of the target hole region.

[0077] Preferably, in some possible implementation manners of the embodiments of the present invention, the method for obtaining the final area change amount function includes: performing non-linear analysis on the number of branch levels, the target branch angle, and the change amount of the area, constructing an initial area change amount function for each same-branch image sequence, and the initial area change amount function includes a plurality of regression coefficients; obtaining the historical branch image sequence of each same-branch image sequence, taking the difference between the area of the target hole region of each frame image in the historical branch image sequence and the area of the target hole region of its previous frame image as the change amount of the area of the target hole region of each frame image in the historical branch image sequence; substituting the number of branch levels and the target branch angle of each frame image in the historical branch image sequence into the initial area change amount function to obtain the area change fitting value of each frame image in the historical branch image sequence; obtaining the error function of the historical branch image sequence according to the difference between the area change fitting value and the change amount of the area of each image in the historical branch image sequence; taking the numerical value of the regression coefficient corresponding to the minimum value of the error function as the optimal numerical value of each regression coefficient in the initial area change amount function; substituting the optimal numerical value of the regression coefficient into the initial area change amount function to obtain the final area change amount function of each same-branch image sequence.

[0078] It should be noted that the sequence formed by the internal gray-scale images of the bronchus between the first-frame internal gray-scale image of the bronchus and the first-frame bifurcation position image in the video image is denoted as the first same-branch image sequence; the sequence formed by the internal gray-scale images of the bronchus between the last-frame bifurcation position image and the last-frame internal gray-scale image of the bronchus in the video image is denoted as the last same-branch image sequence; each frame of bifurcation position image belongs to the previous same-branch image sequence.

[0079] Under the condition of ensuring a relatively low function complexity, in order to better capture the non-linear relationships between the number of branches, the target branch angle, and the area change amount respectively, a quadratic polynomial regression equation is selected in this embodiment to construct the relationships between the number of branches, the target branch angle, and the area change amount respectively. To comprehensively analyze the combined effects of the number of branches and the target branch angle on the area change amount, the above two quadratic polynomial regression equations are added to obtain the initial area change amount function, so as to retain the influence of each independent variable on the dependent variable and reduce the function complexity. It should be noted that constructing the functional relationships between the number of branches, the target branch angle, and the area change amount respectively affects the number of regression coefficients in the initial area change amount function.

[0080] In a specific implementation manner of the embodiment of the present invention, the initial area change amount function of each same-branch image sequence is expressed by the formula:

[0081] ;

[0082] In the formula, is the area change amount of the target hole region of the internal grayscale image of the bronchus; is the number of branches of the internal grayscale image of the bronchus; is the target branch angle of the internal grayscale image of the bronchus; are regression coefficients, all of which are unknowns.

[0083] Obtain the historical branch image sequence of each same-branch image sequence; the analysis objects of each same-branch image sequence and its historical branch image sequence are the same, and the analysis object is the bronchus corresponding to the same-branch image sequence in the virtual bronchial tree. The same-branch image sequence is composed of the internal grayscale images of the bronchus collected during the process of the bronchoscope passing through this bronchus; the difference is that the same-branch image sequence is obtained by the user to be measured during the current bronchus examination; while the historical branch image sequence of the same-branch image sequence is obtained by the user to be measured during the historical bronchus examination, and it is necessary to ensure that the bronchial tree of the user to be measured is healthy, that is, there are no abnormal structures.

[0084] It should be noted that since the number of regression coefficients in the initial area change amount function in this embodiment is 6, in order to determine the specific values of the regression coefficients, it is necessary to ensure that the total number of images in the historical branch image sequence is greater than or equal to 6. Since the analysis objects of the same-branch image sequence and its historical branch image sequence are the same section of the bronchus corresponding to the same-branch image sequence in the virtual bronchial tree, the target branch angles of all the images in the historical branch image sequence are equal, the number of branches is equal, and the methods for obtaining the target branch angles, the number of branches, and the area change amount in the same-branch image sequence and its historical branch image sequence are the same.

[0085] The degree of area change is measured by the area difference of the target hole region in the internal gray-scale images of the bronchus in adjacent frames. In this embodiment, the absolute value of the difference between the area of the target hole region in each frame of the internal gray-scale image of the bronchus and the area of the target hole region in its previous frame of the internal gray-scale image of the bronchus is used as the area change amount of the target hole region corresponding to the image. Among them, the area of the target hole region is the total number of pixel points in the target hole region. Substitute the branch level and the target branch angle of each frame of the image in the historical branch image sequence into the initial area change amount function to obtain the area change fitting value of each frame of the image in the historical branch image sequence; according to the difference between the area change fitting value and the area change amount of each image in the historical branch image sequence, obtain the error function of the historical branch image sequence. In a specific implementation manner of the embodiment of the present invention, for each same-branch image sequence, the error function RSS of the historical branch image sequence of the same-branch image sequence is expressed by the formula:

[0086] ;

[0087] In the formula, I is the total number of images in the historical branch image sequence of the same-branch image sequence; is the area change amount of the target hole region of the i-th image in the historical branch image sequence of the same-branch image sequence; is the branch level of the i-th image in the historical branch image sequence of the same-branch image sequence; is the target branch angle of the i-th image in the historical branch image sequence of the same-branch image sequence; is the regression coefficient, all of which are unknowns; is the area change fitting value of the i-th image in the historical branch image sequence of the same-branch image sequence; is the absolute value function.

[0088] Take the numerical value of the regression coefficient corresponding to the minimum value of the error function as the optimal numerical value of each regression coefficient in the initial area change amount function; substitute the optimal numerical value of the regression coefficient into the initial area change amount function to obtain the final area change amount function of each same-branch image sequence.

[0089] Step S230: Substitute the target branch angle and the branch level of each frame of the internal gray-scale image of the bronchus into the final area change amount function of the same-branch image sequence where the internal gray-scale image of the bronchus is located to obtain the area correction amount of the target hole region of each frame of the internal image of the bronchus.

[0090] Since the analysis objects of the same-branch image sequence and its historical branch image sequence are the same, the final area change amount function obtained from the historical branch image sequence can be used to correct the area of the target hole region of each frame of the internal gray-scale image of the bronchus in the same-branch image sequence, so as to obtain the true area of the target hole.

[0091] Step S3: According to the distance between the corresponding position of each frame of the internal bronchial gray-scale image in the virtual bronchial tree and its next bifurcation, and the area correction amount, correct the area of the target hole region to obtain the true area of the target hole region of each frame of the internal bronchial gray-scale image.

[0092] The internal bronchial gray-scale images in the first half of the same-branch image sequence are farther from the next bifurcation. There is only one hole region representing the bronchoscope passing through the bronchus in the image, and the size of the hole region is not affected by the perspective phenomenon. The areas of the target hole regions of these images are relatively close; the internal bronchial gray-scale images in the second half of the same-branch image sequence are closer to the next bifurcation. There are multiple hole regions of the next bronchus that the bronchoscope may pass through in the image. As the bronchoscope moves towards the bifurcation position, the perspective phenomenon causes the areas of the target hole regions of these images to show a gradually increasing pattern.

[0093] Therefore, the closer the distance between the corresponding position of the internal bronchial gray-scale image in the virtual bronchial tree and its next bifurcation, the greater the change in the area of the target hole region; combined with the area correction amount reflecting the degree of change in the area of the target hole region of adjacent frames of the internal bronchial gray-scale images for analysis, the accuracy of the area correction of the target hole region is improved.

[0094] Preferably, in some possible implementation manners of the embodiment of the present invention, the method for obtaining the true area of the target hole region includes: judging whether the number of the original hole regions in each frame of the internal bronchial gray-scale image is greater than the constant 1. If not, taking the area of the original hole region as the true area of the target hole region of the corresponding image; if so, obtaining the area correction coefficient of the target hole region of the corresponding image according to the distance between the corresponding position of each frame of the internal bronchial gray-scale image in the virtual bronchial tree and its next bifurcation, and the area correction amount; weighting the area of the target hole region of each frame of the internal bronchial gray-scale image by using the area correction coefficient to obtain the true area of the target hole region of each frame of the internal bronchial gray-scale image.

[0095] If there is only one original hole region in the internal bronchial gray-scale image, the position represented by the original hole region changes as the bronchoscope moves forward, and the size of the hole region is not affected by the perspective phenomenon, then the area of the target hole region does not need to be corrected; if there are multiple original hole regions in the internal bronchial gray-scale image, the target hole region is equivalent to having a fixed position. As the bronchoscope moves towards the bifurcation position, the perspective phenomenon causes the area of the target hole region to show a gradually increasing pattern, then the area of the target hole region needs to be corrected.

[0096] If the distance between the corresponding position of the internal bronchial gray-scale image in the virtual bronchial tree and its next bifurcation is smaller, the more severely the image is affected by the perspective phenomenon, and the greater the degree of change in the area of the target hole region between the internal bronchial gray-scale image and its previous frame image. Therefore, the distance between the corresponding position of each frame of the internal bronchial gray-scale image in the virtual bronchial tree and its next bifurcation is negatively correlated and mapped, the product of the mapping result and the area correction amount is normalized, and the sum of the normalized result and the constant 1 is used as the area correction coefficient.

[0097] It should be noted that the corresponding position of the internal bronchial gray-scale image on the center line of the virtual bronchial tree is recorded as the first position, the next bifurcation point of this position on the center line is recorded as the second position, and the Euclidean distance between the first position and the second position is used as the distance between the corresponding position of the internal bronchial gray-scale image in the virtual bronchial tree and its next bifurcation.

[0098] In a specific implementation manner of the embodiment of the present invention, the true area of the target hole region is expressed by the formula:

[0099] ;

[0100] In the formula, is the true area of the target hole region of each frame of the internal bronchial gray-scale image; is the area correction amount of the target hole region of each frame of the internal bronchial image; is the distance between the corresponding position of each frame of the internal bronchial gray-scale image in the virtual bronchial tree and its next bifurcation; is the area of the target hole region of each frame of the internal bronchial gray-scale image, that is, the total number of pixel points in the target hole region; is the area correction coefficient of the target hole region of each frame of the internal bronchial gray-scale image; is a preset positive number, taking an empirical value of 0.01, which is used to prevent the fraction from being meaningless due to the denominator being zero; Norm is the normalization function.

[0101] It should be noted that since there is a certain angle between the bronchus where the bronchoscope is located and the bronchus connected to the next bifurcation, the area of the target hole region of the internal image of the bronchoscope is smaller than the actual area, so the area correction coefficient is greater than the constant 1, and the true area obtained after correcting the area of the target hole region is greater than the area s before correction.

[0102] Step S4: Assist the bronchoscope in positioning according to the distance between the corresponding position of the last frame of the bronchial internal grayscale image in the video image and its next bifurcation in the virtual bronchial tree, and the change in the true area of the target hole region of the bronchial internal grayscale image between the corresponding position of the last frame of the bronchial internal grayscale image in the virtual bronchial tree and its previous bifurcation.

[0103] It is known that the area of the target hole region of the bronchial internal grayscale image collected by the bronchoscope shows a gradually increasing pattern due to the perspective phenomenon. By analyzing the change in the true area of the target hole region of the bronchial internal grayscale images in the same bronchus as the bronchial internal grayscale image, the degree of the area change pattern of the target hole region is measured. The closer the distance between the corresponding position of each frame of the bronchial internal grayscale image in the virtual bronchial tree and its next bifurcation, the area mutation caused by the branch angle may be recognized as the area mutation caused by abnormal structures.

[0104] If there are no abnormal structures such as foreign bodies or tumors in a bronchus, the area change of the target hole region in the bronchial internal grayscale image corresponding to this bronchus has a stronger regularity; if there are abnormal structures, the area change of the target hole region in the bronchial internal grayscale image corresponding to this bronchus has a weaker regularity, that is, the area of the target hole region at some positions in the bronchus shows a mutation, thereby assisting the bronchoscope in positioning. The specific method is as follows:

[0105] Step S410: Denote the last frame of the bronchial internal grayscale image in the video image as the current image, and denote the sequence of the bronchial internal grayscale images between the corresponding position of the current image in the virtual bronchial tree and its previous bifurcation as the analysis image sequence; obtain the area change pattern value of the current image according to the change in the true area of the target hole region of the bronchial internal grayscale images in the analysis image sequence.

[0106] This solution provides real-time assistance for the operation process of the bronchoscope in the bronchial tree of the user to be measured, and only needs to analyze the bronchial internal grayscale image at the current moment, that is, the last frame of the bronchial internal grayscale image in the video image. Since the bronchial internal grayscale images at different bifurcation positions are blocked by different branch angles, resulting in significant differences in the area mutation of the target hole region, which affects the accuracy of the analysis. To avoid the above influence, the bronchial internal grayscale images corresponding to the same bronchus are analyzed. In this embodiment, the sequence of the bronchial internal grayscale images between the corresponding position of the current image in the virtual bronchial tree and its previous bifurcation is denoted as the analysis image sequence. By analyzing the change in the true area of the target hole region of the bronchial internal grayscale images in the analysis image sequence, the area change pattern value is obtained; the larger the area change pattern value, the smaller the possibility that there are abnormal structures affecting in the bronchus.

[0107] Preferably, in some possible implementation manners of the embodiments of the present invention, the method for obtaining the area change rule value includes: recording the sequence composed of the true areas of the target hole regions in the internal gray-scale images of the bronchus in the analyzed image sequence as the area sequence; respectively obtaining the first-order difference sequence and the second-order difference sequence of the area sequence, and recording the mean value of the elements in the second-order difference sequence as the area change rate of the current image; obtaining the area discrete value of the current image according to the degree of dispersion of the elements in the first-order difference sequence; taking the number of changes in the signs of two adjacent elements in the first-order difference sequence as the area trend value of the current image; and obtaining the area change rule value of the current image according to the area change rate, the area discrete value, and the area trend value.

[0108] When there is no abnormal structure in the bronchus where the bronchoscope is located at the current moment, if the position of the current image in the virtual bronchial tree is far from its next bifurcation, the target hole region in the internal gray-scale image of the bronchus in the analyzed image sequence is less affected by the perspective phenomenon, and the actual areas of the target hole regions of these images are relatively close; if the position of the current image in the virtual bronchial tree is far from its next bifurcation, the target hole region in the internal gray-scale image of the bronchus in the analyzed image sequence is more affected by the perspective phenomenon, and the area of the target hole region shows a gradually increasing rule. Therefore, when there is no foreign object structure in the bronchial tree, the elements in the area sequence show a trend of first remaining basically unchanged and then gradually increasing, and the increase amplitude is relatively stable.

[0109] The area change rate reflects the stability of the area change rate of the target hole region in the internal gray-scale image of the bronchus in the analyzed image sequence. If the area change rate is smaller, and the elements in the image sequence show a gradually increasing trend with a relatively stable increase amplitude, it indicates that the true area change of the target hole region in the internal gray-scale image of the bronchus in the analyzed image sequence has stronger regularity.

[0110] The degree of dispersion of the elements in the first-order difference sequence of the area sequence reflects the regularity of the area change of the target hole region in the internal gray-scale image of the bronchus in the analyzed image sequence. Variance, standard deviation, range, etc. can all reflect the degree of dispersion. In this embodiment, the variance of the elements in the first-order difference sequence of the area sequence of each frame of the internal gray-scale image of the bronchus is recorded as the area discrete value of the corresponding image; in other embodiments, the variance can be replaced by the standard deviation or the range. If the area discrete value is smaller, it indicates that the true area change of the target hole region in the internal gray-scale image of the bronchus in the analyzed image sequence is more stable, and the area change has stronger regularity.

[0111] As an example, if the first-order difference sequence N1 of the area sequence is (1, 2, -1, 4), taking the element 1 as the element to be updated, both the element 1 and the element 2 are positive numbers, the actual area of the target hole area satisfies an increasing trend, the interval change value remains unchanged, which is still the constant 0, and the element 2 is taken as the new element to be updated; the element 2 and the element -1 have different signs, which destroys the increasing trend of the area of the target hole area. There may be abnormal structures at the corresponding position of the image corresponding to the element -1 in the bronchial tree. The interval change value is added with the constant 1, and the updated interval change value is 1. The element -1 is taken as the new element to be updated; the element -1 and the element 4 have different signs, which destroys the increasing trend of the area of the target hole area. The interval change value is added with the constant 1, and the updated interval change value is 2; after traversing the first-order difference sequence N1, the updated interval change value is 2. If the first-order difference sequence is N2(1, 2, 1, 4), then the finally updated interval change value, that is, the area trend value, is 0. The smaller the area trend value is, the more the true area of the target hole area in the grayscale image inside the bronchus in the analyzed image sequence conforms to the increasing trend, and the stronger the regularity of the area change of the target hole area is.

[0112] The smaller the area change regularity value is, the stronger the regularity of the true area change of the target hole area in the grayscale image inside the bronchus in the analyzed image sequence is. Therefore, the area change rate, the area discrete value, and the area trend value are all positively correlated with the area change regularity index. In a specific implementation manner of the embodiment of the present invention, the area change regularity value of each frame of the grayscale image inside the bronchus is expressed by the formula:

[0113] ;

[0114] In the formula, is the area change regularity value of the current image; W is the total number of elements in the second-order difference sequence of the area sequence of the current image; is the w-th element in the second-order difference sequence of the area sequence of the current image; is the area change rate of the current image; is the area discrete value of the current image; is the area trend value of the current image.

[0115] Step S420: Obtain the abnormal possibility of the current image according to the distance between the corresponding position of the current image in the virtual bronchial tree and its previous bifurcation, and the area change regularity value; assist the bronchoscope to position according to the abnormal possibility.

[0116] If the value of the area change law is smaller, the regularity of the true area change of the target hole area in the gray-scale image inside the bronchus in the analyzed image sequence is stronger, and the possibility of an abnormal structure appearing in front of the corresponding position in the virtual bronchial tree at the current moment is smaller; if the distance between the corresponding position of the current image in the virtual bronchial tree and its next bifurcation is closer, the possibility of an area mutation occurring in the target hole area due to the branch angle at the bifurcation position is greater, and the foreign object structure in the bronchus will also cause an area mutation in the target hole area, then the possibility of an abnormal structure appearing in front of the corresponding position in the virtual bronchial tree at the current moment is greater. Therefore, the value of the area change law and the abnormal possibility are in a positive correlation, and the distance between the corresponding position of the current image in the virtual bronchial tree and its next bifurcation and the abnormal possibility are in a negative correlation.

[0117] In this embodiment, the distance between the corresponding position of the current image in the virtual bronchial tree and its next bifurcation is subjected to a negative correlation mapping, and the product of the mapping result and the value of the area change law is normalized to obtain the abnormal possibility at the current moment. In a specific implementation manner of the embodiment of the present invention, the abnormal possibility is expressed by the formula:

[0118] ;

[0119] In the formula, is the abnormal possibility of the current image; is the distance between the corresponding position of the current image in the virtual bronchial tree and its next bifurcation; is the value of the area change law of the current image; is the exponential function with the natural constant e as the base. It should be noted that the abnormal possibility The greater it is, the greater the possibility of an abnormal structure appearing in front of the corresponding position in the virtual bronchial tree at the current moment.

[0120] When the abnormal possibility is greater than the preset normal threshold, there is an abnormal structure in front of the corresponding position of the current image in the virtual bronchial tree, which may be a foreign object, a tumor or tuberculosis, etc., and the bronchoscope gives an early warning, so as to remind the doctor to change the direction or path of the bronchoscope to make the operation safer; when the abnormal possibility is less than or equal to the preset normal threshold, there is no abnormal structure in front of the corresponding position of the current image in the virtual bronchial tree. It should be noted that in the embodiment of the present invention, the preset normal threshold takes an empirical value of 0.5, and the implementer can set it by himself according to the specific situation.

[0121] So far, the present invention is completed.

[0122] Embodiment 2:

[0123] The present invention provides a bronchoscope-assisted positioning system for tuberculosis care. Please refer to Figure 4, which shows the structural diagram of a bronchoscope-assisted positioning system for tuberculosis care provided by an embodiment of the present invention. The system includes:

[0124] A data acquisition module 510, configured to collect video images of the bronchus of a user to be measured in real time through a bronchoscope and obtain the virtual bronchial tree of the user to be measured;

[0125] A hole correction analysis module 520, configured to obtain the target hole area of the grayscale image inside the bronchus in each frame of the video image; according to the corresponding position of each frame of the grayscale image inside the bronchus in the virtual bronchial tree, the branch angle of the next bifurcation of the position, and the area difference between the target hole area of each frame of the grayscale image inside the bronchus and the target hole area of the grayscale image inside the bronchus before it, obtain the area correction amount of the target hole area of each frame of the grayscale image inside the bronchus;

[0126] A hole area correction module 530, configured to correct the area of the target hole area according to the distance between the corresponding position of each frame of the grayscale image inside the bronchus in the virtual bronchial tree and its next bifurcation and the area correction amount, and obtain the true area of the target hole area of each frame of the grayscale image inside the bronchus;

[0127] An auxiliary positioning module 540, configured to assist in positioning the bronchoscope according to the distance between the corresponding position of the last frame of the grayscale image inside the bronchus in the video image and its next bifurcation in the virtual bronchial tree, and the change in the true area of the target hole area of the grayscale image inside the bronchus between the corresponding position of the last frame of the grayscale image inside the bronchus in the virtual bronchial tree and its previous bifurcation.

[0128] It should be noted that: for the device provided in the above embodiment, only the above-mentioned functional module division is used for illustration. In practical applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the functions described above. In addition, a bronchoscope-assisted positioning system for tuberculosis care provided in the above embodiment and a method embodiment of a bronchoscope-assisted positioning method for tuberculosis care belong to the same concept. For the specific implementation process, please refer to the method embodiment, which will not be elaborated here.

[0129] Embodiment 3:

[0130] Figure 5 It is a schematic diagram of a computer device of a bronchoscope-assisted positioning device for tuberculosis care provided by an embodiment of the present invention. Exemplarily, such as Figure 5As shown in the figure, the computer device includes: a memory 601, a processor 602, and a computer program 603 stored in the memory 601 and running on the processor 602. When the processor 602 executes the computer program 603, the computer device can execute any one of the bronchoscope-assisted positioning methods for tuberculosis care introduced above.

[0131] In addition, an embodiment of the present application also protects a device, which may include a memory and a processor. Among them, an executable program code is stored in the memory, and the processor is used to call and execute the executable program code to execute a bronchoscope-assisted positioning method for tuberculosis care provided by an embodiment of the present application.

[0132] In this embodiment, the device can be divided into functional modules according to the above method examples. For example, it can correspond to each functional module, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is illustrative, only a logical function division, and there may be other division methods in actual implementation.

[0133] It should be understood that the device provided in this embodiment is used to execute the above-mentioned bronchoscope-assisted positioning method for tuberculosis care, so it can achieve the same effect as the above implementation method.

[0134] In the case of adopting an integrated unit, the device may include a processing module and a storage module. Among them, when the device is applied to a device, the processing module can be used to control and manage the actions of the device. The storage module can be used to support the device to execute mutual program codes, etc.

[0135] Among them, the processing module can be a processor or a controller, which can implement or execute various exemplary logical blocks, modules, and circuits included in the disclosure of the present application. The processor can also be a combination that realizes computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc. The storage module can be a memory.

[0136] It should be noted that the above sequence of embodiments of the present invention is only for description and does not represent the superiority or inferiority of the embodiments. The processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0137] Each embodiment in this specification is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other, and the key point of each embodiment is to illustrate the differences from other embodiments.

[0138] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A bronchoscope auxiliary positioning system for tuberculosis care, characterized in that: The system includes: A data acquisition module, used to acquire video images of the bronchi of the user to be tested in real time through a bronchoscope, and obtain a virtual bronchial tree of the user to be tested; a hole correction analysis module, for obtaining a target hole area of ​​each frame of the grayscale image of the bronchus in the video image; and obtaining an area correction amount of the target hole area of ​​each frame of the grayscale image of the bronchus in the video image according to the corresponding position of each frame of the grayscale image of the bronchus in the virtual bronchial tree, the branching angle of the next bifurcation of the position, and the area difference between each frame of the grayscale image of the bronchus in the video image and the target hole area of ​​the grayscale image of the bronchus in the video image; A hole area correction module is used to correct the area of ​​the target hole region according to the distance between the corresponding position of each frame of the internal bronchial grayscale image in the virtual bronchial tree and the next bifurcation and the area correction amount, so as to obtain the real area of ​​the target hole region of each frame of the internal bronchial grayscale image; An auxiliary positioning module is used to assist the bronchoscope in positioning according to the distance between the corresponding position of the last frame of the internal bronchial grayscale image in the video image in the virtual bronchial tree and its next bifurcation, and the change in the real area of ​​the target hole region of the internal bronchial grayscale image between the corresponding position of the last frame of the internal bronchial grayscale image in the virtual bronchial tree and its previous bifurcation.

2. A bronchoscope auxiliary positioning system for tuberculosis care according to claim 1, characterized in that: The step of obtaining the target hole area of ​​the grayscale image of the inside of the bronchus in each frame of the video image comprises: For each frame of the grayscale image of the inside of the bronchus, the maximum inter-class variance method is used to obtain the segmentation threshold for the grayscale values ​​of the pixels in the grayscale image of the inside of the bronchus, and the connected domain formed by the pixels whose grayscale values ​​are less than the segmentation threshold is taken as the original hole area; The original hole region that the bronchoscope is going to pass through in the grayscale image inside the bronchus in the video image is recorded as the target hole region of each frame of the grayscale image inside the bronchus.

3. A bronchoscope auxiliary positioning system for tuberculosis care according to claim 2, characterized in that: The step of obtaining the area correction amount of the target hole region of each frame of the grayscale image inside the bronchus includes: According to the corresponding position of each frame of the grayscale image inside the bronchus in the virtual bronchial tree, the branching level of the corresponding image is determined; the branching angle of the next bifurcation at the corresponding position of each frame of the grayscale image inside the bronchus in the virtual bronchial tree is recorded as the target branching angle of the corresponding image; Recording the grayscale image of the inside of the bronchus at each bifurcation position of the virtual bronchial tree as a bifurcation position image; in the video image, recording the sequence of grayscale images of the inside of the bronchus between each bifurcation position image and the bifurcation position image of the previous frame as a same-branch image sequence; obtaining the final area change function of each same-branch image sequence; The target branch angle and the branch order of each frame of the internal bronchial grayscale image are substituted into the final area change function of the same branch image sequence in which the internal bronchial grayscale image is located to obtain the area correction of the target hole area of ​​each frame of the internal bronchial image.

4. A bronchoscope auxiliary positioning system for tuberculosis care according to claim 3, characterized in that: The method of obtaining the final area variation function of each same-branch image sequence includes: A nonlinear analysis is performed on the branching order, the target branching angle and the area variation, and an initial area variation function of each image sequence of the same branch is constructed. The initial area variation function includes several regression coefficients. Obtain a historical branch image sequence for each same branch image sequence, and use the difference between the area of ​​the target hole region of each frame image in the historical branch image sequence and the area of ​​the target hole region of the previous frame image as the area change of the target hole region of each frame image in the historical branch image sequence; Substituting the branching level and target branching angle of each frame image in the historical branch image sequence into the initial area change function to obtain an area change fitting value of each frame image in the historical branch image sequence; According to the difference between the area change fitting value and the area change of each image in the historical branch image sequence, the error function of the historical branch image sequence is obtained; the value of the regression coefficient corresponding to the minimum value of the error function is used as the optimal value of each regression coefficient in the initial area change function; the optimal value of the regression coefficient is substituted into the initial area change function to obtain the final area change function of each same-branch image sequence.

5. A bronchoscope auxiliary positioning system for tuberculosis care according to claim 4, characterized in that: The step of obtaining the real area of ​​the target hole region of each frame of the grayscale image of the interior of the bronchus comprises: Determine whether the number of the original hole regions in each frame of the grayscale image of the inside of the bronchus is greater than a constant 1, and if not, use the area of ​​the original hole region as the true area of ​​the target hole region of the corresponding image; If yes, then according to the distance between the corresponding position of each frame of the grayscale image of the inside of the bronchus in the virtual bronchial tree and the next bifurcation thereof, and the area correction amount, obtain the area correction coefficient of the target hole area of ​​the corresponding image; The area correction coefficient is used to weight the area of ​​the target hole region of each frame of the bronchial internal grayscale image to obtain the true area of ​​the target hole region of each frame of the bronchial internal grayscale image.

6. A bronchoscope auxiliary positioning system for tuberculosis care according to claim 5, characterized in that: The method of assisting the bronchoscope in positioning according to the distance between the corresponding position of the last frame of the bronchial internal grayscale image in the video image and its next bifurcation in the virtual bronchial tree, and the change of the real area of ​​the target hole region of the bronchial internal grayscale image between the position and its previous bifurcation, comprises: The last frame of the grayscale image of the bronchus in the video image is recorded as the current image, and the sequence of grayscale images of the bronchus between the corresponding position of the current image in the virtual bronchial tree and the previous bifurcation is recorded as the analysis image sequence; according to the change of the real area of ​​the target hole area of ​​the grayscale image of the bronchus in the analysis image sequence, the area change law value of the current image is obtained; According to the distance between the corresponding position of the current image in the virtual bronchial tree and its next bifurcation, and the area change law value, the abnormal possibility of the current image is obtained; and the bronchoscope is assisted in positioning according to the abnormal possibility.

7. A bronchoscope auxiliary positioning system for tuberculosis care according to claim 6, characterized in that: The step of obtaining the area change regularity value of the current image includes: Record the sequence consisting of the real areas of the target hole regions of the grayscale images inside the bronchus in the analysis image sequence as an area sequence; obtain the first-order difference sequence and the second-order difference sequence of the area sequence respectively, and record the mean of the elements in the second-order difference sequence as the area change rate of the current image; obtain the area discrete value of the current image according to the discrete degree of the elements in the first-order difference sequence; and take the number of changes in the signs of two adjacent elements in the first-order difference sequence as the area trend value of the current image; The area change regularity value of the current image is obtained according to the area change rate, the area discrete value and the area trend value.

8. A bronchoscope auxiliary positioning system for tuberculosis care according to claim 7, characterized in that: The method of assisting the bronchoscope in positioning according to the possibility of abnormality includes: When the abnormal possibility is greater than the preset normal threshold, there is an abnormal structure in front of the corresponding position in the virtual bronchial tree of the current image; when the abnormal possibility is less than or equal to the preset normal threshold, there is no abnormal structure in front of the corresponding position in the virtual bronchial tree of the current image.

9. A bronchoscope auxiliary positioning system for tuberculosis care according to claim 1, characterized in that: The total number of pixels in the target hole area is the area of ​​the target hole area.

Images