Path generation method, system and device and readable medium

Abstract

The invention provides a path generation method, system and device and a readable medium. The path generation method comprises the following steps: reconstructing a bronchial tree 3D mask according to a CT image, extracting a center line point set of the bronchial tree 3D mask and carrying out structured processing to obtain a center line tree, selecting a drawing starting point and a target area of a navigation schematic diagram of the bronchoscope, starting to traverse the center line tree from a target node corresponding to the target area, and gradually backtracking a father node of the target node in the centerline tree until the father node reaches the drawing starting point to form a tracking path, recording nodes included in the tracking path to generate tracking data, and drawing a navigation schematic diagram from the drawing starting point to the target area. Through the scheme, subjective errors of manual operation can be reduced, the time of manual navigation path planning is shortened, and the working efficiency of doctors can be remarkably improved through rapid and automatic path generation.

Description

Technical Field

[0001] The present invention belongs to the field of medical data processing technology, and in particular, relates to a path generation method, system, device and readable medium for reaching a target area to be inspected or treated during bronchoscopy. Background Art

[0002] During bronchoscopy, the doctor first needs to determine the location of the target area under the CT image and plan the inspection path in advance. Hand-drawn navigation is a simple and effective navigation method. This method uses a layered schematic diagram to indicate the direction in which each fork under the bronchoscope should go, thereby achieving the purpose of navigation. Figure 1 As shown in the figure, one guide curve represents one lumen, and two guide curves at the same level represent the emergence of two lumen branches. The use of nested guide curves can represent the lumens of different layers of the bronchus. Figure 1 The solid area in the middle is the final lesion inspection area. The doctor can operate the bronchoscope into the corresponding cavity at each bifurcation according to the hand-drawn navigation, and finally reach the target area corresponding to the target site of bronchoscopy.

[0003] In the actual operation of the hospital, the usual practice is for doctors to manually draw hand-drawn navigation diagrams based on CT images, which largely relies on the doctor's experience and professional skills. It is difficult for doctors with insufficient experience to accurately draw the optimal path to the target area (such as lesions, biopsy), which increases the risk during the operation. In the manual drawing process, any small error may be magnified during the navigation process, especially when the path involves multiple branches. This may cause the navigation path to deviate from the target area, thereby affecting the success rate of the operation. The manually drawn path may not fully consider the physiological characteristics of the bronchi, such as curvature and diameter changes, resulting in the selected path being unfavorable for the smooth progress of bronchoscopic navigation. Although this method can provide personalized navigation guidance based on the doctor's professional knowledge and experience, it also has problems such as accuracy, drawing difficulty and lack of real-time performance. In the complex bronchial tree structure, manual drawing is prone to introduce errors and is difficult to adapt to the surgical process. Summary of the invention

[0004] The present invention proposes a path generation method, system, device and readable medium, which can automatically output a hand-drawn navigation diagram based on the input CT image to improve the doctor's work efficiency. The first aspect of the present application provides a path generation method, the method comprising:

[0005] Reconstruct the bronchial tree 3D mask based on the CT image;

[0006] Extract the centerline point set of the bronchial tree 3D mask, and perform structured processing on the centerline point set to obtain the centerline tree;

[0007] Select the starting point and target area for drawing the navigation diagram of the bronchoscope on the centerline tree;

[0008] Starting from the target node corresponding to the target area, traversing the centerline tree, gradually tracing back to the parent node of the target node in the centerline tree until reaching the drawing starting point to form a tracking path, recording the nodes in the tracking path to generate tracking data, and also generating tracking data for all nodes included in the tracking path;

[0009] Draw a navigation diagram from the starting point to the target area based on the tracking data.

[0010] In an implementation of the first aspect, the centerline tree is traversed starting from the target node corresponding to the target area, and the parent node of the target node in the centerline tree is gradually traced back until the drawing starting point is reached to form a tracking path, and the nodes in the tracking path are recorded to generate tracking data, including: selecting pixel points on the centerline tree as reference points, and the reference points at least include pixel points at bifurcations; forming a local neighborhood around the reference points; traversing the reference points on the tracking path, identifying one or more connected domains in the local neighborhood corresponding to the cross-section of the current layer of the bronchial tree 3D mask, and the connected domain is a set of interconnected pixel points in the local neighborhood; analyzing the connectivity of the connected domain to generate the tracking data.

[0011] In an implementation of the first aspect, forming a local neighborhood within a predetermined area around a reference point includes: a predetermined area formed with the reference point as the center and a predetermined distance as a radius is the local neighborhood; wherein the predetermined distance can be determined based on the corresponding bronchial diameter at the reference point, and the local neighborhood includes a connected domain related to the bronchial tracing path, which helps to accurately identify the bronchial morphology, especially the bronchial morphology when the lumen diameter changes or the path bifurcates.

[0012] In an implementation of the first aspect, one or more connected domains are identified in a local neighborhood corresponding to a cross-section of a current layer of a bronchial tree 3D mask, including: extracting a cross-section corresponding to the local neighborhood in the bronchial tree 3D mask; selecting an unmarked pixel point as an initial point in the local neighborhood corresponding to the cross-section of the current layer; starting from the initial point, checking all adjacent pixel points of the initial point, and marking adjacent pixel points that meet connectivity conditions; starting from the initial point, traversing and checking along adjacent pixel points until all pixel points connected to the initial point are marked, forming one or more connected domains consisting of multiple mutually connected marked pixel points.

[0013] In an implementation of the first aspect, analyzing the connectivity status of the connectivity domain and generating tracking data includes:

[0014] According to the correlation between a connected domain of the current layer and the reference point, the connected domain is identified and tracked;

[0015] Traverse and check each tracking connected domain, and generate tracking data based on the connectivity between the tracking connected domain of the current layer and the connected domain of the adjacent layer of the current layer.

[0016] In an implementation of the first aspect, according to the association between a connected domain of the current layer and the reference point, identifying and tracking the connected domain includes:

[0017] In the case where a connected domain of the current layer includes a reference point of a local neighborhood corresponding to the current layer or is directly connected to a reference point in the local neighborhood, it is identified as a tracking connected domain.

[0018] In an implementation of the first aspect, drawing a navigation diagram from a drawing starting point to a target area according to tracking data includes:

[0019] Draw multiple guide curves based on the tracking data, where the guide curves are used to indicate the multi-layer branch structure that needs to be passed from the drawing starting point to the target area;

[0020] Each guiding curve corresponds to a bronchial branch entrance in the multi-layer branch structure; wherein a bronchial branch entrance is a bronchial opening of a bronchial segment where a target area is located.

[0021] In an implementation of the first aspect, the navigation schematic diagram is rotated or flipped according to the detection image of the bronchoscope, so that the navigation schematic diagram matches the detection orientation corresponding to the bronchoscope.

[0022] A second aspect of the present application provides a path generation system for executing the path generation method in the first aspect, the system comprising:

[0023] The reconstruction module is used to reconstruct the bronchial tree 3D mask based on the CT image;

[0024] The structured module is used to extract the centerline point set of the bronchial tree 3D mask and perform structured processing on the centerline point set to obtain the centerline tree;

[0025] A selection module, used for selecting a drawing starting point and a target area of ​​a navigation diagram of a bronchoscope on a centerline tree;

[0026] A data generation module is used to traverse the centerline tree starting from the target node corresponding to the target area, gradually trace back the parent node of the target node in the centerline tree until reaching the drawing starting point to form a tracking path, and record the nodes or all nodes included in the tracking path to generate tracking data;

[0027] The drawing module draws a navigation diagram from the drawing starting point to the target area according to the tracking data.

[0028] The third aspect of the present application provides an electronic device for executing the path generation method in the first aspect, the device comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions.

[0029] A fourth aspect of the present application provides a computer-readable storage medium for executing the path generation method in the first aspect, wherein the computer-readable storage medium stores computer instructions.

[0030] The technical solution of the present application is used to obtain the centerline tree of the bronchus through the reconstructed 3D mask of the bronchial tree. First, the specific node of the target area in the centerline tree is identified, and its parent node is gradually traced upward from the target area node, and all reference points of the entire centerline tree are traversed until the starting point node corresponding to the specified drawing starting point (such as the entrance of the trachea, a bronchial opening, etc.) is reached. According to the traversal data generated during the traversal process, a navigation diagram for guiding the movement of the bronchoscope is formed. A clear navigation diagram that can identify the bronchus from the bronchoscope entering the bronchus until reaching the target area is generated based on the tracking data. The navigation diagram can clearly indicate the starting point, the direction of the path, each bifurcation point, and the target area. This solution can automatically generate navigation diagrams, reduce subjective errors in manual operations, and reduce the time for manual navigation path planning. Rapid automated path generation can significantly improve the doctor's work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0032] Figure 1 A schematic diagram of hand-drawn navigation in the prior art is shown;

[0033] Figure 2 According to one embodiment of the present application, a schematic structural diagram of a bronchoscope is shown;

[0034] Figure 3 According to an embodiment of the present application, a schematic diagram of a process of a path generation method is shown;

[0035] Figure 4 According to an embodiment of the present application, a schematic diagram of a process for generating tracking data is shown;

[0036] Figure 5According to an embodiment of the present application, a flowchart of a specific tracking process is shown;

[0037] Figure 6 According to an embodiment of the present application, a system structure diagram of a path generation system is shown. DETAILED DESCRIPTION

[0038] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0039] It should be noted that the following detailed descriptions are all illustrative and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.

[0040] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0041] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0042] A bronchoscope typically includes an image capture device, such as a camera, located at the front end of the bronchoscope to be inserted into a patient and connected to a display to provide medical personnel with a view of the portion of the airway where the front end of the bronchoscope is located.

[0043] The target areas usually include the distal bronchial small airways, lung lobes or segments, lymph nodes, parabronchial nodules, etc., where there are lesions such as polyps, tumors, foreign bodies, etc. in the bronchi, which need to be directly reached for biopsy or removal.

[0044] The starting point for drawing is usually the starting point of bronchial navigation, which can be selected according to the entry method of the clinical equipment and the patient's condition, including navigation starting points such as the tracheal entrance, bronchial opening, subglottis, and main bronchial bifurcation point.

[0045] See also Figure 2 , shows a schematic structural diagram of a bronchoscope 1. The bronchoscope 1 may be disposable or reusable. The bronchoscope 1 includes an insertion portion 2. A handle 3 for operating the bronchoscope 1 is arranged at the base end of the insertion portion 2. A head portion 5 is provided at the front end of the insertion portion 2. A camera assembly 6 is positioned in the end portion 5 and is configured to transmit an image signal to a display 11 through a processing device 12 of the bronchoscope 1. The display 11 may allow an operator to view images captured by the camera assembly 6 of the bronchoscope 1 and a target navigation path for assisting the operation of the bronchoscope 1.

[0046] The display 11 is coupled to the processor and is operable to render an image of the patient's anatomy. Such an image may be based on a set of images obtained before surgery (e.g., CT or MRI scans, 3D mapping, etc.). The view of the patient's anatomy provided by the display 11 may also be dynamically changed based on the current identified real-time location. For example, as the bronchoscope 1 moves within the patient's body, its real-time position data may cause the processor to update the view of the patient's anatomy in the display 11 in real time to depict the area of ​​the patient's anatomy captured by the camera assembly 6 of the bronchoscope 1.

[0047] See also Figure 3 , which shows a flow chart of a path generation method provided by an embodiment of the present application. The method can be implemented by a computing device connected to the bronchoscope 1, for example, a computer terminal. As an optional method, the computing device and the bronchoscope 1 can be combined into one device. For ease of explanation, the following embodiments are all based on the computing device as the execution subject. The path generation method specifically includes:

[0048] Step S10: reconstructing a bronchial tree 3D mask from the segmented CT image.

[0049] In the technical solution provided in step S10, data after block division and normalization processing are selected from the thin-layer CT image corresponding to the bronchus as input objects and input into the pre-trained segmentation model for segmentation to obtain the bronchial tree 3D mask.

[0050] Among them, each frame of CT image obtained in the CT examination of the bronchus can be a cross-section of the bronchus. First, its thin-layer CT image data (generally less than 1mm) is selected as the analysis object, and the CT image is divided into multiple small blocks. For example, the block division is performed in a block manner of 64x64x64 size and a step size of 16 to form multiple sub-blocks. After the block division, the magnitude difference of the image gray value is eliminated by normalization processing such as linear normalization or Z-Score. The data block is input as an input object into the pre-trained segmentation model for inference prediction, and the voxels that meet the bronchial location characteristics in the prediction analysis process are screened based on the output score. Specifically, for voxels whose output results are greater than or equal to the preset threshold, they are determined to be bronchial voxels, and voxels less than the preset threshold are considered to be other backgrounds. The corresponding output mask of the data block is obtained. Specifically, when the prediction score obtained by the segmentation model is in the range of 0 to 1, the preset threshold can be set to 0.5. When the prediction score is greater than or equal to 0.5, the corresponding voxel is considered to be a bronchial voxel, otherwise it is considered to be a background or other tissue-related voxel.

[0051] The 3D bronchial tree is a mask used to represent the bronchial tree structure in three-dimensional space. In the bronchial tree 3Dmask, the mask is a three-dimensional data set, in which the value of each volume pixel can be used to indicate whether it belongs to the bronchial tree structure.

[0052] All input blocks of all relevant data for bronchial reconstruction are input into the segmentation model, and the output blocks including bronchial voxels, background, and other tissues in the segmentation model are reassembled, restored, and spliced ​​according to the spatial position of the original CT data to obtain a 3D mask of the bronchial tree based on the 3D sequence. The bronchial tree 3D mask includes a detailed three-dimensional representation of the branch structure of the entire bronchial system from the main bronchi to the lobar bronchi, segmental bronchi, and even finer bronchial branches.

[0053] The training process includes: selecting the entire bronchial structure CT data, screening the CT data that meets the layer thickness conditions (thin layer) as the annotation object, manually annotating the main bronchi, secondary bronchi and their branches, bronchial regional boundaries, etc. that can form the bronchial tree mask, and dividing the training set, validation set, and test set into units of CT sequences after the manual annotation is completed. The number of training sets, validation sets, and test sets is 8:1:1, and a 3D segmentation network model Vnet is constructed. The input size of the network model is set to 64x64x64, and the size of the output layer is also 64x64x64. The final output layer of the network is activated by the sigmoid function, and the output range of the output layer output value is between 0 and 1. The value of each output node represents the probability that the voxel belongs to a certain part of the bronchial structure. According to the preset threshold, the value exceeding the preset threshold is selected as the recognition result corresponding to the part.

[0054] Since the dimension of a CT sequence is generally 512x512xZ or 1024x1024xZ (Z is the sequence depth), it cannot be input into the network model for a row calculation and training, and needs to be cut into blocks. The entire process can refer to the network input size of Vnet, and the original 3DCT data and the corresponding annotated 3Dmask are divided into blocks according to the size of 64x64x64 and the step size of 16 to obtain the training data and corresponding labels that meet the network input, thereby improving the stability and generalization ability of the pre-trained prediction model. At the same time, the training data is normalized to between 0 and 1 according to the window width WW=1500 and the window level WL=-400. The normalization formula is as follows:

[0055] Where x is the original CT value and f(x) is the normalized result:

[0056]

[0057] The entire model training process also includes randomly initializing the network model, setting dice loss as the target loss function for network model training, and using the Adam optimizer to optimize the model parameters to obtain a pre-trained prediction model.

[0058] In some implementations, for CT image methods, bronchial segmentation can be performed on the lung bronchi by traditional image algorithms, such as region growing algorithm and / or Hessian matrix algorithm, etc., or bronchial segmentation can be performed on the lung bronchi by artificial intelligence methods such as deep learning, for example, bronchial segmentation can be performed on the lung bronchi based on a 2D or 3D UNet deep learning segmentation network, which is not described in detail here.

[0059] Step S20: Extract the centerline point set of the bronchial tree 3D mask, and perform structured processing on the centerline point set to obtain the centerline tree. Perform point extraction on the centerline of the bronchial tree 3D mask to obtain a discrete centerline point set, select the point in the centerline point set closest to the CT head direction as the starting point of the bronchial tree, and use the starting point as the root node R of the centerline tree of the bronchial tree 3D mask. Starting from the root node, mark it as visited, and for each unvisited adjacent point, recursively perform a depth-first traversal, and add these points to the tree as child nodes of the current node. Then, all centerline points connected to it can be continuously added to the tree structure from the root node until all points are included in the tree, and finally a tree-type data storage structure is formed.

[0060] In this process, all non-leaf nodes have child nodes, and all nodes except the root node have parent nodes, thus forming a standard tree structure. In constructing the tree structure, given a target point A (the target area corresponds to the target area to be reached by bronchoscopy), the path represents the main route from the root node R (the starting point of the bronchial tree) to the target point A (a specific position in the bronchial tree). The route can start from the target point A and continuously track its parent nodes along the centerline tree until it reaches the root node R. .

[0061] In one embodiment, the centerline point set may include all pixel points located on the centerline of the bronchial tree 3D mask.

[0062] Generally speaking, the human airway tree has about 24 branches from the first level to the alveoli, which is shaped like an inverted tree, so it can be extended to the concept of tree structure in data structure. The trachea is divided into the left and right main bronchi at the bifurcation, and the bronchial entrance can be compared to the root node.

[0063] In the technical solution of step S20, the method for extracting the centerline point set of the bronchial tree 3D mask may include: extracting the centerline of the bronchial tree 3D mask based on a skeleton extraction algorithm. The centerline point set of the bronchial tree 3D mask is obtained by point extraction of the centerline, and the centerline point set includes multiple target key points. The target key point may include at least a non-leaf node, or each pixel point on the centerline in a discrete state.

[0064] Step S30: Select the drawing starting point and target area of ​​the navigation diagram of the bronchoscope 1 on the centerline tree. It is understandable that the drawing starting point is usually selected from a node of the centerline tree, and the node represents the main bronchial entrance where the navigation of the bronchoscope 1 starts, or a suitable anatomical position selected according to clinical needs. The drawing starting point can be calibrated at the corresponding bronchial entrance using a visual marking method based on the real-time position of the bronchoscope 1, but it is also possible to calibrate the corresponding bronchial entrance based on the mapping relationship automatically established between the real-time position identified by the bronchoscope and the drawing starting point, which is not limited here.

[0065] Step S40: Traverse the centerline tree starting from the target node corresponding to the target area, gradually trace back the parent node of the target node in the centerline tree until reaching the drawing starting point to form a tracking path, and record the nodes or all nodes included in the tracking path to generate tracking data. First, identify the target node of the target area (such as biopsy, lesion) in the centerline tree, start tracing the parent node of the target area from the target area node, and gradually trace its parent node upward in the centerline tree in an iterative loop until reaching the drawing starting point (such as the tracheal entrance, a bronchial opening, etc.) to form a tracking path. The parent node is continuously tracked along the centerline on the tracking path in an iterative loop to ensure that the tracking process can cover as many or all nodes as possible between the target node and the starting point, and all nodes tracked during the backtracking process will be stored to generate tracking data.

[0066] Specifically, a tracking list can be initialized before traversal to store tracking data during the tracking process. Starting from the target node, recursively traverse the nodes of the entire centerline tree, add the current node to the tracking list, and if the current node is the drawing starting point, stop the loop; update the current node to the parent node of the current node and continue to backtrack until the current node is the drawing starting point. The entire tracking process forms a tracking path, and the corresponding traversed nodes are saved as complete tracking data.

[0067] Step S50: Generate a navigation diagram from the drawing starting point to the target area according to the tracking data. The navigation diagram generated according to the tracking data may include multiple guide routes. For example, the guide route may be a closed / non-closed curve, and the shape and position of each curve in the guide curve, each closed / non-closed curve can reflect the position and size of the corresponding bronchial opening in the bronchial tree, and multiple closed / non-closed curves can reflect that it gradually moves from the drawing starting point to the target area in a multi-level progressive manner. The nested relationship between the guide curves represents the multi-layer branch structure of the trachea, which can help doctors or navigation systems identify the specific tracheal branches where the bronchoscope 1 is located. The navigation diagram can guide the bronchoscope 1 to move from the drawing starting point to the target area in a layer-by-layer progressive manner. The formation of the navigation diagram can apply a streamline algorithm, a Bezier curve algorithm, and other graphic drawing algorithms to convert the path into a hand-drawn style image that is easy for doctors to understand, and arrows, annotations, and other symbols can also be added to the navigation diagram to assist doctors in intuitively understanding the navigation path.

[0068] See also Figure 4 , shows a schematic diagram of a process for generating tracking data provided by an embodiment of the present application. The process of generating a tracking path in the technical solution in step S40 includes:

[0069] Step S41: Select pixel points on the centerline tree as reference points, and the reference points at least include pixel points at the bifurcation.

[0070] It can be understood that the centerline point set included in the centerline tree is composed of a series of pixel points, and the reference points selected from the pixel points are used to identify key positions in the bronchial path, usually including but not limited to bronchial bifurcation points and path turning points, etc. According to the selected reference points, a complete tracking path from the drawing starting point to the target area is constructed by continuously tracing back from the target area, and further connected domain analysis is performed to form tracking data.

[0071] Step S42: forming a local neighborhood in a predetermined area around the reference point. The local neighborhood of the predetermined area is formed with the reference point as the center and the predetermined distance as the radius; wherein the predetermined distance is determined according to the bronchial diameter corresponding to the reference point. For example, a predetermined area formed with the reference point as the center and two or three times the bronchial diameter corresponding to the reference point can be the local neighborhood.

[0072] It is understandable that the size of the local neighborhood area is very important for the accuracy of the connected domain analysis. A local neighborhood area that is too large may include irrelevant bronchial branches, while a local neighborhood area that is too small may lose some bronchial information. The local neighborhood includes a pixel point and a set of pixels within a fixed range around it, usually a regular area such as a rectangle or circle. In this application, the local neighborhood can be defined as a predetermined area formed by a circle with a radius of twice the current lumen diameter and a selected reference point as the center, to ensure that the formed local neighborhood contains sufficient bronchial structure information.

[0073] Step S43: traverse the reference points on the tracing path, and identify one or more connected domains in the local neighborhood corresponding to the cross-section of the current layer of the bronchial tree 3D mask. The connected domain is a set of interconnected pixel points in the local neighborhood. The formation of the connected domain in the local neighborhood can be achieved by traversing all the pixel points in the local neighborhood, and gradually forming one or more connected domains from the marked interconnected point sets. Specifically, starting from the starting point of the centerline tree, traverse each target key point or reference point in the centerline tree one by one. The target key point or reference point includes a bifurcation point or other meaningful position point (such as an important path node), extract the cross-section of the current layer corresponding to the local neighborhood where the current reference point is located, and identify one or more connected domains in the cross-section of the current layer. The connected domain is an area composed of interconnected pixels or voxels, which represents the local structure of the bronchial tree 3D mask on the cross-section.

[0074] In step S43, the formation of each connected domain starts from the initial point, and continuously expands by detecting the connectivity of adjacent points until the entire connected domain is identified and formed. Identifying one or more connected domains in the local neighborhood corresponding to the cross-section of the current layer of the bronchial tree 3D mask includes: extracting the cross-section corresponding to the local neighborhood in the bronchial tree 3D mask; selecting unmarked pixels as the initial points in the local neighborhood corresponding to the cross-section of the current layer. Starting from the initial point, all adjacent pixels of the initial point are checked, and the adjacent pixels that meet the connectivity conditions are marked. The new initial point is continuously iterated through the adjacent pixels, so that it is traversed and checked along the adjacent pixels after the iteration until the pixels connected to the initial point are marked, forming one or more connected domains composed of multiple marked pixels connected to each other.

[0075] It can be understood that in the bronchial tree 3D mask, a cross section corresponding to the local neighborhood is extracted, which represents the structure of the bronchial tree on the plane where the reference point is located. Select the pixel points that have not been assigned or classified in the process of connected domain identification as the initial point, and check the adjacent pixel points of its starting point step by step in an iterative manner. Whenever an adjacent pixel point that meets the connectivity condition is identified, it is used as a new starting point and continues to expand until no new adjacent pixel points that meet the connectivity condition can be found or the boundary of the local neighborhood is reached, and the set of all pixel points that meet the connectivity condition is identified as a connected domain. Then, continue to look for new connected domains from the unlabeled pixels in the local neighborhood until all local neighborhoods are traversed.

[0076] It can be understood that the two-dimensional plane cut along a certain axis of the bronchial tree 3D mask can be understood as a cross-section of the bronchial tree, showing cross-sections of different layers. The cross-section of each layer can observe the structure and morphology of the bronchial tree in the corresponding layer.

[0077] It can be understood that in the bronchial 3D mask, one or more connected domains can be identified in each cross-section layer. When a connected domain is identified, the connectivity conditions may include 4-connectivity where the initial point is connected to 4 pixels above, below, left, and right, and 8-connectivity where the initial point is connected to pixels in 8 directions around it. The connectivity method can usually be confirmed by traversing and marking all interconnected pixel points including depth-first search (DFS), breadth-first search (BFS) and flood fill algorithm to form a connected domain.

[0078] Step S44: Analyze the connectivity of the connected domain and generate tracking data. The identified connected domain is analyzed for connectivity. According to the analysis results, the node information passed through in the path from the target area to the starting point of drawing is recorded, the connectivity status of each connected domain is marked (e.g., directly connected, indirectly connected), and the distance or position relationship with the target node is recorded. The irrelevant connected domains are deleted to generate tracking data.

[0079] Step S44 also includes: identifying a tracking connected domain according to the association between a connected domain of the current layer and the reference point; traversing and checking each tracking connected domain, and generating tracking data according to the connectivity between the tracking connected domain of the current layer and the connected domain of the adjacent layer of the current layer.

[0080] Among them, according to the correlation between a connected domain of the current layer and the reference point, identifying and tracking the connected domain includes: when a connected domain of the current layer includes the reference point of the local neighborhood corresponding to the current layer, or when the connected domain is directly connected to the reference point in the local neighborhood, it can be identified as a tracking connected domain.

[0081] It is understandable that the existence of direct or indirect connections between the connected domain and the reference points of the local neighborhood corresponding to the cross section of the current layer is crucial for the acquisition of tracking data. A connected domain containing reference points in the local neighborhood of the current layer or a connected domain with no interruption or gap between the reference points and the connected domain, and a connected path between the two can be identified as a tracking connected domain, and the tracking connected domain corresponds to the target area for tracking and analysis in multi-level data.

[0082] Among them, there is no interruption or gap between the reference point and the identified connected domain, and there is a connected path between the two. The situation includes: when a reference point is located at the bifurcation of the bronchial tree, the reference point is not in the connected domain, but the connected domain reaches the reference point through a continuous path (such as a group of connected pixels), then this connected domain is directly connected to the reference point. When the number of connected domains in the local neighborhoods of adjacent layers is different, and the number of connected domains identified in the local neighborhood of the current layer increases compared with the previous layer, the bronchus corresponding to the current layer is bifurcated.

[0083] Furthermore, in multi-layer data, the identification of connected domains not only focuses on the current layer, but also needs to consider the relationship between the current layer and the adjacent layer (the previous layer of the current layer). If the tracking connected domain of the current layer is connected to the tracking connected domain of its adjacent layer, then it may be identified as forming part of the tracking path. Assume that there is a connected domain A in the first layer and a connected domain B in the second layer, and the boundary of B is directly adjacent to A in a certain axis direction, then the connected domain A and the connected domain B are directly adjacent. When a connected domain identified in the local neighborhood of the current layer does not belong to the tracking connected domain, and the connected domain is not connected to the tracking connected domain in the adjacent layer, the connected domain should be deleted from the tracking data to avoid entering other irrelevant branches of the bronchus by mistake.

[0084] See also Figure 5 , showing a flow chart of a specific tracking process in an embodiment of the present application. Initialize an empty tracking data storage list from the drawing starting point S to the target area E, which is used to record the tracking data of the main path L from the drawing starting point S to the target area E; then, along the extracted tracking path bronchial centerline, traverse all pixel points from the drawing starting point S to the target area E (bronchial terminal or lesion, biopsy point) step by step. If the current point is P i As the starting point and traversing, the current point P i The local neighborhood is formed by twice the diameter of the bronchus corresponding to the current point of the circle. The local neighborhood range is used to analyze the connected area around the current point Pi. i , analyze the connected area on the cross section within the neighborhood range, and obtain one or more connected domains C of the current layer i ; Continue traversal: According to the above steps, process each reference point in the path and its corresponding connected domain in turn, filter out irrelevant connected domains and delete them according to the correlation between the connected domain and the reference point and the connectivity between multiple layers of connected domains, continuously update the connected domain set of the current layer, and perform the same analysis and judgment on the newly added connected domain when bifurcation occurs, until traversing to the target area E, and store the node data corresponding to all tracked connected domains as tracking data.

[0085] Among them, filtering irrelevant connected domains includes: checking C i For each connected domain in the layer, any domain that is neither a trace path connected domain nor connected to the previous layer C i-1 The connected domains connected to the trace path connected domains in will be removed. Among them, the method for determining the bifurcation of the bronchial branch includes: in the process of traversing the cross-section extracted by the 3D mask of the bronchial tree, when the i-th layer P i The number of connected domains corresponding to the reference point C i More than the i-1th layer P i-1 C corresponding to the reference point i-1When , it means that from the i-1th layer to the i-th layer, the bronchial branches enter multiple paths, indicating that the bronchial path bifurcates at the i-th layer, P i It corresponds to the bifurcation of the bronchus. Every time the path bifurcates, a new branch structure will be added on the basis of the original path. i Analyze the corresponding increased connected domain to obtain P i All connected domains corresponding to the trace connected domains connected to the main path L.

[0086] Specifically, judging C i Whether it is related to C i-1 The method for connecting any connected domains in the method also includes: using the intersection or adjacency matrix method to determine whether two connected domains are adjacent or have common pixels to determine the connectivity of any two connected domains, and determining two connected domains that are not adjacent and have no common pixels as unrelated connected domains.

[0087] In step S50, the tracking data draws a navigation diagram from the drawing starting point to the target area, including: drawing multiple guide curves based on the tracking data, the guide curves are used to indicate the multi-layer branch structure that needs to be passed from the drawing starting point to the target area; each guide curve corresponds to a bronchial branch entrance in the multi-layer branch structure; one of the bronchial branch entrances is the bronchial opening of the bronchial segment where the target area is located.

[0088] According to the tracking data and the drawing starting point and target area, multiple guide curves are drawn along the drawing starting point according to the multi-layer branching structure of the bronchus until the target area is reached, and each guide curve corresponds to a bronchial branch entrance in the multi-layer branching structure; wherein a bronchial branch entrance is the bronchial opening of the bronchial segment where the target area is located. It is understandable that if the same bronchial segment has multiple bronchial entrances, the guide marks of the multiple bronchial entrances are different to achieve the distinction between different bronchial entrances. The bronchoscope 1 is used to reach the target area matched by the lesion under the guidance of the navigation schematic diagram. The target area can be understood as the location of the lesion or the vicinity of the lesion.

[0089] In some embodiments, due to the different imaging angles and orientations of CT imaging and bronchoscope 1, the hand-drawn navigation diagram needs to be rotated or flipped to ensure that it is consistent with the imaging direction of the bronchoscope. The purpose of this transformation is to make the imaging observed by the doctor when using the bronchoscope consistent with the path on the navigation diagram, thereby improving the accuracy of the operation. According to the starting navigation point of the bronchoscope, the navigation diagram is rotated or flipped to make the navigation diagram consistent with the specific imaging orientation corresponding to the bronchoscope 1. Specifically, it includes: adaptively rotating or flipping according to the position of the drawing starting point S. For example, the transformation rule can be implemented in the following way: when the drawing starting point S is located in the left upper lobe branch (also known as the intrinsic segment), the navigation diagram G needs to be rotated 90 degrees clockwise; when the drawing starting point S is located in the right upper lobe, the navigation diagram G needs to be rotated 90 degrees counterclockwise; when the drawing starting point S is located in the lingual segment, the left lower lobe, and the right lower lobe basal segment, the navigation diagram G needs to be flipped horizontally. By adjusting the direction of the navigation diagram to keep it consistent with the imaging orientation detected by the bronchoscope, doctors can more accurately judge the bifurcations and paths within the bronchus and avoid getting lost in complex airways. Doctors can obtain more intuitive and accurate navigation support in actual operations, thereby improving the safety and effectiveness of bronchoscopic surgery.

[0090] See also Figure 6 , shows a system structure diagram of the path generation system 6. The system includes:

[0091] A reconstruction module 61 is used to reconstruct a bronchial tree 3D mask based on the CT image;

[0092] A structuring module 62 is used to extract a centerline point set of the bronchial tree 3D mask, and perform structured processing on the centerline point set to obtain a centerline tree;

[0093] A selection module 63 is used to select a drawing starting point and a target area of ​​the navigation diagram of the bronchoscope on the centerline tree;

[0094] The data generation module 64 is used to traverse the centerline tree starting from the target node corresponding to the target area, gradually trace back the parent node of the target node in the centerline tree until reaching the drawing starting point to form a tracking path, and record the nodes or all nodes included in the tracking path to generate tracking data.

[0095] The drawing module 65 draws a navigation diagram from the drawing starting point to the target area according to the tracking data.

[0096] The above system can execute the path generation method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the path generation method.

[0097] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, a device or a device or used in combination with it.

[0098] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0099] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0100] Computer program code for performing the operation of embodiments of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0101] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A path generation method, characterized in that: The method includes: Reconstruct the bronchial tree 3D mask based on the CT image; Extracting a centerline point set of the bronchial tree 3D mask, and performing structured processing on the centerline point set to obtain a centerline tree; Selecting a drawing starting point and a target area for the navigation diagram of the bronchoscope on the centerline tree; Traversing the centerline tree from the target node corresponding to the target area, gradually tracing back to the parent node of the target node in the centerline tree until reaching the drawing starting point to form a tracking path, and recording the nodes included in the tracking path to generate tracking data; A navigation diagram is drawn from the drawing starting point to the target area according to the tracking data.

2. The path generation method according to claim 1, characterized in that: Traversing the centerline tree from the target node corresponding to the target area, and gradually tracing back to the parent node of the target node in the centerline tree until reaching the drawing starting point to form a tracking path, and recording the nodes included in the tracking path to generate tracking data includes: Selecting pixel points on the centerline tree as reference points, wherein the reference points at least include pixel points at bifurcations; forming a local neighborhood within a predetermined area around the reference point; Traversing the reference points on the tracing path, identifying one or more connected domains in the local neighborhood corresponding to the cross section of the bronchial tree 3D mask at the current layer, the connected domain being a set of interconnected pixel points in the predetermined area; The connectivity status of the connected domain is analyzed to generate the tracking data.

3. The path generation method according to claim 2, characterized in that: Forming a local neighborhood within a predetermined area around the reference point includes: Forming a local neighborhood with the reference point as the center and a predetermined distance as the radius; Wherein, the predetermined distance is determined according to the bronchial diameter corresponding to the reference point.

4. The path generation method according to claim 2, characterized in that: Identifying one or more connected domains in the local neighborhood corresponding to the cross section of the bronchial tree 3D mask at the current layer, including: extracting the cross section corresponding to the local neighborhood in the bronchial tree 3D mask; Selecting the unmarked pixel point in the local neighborhood corresponding to the cross section of the current layer as an initial point; Starting from the initial point, all adjacent pixels of the initial point are checked, and adjacent pixels that meet the connectivity condition are marked; A traversal check is performed along the adjacent pixel points starting from the initial point until all the pixel points connected to the initial point are marked, thereby forming one or more connected domains consisting of a plurality of the marked pixel points connected to each other.

5. The path generation method according to claim 2, characterized in that: Analyzing the connectivity status of the connected domain and generating the tracking data includes: Identify and track a connected domain according to the association between a connected domain of the current layer and the reference point; Traversing and checking each of the tracking connected domains, and generating tracking data according to the connectivity between the tracking connected domain of the current layer and the connected domain of the adjacent layer of the current layer; Wherein, identifying the tracking connectivity domain includes: In the case where a connected domain of the current layer includes the reference point of the corresponding local neighborhood, it is identified as a tracking connected domain.

6. The path generation method according to claim 1, characterized in that: Drawing a navigation diagram from the drawing starting point to the target area according to the tracking data includes: Drawing a plurality of guide curves based on the tracking data, wherein the guide curves are used to indicate a multi-layer branch structure to be passed through when going from the drawing starting point to the target area; Each of the guiding curves corresponds to a bronchial branch entrance in the multi-layer branch structure; wherein a bronchial branch entrance is a bronchial opening of the bronchial segment where the target area is located.

7. The path generation method according to claim 1, characterized in that: The method further comprises: The navigation schematic diagram is rotated or flipped according to the detection image of the bronchoscope, so that the navigation schematic diagram matches the detection orientation corresponding to the bronchoscope.

8. A path generation system, applied to the path generation method according to any one of claims 1 to 7, characterized in that: The system comprises: The reconstruction module is used to reconstruct the bronchial tree 3D mask based on the CT image; A structuring module, used to extract a centerline point set of the bronchial tree 3D mask, and perform structured processing on the centerline point set to obtain a centerline tree; A selection module, used for selecting a drawing starting point and a target area of ​​the navigation diagram of the bronchoscope on the centerline tree; A data generation module, configured to traverse the centerline tree starting from a target node corresponding to the target area, gradually trace back to the parent node of the target node in the centerline tree until reaching the drawing starting point to form a tracking path, and record the nodes in the tracking path to generate tracking data; A drawing module draws a navigation diagram from the drawing starting point to the target area according to the tracking data.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the path generation method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the path generation method according to any one of claims 1 to 7.

Images