A method and system for expanding intraoperative maps during electromagnetic navigation bronchoscopy combined with medical imaging information

Abstract

The present invention discloses a method and system for map expansion during electromagnetic navigation bronchoscopy combined with medical imaging information, the method comprising: 1) sequentially traversing and taking candidate registration points as seed points, performing threshold segmentation of connected areas on mask CT data, and forming a binary representation of static segmentation result area; 2) connecting the segmentation result area to establish a binary representation of a cylindrical area; 3) if information is missing, using the points in the list of preliminary registration points to establish a binary representation of a cylindrical area; 4) when map expansion is automatically triggered by the system, the real-time registration points should also be traversed as seed points to perform threshold segmentation of connected areas on mask CT data, and form a binary representation of a real-time segmentation result area; 5) performing a combined operation on the binary representation of the cylindrical area, the binary representation of the static segmentation result area, and the binary representation of the real-time segmentation result area, and processing to obtain new airway information. The present invention uses the low-density value area of ​​CT data as a reference for the area where the new airway is located, which is more conducive to reflecting the actual airway morphology.

Description

Technical Field

[0001] The present invention belongs to the technical field of surgical navigation, and relates to a method and system for expanding a map during an electromagnetic navigation bronchoscopy operation combined with medical imaging information. Background Art

[0002] Bronchoscope is one of the important diagnostic and treatment instruments for respiratory diseases, used to observe the bronchial cavity and perform lung tissue biopsy, etc. Bronchoscope navigation technology can indicate the position of the instrument tip in the patient's bronchus during bronchoscopic surgery and guide it to the target to be observed and the biopsy target.

[0003] The current bronchoscopic navigation technology can be divided into electromagnetic navigation bronchoscope, image navigation bronchoscope, and mixed modality navigation bronchoscope according to the information modality used. Electromagnetic navigation bronchoscope is a relatively reliable and widely used technology.

[0004] The electromagnetic navigation bronchoscope is equipped with a magnetic positioning sensor in the instrument, and its implementation process includes:

[0005] 1) Perform hand-eye calibration on the instrument to obtain the end coordinate system of the instrument and magnetic positioning sensor coordinate system The coordinate transformation matrix between (hereinafter referred to as "hand-eye calibration coordinate transformation matrix") The schematic diagram of the two coordinate systems is as follows Figure 1 shown.

[0006] and are the sampling information of the bronchoscope tip posture and magnetic positioning sensor posture in hand-eye calibration, and n is the number of collected postures. The solution is shown in formula (1).

[0007] (1)

[0008] The calculation method of is to first use the singular value decomposition method to make an initial estimate, and then use the iterative nearest point method to optimize the solution.

[0009] If the position of the end of the instrument is known , the position and posture of the magnetic positioning sensor can be calculated by formula (2)

[0010] (2)

[0011] 2) Process the preoperative CT data to obtain the three-dimensional morphological description information of the bronchial tree (hereinafter referred to as "map information"), which is in the form of the surface information of the bronchial tree and the binary representation of the bronchial tree area, or the surface information of the bronchial tree, the binary representation of the bronchial tree area and the centerline information of each airway in the bronchial tree. Based on the map information, the three-dimensional morphological map of the bronchial tree (hereinafter referred to as "map") can be displayed on a computer graphical interface.

[0012] 3) At the beginning of the operation, perform initial registration. Use the bronchoscope to carry the instrument and move it in the bronchus. Place the end of the instrument in multiple positions in sequence according to the multiple coordinates recorded by the magnetic positioning sensor. , and the coordinates of the map feature closest to where the instrument tip is placed , and the hand-eye calibration coordinate transformation matrix , calculate the initial registration coordinate transformation matrix The coordinate transformation relationship is shown in formula (3).

[0013] (3)

[0014] m is the number of coordinates collected for the initial registration. The calculation method of is calculated in the same way.

[0015] Initial registration coordinate transformation matrix Describe the magnetic positioning sensor pose and the magnetic positioning sensor pose in the map coordinate system The transformation between them is shown in formula (4): The end position of the instrument.

[0016] (4)

[0017] 4) During the operation, real-time registration is performed using formula (5-6).

[0018] Get the hand-eye calibration coordinate transformation matrix and the initial registration coordinate transformation matrix Then, according to formula (5), the real-time coordinates output by the magnetic positioning sensor during the operation can be (hereinafter referred to as "magnetic navigation trajectory points"), and points on the centerline of each airway in the bronchial tree Perform registration and obtain the registration coordinate transformation matrix k is the number of points involved in the registration. Use the ICP algorithm to solve formula (5) to As initial estimate of .

[0019] get Then, the coordinates of the map coordinate system corresponding to the current magnetic navigation trajectory point after registration are calculated by formula (6): (hereinafter referred to as "registration point") and displayed on the computer's graphical interface. The coordinates of this registration point in the map represent the position of the instrument tip in the patient's bronchial cavity.

[0020] (5)

[0021] (6)

[0022] There are many registration techniques used in electromagnetic navigation bronchoscopes, such as constructing a cost function to calculate the optimal registration coordinate transformation matrix, or integrating the coordinate and angle information output by the magnetic positioning sensor with the centerline information of each airway in the bronchial tree for ICP registration.

[0023] Bronchoscopy is performed by the bronchoscope body and instrument consumables. Instrument consumables include visual probes, biopsy forceps, etc. (hereinafter referred to as "instruments"). The instrument extends from the instrument channel of the mother scope and can carry magnetic positioning sensors into thinner bronchi. If the instrument can enter the airway that cannot be distinguished when the CT data is reconstructed in three dimensions, new map information (hereinafter referred to as "new airway information") can be recorded during the operation.

[0024] CN117257459A discloses a method for expanding the map of electromagnetic navigation bronchoscope during operation with resistance to respiratory interference, which can record new airway information and update map information during operation according to the magnetic navigation trajectory after the transformation of the registration coordinate transformation matrix. In the method, the states of the electromagnetic navigation bronchoscope during operation are divided into three types, namely, the normal state, the map preparation state and the map construction state, and whether the state is transferred is determined by the map preparation judgment condition, the map construction judgment condition and the map construction end judgment condition; when the navigation system determines that the map preparation judgment condition is met, the preparation registration point list is established and updated; when the navigation system determines that the map construction judgment condition is met (that is, it is determined that the instrument is in an airway without map information), the map construction state is entered; when recording new map information, due to respiratory and other interferences, the magnetic navigation trajectory often cannot reflect the actual airway morphology. Although the new airway can be recorded as a cylindrical or curved pipe shape in an approximate manner, it may still be different from the actual airway morphology. Based on this, the present invention proposes an intraoperative map expansion method for electromagnetic navigation bronchoscopy combined with medical imaging information. The method can combine preoperative computed tomography (CT) information during the intraoperative map expansion process, and use the position and morphology of the low-density area as a reference for recording the new airway information, so that the new airway morphology is closer to the actual airway. Summary of the invention

[0025] The purpose of the present invention is to address the deficiencies in the prior art and to provide an electromagnetic navigation bronchoscopy intraoperative map expansion method and system that combines medical imaging information to solve the problem that the airway morphology in the new airway information recorded by the existing intraoperative map expansion method is significantly different from the actual airway morphology.

[0026] The technical solution adopted by the present invention is as follows:

[0027] In a first aspect, an embodiment of the present invention provides a method for expanding a map during electromagnetic navigation bronchoscopy combined with medical imaging information, the method comprising:

[0028] After triggering the map expansion, at the beginning of the map expansion process, the preparation work includes: creating a copy of the CT data and setting some pixels to high density values ​​to form mask CT data; filtering and sorting the list of preliminary registration points established in the map preparation state to obtain a list of candidate registration points;

[0029] Carry out map expansion, including the following:

[0030] 1) Connected region threshold segmentation: If the number of registration points in the candidate registration point list is not 0, the candidate registration points are traversed in the order of the candidate registration point list and taken as seed points. The connected region threshold segmentation is performed on the mask CT data. The static segmentation result region binary representation is formed by combining the threshold judgment and added to the static segmentation result region binary representation list;

[0031] 2) Segmentation result area connection: After the connected area threshold segmentation is completed, if the number of items in the binary representation list of the static segmentation result area is not 0, the nearest point of the retained airway is taken as the current starting point, and the items in the binary representation list of the static segmentation result area are traversed in order from front to back. For each item, its corresponding proximal pixel point and distal pixel point are obtained, and the current starting point and the proximal pixel point are used as the endpoints of the line segment to establish a cylindrical area binary representation;

[0032] 3) Recording of airway segment information when information is missing: If the number of items in the candidate registration point list is 0, or after the connected region threshold segmentation is performed, the number of items in the static segmentation result region binary representation list is 0, then the nearest neighbor pixel point of the nearest point of the retained airway is taken as the starting point, and the point in the preliminary registration point list farthest from the starting point is found, and its nearest neighbor pixel point is taken as the ending point, and the starting point and the ending point are taken as the end points of the line segment to establish a cylindrical region binary representation;

[0033] 4) Processing real-time registration points: When the execution of map expansion is automatically triggered by the system, the real-time registration points after the map expansion is triggered are added to the real-time registration point list. After steps 1)-3) are completed, the real-time registration points in the real-time registration point list are screened out, and the remaining real-time registration points are used as seed points in turn to perform connected area threshold segmentation. Combined with the threshold judgment, a binary representation of the real-time segmentation result area is formed, and added to the real-time segmentation result area binary representation list;

[0034] In a unit time period, if the number of items in the binary representation list of the real-time segmentation result region is not 0, then the far pixel point of the processed last static segmentation result region binary representation or the real-time segmentation result region binary representation is taken as the current starting point, and the items in the real-time segmentation result region binary representation list are traversed in order from front to back, and for each item, its corresponding near pixel point and far pixel point are obtained, and the current starting point and the near pixel point are used as the end points of the line segment to establish a cylindrical region binary representation; if the number of items in the binary representation list of the real-time segmentation result region is 0, then the far pixel point of the processed last static segmentation result region binary representation is taken as the starting point, and the point farthest from the starting point in the real-time registration point list is found, and its nearest neighbor pixel point is taken as the end point, and the starting point and the end point are used as the end points of the line segment to establish a cylindrical region binary representation;

[0035] 5) Generate new airway information: Combine the binary representations of all cylindrical areas, the binary representations of static segmentation result areas, and the binary representations of real-time segmentation result areas established in the above process to obtain the binary representation of the new airway area. Use the marching cube algorithm to extract the isosurface to obtain the new airway surface information. The binary representation of the new airway area and the new airway surface information are used together as the new airway information obtained by map expansion.

[0036] In a second aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for expanding an intraoperative map of electromagnetic navigation bronchoscopy combined with medical imaging information as described in any embodiment.

[0037] In a third aspect, an embodiment of the present invention provides an electromagnetic navigation bronchoscope system, the system comprising:

[0038] one or more processors;

[0039] A memory for storing one or more programs;

[0040] When the one or more programs are executed by the one or more processors, the one or more processors implement the electromagnetic navigation bronchoscopy intraoperative map expansion method combined with medical imaging information as described in any embodiment.

[0041] The technical solution provided in the embodiment of the present invention simultaneously utilizes CT information and magnetic positioning information when expanding the map during electromagnetic navigation bronchoscopy, and uses the low-density value area in the CT data as a reference for the area where the new airway is located. In the presence of interference such as respiratory movement, the obtained results can better reflect the actual airway morphology compared to the method of only using magnetic positioning information. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic diagram of the coordinate system of the instrument end and the coordinate system of the magnetic positioning sensor;

[0043] Figure 2 A flow chart of a method for expanding an intraoperative map triggered by a user provided in Embodiment 1 of the present invention;

[0044] Figure 3 A flow chart of an intraoperative map expansion method automatically triggered by a system provided in Embodiment 2 of the present invention;

[0045] Figure 4-7 A partial schematic diagram of a method for expanding an intraoperative map triggered by a user provided in Embodiment 1 of the present invention.

[0046] Figure 8-10 This is a partial schematic diagram of an intraoperative map expansion method automatically triggered by the system provided in Example 2 of the present invention.

[0047] Fig.11 This is the effect diagram of intraoperative map expansion using the patented CN117257459A method.

[0048] Fig.12 This is the effect diagram of intraoperative map expansion using this method.

[0049] In the figure:

[0050] 1 indicates an airway that actually exists but is not included in the map information.

[0051] 2 indicates a point in the list of preliminary registration points.

[0052] 3 represents the binary representation of the static segmentation result area.

[0053] 4 indicates binary representation of cylindrical area.

[0054] 5 indicates binary representation of the new airway area.

[0055] 6 represents the points in the real-time registration point list.

[0056] 7 represents the binary representation of the real-time segmentation result area. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] The method for intraoperative map expansion of electromagnetic navigation bronchoscope combined with medical imaging information of the present invention is a further improvement on the method of CN117257459A. The method of the present invention adopts the same method as CN117257459A to convert the normal state / map preparation state / map construction state, but in the map preparation state and the map construction state, the specific execution method of the system is different. For the electromagnetic navigation bronchoscope, the initial registration is performed at the beginning of the operation to obtain the registration coordinate transformation matrix;

[0059] Under normal conditions, real-time registration is performed and the registration coordinate transformation matrix is ​​updated in real time; the coordinates of the current registration point in the map are determined by the real-time registration coordinate transformation matrix and the magnetic navigation trajectory points; the map and the registration point can be displayed in real time on a computer graphical interface.

[0060] When the mapping preparatory judgment conditions are met, it is considered that the end of the bronchoscope has entered an airway where map information is missing. At this time, in the present invention, the point on the bronchial tree surface or the center line of each airway in the bronchial tree that is closest to the registration point is recorded as the closest point to the retained airway, and the real-time registration point begins to be stored in the preparatory registration point list. At this time, map expansion can be executed, that is, the transition to the mapping state can be triggered; the triggering execution of map expansion can be triggered by the user (the process is as follows Figure 2 ), or it can be triggered automatically by the system (process such as Figure 3 ), where the conditions for the system to automatically trigger map expansion are described in the "Mapping Determination Conditions" in patent CN117257459A. After the map expansion is triggered, the system stops updating the list of preliminary registration points.

[0061] When the execution of map extension is triggered by the user, the map extension execution process is terminated after processing all data in the list of preliminary registration points.

[0062] When the execution of map expansion is automatically triggered by the system, the collection and processing of real-time registration points are still performed during the execution of map expansion. The execution of map expansion is terminated when the navigation system determines that the device is in the airway with map information.

[0063] When the navigation system determines that the device is in an airway with map information (i.e., in a normal state), map expansion cannot be performed, the collection of real-time registration points is stopped, and the record information of the reserved airway closest point and the list of preliminary registration points is deleted.

[0064] The present invention is described with reference to the following examples:

[0065] Example 1

[0066] During surgery, when the navigation system determines that the instrument is in an airway without map information, the point on the bronchial tree surface closest to the registration point is recorded as the nearest point of the reserved airway (according to different implementations, if the map information includes the centerline information of each airway in the bronchial tree, the point on the centerline of each airway in the bronchial tree closest to the registration point is recorded as the nearest point of the reserved airway), and the real-time registration point is stored in the list of prepared registration points. At this time, map expansion can be executed. In this example, the execution of map expansion is triggered by the user.

[0067] After the map expansion is triggered, the system stops updating the list of preliminary registration points. The map expansion execution process ends after processing all the data in the list of preliminary registration points.

[0068] At the beginning of the map expansion process, the following preparations are made:

[0069] 1. Create a copy of the CT data, in which the values ​​of the pixels labeled 1 in the binary representation of the bronchial tree region and the binary representation of the existing new airway region (if any) are set to the high density value H as the mask CT data. H can take any value between -999 and +∞ that is greater than the air CT density value, and here H=0.

[0070] 2. Process the list of preliminary registration points. If the nearest neighbor pixel of a preliminary registration point is labeled as 1 in the binary representation of the bronchial tree region or the binary representation of the existing new airway region (if any), delete this preliminary registration point. If the CT density value corresponding to the coordinates of a preliminary registration point is greater than the upper threshold T_seg_upper of the subsequent connected area threshold segmentation, delete this preliminary registration point. The remaining registration points are regarded as candidate registration points. They are arranged in ascending order of distance to the nearest point of the retained airway and stored in the candidate registration point list.

[0071] The following map expansion process is as follows:

[0072] 1. Perform connected area threshold segmentation.

[0073] If the number of registration points in the candidate registration point list is not 0, the connected region threshold segmentation is started. According to the order in the candidate registration point list, the candidate registration points are taken as seed points in turn, and the connected region threshold segmentation is performed on the mask CT data with the preset upper and lower thresholds of the connected region threshold segmentation, T_seg_upper and T_seg_lower, to obtain a binary representation, which is called "static segmentation result region binary representation".

[0074] T_seg_lower can take any value between -∞ and -1001 that is less than the air CT density value. In this embodiment, T_seg_lower=2000; T_seg_upper can take any value between -999 and 0 that is between the air CT density value and the water CT density value. In this embodiment, T_seg_upper=-800.

[0075] Count the number of pixels labeled 1 in the binary representation of the static segmentation result area. If the number of pixels is less than the lower limit of the pixel number threshold T_pixels_lower of the segmentation result area, discard the segmentation result; if the number of pixels is between the lower limit T_pixels_lower and the upper limit T_pixels_upper of the pixel number threshold of the segmentation result area, retain it and add it to the binary representation list of the static segmentation result area; if the number of pixels is greater than the upper limit T_pixels_upper of the pixel number threshold of the segmentation result area, reduce the upper limit T_seg_upper of the threshold segmentation of the connected area, as shown in formula (7).

[0076] (7)

[0077] Wherein n is an integer greater than 0.

[0078] Then the connected area threshold segmentation is performed again until the number of pixels labeled 1 in the binary representation of the static segmentation result area is less than or equal to the upper limit of the pixel number threshold T_pixels_upper of the segmentation result area. At this time, the binary representation of the static segmentation result area is retained and added to the binary representation list of the static segmentation result area.

[0079] T_pixels_lower can take any value between 1 and 100 that can be considered as an invalid segmentation area pixel number. In this embodiment, T_pixels_lower=10; T_pixels_upper can take any value between 101 and +∞ that can be considered as an excessively large segmentation area pixel number. In this embodiment, T_pixels_upper=5000.

[0080] The value n in formula (7) can be between 1 and 100, and can also be changed according to actual conditions during the calculation process. In this embodiment, n=1.

[0081] When the next candidate registration point is taken as the seed point for threshold segmentation, the upper threshold limit T_seg_upper of the connected region threshold segmentation is restored to the initial value. For any item in the binary representation list of the static segmentation result area, if the nearest neighbor pixel of a candidate registration point is labeled 1 in this binary representation, this candidate registration point is no longer used as a seed point. Other candidate registration points in the candidate registration point list are sequentially selected as seed points for connected region threshold segmentation until the candidate registration point list is traversed.

[0082] 2. Connect the binary representation of the static segmentation result area.

[0083] After the connected area threshold segmentation is completed, check the binary representation list of the static segmentation result area. If the number of items in the list is not 0, take the nearest point of the retained airway as the "current starting point" and traverse the items in the binary representation list of the static segmentation result area in order from front to back. For each item, obtain the pixel points closest to and farthest from the current starting point with a label of 1, which are called the proximal pixel point and the distal pixel point, respectively. For the line segment between the current starting point and the near-end pixel point of the current static segmentation result area binary representation, a series of pixel points closest to the line segment are calculated using the three-dimensional Bresenham algorithm, which is called the "connected pixel point set" (the Bresenham algorithm can be found in Bresenham J E. Algorithm for computer control of a digital plotter [M]. Seminal graphics: pioneering efforts that shaped the field. 1998: 1-6.), and the pixel points whose distance from any pixel point in the connected pixel point set is less than or equal to k pixels are recorded, which is called the "filled pixel point set". The two point sets form a cylindrical area binary representation, and the labels of the pixels in the two point sets are set to 1, and the labels of other pixels are set to 0, and then the current starting point is reset to the far-end pixel point of the current static segmentation result area binary representation. Each item in the static segmentation result area binary representation list is processed in the above manner until the near-end pixel point of the current starting point and the last item in the static segmentation result area binary representation list is used as the end point of the line segment to establish a cylindrical area binary representation.

[0084] Here, k can take a value between 0 and 100, and in this embodiment, k=2.

[0085] 3. Recording of airway segment information when information is missing

[0086] If the number of items in the candidate registration point list is 0, or after the connected region threshold segmentation, the number of items in the binary representation list of the static segmentation result region is 0, then the nearest neighbor pixel point of the nearest point of the retained airway is used as the starting point, and the point farthest from the starting point in the list of preliminary registration points is found (that is, there is no available low-density CT information at this time, no candidate registration point is obtained, and only the preliminary registration point can be used), and its nearest neighbor pixel point is used as the end point. For the line segment between the starting point and the end point, the three-dimensional Bresenham algorithm is used to calculate a series of pixels closest to this line segment, which is called the "connected pixel point set", and the pixel points whose distance from any pixel point in the connected pixel point set is less than or equal to k pixels are recorded, which is called the "filled pixel point set". The two point sets form a cylindrical region binary representation, and the labels of the pixels in the two point sets are set to 1, and the labels of other pixels are set to 0.

[0087] 4. Generate New Airway Information

[0088] All the binary representations of cylindrical regions and the binary representations of the static segmentation result regions (if any) established in the above process are combined to obtain the binary representation of the new airway region. This binary representation is used to extract the isosurface using the marching cube algorithm to obtain the "new airway surface information", which is used for display in the computer graphics interface. All the binary representations of cylindrical regions and the binary representations of the static segmentation result regions are deleted. The binary representation of the new airway region and the new airway surface information are used together as the new airway information obtained by map expansion.

[0089] According to different implementations, if the map information includes the centerline information of each airway in the bronchial tree, it is necessary to extract the centerline from the surface information of the new airway. The specific method is to take the nearest point of the retained airway as the starting point of the centerline, take the point farthest from the starting point of the centerline in the binary representation of the new airway area as the ending point of the centerline, and use the maximum inscribed sphere method to extract the centerline between the starting point and the ending point based on the surface information of the new airway.

[0090] The above process realizes intraoperative map expansion combined with CT density information, such as Figure 4-7 As shown, it is a schematic diagram corresponding to this embodiment, Figure 4 For the airway 1 that actually exists but is not included in the map information, the above method is used to obtain point 2 in the list of preliminary registration points (such as Figure 5 ), after processing, we get the binary representation of the static segmentation result area 3 and the binary representation of the cylindrical area 4 (such as Figure 6 ) and generate a new airway region binary representation 5 (such as Figure 7 ).

[0091] Embodiment 2:

[0092] During surgery, when the navigation system determines that the instrument is in an airway without map information, the point on the bronchial tree surface closest to the registration point is recorded as the closest point to the reserved airway (according to different implementations, if the map information includes the centerline information of each airway in the bronchial tree, the point on the centerline of each airway in the bronchial tree closest to the registration point is recorded as the closest point to the reserved airway), and the real-time registration point is stored in the list of prepared registration points. At this time, map expansion can be executed. In this example, the execution of map expansion is automatically triggered by the system, and the triggering conditions are shown in the "Mapping Determination Conditions" in patent CN117257459A.

[0093] After the map expansion is triggered, the system stops updating the list of preliminary registration points. During the execution of the map expansion, the real-time registration points are still collected and processed. The map expansion execution process ends when the navigation system determines that the device is in the airway with map information.

[0094] At the beginning of the map expansion process, the following preparations are made:

[0095] 1. Create a copy of the CT data, in which the values ​​of the pixels labeled 1 in the binary representation of the bronchial tree region and the binary representation of the existing new airway region (if any) are set to the high density value H as the mask CT data. H can take any value between -999 and +∞ that is greater than the air CT density value, and here H=0.

[0096] 2. Process the list of preliminary registration points. If the nearest neighbor pixel of a preliminary registration point is labeled as 1 in the binary representation of the bronchial tree region or the binary representation of the existing new airway region (if any), delete this preliminary registration point. If the CT density value corresponding to the coordinates of a preliminary registration point is greater than the upper threshold T_seg_upper of the subsequent connected area threshold segmentation, delete this preliminary registration point. The remaining registration points are regarded as candidate registration points. They are arranged in ascending order of distance to the nearest point of the retained airway and stored in the candidate registration point list.

[0097] The following map expansion process is as follows:

[0098] 1. Perform connected area threshold segmentation.

[0099] If the number of registration points in the candidate registration point list is not 0, the connected region threshold segmentation is started. According to the order in the candidate registration point list, the candidate registration points are taken as seed points in turn, and the connected region threshold segmentation is performed on the mask CT data with the preset upper and lower thresholds of the connected region threshold segmentation, T_seg_upper and T_seg_lower, to obtain a binary representation, which is called "static segmentation result region binary representation".

[0100] T_seg_lower can take any value between -∞ and -1001 that is less than the air CT density value. In this embodiment, T_seg_lower=2000; T_seg_upper can take any value between -999 and 0 that is between the air CT density value and the water CT density value. In this embodiment, T_seg_upper=-800.

[0101] The number of pixels with the label 1 in the binary representation of the static segmentation result region is counted. If the number of pixels is less than the lower limit of the pixel number threshold T_pixels_lower of the segmentation result region, the segmentation result is discarded; if the number of pixels is between the lower limit T_pixels_lower and the upper limit T_pixels_upper of the pixel number threshold of the segmentation result region, it is retained and added to the list of binary representations of the static segmentation result region; if the number of pixels is greater than the upper limit T_pixels_upper of the pixel number threshold of the segmentation result region, the upper limit T_seg_upper of the threshold segmentation of the connected region is reduced, as shown in formula (7), and then the threshold segmentation of the connected region is performed again until the number of pixels with the label 1 in the binary representation of the static segmentation result region is less than or equal to the upper limit T_pixels_upper of the pixel number threshold of the segmentation result region. At this time, the binary representation of the static segmentation result region is retained and added to the list of binary representations of the static segmentation result region.

[0102] T_pixels_lower can take any value between 1 and 100 that can be considered as an invalid segmentation area pixel number. In this embodiment, T_pixels_lower=10; T_pixels_upper can take any value between 101 and +∞ that can be considered as an excessively large segmentation area pixel number. In this embodiment, T_pixels_upper=5000.

[0103] The value n in formula (7) can be between 1 and 100, and can also be changed according to actual conditions during the calculation process. In this embodiment, n=1.

[0104] When the next candidate registration point is taken as the seed point for threshold segmentation, the upper threshold limit T_seg_upper of the connected region threshold segmentation is restored to the initial value. For any item in the binary representation list of the static segmentation result area, if the nearest neighbor pixel of a candidate registration point is labeled 1 in this binary representation, this candidate registration point is no longer used as a seed point. Other candidate registration points in the candidate registration point list are sequentially selected as seed points for connected region threshold segmentation until the candidate registration point list is traversed.

[0105] 2. Connect and display the binary representation of the static segmentation result area.

[0106] After the connected region threshold segmentation is completed, the binary representation list of the static segmentation result region is checked. If the number of items in the list is not 0, the nearest point of the retained airway is taken as the "current starting point", and the items in the binary representation list of the static segmentation result region are traversed in order from front to back. For each item, the pixel points closest to and farthest from the current starting point with a label of 1 are obtained, which are called the proximal pixel point and the distal pixel point, respectively. For the line segment between the current starting point and the proximal pixel point of the current static segmentation result region binary representation, a series of pixel points closest to this line segment are calculated using the three-dimensional Bresenham algorithm, which are called the "connected pixel point set". The pixel points whose distance from any pixel point in the connected pixel point set is less than or equal to k pixels are recorded, which are called the "filled pixel point set". The two point sets form a cylindrical region binary representation. The labels of the pixels in the two point sets are set to 1, and the labels of other pixels are set to 0. Then the current starting point is reset to the distal pixel point of the current static segmentation result region binary representation. Each item in the binary representation list of the static segmentation result area is processed in the above manner until the current starting point and the proximal pixel point of the last item in the binary representation list of the static segmentation result area are used as the endpoints of the line segment to establish a cylindrical area binary representation.

[0107] Here, k can take a value between 0 and 100, and in this embodiment, k=2.

[0108] The binary representations of all cylindrical regions (if any) and the binary representations of the static segmentation result regions (if any) are used to extract isosurfaces using the marching cube algorithm, which is called "temporary surface information" and is used for display in a computer graphics interface.

[0109] 3. Recording and display of airway segment information when information is missing

[0110] If the number of items in the candidate registration point list is 0, or after the connected region threshold segmentation, the number of items in the binary representation list of the static segmentation result region is 0, the nearest neighbor pixel point of the nearest point of the retained airway is used as the starting point, and the point farthest from the starting point in the preliminary registration point list is found, and its nearest neighbor pixel point is used as the ending point. For the line segment between the starting point and the ending point, a series of pixel points closest to this line segment are calculated using the three-dimensional Bresenham algorithm, which is called the "connected pixel point set". The pixel points whose distance to any pixel point in the connected pixel point set is less than or equal to k pixels are recorded, which is called the "filled pixel point set". The cylindrical region binary representation is established, and the labels of the pixels in the two point sets are set to 1, and the labels of other pixels are set to 0.

[0111] The binary representation of the cylindrical region is used to extract isosurfaces using the marching cubes algorithm to obtain temporary surface information for display in a computer graphics interface.

[0112] 4. Process real-time registration points.

[0113] The system adds the real-time registration points after the map expansion is triggered to the real-time registration point list, such as Figure 8 Point 6 in the real-time registration point list. After the above steps 1, 2, and 3 are completed, the real-time registration points whose coordinates correspond to CT density values ​​greater than the upper threshold T_seg_upper of the connected region threshold segmentation, as well as the binary representation of the nearest neighbor pixel in the bronchial tree area, the binary representation of the existing new airway area (if any), the binary representation of all static segmentation result areas (if any) or the real-time registration points with a label of 1 in the binary representation of the cylindrical area are excluded from the real-time registration point list. The remaining real-time registration points are used as seed points in turn, and the connected region threshold segmentation is performed as described in step 1. The segmentation result is used as the real-time segmentation result area binary representation (such as Fig. 9 The real-time segmentation result area binary representation 7) is added to the real-time segmentation result area binary representation list.

[0114] In a unit time period, if the number of items in the binary representation list of the real-time segmentation result area is not 0, the distal pixel point of the last processed binary representation of the static segmentation result area or the real-time segmentation result area is taken as the "current starting point", and the static segmentation result area binary representation and the real-time segmentation result area binary representation are connected and displayed according to the method described in step 2; if the number of items in the binary representation list of the real-time segmentation result area is 0, the distal pixel point of the last processed binary representation of the static segmentation result area is taken as the starting point, and the point farthest from the starting point in the real-time registration point list is found, and its nearest neighbor pixel point is taken as the ending point, and the airway segment information is recorded and displayed according to the method described in step 3.

[0115] The processing of the real-time registration points ends when the navigation system determines that the instrument is in the airway with map information.

[0116] 5. Generate new airway information

[0117] All the binary representations of the cylindrical regions, the binary representations of the static segmentation result regions, and the binary representations of the real-time segmentation result regions established in the above process are combined to obtain the binary representation of the new airway region, such as Fig.10 . Use the marching cube algorithm to extract the isosurface from this binary representation to obtain the "new airway surface information". Delete all temporary surface information and use the new airway surface information for display in the computer graphics interface. Delete all binary representations of cylindrical areas, as well as binary representations of static segmentation result areas and binary representations of real-time segmentation result areas. The binary representation of the new airway area and the new airway surface information are used together as the new airway information obtained by map expansion.

[0118] According to different implementations, if the map information includes the centerline information of each airway in the bronchial tree, it is necessary to extract the centerline from the surface information of the new airway. The specific method is to take the nearest point of the retained airway as the starting point of the centerline, take the point farthest from the starting point of the centerline in the binary representation of the new airway area as the ending point of the centerline, and use the maximum inscribed sphere method to extract the centerline between the starting point and the ending point based on the surface information of the new airway.

[0119] The above process realizes intraoperative map expansion combined with CT density information.

[0120] The method of the present invention can be well combined with CT density information to expand the intraoperative map, so that the new airway information finally constructed can be closer to the actual airway situation, such as Fig.11 and Fig.12 As shown, the new airways obtained by performing intraoperative map expansion using the CN117257459A method and the method of the present invention in the same example scenario respectively. It can be seen that the method of the present invention uses the low-density value area in the CT data as a reference for the area where the new airway is located. In the presence of interference such as respiratory movement, the obtained results can better reflect the actual airway morphology compared to the method that only uses magnetic positioning information.

[0121] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which implements any of the above-mentioned intraoperative map expansion methods when executed by a processor.

[0122] The present application also provides an electromagnetic navigation bronchoscope system, the system comprising:

[0123] one or more processors;

[0124] A memory for storing one or more programs;

[0125] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the above-mentioned intraoperative map expansion methods.

[0126] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0127] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0128] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0129] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0130] The present invention is not limited to the above embodiments, and various changes can be made within the knowledge of those skilled in the art without departing from the purpose of the present invention, for example, the present invention can be extended to a multi-machine collaborative exploration system.

Claims

1. A method for expanding the map during electromagnetic navigation bronchoscopy combined with medical imaging information, characterized in that: The method includes: After triggering the map expansion, at the beginning of the map expansion process, the preparation work includes: creating a copy of the CT data and setting some pixels to high density values ​​to form mask CT data; filtering and sorting the pre-established preliminary registration point list to obtain a candidate registration point list; Carry out map expansion, including the following: 1) Connected region threshold segmentation: If the number of registration points in the candidate registration point list is not 0, the candidate registration points are traversed in the order of the candidate registration point list and taken as seed points. The connected region threshold segmentation is performed on the mask CT data. The static segmentation result region binary representation is formed by combining the threshold judgment and added to the static segmentation result region binary representation list; 2) Segmentation result area connection: After the connected area threshold segmentation is completed, if the number of items in the binary representation list of the static segmentation result area is not 0, the nearest point of the retained airway is taken as the current starting point, and the items in the binary representation list of the static segmentation result area are traversed in order from front to back. For each item, its corresponding proximal pixel point and distal pixel point are obtained, and the current starting point and the proximal pixel point are used as the end points of the line segment to establish a cylindrical area binary representation, and then the current starting point is reset to the distal pixel point of the current static segmentation result area binary representation, thereby processing each item in the binary representation list of the static segmentation result area, until the current starting point and the proximal pixel point of the last item in the binary representation list of the static segmentation result area are used as the end points of the line segment to establish a cylindrical area binary representation; 3) Recording of airway segment information when information is missing: If the number of items in the candidate registration point list is 0, or after the connected region threshold segmentation is performed, the number of items in the static segmentation result region binary representation list is 0, then the nearest neighbor pixel point of the nearest point of the retained airway is taken as the starting point, and the point in the preliminary registration point list farthest from the starting point is found, and its nearest neighbor pixel point is taken as the ending point, and the starting point and the ending point are taken as the end points of the line segment to establish a cylindrical region binary representation; 4) Processing real-time registration points: When the execution of map expansion is automatically triggered by the system, the real-time registration points after the map expansion is triggered are added to the real-time registration point list. After steps 1)-3) are completed, the real-time registration points in the real-time registration point list are screened out, and the remaining real-time registration points are used as seed points in turn to perform connected area threshold segmentation. Combined with the threshold judgment, a binary representation of the real-time segmentation result area is formed, and added to the real-time segmentation result area binary representation list; In a unit time period, if the number of items in the binary representation list of the real-time segmentation result region is not 0, then the far pixel point of the processed last static segmentation result region binary representation or the real-time segmentation result region binary representation is taken as the current starting point, and the items in the real-time segmentation result region binary representation list are traversed in order from front to back, and for each item, its corresponding near pixel point and far pixel point are obtained, and the current starting point and the near pixel point are used as the end points of the line segment to establish a cylindrical region binary representation; if the number of items in the binary representation list of the real-time segmentation result region is 0, then the far pixel point of the processed last static segmentation result region binary representation is taken as the starting point, and the point farthest from the starting point in the real-time registration point list is found, and its nearest neighbor pixel point is taken as the end point, and the starting point and the end point are used as the end points of the line segment to establish a cylindrical region binary representation; 5) Generate new airway information: Combine the binary representations of all cylindrical areas, the binary representations of static segmentation result areas, and the binary representations of real-time segmentation result areas established in the above process to obtain the binary representation of the new airway area. Use the marching cube algorithm to extract the isosurface to obtain the new airway surface information. The binary representation of the new airway area and the new airway surface information are used together as the new airway information obtained by map expansion.

2. The method for expanding the map during electromagnetic navigation bronchoscopy combined with medical imaging information according to claim 1, characterized in that: The method for forming the mask CT data is as follows: a copy of the CT data is created, and the values ​​of the pixels labeled as 1 in the binary representation of the bronchial tree region and the binary representation of the existing new airway region are set to the high density value H.

3. The method for expanding the map during electromagnetic navigation bronchoscopy combined with medical imaging information according to claim 1, characterized in that: The screening and sorting of the list of preliminary registration points is specifically as follows: if the label of the nearest neighbor pixel of a certain preliminary registration point in the binary representation of the bronchial tree region or the binary representation of the existing new airway region is 1, then the preliminary registration point is deleted; if the CT density value corresponding to the coordinates of a certain preliminary registration point is greater than the upper threshold limit T_seg_upper of the subsequent connected area threshold segmentation, then the preliminary registration point is also deleted; the remaining registration points are called "candidate registration points", which are arranged in ascending order of distance from the nearest point of the retained airway, and stored in the candidate registration point list.

4. The method for expanding the map during electromagnetic navigation bronchoscopy combined with medical imaging information according to claim 1, characterized in that: The connected region threshold segmentation specifically includes: taking candidate / real-time registration points as seed points in sequence according to the order in the candidate / real-time registration point list, performing connected region threshold segmentation on the mask CT data with the preset connected region threshold segmentation upper limit T_seg_upper and lower limit T_seg_lower, and obtaining a binary representation of the static / real-time segmentation result area; After each connected region threshold segmentation calculation, the number of pixels with a label of 1 in the binary representation of the static / real-time segmentation result region is counted. If the number of pixels is less than the lower limit T_pixels_lower of the pixel number threshold of the segmentation result region, the segmentation result is discarded; if the number of pixels is between the lower limit T_pixels_lower and the upper limit T_pixels_upper of the pixel number threshold of the segmentation result region, it is retained and added to the list of binary representations of the static / real-time segmentation result region; if the number of pixels is greater than the upper limit T_pixels_upper of the pixel number threshold of the segmentation result region, the upper limit T_seg_upper of the threshold segmentation of the connected region is reduced, and then the threshold segmentation of the connected region is performed again until the number of pixels with a label of 1 in the binary representation of the static / real-time segmentation result region is less than or equal to the upper limit T_pixels_upper of the pixel number threshold of the segmentation result region. At this time, the binary representation of the static / real-time segmentation result region is retained and added to the list of binary representations of the static / real-time segmentation result region.

5. The method for expanding the map during electromagnetic navigation bronchoscopy combined with medical imaging information according to claim 4, characterized in that: When a new candidate / real-time registration point is taken as a seed point for threshold segmentation in the candidate / real-time registration point list, the upper threshold limit T_seg_upper of the connected area threshold segmentation is restored to the initial value; for any item in the binary representation list of the static / real-time segmentation result area, if the nearest neighbor pixel point of a candidate / real-time registration point is labeled 1 in this binary representation, then this candidate / real-time registration point is no longer used as a seed point; other candidate / real-time registration points in the candidate / real-time registration point list are sequentially selected as seed points for connected area threshold segmentation until the candidate / real-time registration point list is traversed.

6. The method for expanding the map during electromagnetic navigation bronchoscopy combined with medical imaging information according to claim 1, characterized in that: The establishment of the binary representation of the cylindrical area is specifically as follows: a series of pixel points closest to the line segment are calculated using a three-dimensional Bresenham algorithm to form a line pixel point set, and pixel points whose distance from any pixel point in the line pixel point set is less than or equal to k pixels are recorded, which is called a filled pixel point set. Thus, a binary representation of the cylindrical area is formed based on the line pixel point set and the filled pixel point set, and the labels of the pixels in these two point sets are set to 1, and the labels of other pixels are set to 0.

7. The method for expanding the map during electromagnetic navigation bronchoscopy combined with medical imaging information according to claim 1, characterized in that: If the execution of map expansion is automatically triggered by the system, all cylindrical area binary representations and segmentation result area binary representations are extracted using the Marching Cubes algorithm to extract isosurfaces as temporary surface information for display in a computer graphics interface.

8. The method for expanding the map during electromagnetic navigation bronchoscopy combined with medical imaging information according to claim 1, characterized in that: After obtaining the new airway surface information, all temporary surface information is deleted, and the new airway surface information is used for display in a computer graphics interface, and all binary representations of cylindrical regions and binary representations of segmentation result regions are deleted at the same time; If the map information includes the centerline information of each airway in the bronchial tree, the new airway information also includes the centerline extracted from the surface information of the new airway, and the method for extracting the centerline from the surface information of the new airway is to take the nearest point of the retained airway as the starting point of the centerline, and take the point farthest from the starting point of the centerline in the binary representation of the new airway area as the ending point of the centerline; according to the surface information of the new airway, use the maximum inscribed sphere method to extract the centerline between the starting point and the ending point.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for expanding the intraoperative map of electromagnetic navigation bronchoscopy combined with medical imaging information as described in any one of claims 1 to 8 is implemented.

10. An electromagnetic navigation bronchoscope system, characterized in that: The system comprises: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the electromagnetic navigation bronchoscopy intraoperative map expansion method combined with medical imaging information as described in any one of claims 1-8.

Images