Bronchoscope dynamic positioning method and device

Abstract

The disclosure provides a bronchoscope dynamic positioning method and device, which can be applied to the technical field of surgical navigation and electromagnetic navigation. The method comprises: processing image data corresponding to a real bronchial model to generate virtual bronchial model data; performing registration operation on the real bronchial model data and the virtual bronchial model data based on a preset registration rule to generate a first conversion matrix; mapping real bronchoscope pose data to virtual bronchoscope pose data based on the first conversion matrix and a preset mapping rule; in the case that the pose data of the virtual bronchoscope exceeds a preset threshold, updating the pose data of the virtual bronchoscope by using a preset updating rule to determine target pose data corresponding to the virtual bronchoscope; and dynamically positioning the bronchoscope based on the target pose data, thereby improving the accuracy of bronchoscope positioning in real and virtual space.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of surgical navigation and electromagnetic navigation, and in particular to a bronchoscope dynamic positioning method and device. BACKGROUND

[0002] In a bronchoscopy related surgery, a doctor is prone to lose direction in a complex and numerous bronchial passage. Even if a surgical robot can move autonomously under image guidance, the doctor is prone to make airway selection errors and selection takes a long time, thereby reducing the efficiency of the surgery. In related technologies, the positioning method of a bronchoscope includes image registration and additional sensors. The image registration is to align and register bronchial images at different time points or different imaging modalities to achieve bronchial positioning and navigation.

[0003] In the process of conceiving the present disclosure, the inventors found that the conventional bronchoscope positioning method excessively relies on the structural information similarity (such as texture features, bronchial lumen features, etc.) between real bronchoscope images and virtual bronchoscope images, which leads to bronchoscope tracking failure in some cases (such as in the case of large changes in illumination, sudden motion changes, or image blur). In the actual surgical process, the motion of the bronchoscope tip is difficult to predict, which to some extent causes deviation in navigation and reduces the efficiency and accuracy of the surgery. SUMMARY

[0004] Therefore, the present disclosure provides a bronchoscope dynamic positioning method, device, equipment, storage medium and program product.

[0005] According to a first aspect of the present disclosure, a bronchoscope dynamic positioning method is provided, comprising: processing image data corresponding to a real bronchial model to generate virtual bronchial model data; performing registration operation on the real bronchial model data and the virtual bronchial model data based on a preset registration rule to generate a first conversion matrix; mapping real bronchoscope pose data to virtual bronchoscope pose data based on the first conversion matrix and a preset mapping rule, wherein the pose data includes position data and direction data corresponding to the bronchoscope; in the case that the pose data of the virtual bronchoscope exceeds a preset threshold, updating the pose data of the virtual bronchoscope using a preset update rule to determine target pose data corresponding to the virtual bronchoscope, wherein the preset update rule is associated with the position data and direction data of the virtual bronchoscope at a target time point before the preset threshold is exceeded; and dynamically positioning the bronchoscope based on the target pose data.

[0006] A second aspect of the present disclosure provides a bronchoscope dynamic positioning device, characterized in that the device comprises:

[0007] The model data generation module is configured to process image data corresponding to the real bronchial model to generate virtual bronchial model data.

[0008] The first conversion matrix generation module is configured to perform registration operation on the real bronchial model data and the virtual bronchial model data based on a preset registration rule to generate a first conversion matrix.

[0009] The mapping module is configured to map the real bronchoscope pose data to virtual bronchoscope pose data based on the first conversion matrix and a preset mapping rule, wherein the pose data includes position data and direction data corresponding to the bronchoscope.

[0010] The target pose data determination module is configured to update the pose data of the virtual bronchoscope using a preset update rule when the pose data of the virtual bronchoscope exceeds a preset threshold to determine target pose data corresponding to the virtual bronchoscope, wherein the preset update rule is associated with position data and direction data of the virtual bronchoscope at a target time before the preset threshold is exceeded.

[0011] The positioning module is configured to dynamically position the bronchoscope based on the target pose data.

[0012] The third aspect of the present disclosure provides an electronic device, comprising: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the method.

[0013] The fourth aspect of the present disclosure further provides a computer-readable storage medium having stored executable instructions, which are executed by a processor to cause the processor to perform the method.

[0014] The fifth aspect of the present disclosure further provides a computer program product comprising a computer program, which is executed by a processor to implement the method.

[0015] The bronchoscope dynamic positioning method, device, equipment, storage medium and program product provided according to the present disclosure, through the preset registration rule, the real bronchoscope model data and the virtual bronchoscope model data are registered, and the first conversion matrix is generated, so that the real bronchoscope pose data is mapped to the virtual bronchoscope pose data based on the first conversion matrix and the preset mapping rule, and then in the case that the pose data of the virtual bronchoscope exceeds the preset threshold, the position data and the direction data of the target time before the preset threshold is not exceeded are used to update and optimize the pose data of the virtual bronchoscope, and the target pose data corresponding to the virtual bronchoscope is obtained, and then the dynamic positioning of the bronchoscope is realized, which avoids the situation that the tracking fails or deviates due to excessive dependence on the similarity between the real and virtual bronchoscope image structure information, improves the accuracy of the bronchoscope positioning, and further improves the efficiency of the bronchoscope examination. BRIEF DESCRIPTION OF DRAWINGS

[0016] The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments of the present disclosure, taken in conjunction with the accompanying drawings, in which:

[0017] Figure 1 An application scenario diagram of the bronchoscope dynamic positioning method and device according to an embodiment of the present disclosure is schematically shown;

[0018] Figure 2 A flowchart of the bronchoscope dynamic positioning method according to an embodiment of the present disclosure is schematically shown;

[0019] Figure 3 A position correction schematic diagram based on a virtual bronchial model space boundary normal vector according to an embodiment of the present disclosure is schematically shown;

[0020] Figure 4 A bronchoscope motion mapping schematic diagram according to an embodiment of the present disclosure is schematically shown;

[0021] Figure 5 A bronchial model and central path schematic diagram according to an embodiment of the present disclosure is schematically shown;

[0022] Figure 6 A marker positioning schematic diagram on a real bronchial model according to an embodiment of the present disclosure is schematically shown;

[0023] Figure 7 A bronchoscope coordinate system and sensor coordinate system schematic diagram in a real scene according to an embodiment of the present disclosure is schematically shown;

[0024] Figure 8 A sensor and camera coordinate system relationship schematic diagram according to an embodiment of the present disclosure is schematically shown;

[0025] Figure 9A schematic diagram of a real bronchoscope and a virtual bronchoscope mapping relationship is shown according to an embodiment of the present disclosure.

[0026] Figure 10A A schematic diagram of a system architecture of a bronchoscope dynamic positioning method is shown according to an embodiment of the present disclosure.

[0027] Figure 10B A schematic diagram of a virtual bronchoscope dynamic positioning effect is shown according to an embodiment of the present disclosure.

[0028] Figure 11 A structural block diagram of a bronchoscope dynamic positioning device is shown according to an embodiment of the present disclosure.

[0029] Figure 12 A block diagram of an electronic device suitable for implementing a bronchoscope dynamic positioning method is shown according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood, however, that the description which follows is merely exemplary and is not intended to limit the scope of the present disclosure. In the following detailed description of embodiments of the present disclosure, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it would be apparent to one skilled in the art that the present disclosure can be practiced without these specific details. In other instances, well-known structures and functions have not been described in detail in order to avoid obscuring the concepts of the present disclosure.

[0031] The terms used herein are merely used to describe specific embodiments and are not intended to limit the present disclosure. The terms "include", "comprise" and the like used herein indicate the presence of the described features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0032] All terms used herein, including technical and scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of the present specification, and should not be interpreted in an idealized or overly formal manner.

[0033] In the case of using expressions similar to "at least one of A, B, and C, etc.", it should be generally interpreted as including one or more of the items enumerated in the list (e.g., "a system having at least one of A, B, and C" should include, but not be limited to, a system having A alone, a system having B alone, a system having C alone, a system having both A and B, a system having both A and C, a system having both B and C, and / or a system having A, B, and C, etc.).

[0034] In the technical solutions of the present disclosure, the user information (including but not limited to user personal information, user image information, user equipment information such as location information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of related data all comply with relevant laws, regulations and standards, take necessary security measures, do not violate public order and good customs, and provide corresponding operation portals for user selection authorization or refusal.

[0035] In the process of conceiving the present disclosure, the inventors found that in the related art, the positioning of the bronchoscope excessively relies on the similarity of the structural information between the real bronchoscope image and the virtual bronchoscope image, which leads to bronchoscope tracking failure or yaw, reducing the efficiency and accuracy of the operation. Therefore, the present disclosure generates a first conversion matrix by pre-setting a registration rule to register the real bronchoscope model data and the virtual bronchoscope model data, and then maps the real bronchoscope pose data to the virtual bronchoscope pose data based on the first conversion matrix and a pre-set mapping rule. In the case that the pose data of the virtual bronchoscope exceeds a pre-set threshold, the pose data of the virtual bronchoscope is updated and optimized using the position data and direction data of the target time before the pre-set threshold is exceeded, to obtain the target pose data corresponding to the virtual bronchoscope, thereby realizing dynamic positioning of the bronchoscope, avoiding the situation that excessive reliance on the similarity between the structural information of the real and virtual bronchoscope images leads to tracking failure or yaw, improving the accuracy of bronchoscope positioning, and further improving the efficiency of bronchoscope examination.

[0036] Embodiments of the present disclosure provide a bronchoscope dynamic positioning method, device, equipment, storage medium and program product. The method comprises: processing image data corresponding to a real bronchus model to generate virtual bronchus model data; performing registration operation on real bronchus model data and virtual bronchus model data based on a pre-set registration rule to generate a first conversion matrix; mapping real bronchoscope pose data to virtual bronchoscope pose data based on the first conversion matrix and a pre-set mapping rule, wherein the pose data comprises position data and direction data corresponding to the bronchoscope; in the case that the pose data of the virtual bronchoscope exceeds a pre-set threshold, updating the pose data of the virtual bronchoscope using a pre-set update rule to determine target pose data corresponding to the virtual bronchoscope, wherein the pre-set update rule is associated with the position data and direction data of the virtual bronchoscope at the target time before the pre-set threshold is exceeded; and dynamically positioning the bronchoscope based on the target pose data.

[0037] Figure 1An application scenario diagram of a bronchoscope dynamic positioning method and device according to an embodiment of the present disclosure is schematically shown.

[0038] As shown in Figure 1 the application scenario according to this embodiment can include a magnetic navigation bronchoscope system 110, a control terminal device 120 and a network 130, wherein the magnetic navigation bronchoscope system 110 can include a magnetic navigation sensor 111, a bronchoscope camera 112, a magnetic field generator 113 and a display terminal device 114, the magnetic navigation sensor 111 is connected with the control terminal device 120 through the network 130, the bronchoscope camera 112 is connected with the control terminal device 120 and the display terminal device 114 through the network 130, the control terminal device 120 can be used for receiving preoperative CT image data of a patient and performing corresponding processing to obtain three-dimensional reconstruction model data, the magnetic field generator 113 can be used for acquiring positioning and navigation data fed back by the magnetic navigation sensor 111. The display terminal device can be used for displaying image / video data shot by the bronchoscope camera 112.

[0039] The magnetic navigation sensor 111 is used for position tracking and data transmission in a bronchoscope related operation, the magnetic navigation sensor 111 is usually located at a tip or other key parts of the bronchoscope, can detect the spatial position and direction of the bronchoscope in the body in real time. At the same time, the magnetic navigation sensor 111 can transmit the position and direction data to a navigation system to display the position of the bronchoscope in real time; the bronchoscope camera 112 can also be called a bronchoscope, is used for acquiring real-time images inside the airway, can help medical staff to observe and examine the internal structure of the bronchus and lung; the magnetic field generator 113 can generate a magnetic field with known characteristics around the operation area of the bronchoscope, through cooperation with the magnetic navigation sensor 111, the magnetic field generator 113 can determine the accurate position and direction of the bronchoscope 112. The display terminal device 114 (such as a monitor or screen) can be used for displaying real-time images captured by the bronchoscope camera 112 and three-dimensional position data generated by the navigation system. The current position and path of the bronchoscope 112 in the airway are displayed in real time, providing visual navigation information for doctors to accurately guide the bronchoscope to reach the target area (lesion).

[0040] In a feasible embodiment, the control terminal device 120 can also be used for acquiring data fed back by the magnetic navigation sensor 111; or the magnetic field generator 113 can be connected with the control terminal device 120 through the network 130 to transmit the acquired data to the control terminal device 120. It should be explained that the above connection mode is only schematic and is not used for limiting the present disclosure, and can be determined according to actual conditions.

[0041] It should be understood that Figure 1The number of the magnetic navigation bronchoscope system, the control terminal device and the network in the figure is only illustrative. According to the implementation needs, there can be any number of magnetic navigation bronchoscope systems, control terminal devices and networks.

[0042] Figure 2 A flowchart of a bronchoscope dynamic positioning method according to an embodiment of the present disclosure is illustratively shown.

[0043] As Figure 2 shown, the bronchoscope dynamic positioning method of this embodiment includes operations S210-S250.

[0044] In operation S210, image data corresponding to a real bronchial model is processed to generate virtual bronchial model data.

[0045] According to an embodiment of the present disclosure, the image data can be computed tomography data, which can represent detailed structure and tissue information inside the human body. The computed tomography data is two-dimensional data, and a three-dimensional virtual bronchial model data can be constructed by segmenting and stacking the computed tomography data.

[0046] In operation S220, a registration operation is performed on the real bronchial model data and the virtual bronchial model data based on a preset registration rule to generate a first conversion matrix.

[0047] According to an embodiment of the present disclosure, the preset registration rule can be a rule for determining feature point data in the virtual bronchial model based on feature point data in the real bronchial model data, thereby performing a registration operation to determine a registration conversion matrix, which can also be referred to as the first conversion matrix.

[0048] In operation S230, real bronchoscope pose data is mapped to virtual bronchoscope pose data based on the first conversion matrix and a preset mapping rule, wherein the pose data includes position data and direction data corresponding to the bronchoscope.

[0049] According to an embodiment of the present disclosure, after the real and virtual bronchial models are registered, the position data of the calibration points on the calibration board in the bronchoscope coordinate system and the sensor coordinate system can be used to determine the conversion matrix between the real bronchoscope and the target sensor (magnetic navigation sensor), and then the pose data of the real bronchoscope is mapped to the pose data of the virtual bronchoscope. It should be noted that the pose data in the present disclosure can represent any point on the center line of the virtual bronchial model data, i.e., any point in the center path data generated from the planned tracheal entrance (starting point) to the lesion (target point), and the center path data is also the motion path data of the bronchoscope.

[0050] In operation S240, in a case where the pose data of the virtual bronchoscope exceeds the preset threshold, the pose data of the virtual bronchoscope is updated by using a preset update rule to determine target pose data corresponding to the virtual bronchoscope, wherein the preset update rule is associated with position data and direction data of the virtual bronchoscope at a target moment before the preset threshold is exceeded.

[0051] According to an embodiment of the present disclosure, the preset threshold can represent a boundary threshold of a lens of the virtual bronchoscope, or can represent a frequency threshold at which the lens of the bronchoscope exceeds the boundary threshold, or a combination of the boundary threshold and the frequency threshold. The target pose data can represent pose data of the virtual bronchoscope when the virtual bronchoscope does not collide with the virtual bronchial wall. When the virtual bronchoscope collides with the virtual bronchial wall, the pose of the virtual bronchoscope can be updated based on the position of the virtual bronchial wall spatial boundary.

[0052] In operation S250, the bronchoscope is dynamically positioned based on the target pose data.

[0053] According to an embodiment of the present disclosure, when the virtual bronchoscope collides with the virtual bronchial wall, the position of the virtual bronchoscope can be corrected based on a position correction strategy of the virtual spatial boundary. When the virtual bronchoscope does not collide with the spatial boundary, the virtual bronchoscope will continue to maintain the current motion state of the real bronchoscope to achieve dynamic positioning of the virtual bronchoscope.

[0054] In a feasible embodiment, a simple two-dimensional human body scan data can be processed to obtain a three-dimensional virtual model by using an image processing tool, and visualized. For example, taking CT (Computed Tomography) scan data as an example, high-resolution CT image data in a specific range of the bronchus of a patient is obtained by computer tomography, so that the CT image data is loaded by using VTK (Visualization Toolkit) and related algorithms (for example, Marching Cubes algorithm) to construct a three-dimensional bronchial model.

[0055] According to an embodiment of the present disclosure, the real bronchoscope model data and the virtual bronchoscope model data are registered by a preset registration rule to generate a first conversion matrix, the real bronchoscope pose data is mapped to the virtual bronchoscope pose data based on the first conversion matrix and a preset mapping rule, and in a case where the pose data of the virtual bronchoscope exceeds a preset threshold, the pose data of the virtual bronchoscope is updated and optimized by using the position data and the direction data of a target time point before the preset threshold is exceeded to obtain target pose data corresponding to the virtual bronchoscope, thereby realizing dynamic positioning of the bronchoscope and avoiding a situation of tracking failure or yawing caused by excessive dependence on the similarity between the real and virtual bronchoscope image structure information, improving the accuracy of bronchoscope positioning and further improving the efficiency of bronchoscope examination.

[0056] According to an embodiment of the present disclosure, the preset threshold includes a boundary threshold and a frequency threshold, and the preset update rule includes a first update rule and a second update rule; in a case where the pose data of the virtual bronchoscope exceeds the preset threshold, the pose data of the virtual bronchoscope is updated by using the preset update rule to determine the target pose data corresponding to the virtual bronchoscope, including: in a case where the pose data of the virtual bronchoscope is greater than the boundary threshold, the pose data of the virtual bronchoscope is updated by using the first update rule to determine the k-1 pose data; the k-1 pose data is compared with the boundary threshold, and in a case where the k-1 pose data is greater than the boundary threshold, the k-1 frequency data is determined; the k-1 frequency data is compared with the frequency threshold to generate a comparison result; in a case where the comparison result represents that the k-1 frequency data is greater than the frequency threshold, the k-1 pose data is updated by using the second update rule to generate the k pose data; and in a case where the k pose data is less than the boundary threshold, the k pose data is determined as the target pose data corresponding to the virtual bronchoscope.

[0057] According to an embodiment of the present disclosure, the boundary threshold can represent the motion boundary or range of the virtual bronchoscope under normal circumstances in the virtual bronchus. The frequency threshold can represent the normal number or frequency of the virtual bronchoscope exceeding the boundary threshold. From the actual bronchoscope intervention scene, it can be known that even if the virtual bronchoscope is mapped to the outside of the virtual space, the motion of the real bronchoscope is always limited within the bronchus entity space. Therefore, in a case where the pose data of the virtual bronchoscope is greater than the boundary threshold and the frequency threshold, it represents that the real bronchoscope has been fitted to the bronchus wall for motion. When the real bronchoscope is moving, the motion direction thereof is basically perpendicular to the normal vector of the bronchus wall closest to the bronchoscope. Therefore, when the virtual bronchoscope collides with the virtual bronchus wall, the normal vector of the space boundary of the virtual bronchus wall can be used to correct the position of the virtual bronchoscope.

[0058] According to an embodiment of the present disclosure, the k-1th pose data is obtained by updating the k-2th pose data (i.e., the pose data of the bronchoscope exceeding the boundary threshold) using the first updating rule and the second updating rule. Considering that in the actual application process, the corrected pose of the bronchoscope (e.g., the k-1th pose data) may also continue to exceed the spatial boundary, in other words, after the pose data of the virtual bronchoscope is updated and corrected, the virtual bronchoscope still has the possibility of continuously colliding with the virtual bronchial wall. In the case that the k-1th pose data exceeds the boundary threshold, the k-1th pose data can be continuously updated and corrected, and the k-1th frequency data (i.e., the number of times that the k-1th pose data exceeds the boundary threshold) is determined, so as to compare the k-1th frequency data with the frequency threshold, and in the case that the k-1th frequency data is greater than the frequency threshold (e.g., 3 times), it is represented that the real bronchoscope has been in motion adhering to the bronchial wall.

[0059] According to an embodiment of the present disclosure, in the case that the k-1th frequency data is greater than the frequency threshold, the k-1th pose data can be updated and corrected using the second updating rule, for example, in the case that the k-1th pose data is regarded as the current point, the spatial point obtained by extending the normal vector of the boundary point closest to the current point to the inside of the virtual bronchial space by a preset distance (e.g., 1.5 millimeters) is taken as the updated point (i.e., the kth pose data) after collision. It should be noted that the preset distance can be related to the radius distance (e.g., 1.5 millimeters) of the bronchoscope tip.

[0060] According to an embodiment of the present disclosure, in the case that the pose data of the virtual bronchoscope is greater than the boundary threshold, the pose data of the virtual bronchoscope is updated using the first updating rule to determine the k-1th pose data, including: in the case that the k-2th pose data is greater than the boundary threshold, the k-3th pose data is obtained; from the point cloud data set corresponding to the virtual bronchial model data, the target point data corresponding to the k-3th pose data is determined, wherein the target point data is associated with the position data; based on the target point data and the k-3th pose data, the target vector corresponding to the k-3th pose data is determined, wherein the target vector is associated with the direction data; and the k-2th pose data is updated based on the target vector and the first updating rule to determine the k-1th pose data.

[0061] According to an embodiment of the present disclosure, the first updating rule can represent a strategy rule for updating the bronchoscope (the k-2th pose data) that exceeds the virtual bronchial boundary range based on the normal vector of the virtual bronchial model space boundary. It can be understood that the k-3th pose data is the pose data of the virtual bronchoscope before exceeding the preset threshold, and the k-1th pose data is the pose data obtained by updating the k-2th pose data.

[0062] In one feasible embodiment, assume P t-1 This refers to the position data of the virtual bronchoscope at the moment of impact, located inside the virtual bronchus; P t This refers to the first point data that crosses the spatial boundary of the virtual bronchial model after a collision, located outside the model; the virtual bronchial model is then converted into a point cloud (assuming the point set is π), and indexed to a point in the virtual bronchial model point cloud at a distance P. t-1 Calculate the normal vector of the most recent point data. The specific formula is shown in formula (1) below:

[0063] (1);

[0064] Where, x i Characterization distance P t-1 The most recent point data, the dist function represents returning two points (P t-1 With x i The distance between them The function can obtain the normal vector of the point on the virtual bronchus model.

[0065] Figure 3 The illustration shows a schematic diagram of the position correction of the spatial boundary normal vector of a virtual bronchus model according to an embodiment of the present disclosure.

[0066] like Figure 3 As shown, the corrected location point is Its corrected motion vector Perpendicular to the normal vector To obtain the corrected motion vector, we can first calculate... and The angle between the vectors is shown in the following formula (2):

[0067] (2);

[0068] According to the rules of spatial vector operations, the corrected direction vector is expressed as shown in the following formula (3):

[0069] (3);

[0070] Where α is a correction coefficient, the value of which can be determined according to the actual situation. In this disclosure, α can be 1. After obtaining the corrected direction vector, the corrected point... It can be shown in the following formula (4):

[0071] (4);

[0072] Therefore, the original mapping point P outside the spatial boundary of the virtual bronchus model tThe corrected point is .

[0073] In one feasible embodiment, this disclosure takes into account that after the pose data of the virtual bronchoscope that has crossed the boundary is corrected and updated, the form of its motion mapping will also change. Therefore, the position data cannot be directly mapped, but the motion state of the real bronchoscope needs to be used and updated and optimized and mapped onto the virtual bronchial model.

[0074] Figure 4 A schematic diagram of bronchoscopic motion mapping according to an embodiment of the present disclosure is shown.

[0075] like Figure 4 As shown, correction point The motion to the next moment and the original mapping point The motion to its next mapping point is the same. The purpose of maintaining the original mapping point motion state after the position data correction is to better track the motion of the bronchoscope and suppress the distortion of motion state caused by the correction of position data.

[0076] In one feasible embodiment, considering that the virtual bronchoscope pose data may continue to exceed the spatial boundary of the virtual bronchial model during motion mapping, this disclosure incorporates a virtual bronchoscope spatial boundary out-of-bounds position data correction strategy, and the motion mapping update optimization strategy method includes operations S410 to S460.

[0077] When operating S410, the threshold for the virtual bronchoscope to continuously extend beyond the bronchial wall can be initialized. Let the correction marker be... , It can characterize the bronchoscopic pose data beyond the virtual bronchial wall.

[0078] When operating the S420, the motion vector of the actual bronchoscope can be recorded. The virtual bronchoscope follows the movement of the real bronchoscope. If the virtual bronchoscope remains inside the virtual bronchus, it can maintain the movement state of the real bronchoscope. The virtual bronchoscope's current position data is updated using its initial mapping posture. .

[0079] In operation S430, perform boundary collision detection. If the virtual bronchoscope does not collide with the virtual bronchial wall, operation S420 can be returned; if a boundary collision occurs, the distance from the point of collision can be determined according to formula (1). The normal vector of the nearest point And obtain the corrected point according to formula (4). After the virtual bronchial space boundary outer point correction is performed, the motion state can be continuously maintained: Operation S440 is performed.

[0080] In operation S440, continuous out-of-bound collision detection after position correction. Assuming that the frequency data (number of times) of collision between the virtual bronchoscope and the virtual bronchial wall is j, the initial value is 0, and the position of the virtual bronchoscope is updated according to formula (3) as follows: The out-of-bound collision detection is continuously performed, which can be divided into three cases: if no collision occurs, operation S420 is returned; if the collision continues, the correction is continuously performed according to formula (3), and the corrected position can be , and the number of collisions is recorded , the motion state is continuously maintained, , and operation S450 is performed.

[0081] In operation S450, if the number of continuous collisions is less than the frequency threshold , operation S440 is returned; when the number of continuous collisions exceeds the set frequency threshold , it can be understood that the accumulated error of the pose data is large, and the real bronchoscope has been in contact with the bronchus, and the pose data at a position where a normal vector of the boundary point closest to the current pose data is extended to the inside of the virtual bronchial model space by 1.5 millimeters can be set as the corrected point after the collision, and the radius of the bronchoscope tip is 1.5 millimeters. Assuming that the coordinates of the boundary point closest to the current bronchoscope position are , then , the corrected marker point is updated, and operation S460 is performed.

[0082] In operation S460, the motion position of the corrected marker point is tracked, if the corrected marker point is in the virtual bronchial model space, operation S420 is returned; otherwise, the position data update strategy in operation S450 is used to update the corrected marker point .

[0083] According to the embodiments of the present disclosure, the bronchoscope that exceeds the virtual bronchial boundary range is updated based on the normal vector of the virtual bronchial model space boundary, which can avoid the error caused by relying on simple position coordinates for judgment, and does not need to excessively rely on the similarity between the real and virtual bronchoscope image structure information, so that the position relationship between the bronchoscope and the virtual bronchial model can be more accurately determined. At the same time, the updating method based on the normal vector can dynamically adjust the position and attitude of the bronchoscope according to specific conditions to adapt to virtual bronchial models of different shapes and sizes, thereby improving the applicability and flexibility of the system.

[0084] ​According to an embodiment of the present disclosure, the image data corresponding to the real bronchial model is processed to generate virtual bronchial model data, including: comparing the gray value corresponding to the image data with a preset gray threshold to generate a comparison result; in the case that the comparison result represents that the gray value is greater than the preset gray threshold, the image data is determined as target image data corresponding to the real bronchial model; and the target image data is stacked to generate the virtual bronchial model data.

[0085] According to an embodiment of the present disclosure, after the image data (for example, CT or MRI scanning data) is obtained, the volume data can be segmented, and the segmentation method can include a threshold segmentation method, a region growing segmentation method, and a machine learning-based segmentation method. On this basis, the segmented image data can be denoised and smoothed to eliminate noise data (for example, artifacts and small holes); morphological operations (such as dilation, erosion, opening operation, and closing operation) are applied to further optimize the segmentation result; and finally, analysis of connected components can be performed to retain the largest connected component and remove scattered small components.

[0086] In a feasible embodiment, after the image data is segmented, the processed 2D slice data can be stacked in a specific order to form a 3D volume data; at the same time, surface extraction is performed, for example, using the MarchingCubes algorithm or other surface extraction algorithms to extract the surface mesh of the bronchus from the 3D volume data, and then using a mesh optimization strategy to simplify and smooth the extracted mesh to improve the quality and visualization effect of the model.

[0087] In a feasible embodiment, the machine learning-based image segmentation algorithm can include: performing classification through traditional machine learning, for example, using random forest, support vector machine, and the like; on this basis, deep learning is performed, and a convolutional neural network (for example, U-Net, V-Net) is used for automatic segmentation, which can include: collecting and labeling image data, for example, CT, MRI, and the like, and adding correct labels to each pixel or region for training the model; designing a CNN architecture suitable for medical image segmentation tasks, for example, U-Net, so that the network structure can effectively capture local and global information in the image and retain spatial context information; performing enhancement (rotation, flipping, scaling) processing on the training data to expand the training data set; using the prepared data set to train the CNN model, and constantly adjusting the model parameters through the back propagation algorithm to enable the model to accurately predict the bronchial structure in the image data; and using the trained model to evaluate the image data, and using pixel-level evaluation indicators (for example, Dice coefficient, IoU, and the like) to measure the performance of the model in the segmentation task.

[0088] According to an embodiment of the present disclosure, by segmenting image data through a convolutional neural network, feature representation in medical images can be automatically learned without manually designing a feature extractor, so that complex features and structures in the images can be better captured, and spatial context information of the images can be preserved through convolutional operations and pooling operations, which is conducive to accurate positioning and segmentation of structures in medical images.

[0089] According to an embodiment of the present disclosure, the virtual bronchial model data includes central path data from the starting point to the target point, and the method further includes: determining the starting point and the target point corresponding to the virtual bronchial model data; determining initial central path data from the starting point to the target point based on a preset path planning algorithm; and updating the initial central path data based on a path curvature update rule to determine the central path data.

[0090] According to an embodiment of the present disclosure, after the virtual three-dimensional bronchial model is constructed, preoperative path planning can be performed in the virtual three-dimensional model to plan a centerline from a tracheal entrance (starting point) to a lesion (target point) to generate initial central path data. The path curvature update rule can include a curvature adjustment algorithm and a minimum curvature path planning method to finally determine the central path data. For example, the way of updating the initial central path data by using the curvature adjustment algorithm can include adjusting the curvature of each point on the path through the algorithm to optimize the balance between turns and straight segments of the path, thereby improving the efficiency and safety of the navigation system.

[0091] Figure 5 A bronchial model and a center path diagram according to an embodiment of the present disclosure are schematically shown.

[0092] As shown in Figure 5 Fig. (a) is a schematic diagram of a real bronchial model, Fig. (b) is a schematic diagram of a virtual bronchial model, and Fig. (c) is a schematic diagram of a bronchial center path.

[0093] In a feasible embodiment, the planning method of the bronchial center path can include: denoising, enhancing, and image segmentation processing of the volume data (such as CT scan data), using an image segmentation algorithm (such as threshold segmentation, edge detection, etc.) to preliminarily segment the preprocessed image to obtain a candidate region possibly containing a bronchus; further extracting information related to bronchial features from the candidate region, such as shape, texture, gray scale, and other features; and then using a curve tracking or a topological structure-based method to extract the bronchial center path from the candidate region.

[0094] In a feasible embodiment, in consideration of the fact that the extracted center path may have some sharp inflection points or noises, the peaks and valleys in the path can be reduced through Gaussian filtering or spline interpolation, and the center path is smoothed to make the path more natural and continuous.

[0095] According to an embodiment of the present disclosure, the real bronchial model data and the virtual bronchial model data are registered based on a preset registration rule to generate a first conversion matrix, including: selecting a plurality of first feature point data from the point cloud corresponding to the real bronchial model data; determining a plurality of second feature point data corresponding to the virtual bronchial model data based on the plurality of first feature point data; and generating the first conversion matrix based on the preset registration rule, the plurality of first feature point data and the plurality of second feature point data.

[0096] According to an embodiment of the present disclosure, the first feature point data can represent a feature reference point in the real bronchial model, and the second feature point data can represent a feature reference point in the virtual bronchial model. In order to realize the spatial registration of the real and virtual scene coordinate systems, the marker points in the real scene can be defined first, and the marker points in the real scene are correspondingly positioned in the virtual bronchial scene.

[0097] Figure 6 The marker positioning diagram on the real bronchial model according to an embodiment of the present disclosure is schematically shown.

[0098] As Figure 6 shown, the four corner points of the real bronchial model are defined as respectively, and the corresponding marker points in the virtual bronchial model are positioned. In order to obtain the coordinates of the four corner points in the magnetic field generator coordinate system (i.e. the sensor coordinate system) in the real scene, the target sensor (such as a magnetic navigation sensor) is placed at the four marker points for measurement and recording, and for the virtual bronchoscope system scene, the four marker points in the virtual scene can be obtained through VTK interface interaction. It should be noted that the four corner point marker positioning can be determined according to the actual situation, which is not limited here.

[0099] In a feasible embodiment, the optimal transformation matrix T can be solved based on the nonlinear optimization registration method using the Gauss-Newton iteration method, and the objective function of the optimization can be shown in the following formula (5):

[0100] (5);

[0101] Wherein, is a feature point in the virtual bronchial model, is a feature point in the real bronchial model, and the error function can be The exponential mapping relationship of the Lie group Lie algebra can obtain the following formula (6) as shown:

[0102] (6) ;

[0103] wherein, is a six-dimensional vector, the symbol represents converting the Lie algebra into a 4x4 matrix, so as to obtain the incremental equation thereof as shown in the following formula (7):

[0104] (7) ;

[0105] After Taylor expansion, formula (8) can be obtained:

[0106] (8) ;

[0107] wherein, is the derivative of the error function to the disturbance , the matrix dimension is 4x6, and the Jacobian matrix of the disturbance model can be shown in the following formula (9):

[0108] (9) ;

[0109] wherein, the function represents the anti-symmetric matrix of the orientation vector, and the returned matrix dimension is 3x3.

[0110] According to embodiments of the present disclosure, the algorithm based on Gauss-Newton iterative optimization can include operations S610-S640.

[0111] In operation S610, the transformation matrix is initialized as , the iteration number k = 0, and the iteration termination condition eps is set.

[0112] In operation S620, the current Jacobian matrix and the error can be calculated according to the above formula (9), and , .

[0113] In operation S630, the matrix equation can be solved by using the LU Decomposition decomposition method.

[0114] In operation S640, the transformation matrix is updated: , and it is checked whether the termination condition is met, otherwise the above operations are repeatedly executed.

[0115] According to an embodiment of the present disclosure, the nonlinear optimization-based registration method can directly solve the conversion matrix without the need for secondary solving, simplifying the calculation process, saving the resource consumption of the system, improving the registration efficiency between the real bronchial model and the virtual bronchial model, and the nonlinear optimization registration method can process various types of transformation models and error metrics, and has good flexibility.

[0116] According to an embodiment of the present disclosure, the mapping of the pose data of the real bronchoscope to the pose data of the virtual bronchoscope based on the first conversion matrix and the preset mapping rule comprises: obtaining first position data and second position data of the calibration point position on the calibration plate relative to the bronchoscope coordinate system and the sensor coordinate system respectively; determining a second conversion matrix of the real bronchoscope and the target sensor based on the first position data and the second position data; and determining the mapping of the pose data of the real bronchoscope to the pose data of the virtual bronchoscope based on the first conversion matrix, the second conversion matrix and the preset mapping rule.

[0117] According to an embodiment of the present disclosure, the first position data can represent the position data of the calibration point position on the calibration plate in the bronchoscope coordinate system, and the second position data can represent the position data of the calibration point in the sensor coordinate system. The second conversion matrix can represent the conversion relationship between the real bronchoscope and the target sensor. After the real and virtual bronchial models are registered, the pose data of the real bronchoscope can be mapped to the pose data of the virtual bronchoscope.

[0118] Figure 7 A schematic diagram of a bronchoscope coordinate system and a sensor coordinate system in a real scene according to an embodiment of the present disclosure is shown.

[0119] As shown in the coordinate system relationship in the real scene, Figure 7 , the magnetic field generator coordinate system 710 (i.e. the sensor coordinate system) is defined as the world coordinate system (O w -X w -Y w -Z w ), and the bronchoscope coordinate system 720 (O c -X c -Y c -Z c ) is defined as the image coordinate system (O c , the focal point of the bronchoscope, the Z c axis is perpendicular to the imaging plane 730 passing through the optical center O1, the X c and Y c axes are parallel to the u and v axes of the image coordinate system, and the distance from the imaging plane to the bronchoscope coordinate system is the focal length f.

[0120] The relative position relationship of the bronchoscope when installing the target sensor is random, and the poses of the two exist a certain degree of attitude and position deviation, so the value of the target sensor cannot be directly used as the pose of the bronchoscope for calculation; the conversion relationship between the target sensor and the pose of the bronchoscope (camera) can be obtained through calibration; the calibration uses a 7*7 circle dot type calibration board 740, which is fixed on the operating table; in the sensor coordinate system 710, the coordinates of each circle dot are measured by the target sensor, so that the bronchoscope is fixed at a specific position, the pixel position of each circle dot recognized by the bronchoscope is recorded and corresponds to the circle dot coordinate on the calibration board 740, and the change matrix T c,s from the target sensor to the bronchoscope is calibrated.

[0121] Figure 8 A sensor and camera coordinate system relationship diagram according to an embodiment of the disclosure is schematically shown.

[0122] As shown in Figure 8 , in order to obtain a spatial scene consistent with the real bronchoscope in the virtual bronchoscope, a pose mapping from the magnetic navigation sensor 810 to the virtual bronchoscope 820 can be established. The pose of the virtual bronchoscope in the virtual world coordinate (X ) can be defined as , the pose of the real bronchoscope in the sensor coordinate system is , and the pose of the sensor output by the magnetic navigation sensor is T sensor , then the mapping function from the magnetic navigation sensor to the virtual bronchoscope can be obtained as .

[0123] In a feasible embodiment, the updated change matrix (first conversion matrix) T k+1 obtained according to the above-mentioned algorithm based on Gauss-Newton iteration optimization can be used to know the spatial coordinate mapping relationship from the real scene to the virtual scene, the first conversion matrix is defined as T V,R here, and the second matrix of the sensor pose to the bronchoscope can be calibrated . For the spatial coordinate mapping (first conversion matrix) , it plays a bridging role in the mapping of the bronchoscope, that is, the three-dimensional coordinates or the pose of the bronchoscope in the real scene can be mapped to the virtual scene.

[0124] According to an embodiment of the disclosure, the pose data of the bronchoscope in the real scene can be defined as point P r , and the pose data of the bronchoscope in the virtual scene is point P v , then the following formula (10) can be obtained:

[0125] (10);

[0126] Similarly, for the bridging mapping equation from the real bronchoscope to the virtual bronchoscope, it can be shown as formula (11) as follows:

[0127] (11) ;

[0128] wherein, is the pose of the virtual bronchoscope in the virtual world coordinate (xv, yv, zv), is the pose of the real bronchoscope in the sensor coordinate system.

[0129] Figure 9 A real bronchoscope and virtual bronchoscope mapping relationship diagram according to an embodiment of the present disclosure is schematically shown.

[0130] As shown in FIG. 9, the pose data of the sensor can be first obtained from the magnetic field generator, converted to the real bronchoscope 910 (camera) pose, and then bridged to the virtual bronchoscope 920 using the first conversion matrix T Figure 9 of spatial registration, and the specific mapping function expression is shown as formula (12) as follows:

[0131] (12) ;

[0132] wherein, T sensor is the sensor pose output by the magnetic navigation sensor.

[0133] After the real and virtual space registration (the first conversion matrix T V,R ) and the conversion matrix (the second matrix T ) of the target sensor to the bronchoscope are determined, the data stream output by the magnetic navigation sensor can be converted to the pose of the virtual bronchoscope through the mapping function , so as to realize the pose mapping of the real bronchoscope to the virtual bronchoscope.

[0134] According to an embodiment of the present disclosure, the initial center path data is updated based on a path curvature update rule to determine the center path data, including: determining target path point data in the initial center path data, wherein the target path point data includes turning point data and straight section point data; and updating the initial center path data based on the path curvature update rule, the turning point data and the straight section point data to determine the center path data.

[0135] ​​According to an embodiment of the present disclosure, the path curvature updating rule can be a method of obtaining clearer and more realistic path data by smoothing the planned initial center path. The turning point data and the straight segment point data of the initial center path can be smoothed by a curvature adjustment algorithm, which can include: initialization based on the initial path or according to the specific requirements of the center path; then calculating the quality of the current path according to the defined objective function, and then adjusting the curvature at each path point by using the selected optimization algorithm (such as Newton method) to minimize the objective function; repeat the above steps until the convergence condition is reached (such as the objective function converges to a stable value, or the number of iterations reaches a preset value).

[0136] According to an embodiment of the present disclosure, by optimizing the curvature, the sharp changes of the turning on the path can be reduced, especially in the motion control of the robot (bronchoscope), the stability and smoothness of the path can be improved, and the incoherence of the motion can be reduced.

[0137] According to an embodiment of the present disclosure, the first conversion matrix is generated based on the preset registration rule, the plurality of first feature point data and the plurality of second feature point data, including: determining the error parameter and the transformation matrix corresponding to the initialization conversion matrix based on the preset registration rule, the plurality of first feature point data and the plurality of second feature point data; determining the parameter variable based on the error parameter and the transformation matrix; and updating the initialization conversion matrix based on the parameter variable to obtain the first conversion matrix.

[0138] According to an embodiment of the present disclosure, as shown in the above formula (5) to formula (9), the initialization conversion matrix can be updated and optimized based on the Gauss-Newton iterative optimization algorithm, using the error parameter and the transformation matrix to obtain the first conversion matrix.

[0139] Figure 10A The system architecture diagram of the bronchoscope dynamic positioning method according to an embodiment of the present disclosure is schematically shown.

[0140] As Figure 10AAs shown, the real bronchial model can be first CT scanned to obtain image data 1010 (for example, CT or MRI scanning data), the image data 1010 is segmented and stacked, the virtual bronchial model data 1020 is reconstructed and generated, and the bronchial inlet to the center path data 1030 of the lesion is planned on this basis; the real bronchial model data 1040 and the virtual bronchial model data 1020 are registered based on the preset registration rule, the first conversion matrix 1050 is generated, and the real bronchoscope pose data is mapped to the virtual bronchoscope pose data 1060 based on the first conversion matrix 1050 and the preset mapping rule; in the case that the pose data of the virtual bronchoscope exceeds the preset threshold, the pose data of the virtual bronchoscope is updated by using the preset update rule, the target pose data 1070 corresponding to the virtual bronchoscope is determined, and the virtual bronchoscope is dynamically positioned based on the target pose data.

[0141] Figure 10B The effect diagram of the virtual bronchoscope dynamic positioning according to the embodiment of the present disclosure is schematically shown.

[0142] The virtual bronchoscope dynamic positioning optimization effect based on the magnetic navigation sensor is as shown in the following figure. Figure 10B As shown, the green symbol represents the real position of the bronchoscope under the bronchial model, and the blue and red symbols represent the positions of the virtual bronchoscope after the dynamic optimization and without dynamic optimization, respectively. From the effect diagram, it can be seen that the position of the virtual bronchoscope in the virtual scene space after the optimization update using the optimization mapping strategy provided by the present disclosure can better reflect the position data of the real bronchoscope.

[0143] According to the embodiment of the present disclosure, the dynamic positioning optimization strategy is proposed by using the boundary relationship of the virtual bronchial space, the space out-of-bound position of the virtual bronchoscope is corrected, and the mapping of the motion behavior of the virtual bronchoscope out of the space boundary is updated and optimized. Figure 10B The three examples are that the virtual bronchial model space mapping out-of-bound occurs in the second and third bronchial parts of the bronchus, and the bronchoscope dynamic positioning optimization strategy ensures the bronchoscope mapping accuracy under the registration and reconstruction error, realizes the one-to-one correspondence of the bronchoscope image of the real bronchial model space and the virtual bronchial model space, and is beneficial to the doctor to accurately and quickly position the lesion and the position relationship of the endoscope (bronchoscope) in the bronchoscope examination and diagnosis process.

[0144] Based on the above bronchoscope dynamic positioning method, the present disclosure further provides a bronchoscope dynamic positioning device. The device will be described in detail below. Figure 11 The device is described in detail.

[0145] Figure 11A structural block diagram of a bronchoscope dynamic positioning apparatus according to an embodiment of the present disclosure is shown schematically.

[0146] As shown in Figure 11 The bronchoscope dynamic positioning apparatus of this embodiment includes a model data generation module 1110, a first conversion matrix generation module 1120, a mapping module 1130, a target pose data determination module 1140, and a positioning module 1150.

[0147] The model data generation module 1110 is configured to process image data corresponding to a real bronchial model to generate virtual bronchial model data. In an embodiment, the model data generation module 1110 can be configured to perform operation S210 described above, and thus repeated details are not provided herein.

[0148] The first conversion matrix generation module 1120 is configured to perform a registration operation on the real bronchial model data and the virtual bronchial model data based on a preset registration rule to generate a first conversion matrix. In an embodiment, the first conversion matrix generation module 1120 can be configured to perform operation S220 described above, and thus repeated details are not provided herein.

[0149] The mapping module 1130 is configured to map real bronchoscope pose data to virtual bronchoscope pose data based on the first conversion matrix and a preset mapping rule, wherein the pose data includes position data and direction data corresponding to the bronchoscope. In an embodiment, the mapping module 1130 can be configured to perform operation S230 described above, and thus repeated details are not provided herein.

[0150] The target pose data determination module 1140 is configured to update the pose data of the virtual bronchoscope using a preset update rule in a case where the pose data of the virtual bronchoscope exceeds a preset threshold, to determine target pose data corresponding to the virtual bronchoscope, wherein the preset update rule is associated with position data and direction data of the virtual bronchoscope at a target time point before the preset threshold is exceeded. In an embodiment, the target pose data determination module 1140 can be configured to perform operation S240 described above, and thus repeated details are not provided herein.

[0151] The positioning module 1150 is configured to perform dynamic positioning of the bronchoscope based on the target pose data. In an embodiment, the positioning module 1150 can be configured to perform operation S250 described above, and thus repeated details are not provided herein.

[0152] According to an embodiment of the present disclosure, by means of the model data generation module 1110, the first conversion matrix generation module 1120, the mapping module 1130, the target pose data determination module 1140 and the positioning module 1150 in the bronchoscope dynamic positioning device, the real bronchoscope model data and the virtual bronchoscope model data are registered by a preset registration rule, a first conversion matrix is generated, and then the real bronchoscope pose data is mapped to the virtual bronchoscope pose data based on the first conversion matrix and a preset mapping rule. In the case that the pose data of the virtual bronchoscope exceeds a preset threshold, the pose data of the virtual bronchoscope is updated and optimized by using the position data and the direction data at the target time before the preset threshold is exceeded, the target pose data corresponding to the virtual bronchoscope is obtained, and the dynamic positioning of the bronchoscope is realized. This avoids the situation that the tracking fails or deviates due to excessive dependence on the similarity between the real and virtual bronchoscope image structure information, improves the accuracy of bronchoscope positioning, and further improves the efficiency of bronchoscope examination.

[0153] According to an embodiment of the present disclosure, the preset threshold includes a boundary threshold and a frequency threshold, and the preset update rule includes a first update rule and a second update rule. The target pose data determination module includes a (k-1)th pose data determination submodule, a (k-1)th frequency data determination submodule, a comparison result generation submodule, a kth pose data generation submodule and a target pose data determination submodule.

[0154] The (k-1)th pose data determination submodule is configured to update the pose data of the virtual bronchoscope by using the first update rule in the case that the pose data of the virtual bronchoscope is greater than the boundary threshold, and determine the (k-1)th pose data.

[0155] The (k-1)th frequency data determination submodule is configured to compare the (k-1)th pose data with the boundary threshold, and determine the (k-1)th frequency data in the case that the (k-1)th pose data is greater than the boundary threshold.

[0156] The comparison result generation submodule is configured to compare the (k-1)th frequency data with the frequency threshold, and generate a comparison result.

[0157] The kth pose data generation submodule is configured to update the (k-1)th pose data by using the second update rule in the case that the comparison result represents that the (k-1)th frequency data is greater than the frequency threshold, and generate the kth pose data.

[0158] The target pose data determination submodule is configured to determine the kth pose data as the target pose data corresponding to the virtual bronchoscope in the case that the kth pose data is less than the boundary threshold.

[0159] According to an embodiment of the present disclosure, the k-1 pose data determination sub-module comprises: a k-3 pose data acquisition unit, a target point position data determination unit, a target vector determination unit, and a k-1 pose data determination unit.

[0160] The k-3 pose data acquisition unit is configured to acquire k-3 pose data when the k-2 pose data is greater than the boundary threshold.

[0161] The target point position data determination unit is configured to determine target point position data corresponding to the k-3 pose data from a point cloud data set corresponding to the virtual bronchial model data, wherein the target point position data is associated with position data.

[0162] The target vector determination unit is configured to determine a target vector corresponding to the k-3 pose data based on the target point position data and the k-3 pose data, wherein the target vector is associated with direction data.

[0163] The k-1 pose data determination unit is configured to update the k-2 pose data based on the target vector and a first update rule to determine k-1 pose data.

[0164] According to an embodiment of the present disclosure, the model data generation module comprises: a comparison result generation sub-module, a target image data determination sub-module, and a model data generation sub-module.

[0165] The comparison result generation sub-module is configured to compare the grayscale value corresponding to the image data with a preset grayscale threshold to generate a comparison result.

[0166] The target image data determination sub-module is configured to determine the image data as target image data corresponding to the real bronchial model when the comparison result indicates that the grayscale value is greater than the preset grayscale threshold.

[0167] The model data generation sub-module is configured to stack the target image data to generate virtual bronchial model data.

[0168] According to an embodiment of the present disclosure, the virtual bronchial model data comprises center path data from a starting point to a target point, and the device further comprises: a starting point and target point determination module, an initial center path data determination module, and a center path data determination module.

[0169] The starting point and target point determination module is configured to determine a starting point and a target point corresponding to the virtual bronchial model data.

[0170] The initial center path data determination module is configured to determine initial center path data from the starting point to the target point based on a preset path planning algorithm.

[0171] The center path data determination module is configured to update the initial center path data based on a path curvature update rule to determine the center path data.

[0172] According to an embodiment of the present disclosure, the first conversion matrix generation module comprises a first feature point data selection sub-module, a second feature point data determination sub-module, and a first conversion matrix generation sub-module.

[0173] The first feature point data selection sub-module is configured to select a plurality of first feature point data from the point cloud corresponding to the real bronchial model data.

[0174] The second feature point data determination sub-module is configured to determine a plurality of second feature point data corresponding to the virtual bronchial model data based on the plurality of first feature point data.

[0175] The first conversion matrix generation sub-module is configured to generate the first conversion matrix based on a preset registration rule, the plurality of first feature point data, and the plurality of second feature point data.

[0176] According to an embodiment of the present disclosure, the mapping module comprises a position data acquisition sub-module, a second conversion matrix determination sub-module, and a pose data determination sub-module.

[0177] The position data acquisition sub-module is configured to acquire first position data and second position data of a calibration point on a calibration plate relative to a bronchoscope coordinate system and a sensor coordinate system, respectively.

[0178] The second conversion matrix determination sub-module is configured to determine a second conversion matrix of a real bronchoscope and a target sensor based on the first position data and the second position data.

[0179] The pose data determination sub-module is configured to determine pose data of the real bronchoscope mapped to pose data of a virtual bronchoscope based on the first conversion matrix, the second conversion matrix, and a preset mapping rule.

[0180] According to an embodiment of the present disclosure, the center path data determination module comprises a target path point data determination sub-module and a center path data determination sub-module.

[0181] The target path point data determination sub-module is configured to determine target path point data in the initial center path data, wherein the target path point data comprises turning point data and straight segment interval point data.

[0182] The center path data determination sub-module is configured to update the initial center path data based on a path curvature update rule, the turning point data, and the straight segment interval point data to determine the center path data.

[0183] According to an embodiment of the present disclosure, the first conversion matrix generation submodule comprises: a parameter and matrix determination unit, a parameter variable determination unit, and an initialized conversion matrix updating unit.

[0184] The parameter and matrix determination unit is configured to determine an error parameter and a transformation matrix corresponding to the initialized conversion matrix based on a preset registration rule, the plurality of first feature point data, and the plurality of second feature point data.

[0185] The parameter variable determination unit is configured to determine a parameter variable based on the error parameter and the transformation matrix.

[0186] The initialized conversion matrix updating unit is configured to update the initialized conversion matrix based on the parameter variable to obtain the first conversion matrix.

[0187] According to an embodiment of the present disclosure, any one or more of the model data generation module 1110, the first conversion matrix generation module 1120, the mapping module 1130, the target pose data determination module 1140, and the positioning module 1150 can be combined in one module, or any one of them can be split into multiple modules. Alternatively, at least part of the function of one or more of these modules can be combined with at least part of the function of other modules, and implemented in one module. According to an embodiment of the present disclosure, at least one of the model data generation module 1110, the first conversion matrix generation module 1120, the mapping module 1130, the target pose data determination module 1140, and the positioning module 1150 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application specific integrated circuit (ASIC), or any other reasonable way of hardware or firmware that can be integrated or packaged into a circuit, or any one of software, hardware and firmware or any appropriate combination of several of them. Alternatively, at least one of the model data generation module 1110, the first conversion matrix generation module 1120, the mapping module 1130, the target pose data determination module 1140, and the positioning module 1150 can be at least partially implemented as a computer program module that can perform corresponding functions when the computer program module is run.

[0188] Figure 12 A block diagram of an electronic device suitable for implementing the bronchoscope dynamic positioning method according to an embodiment of the present disclosure is schematically shown.

[0189] As Figure 12As shown, the electronic device according to embodiments of the present disclosure includes a processor 1201, which can perform various appropriate actions and processes according to programs stored in a read only memory (ROM) 1202 or loaded from a storage section 1208 into a random access memory (RAM) 1203. The processor 1201 can include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor, and / or a related chip set, and / or a special purpose microprocessor (e.g., an application specific integrated circuit (ASIC)), and / or the like. The processor 1201 can also include on-board memory for cache purposes. The processor 1201 can include a single processing unit or multiple processing units to perform the various actions of the method processes according to embodiments of the present disclosure.

[0190] In the RAM 1203, various programs and data required for the operation of the electronic device 1200 are stored. The processor 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. The processor 1201 performs various operations of the method processes according to embodiments of the present disclosure by executing the programs in the ROM 1202 and / or the RAM 1203. Note that the programs can also be stored in one or more memories other than the ROM 1202 and the RAM 1203. The processor 1201 can also perform various operations of the method processes according to embodiments of the present disclosure by executing the programs stored in the one or more memories.

[0191] According to embodiments of the present disclosure, the electronic device 1200 can also include an input / output (I / O) interface 1205, which is also connected to the bus 1204. The electronic device 1200 can also include one or more of the following components connected to the input / output (I / O) interface 1205: an input section 1206 including a keyboard, a mouse, etc.; an output section 1207 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN card, a modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the input / output (I / O) interface 1205 as necessary. A removable medium 1211 such as a magnetic disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 1210 as necessary, so that a computer program read out therefrom is installed into the storage section 1208 as necessary.

[0192] The present disclosure also provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments, or exist independently without being assembled into the device / apparatus / system. The above computer readable storage medium carries one or more programs, which when executed, implement the method according to the embodiments of the present disclosure.

[0193] According to the embodiments of the present disclosure, the computer readable storage medium can be a non-volatile computer readable storage medium, which can include, but is not limited to, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination thereof. In the present disclosure, the computer readable storage medium can be any tangible medium that contains or stores a program, which can be used by or in connection with an instruction execution system, apparatus, or device. For example, according to the embodiments of the present disclosure, the computer readable storage medium can include one or more memories of the ROM 1202 and / or the RAM 1203 described above and / or one or more memories other than the ROM 1202 and the RAM 1203.

[0194] The embodiments of the present disclosure also include a computer program product, which includes a computer program containing program codes for executing the methods shown in the flowcharts. When the computer program product is run in a computer system, the program codes are used to make the computer system implement the bronchoscope dynamic positioning method provided by the embodiments of the present disclosure.

[0195] The above functions defined in the system / apparatus of the embodiments of the present disclosure are performed when the computer program is executed by the processor 1201. According to the embodiments of the present disclosure, the system, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0196] In one embodiment, the computer program can rely on a tangible storage medium such as an optical storage device, a magnetic storage device, etc. In another embodiment, the computer program can also be transmitted, distributed, and downloaded in the form of a signal via a network medium, and be installed and executed through the communication part 1209 and / or the detachable medium 1211. The program codes contained in the computer program can be transmitted via any appropriate network medium, including but not limited to wireless, wired, etc., or any appropriate combination thereof.

[0197] In such embodiments, the computer program can be downloaded and installed from the network through the communication section 1209, and / or installed from the removable medium 1211. When the computer program is executed by the processor 1201, the above-described functions defined in the system of the embodiments of the present disclosure are performed. According to the embodiments of the present disclosure, the system, device, apparatus, module, unit, and the like described above can be implemented by computer program modules.

[0198] According to the embodiments of the present disclosure, the program code for executing the computer program provided by the embodiments of the present disclosure can be written in any combination of one or more programming languages, and specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming language, and / or assembly / machine language. The programming language includes, but is not limited to, such as Java, C++, python, "C" language or similar programming language. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case involving a remote computing device, the remote computing device can be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, connected to the Internet through an Internet service provider).

[0199] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams can represent a module, a segment, or a portion of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in a different order than that shown in the figures. For example, two blocks noted in succession can actually be executed substantially concurrently, or they can sometimes be executed in reverse order, depending on the functionality involved. It should also be noted that each block in the flowcharts or block diagrams, and combinations of blocks in the flowcharts or block diagrams, can be implemented by dedicated hardware-based systems that perform the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0200] Those skilled in the art can understand that the features described in various embodiments of the present disclosure and / or claims can be combined or / and integrated, even if such combinations or integrations are not explicitly described in the present disclosure. In particular, the features described in various embodiments of the present disclosure and / or claims can be combined and / or integrated in various combinations, without departing from the spirit and teachings of the present disclosure. All these combinations and / or integrations fall within the scope of the present disclosure.

[0201] The above described embodiments of the present disclosure. However, these embodiments are merely for illustrative purposes, and are not intended to limit the scope of the present disclosure. Although each embodiment is described above separately, this does not mean that the measures in each embodiment cannot be advantageously used in combination. The scope of the present disclosure is defined by the appended claims and their equivalents. Those skilled in the art can make various substitutions and modifications without departing from the scope of the present disclosure, and all such substitutions and modifications shall fall within the scope of the present disclosure.

Claims

1. A bronchoscope dynamic positioning method, characterized in that, The method comprises: processing image data corresponding to a real bronchial model to generate virtual bronchial model data; performing registration operation on the real bronchial model data and the virtual bronchial model data based on a preset registration rule to generate a first conversion matrix; mapping real bronchoscope pose data to virtual bronchoscope pose data based on the first conversion matrix and a preset mapping rule, wherein the pose data comprises position data and direction data corresponding to the bronchoscope; in a case where the pose data of the virtual bronchoscope exceeds a preset threshold, updating the pose data of the virtual bronchoscope by using a preset update rule to determine target pose data corresponding to the virtual bronchoscope, wherein the preset update rule is associated with position data and direction data of the virtual bronchoscope at a target time before the virtual bronchoscope exceeds the preset threshold; and dynamically positioning the bronchoscope based on the target pose data; wherein the preset threshold comprises a boundary threshold and a frequency threshold, and the preset update rule comprises a first update rule and a second update rule; wherein, in a case where the pose data of the virtual bronchoscope exceeds a preset threshold, updating the pose data of the virtual bronchoscope by using a preset update rule to determine target pose data corresponding to the virtual bronchoscope, comprises: in a case where the pose data of the virtual bronchoscope is greater than the boundary threshold, updating the pose data of the virtual bronchoscope by using the first update rule to determine k-1 pose data; comparing the k-1 pose data with the boundary threshold, and in a case where the k-1 pose data is greater than the boundary threshold, determining k-1 frequency data; comparing the k-1 frequency data with the frequency threshold to generate a comparison result; in a case where the comparison result indicates that the k-1 frequency data is greater than the frequency threshold, updating the k-1 pose data by using the second update rule to generate k pose data; and in a case where the k pose data is less than the boundary threshold, determining the k pose data as the target pose data corresponding to the virtual bronchoscope.

2. The method of claim 1, wherein, in a case where the pose data of the virtual bronchoscope is greater than the boundary threshold, updating the pose data of the virtual bronchoscope by using the first update rule to determine k-1 pose data, comprises: in a case where k-2 pose data is greater than the boundary threshold, obtaining k-3 pose data; from a point cloud data set corresponding to the virtual bronchial model data, determining target point data corresponding to the k-3 pose data, wherein the target point data is associated with the position data; based on the target point data and the k-3 pose data, determining a target vector corresponding to the k-3 pose data, wherein the target vector is associated with the direction data; and updating the k-2 pose data based on the target vector and the first update rule to determine the k-1 pose data.

3. The method of claim 1, wherein, Processing image data corresponding to a real bronchial model to generate virtual bronchial model data, comprising: Comparing a grayscale value corresponding to the image data with a preset grayscale threshold to generate a comparison result; In a case where the comparison result represents that the grayscale value is greater than the preset grayscale threshold, determining the image data as target image data corresponding to the real bronchial model; Stacking the target image data to generate the virtual bronchial model data.

4. The method of claim 3, wherein, The virtual bronchial model data includes center path data from a starting point to a target point, and the method further comprises: Determining the starting point and the target point corresponding to the virtual bronchial model data; Determining initial center path data from the starting point to the target point based on a preset path planning algorithm; and Updating the initial center path data based on a path curvature update rule to determine the center path data.

5. The method of claim 1, wherein, The registration operation of the real bronchial model data and the virtual bronchial model data based on the preset registration rule generates a first conversion matrix, comprising: Selecting a plurality of first feature point data from the point cloud corresponding to the real bronchial model data; Determining a plurality of second feature point data corresponding to the virtual bronchial model data based on a plurality of the first feature point data; and Generating the first conversion matrix based on the preset registration rule, a plurality of the first feature point data and a plurality of the second feature point data.

6. The method of claim 1, wherein, The mapping of the pose data of the real bronchoscope to the pose data of the virtual bronchoscope based on the first conversion matrix and a preset mapping rule, comprising: Obtaining first position data and second position data of a calibration point on a calibration board relative to a bronchoscope coordinate system and a sensor coordinate system respectively; Determining a second conversion matrix of the real bronchoscope and the target sensor based on the first position data and the second position data; and Determining the mapping of the pose data of the real bronchoscope to the pose data of the virtual bronchoscope based on the first conversion matrix, the second conversion matrix and the preset mapping rule.

7. The method of claim 4, wherein, Updating the initial center path data based on the path curvature update rule to determine the center path data, comprising: Determining target path point data in the initial center path data, wherein the target path point data includes turning point data and straight segment point data; and Updating the initial center path data based on the path curvature update rule, the turning point data and the straight segment point data to determine the center path data.

8. The method of claim 5, wherein, Generating the first conversion matrix based on the preset registration rule, a plurality of the first feature point data and a plurality of the second feature point data, comprising: Determining an error parameter and a transformation matrix corresponding to an initialization conversion matrix based on the preset registration rule, a plurality of the first feature point data and a plurality of the second feature point data; Determining a parameter variable based on the error parameter and the transformation matrix; and Updating the initialization conversion matrix based on the parameter variable to obtain the first conversion matrix.

9. A bronchoscope dynamic positioning device, characterized by, The device comprises: The model data generation module is configured to process image data corresponding to the real bronchial model to generate virtual bronchial model data. The first conversion matrix generation module is configured to perform registration operation on the real bronchial model data and the virtual bronchial model data based on a preset registration rule to generate a first conversion matrix. The mapping module is configured to map real bronchoscope pose data to virtual bronchoscope pose data based on the first conversion matrix and a preset mapping rule, wherein the pose data includes position data and direction data corresponding to the bronchoscope. The target pose data determination module is configured to update the pose data of the virtual bronchoscope using a preset update rule when the pose data of the virtual bronchoscope exceeds a preset threshold to determine target pose data corresponding to the virtual bronchoscope, wherein the preset update rule is associated with position data and direction data of the virtual bronchoscope at a target time before the preset threshold is exceeded. The positioning module is configured to dynamically position the bronchoscope based on the target pose data. The preset threshold includes a boundary threshold and a frequency threshold, and the preset update rule includes a first update rule and a second update rule. The k-1th pose data determination submodule is configured to update the pose data of the virtual bronchoscope using the first update rule when the pose data of the virtual bronchoscope is greater than the boundary threshold to determine the k-1th pose data. The k-1th frequency data determination submodule is configured to compare the k-1th pose data with the boundary threshold and determine the k-1th frequency data when the k-1th pose data is greater than the boundary threshold. The comparison result generation submodule is configured to compare the k-1th frequency data with the frequency threshold to generate a comparison result. The kth pose data generation submodule is configured to update the k-1th pose data using the second update rule when the comparison result indicates that the k-1th frequency data is greater than the frequency threshold to generate the kth pose data. The target pose data determination submodule is configured to determine the kth pose data as the target pose data corresponding to the virtual bronchoscope when the kth pose data is less than the boundary threshold.

Images