Real-time navigation method and system for interventional surgical robot and storage medium
By using CTA and DSA data to determine the position of the three-dimensional vascular camera in interventional surgery, and mapping the slender interventional instruments to the three-dimensional vascular, solving the problem of lack of 3D visual navigation guidance in the prior art, improving navigation intuitiveness and surgery Security.
Patent Information
- Application Number
- CN202411095483.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-08-09
AI Technical Summary
The existing shortage of 3D vision navigation guidelines for elongated interventional instruments in interventional surgery, resulting in unintuitive navigation and prone to artificial errors, affecting the safety of the surgery.
By obtaining preoperative CTA data, extracting the three-dimensional blood vessels and blood vessel centerlines, combining the two-dimensional images collected by the DSA device in the operation, the final camera position of the three-dimensional blood vessels to the two-dimensional blood vessels is determined, and the target two-dimensional elongated intervention is carried out. The instrument maps to three-dimensional blood vessels to achieve 3D visual navigation.
It improves the navigational intuitiveness of interventional surgery, reduces artificial errors, enhances surgical safety, and avoids the complexity of judging the location of slender interventional instruments based on 2D images.
Smart Images

Figure CN119033464A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of medical technology, and in particular to a real-time navigation method, system and storage medium for an interventional surgical robot. Background Art
[0002] In interventional surgery, existing vascular navigation solutions are mainly based on 2D and 3D vascular registration. Doctors determine the position of slender interventional devices such as catheters and guidewires based on 2D images and use 3D information as a reference. However, there is a lack of 3D vision for navigation guidance of slender interventional devices. Summary of the invention
[0003] In response to the above technical problems, the purpose of the present application is to provide a real-time navigation method, system and storage medium for an interventional surgical robot, aiming to solve the current problem of navigation guidance for slender interventional instruments that lack 3D vision.
[0004] In a first aspect, an embodiment of the present application provides a real-time navigation method for an interventional surgical robot, comprising:
[0005] Obtain preoperative CTA data;
[0006] Extracting three-dimensional blood vessels and three-dimensional blood vessel centerlines based on the CTA data;
[0007] Acquire the two-dimensional images collected by the DSA device during surgery;
[0008] extracting a target two-dimensional blood vessel centerline and a target two-dimensional elongated interventional device from the two-dimensional image acquired by the DSA device;
[0009] Determine a final camera pose from the three-dimensional blood vessel to the two-dimensional blood vessel based on the three-dimensional blood vessel centerline and the target two-dimensional blood vessel centerline;
[0010] Based on the final camera pose, the target two-dimensional elongated interventional device is mapped to the three-dimensional blood vessel for navigation.
[0011] Furthermore, the step of determining the final camera pose from the three-dimensional blood vessel to the two-dimensional blood vessel based on the three-dimensional blood vessel centerline and the target two-dimensional blood vessel centerline includes:
[0012] For each variable in the projection parameters, a minimum value, a maximum value and a step length are set, and then all combination items are continuously traversed according to the step length, and each combination item is used to project the three-dimensional blood vessel centerline onto a two-dimensional plane to obtain a corresponding projected two-dimensional curve image;
[0013] During intraoperative registration, the similarity between the target two-dimensional blood vessel centerline and each of the projected two-dimensional curve images is calculated respectively, and the combination item corresponding to the maximum similarity is taken as the initial camera pose;
[0014] An iterative solution is performed based on the initial camera pose to obtain a final camera pose.
[0015] Furthermore, the step of determining the final camera pose from the three-dimensional blood vessel to the two-dimensional blood vessel based on the three-dimensional blood vessel centerline and the target two-dimensional blood vessel centerline includes:
[0016] For each variable in the projection parameters, a minimum value, a maximum value and a step length are set, and then all combination items are continuously traversed according to the step length, and each combination item is used to project the three-dimensional blood vessel centerline onto a two-dimensional plane to obtain a corresponding projected two-dimensional curve image;
[0017] Using the projected two-dimensional curve image as input and the combined item as output to train a convolutional neural network, and after the training is completed, outputting the trained model;
[0018] Inputting the target two-dimensional blood vessel centerline into the trained model to obtain a combination item corresponding to the target two-dimensional blood vessel centerline;
[0019] Taking the combination item corresponding to the centerline of the target two-dimensional blood vessel as the initial camera pose;
[0020] An iterative solution is performed based on the initial camera pose to obtain a final camera pose.
[0021] Furthermore, the step of performing iterative solution based on the initial camera pose to obtain the final camera pose includes:
[0022] Using a search algorithm to find the nearest neighbor point of each point in the two-dimensional curve to be registered in the center line of the target two-dimensional blood vessel;
[0023] Constructing an objective function based on each point in the two-dimensional curve to be registered and the corresponding nearest neighbor point;
[0024] Based on the initial camera pose, the final camera pose is iteratively solved with the goal of minimizing the objective function; wherein the two-dimensional curve to be registered is a projected two-dimensional curve corresponding to the camera pose during iteration;
[0025] When the objective function converges, the camera pose corresponding to the minimum value of the objective function is the final camera pose.
[0026] Furthermore, the step of respectively calculating the similarity between the target two-dimensional blood vessel centerline and each of the projected two-dimensional curve images comprises:
[0027] The similarity between the target two-dimensional blood vessel centerline and each of the projected two-dimensional curve images is calculated according to the following formulas respectively;
[0028]
[0029] Where x represents the target 2D blood vessel centerline, y represents the projected 2D curve, and μ x is the average value of the target 2D blood vessel centerline; μ y is the variance of the projected two-dimensional curve; σ x is the variance of the target 2D vessel centerline, is the variance of the projected two-dimensional curve, σ xy is the covariance of x and y, and C1 and C2 are nonzero constants.
[0030] Furthermore, the step of constructing an objective function based on each point in the two-dimensional curve to be registered and the corresponding nearest neighbor point includes:
[0031] The objective function is constructed according to the following formula:
[0032]
[0033]
[0034] in, represents the i-th point of the two-dimensional curve to be registered; Indicates that the i-th point in the two-dimensional curve to be registered is at the center x of the target two-dimensional blood vessel Target The nearest neighbor point in the 2D point; σ is the bandwidth parameter of the Gaussian kernel, which uses the maximum distance between two-dimensional points; k i is the value of the Gaussian kernel, which measures The similarity between two points; K is the objective function, and n is the total number of points of the two-dimensional curve to be registered.
[0035] Furthermore, the step of mapping the target two-dimensional elongated interventional instrument to the three-dimensional blood vessel for navigation based on the final camera pose includes:
[0036] Obtaining a projected two-dimensional curve corresponding to the final camera pose;
[0037] The projected two-dimensional curve corresponding to the final camera posture is used as the projected two-dimensional curve;
[0038] Using a search algorithm, matching the point of the target two-dimensional elongated interventional device to the closest projected two-dimensional curve point;
[0039] Based on the final camera position, determining the coordinates of the center point of the three-dimensional blood vessel through the projected two-dimensional curve point closest to it;
[0040] The color of the coordinates of the center point of the three-dimensional blood vessel is changed to the color of the slender interventional instrument, thereby completing the mapping of the target two-dimensional slender interventional instrument to the three-dimensional blood vessel.
[0041] Furthermore, the step of iteratively solving the final camera pose with the objective of minimizing the objective function includes:
[0042] The final camera position is solved iteratively by a nonlinear least squares algorithm with the goal of minimizing the objective function.
[0043] In a second aspect, an embodiment of the present application provides a real-time navigation system for an interventional surgical robot, the system comprising:
[0044] CTA data acquisition module, used to acquire preoperative CTA data;
[0045] A first extraction module, configured to extract a three-dimensional blood vessel and a three-dimensional blood vessel centerline based on the CTA data;
[0046] An acquisition module is used to obtain two-dimensional images collected by the DSA device during surgery;
[0047] A second extraction module, used to extract a target two-dimensional blood vessel centerline and a target two-dimensional elongated interventional device from the two-dimensional image acquired by the DSA device;
[0048] A final camera pose determination module, used to determine a final camera pose from a three-dimensional blood vessel to a two-dimensional blood vessel based on the three-dimensional blood vessel centerline and the target two-dimensional blood vessel centerline;
[0049] A mapping module is used to map the target two-dimensional elongated interventional instrument to the three-dimensional blood vessel based on the final camera posture for navigation.
[0050] In a third aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the real-time navigation method for an interventional surgical robot as described above are implemented.
[0051] The real-time navigation method for an interventional surgical robot provided in an embodiment of the present application includes: obtaining preoperative CTA data, extracting three-dimensional blood vessels and three-dimensional blood vessel centerlines based on the CTA data, obtaining two-dimensional images collected by a DSA device during surgery, extracting a target two-dimensional blood vessel centerline and a target two-dimensional slender interventional instrument from the two-dimensional images collected by the DSA device, determining the final camera pose from the three-dimensional blood vessel to the two-dimensional blood vessel based on the three-dimensional blood vessel centerline and the target two-dimensional blood vessel centerline, and mapping the target two-dimensional slender interventional instrument to the three-dimensional blood vessel based on the final camera pose for navigation. In this way, 3D visual navigation guidance for slender interventional instruments is achieved, and doctors can directly see the position of slender interventional instruments in three-dimensional blood vessels. Compared with judging the position of slender interventional instruments based on 2D images, the method of using 3D information as a reference is more intuitive, and at the same time avoids the human eye from matching the position of 2D slender interventional instruments in 3D blood vessels, avoiding human errors, and thus improving surgical safety. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the implementation manner will be briefly introduced below. Obviously, the drawings described below are only some implementation manners of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0053] Figure 1 is a flow chart of a real-time navigation method for an interventional surgical robot provided in an embodiment of the present application;
[0054] Figure 2 It is a schematic diagram of the structure of a real-time navigation system for an interventional surgical robot provided in an embodiment of the present application. DETAILED DESCRIPTION
[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0056] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "a", "an", "above", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of features, integers, steps, operations, elements, modules, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, modules, components, and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any module and all combinations of one or more associated listed items.
[0057] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.
[0058] See also Figure 1 , an embodiment of the present application provides a real-time navigation method for an interventional surgery robot, the method comprising:
[0059] S1. Obtain preoperative CTA data.
[0060] In step S1, the full name of CTA is Computed Tomography Angiography, which is a method of observing the cardiovascular system using computed tomography (CT) technology. CTA combines CT scanning and angiography to form non-invasive high-resolution vascular imaging, which is clinically used to evaluate vascular lesions and diseases. Before surgery, the patient's preoperative CTA data can be obtained by performing a CTA examination on the patient.
[0061] S2. Extracting three-dimensional blood vessels and three-dimensional blood vessel centerlines based on the CTA data.
[0062] In step S2, the CTA data obtained in step S1 are preprocessed before surgery, including CTA three-dimensional reconstruction to obtain three-dimensional blood vessels, and three-dimensional blood vessel segmentation to extract the three-dimensional blood vessel centerline.
[0063] S3. Acquire the two-dimensional image collected by the DSA device during the operation.
[0064] In step S3, DSA is the full name of Digital Subtraction Angiography in English, and its Chinese name is digital subtraction angiography technology. It is an X-ray imaging system that is a combination of conventional angiography and electronic computer image processing technology. Digital subtraction angiography requires the removal of interfering images overlapping with blood vessels in order to obtain clear vascular images. During interventional surgery, the DSA device will collect images of the patient, so that the two-dimensional images collected by the DSA device during the patient's surgery can be obtained.
[0065] S4. Extracting the target two-dimensional blood vessel centerline and the target two-dimensional slender interventional instrument from the two-dimensional image acquired by the DSA device.
[0066] In step S4, it should be understood that interventional surgery is a minimally invasive surgical method for diagnosing and treating diseases under the guidance of imaging equipment through slender interventional instruments such as guidewires and catheters. Therefore, the two-dimensional image acquired by the intraoperative DSA device obtained in step S3 includes images of blood vessels and slender interventional instruments. After obtaining the two-dimensional image acquired by the DSA device, the blood vessels and slender interventional instruments are segmented respectively, and the blood vessel centerline and the slender interventional instrument are extracted. The extracted blood vessel centerline is the target two-dimensional blood vessel centerline, and the extracted slender interventional instrument is the target two-dimensional slender interventional instrument.
[0067] S5. Determine a final camera pose from the three-dimensional blood vessel to the two-dimensional blood vessel based on the three-dimensional blood vessel centerline and the target two-dimensional blood vessel centerline.
[0068] In the 3D and 2D registration of blood vessels, the main problem to be solved is the Perspective-n-Point (PnP) problem, that is, solving a set of camera position and posture parameters (referred to as camera posture) in world coordinates, so that the 2D image after 3D blood vessel projection coincides with the feature points of the 2D blood vessel image collected by the DSA device. Suppose the image directly obtained by the DSA device is I fix , which is a fixed reference image; the three-dimensional blood vessel V is * The two-dimensional image obtained after projection under the parameters is The projection mapping function from three dimensions to two dimensions is T p* , subscript p * Represents the parameters corresponding to the camera pose, where p * ∈{p1,p2,p3…},p * represents the optimal parameter, S is set as the similarity metric function, and the similarity of the two sets of data is evaluated. Then It can be seen that this method is to find a set of parameters P = [Rx; Ry; Rz; Tx; Ty; Tz] within the spatial range so that the projected two-dimensional point is as similar as possible to the target two-dimensional point, where Tx, Ty, Tz represent the translation components along the x, y, and z axes; Rx, Ry, Rz represent the rotation components along the x, y, and z axes. The present invention specifically determines the final camera pose of the three-dimensional blood vessel to the two-dimensional blood vessel based on the three-dimensional blood vessel centerline and the target two-dimensional blood vessel centerline. The blood vessel centerline, as an important feature of the blood vessel structure, can well represent the overall morphology and direction of the blood vessel. In addition, in both three-dimensional and two-dimensional images, the blood vessel centerline is relatively clear and easy to extract. Therefore, using the blood vessel centerline as a feature point for registration can simplify the complexity of the registration process and improve the accuracy and robustness of the registration. In addition, compared with using the three-dimensional points and two-dimensional points of the entire blood vessel for registration, using only the centerline can greatly reduce the amount of data to be processed. This not only reduces the computational complexity, but also shortens the time required for registration.
[0069] S6. Based on the final camera pose, map the target two-dimensional elongated interventional instrument to the three-dimensional blood vessel for navigation.
[0070] In step S6, after obtaining the final camera pose, the target two-dimensional slender interventional device can be mapped to the three-dimensional blood vessel using the final camera pose and displayed through a display device, so that the doctor can directly see the position of the slender interventional device in the three-dimensional blood vessel. Compared with judging the position of the slender interventional device based on the 2D image, the method of using 3D information as a reference is more intuitive, and at the same time avoids the human eye from matching the position of the 2D slender interventional device in the 3D blood vessel, avoiding human errors, thereby improving the safety of the operation.
[0071] The real-time navigation method for an interventional surgical robot provided in an embodiment of the present application includes: obtaining preoperative CTA data, extracting three-dimensional blood vessels and three-dimensional blood vessel centerlines based on the CTA data, obtaining two-dimensional images collected by a DSA device during surgery, extracting a target two-dimensional blood vessel centerline and a target two-dimensional slender interventional instrument from the two-dimensional images collected by the DSA device, determining the final camera pose from the three-dimensional blood vessel to the two-dimensional blood vessel based on the three-dimensional blood vessel centerline and the target two-dimensional blood vessel centerline, and mapping the target two-dimensional slender interventional instrument to the three-dimensional blood vessel based on the final camera pose for navigation. In this way, 3D visual navigation guidance for slender interventional instruments is achieved, and doctors can directly see the position of slender interventional instruments in three-dimensional blood vessels. Compared with judging the position of slender interventional instruments based on 2D images, the method of using 3D information as a reference is more intuitive, and at the same time avoids the human eye from matching the position of 2D slender interventional instruments in 3D blood vessels, avoiding human errors, and thus improving surgical safety.
[0072] In one embodiment, the step of determining a final camera pose from a three-dimensional blood vessel to a two-dimensional blood vessel based on the three-dimensional blood vessel centerline and the target two-dimensional blood vessel centerline comprises:
[0073] For each variable in the projection parameters, a minimum value, a maximum value and a step length are set, and then all combination items are continuously traversed according to the step length, and each combination item is used to project the three-dimensional blood vessel centerline onto a two-dimensional plane to obtain a corresponding projected two-dimensional curve image;
[0074] During intraoperative registration, the similarity between the target two-dimensional blood vessel centerline and each of the projected two-dimensional curve images is calculated respectively, and the combination item corresponding to the maximum similarity is taken as the initial camera pose;
[0075] An iterative solution is performed based on the initial camera pose to obtain a final camera pose.
[0076] In the embodiment of the present application, when solving the PNP problem, it is usually necessary to give an initial value so that more accurate results can be obtained by iterative solution during the optimization process. In the PNP problem, the selection of the initial value is crucial to the stability of the solution process and the accuracy of the final result. If the initial value is not properly selected, the solution process may fall into a local optimal solution, or even fail to converge to the correct solution. In order to select a suitable initial value, the present application adopts a spatial traversal method, and sets the minimum value, maximum value and step size for each variable in the projection parameters (the projection parameters here are actually camera pose parameters, also including Tx, Ty, Tz, Rx, Ry, Rz), and then continuously traverses all combinations according to the step size, and uses each combination item to project all three-dimensional blood vessel centerlines to a two-dimensional plane, obtains the corresponding two-dimensional curve image after projection, and records and saves all the two-dimensional curve images after projection. Then, during the intraoperative registration, the similarity between the target two-dimensional blood vessel centerline (it should be understood that the target two-dimensional blood vessel centerline described here is an image) and each of the projected two-dimensional curve images is calculated respectively, and then the combination item corresponding to the maximum similarity is taken as the initial value of the PNP algorithm. This initial value will be used as the initial camera pose, and the final camera pose can be obtained by iteratively solving according to the PNP algorithm and the initial camera pose. The embodiment of the present application selects initial values through spatial traversal, which can cover different areas of the parameter space as widely as possible. Compared with randomly selecting initial values, this method is more likely to approach or find the global optimal solution, which helps to converge to the final solution faster in the subsequent iterative solution process and improve the accuracy of the solution. It should be understood that the final solution described here refers to the final camera pose.
[0077] In one embodiment, the step of determining a final camera pose from a three-dimensional blood vessel to a two-dimensional blood vessel based on the three-dimensional blood vessel centerline and the target two-dimensional blood vessel centerline comprises:
[0078] For each variable in the projection parameters, a minimum value, a maximum value and a step length are set, and then all combination items are continuously traversed according to the step length, and each combination item is used to project the three-dimensional blood vessel centerline onto a two-dimensional plane to obtain a corresponding projected two-dimensional curve image;
[0079] Using the projected two-dimensional curve image as input and the combined item as output to train a convolutional neural network, and after the training is completed, outputting the trained model;
[0080] Inputting the target two-dimensional blood vessel centerline into the trained model to obtain a combination item corresponding to the target two-dimensional blood vessel centerline;
[0081] The combination item corresponding to the center line of the target two-dimensional blood vessel is used as the initial camera pose;
[0082] An iterative solution is performed based on the initial camera pose to obtain a final camera pose.
[0083] In the embodiment of the present application, when solving the PNP problem, it is usually necessary to give an initial value so that a more accurate result can be obtained by iterative solution during the optimization process. In the PNP problem, the selection of the initial value is crucial to the stability of the solution process and the accuracy of the final result. If the initial value is not properly selected, the solution process may fall into a local optimal solution, or even fail to converge to the correct solution. In order to select a suitable initial value, the present application adopts a machine learning method, first generating a training sample, and the method for generating the training sample is specifically as follows: for each variable in the projection parameter, a minimum value, a maximum value and a step length are set, and then all combinations (i.e., a set of projection parameters) are continuously traversed according to the step length, and each combination item is used to project all three-dimensional blood vessel centerlines onto a two-dimensional plane, and the corresponding two-dimensional curve image after projection is obtained and all the two-dimensional curve images after projection are recorded and saved. Then each combination item and the corresponding two-dimensional curve image after projection are used as a set of training data, so that the training data is obtained. Afterwards, the projected two-dimensional curve image is used as input, and the combination item is used as output to train the convolutional neural network. After the training is completed, the trained model is output; the target two-dimensional blood vessel centerline is input into the trained model, and the result of the model output can be used as the initial camera pose. The final camera pose can be obtained by iteratively solving the PNP algorithm and the initial camera pose. The embodiment of the present application can learn the complex mapping relationship between projection parameters and two-dimensional curve images from training data through machine learning. This data-driven method can capture subtle features that are difficult to capture with traditional methods, so that the predicted projection parameters corresponding to the target two-dimensional blood vessel centerline are better, that is, the obtained initial pose parameters are better.
[0084] In one embodiment, the step of performing iterative solution based on the initial camera pose to obtain the final camera pose comprises:
[0085] Using a search algorithm to find the nearest neighbor point of each point in the two-dimensional curve to be registered in the center line of the target two-dimensional blood vessel;
[0086] Constructing an objective function based on each point in the two-dimensional curve to be registered and the corresponding nearest neighbor point;
[0087] Based on the initial camera pose, the final camera pose is iteratively solved with the goal of minimizing the objective function; wherein the two-dimensional curve to be registered is a projected two-dimensional curve corresponding to the camera pose during iteration;
[0088] When the objective function converges, the camera pose corresponding to the minimum value of the objective function is the final camera pose.
[0089] In the embodiment of the present application, the iteration starts from the initial camera pose. In each iteration, the three-dimensional blood vessel is projected onto a two-dimensional plane according to the current camera pose to obtain a new two-dimensional curve to be registered. The nearest neighbor point of each point in the new two-dimensional curve to be registered on the centerline of the target two-dimensional blood vessel is found by a search algorithm. The objective function value is updated according to each point in the new two-dimensional curve to be registered and the corresponding nearest neighbor point. According to the gradient information of the objective function, the update amount of the camera pose is calculated by the optimization algorithm. The camera pose is updated and the next iteration is prepared. When the value of the objective function reaches below a preset threshold, or the update amount of multiple consecutive iterations is less than a certain threshold, it is considered that the algorithm has converged, and the camera pose corresponding to the minimum value of the objective function at this time is the final camera pose. The pose parameters of the camera can be gradually optimized through the iterative solution process, so that the similarity between the two-dimensional curve projected by the camera in the current pose and the centerline of the target two-dimensional blood vessel is maximized (or the error is minimized). This optimization process can significantly improve the accuracy of the camera pose estimation, thereby improving the accuracy of the position of the target two-dimensional slender interventional device mapped to the three-dimensional blood vessel.
[0090] In addition, the search algorithm may specifically be a k-nearest neighbors search algorithm, and the objective function is specifically constructed based on a Gaussian kernel function, and the Gaussian kernel function is used to calculate the similarity between two points. It should be noted that the number of points on the centerline of the target two-dimensional blood vessel and the number of points on the two-dimensional curve after the three-dimensional blood vessel projection are not matched, and the image actually displayed may only be a part of the three-dimensional projection. The present invention uses a k-nearest neighbor search algorithm combined with a Gaussian kernel function to not only handle the problem of mismatched number of data points, but also effectively deal with local data missing or noise interference.
[0091] In one embodiment, the step of respectively calculating the similarity between the target two-dimensional blood vessel centerline and each of the projected two-dimensional curve images comprises:
[0092] The similarity between the target two-dimensional blood vessel centerline and each of the projected two-dimensional curve images is calculated according to the following formulas respectively;
[0093]
[0094] Where x represents the target 2D blood vessel centerline, y represents the projected 2D curve, and μ x is the average value of the target 2D blood vessel centerline; μ y is the variance of the projected two-dimensional curve; σ x is the variance of the target 2D vessel centerline, is the variance of the projected two-dimensional curve, σ xy is the covariance of x and y, and C1 and C2 are nonzero constants.
[0095] In the embodiment of the present application, it should be understood that the target two-dimensional blood vessel centerline is presented in the form of an image, and the image is composed of pixel points. Calculating the similarity between the target two-dimensional blood vessel centerline and the projected two-dimensional curve image is actually calculating the similarity between the two images. In the embodiment of the present application, the method for calculating the similarity between the two images preferably determines the similarity of the two images by calculating the structural similarity (SSIM, Structural Similarity Index) of the images. SSIM is an indicator for measuring the similarity between two images, commonly used in the fields of image processing and computer vision. Compared with simple pixel differences, SSIM pays more attention to the structural information of the image and is a method of perception that is closer to the human visual system. The SSIM indicator is used to measure the similarity between two images. It considers the similarity of images from three aspects: brightness, contrast and structure. Its value range is [-1,1], where 1 represents an identical image. Since SSIM comprehensively considers brightness, contrast and structural information, SSIM is used to determine the similarity between the target two-dimensional blood vessel centerline and each of the projected two-dimensional curve images. The similarity between the blood vessel centerline and each of the projected two-dimensional curve images can be more accurately evaluated. Compared with simple pixel differences, an image that is more similar to the target two-dimensional blood vessel centerline can be found, thereby finding a better initial camera pose, and then the final camera pose determined by iterative solution based on the initial camera pose is more accurate, and the position of the target two-dimensional slender interventional device mapped to the three-dimensional blood vessel is more accurate.
[0096] In one embodiment, the step of constructing an objective function based on each point in the two-dimensional curve to be registered and the corresponding nearest neighbor point comprises:
[0097] The objective function is constructed according to the following formula:
[0098]
[0099] in, represents the i-th point of the two-dimensional curve to be registered; represents the nearest neighbor point of the i-th point in the two-dimensional curve to be registered in the center line of the target two-dimensional blood vessel; σ is the bandwidth parameter of the Gaussian kernel, and the bandwidth parameter of the Gaussian kernel adopts the maximum distance value between the two-dimensional points; k i is the value of the Gaussian kernel, which measures The similarity between two points; K is the objective function, and n is the total number of points of the two-dimensional curve to be registered.
[0100] In the embodiments of the present application, express The Euclidean distance between two points, σ is the Gaussian kernel bandwidth parameter, which controls the width of the Gaussian kernel function. The maximum distance between two-dimensional points is calculated as: Among them, maxDistance is the maximum distance between two-dimensional points, C is the Curve_Reg matrix, the size is m×n, where m is the number of rows and n is the number of columns, which records the points on the two-dimensional curve to be registered. T is the Target matrix, the size is p×n, where p is the number of rows and n is the number of columns, which records the points on the centerline of the target two-dimensional blood vessel.
[0101] In one embodiment, the step of mapping the target two-dimensional elongated interventional device to the three-dimensional blood vessel for navigation based on the final camera pose comprises:
[0102] Obtaining a projected two-dimensional curve corresponding to the final camera pose;
[0103] The projected two-dimensional curve corresponding to the final camera posture is used as the projected two-dimensional curve;
[0104] Using a search algorithm, matching the point of the target two-dimensional elongated interventional device to the closest projected two-dimensional curve point;
[0105] Based on the final camera position, determining the coordinates of the center point of the three-dimensional blood vessel through the projected two-dimensional curve point closest to it;
[0106] The color of the coordinates of the center point of the three-dimensional blood vessel is changed to the color of the slender interventional instrument, thereby completing the mapping of the target two-dimensional slender interventional instrument to the three-dimensional blood vessel.
[0107] In the embodiment of the present application, the projected two-dimensional curve corresponding to the final camera posture refers to the image obtained by projecting the three-dimensional blood vessel centerline onto a two-dimensional plane using the final camera posture. For the convenience of description, the projected two-dimensional curve corresponding to the final camera posture is used as the projected two-dimensional curve. Through a search algorithm, such as a k-nearest neighbor search algorithm, the point of the target two-dimensional slender interventional device is matched to the nearest projected two-dimensional curve point. Specifically, the target two-dimensional slender interventional device consists of a point set N, and the projected two-dimensional curve consists of a point set W. Then, the k-nearest neighbor search algorithm is used to find the nearest w points corresponding to each point in the point set N, where the w points are points in the point set W. Then, based on the final camera posture, the corresponding three-dimensional blood vessel center point coordinates are found through the w points. Finally, the color of the three-dimensional blood vessel center point coordinates is changed to the color of the slender interventional device. In this way, medical staff can quickly find the position of the slender interventional device through the color change, and can more accurately control the moving direction and depth of the interventional device, avoiding problems such as vascular damage or device retention caused by misoperation. In addition, the color-coded navigation method can reflect the position changes of interventional instruments in real time, provide medical staff with instant feedback information, and help them better identify the path and reduce surgical risks. Color coding makes the navigation process more intuitive and easy to understand, allowing medical staff to focus more on the surgery itself without being distracted by complex navigation operations, thus reducing surgical risks.
[0108] In one embodiment, the step of iteratively solving the final camera pose with the objective of minimizing the objective function comprises:
[0109] The final camera position is solved iteratively by a nonlinear least squares algorithm with the goal of minimizing the objective function.
[0110] In the embodiment of the present application, since the nonlinear least squares algorithm such as the Levenberg-Marquardt algorithm combines the advantages of the Gauss-Newton algorithm and the gradient descent, it can converge quickly and improve the accuracy of the solution. The nonlinear least squares algorithm is usually used for curve fitting problems, and it can find the minimum value of a function in a multidimensional space. Therefore, the present invention uses a nonlinear least squares algorithm to iteratively solve the final camera pose, and can quickly find the camera pose that best matches the observed data, that is, the final camera pose obtained by the final solution is more accurate, so that the position of the target two-dimensional slender interventional device mapped to the three-dimensional blood vessel is more accurate.
[0111] like Figure 2 As shown, in a second aspect, an embodiment of the present application provides a real-time navigation system for an interventional surgical robot, the system comprising:
[0112] CTA data acquisition module 1, used to acquire preoperative CTA data;
[0113] A first extraction module 2, used to extract three-dimensional blood vessels and three-dimensional blood vessel centerlines based on the CTA data;
[0114] Acquisition module 3, used to acquire two-dimensional images acquired by the DSA device during surgery;
[0115] A second extraction module 4, used to extract a target two-dimensional blood vessel centerline and a target two-dimensional elongated interventional device from the two-dimensional image acquired by the DSA device;
[0116] A final camera pose determination module 5, used to determine a final camera pose from a three-dimensional blood vessel to a two-dimensional blood vessel based on the three-dimensional blood vessel centerline and the target two-dimensional blood vessel centerline;
[0117] A mapping module 6 is used to map the target two-dimensional elongated interventional instrument to the three-dimensional blood vessel based on the final camera posture for navigation.
[0118] In one embodiment, the final camera pose determination module 5 includes:
[0119] The first traversal projection unit is used to set a minimum value, a maximum value and a step length for each variable in the projection parameters, and then continuously traverse all combination items according to the step length, and use each combination item to project the three-dimensional blood vessel centerline onto a two-dimensional plane to obtain a corresponding projected two-dimensional curve image;
[0120] A similarity calculation unit, used for respectively calculating the similarity between the target two-dimensional blood vessel centerline and each of the projected two-dimensional curve images during intraoperative registration, and taking the combination item corresponding to the maximum similarity as the initial camera pose;
[0121] The first iterative solution unit is used to perform iterative solution based on the initial camera pose to obtain a final camera pose.
[0122] In one embodiment, the steps of the final camera pose determination module 5 include:
[0123] A second traversal projection unit is used to set a minimum value, a maximum value and a step length for each variable in the projection parameters, and then continuously traverse all combination items according to the step length, and use each combination item to project the three-dimensional blood vessel centerline onto a two-dimensional plane to obtain a corresponding projected two-dimensional curve image;
[0124] A training unit, used to train a convolutional neural network using the projected two-dimensional curve image as input and the combination item as output, and output a trained model after training;
[0125] An input unit, used to input the target two-dimensional blood vessel centerline into the trained model to obtain a combination item corresponding to the target two-dimensional blood vessel centerline;
[0126] A first equivalent unit is used to use the combination item corresponding to the center line of the target two-dimensional blood vessel as an initial camera pose;
[0127] The second iterative solution unit is used to perform iterative solution based on the initial camera pose to obtain a final camera pose.
[0128] In one embodiment, the first iterative solution unit includes:
[0129] A search subunit, used for finding the nearest neighbor point of each point in the two-dimensional curve to be registered in the center line of the target two-dimensional blood vessel by using a search algorithm;
[0130] A construction subunit, used for constructing an objective function based on each point in the two-dimensional curve to be registered and the corresponding nearest neighbor point;
[0131] A solving subunit, configured to iteratively solve the final camera pose based on the initial camera pose with the objective of minimizing the objective function; wherein the two-dimensional curve to be registered is a projected two-dimensional curve corresponding to the camera pose during iteration;
[0132] The output subunit is used for, when the objective function converges, the camera pose corresponding to the minimum value of the objective function is the final camera pose.
[0133] In one embodiment, the second iterative solution unit includes:
[0134] A search subunit, used for finding the nearest neighbor point of each point in the two-dimensional curve to be registered in the center line of the target two-dimensional blood vessel by using a search algorithm;
[0135] A construction subunit, used for constructing an objective function based on each point in the two-dimensional curve to be registered and the corresponding nearest neighbor point;
[0136] A solving subunit, configured to iteratively solve the final camera pose based on the initial camera pose with the objective of minimizing the objective function; wherein the two-dimensional curve to be registered is a projected two-dimensional curve corresponding to the camera pose during iteration;
[0137] The output subunit is used for, when the objective function converges, the camera pose corresponding to the minimum value of the objective function is the final camera pose.
[0138] In one embodiment, the similarity calculation unit package is specifically used for:
[0139] The similarity between the target two-dimensional blood vessel centerline and each of the projected two-dimensional curve images is calculated according to the following formulas respectively;
[0140]
[0141] Where x represents the target 2D blood vessel centerline, y represents the projected 2D curve, and μ x is the average value of the target 2D blood vessel centerline; μ y is the variance of the projected two-dimensional curve; σ x is the variance of the target 2D vessel centerline, is the variance of the projected two-dimensional curve, σ xy is the covariance of x and y, and C1 and C2 are nonzero constants.
[0142] In one embodiment, the construction subunit is specifically used for:
[0143] The objective function is constructed according to the following formula:
[0144]
[0145] in, represents the i-th point of the two-dimensional curve to be registered; represents the nearest neighbor point of the i-th point in the two-dimensional curve to be registered in the center line of the target two-dimensional blood vessel; σ is the bandwidth parameter of the Gaussian kernel, and the bandwidth parameter of the Gaussian kernel adopts the maximum distance value between the two-dimensional points; k i is the value of the Gaussian kernel, which measures The similarity between two points; K is the objective function, and n is the total number of points of the two-dimensional curve to be registered.
[0146] In one embodiment, the mapping module includes:
[0147] An acquisition unit, which acquires a projected two-dimensional curve corresponding to the final camera posture;
[0148] A second equivalent unit is used to use the projected two-dimensional curve corresponding to the final camera posture as the projected two-dimensional curve;
[0149] A matching unit, configured to match the point of the target two-dimensional elongated interventional device to the closest projection two-dimensional curve point thereof by using a search algorithm;
[0150] A three-dimensional blood vessel center point coordinate determination unit, configured to determine the three-dimensional blood vessel center point coordinates based on the final camera position and through the closest projected two-dimensional curve point;
[0151] The rendering unit is used to change the color of the coordinates of the center point of the three-dimensional blood vessel into the color of the slender interventional instrument, so as to complete the mapping of the target two-dimensional slender interventional instrument to the three-dimensional blood vessel.
[0152] In one embodiment, the solving subunit is specifically used for:
[0153] The final camera position is solved iteratively by a nonlinear least squares algorithm with the goal of minimizing the objective function.
[0154] Embodiment three:
[0155] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, a real-time navigation method for an interventional surgical robot is implemented, comprising the following steps: acquiring preoperative CTA data; extracting three-dimensional blood vessels and three-dimensional blood vessel centerlines based on the CTA data; acquiring two-dimensional images acquired by a DSA device during the operation; extracting a target two-dimensional blood vessel centerline and a target two-dimensional slender interventional instrument from the two-dimensional images acquired by the DSA device; determining a final camera pose from a three-dimensional blood vessel to a two-dimensional blood vessel based on the three-dimensional blood vessel centerline and the target two-dimensional blood vessel centerline; and mapping the target two-dimensional slender interventional instrument to the three-dimensional blood vessel based on the final camera pose for navigation.
[0156] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided in this application and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.
[0157] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.
[0158] The above description is only a preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. A real-time navigation method for an interventional surgical robot, characterized in that: The method comprises: Obtain preoperative CTA data; Extracting three-dimensional blood vessels and three-dimensional blood vessel centerlines based on the CTA data; Acquire the two-dimensional images collected by the DSA device during surgery; Extracting a target two-dimensional blood vessel centerline and a target two-dimensional elongated interventional device from the two-dimensional image acquired by the DSA device; Determine a final camera pose from the three-dimensional blood vessel to the two-dimensional blood vessel based on the three-dimensional blood vessel centerline and the target two-dimensional blood vessel centerline; Based on the final camera pose, the target two-dimensional elongated interventional device is mapped to the three-dimensional blood vessel for navigation.
2. The real-time navigation method for an interventional surgical robot according to claim 1, characterized in that: The step of determining the final camera pose from the three-dimensional blood vessel to the two-dimensional blood vessel based on the three-dimensional blood vessel centerline and the target two-dimensional blood vessel centerline comprises: For each variable in the projection parameters, a minimum value, a maximum value and a step length are set, and then all combination items are continuously traversed according to the step length, and each combination item is used to project the three-dimensional blood vessel centerline onto a two-dimensional plane to obtain a corresponding projected two-dimensional curve image; During intraoperative registration, the similarity between the target two-dimensional blood vessel centerline and each of the projected two-dimensional curve images is calculated respectively, and the combination item corresponding to the maximum similarity is taken as the initial camera pose; An iterative solution is performed based on the initial camera pose to obtain a final camera pose.
3. The real-time navigation method for an interventional surgery robot according to claim 1, characterized in that: The step of determining the final camera pose from the three-dimensional blood vessel to the two-dimensional blood vessel based on the three-dimensional blood vessel centerline and the target two-dimensional blood vessel centerline comprises: For each variable in the projection parameters, a minimum value, a maximum value and a step length are set, and then all combination items are continuously traversed according to the step length, and each combination item is used to project the three-dimensional blood vessel centerline onto a two-dimensional plane to obtain a corresponding projected two-dimensional curve image; Using the projected two-dimensional curve image as input and the combined item as output to train a convolutional neural network, and after the training is completed, outputting the trained model; Inputting the target two-dimensional blood vessel centerline into the trained model to obtain a combination item corresponding to the target two-dimensional blood vessel centerline; Taking the combination item corresponding to the center line of the target two-dimensional blood vessel as the initial camera pose; An iterative solution is performed based on the initial camera pose to obtain a final camera pose.
4. The real-time navigation method for an interventional surgical robot according to claim 2 or 3, characterized in that: The step of performing iterative solution based on the initial camera pose to obtain the final camera pose comprises: Using a search algorithm to find the nearest neighbor point of each point in the two-dimensional curve to be registered in the center line of the target two-dimensional blood vessel; Constructing an objective function based on each point in the two-dimensional curve to be registered and the corresponding nearest neighbor point; Based on the initial camera pose, the final camera pose is iteratively solved with the goal of minimizing the objective function; wherein the two-dimensional curve to be registered is a projected two-dimensional curve corresponding to the camera pose during iteration; When the objective function converges, the camera pose corresponding to the minimum value of the objective function is the final camera pose.
5. The real-time navigation method for an interventional surgery robot according to claim 2, characterized in that: The step of respectively calculating the similarity between the target two-dimensional blood vessel centerline and each of the projected two-dimensional curve images comprises: The similarity between the target two-dimensional blood vessel centerline and each of the projected two-dimensional curve images is calculated according to the following formulas respectively; Where x represents the target 2D blood vessel centerline, y represents the projected 2D curve, and μ x is the average value of the target 2D blood vessel centerline; μ y is the variance of the projected two-dimensional curve; σ x is the variance of the target 2D vessel centerline, is the variance of the projected two-dimensional curve, σ xy is the covariance of x and y, and C1 and C2 are nonzero constants.
6. The real-time navigation method for an interventional surgery robot according to claim 4, characterized in that: The step of constructing an objective function based on each point in the two-dimensional curve to be registered and the corresponding nearest neighbor point comprises: The objective function is constructed according to the following formula: in, represents the i-th point of the two-dimensional curve to be registered; represents the nearest neighbor point of the i-th point in the two-dimensional curve to be registered in the center line of the target two-dimensional blood vessel; σ is the bandwidth parameter of the Gaussian kernel, and the bandwidth parameter of the Gaussian kernel adopts the maximum distance value between the two-dimensional points; k i is the value of the Gaussian kernel, which measures and The similarity between two points; K is the objective function, and n is the total number of points of the two-dimensional curve to be registered.
7. The real-time navigation method for an interventional surgery robot according to claim 4, characterized in that: The step of mapping the target two-dimensional elongated interventional device to the three-dimensional blood vessel for navigation based on the final camera pose comprises: Obtaining a projected two-dimensional curve corresponding to the final camera pose; The projected two-dimensional curve corresponding to the final camera posture is used as the projected two-dimensional curve; Using a search algorithm, matching the point of the target two-dimensional elongated interventional device to the closest projected two-dimensional curve point; Based on the final camera position, determining the coordinates of the center point of the three-dimensional blood vessel through the projected two-dimensional curve point closest to it; The color of the coordinates of the center point of the three-dimensional blood vessel is changed to the color of the slender interventional instrument, thereby completing the mapping of the target two-dimensional slender interventional instrument to the three-dimensional blood vessel.
8. The real-time navigation method for an interventional surgery robot according to claim 4, characterized in that: The step of iteratively solving the final camera pose with the objective of minimizing the objective function comprises: The final camera position is solved iteratively by a nonlinear least squares algorithm with the goal of minimizing the objective function.
9. A real-time navigation system for an interventional surgical robot, characterized in that: The system comprises: CTA data acquisition module, used to acquire preoperative CTA data; A first extraction module, configured to extract a three-dimensional blood vessel and a three-dimensional blood vessel centerline based on the CTA data; An acquisition module is used to obtain two-dimensional images collected by the DSA device during surgery; A second extraction module, used to extract a target two-dimensional blood vessel centerline and a target two-dimensional elongated interventional device from the two-dimensional image acquired by the DSA device; A final camera pose determination module, used to determine a final camera pose from a three-dimensional blood vessel to a two-dimensional blood vessel based on the three-dimensional blood vessel centerline and the target two-dimensional blood vessel centerline; A mapping module is used to map the target two-dimensional elongated interventional instrument to the three-dimensional blood vessel based on the final camera posture for navigation.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the real-time navigation method for an interventional surgical robot are implemented as claimed in any one of claims 1 to 8.
Citation Information
Patent Citations
3D-2D vascular elastic registration method based on CT and DSA
CN114419117A
Surgical operating system and method
CN115804652A
Vascular interventional operation navigation method and device based on multi-modal image fusion
CN116531092A
Method and apparatus for tracking catheter location using radiographic imaging
KR102396620B1
Cited By
Catheter positioning method and system for vascular intervention and storage medium
CN119867936A
Catheter Positioning Method, System and Storage Medium for Vascular Intervention
CN119867936B
