Real-time navigation method, system and storage medium for interventional surgical robots

Abstract

The application relates to the medical technology field and provides a real-time navigation method, system and storage medium for an interventional operation robot, wherein the method comprises the following steps: acquiring preoperative CTA data; extracting a three-dimensional blood vessel and a three-dimensional blood vessel center line based on the CTA data; acquiring a two-dimensional image collected by a DSA device during operation; extracting a target two-dimensional blood vessel center line and a target two-dimensional elongated interventional instrument from the two-dimensional image collected by the DSA device; determining a final camera pose of a three-dimensional blood vessel to a two-dimensional blood vessel based on the three-dimensional blood vessel center line and the target two-dimensional blood vessel center line; and mapping the target two-dimensional elongated interventional instrument to the three-dimensional blood vessel based on the final camera pose, so as to perform navigation. The application realizes 3D visual elongated interventional instrument navigation guidance, and doctors can directly see the position of the elongated interventional instrument in the three-dimensional blood vessel.

Description

Technical Field

[0001] This application relates to the field of medical technology, and in particular to a real-time navigation method, system, and storage medium for interventional surgical robots. Background Technology

[0002] In interventional surgery, existing vascular navigation solutions mainly rely on 2D and 3D vascular registration. Doctors determine the position of slender interventional instruments such as catheters and guidewires based on 2D images and use 3D information as a reference, but there is a lack of 3D vision for navigation guidance of slender interventional instruments. Summary of the Invention

[0003] To address the aforementioned technical problems, the purpose of this application is to provide a real-time navigation method, system, and storage medium for interventional surgical robots, aiming to solve the problem of navigation guidance for slender interventional instruments that currently lack 3D vision.

[0004] In a first aspect, embodiments of this application provide a real-time navigation method for an interventional surgical robot, comprising: Obtain preoperative CTA data; Based on the CTA data, three-dimensional blood vessels and three-dimensional blood vessel centerlines were extracted; Acquire two-dimensional images captured by the DSA device during the operation; The target two-dimensional blood vessel centerline and the target two-dimensional slender interventional device are extracted from the two-dimensional images acquired by the DSA device; The final camera pose from the three-dimensional blood vessel to the two-dimensional blood vessel is determined based on the three-dimensional blood vessel centerline and the target two-dimensional blood vessel centerline. Based on the final camera pose, the target two-dimensional slender interventional device is mapped onto the three-dimensional blood vessel for navigation.

[0005] Furthermore, the step of determining the final camera pose from the three-dimensional blood vessel to the two-dimensional blood vessel based on the three-dimensional blood vessel centerline and the target two-dimensional blood vessel centerline includes: For each variable in the projection parameters, a minimum value, a maximum value, and a step size are set. Then, according to the step size, all combination items are continuously traversed, and the three-dimensional blood vessel centerline is projected onto a two-dimensional plane using each combination item to obtain the corresponding projected two-dimensional curve image. During intraoperative registration, the similarity between the target two-dimensional vessel centerline and each projected two-dimensional curve image is calculated, and the combination with the highest similarity is taken as the initial camera pose. The final camera pose is obtained by iteratively solving based on the initial camera pose.

[0006] Furthermore, the step of determining the final camera pose from the three-dimensional blood vessel to the two-dimensional blood vessel based on the three-dimensional blood vessel centerline and the target two-dimensional blood vessel centerline includes: For each variable in the projection parameters, a minimum value, a maximum value, and a step size are set. Then, according to the step size, all combination items are continuously traversed, and the three-dimensional blood vessel centerline is projected onto a two-dimensional plane using each combination item to obtain the corresponding projected two-dimensional curve image. The projected two-dimensional curve image is used as input, and the combined terms are used as output to train a convolutional neural network. After training, the trained model is output. Input the target two-dimensional blood vessel centerline into the trained model to obtain the combination item corresponding to the target two-dimensional blood vessel centerline; Use the combination of items corresponding to the center line of the target two-dimensional blood vessel as the initial camera pose; The final camera pose is obtained by iteratively solving based on the initial camera pose.

[0007] Furthermore, the step of iteratively solving based on the initial camera pose to obtain the final camera pose includes: A search algorithm is used to find the nearest neighbor of each point in the two-dimensional curve to be registered in the centerline of the target two-dimensional blood vessel. A target function is constructed based on each point in the two-dimensional curve to be registered and its corresponding nearest neighbor. Based on the initial camera pose, the final camera pose is iteratively solved with the objective function being minimized; wherein, the two-dimensional curve to be registered is the projected two-dimensional curve corresponding to the camera pose during iteration; When the objective function converges, the camera pose corresponding to the minimum value of the objective function is the final camera pose.

[0008] Furthermore, the step of calculating the similarity between the target two-dimensional blood vessel centerline and each of the projected two-dimensional curve images includes: The similarity between the target two-dimensional blood vessel centerline and each of the projected two-dimensional curve images is calculated according to the following formulas;

[0009] in, Let y represent the centerline of the target two-dimensional blood vessel, and y represent the projected two-dimensional curve. It is the average value of the center line of the target two-dimensional blood vessel; It is the variance of the projected two-dimensional curve; It is the variance of the centerline of the target two-dimensional blood vessel. It is the variance of the projected two-dimensional curve. It is the covariance of x and y. and It is a non-zero constant.

[0010] Furthermore, the step of constructing the objective function based on each point in the two-dimensional curve to be registered and its corresponding nearest neighbor includes: Construct the objective function based on the following formula: ;

[0011] in, This represents the i-th point of the two-dimensional curve to be registered; This indicates that the i-th point in the two-dimensional curve to be registered is located at the center of the target two-dimensional blood vessel. The nearest neighbor in; It is the Gaussian kernel bandwidth parameter, which uses the maximum distance between two-dimensional points. It is the value of the Gaussian kernel, which measures... and The similarity between two points; K is the objective function, and n is the total number of points on the two-dimensional curve to be registered.

[0012] Furthermore, the step of mapping the target two-dimensional slender interventional device onto the three-dimensional blood vessel based on the final camera pose for navigation includes: Obtain the projected two-dimensional curve corresponding to the final camera pose; The projected two-dimensional curve corresponding to the final camera pose is used as the projected two-dimensional curve. A search algorithm is used to match the points of the target two-dimensional slender interventional device to the nearest projected two-dimensional curve point; Based on the final camera pose, the coordinates of the three-dimensional blood vessel center point are determined by the nearest projected two-dimensional curve point. The color of the three-dimensional blood vessel center point coordinates is changed to the color of the slender interventional device, thus completing the mapping of the target two-dimensional slender interventional device onto the three-dimensional blood vessel.

[0013] Furthermore, the step of iteratively solving for the final camera pose with the objective function as the goal includes: With the objective function being minimized, the final camera pose is solved iteratively using a nonlinear least squares algorithm.

[0014] Secondly, embodiments of this application provide a real-time navigation system for an interventional surgical robot, the system comprising: The CTA data acquisition module is used to acquire preoperative CTA data. The first extraction module is used to extract three-dimensional blood vessels and three-dimensional blood vessel centerlines based on the CTA data; The acquisition module is used to acquire two-dimensional images captured by the DSA device during surgery; The second extraction module is used to extract the target two-dimensional blood vessel centerline and the target two-dimensional slender interventional device from the two-dimensional image acquired by the DSA device. The final camera pose determination module is used to determine the final camera pose from the three-dimensional blood vessel to the two-dimensional blood vessel based on the three-dimensional blood vessel centerline and the target two-dimensional blood vessel centerline. The mapping module is used to map the target two-dimensional slender interventional device onto the three-dimensional blood vessel based on the final camera pose for navigation.

[0015] Thirdly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the real-time navigation method for interventional surgical robots described in any of the preceding claims.

[0016] The real-time navigation method for interventional surgical robots provided in this application includes: acquiring preoperative CTA data; extracting three-dimensional blood vessels and three-dimensional blood vessel centerlines based on the CTA data; acquiring two-dimensional images acquired by an intraoperative DSA device; extracting a target two-dimensional blood vessel centerline and a target two-dimensional slender interventional device from the two-dimensional images acquired by the DSA device; determining the final camera pose from the three-dimensional blood vessel centerline to the two-dimensional blood vessel based on the three-dimensional blood vessel centerline and the target two-dimensional blood vessel centerline; and mapping the target two-dimensional slender interventional device onto the three-dimensional blood vessel based on the final camera pose for navigation. In this way, 3D vision-based navigation guidance for slender interventional devices is achieved. Surgeons can directly see the position of the slender interventional device in the three-dimensional blood vessel. Compared to determining the position of the slender interventional device based on 2D images, using 3D information as a reference is more intuitive and avoids the need for the human eye to match the position of the 2D slender interventional device in the 3D blood vessel, thus avoiding human error and improving surgical safety. Attached Figure Description

[0017] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the real-time navigation method for an interventional surgical robot provided in an embodiment of this application; Figure 2This is a schematic diagram of the structure of a real-time navigation system for an interventional surgical robot provided in an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0020] Those skilled in the art will understand that, unless explicitly stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in the specification of this application means the presence of features, integers, steps, operations, elements, modules, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, modules, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any modules and all combinations of one or more associated listed items.

[0021] Those skilled in the art will understand that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0022] Please see Figure 1 This application provides a real-time navigation method for an interventional surgical robot, the method comprising: S1. Obtain preoperative CTA data.

[0023] In step S1, CTA stands for Computed Tomography Angiography, a method that uses computed tomography (CT) technology to visualize the cardiovascular system. CTA combines CT scanning and angiography to create non-invasive, high-resolution vascular images, clinically used to assess vascular lesions and diseases. Preoperative CTA examinations can obtain preoperative CTA data for the patient.

[0024] S2. Extract three-dimensional blood vessels and three-dimensional blood vessel centerlines based on the CTA data.

[0025] In step S2, the CTA data obtained in step S1 is preprocessed before surgery, including CTA three-dimensional reconstruction, obtaining three-dimensional blood vessels, segmenting the three-dimensional blood vessels, and extracting the three-dimensional blood vessel centerline.

[0026] S3. Acquire two-dimensional images captured by the DSA device during the operation.

[0027] In step S3, DSA stands for Digital Subtraction Angiography, a type of X-ray imaging system that combines conventional angiography with computer image processing technology. Digital subtraction angiography aims to remove interfering images overlapping with blood vessels to obtain clear vascular images. During interventional procedures, the DSA device acquires images of the patient, thus obtaining two-dimensional images of the patient captured by the DSA device during the procedure.

[0028] S4. Extract the target two-dimensional blood vessel centerline and the target two-dimensional slender interventional device from the two-dimensional image acquired by the DSA device.

[0029] In step S4, it should be understood that interventional surgery is a minimally invasive surgical procedure that uses slender interventional instruments such as guidewires and catheters to diagnose and treat diseases under the guidance of imaging equipment. Therefore, the two-dimensional images acquired by the intraoperative DSA equipment in step S3 include images of blood vessels and slender interventional instruments. After obtaining the two-dimensional images acquired by the DSA equipment, the blood vessels and slender interventional instruments are segmented separately, and the center lines of the blood vessels and the slender interventional instruments are extracted. The extracted center line of the blood vessels is the target two-dimensional blood vessel center line, and the extracted slender interventional instruments are the target two-dimensional slender interventional instruments.

[0030] S5. Determine the final camera pose from the three-dimensional blood vessel to the two-dimensional blood vessel based on the three-dimensional blood vessel centerline and the target two-dimensional blood vessel centerline.

[0031] In 3D and 2D registration of blood vessels, the main problem to be solved is the Perspective-n-Point (PnP) problem, which involves finding a set of camera position and pose parameters in world coordinates (referred to as camera pose) such that the 2D image projected from the 3D blood vessel coincides with the feature points of the 2D blood vessel image acquired by the DSA device. Let the image directly acquired by the DSA device be... This image is a fixed, unchanging reference image; the three-dimensional blood vessel V is... The two-dimensional image obtained after projection under the parameters is The projection mapping function from 3D to 2D is: subscript Represents the parameters corresponding to the camera pose, where, where , Let S represent the optimal parameters, and let S be the similarity metric function used to evaluate the similarity between two sets of data. , Therefore, this method aims to find a set of parameters P=[Rx; Ry; Rz; Tx; Ty; Tz] within a spatial range to make the projected 2D points as similar as possible to the target 2D points, where Tx, Ty, and Tz represent the translation components along the x, y, and z axes, and Rx, Ry, and Rz represent the rotation components along the x, y, and z axes. Specifically, this invention determines the final camera pose of the 3D to 2D blood vessel based on the 3D blood vessel centerline and the target 2D blood vessel centerline. The blood vessel centerline, as an important feature of the blood vessel structure, can well represent the overall shape and direction of the blood vessel. Furthermore, the blood vessel centerline is relatively clear and easy to extract in both 3D and 2D images. Therefore, using the blood vessel centerline as the registration feature point can simplify the complexity of the registration process and improve the accuracy and robustness of the registration. In addition, compared to using the entire blood vessel's 3D and 2D points for registration, using only the centerline can significantly reduce the amount of data that needs to be processed. This not only reduces computational complexity but also shortens the registration time.

[0032] S6. Based on the final camera pose, map the target two-dimensional slender interventional device onto the three-dimensional blood vessel for navigation.

[0033] In step S6, after obtaining the final camera pose, the target two-dimensional slender interventional device can be mapped onto a three-dimensional blood vessel using this final camera pose and displayed on a display device. In this way, the doctor can directly see the position of the slender interventional device in the three-dimensional blood vessel. Compared with judging the position of the slender interventional device based on 2D images, the method of using 3D information as a reference is more intuitive. At the same time, it avoids the human eye matching the position of the 2D slender interventional device in the 3D blood vessel, avoids human error, and thus improves the safety of the operation.

[0034] The real-time navigation method for interventional surgical robots provided in this application includes: acquiring preoperative CTA data; extracting three-dimensional blood vessels and three-dimensional blood vessel centerlines based on the CTA data; acquiring two-dimensional images acquired by an intraoperative DSA device; extracting a target two-dimensional blood vessel centerline and a target two-dimensional slender interventional device from the two-dimensional images acquired by the DSA device; determining the final camera pose from the three-dimensional blood vessel centerline to the two-dimensional blood vessel based on the three-dimensional blood vessel centerline and the target two-dimensional blood vessel centerline; and mapping the target two-dimensional slender interventional device onto the three-dimensional blood vessel based on the final camera pose for navigation. In this way, 3D vision-based navigation guidance for slender interventional devices is achieved. Surgeons can directly see the position of the slender interventional device in the three-dimensional blood vessel. Compared to determining the position of the slender interventional device based on 2D images, using 3D information as a reference is more intuitive and avoids the need for the human eye to match the position of the 2D slender interventional device in the 3D blood vessel, thus avoiding human error and improving surgical safety.

[0035] In one embodiment, the step of determining the final camera pose from the three-dimensional blood vessel to the two-dimensional blood vessel based on the three-dimensional blood vessel centerline and the target two-dimensional blood vessel centerline includes: For each variable in the projection parameters, a minimum value, a maximum value, and a step size are set. Then, according to the step size, all combination items are continuously traversed, and the three-dimensional blood vessel centerline is projected onto a two-dimensional plane using each combination item to obtain the corresponding projected two-dimensional curve image. During intraoperative registration, the similarity between the target two-dimensional vessel centerline and each projected two-dimensional curve image is calculated, and the combination with the highest similarity is taken as the initial camera pose. The final camera pose is obtained by iteratively solving based on the initial camera pose.

[0036] In this embodiment, when solving the PNP problem, an initial value is usually required to obtain a more accurate result through iterative solution during the optimization process. The choice of initial value is crucial for the stability of the solution process and the accuracy of the final result in the PNP problem. An inappropriate initial value may cause the solution process to get stuck in a local optimum or even fail to converge to the correct solution. To select a suitable initial value, this application employs a spatial traversal approach. For each variable within the projection parameters (which are essentially camera pose parameters, including Tx, Ty, Tz, Rx, Ry, Rz), a minimum value, a maximum value, and a step size are set. Then, based on the step size, all combination terms are iterated continuously, and each combination term is used to project all three-dimensional blood vessel centerlines onto a two-dimensional plane, obtaining the corresponding projected two-dimensional curve images. All projected two-dimensional curve images are then recorded and saved. Then, during intraoperative registration, the similarity between the target two-dimensional vessel centerline (which should be understood as an image) and each projected two-dimensional curve image is calculated. The combination with the highest similarity is then taken as the initial value for the PNP algorithm. This initial value serves as the initial camera pose. Subsequent iterative solutions based on the PNP algorithm and the initial camera pose yield the final camera pose. This embodiment selects the initial value through spatial traversal, which can cover different regions of the parameter space as broadly as possible. Compared to randomly selecting initial values, this method is more likely to approach or discover the global optimum, which helps to converge to the final solution faster in subsequent iterative solutions and improves the accuracy of the solution. It should be understood that the final solution referred to here is the final camera pose.

[0037] In one embodiment, the step of determining the final camera pose from the three-dimensional blood vessel to the two-dimensional blood vessel based on the three-dimensional blood vessel centerline and the target two-dimensional blood vessel centerline includes: For each variable in the projection parameters, a minimum value, a maximum value, and a step size are set. Then, according to the step size, all combination items are continuously traversed, and the three-dimensional blood vessel centerline is projected onto a two-dimensional plane using each combination item to obtain the corresponding projected two-dimensional curve image. The projected two-dimensional curve image is used as input, and the combined terms are used as output to train a convolutional neural network. After training, the trained model is output. Input the target two-dimensional blood vessel centerline into the trained model to obtain the combination item corresponding to the target two-dimensional blood vessel centerline; The combination of items corresponding to the center line of the target two-dimensional blood vessel is used as the initial camera pose. The final camera pose is obtained by iteratively solving based on the initial camera pose.

[0038] In this embodiment, when solving the PNP problem, an initial value is typically required to obtain a more accurate result through iterative optimization. The choice of initial value is crucial for the stability of the solution process and the accuracy of the final result in the PNP problem. An inappropriate initial value may cause the solution process to get stuck in a local optimum or even fail to converge to the correct solution. To select a suitable initial value, this application employs a machine learning method. First, training samples are generated. Specifically, for each variable within the projection parameters, a minimum value, a maximum value, and a step size are set. Then, based on the step size, all combination terms (i.e., a set of projection parameters) are iterated over, and each combination term is used to project all three-dimensional blood vessel centerlines onto a two-dimensional plane, obtaining the corresponding projected two-dimensional curve images. All projected two-dimensional curve images are recorded and saved. Then, each combination term and its corresponding projected two-dimensional curve image are used as a set of training data, thus obtaining the training data. Subsequently, using the projected two-dimensional curve image as input and the combined terms as output, a convolutional neural network is trained. After training, the trained model is output. The target two-dimensional blood vessel centerline is input into the trained model, and the model's output can be used as the initial camera pose. The final camera pose can then be obtained by iteratively solving the problem using the PNP algorithm and the initial camera pose. This embodiment of the application utilizes machine learning to learn the complex mapping relationship between projection parameters and two-dimensional curve images from training data. This data-driven method can capture subtle features that are difficult to capture using traditional methods, resulting in better predicted projection parameters for the target two-dimensional blood vessel centerline, i.e., better initial pose parameters.

[0039] In one embodiment, the step of iteratively solving based on the initial camera pose to obtain the final camera pose includes: A search algorithm is used to find the nearest neighbor of each point in the two-dimensional curve to be registered in the centerline of the target two-dimensional blood vessel. A target function is constructed based on each point in the two-dimensional curve to be registered and its corresponding nearest neighbor. Based on the initial camera pose, the final camera pose is iteratively solved with the objective function being minimized; wherein, the two-dimensional curve to be registered is the projected two-dimensional curve corresponding to the camera pose during iteration; When the objective function converges, the camera pose corresponding to the minimum value of the objective function is the final camera pose.

[0040] In this embodiment, the iteration begins from the initial camera pose. In each iteration, the 3D blood vessel is projected onto a 2D plane based on the current camera pose, resulting in a new 2D curve to be registered. A search algorithm is used to find the nearest neighbor of each point in the new 2D curve to be registered on the centerline of the target 2D blood vessel. The objective function value is updated based on each point in the new 2D curve to be registered and its corresponding nearest neighbor. Based on the gradient information of the objective function, the update amount of the camera pose is calculated using an optimization algorithm. The camera pose is updated, and preparation is made for the next iteration. When the value of the objective function falls below a preset threshold, or the update amount of multiple consecutive iterations is less than a certain threshold, the algorithm is considered to have converged, and the camera pose corresponding to the minimum value of the objective function at this point is the final camera pose. Through the iterative solution process, the camera pose parameters can be gradually optimized, maximizing the similarity (or minimizing the error) between the 2D curve projected by the camera in the current pose and the centerline of the target 2D blood vessel. This optimization process can significantly improve the accuracy of camera pose estimation, thereby improving the accuracy of mapping the target 2D slender interventional device to the 3D blood vessel.

[0041] Furthermore, the search algorithm can specifically be a k-nearest neighbors search algorithm, and the objective function is specifically constructed based on a Gaussian kernel function, which is used to calculate the similarity between two points. It should be noted that the number of points on the center line of the target two-dimensional blood vessel and the number of points on the two-dimensional curve after the three-dimensional blood vessel projection do not match. Additionally, the actual displayed image may only be a local part of the three-dimensional projection. This invention, using a k-nearest neighbors search algorithm combined with a Gaussian kernel function, can not only handle the problem of mismatched data point numbers but also effectively cope with local data loss or noise interference.

[0042] In one embodiment, the step of calculating the similarity between the target two-dimensional blood vessel centerline and each of the projected two-dimensional curve images includes: The similarity between the target two-dimensional blood vessel centerline and each of the projected two-dimensional curve images is calculated according to the following formulas;

[0043] in, Let y represent the centerline of the target two-dimensional blood vessel, and y represent the projected two-dimensional curve. It is the average value of the center line of the target two-dimensional blood vessel; It is the variance of the projected two-dimensional curve; It is the variance of the centerline of the target two-dimensional blood vessel. It is the variance of the projected two-dimensional curve. It is the covariance of x and y. and It is a non-zero constant.

[0044] In this embodiment, it should be understood that the target two-dimensional blood vessel centerline is displayed in image form, and the image is composed of pixels. Calculating the similarity between the target two-dimensional blood vessel centerline and the projected two-dimensional curve image is actually calculating the similarity between the two images. In this embodiment, the method for calculating the similarity between two images is preferably to determine the similarity between the two images by calculating the Structural Similarity Index (SSIM). SSIM is an index used to measure the similarity between two images and is commonly used in image processing and computer vision. Compared to simple pixel differences, SSIM focuses more on the structural information of the image and is a method that is closer to the human visual system. The SSIM index is used to measure the similarity between two images. It considers the similarity of images from three aspects: brightness, contrast, and structure. Its value range is [-1, 1], where 1 represents an identical image. Because SSIM comprehensively considers brightness, contrast, and structural information, it determines the similarity between the target two-dimensional blood vessel centerline and each projected two-dimensional curve image. This allows for a more refined assessment of the similarity between the blood vessel centerline and each projected two-dimensional curve image. Compared to simple pixel differences, it can find images that are more similar to the target two-dimensional blood vessel centerline, thus finding a better initial camera pose. Consequently, the final camera pose determined by iterative solving based on the initial camera pose is more accurate, and the position of the target two-dimensional slender interventional device mapped into the three-dimensional blood vessel is more precise.

[0045] In one embodiment, the step of constructing an objective function based on each point in the two-dimensional curve to be registered and its corresponding nearest neighbor includes: Construct the objective function based on the following formula: ;

[0046] in, This represents the i-th point of the two-dimensional curve to be registered; This represents the nearest neighbor of the i-th point in the two-dimensional curve to be registered in the centerline of the target two-dimensional blood vessel; It is the Gaussian kernel bandwidth parameter, which uses the maximum distance between two-dimensional points. It is the value of the Gaussian kernel, which measures... and The similarity between two points; K is the objective function, and n is the total number of points on the two-dimensional curve to be registered.

[0047] In the embodiments of this application, express and Euclidean distance between two points This is the Gaussian kernel bandwidth parameter, which controls the width of the Gaussian kernel function. The formula for calculating the maximum distance between two-dimensional points is: ;in, Let C be the maximum distance between two-dimensional points, where C is the Curve_Reg matrix of size m×n, where m is the number of rows and n is the number of columns, and it records the points on the two-dimensional curve to be registered. T is the Target matrix of size p×n, where p is the number of rows and n is the number of columns, and it records the points on the center line of the target two-dimensional blood vessel.

[0048] In one embodiment, the step of mapping the target two-dimensional elongated interventional device onto the three-dimensional blood vessel based on the final camera pose for navigation includes: Obtain the projected two-dimensional curve corresponding to the final camera pose; The projected two-dimensional curve corresponding to the final camera pose is used as the projected two-dimensional curve. A search algorithm is used to match the points of the target two-dimensional slender interventional device to the nearest projected two-dimensional curve point; Based on the final camera pose, the coordinates of the three-dimensional blood vessel center point are determined by the nearest projected two-dimensional curve point. The color of the three-dimensional blood vessel center point coordinates is changed to the color of the slender interventional device, thus completing the mapping of the target two-dimensional slender interventional device onto the three-dimensional blood vessel.

[0049] In this embodiment, the projected two-dimensional curve corresponding to the final camera pose refers to the image obtained by projecting the three-dimensional vessel centerline onto a two-dimensional plane using the final camera pose. For ease of description, the projected two-dimensional curve corresponding to the final camera pose is referred to as the projected two-dimensional curve. A search algorithm, such as the k-nearest neighbor search algorithm, is used to match the points of the target two-dimensional slender interventional device to its nearest projected two-dimensional curve point. Specifically, the target two-dimensional slender interventional device consists of a point set N, and the projected two-dimensional curve consists of a point set W. The k-nearest neighbor search algorithm is then used to find the nearest w point for each point in point set N, where w point is a point in point set W. Then, based on the final camera pose, the coordinates of the corresponding three-dimensional vessel center point are found through the w point. Finally, the color of the three-dimensional vessel center point coordinates is changed to the color of the slender interventional device. In this way, medical personnel can quickly locate the position of the slender interventional device through the color change, allowing for more accurate control of the device's movement direction and depth, avoiding problems such as vascular damage or device retention caused by misoperation. Furthermore, color-coded navigation can reflect the real-time positional changes of interventional instruments, providing immediate feedback to medical staff. This helps them better identify the path and reduce surgical risks. Color coding makes the navigation process more intuitive and easier to understand, allowing medical staff to focus more on the surgery itself without being distracted by complex navigation operations, thus reducing surgical risks.

[0050] In one embodiment, the step of iteratively solving for the final camera pose with the objective function as the goal includes: With the objective function being minimized, the final camera pose is solved iteratively using a nonlinear least squares algorithm.

[0051] In this embodiment, nonlinear least squares algorithms such as the Levenberg-Marquardt algorithm combine the advantages of the Gauss-Newton algorithm and gradient descent, thus enabling rapid convergence and improved solution accuracy. Nonlinear least squares algorithms are commonly used for curve fitting problems, finding the minimum value of a function in a multidimensional space. Therefore, this invention employs a nonlinear least squares algorithm to iteratively solve for the final camera pose, quickly finding the camera pose that best matches the observation data. This results in a more accurate final camera pose, leading to a more precise mapping of the target two-dimensional slender interventional device to the three-dimensional blood vessel.

[0052] like Figure 2 As shown, in a second aspect, embodiments of this application provide a real-time navigation system for an interventional surgical robot, the system comprising: CTA data acquisition module 1 is used to acquire preoperative CTA data; The first extraction module 2 is used to extract three-dimensional blood vessels and three-dimensional blood vessel centerlines based on the CTA data; Acquisition module 3 is used to acquire two-dimensional images captured by the DSA device during the operation; The second extraction module 4 is used to extract the target two-dimensional blood vessel centerline and the target two-dimensional slender interventional device from the two-dimensional image acquired by the DSA device. The final camera pose determination module 5 is used to determine the final camera pose from the three-dimensional blood vessel to the two-dimensional blood vessel based on the three-dimensional blood vessel centerline and the target two-dimensional blood vessel centerline. The mapping module 6 is used to map the target two-dimensional slender interventional device onto the three-dimensional blood vessel based on the final camera pose for navigation.

[0053] In one embodiment, the final camera pose determination module 5 includes: The first traversal projection unit is used to set the minimum value, maximum value and step size for each variable in the projection parameters. Then, according to the step size, it continuously traverses all combination items and uses each combination item to project the three-dimensional blood vessel centerline onto the two-dimensional plane to obtain the corresponding projected two-dimensional curve image. The similarity calculation unit is used to calculate the similarity between the target two-dimensional blood vessel centerline and each of the projected two-dimensional curve images during intraoperative registration, and take the combination corresponding to the maximum similarity as the initial camera pose. The first iterative solution unit is used to iteratively solve the initial camera pose to obtain the final camera pose.

[0054] In one embodiment, the steps of the final camera pose determination module 5 include: The second traversal projection unit is used to set a minimum value, a maximum value, and a step size for each variable in the projection parameters. Then, based on the step size, it continuously traverses all combination items and uses each combination item to project the three-dimensional blood vessel centerline onto a two-dimensional plane to obtain the corresponding projected two-dimensional curve image. The training unit is used to train a convolutional neural network with the projected two-dimensional curve image as input and the combined terms as output, and outputs the trained model after training is completed. An input unit is used to input the target two-dimensional blood vessel centerline into the trained model to obtain the combination item corresponding to the target two-dimensional blood vessel centerline. The first equivalent unit is used to take the combination item corresponding to the center line of the target two-dimensional blood vessel as the initial camera pose. The second iterative solution unit is used to iteratively solve the initial camera pose to obtain the final camera pose.

[0055] In one embodiment, the first iterative solution unit includes: The search subunit is used to find the nearest neighbor of each point in the two-dimensional curve to be registered in the centerline of the target two-dimensional blood vessel using a search algorithm. Construct sub-units to build objective functions based on each point in the two-dimensional curve to be registered and its corresponding nearest neighbor; The solution subunit is used to iteratively solve the final camera pose based on the initial camera pose, with the objective function being minimized; wherein, the two-dimensional curve to be registered is the projected two-dimensional curve corresponding to the camera pose during iteration; The output sub-unit is used to determine the final camera pose when the objective function converges, based on the minimum value of the objective function.

[0056] In one embodiment, the second iterative solution unit includes: The search subunit is used to find the nearest neighbor of each point in the two-dimensional curve to be registered in the centerline of the target two-dimensional blood vessel using a search algorithm. Construct sub-units to build objective functions based on each point in the two-dimensional curve to be registered and its corresponding nearest neighbor; The solution subunit is used to iteratively solve the final camera pose based on the initial camera pose, with the objective function being minimized; wherein, the two-dimensional curve to be registered is the projected two-dimensional curve corresponding to the camera pose during iteration; The output sub-unit is used to determine the final camera pose when the objective function converges, based on the minimum value of the objective function.

[0057] In one embodiment, the similarity calculation unit package is specifically used for: The similarity between the target two-dimensional blood vessel centerline and each of the projected two-dimensional curve images is calculated according to the following formulas;

[0058] in, Let y represent the centerline of the target two-dimensional blood vessel, and y represent the projected two-dimensional curve. It is the average value of the center line of the target two-dimensional blood vessel; It is the variance of the projected two-dimensional curve; It is the variance of the centerline of the target two-dimensional blood vessel. It is the variance of the projected two-dimensional curve. It is the covariance of x and y. and It is a non-zero constant.

[0059] In one embodiment, the building subunit is specifically used for: Construct the objective function based on the following formula: ;

[0060] in, This represents the i-th point of the two-dimensional curve to be registered; This represents the nearest neighbor of the i-th point in the two-dimensional curve to be registered in the centerline of the target two-dimensional blood vessel; It is the Gaussian kernel bandwidth parameter, which uses the maximum distance between two-dimensional points. It is the value of the Gaussian kernel, which measures... and The similarity between two points; K is the objective function, and n is the total number of points on the two-dimensional curve to be registered.

[0061] In one embodiment, the mapping module includes: The acquisition unit acquires the projected two-dimensional curve corresponding to the final camera pose; The second equivalent unit is used to take the projected two-dimensional curve corresponding to the final camera pose as the projected two-dimensional curve. The matching unit is used to match the points of the target two-dimensional slender interventional device to the nearest projected two-dimensional curve point using a search algorithm; The three-dimensional blood vessel center point coordinate determination unit is used to determine the coordinates of the three-dimensional blood vessel center point based on the final camera pose and through the nearest projected two-dimensional curve point. The rendering unit is used to change the color of the three-dimensional blood vessel center point coordinates to the color of the slender interventional device, thereby completing the mapping of the target two-dimensional slender interventional device onto the three-dimensional blood vessel.

[0062] In one embodiment, the solving subunit is specifically used for: With the objective function being minimized, the final camera pose is solved iteratively using a nonlinear least squares algorithm.

[0063] Example 3: One embodiment of this application also provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a real-time navigation method for an interventional surgical robot, comprising the following steps: acquiring preoperative CTA data; extracting three-dimensional blood vessels and three-dimensional blood vessel centerlines based on the CTA data; acquiring two-dimensional images acquired by an intraoperative DSA device; extracting a target two-dimensional blood vessel centerline and a target two-dimensional slender interventional device from the two-dimensional images acquired by the DSA device; determining the final camera pose from the three-dimensional blood vessel centerline to the two-dimensional blood vessel based on the three-dimensional blood vessel centerline and the target two-dimensional blood vessel centerline; and mapping the target two-dimensional slender interventional device onto the three-dimensional blood vessel based on the final camera pose for navigation.

[0064] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in this application and in the embodiments can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual-speed SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0065] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0066] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A real-time navigation system for interventional surgical robots, characterized in that, The system includes: The CTA data acquisition module is used to acquire preoperative CTA data. The first extraction module is used to extract three-dimensional blood vessels and three-dimensional blood vessel centerlines based on the CTA data; The acquisition module is used to acquire two-dimensional images captured by the DSA device during surgery; The second extraction module is used to extract the target two-dimensional blood vessel centerline and the target two-dimensional slender interventional device from the two-dimensional image acquired by the DSA device. The final camera pose determination module is used to determine the final camera pose from the three-dimensional blood vessel to the two-dimensional blood vessel based on the three-dimensional blood vessel centerline and the target two-dimensional blood vessel centerline. The mapping module is used to: obtain the projected two-dimensional curve corresponding to the final camera pose; use the projected two-dimensional curve corresponding to the final camera pose as the projected two-dimensional curve; use a search algorithm to match the points of the target two-dimensional slender interventional device to the nearest projected two-dimensional curve point; determine the coordinates of the three-dimensional blood vessel center point based on the final camera pose and the nearest projected two-dimensional curve point; and change the color of the three-dimensional blood vessel center point coordinates to the color of the slender interventional device, thus completing the mapping of the target two-dimensional slender interventional device to the three-dimensional blood vessel.

2. The real-time navigation system for interventional surgical robots according to claim 1, characterized in that, The final camera pose determination module includes: The first traversal projection unit is used to set the minimum value, maximum value and step size for each variable in the projection parameters. Then, according to the step size, it continuously traverses all combination items and uses each combination item to project the three-dimensional blood vessel centerline onto the two-dimensional plane to obtain the corresponding projected two-dimensional curve image. The similarity calculation unit is used to calculate the similarity between the target two-dimensional blood vessel centerline and each of the projected two-dimensional curve images during intraoperative registration, and take the combination corresponding to the maximum similarity as the initial camera pose. The first iterative solution unit is used to iteratively solve the initial camera pose to obtain the final camera pose.

3. The real-time navigation system for interventional surgical robots according to claim 1, characterized in that, The final camera pose determination module includes: The second traversal projection unit is used to set a minimum value, a maximum value, and a step size for each variable in the projection parameters. Then, based on the step size, it continuously traverses all combination items and uses each combination item to project the three-dimensional blood vessel centerline onto a two-dimensional plane to obtain the corresponding projected two-dimensional curve image. The training unit is used to train a convolutional neural network with the projected two-dimensional curve image as input and the combined terms as output, and outputs the trained model after training is completed. An input unit is used to input the target two-dimensional blood vessel centerline into the trained model to obtain the combination item corresponding to the target two-dimensional blood vessel centerline. The first equivalent unit is used to take the combination item corresponding to the center line of the target two-dimensional blood vessel as the initial camera pose. The second iterative solution unit is used to iteratively solve the initial camera pose to obtain the final camera pose.

4. The real-time navigation system for interventional surgical robots according to claim 2, characterized in that, The first iterative solution unit includes: The search subunit is used to find the nearest neighbor of each point in the two-dimensional curve to be registered in the centerline of the target two-dimensional blood vessel using a search algorithm. Construct sub-units to build an objective function based on each point in the two-dimensional curve to be registered and its corresponding nearest neighbor; The solution subunit is used to iteratively solve the final camera pose based on the initial camera pose, with the objective function being minimized; wherein, the two-dimensional curve to be registered is the projected two-dimensional curve corresponding to the camera pose during iteration; The output sub-unit is used to determine the final camera pose when the objective function converges, based on the minimum value of the objective function.

5. The real-time navigation system for interventional surgical robots according to claim 2, characterized in that, The similarity calculation unit is specifically used for: The similarity between the target two-dimensional blood vessel centerline and each of the projected two-dimensional curve images is calculated according to the following formulas; in, Let y represent the centerline of the target two-dimensional blood vessel, and y represent the projected two-dimensional curve. It is the average value of the center line of the target two-dimensional blood vessel; It is the average value of the projected two-dimensional curve; It is the variance of the centerline of the target two-dimensional blood vessel. It is the variance of the projected two-dimensional curve. It is the covariance of x and y. and It is a non-zero constant.

6. The real-time navigation system for interventional surgical robots according to claim 4, characterized in that, The construction subunit is specifically used for: Construct the objective function based on the following formula: ； in, This represents the i-th point of the two-dimensional curve to be registered; This represents the nearest neighbor of the i-th point in the two-dimensional curve to be registered in the centerline of the target two-dimensional blood vessel; It is the Gaussian kernel bandwidth parameter, which uses the maximum distance between two-dimensional points. It is the value of the Gaussian kernel, which measures... and The similarity between two points; K is the objective function, and n is the total number of points on the two-dimensional curve to be registered.

7. The real-time navigation system for interventional surgical robots according to claim 4, characterized in that, The solution subunit is specifically used for: With the objective function being minimized, the final camera pose is solved iteratively using a nonlinear least squares algorithm.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements a real-time navigation method for an interventional surgical robot, including: Obtain preoperative CTA data; Based on the CTA data, three-dimensional blood vessels and three-dimensional blood vessel centerlines were extracted; Acquire two-dimensional images captured by the DSA device during the operation; The target two-dimensional blood vessel centerline and the target two-dimensional slender interventional device are extracted from the two-dimensional images acquired by the DSA device; The final camera pose from the three-dimensional blood vessel to the two-dimensional blood vessel is determined based on the three-dimensional blood vessel centerline and the target two-dimensional blood vessel centerline. Based on the final camera pose, a projected two-dimensional curve corresponding to the final camera pose is obtained; the projected two-dimensional curve corresponding to the final camera pose is used as the projected two-dimensional curve; a search algorithm is used to match the points of the target two-dimensional slender interventional device to the nearest projected two-dimensional curve point; based on the final camera pose, the coordinates of the three-dimensional blood vessel center point are determined through the nearest projected two-dimensional curve point; the color of the three-dimensional blood vessel center point coordinates is changed to the color of the slender interventional device, thus completing the mapping of the target two-dimensional slender interventional device to the three-dimensional blood vessel.

9. The computer-readable storage medium according to claim 8, characterized in that, The process of determining the final camera pose from the three-dimensional blood vessel to the two-dimensional blood vessel based on the three-dimensional blood vessel centerline and the target two-dimensional blood vessel centerline includes: For each variable in the projection parameters, a minimum value, a maximum value, and a step size are set. Then, according to the step size, all combination items are continuously traversed, and the three-dimensional blood vessel centerline is projected onto a two-dimensional plane using each combination item to obtain the corresponding projected two-dimensional curve image. During intraoperative registration, the similarity between the target two-dimensional vessel centerline and each projected two-dimensional curve image is calculated, and the combination with the highest similarity is taken as the initial camera pose. The final camera pose is obtained by iteratively solving based on the initial camera pose.

10. The computer-readable storage medium according to claim 8, characterized in that, The process of determining the final camera pose from the three-dimensional blood vessel to the two-dimensional blood vessel based on the three-dimensional blood vessel centerline and the target two-dimensional blood vessel centerline includes: For each variable in the projection parameters, a minimum value, a maximum value, and a step size are set. Then, according to the step size, all combination items are continuously traversed, and the three-dimensional blood vessel centerline is projected onto a two-dimensional plane using each combination item to obtain the corresponding projected two-dimensional curve image. The projected two-dimensional curve image is used as input, and the combined terms are used as output to train a convolutional neural network. After training, the trained model is output. Input the target two-dimensional blood vessel centerline into the trained model to obtain the combination item corresponding to the target two-dimensional blood vessel centerline; The combination of items corresponding to the center line of the target two-dimensional blood vessel is used as the initial camera pose; The final camera pose is obtained by iteratively solving based on the initial camera pose.

11. The computer-readable storage medium according to claim 9, characterized in that, The iterative solution based on the initial camera pose to obtain the final camera pose includes: A search algorithm is used to find the nearest neighbor of each point in the two-dimensional curve to be registered in the centerline of the target two-dimensional blood vessel. A target function is constructed based on each point in the two-dimensional curve to be registered and its corresponding nearest neighbor. Based on the initial camera pose, the final camera pose is iteratively solved with the objective function being minimized; wherein, the two-dimensional curve to be registered is the projected two-dimensional curve corresponding to the camera pose during iteration; When the objective function converges, the camera pose corresponding to the minimum value of the objective function is the final camera pose.

12. The computer-readable storage medium according to claim 9, characterized in that, During intraoperative registration, the similarity between the target two-dimensional vessel centerline and each projected two-dimensional curve image is calculated, and the combination with the highest similarity is taken as the initial camera pose, including: The similarity between the target two-dimensional blood vessel centerline and each of the projected two-dimensional curve images is calculated according to the following formulas; in, Let y represent the centerline of the target two-dimensional blood vessel, and y represent the projected two-dimensional curve. It is the average value of the center line of the target two-dimensional blood vessel; It is the average value of the projected two-dimensional curve; It is the variance of the centerline of the target two-dimensional blood vessel. It is the variance of the projected two-dimensional curve. It is the covariance of x and y. and It is a non-zero constant.

13. The computer-readable storage medium according to claim 11, characterized in that, The construction of the objective function based on each point in the two-dimensional curve to be registered and its corresponding nearest neighbor includes: Construct the objective function based on the following formula: ； in, This represents the i-th point of the two-dimensional curve to be registered; This represents the nearest neighbor of the i-th point in the two-dimensional curve to be registered in the centerline of the target two-dimensional blood vessel; It is the Gaussian kernel bandwidth parameter, which uses the maximum distance between two-dimensional points. It is the value of the Gaussian kernel, which measures... and The similarity between two points; K is the objective function, and n is the total number of points on the two-dimensional curve to be registered.

14. The computer-readable storage medium according to claim 11, characterized in that, The step of iteratively solving for the final camera pose based on the initial camera pose, with the objective function being minimized, includes: With the objective function being minimized, the final camera pose is solved iteratively using a nonlinear least squares algorithm.

Images