Vascular interventional surgery navigation method and device based on multi-modal image fusion

Abstract

The application belongs to the technical field of surgical navigation, and discloses a blood vessel interventional surgical navigation method and device based on multi-modal image fusion. The method comprises the following steps: taking a preoperative coronary image and a real-time coronary angiography image, wherein the preoperative coronary image is 3D-CTA data, and the real-time coronary angiography image is 2D-DSA data; determining the length of a guide wire inserted into a coronary artery, wherein the length of the guide wire inserted into the coronary artery is the length from the entry point of the coronary artery to the corresponding position of the end point of the guide wire; determining the name of the coronary vessel where the guide wire is located through the real-time coronary angiography image; and matching the real-time position of the end point of the guide wire in the preoperative coronary image based on the name of the vessel and the length of the guide wire inserted into the coronary artery. In the above manner, the real-time position of the end point of the guide wire can be accurately displayed.

Description

Technical Field

[0001] This invention relates to the field of surgical navigation technology, and in particular to a method and device for vascular interventional surgical navigation based on multimodal image fusion. Background Technology

[0002] Coronary angiography is an invasive diagnostic technique widely used in clinical practice. Doctors typically use intraoperative 2D angiographic images (such as DSA) as guiding images in real-time. However, these modal images are acquired through projection, thus lacking three-dimensional depth information. The tissue overlap caused by projection makes it difficult for doctors to make clear and intuitive decisions regarding interventional methods and treatment plans during surgery. Preoperative CTA, on the other hand, provides better three-dimensional information. Through 3D reconstruction, it can intuitively and three-dimensionally present the three-dimensional spatial information of blood vessels, and can generate functional information such as 3D VR, vessel cross-sections, 3D MIP, planar CPR, line CPR, slice images, and CT-FFR. Therefore, fusing real-time 2D angiographic images with CTA images containing spatial structural information can quickly assist doctors in making accurate diagnoses from multiple perspectives.

[0003] Currently, multimodal (3D / 2D) image fusion technology is based on the registration method of coronary artery centerlines, that is, registering the centerlines obtained from 2D images with the centerlines obtained from 3D images. However, the length and shape of the centerlines corresponding to these two modalities are inconsistent. Moreover, when performing centerline registration, the position of the guidewire starting point is often not accurately obtained because the location of the coronary angiography entry point in the image is not clear. All of these will lead to poor centerline registration, thus affecting the accuracy of the registration between the angiographic coronary artery and the preoperative coronary artery. Summary of the Invention

[0004] The main objective of this invention is to provide a navigation method and device for vascular interventional surgery based on multimodal image fusion, which aims to solve the technical problem in the prior art that the real-time position of the guidewire tip is affected by poor registration between the center lines of 2D and 3D images.

[0005] To achieve the above objectives, the present invention provides a vascular interventional surgery navigation method based on multimodal image fusion, the method comprising the following steps:

[0006] Acquire preoperative coronary artery images and real-time coronary angiography images, wherein the preoperative coronary artery images are 3D-CTA data and the real-time coronary angiography images are 2D-DSA data;

[0007] The preoperative coronary artery image is input into the trained 3D segmentation model to obtain the initial 3D vessel segmentation result;

[0008] The 3D vessel centerline extraction result in the initial 3D vessel segmentation result is determined based on the Region Grow algorithm or machine learning algorithm, / or the 3D vessel centerline extraction result in the initial 3D vessel segmentation result is determined based on minimal-path.

[0009] All centerlines in the 3D vessel centerline extraction results are named using deep learning or matching methods to complete the naming of the coronary vessel centerlines in the preoperative coronary artery image.

[0010] Determine the length of the guidewire that penetrates into the coronary artery, wherein the length of the guidewire is the length from the inlet point of the coronary artery to the corresponding position of the end point of the guidewire;

[0011] The name of the coronary artery where the guidewire is located is determined by the real-time coronary angiography images.

[0012] Based on the vessel name and the length of the coronary artery penetration, the real-time position of the guidewire tip in the preoperative coronary artery image is matched.

[0013] Optionally, determining the depth of the guidewire into the coronary artery includes:

[0014] Determine the gear rotation speed and drive device diameter in the interventional robot;

[0015] The forward length / retraction length of the guide wire is determined based on the gear rotation speed, the diameter of the drive device, the motion state, and the motion time.

[0016] Based on the forward length / retraction length and the marked coronary artery inlet as the guidewire starting point, the guidewire's insertion length into the coronary artery is determined.

[0017] Optionally, determining the name of the coronary artery where the guidewire is located using the real-time coronary angiography image includes:

[0018] The real-time coronary angiography image is input into the trained angiography vessel naming model to obtain the predicted result of the vessel name classification of the coronary vessel where the guidewire is located.

[0019] The name of the coronary artery where the guidewire is located is determined based on the prediction results.

[0020] Optionally, before inputting the real-time coronary angiography image into the trained angiography vessel naming model, the method further includes:

[0021] The labeled DSA data is input into a deep learning network to obtain the predicted classification result of the coronary artery where the guidewire is located;

[0022] Based on the label data and the predicted classification results, determine the cross-entropy loss;

[0023] The parameters in the angiographic vessel naming model are updated based on the cross-entropy loss to obtain a trained angiographic vessel naming model.

[0024] Optionally, after acquiring preoperative coronary artery images, the procedure may also include:

[0025] The preoperative coronary artery image is input into the trained 3D segmentation model to obtain the initial 3D vessel segmentation result;

[0026] The 3D vessel centerline extraction result in the initial 3D vessel segmentation result is determined based on the Region Grow algorithm or a machine learning algorithm; or...

[0027] The 3D vessel centerline extraction result in the initial 3D vessel segmentation result is determined based on the minimal-path.

[0028] All centerlines in the 3D vascular centerline extraction results are named using deep learning or matching methods.

[0029] Optionally, before inputting the preoperative coronary artery image into the trained 3D segmentation model, the method further includes:

[0030] CTA data labeled with real blood vessel information is input into a 3D segmentation model to obtain predicted blood vessel information;

[0031] Based on the real blood vessel information and the predicted blood vessel information, the prediction error is determined;

[0032] The parameters in the 3D segmentation model are updated through backpropagation based on the prediction error to obtain a trained 3D segmentation model.

[0033] Optionally, matching the guidewire tip location in the preoperative coronary image based on the vessel name and the length of penetration into the coronary artery includes:

[0034] Based on the name of the blood vessel, determine the position of the guidewire in the preoperative coronary artery image;

[0035] Based on the location and the length of penetration into the coronary artery, the real-time position of the guidewire tip in the preoperative coronary artery image is matched.

[0036] Furthermore, to achieve the above objectives, the present invention also proposes a vascular interventional surgery navigation device based on multimodal image fusion, the vascular interventional surgery navigation device based on multimodal image fusion comprising:

[0037] The acquisition module is used to acquire preoperative coronary artery images and real-time coronary angiography images, wherein the preoperative coronary artery images are 3D-CTA data and the real-time coronary angiography images are 2D-DSA data;

[0038] The determination module is used to input the preoperative coronary artery image into the trained 3D segmentation model to obtain the initial 3D vessel segmentation result;

[0039] The determining module is used to determine the 3D vessel centerline extraction result in the initial 3D vessel segmentation result based on the Region grow algorithm or machine learning algorithm, / or to determine the 3D vessel centerline extraction result in the initial 3D vessel segmentation result based on minimal-path.

[0040] The naming module is used to name all the centerlines in the 3D vessel centerline extraction result through deep learning or matching methods, so as to complete the naming of the coronary vessel centerlines in the preoperative coronary artery image.

[0041] The determining module is also used to match the real-time position of the guidewire tip in the preoperative coronary artery image based on the vessel name and the length of penetration into the coronary artery;

[0042] The determining module is used to determine the name of the coronary artery where the guidewire is located through the real-time coronary angiography image;

[0043] The matching module is used to match the real-time position of the guidewire tip in the preoperative coronary artery image based on the vessel name and the length of penetration into the coronary artery.

[0044] Furthermore, to achieve the above objectives, the present invention also proposes a vascular interventional surgery navigation device based on multimodal image fusion. The vascular interventional surgery navigation device based on multimodal image fusion includes: a memory, a processor, and a vascular interventional surgery navigation program based on multimodal image fusion stored in the memory and executable on the processor. The vascular interventional surgery navigation program based on multimodal image fusion is configured to implement the steps of the vascular interventional surgery navigation method based on multimodal image fusion as described above.

[0045] Furthermore, to achieve the above objectives, the present invention also proposes a storage medium storing a vascular interventional surgery navigation program based on multimodal image fusion, wherein when the vascular interventional surgery navigation program based on multimodal image fusion is executed by a processor, it implements the steps of the vascular interventional surgery navigation method based on multimodal image fusion as described above.

[0046] The present invention provides a vascular interventional surgery navigation method based on multimodal image fusion, which acquires preoperative coronary artery images and real-time coronary angiography images, wherein the preoperative coronary artery images are 3D-CTA data and the real-time coronary angiography images are 2D-DSA data; the preoperative coronary artery images are input into a trained 3D segmentation model to obtain initial 3D vessel segmentation results; based on Region The GROUP algorithm or machine learning algorithm is used to determine the 3D vessel centerline extraction result in the initial 3D vessel segmentation result, / or the 3D vessel centerline extraction result is determined based on minimal-path; all centerlines in the 3D vessel centerline extraction result are named using deep learning or matching methods to complete the naming of the coronary vessel centerlines in the preoperative coronary artery image; the guidewire penetration length in the coronary artery is determined, wherein the penetration length in the coronary artery is the length of the guidewire from the coronary artery inlet point to the corresponding position of the guidewire tip; the vessel name of the coronary artery where the guidewire is located is determined using the real-time coronary angiography image; based on the vessel name and the penetration length in the coronary artery, the real-time position of the guidewire tip in the preoperative coronary artery image is matched. The above method first obtains the name of the coronary artery where the guidewire is located and the length of the guidewire's penetration into the coronary artery from the DSA image. Then, based on the name of the vessel and the length of penetration into the coronary artery, the corresponding position of the guidewire tip in the preoperative coronary artery image is determined. This allows for accurate and effective determination of the real-time position of the guidewire tip in the preoperative coronary artery image, thereby assisting doctors in making accurate diagnoses of the patient's lesion location from multiple perspectives. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the structure of a vascular interventional surgery navigation device based on multimodal image fusion, which is part of the hardware operating environment of the embodiment of the present invention.

[0048] Figure 2 This is a flowchart illustrating the first embodiment of the vascular interventional surgery navigation method based on multimodal image fusion of the present invention;

[0049] Figure 3 This is a diagram of the deep learning network structure used to predict the name of the coronary artery where the guidewire is located in the first embodiment of the vascular interventional surgery navigation method based on multimodal image fusion of the present invention.

[0050] Figure 4 This is a flowchart illustrating the second embodiment of the vascular interventional surgery navigation method based on multimodal image fusion of the present invention;

[0051] Figure 5 This is a multimodal image of the second embodiment of the vascular interventional surgery navigation method based on multimodal image fusion of the present invention;

[0052] Figure 6This is a structural block diagram of the first embodiment of the vascular interventional surgery navigation device based on multimodal image fusion of the present invention.

[0053] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0054] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0055] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of a vascular interventional surgery navigation device based on multimodal image fusion, which is part of the hardware operating environment of the embodiment of the present invention.

[0056] like Figure 1 As shown, the vascular interventional surgery navigation device based on multimodal image fusion may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0057] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on vascular interventional surgical navigation devices based on multimodal image fusion, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0058] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and a vascular interventional surgery navigation program based on multimodal image fusion.

[0059] exist Figure 1In the vascular interventional surgery navigation device based on multimodal image fusion shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and memory 1005 in the vascular interventional surgery navigation device based on multimodal image fusion of the present invention can be set in the vascular interventional surgery navigation device based on multimodal image fusion. The vascular interventional surgery navigation device based on multimodal image fusion calls the vascular interventional surgery navigation program based on multimodal image fusion stored in the memory 1005 through the processor 1001 and executes the vascular interventional surgery navigation method based on multimodal image fusion provided in the embodiment of the present invention.

[0060] Based on the above hardware structure, an embodiment of the vascular interventional surgery navigation method based on multimodal image fusion of the present invention is proposed.

[0061] Reference Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of a vascular interventional surgical navigation method based on multimodal image fusion according to the present invention.

[0062] In this embodiment, the vascular interventional surgery navigation method based on multimodal image fusion includes the following steps:

[0063] Step S10: Acquire preoperative coronary artery images and real-time coronary angiography images, wherein the preoperative coronary artery images are 3D-CTA data and the real-time coronary angiography images are 2D-DSA data.

[0064] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a mobile phone, tablet computer, or personal computer, or an electronic device capable of performing the above functions or a vascular interventional surgery navigation device based on multimodal image fusion. The following description uses the vascular interventional surgery navigation device based on multimodal image fusion as an example to illustrate this embodiment and the subsequent embodiments.

[0065] It's important to note that preoperative coronary angiography images are CTA images acquired before the interventional procedure. CTA images are obtained through CT angiography and are a crucial part of clinical CT applications. Due to the poor natural contrast between coronary arteries and their background soft tissues, conventional CT scans often fail to visualize the coronary vessels. During CTA examinations, contrast agents are introduced to alter the image contrast between the coronary vessels and background tissues, thereby highlighting the coronary vessels. Preoperative CTA images are 3D images with good three-dimensional information. Three-dimensional reconstruction of CTA images can intuitively and three-dimensionally present the three-dimensional spatial information of the coronary vessels. Furthermore, functional information such as 3DVR, vessel cross-sections, 3D MIP, planar CPR, line CPR, slice images, and CT-FFR can be derived from CTA images. Real-time coronary angiography images, also known as DSA images, are acquired in real-time during the interventional procedure. These are 2D data. During the interventional procedure, the physician injects contrast agent into the coronary arteries to visualize them, which are then captured by X-ray imaging to obtain clear images. During interventional procedures, doctors need to insert a guidewire into the coronary artery; therefore, real-time coronary angiography images are coronary angiography images with a guidewire.

[0066] Understandably, by fusing real-time 2D angiography images with CTA images containing spatial structural information, doctors can present the patient's plaque information and coronary artery functional information in the form of images during interventional procedures, thus providing rapid and accurate diagnosis from multiple perspectives.

[0067] Step S20: Input the preoperative coronary artery image into the trained 3D segmentation model to obtain the initial 3D vessel segmentation result.

[0068] It should be noted that the 3D segmentation model is a deep learning network.

[0069] It should be noted that inputting preoperative coronary artery images into a trained 3D segmentation model can segment the coronary arteries from the preoperative coronary artery images.

[0070] Step S30: Determine the 3D vessel centerline extraction result in the initial 3D vessel segmentation result based on the Region Grow algorithm or machine learning algorithm, / or determine the 3D vessel centerline extraction result in the initial 3D vessel segmentation result based on minimal-path.

[0071] Step S40: Name all the centerlines in the 3D vessel centerline extraction results using deep learning or matching methods to complete the naming of the coronary vessel centerlines in the preoperative coronary artery image.

[0072] In practical implementation, the OP segment centerline can be determined from the 3D vessel centerline extraction results based on the location O of the coronary artery inlet point in the preoperative coronary artery image and the real-time location of the guidewire tip in the preoperative coronary artery image. Then, coronary artery morphological or functional information can be obtained based on the OP segment centerline. Figure 5 As shown, it can provide doctors with more comprehensive and rich information on coronary arteries and surgical navigation through 3D VR, vascular cross sections, 3D MIP, surface CPR, line CPR, and slice images of preoperative coronary artery images.

[0073] Step S50: Determine the length of the guidewire penetrating the coronary artery, wherein the length of the guidewire penetrating the coronary artery is the length from the inlet point of the coronary artery to the corresponding position of the end point of the guidewire.

[0074] It should be noted that during interventional surgery, a guidewire needs to be inserted into the coronary artery. Doctors usually control the movement of the guidewire by controlling an interventional robot. Therefore, the length of the guidewire's insertion into the coronary artery can be determined by the interventional robot.

[0075] It should be noted that the guidewire end point is the current distal end of the guidewire, that is, the point on the guidewire that is farthest from the interventional robot at the current moment, and it is also the starting point of the guidewire's movement. Since the other end point of the guidewire is fixed on the interventional robot, the guidewire end point mentioned in this invention refers to the point on the guidewire that is farthest from the interventional robot at the current moment.

[0076] It is understandable that, since the guidewire advances or retracts along the coronary artery centerline when it is inserted into the coronary artery, the length of the guidewire penetrating the coronary artery is equivalent to the length of the coronary artery centerline from the coronary artery inlet point to the current distal end of the guidewire.

[0077] In practice, the depth of the guidewire into the coronary artery can also be determined by external equipment or interventional robots.

[0078] In one embodiment, determining the depth of the guidewire into the coronary artery using an interventional robot includes:

[0079] Determine the gear rotation speed and drive device diameter in the interventional robot;

[0080] The forward length / retraction length of the guide wire is determined based on the gear rotation speed, the diameter of the drive device, the motion state, and the motion time.

[0081] Based on the forward length / retraction length and the marked coronary artery inlet as the guidewire starting point, the guidewire's insertion length into the coronary artery is determined.

[0082] It should be noted that the motion state refers to the forward movement or retraction of the guidewire.

[0083] It should be noted that the formula for determining the forward length / retraction length of the guide wire based on the gear speed, the diameter of the driving device, and the movement time is: L = 0.5R * V * T, where L represents the forward length / retraction length of the guide wire, R represents the diameter of the driving device, and T represents the movement time.

[0084] In the specific implementation, the gear speed of the intervention robot is 10 rad / s, the diameter of the drive device is 0.2 m, and the movement time is 0.1 s. Therefore, the forward length / retraction length of the guide wire can be determined to be 0.1 m based on L = 0.5R * V * T.

[0085] It should be noted that if the length of the guidewire penetrating the coronary artery at time t1 is known, then the length of the guidewire penetrating the coronary artery at time t2 can be determined based on the forward length / retraction length of the guidewire from time t1 to time t2.

[0086] It should be noted that, since most of the guidewire is transparent and has no distinctive structure, it is very difficult to directly detect the length of the guidewire's penetration into the coronary artery based on real-time intraoperative coronary artery images.

[0087] In this embodiment, the guidewire's forward / retraction length is determined by using an interventional robot, thereby determining the guidewire's depth into the coronary artery. This allows for rapid and accurate acquisition of the guidewire's depth into the coronary artery, effectively improving the accuracy of the real-time position display of the guidewire's tip.

[0088] Step S60: Determine the name of the coronary artery where the guidewire is located using the real-time coronary angiography image.

[0089] It should be noted that, apart from the guidewire head and guidewire tip, most of the guidewire area is transparent and has no characteristic structure.

[0090] In practice, since the guidewire head and guidewire tip are non-transparent structures, the current position of the guidewire's distal end in the real-time coronary angiography image can be obtained by image processing of the real-time coronary angiography image, and then the name of the coronary vessel where the guidewire is located can be determined based on the position of the current distal end of the guidewire.

[0091] In one embodiment, determining the name of the coronary artery where the guidewire is located using the real-time coronary angiography image includes:

[0092] The real-time coronary angiography image is input into the trained angiography vessel naming model to obtain the predicted result of the vessel name classification of the coronary vessel where the guidewire is located.

[0093] The name of the coronary artery where the guidewire is located is determined based on the prediction results.

[0094] It should be noted that the deep learning network in this invention is a multi-classification task deep learning network. The process of inputting real-time intraoperative images into the trained deep learning network for prediction is as follows: Figure 3 As shown.

[0095] In practice, real-time coronary angiography images are input into a trained angiography vessel naming model to predict the probability of the coronary vessel where the guidewire is located belonging to each vessel category. The vessel category with the highest probability is then determined as the vessel name of the coronary vessel where the guidewire is located. For example, if a coronary angiography image contains four types of coronary vessels: type A, type B, type C, and type D, and the real-time coronary angiography image is input into a trained deep learning network, the probability of predicting that the coronary vessel where the guidewire is located is type A is 0.2, type B is 0.3, type C is 0.1, and type D is 0.4 can be determined based on the prediction results (i.e., the category with the highest prediction probability). The vessel name of the coronary vessel where the guidewire is located can be determined as type D. Note that the vessel categories in the coronary angiography image are not limited to four; specific vessel categories can include LAD, LCX, RCA, D1, D2, OM1, OM2, RI, RPDA, etc.

[0096] In this embodiment, by inputting real-time coronary angiography images into a trained deep learning network, the name of the coronary vessel where the guidewire is located can be predicted quickly and accurately, thereby improving the matching rate of the real-time position of the guidewire tip.

[0097] In one embodiment, before inputting the real-time coronary angiography image into the trained angiography vessel naming model, the method further includes:

[0098] The labeled DSA data is input into the angiography vessel naming model to obtain the predicted classification result of the coronary vessel where the guidewire is located.

[0099] Based on the label data and the predicted classification results, determine the cross-entropy loss;

[0100] The parameters in the angiographic vessel naming model are updated based on the cross-entropy loss to obtain a trained angiographic vessel naming model.

[0101] It should be noted that DSA data refers to coronary angiography images obtained during interventional surgery when a guidewire is inserted into the coronary artery. The label data is pre-labeled by the doctor based on the guidewire tip in the DSA data. Specifically, if the guidewire tip in the DSA data is located in a coronary artery named RCA, then the label data in the DSA data is RCA.

[0102] It should be noted that the formula for calculating the cross-entropy loss, the loss function for classification tasks in deep learning, is as follows: In the formula, i represents the i-th category, k represents the number of coronary artery types in the DSA data, y represents the label data, and p represents the probability that the deep learning network predicts the coronary artery category to be i.

[0103] It should be noted that when the label data is of the i-th category, y i =1, otherwise 0.

[0104] In the specific implementation, the coronary vessels labeled in the DSA data are classified as type i. The DSA data with labeled coronary vessel names is input into a deep learning network to predict the vessel name of the coronary vessel where the guidewire is located, obtaining the predicted classification result. Specifically, if the DSA data contains four types of coronary vessels: A, B, C, and D, and the labeled coronary vessels are type D, inputting the DSA data with type D coronary vessels into the deep learning network yields the following predicted classification results: the prediction probability for type A coronary vessels is 0.2, for type B coronary vessels is 0.3, for type C coronary vessels is 0.1, and for type D coronary vessels is 0.4. Therefore, according to... The cross-entropy loss can be determined as follows:

[0105] Step S70: Based on the vessel name and the length of the in-depth coronary artery, match the real-time position of the guidewire tip in the preoperative coronary artery image.

[0106] It's important to note that preoperative coronary artery images are CTA images acquired before the interventional procedure, while real-time coronary angiography images are DSA images acquired during the procedure. During the interventional procedure, it's impossible to obtain three-dimensional imaging information about the guidewire tip location from the CTA images. Therefore, during the interventional procedure, the only way to determine the guidewire tip location in the preoperative coronary artery images is to first acquire the DSA images and then use the guidewire information within the DSA images.

[0107] In practice, the process begins by acquiring DSA images of the patient undergoing interventional surgery. Then, based on the DSA images, the name of the coronary artery where the guidewire is located is determined, as well as the length of the guidewire's penetration into the coronary artery. Next, the guidewire's position in the preoperative coronary artery image is preliminarily located based on the name of the coronary artery where the guidewire is located. Finally, the position of the guidewire's distal end point in the preoperative coronary artery image is precisely determined based on the length of penetration into the coronary artery. In other words, the real-time position of the current distal end point in the preoperative coronary artery image is matched based on the vessel name and the length of penetration into the coronary artery.

[0108] Understandably, after determining the real-time location of the guidewire tip in the preoperative coronary artery image, doctors can use the preoperative coronary artery image to understand the patient's three-dimensional spatial information at the guidewire tip, thereby assisting doctors in making accurate diagnoses from multiple perspectives.

[0109] In one embodiment, matching the guidewire tip location in the preoperative coronary image based on the vessel name and the length of penetration into the coronary artery includes:

[0110] Based on the name of the blood vessel, determine the position of the guidewire in the preoperative coronary artery image;

[0111] Based on the location and the length of penetration into the coronary artery, the real-time position of the guidewire tip in the preoperative coronary artery image is matched.

[0112] In this embodiment, the corresponding position of the guidewire in the preoperative coronary artery image is initially determined by the name of the blood vessel where the guidewire is located. Then, the real-time position of the guidewire tip in the preoperative coronary artery image is accurately determined based on the length of the guidewire's penetration into the coronary artery, which can effectively improve the accuracy of real-time position determination.

[0113] This embodiment acquires preoperative coronary artery images and real-time coronary angiography images, wherein the preoperative coronary artery images are 3D-CTA data and the real-time coronary angiography images are 2D-DSA data; determines the guidewire's penetration length in the coronary artery, wherein the penetration length is the length from the coronary artery inlet point to the corresponding position of the guidewire tip; determines the name of the coronary vessel where the guidewire is located using the real-time coronary angiography image; and matches the real-time position of the guidewire tip in the preoperative coronary artery image based on the vessel name and the penetration length. Through this method, the name of the coronary vessel where the guidewire is located and the penetration length of the guidewire are first obtained from the DSA image, and then the corresponding position of the guidewire tip in the preoperative coronary artery image is determined based on the vessel name and penetration length. This allows for accurate and effective determination of the real-time position of the guidewire tip in the preoperative coronary artery image, thereby assisting physicians in accurately diagnosing the location of lesions from multiple perspectives.

[0114] refer to Figure 4 , Figure 4 This is a flowchart illustrating a second embodiment of a vascular interventional surgical navigation method based on multimodal image fusion according to the present invention.

[0115] Based on the first embodiment described above, this embodiment of the vascular interventional surgery navigation method based on multimodal image fusion further includes, after step S10:

[0116] Step S101: Input the preoperative coronary artery image into the trained 3D segmentation model to obtain the initial 3D vessel segmentation result.

[0117] It should be noted that the 3D segmentation model is a deep learning network.

[0118] It should be noted that inputting preoperative coronary artery images into a trained 3D segmentation model can segment the coronary arteries from the preoperative coronary artery images.

[0119] In one embodiment, before inputting the preoperative coronary artery image into the trained 3D segmentation model, the method further includes:

[0120] CTA data labeled with real blood vessel information is input into a 3D segmentation model to obtain predicted blood vessel information;

[0121] Based on the real blood vessel information and the predicted blood vessel information, the prediction error is determined;

[0122] The parameters in the 3D segmentation model are updated through backpropagation based on the prediction error to obtain a trained 3D segmentation model.

[0123] It should be noted that the actual vascular information is obtained by doctors in advance by annotating the coronary vessels in the CTA data.

[0124] Step S102: Determine the 3D vessel centerline extraction result in the initial 3D vessel segmentation result based on the Region Grow algorithm or machine learning algorithm.

[0125] Step S103: Determine the 3D vessel centerline extraction result in the initial 3D vessel segmentation result based on minimal-path.

[0126] In practice, the 3D vessel centerline extraction result can also be obtained by performing skeletonization processing on the initial segmentation result of the 3D vessel.

[0127] In practical implementation, the OP segment centerline can be determined from the 3D vessel centerline extraction results based on the location O of the coronary artery inlet point in the preoperative coronary artery image and the real-time location of the guidewire tip in the preoperative coronary artery image. Then, coronary artery morphological or functional information can be obtained based on the OP segment centerline. Figure 5 As shown, it can provide doctors with more comprehensive and rich information on coronary arteries and surgical navigation through 3D VR, vascular cross sections, 3D MIP, surface CPR, line CPR, and slice images of preoperative coronary artery images.

[0128] This embodiment inputs the preoperative coronary artery image into a trained 3D segmentation model to obtain the initial 3D vessel segmentation result. The 3D vessel centerline extraction result is then determined based on the Region Grow algorithm or a machine learning algorithm; alternatively, the 3D vessel centerline extraction result is determined based on the minimal-path algorithm. Through this method, the centerline of each coronary artery in the preoperative coronary artery image can be determined, thereby providing physicians with more detailed impact or functional information after determining the corresponding position of the guidewire tip in the preoperative coronary artery image.

[0129] Furthermore, this embodiment of the invention also proposes a storage medium storing a vascular interventional surgery navigation program based on multimodal image fusion. When the vascular interventional surgery navigation program based on multimodal image fusion is executed by a processor, it implements the steps of the vascular interventional surgery navigation method based on multimodal image fusion as described above.

[0130] Reference Figure 6 , Figure 6 This is a structural block diagram of the first embodiment of the vascular interventional surgery navigation device based on multimodal image fusion of the present invention.

[0131] like Figure 5 As shown, the vascular interventional surgery navigation device based on multimodal image fusion proposed in this embodiment of the invention includes:

[0132] The acquisition module 10 is used to acquire preoperative coronary artery images and real-time coronary angiography images, wherein the preoperative coronary artery images are 3D-CTA data and the real-time coronary angiography images are 2D-DSA data.

[0133] The determination module 20 is used to input the preoperative coronary artery image into the trained 3D segmentation model to obtain the initial 3D vessel segmentation result;

[0134] The determining module 20 is further configured to determine the 3D vessel centerline extraction result in the initial 3D vessel segmentation result based on the Region grow algorithm or machine learning algorithm, / or determine the 3D vessel centerline extraction result in the initial 3D vessel segmentation result based on minimal-path.

[0135] The naming module 30 is used to name all the center lines in the 3D vessel center line extraction result through deep learning or matching methods, so as to complete the naming of the coronary vessel center lines in the preoperative coronary artery image.

[0136] The determining module 20 is used to determine the length of the guidewire penetrating the coronary artery, wherein the length of the guidewire penetrating the coronary artery is the length from the coronary artery inlet point to the corresponding position of the guidewire end point.

[0137] The determining module 20 is used to determine the name of the coronary artery where the guidewire is located through the real-time coronary angiography image.

[0138] The matching module 40 is used to match the current distal endpoint to the real-time position in the preoperative coronary artery image based on the vessel name and the length of the deep coronary artery.

[0139] It should be understood that the above are merely illustrative examples and do not constitute any limitation on the technical solutions of the present invention. In specific applications, those skilled in the art can make settings as needed, and the present invention does not impose any restrictions on this.

[0140] This embodiment acquires preoperative coronary artery images and real-time coronary angiography images, wherein the preoperative coronary artery images are 3D-CTA data and the real-time coronary angiography images are 2D-DSA data; determines the guidewire's penetration length in the coronary artery, wherein the penetration length is the length from the guidewire's entry point in the coronary artery to the corresponding position of the guidewire's tip; determines the name of the coronary vessel where the guidewire is located using the real-time coronary angiography image; and matches the real-time position of the guidewire's tip in the preoperative coronary artery image based on the vessel name and the penetration length. Through this method, the name of the coronary vessel where the guidewire is located and the penetration length of the guidewire are first obtained from the DSA image, and then the corresponding position of the guidewire's tip in the preoperative coronary artery image is determined based on the vessel name and penetration length. This allows for accurate and effective determination of the real-time position of the guidewire's tip in the preoperative coronary artery image, thereby assisting physicians in accurately diagnosing the location of the patient's lesions from multiple perspectives.

[0141] In one embodiment, the acquisition module 10 is further configured to:

[0142] Determine the gear rotation speed and drive device diameter in the interventional robot;

[0143] The forward length / retraction length of the guide wire is determined based on the gear rotation speed, the diameter of the drive device, the motion state, and the motion time.

[0144] Based on the forward / retraction length and the guidewire starting point position, the depth of the guidewire into the coronary artery is determined.

[0145] In one embodiment, the determining module 20 is further configured to:

[0146] The real-time coronary angiography image is input into the trained angiography vessel naming model to obtain the predicted result of the vessel name classification of the coronary vessel where the guidewire is located.

[0147] The name of the coronary artery where the guidewire is located is determined based on the prediction results.

[0148] In one embodiment, the determining module 20 is further configured to:

[0149] The labeled DSA data is input into the angiography vessel naming model to obtain the predicted classification result of the coronary vessel where the guidewire is located.

[0150] Based on the label data and the predicted classification results, determine the cross-entropy loss;

[0151] The parameters in the angiographic vessel naming model are updated based on the cross-entropy loss to obtain a trained angiographic vessel naming model.

[0152] In one embodiment, the acquisition module 10 is further configured to:

[0153] The preoperative coronary artery image is input into the trained 3D segmentation model to obtain the initial 3D vessel segmentation result;

[0154] The 3D vessel centerline extraction result in the initial 3D vessel segmentation result is determined based on the Region Grow algorithm or a machine learning algorithm; or...

[0155] The 3D vessel centerline extraction result in the initial 3D vessel segmentation result is determined based on the minimal-path.

[0156] All centerlines in the 3D vascular centerline extraction results are named using deep learning or matching methods.

[0157] In one embodiment, the acquisition module 10 is further configured to:

[0158] CTA data labeled with real blood vessel information is input into a 3D segmentation model to obtain predicted blood vessel information;

[0159] Based on the real blood vessel information and the predicted blood vessel information, the prediction error is determined;

[0160] The parameters in the 3D segmentation model are updated through backpropagation based on the prediction error to obtain a trained 3D segmentation model.

[0161] In one embodiment, the matching module 40 is further configured to:

[0162] Based on the name of the blood vessel, determine the position of the guidewire in the preoperative coronary artery image;

[0163] Based on the location and the length of penetration into the coronary artery, the real-time position of the guidewire tip in the preoperative coronary artery image is matched.

[0164] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this invention. In practical applications, those skilled in the art can select some or all of the workflow to achieve the purpose of this embodiment according to actual needs, and no restrictions are imposed here.

[0165] In addition, for technical details not described in detail in this embodiment, please refer to the vascular interventional surgery navigation method based on multimodal image fusion provided in any embodiment of the present invention, which will not be repeated here.

[0166] Furthermore, it should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0167] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0168] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0169] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A vascular interventional surgery navigation device based on multimodal image fusion, characterized in that, The vascular interventional surgery navigation device based on multimodal image fusion is used to execute a vascular interventional surgery navigation method based on multimodal image fusion, the method comprising: Acquire preoperative coronary artery images and real-time coronary angiography images, wherein the preoperative coronary artery images are 3D-CTA data and the real-time coronary angiography images are 2D-DSA data; The preoperative coronary artery image is input into the trained 3D segmentation model to obtain the initial 3D vessel segmentation result; The 3D vessel centerline extraction result in the initial 3D vessel segmentation result is determined based on the Region Grow algorithm or machine learning algorithm, / or the 3D vessel centerline extraction result in the initial 3D vessel segmentation result is determined based on minimal-path. All centerlines in the 3D vessel centerline extraction results are named using deep learning or matching methods to complete the naming of the coronary vessel centerlines in the preoperative coronary artery image. Determine the length of the guidewire that penetrates into the coronary artery, wherein the length of the guidewire is the length from the inlet point of the coronary artery to the corresponding position of the end point of the guidewire; The name of the coronary artery where the guidewire is located is determined by the real-time coronary angiography images. Based on the vessel name and the length of the coronary artery penetration, the real-time position of the guidewire tip in the preoperative coronary artery image is matched.

2. The apparatus as claimed in claim 1, characterized in that, Determining the depth of the guidewire into the coronary artery includes: Determine the gear rotation speed and drive device diameter in the interventional robot; The forward length / retraction length of the guide wire is determined based on the gear rotation speed, the diameter of the drive device, the motion state, and the motion time. Based on the forward length / retraction length and the marked coronary artery inlet as the guidewire starting point, the guidewire's insertion length into the coronary artery is determined.

3. The apparatus as described in claim 1, characterized in that, The step of determining the name of the coronary artery where the guidewire is located using the real-time coronary angiography image includes: The real-time coronary angiography image is input into the trained angiography vessel naming model to obtain the predicted result of the vessel name classification of the coronary vessel where the guidewire is located. The name of the coronary artery where the guidewire is located is determined based on the prediction results.

4. The apparatus as described in claim 3, characterized in that, Before inputting the real-time coronary angiography images into the trained angiography vessel naming model, the process also includes: The labeled DSA data is input into the angiography vessel naming model to obtain the predicted classification result of the coronary vessel where the guidewire is located. Based on the label data and the predicted classification results, determine the cross-entropy loss; The parameters in the angiographic vessel naming model are updated based on the cross-entropy loss to obtain a trained angiographic vessel naming model.

5. The apparatus as claimed in claim 1, characterized in that, Before inputting the preoperative coronary artery images into the trained 3D segmentation model, the following steps are also included: CTA data labeled with real blood vessel information is input into a 3D segmentation model to obtain predicted blood vessel information; Based on the real blood vessel information and the predicted blood vessel information, the prediction error is determined; The parameters in the 3D segmentation model are updated through backpropagation based on the prediction error to obtain a trained 3D segmentation model.

6. The apparatus as claimed in any one of claims 1 to 5, characterized in that, The process of matching the guidewire tip to the specific location in the preoperative coronary artery image based on the vessel name and the length of penetration into the coronary artery includes: Based on the name of the blood vessel, determine the position of the guidewire in the preoperative coronary artery image; Based on the location and the length of penetration into the coronary artery, the real-time position of the guidewire tip in the preoperative coronary artery image is matched.

Images