Positioning method and device of interventional device, electronic device and storage medium

Abstract

The application discloses a positioning method and device of an interventional device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a first medical image of a target patient's vasculature, generating a first vasculature model based on the first medical image, acquiring a plurality of continuous second medical images in the blood vessels of the vasculature of the target patient through an interventional device arranged in the blood vessels of the vasculature of the target patient, generating a second vasculature model based on the second medical images, performing registration processing on the first vasculature model and the second vasculature model to obtain a target registration matrix corresponding to the target patient, acquiring a third medical image in the blood vessels of the vasculature of the target patient through the interventional device in real time, and mapping the third medical image onto the first vasculature model based on the target registration matrix to display the position of the interventional device on the first vasculature model in real time. According to the application, the interventional device can be accurately positioned in the vasculature without a lead plate operating room.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical auxiliary technology, and in particular to a positioning method and device of an interventional device, an electronic device and a storage medium. BACKGROUND

[0002] Human aortic dissection is a disease that seriously endangers life safety, and once found, it needs to be treated immediately. Part of Standford A dissection and all Standford B dissection can be treated by minimally invasive interventional stent implantation. Another part of Standford A dissection needs to be replaced by artificial blood vessels through open chest surgery, and then combined with interventional surgery. In addition, most of the aortic aneurysm and penetrating ulcer can be effectively treated by interventional surgery, which has the advantages of small trauma and quick recovery. CTA (Computed Tomography Angiography, Angiography) and MRI (Magnetic Resonance Imaging, Magnetic Resonance Imaging) are medical examination methods that can clearly show the human vascular system, and are commonly used for preoperative examination of vascular system diseases, but both can only show the overall outline of the vascular system, and cannot obtain specific information of the lesions in the vascular system. In order to obtain the specific information of the lesions in the vascular system, it is necessary to rely on intravascular ultrasound and OCT (Optical Coherence Tomography, Optical Coherence Tomography) examination to obtain. Among them, the two modalities (CTA and MRI, intravascular ultrasound and OCT) are complementary but independent of each other, and are currently only used for disease condition judgment, and cannot be combined in the operation process to accurately locate the lesion site and guide the operation.

[0003] In the related art, in order to accurately position the lesion site and perform an interventional operation, it is usually necessary to perform the operation in a lead plate operating room capable of preventing the penetration of rays, and the doctor needs to wear a heavy lead suit. During the operation, the vascular stent is sent into the patient's vasculature with the assistance of a delivery system, and the patient needs to be continuously injected with contrast agent to display the position and state of the implanted interventional device in the blood vessels. At the same time, the vascular stent needs to be X-rayed multiple times during and after release to determine the position of the vascular stent and the release effect. Multiple X-ray exposures are undoubtedly very disadvantageous to the patient and the operating medical staff, and the toxicity and allergenicity of the contrast agent to the human body also increase the iatrogenic risk of the patient. In addition, the lead plate operating room is expensive and limited in quantity, and the queuing for the lead plate operating room for interventional operation makes many medical staff and patients wait until late at night, which is undoubtedly a great waste of human resources, and also brings uncertainty to the treatment of diseases, and sometimes patients have anxiety during the waiting process, which leads to the occurrence of malignant cardiovascular events. Further, since surgical operation mainly intervenes in the proximal vascular region of the lesion, and interventional operation treats the distal vascular region of the lesion. Therefore, if the timing of surgical operation and the time when the lead plate operating room is available cannot be matched, the patient under general anesthesia will be in a state of maintaining anesthesia while waiting for the lead plate operating room, which brings a heavy burden to the patient, medical staff and the medical system.

[0004] Therefore, there is an urgent need for a new method for accurately positioning an interventional device in the vasculature without a lead plate operating room. SUMMARY

[0005] The purpose of the embodiments of the present application is to provide an interventional device positioning method, device, electronic device and storage medium to solve the technical problems in the background art.

[0006] In a first aspect, the embodiments of the present application provide an interventional device positioning method, comprising:

[0007] obtaining a first medical image of a vasculature in a target patient, and generating a corresponding first vasculature model based on the first medical image, wherein the first medical image comprises an angiogram and a magnetic resonance image;

[0008] obtaining a second medical image of a plurality of continuous frames in the blood vessel by an interventional device arranged in the vasculature of the target patient, and generating a three-dimensionally reconstructed second vasculature model based on the second medical image of the plurality of continuous frames, wherein the second medical image comprises an intravascular ultrasound image and an optical coherence tomography image;

[0009] performing registration processing on the first vasculature model and the second vasculature model to obtain a target registration matrix corresponding to the target patient;

[0010] acquiring a third medical image of a vessel system in the target patient in real time by the interventional device, and mapping the third medical image onto the first vessel system model based on the target registration matrix to display the position of the interventional device on the first vessel system model in real time.

[0011] In some embodiments, before the step of acquiring the first medical image of the vessel system in the target patient, the method further comprises:

[0012] acquiring a first initial image of a vessel system in a different patient, the first initial image comprising an angiography image and a magnetic resonance image;

[0013] rendering and segmenting the first initial image to obtain a plurality of first initial three-dimensional vessel system models corresponding to the different patients;

[0014] training a deep learning neural network model to be trained until convergence by taking the first initial image as input and taking the first initial three-dimensional vessel system model corresponding to the first initial image as output;

[0015] The step of generating a first vessel system model corresponding to the first medical image comprises:

[0016] calling the trained first generation model to generate a first vessel system model corresponding to the first medical image by generating processing the first medical image.

[0017] In some embodiments, before the step of acquiring a plurality of second medical images of the vessel system in the target patient in real time by the interventional device arranged in the vessel of the vessel system in the target patient, the method further comprises:

[0018] acquiring a plurality of second initial images of the vessel system in a different patient in real time by an interventional device entering the vessel system in the different patient, the second initial image comprising an intravascular ultrasound image and an optical coherence tomography image;

[0019] determining a three-dimensional coordinate of the vessel system in each frame of the second initial image according to first position information of the interventional device in each frame of the second initial image and second position information of the interventional device relative to the intravascular wall;

[0020] performing three-dimensional reconstruction of the vessel system in the different patient based on the three-dimensional coordinates of the vessel system in the plurality of second initial images to obtain a second three-dimensional human vessel system model corresponding to the different patient;

[0021] The second initial images corresponding to different patients are taken as inputs, and the second initial three-dimensional vessel system models corresponding to the second initial images are taken as outputs, and a deep learning neural network model to be trained is trained until convergence, and a trained second generation model is obtained;

[0022] The step of generating the three-dimensional reconstructed second vessel system model based on the second medical images includes:

[0023] The trained second generation model is called to generate the second vessel system model corresponding to the second medical images.

[0024] In some embodiments, the registration processing includes rigid registration processing, and the step of registering the first vessel system model and the second vessel system model to obtain a target registration matrix corresponding to the target patient includes:

[0025] The second vessel system model is rotated and translated to spatially align the second vessel system model with the first vessel system model.

[0026] The target registration matrix corresponding to the target patient is determined according to the second vessel system model before and after spatial alignment.

[0027] In some embodiments, the registration processing also includes non-rigid registration processing, and the step of registering the first vessel system model and the second vessel system model to obtain a target registration matrix corresponding to the target patient includes:

[0028] The second vessel system model is elastically transformed and thin-plate spline operated to spatially align the second vessel system model with the first vessel system model.

[0029] The target registration matrix corresponding to the target patient is determined according to the second vessel system model before and after spatial alignment.

[0030] In some embodiments, the interventional device includes a probe, a vascular stent, and a delivery rod, the probe is arranged at the end of the delivery rod, the vascular stent is arranged around and tightly against the outer sidewall of the delivery rod, and the method further includes:

[0031] When the position of the probe displayed in real time on the first vessel system model is a target position, prompt information to reach the target position is displayed.

[0032] In response to a vascular stent release instruction, the vascular stent is controlled to be released from the outer sidewall of the delivery rod and to perform expansion operation.

[0033] In some embodiments, before the step of controlling the release of the vascular stent from the outer side wall of the delivery rod and the expansion operation in response to the vascular stent release instruction, the method further comprises:

[0034] acquiring an intravascular image captured by the probe;

[0035] in the case that the position of the probe in the intravascular image is not the target position and a vascular stent release instruction is detected, stopping the execution of the vascular stent release operation and displaying prompt information indicating a position error.

[0036] In a second aspect, an embodiment of the present application provides a positioning device of an interventional device, comprising:

[0037] a first acquisition module configured to acquire a first medical image of a vasculature in a target patient, and generate a first vasculature model corresponding to the first medical image based on the first medical image, the first medical image comprising an angiogram image and a magnetic resonance image;

[0038] a second acquisition module configured to acquire a second medical image of a plurality of continuous frames in a blood vessel of the vasculature in the target patient by an interventional device arranged in the blood vessel of the vasculature in the target patient, and generate a three-dimensionally reconstructed second vasculature model based on the second medical image of the plurality of continuous frames, the second medical image comprising an intravascular ultrasound image and an optical coherence tomography image;

[0039] a registration module configured to perform registration processing on the first vasculature model and the second vasculature model to obtain a target registration matrix corresponding to the target patient;

[0040] a positioning module configured to acquire a third medical image in the blood vessel of the vasculature in the target patient by the interventional device in real time, and map the third medical image to the first vasculature model based on the target registration matrix, so as to display the position of the interventional device on the first vasculature model in real time.

[0041] In a third aspect, an embodiment of the present application provides an electronic device, which comprises a processor, a memory, and a computer program stored in the memory and capable of running on the processor, and the processor implements the steps in the positioning method of the interventional device according to any one of the above embodiments when executing the computer program.

[0042] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program implements the steps in the positioning method of the interventional device according to any one of the above embodiments when executed by a processor.

[0043] The embodiment of the application provides a positioning method, device, electronic equipment and storage medium of an interventional device, the method comprises the following steps: acquiring a first medical image of a target patient's vasculature, and generating a corresponding first vasculature model based on the first medical image; acquiring a plurality of continuous second medical images in the blood vessels of the target patient's vasculature through an interventional device arranged in the blood vessels of the target patient's vasculature, and generating a three-dimensionally reconstructed second vasculature model based on the plurality of continuous second medical images; and performing registration processing on the first vasculature model and the second vasculature model to obtain a target registration matrix corresponding to the target patient, and then mapping a third medical image in the blood vessels of the target patient's vasculature to the first vasculature model based on the target registration matrix to display the position of the interventional device on the first vasculature model in real time, so that the interventional device can be accurately positioned in the vasculature without being in a lead plate operating room. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 FIG. 1 is a flowchart of a positioning method of an interventional device provided by the embodiment of the application;

[0045] Figure 2 FIG. 2 is a flowchart of a training method of a first generation model provided by the embodiment of the application;

[0046] Figure 3 FIG. 3 is a flowchart of a training method of a second generation model provided by the embodiment of the application;

[0047] Figure 4 FIG. 4 is a registration effect diagram provided by the embodiment of the application;

[0048] Figure 5 FIG. 5 is an application scenario diagram of acquiring a third medical image provided by the embodiment of the application;

[0049] Figure 6a FIG. 6 is a schematic diagram of a human aortic dissection provided by the embodiment of the application;

[0050] Figure 6b FIG. 7 is a schematic diagram of a vascular stent after expansion in a blood vessel provided by the embodiment of the application;

[0051] Figure 7 FIG. 8 is a structural diagram of a positioning device of an interventional device provided by the embodiment of the application;

[0052] Figure 8 FIG. 9 is a structural diagram of an electronic device provided by the embodiment of the application;

[0053] Figure 9 FIG. 10 is another structural diagram of an electronic device provided by the embodiment of the application. DETAILED DESCRIPTION

[0054] The technical solutions in the embodiments of the present application will be apparently and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all the other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0055] It should be understood that each step recorded in the method embodiments of the present disclosure can be executed in different order and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the execution of the shown steps. The scope of the present disclosure is not limited in this respect.

[0056] The term "comprising" and variations thereof as used in the present disclosure are open-ended, that is "including but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Related definitions will be given in the following description.

[0057] In the related art, in order to accurately position the lesion site and perform the interventional operation, it is usually necessary to perform the operation in a lead plate operating room capable of preventing the penetration of X-rays, and the doctor needs to wear a heavy lead suit. During the operation, the vascular stent is sent into the patient's vascular system with the aid of a delivery system, and the patient needs to be continuously injected with contrast agent to display the position and state of the implanted interventional device in the blood vessels. At the same time, the vascular stent needs to be X-rayed multiple times during and after release to determine the position of the vascular stent and the release effect. Multiple X-ray penetrations are undoubtedly very disadvantageous to the patient and the operating medical staff, and the toxicity and allergenicity of the contrast agent to the human body also increase the iatrogenic risk of the patient. In addition, the lead plate operating room is expensive and limited in quantity, and the queuing for the lead plate operating room for interventional operation makes many medical staff and patients wait until late at night, which is undoubtedly a great waste of human resources, and also brings uncertainty to the treatment of diseases, and sometimes patients have anxiety during the waiting process, which leads to the occurrence of malignant cardiovascular events. Further, since surgical operation mainly intervenes in the proximal vascular region of the lesion, and interventional operation treats the distal vascular region of the lesion. Therefore, if the timing of surgical operation and the time when the lead plate operating room is available cannot be matched, the patient under general anesthesia will be in a state of maintaining anesthesia while waiting for the lead plate operating room, which brings a heavy burden to the patient, medical staff and medical system.

[0058] Therefore, there is an urgent need for a new method for accurately positioning the interventional device in the vascular system without the lead plate operating room.

[0059] To solve the technical problems in the related art, an interventional device positioning method is provided in the embodiments of the present application, please refer to Figure 1 , Figure 1 is a flowchart of the interventional device positioning method provided by the embodiments of the present application, and the method comprises steps 101 to 104.

[0060] Step 101: Obtain a first medical image of a vessel system in a target patient, and generate a corresponding first vessel system model based on the first medical image.

[0061] In the present embodiment, the first medical image provided by the embodiments of the present application can include an angiography image and a magnetic resonance image. The angiography refers to spiral CT scanning after intravenous injection of contrast medium, and in three-dimensional reconstruction, the structures such as skin, muscle and bone that do not need to be displayed are removed, and only the images of three-dimensional blood vessel structure and internal organ structure are displayed. Magnetic resonance is based on the principle of nuclear magnetic resonance (NMR), and according to the different attenuation of the energy released in different structural environments, the electromagnetic waves emitted are detected through an external gradient magnetic field, so that the position and type of atomic nucleus constituting the object can be known, and accordingly the structure inside the object can be drawn. Therefore, the first medical image provided by the embodiments of the present application refers to a 3D image that can display the three-dimensional blood vessel structure of the vessel system in the target patient.

[0062] After obtaining the angiography image and the magnetic resonance image of the vessel system in the target patient, since the angiography image and the magnetic resonance image not only display the three-dimensional blood vessel structure, but also display the image content of the internal organ structure, the embodiments of the present application also need to preprocess the angiography image and the magnetic resonance image to obtain the blood vessel structure image that only belongs to the human vessel system, so as to generate the corresponding three-dimensional first vessel system model.

[0063] Specifically, the preprocessing provided by the embodiments of the present application can include rendering processing, denoising and smoothing processing, contrast enhancement processing, standardization and normalization, and image segmentation processing, so as to accurately remove the blood vessel structure that does not belong to the human vessel system, and generate the three-dimensional first vessel system model.

[0064] In some embodiments, in order to improve the generation efficiency of the three-dimensional first vessel system model, before generating the first vessel system model, the embodiments of the present application can also pre-train a first generation model for automatically generating the corresponding first vessel system model according to the input first medical image. Specifically, please refer to Figure 2 , Figure 2 is a flowchart of the training method of the first generation model provided by the embodiments of the present application, as shown in Figure 2As shown, the training method of the first generation model provided in this embodiment includes steps 201 to 203.

[0065] Step 201, acquiring first initial images of vasculature systems in different patients.

[0066] In this embodiment, the first initial images provided in this embodiment include angiographic images and magnetic resonance images. In order to complete the training of the model, a large number of first initial images of vasculature systems in different patients need to be acquired.

[0067] Step 202, performing rendering processing and segmentation processing on the first initial images to obtain a plurality of first initial three-dimensional vasculature system models corresponding to different patients.

[0068] In this embodiment, since the angiographic images and the magnetic resonance images, that is, the first initial images, not only show the three-dimensional vascular structure, but also show the image content of the internal organ structure, the first initial images need to be rendered and segmented in this embodiment. Processing is needed to remove the vascular structure that does not belong to the human vasculature system to generate a three-dimensional first initial three-dimensional vasculature system model.

[0069] Specifically, before performing the segmentation processing operation on the first initial images, this embodiment can also perform noise removal and smoothing processing, contrast enhancement processing, standardization and normalization, and other operations that can improve the segmentation accuracy and segmentation efficiency of the segmentation processing. In this way, the vascular structure that does not belong to the human vasculature system can be accurately removed to generate a three-dimensional first initial three-dimensional vasculature system model.

[0070] Step 203, taking the first initial images as input and the first initial three-dimensional vasculature system model corresponding to the first initial images as output, training the deep learning neural network model to be trained until convergence, and obtaining the trained first generation model.

[0071] In this embodiment, this embodiment mainly takes the first initial images and the corresponding first initial three-dimensional vasculature system model as training data, and constructs training data sets and validation data sets according to different proportions. The proportion ratio of the training data in the training data set and the validation data set can be pre-set, for example, 8:2 or 9:1, etc. In addition, the deep learning neural network model provided in this embodiment can be any convolutional neural network model that can realize image segmentation function, which is not limited here.

[0072] In this way, the training data set and the validation data set are constructed to train and validate the deep learning neural network model to be trained, which can establish a first generation model that can generate a corresponding three-dimensional human vasculature system model according to the first initial image.

[0073] Specifically, in one case, the manner of training the deep learning neural network model to be trained until convergence can be that the training times reach a preset training times, that is, inputting the first initial image into the deep learning neural network model to be trained for generation processing to obtain a generation result; using a preset loss function to calculate the loss value of the first initial three-dimensional vessel system model corresponding to the input first initial image relative to the generation result, and fine-tuning the model parameters of the deep learning neural network model to be trained according to the loss value; continuously training the deep learning neural network model to be trained and fine-tuning the model parameters using the training data set until the training times reach the preset preset training times, for example, 50000 times or 100000 times, that is, the model converges.

[0074] In another case, the manner of training the deep learning neural network model to be trained until convergence can also be that the accuracy of the generation result of the model reaches a preset accuracy threshold, that is, inputting the first initial image into the deep learning neural network model to be trained for generation processing to obtain a generation result; using a preset loss function to calculate the loss value of the first initial three-dimensional vessel system model corresponding to the input first initial image relative to the generation result, and fine-tuning the model parameters of the deep learning neural network model to be trained according to the loss value; continuously training the deep learning neural network model to be trained and fine-tuning the model parameters using the training data set until the accuracy of the generation result generated by the deep learning neural network model to be trained reaches a preset accuracy threshold, for example, 95% or 98%, that is, the model converges.

[0075] The above only describes two ways of model convergence method, and other methods of determining model convergence can also be used, which are not specifically limited here.

[0076] After obtaining the trained first generation model, the step of generating a first vessel system model corresponding to the first medical image provided by the embodiment can be: calling the trained first generation model to perform generation processing on the first medical image to generate a first vessel system model corresponding to the first medical image. In this way, the generation efficiency of the first vessel system model can be effectively improved.

[0077] In step 102, an interventional device arranged in a blood vessel of a vessel system in the target patient is used to acquire a plurality of continuous frames of second medical images in the blood vessel, and a three-dimensionally reconstructed second vessel system model is generated based on the plurality of continuous frames of second medical images.

[0078] In this embodiment, the second medical image provided by the embodiment can include an intravascular ultrasound image and an optical coherence tomography image. Wherein, the intravascular ultrasound (IVUS) refers to a technique of sending a micro-ultrasound probe into a blood vessel lumen through a catheter to display a blood vessel cross-sectional image, thereby providing an in-vivo blood vessel lumen image. The optical coherence tomography (OCT) is a non-contact and non-invasive optical imaging technology, which uses the interference principle of light to perform high-resolution tomography on biological tissues. Therefore, the second medical image provided by the embodiment also refers to a 3D image capable of displaying the three-dimensional vascular structure of the target patient's vasculature.

[0079] It should be noted that, compared with the first medical image, the second medical image is obtained by sending a micro-probe into the blood vessel lumen, while the first medical image is obtained without sending a micro-probe into the blood vessel lumen. Also, since the second medical image is obtained by sending a micro-probe into the blood vessel lumen, the second medical image can also contain the position information of the micro-probe and the relative position information between the micro-probe and the inner wall of the blood vessel lumen, so that the three-dimensional coordinates of the vasculature can be determined according to the position information, thereby facilitating three-dimensional reconstruction according to the three-dimensional coordinates to generate a second vasculature model corresponding to the second medical image. Further, by using a micro-probe to obtain an ultrasound image or an OCT image in the blood vessel lumen, the risk of nephrotoxicity and fatality of contrast agents can be avoided, and the radiation hazards to patients and medical staff caused by X-ray fluoroscopy can also be avoided, thereby effectively improving the safety of interventional surgery.

[0080] Specifically, the second vasculature model provided by the embodiment can be obtained by three-dimensional reconstruction according to all three-dimensional coordinates of the vasculature, or can be obtained by three-dimensional reconstruction according to the three-dimensional coordinates of several key points of the vasculature. The three-dimensional coordinates of which key points are selected for three-dimensional reconstruction can be set according to actual application requirements, which is not limited here.

[0081] Illustratively, the embodiment can enter the human vasculature through the femoral artery by a micro-probe, and then perform intravascular ultrasound or OCT examination to obtain an ultrasound or OCT image. Then, the three-dimensional coordinates of four vertices in each frame of ultrasound sequence image or each frame of OCT sequence image are extracted by the micro-probe, and a three-dimensional model data is reconstructed by a preset three-dimensional reconstruction algorithm. It should be noted that the ultrasound or OCT sequence image is a group of ultrasound or OCT sequence images acquired by a continuous micro-probe, and an electromagnetic sensor can be provided on the micro-probe, and the electromagnetic sensor can be connected with an electromagnetic positioning system for acquiring the three-dimensional coordinates of the vertices of the ultrasound sequence image or the OCT sequence image.

[0082] In some embodiments, also in order to improve the generation efficiency of the generated three-dimensional second vasculature model, before generating the second vasculature model, the embodiment can also pre-train a second generation model for automatically generating a corresponding second vasculature model according to an input second medical image. Specifically, please refer to Figure 3 , Figure 3 is a flowchart of a training method of a second generation model provided by an embodiment of the present application, as shown in Figure 3 the training method of the second generation model provided by the embodiment includes steps 301 to 304;

[0083] Step 301, using an interventional device to enter the human vasculature to obtain a plurality of frames of second initial images of the vasculature in the body of different patients.

[0084] In the embodiment, the second initial image provided by the embodiment includes intravascular ultrasound images and optical coherence tomography images. The interventional device provided by the embodiment can be a microprobe, and specifically the microprobe can be sent into the blood vessel lumen through catheter technology to perform intravascular ultrasound or OCT examination through the intravascular microprobe to obtain ultrasound or OCT images, i.e. the second initial image.

[0085] Step 302, according to the first position information of the interventional device in each frame of the second initial image and the second position information of the interventional device relative to the intravascular wall, determining the three-dimensional coordinates of the vasculature in each frame of the second initial image.

[0086] In the embodiment, since the interventional device provided by the embodiment is a microprobe, the relative second position information between the intravascular wall and the microprobe in each frame of the second initial image collected by the interventional device can be determined, and then the starting position of the interventional device entering the blood vessel lumen can be taken as the origin to construct a three-dimensional coordinate system, so that the three-dimensional coordinates of the vasculature in different second initial images can be calculated according to the first position information of the interventional device in each frame of the second initial image (which can be calculated according to the moving distance of the interventional device or the relative distance from the origin) and the relative second position information between the intravascular wall and the microprobe in each frame of the second initial image.

[0087] Step 303, based on the three-dimensional coordinates of the vasculature in the plurality of frames of the second initial images, performing three-dimensional reconstruction on the vasculature in the body of different patients to obtain corresponding second three-dimensional human vasculature models of different patients.

[0088] In this embodiment, after obtaining the three-dimensional coordinates of the vessel system in each frame of the second initial image, the three-dimensional coordinates of the vessel system in the target patient can be determined, and then the three-dimensional coordinates are processed by a three-dimensional reconstruction algorithm to obtain a corresponding second three-dimensional human vessel system model.

[0089] In step 304, the continuous multiple frames of the second initial images corresponding to different patients are input, the second initial three-dimensional vessel system model corresponding to the continuous multiple frames of the second initial images is output, the deep learning neural network model to be trained is trained until convergence, and a trained second generation model is obtained.

[0090] In this embodiment, the second initial images and the corresponding second initial three-dimensional vessel system models of the continuous multiple frames are mainly used as training data, and training data sets and validation data sets are constructed according to different proportions. The proportion ratio of the training data in the training data set and the validation data set can be pre-set, for example, 8:2 or 9:1. In addition, the deep learning neural network model provided in this embodiment can be any convolutional neural network model that can realize three-dimensional reconstruction function, which is not specifically limited here.

[0091] In this way, the training and validation of the deep learning neural network model to be trained by the constructed training data set and validation data set can establish a second generation model that can generate a corresponding three-dimensional human vessel system model according to the continuous multiple frames of the second initial images.

[0092] Specifically, in one case, the way to train the deep learning neural network model to be trained until convergence can be that the training times reach a preset training times, that is, the continuous multiple frames of the second initial images are input into the deep learning neural network model to be trained for generation processing to obtain a generation result, a preset loss function is used to calculate the loss value of the second initial three-dimensional vessel system model corresponding to the input continuous multiple frames of the second initial images relative to the generation result, and the model parameters of the deep learning neural network model to be trained are fine-tuned according to the loss value. The training data set is continuously used to train the deep learning neural network model to be trained and fine-tune the model parameters until the calculation training times reach the preset preset training times, for example, 50000 times or 100000 times, that is, the model converges.

[0093] In another case, the manner of training the deep learning neural network model to be trained until convergence can also be that the accuracy of the generated result of the model reaches a preset accuracy threshold, that is, the second initial image of the continuous multiple frames is input into the deep learning neural network model to be trained for generation processing to obtain a generated result; a preset loss function is used to calculate the loss value of the second initial three-dimensional vessel system model corresponding to the input continuous multiple frames of the second initial image relative to the generated result, and the model parameters of the deep learning neural network model to be trained are fine-tuned according to the loss value; the training data set is continuously used to train the deep learning neural network model to be trained and fine-tune the model parameters, until the accuracy of the generated result generated by the deep learning neural network model to be trained reaches a preset accuracy threshold, for example, 95% or 98%, and then it is determined that the model converges.

[0094] Only two ways of model convergence are described above, and other methods of determining model convergence can also be used, which are not specifically limited here.

[0095] After obtaining the trained second generation model, the step of generating a three-dimensional reconstructed second vessel system model based on the continuous multiple frames of the second medical images provided by the embodiment can be specifically: calling the trained second generation model to generate and process the continuous multiple frames of the second medical images to generate a second vessel system model corresponding to the continuous multiple frames of the second medical images. In this way, the generation efficiency of the second vessel system model can be effectively improved.

[0096] Among them, the deep learning neural network model to be trained used by the embodiment in the process of training the first generation model and the second generation model can be a 3D U-Net model, or other neural network models capable of generating 3D models, which are not specifically limited here.

[0097] It should be noted that the steps 101 to 102 provided by the embodiment are steps performed before the interventional surgery, and the steps 103 and the steps after the step 103 are steps performed during the interventional surgery.

[0098] After the trained first and second generation models are obtained, the first medical image (angiography image or magnetic resonance image) of the target patient's vasculature system can be input into the trained first generation model, and the second medical image (intravascular ultrasound image or optical coherence tomography image) of the target patient's vasculature system can be input into the trained second generation model in the application scenario, so that the first and second vasculature system models can be quickly and accurately obtained, and subsequent registration processing can be facilitated, and the position of the interventional device can be mapped to the first vasculature system model in real time during the interventional surgery, so as to provide the navigation information of the interventional device in the target patient's vasculature system for the clinician, and the clinician can accurately position the lesion site and guide the interventional surgery, thereby breaking the hard condition barrier that the interventional device must be used under the protection of a lead plate, so that the interventional surgery can be performed without the need for a fluoroscope in a general operating room, and there is no need to queue for a small number of lead plate operating rooms with high costs, thereby reducing the burden on patients, medical workers and the medical system, and improving the efficiency of interventional surgery.

[0099] Step 103: performing registration processing on the first vasculature system model and the second vasculature system model to obtain a target registration matrix corresponding to the target patient.

[0100] In order to display the navigation information of the interventional device in the blood vessel on the first vasculature system model in real time, the second vasculature system model needs to be accurately fused with the first vasculature system model, so that the positioning information displayed on the second vasculature system model can be equivalent to that displayed on the first vasculature system model. Specifically, the first vasculature system model and the second vasculature system model can be fused into one by performing registration processing on the first vasculature system model and the second vasculature system model, so that the position of any key point on the second vasculature system model can be accurately mapped to the first vasculature system model.

[0101] In the embodiment, the registration processing can include feature extraction, feature matching, transformation model estimation, image resampling, optimization and the like, and the implementation is as follows:

[0102] Step 1: feature extraction, which is to extract key points or regions that can be used for registration from images or point clouds. Common feature extraction methods include but are not limited to corner detection, edge detection, region detection, etc. The first vasculature system model and the second vasculature system model are respectively subjected to feature extraction to obtain corresponding feature points or feature descriptors.

[0103] Step two: feature matching, the purpose is to match the feature points or feature descriptors in the model to be registered and the target model, find the corresponding point pair, the commonly used methods include but are not limited to nearest neighbor matching, RANDSAC (random sample consensus) and the like. A set of matching point pairs is obtained by performing feature matching on the two sets of feature points or feature descriptors obtained in step one.

[0104] Step three: transformation model estimation, the purpose is to estimate the spatial transformation model required for aligning the model to be registered to the target model according to the matching point pairs, the commonly used transformation models include but are not limited to rigid transformation, affine transformation, non-rigid transformation. The commonly used estimation methods include but are not limited to least square method, iterative optimization and the like. The transformation model required for aligning the second vascular system model to the first vascular system model can be output by performing transformation model estimation on the matching point pairs obtained in step two.

[0105] Step four: image resampling, the purpose is to resample the model to be registered according to the estimated transformation model, and generate the aligned model. The commonly used interpolation methods include but are not limited to nearest neighbor interpolation, bilinear interpolation, bicubic interpolation and the like. The second vascular system model is resampled by the transformation model output in step three, and a new model aligned to the first vascular system model is output.

[0106] Step five: optimization, the purpose is to further improve the registration accuracy by iteratively optimizing the transformation model. The commonly used optimization methods include but are not limited to gradient descent method, genetic algorithm, similarity metric algorithm and the like. The similarity metric includes but is not limited to the following methods:

[0107] Mean square error (MSE): suitable for strongly similar images, where I ref ,I float are the intensities of the model to be registered and the target model respectively, (x i ,y i ) and (x i ′,y i ′) are the corresponding point coordinates.

[0108] Mutual information (MI): suitable for multi-modal image registration, where p(a,b) is the joint probability, p ref (a), p float (b) is the marginal probability distribution.

[0109] A highly accurate transformation model is obtained by the optimization method, and the transformation model is the registration matrix for spatially aligning the second vascular system model to the first vascular system model. The specific registration process is as shown in Figure 4 Figure 4 ​is a registration effect schematic diagram provided by the embodiment of the present application, wherein the model to be registered is the second vasculature model, the target model is the first vasculature model, and the result after registration is the conversion of the model to be registered to the spatial coordinate system of the target model.

[0110] In an embodiment, the registration processing provided by the embodiment can include rigid registration processing, and specifically, the step of performing registration processing on the first vasculature model and the second vasculature model to obtain a target registration matrix corresponding to the target patient can specifically be: performing rotation and translation operations on the second vasculature model to spatially align the second vasculature model with the first vasculature model; and determining a target registration matrix corresponding to the target patient according to the second vasculature model before and after spatial alignment.

[0111] wherein the rigid transformation can be defined as: p' = R * p + t, wherein p is the point coordinate in the second vasculature model, p' is the point coordinate after transformation, R is a rotation matrix, and t is a translation vector.

[0112] In this way, the rigid registration processing can achieve spatial alignment between different modal images, thereby ensuring the accuracy of mapping the position of any key point on the second vasculature model to the first vasculature model, and further providing the clinician with accurate navigation positioning information of the interventional device in the vasculature.

[0113] In another embodiment, the registration processing provided by the embodiment can also include non-rigid registration processing, and the step of performing registration processing on the first vasculature model and the second vasculature model to obtain a target registration matrix corresponding to the target patient can specifically be: performing elastic transformation and thin-plate spline operations on the second vasculature model to spatially align the second vasculature model with the first vasculature model; and determining a target registration matrix corresponding to the target patient according to the second vasculature model before and after spatial alignment.

[0114] wherein the non-rigid transformation is divided into B-spline transformation and thin-plate spline (TPS), and the B-spline transformation formula is:

[0115]

[0116] wherein c i,j,k is the control point displacement, β i (u), β j (v), and β k (w) are B-spline basis functions.

[0117] The thin-plate spline formula is:

[0118]

[0119] wherein c i is a control point, w i is a weight, and Φ(r) is a radial basis function.

[0120] Thus, the non-rigid registration processing can solve the problem of low registration accuracy caused by tracheal movement and deformation, thereby further ensuring the accuracy of mapping the position of any key point on the second vasculature model to the first vasculature model, and effectively improving the accuracy of providing the clinician with navigation positioning information of the interventional device in the first vasculature model.

[0121] In addition, the registration processing provided by the embodiment can also include affine transformation registration processing. Specifically, the affine transformation provided by the embodiment is a three-dimensional affine transformation, and the formula thereof is:

[0122]

[0123] wherein a ij (i,j∈(1~3)) is an element of a linear transformation matrix, t x , t y , t z is a translation.

[0124] In step 104, a third medical image of the vasculature in the target patient's body is acquired in real time by the interventional device, and the third medical image is mapped to the first vasculature model based on the target registration matrix, so as to display the position of the interventional device on the first vasculature model in real time.

[0125] Specifically, the second medical image is mainly acquired by the interventional device before the interventional operation, and the third medical image is mainly acquired in real time by the interventional device during the interventional operation.

[0126] After the target registration matrix is obtained through the registration processing, the positioning information of any point in the third medical image acquired in real time can be mapped onto the first vessel system model through the target registration matrix. Specifically, the embodiment mainly maps the positioning information of the interventional device in the third medical image onto the first vessel system model, so that the accurate positioning information of the interventional device can be displayed on the first vessel system model in real time, and the navigation information of the interventional device in the target patient's blood vessel is provided for the clinician, thereby the clinician can accurately position the lesion site and guide the interventional operation, so as to break the hard condition barrier that the interventional device must be used under the protection of the lead plate, so that the interventional operation can be performed without the perspective of the perspective machine in the ordinary operating room, without the need to queue for a small number of lead plate operating rooms with high cost, thereby reducing the burden of patients, medical workers and medical systems, and improving the efficiency of interventional operation.

[0127] Specifically, the acquisition process of the third medical image of the embodiment can be as shown in Figure 5 Figure 5 is an application scenario diagram for acquiring the third medical image provided by the embodiment of the present application. Specifically, the embodiment can be equipped with a positioning device outside the patient's body, the positioning device is connected with the sensor through the connecting line, the sensor is fixed on the probe of the interventional device, the third medical image is acquired through the probe, and the position of the probe of the interventional device in the coordinate system of the positioning device in real time is obtained: p' = M1·M2·p, wherein p is the coordinate position of the probe of the interventional device in the image, M1 is the transformation matrix from the image to the sensor coordinate system, and M2 is the transformation matrix from the sensor coordinate system to the coordinate system of the positioning device. The above transformation can obtain the coordinates of the third medical image in the coordinate system of the positioning device.

[0128] ​In some embodiments, in actual application scenarios, the embodiment can not only display the navigation positioning information of the interventional device in the blood vessel in real time, but also control the interventional device to realize different functions. Specifically, the interventional device provided by the embodiment can include a probe, a blood vessel stent and a delivery rod, the probe is arranged at the end of the delivery rod, and the blood vessel stent is arranged around and tightly on the outer side wall of the delivery rod. Wherein, the probe provided by the embodiment can be a micro probe that can be delivered into the blood vessel cavity; the blood vessel stent can be a mesh cover with elastic deformation function (expandable and contractible), cylindrical and hollow structure, and the mesh cover of the blood vessel stent has the characteristic of isolating blood passing through, so as to ensure that blood can only pass through the hollow position of the mesh cover. In addition, the length of the blood vessel stent can be customized according to the length of the lesion area in the blood vessel; the delivery rod can be a micro catheter that can be delivered into the blood vessel cavity, the delivery rod is designed as a hollow structure, and a data transmission line connected with the probe can be arranged in the hollow structure of the delivery rod. One end of the data transmission line is connected with the probe, and the other end can be connected with a terminal device such as a PC end, the PC end has a display device for displaying a first vasculature model, and for displaying the position of the probe on the first vasculature model in real time. At the same time, a micro motor can also be arranged in the hollow structure of the delivery rod, the micro motor is used to control the bending angle of the end of the delivery rod, so that the end of the delivery rod can be delivered towards different angles / directions, avoiding the delivery rod touching and damaging the inner wall of the vasculature.

[0129] Specifically, the positioning method of the interventional device provided by the embodiment can also include: in the case that the position of the probe displayed on the first vasculature model in real time is a target position, displaying prompt information of reaching the target position; in response to a blood vessel stent release instruction, controlling the blood vessel stent to release from the outer side wall of the delivery rod and perform expansion operation.

[0130] The target position can be a proximal position of the intravascular lesion region, and the length of the vascular stent can be set by pre-calculating the distance between the proximal position and the distal position of the intravascular lesion region, for example, the length of the vascular stent can be selected by setting scale lines on the delivery rod, and then recording the position reached by the probe of the interventional device when the interventional device collects the second intravascular medical image before the interventional operation (when the interventional device collects the second intravascular medical image, the interventional device only includes the probe and the delivery rod, and at this time the delivery rod does not set the vascular stent), specifically: when the probe reaches the distal end of the intravascular lesion region, record the corresponding scale D1 of the delivery rod, when the probe reaches the proximal end of the intravascular lesion region, record the corresponding scale D2 of the delivery rod, then the length of D2-D1 can be determined as the length of the intravascular lesion region, at this time the vascular stent corresponding to the length (D2-D1) can be selected to be set on the delivery rod, and finally in the process of the interventional operation, the delivery rod provided with the vascular stent and the probe is delivered into the vascular lumen of the human body vasculature of the target patient.

[0131] It should be noted that the proximal position and the distal position of the intravascular lesion region can be determined before the interventional operation, that is, step 102 collects the second intravascular medical image by the interventional device. Figure 6a , Figure 6a is a schematic diagram of the human aortic dissection provided by the embodiment of the present application, as shown in Figure 6a , point A in the figure is the proximal position of the intravascular lesion region, point B is the distal position of the intravascular lesion region, and the length AB between points A and B is the length of the intravascular lesion region, which is also the shortest length of the vascular stent, that is, the length between the blood vessels needs to be at least greater than the length AB.

[0132] When the position of the probe displayed in real time on the first medical image is the target position, the vascular stent can be released by the vascular stent release instruction, and the vascular stent can be controlled to release from the outer side wall of the delivery rod and perform expansion operation, and after the release, the vascular stent performs expansion operation and tightly expands against the inner wall of the blood vessel, so that the blood vessel wall of the blood vessel lesion region of the human aortic dissection becomes a plugging state, and the final effect is shown in Figure 6b , Figure 6b is a schematic diagram of the vascular stent after expansion in the blood vessel provided by the embodiment of the present application.

[0133] In some embodiments, in order to improve the accuracy of the positioning of the interventional device, before the step of controlling the release of the stent from the outer side wall of the delivery rod and the expansion operation of the stent in response to the stent release instruction, the method for positioning the interventional device further comprises: acquiring the intravascular image captured by the probe; and in the case that the position of the probe in the intravascular image is not the target position and the stent release instruction is detected, stopping the stent release operation and displaying a prompt message indicating a position error.

[0134] In this way, through the double verification (the verification of the real-time display of the navigation positioning information of the probe in the intravascular image in the first medical image, and the verification of the intravascular imaging image), the accuracy of the positioning of the interventional device in the blood vessel can be effectively ensured.

[0135] As an optional embodiment, in order to further improve the accuracy of the positioning of the interventional device, the embodiment can further combine the scale information recorded by the scale line arranged on the delivery rod to further verify the accuracy of the positioning of the interventional device in the case of the above-mentioned double verification, so as to achieve the purpose of triple verification. Specifically, the distance D3 between the proximal end of the stent and the probe can be determined in advance, when the interventional device loaded with the stent reenters the target patient's vasculature, the positioning information of the probe of the interventional device is displayed in real time on the first vasculature model, and the operator can judge the position of the intravascular stent in the human vasculature in real time through the real-time display of the positioning information of the probe; then, when the probe of the interventional device reaches the proximal end of the intravascular lesion region, the specific position of the interventional device can be determined through the scale D2 on the delivery rod, and the value of D2+D3 is taken as the reference distance of the intravascular stent reaching the proximal end of the intravascular lesion region, so as to determine the position of the intravascular stent in the human vasculature; in addition, the position of the intravascular stent in the human vasculature can also be verified by combining the intravascular ultrasound image or the OCT image collected by the probe of the interventional device in real time, so as to ensure that the stent can accurately reach and completely cover the intravascular lesion region.

[0136] When the stent reaches and completely covers the intravascular lesion region, the release of the stent from the outer side wall of the delivery rod and the expansion operation of the stent can be controlled through the stent release instruction, after the release of the stent, the delivery rod can be controlled to slowly retreat, and in the process of the retreat, the placement position and the apposition of the stent can be observed and detected through the real-time acquisition of the intravascular ultrasound image or the OCT image by the probe, so as to effectively ensure the accuracy of the interventional operation in the ordinary operating room and avoid the exposure of the patient and the medical staff to the radiation environment.

[0137] To sum up, by using the positioning method of the interventional device provided in the embodiment, the method comprises the following steps: acquiring a first medical image of a vessel system in a target patient, and generating a first vessel system model based on the first medical image; acquiring a second medical image of a plurality of continuous frames in the blood vessel by using an interventional device arranged in the blood vessel of the vessel system in the target patient, and generating a second vessel system model based on the second medical image; performing registration processing on the first vessel system model and the second vessel system model to obtain a target registration matrix corresponding to the target patient; acquiring a third medical image in the blood vessel of the vessel system in the target patient by using the interventional device in real time, and mapping the third medical image to the first vessel system model based on the target registration matrix, so as to display the position of the interventional device on the first vessel system model in real time, thereby achieving the purpose of accurately positioning the interventional device in the vessel system without the lead plate operating room, providing navigation information of the interventional device in the blood vessel of the target patient for the clinician, and then facilitating the clinician to accurately position the lesion site and guide the interventional operation, so as to break the hard condition barrier that the interventional device must be used under the protection of the lead plate, and enable the interventional operation to be performed in a general operating room without the fluoroscopy of the fluoroscope, without the need to queue for a small number of lead plate operating rooms with high cost, thereby reducing the burden of patients, medical workers and the medical system, and improving the efficiency of the interventional operation.

[0138] According to the method described in the above embodiment, the present embodiment will be further described from the perspective of the positioning device of the interventional device, which can be implemented as an independent entity or integrated in an electronic device such as a terminal, which can include a mobile phone, a tablet computer and the like.

[0139] Please refer to Figure 7 , Figure 7 is a structural schematic diagram of the positioning device of the interventional device provided in the embodiment, as Figure 7 shown, the positioning device 500 of the interventional device provided in the embodiment comprises a first acquisition module 501, a second acquisition module 502, a registration module 503 and a positioning module 504.

[0140] The first acquisition module 501 is configured to acquire a first medical image of a vessel system in a target patient, and generate a corresponding first vessel system model based on the first medical image, wherein the first medical image comprises an angiogram image and a magnetic resonance image.

[0141] The second acquisition module 502 is configured to acquire a second medical image of a plurality of continuous frames in the blood vessel by using an interventional device arranged in the blood vessel of the vessel system in the target patient, and generate a three-dimensionally reconstructed second vessel system model based on the second medical image of the plurality of continuous frames, wherein the second medical image comprises an intravascular ultrasound image and an optical coherence tomography image.

[0142] a registration module 503, configured to perform a registration process on the first vasculature model and the second vasculature model to obtain a target registration matrix corresponding to the target patient.

[0143] a positioning module 504, configured to acquire a third medical image of a blood vessel in a vasculature in the target patient in real time by using the interventional device, and map the third medical image onto the first vasculature model based on the target registration matrix to display a position of the interventional device on the first vasculature model in real time.

[0144] In some embodiments, the positioning apparatus 500 of the interventional device provided in the embodiment further includes a first training module and a second training module.

[0145] The first training module is configured to acquire first initial images of vasculatures in different patients, the first initial images including angiography images and magnetic resonance images; perform rendering processing and segmentation processing on the first initial images to obtain a plurality of first initial three-dimensional vasculature models corresponding to different patients; train a deep learning neural network model to be trained until convergence by taking the first initial images as input and taking first initial three-dimensional vasculature models corresponding to the first initial images as output, and obtain a trained first generation model.

[0146] At this time, the first acquisition module 501 provided in the embodiment can be further configured to call the trained first generation model to perform generation processing on the first medical image to generate a first vasculature model corresponding to the first medical image.

[0147] The second training module is configured to acquire second initial images of vasculatures in different patients in a continuous manner by using an interventional device to enter a human vasculature, the second initial images including intravascular ultrasound images and optical coherence tomography images; determine three-dimensional coordinates of a vasculature in each frame of the second initial images according to first position information of the interventional device in each frame of the second initial images and second position information of the interventional device relative to a blood vessel wall; perform three-dimensional reconstruction on vasculatures in different patients based on the three-dimensional coordinates of the vasculatures in the continuous multiple frames of the second initial images to obtain second three-dimensional human vasculature models corresponding to different patients; train a deep learning neural network model to be trained until convergence by taking the continuous multiple frames of the second initial images corresponding to different patients as input and taking second initial three-dimensional vasculature models corresponding to the continuous multiple frames of the second initial images as output, and obtain a trained second generation model.

[0148] At this time, the second acquisition module 502 provided in this embodiment can also be used to call the trained second generation model to perform generation processing on the continuous multiple frames of the second medical images, and generate a second vessel system model corresponding to the continuous multiple frames of the second medical images.

[0149] In some embodiments, the registration processing provided in this embodiment includes rigid registration processing, and the registration module 503 is specifically configured to: perform rotation and translation operations on the second vessel system model to spatially align the second vessel system model with the first vessel system model; and determine a target registration matrix corresponding to the target patient according to the second vessel system model before and after spatial alignment.

[0150] In other embodiments, the registration processing provided in this embodiment also includes non-rigid registration processing, and the registration module 503 is specifically configured to: perform elastic transformation and thin plate spline operations on the second vessel system model to spatially align the second vessel system model with the first vessel system model; and determine a target registration matrix corresponding to the target patient according to the second vessel system model before and after spatial alignment.

[0151] As an optional embodiment, the interventional device provided in this embodiment can include a probe, a vascular stent, and a delivery rod, the probe is arranged at the end of the delivery rod, and the vascular stent is arranged around and tightly against the outer sidewall of the delivery rod. The positioning device 500 of the interventional device provided in this embodiment can further include a first prompting module, a control module, a third acquisition module, and a second prompting module.

[0152] The first prompting module is configured to, in a case where the position of the probe displayed on the first vessel system model in real time is a target position, display prompt information of reaching the target position.

[0153] The control module is configured to, in response to a vascular stent release instruction, control the vascular stent to be released from the outer sidewall of the delivery rod and perform an expansion operation.

[0154] Optionally, the intervention device further comprises a stent fixing structure, the delivery rod is provided with a concave fixing groove at a position away from the end portion by a preset distance, the stent fixing structure is arranged in the fixing groove or is used for fixing the blood vessel stent in the fixing groove of the delivery rod, and one end of the blood vessel stent is arranged against a side wall of the fixing groove away from the end portion. The stent fixing structure has a cylindrical hollow structure, and an inner diameter of the stent fixing structure is greater than an outer diameter of the delivery rod. Specifically, the control module is further used for controlling the stent fixing structure to move away from the end portion to release the blood vessel stent from a position close to the end portion, so that the blood vessel stent starts to expand after being separated from the stent fixing structure.

[0155] The third acquisition module is configured to acquire the intravascular image captured by the probe.

[0156] The second prompting module is configured to, in a case where the position of the probe in the intravascular image is not the target position and a blood vessel stent release instruction is detected, stop performing a blood vessel stent release operation and display prompt information indicating a position error.

[0157] In a specific implementation, each of the above modules and / or units can be implemented as an independent entity, or can be combined with any other module and / or unit to be implemented as the same or one or more entities. The specific implementation of each of the above modules and / or units can refer to the method embodiments, and the beneficial effects that can be achieved can refer to the beneficial effects in the method embodiments, which will not be described here again.

[0158] In addition, please refer to Figure 8 , Figure 8 is a structural schematic diagram of an electronic device provided by the embodiment of the application. The electronic device can be a mobile terminal such as a smart phone, a tablet computer or the like. As shown in Figure 8 , the electronic device 600 includes a processor 601 and a memory 602. The processor 601 is electrically connected to the memory 602.

[0159] The processor 601 is the control center of the electronic device 600, and connects each part of the electronic device 600 through various interfaces and lines. The processor 601 executes various functions of the electronic device 600 and processes data by running or loading an application stored in the memory 602 and calling data stored in the memory 602, so as to monitor the whole electronic device 600.

[0160] In the embodiment, the processor 601 in the electronic device 600 loads the instructions corresponding to the processes of one or more application programs into the memory 602, and runs the application programs stored in the memory 602 by the processor 601, so as to implement any step in the positioning method of the interventional device provided in the above embodiments.

[0161] The electronic device 600 can implement the steps in any embodiment of the positioning method of the interventional device provided in the embodiments, and thus can implement the beneficial effects of any positioning method of the interventional device provided in the embodiments. Details are described in the above embodiments, and thus will not be described here.

[0162] Please refer to Figure 9 , Figure 9 is another structural schematic diagram of the electronic device provided in the embodiments, as shown in Figure 9 , Figure 9 shows a specific structural block diagram of the electronic device provided in the embodiments, which can be used to implement the positioning method of the interventional device provided in the above embodiments. The electronic device 700 can be a mobile terminal such as a smart phone or a notebook computer.

[0163] The RF circuit 710 is used to receive and send electromagnetic waves, and to convert electromagnetic waves and electrical signals to each other, so as to communicate with a communication network or other devices. The RF circuit 710 can include various existing circuit elements for performing these functions, for example, an antenna, a radio frequency transceiver, a digital signal processor, an encryption / decryption chip, a subscriber identity module (SIM) card, a memory, and the like. The RF circuit 710 can communicate with various networks, such as the Internet, an intranet, a wireless network, or other devices through the wireless network. The wireless network can include a cellular telephone network, a wireless local area network or metropolitan area network. The wireless network can use various communication standards, protocols and technologies, including but not limited to Global System for Mobile Communication (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), Code Division Access (CDMA), Time Division Multiple Access (TDMA), Wireless Fidelity (Wi-Fi) (such as American National Standards Institute / Institute of Electrical and Electronics Engineers (ANSI / IEEE) 802.11a, 802.11b, 802.11g, and / or 802.11n), Voice over Internet Protocol (VoIP), Worldwide Interoperability for Microwave Access (Wi-Max), other protocols for email, instant messaging and short message service, and any other suitable communication protocol, even those not yet developed as well.

[0164] The memory 720 can be used to store software programs and modules, such as the program instructions / modules corresponding to the positioning method of the interventional device in the above embodiments. The processor 780 executes various functions and the positioning of the interventional device by running the software programs and modules stored in the memory 720.

[0165] Memory 720 can include high-speed random access memory and can also include nonvolatile memory, such as one or more magnetic data storage disks, flash memory, or other nonvolatile solid-state memory. In some examples, memory 720 can further include memory that is remote from processor 780, such as the memory memory of a remote server that is connected to electronic device 700 through a network. Examples of such networks include, without limitation, the Internet, an enterprise intranet, a local area network, a wide area network, a mobile communications network, and combinations thereof.

[0166] Input unit 730 can be used to receive input of numbers or character information, and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function controls. Specifically, input unit 730 can include a touch-sensitive surface 731 and other input devices 732. Touch-sensitive surface 731, also known as a touch display or touchpad, can collect touch operations (such as a user's operation on or near the touch-sensitive surface 731 using a finger, a stylus, or any suitable object or accessory) of a user thereon or there near, and drive corresponding connected devices according to a pre-set program. Optionally, touch-sensitive surface 731 can include two parts, a touch detection device and a touch controller. The touch detection device detects the touch position of the user and detects signals caused by touch operations, and transmits the signals to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch coordinates, and sends it to processor 780, and can also receive commands from processor 780 and execute them. In addition, touch-sensitive surface 731 can be implemented in various types, such as resistive, capacitive, infrared, and surface acoustic wave. In addition to touch-sensitive surface 731, input unit 730 can also include other input devices 732. Specifically, other input devices 732 can include one or more of, but are not limited to, a physical keyboard, function keys (such as volume control keys, on / off keys, etc.), a trackball, a mouse, a joystick, and the like.

[0167] The display unit 740 can be used to display information input by a user or information provided to the user, as well as various graphical user interfaces of the electronic device 700, which can be composed of graphics, text, icons, video, and any combination thereof. The display unit 740 can include a display panel 741, which can be configured in the form of an LCD (Liquid Crystal Display), an OLED (Organic Light-Emitting Diode), or the like, optionally. Further, the touch-sensitive surface 731 can cover the display panel 741, and when the touch-sensitive surface 731 detects a touch operation thereon or adjacent thereto, transmit to the processor 780 to determine the type of touch event, and then the processor 780 provides corresponding visual output on the display panel 741 according to the type of touch event. Although in the figure, the touch-sensitive surface 731 and the display panel 741 are implemented as two independent components to realize the input and output functions, in some embodiments, the touch-sensitive surface 731 and the display panel 741 can be integrated to realize the input and output functions.

[0168] The electronic device 700 can further include at least one sensor 750, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor can include an ambient light sensor that can adjust the brightness of the display panel 741 according to the brightness of ambient light, and a proximity sensor that can generate an interrupt when the cover is closed or turned off. As one of the motion sensors, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally three axes), and when at rest, can detect the magnitude and direction of gravity, which can be used for applications such as identifying the posture of the mobile phone (such as switching between landscape and portrait screens, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometers, tapping), and the like; as well as other sensors that the electronic device 700 can be configured, such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, and the like, which will not be described here.

[0169] The audio circuit 760, the speaker 761, and the microphone 762 can provide an audio interface between the user and the electronic device 700. The audio circuit 760 can convert received audio data into an electrical signal, transmit the electrical signal to the speaker 761, and convert the electrical signal into a sound signal output by the speaker 761; on the other hand, the microphone 762 converts the collected sound signal into an electrical signal, which is received by the audio circuit 760 and converted into audio data, which is output to the processor 780 for processing, and then transmitted to another terminal via the RF circuit 710, or output to the memory 720 for further processing. The audio circuit 760 can also include an earphone jack to provide communication between an external earphone and the electronic device 700.

[0170] The electronic device 700 can help the user to receive requests, send information, etc. through the transmission module 770 (e.g., a Wi-Fi module), which provides the user with wireless broadband Internet access. Although the transmission module 770 is shown in the figure, it can be understood that it does not belong to the essential components of the electronic device 700, and can be omitted as needed without changing the essence of the application.

[0171] The processor 780 is the control center of the electronic device 700, which connects various parts of the entire mobile phone through various interfaces and lines, executes various functions of the electronic device 700 and processes data by running or executing software programs and / or modules stored in the memory 720 and calling data stored in the memory 720, thereby monitoring the entire electronic device. Optionally, the processor 780 can include one or more processing cores; in some embodiments, the processor 780 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application program, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 780.

[0172] The electronic device 700 also includes a power supply 790 (such as a battery) for supplying power to various components, and in some embodiments, the power supply can be logically connected to the processor 780 through a power management system, so that the power management system can realize functions such as management of charging, discharging, and power consumption management. The power supply 790 can also include one or more direct or alternating power sources, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and any other components.

[0173] Although not shown, the electronic device 700 also includes a camera (such as a front camera, a rear camera), a Bluetooth module, etc., which will not be described here. In this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal also includes a memory and one or more programs, wherein one or more programs are stored in the memory and are configured to be executed by one or more processors to implement any one of the steps of the positioning method of the intervention device provided by the above embodiments.

[0174] In specific implementation, each of the above modules can be implemented as an independent entity, or can be combined as the same or several entities, and the specific implementation of each of the above modules can be referred to the method embodiments described above, which will not be described here.

[0175] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions or by controlling relevant hardware by the instructions, which can be stored in a computer readable storage medium and loaded and executed by a processor. To this end, the embodiments of the present application provide a storage medium, in which a plurality of instructions are stored, and the instructions can be executed by a processor to implement any step in the positioning method of the intervention device provided by the above embodiments.

[0176] The storage medium can include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0177] Since the instructions stored in the storage medium can execute the steps in any embodiment of the positioning method of the intervention device provided by the embodiments of the present application, the beneficial effects of any positioning method of the intervention device provided by the embodiments of the present application can be achieved, which are described in detail in the above embodiments and will not be repeated here.

[0178] The above describes in detail the positioning method of the intervention device, the device, the electronic device and the storage medium provided by the embodiments of the present application. The principles and implementation manners of the present application are described by applying specific examples. The above embodiment is only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range can be changed, and the above description should not be understood as limiting the present application. Moreover, for those skilled in the art, some improvements and refinements can be made without departing from the principles of the present application, and these improvements and refinements are also regarded as the protection scope of the present application.

Claims

1. A positioning device for an interventional device, characterized in that The method comprises the following steps: A first obtaining module is configured to obtain a first medical image of a vessel system in a target patient, and generate a corresponding first vessel system model based on the first medical image, wherein the first medical image comprises an angiogram and a magnetic resonance image; A second obtaining module is configured to obtain a plurality of second medical images in succession by using an interventional device arranged in the vessel system in the target patient, and generate a three-dimensionally reconstructed second vessel system model based on the plurality of second medical images in succession, wherein the second medical images comprise intravascular ultrasound images and optical coherence tomography images; A registration module is configured to perform registration processing on the first vessel system model and the second vessel system model to obtain a target registration matrix corresponding to the target patient; A positioning module is configured to obtain a third medical image in real time by using the interventional device, and map the third medical image to the first vessel system model based on the target registration matrix to display the position of the interventional device on the first vessel system model in real time.

2. The positioning device of an interventional instrument according to claim 1, characterized in that The positioning device of the interventional device further comprises a first training module; The first training module is configured to obtain a first initial image of a vessel system in different patients, wherein the first initial image comprises an angiogram and a magnetic resonance image; The first initial image is subjected to rendering processing and segmentation processing to obtain a plurality of first initial three-dimensional vessel system models corresponding to different patients; a deep learning neural network model to be trained is trained until convergence by taking the first initial image as input and taking a first initial three-dimensional vessel system model corresponding to the first initial image as output, to obtain a trained first generation model; The first obtaining module is further configured to call the trained first generation model to perform generation processing on the first medical image to generate a first vessel system model corresponding to the first medical image.

3. The positioning device of an interventional instrument according to claim 1, characterized in that The positioning device of the interventional device further comprises a second training module; The second training module is configured to obtain a plurality of second initial images in succession by using an interventional device to enter a vessel system in different patients, wherein the second initial images comprise intravascular ultrasound images and optical coherence tomography images; determine three-dimensional coordinates of the vessel system in each of the second initial images based on first position information of the interventional device in each of the second initial images and second position information of the interventional device relative to a vessel wall; perform three-dimensional reconstruction on the vessel system in different patients based on the three-dimensional coordinates of the vessel system in the plurality of second initial images in succession to obtain second three-dimensional human vessel system models corresponding to different patients; train a deep learning neural network model to be trained until convergence by taking the plurality of second initial images in succession corresponding to different patients as input and taking second initial three-dimensional vessel system models corresponding to the plurality of second initial images in succession as output, to obtain a trained second generation model. The second acquisition module is further configured to call the trained second generation model to perform generation processing on the continuous multiple frames of the second medical images to generate a second vessel system model corresponding to the continuous multiple frames of the second medical images.

4. The positioning device of an interventional instrument according to claim 1, characterized in that The registration processing includes rigid registration processing, and the registration module is further configured to perform rotation and translation operations on the second vessel system model to spatially align the second vessel system model with the first vessel system model. The target registration matrix corresponding to the target patient is determined according to the second vessel system model before and after spatial alignment.

5. The positioning device of an interventional instrument according to claim 1, characterized in that, The registration processing further includes non-rigid registration processing, and the registration module is further configured to perform elastic transformation and thin plate spline operations on the second vessel system model to spatially align the second vessel system model with the first vessel system model. The target registration matrix corresponding to the target patient is determined according to the second vessel system model before and after spatial alignment.

6. The positioning device of an interventional instrument according to claim 1, characterized in that The interventional device includes a probe, a vascular stent, and a delivery rod, the probe is arranged at an end of the delivery rod, the vascular stent is arranged around and tightly against an outer sidewall of the delivery rod, and the positioning device of the interventional device further includes a first prompt module and a control module. The first prompt module is configured to display prompt information of reaching the target position when the position of the probe displayed on the first vessel system model in real time is the target position. The control module is configured to control the vascular stent to be released from the outer sidewall of the delivery rod and perform expansion operation in response to a vascular stent release instruction.

7. The positioning device of an interventional instrument according to claim 6, characterized in that The positioning device of the interventional device further includes a third acquisition module and a second prompt module. The third acquisition module is configured to acquire an intravascular image captured by the probe. The second prompt module is configured to stop performing the vascular stent release operation and display prompt information of position error when the position of the probe in the intravascular image is not the target position and a vascular stent release instruction is detected.

8. The positioning device of an interventional instrument according to claim 6, characterized in that The interventional device further includes a stent fixing structure, a concave fixing groove is arranged at a position of the delivery rod at a preset distance from the end, the stent fixing structure is arranged in the fixing groove or is configured to fix the vascular stent in the fixing groove of the delivery rod, and one end of the vascular stent is arranged to abut against a sidewall of the fixing groove away from the end. The control module is further configured to control the stent fixing structure to move towards a direction away from the end to release the vascular stent from a position close to the end, so that the vascular stent starts expansion operation after being released from the fixing of the stent fixing structure.

Images