Method and apparatus for monitoring blood vessel and stent in blood vessel, device, and storage medium

Abstract

Disclosed in the present invention is a method for monitoring a blood vessel and a stent in the blood vessel, comprising: acquiring a target blood vessel fusion image of a target blood vessel at a preset angle, wherein the target blood vessel fusion image is an image obtained by fusing a CT angiography image and an X-ray coronary angiography image of the target blood vessel, and the target blood vessel fusion image comprises tunica media features; at the preset angle, acquiring stent radiographic images of a stent at different positions during the movement of the stent in the target blood vessel; segmenting the stent radiographic images to obtain stent segmented images; and performing fusion display on the target blood vessel fusion image and the stent segmented images to obtain a multi-modal fused image of the target blood vessel and the stent. By means of the method, the display of tunica media features in a target blood vessel on an image is fully considered, so that the condition of the stent in the blood vessel can be more clearly observed from the image. Further disclosed in the present invention are an apparatus for monitoring a blood vessel and a stent in the blood vessel, an electronic device, and a computer-readable storage medium.

Description

Method, device, equipment and storage medium for monitoring blood vessels and stents in blood vessels TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing, in particular to a method and device for monitoring blood vessels and stents in blood vessels, an electronic device and a computer readable storage medium. BACKGROUND

[0002] Coronary heart disease is a serious cardiovascular disease that endangers human health, and its incidence and mortality are still on the rise with the aging of the population and the prevalence of cardiovascular disease risk factors. Percutaneous coronary intervention (PCI) is an important treatment for coronary heart disease. The operation mainly provides information on lumen stenosis through X-ray coronary angiography (XA) to determine the necessity of revascularization and guide stent implantation. Specifically, PCI surgery restores blood flow by expanding the narrowed or blocked part of the coronary artery, reducing the symptoms of myocardial ischemia. During PCI surgery, the doctor makes a small incision on the patient's leg or wrist, then inserts a catheter through the vascular system into the heart. Using medical imaging technology, the doctor locates the narrowed or blocked part of the coronary artery. Then, the doctor expands the narrowed blood vessel using a balloon to restore blood flow. If necessary, the doctor implants a stent in the expanded blood vessel to prevent the blood vessel from narrowing or blocking again.

[0003] Although XA is still considered the "gold standard" for the diagnosis of coronary heart disease, the accuracy of simple XA in assessing the severity of the lesion is often affected by multiple factors such as projection position and lesion site, and it cannot provide local plaque composition information, which has obvious limitations in guiding the optimization of PCI treatment. Intracoronary imaging techniques can help obtain more accurate lesion anatomical morphology and plaque composition information, but the additional invasive procedures will significantly increase the operation time and medical expenses, and lead to potential operation-related complication risks, which limits its clinical application.

[0004] Coronary CT angiography (CCTA) as a non-invasive coronary lesion assessment technique can simultaneously obtain three-dimensional lumen structure and plaque composition information of the coronary vascular tree, and has been widely used for coronary heart disease screening and prognosis assessment. In addition, 3D CCTA can also provide optimal XA projection position information, assist in developing chronic occlusive lesion intervention strategies, and has shown broad application prospects in PCI surgery planning.

[0005] During a PCI operation, a doctor needs to observe the position of a stent in a blood vessel and the state of the blood vessel, and move the stent to adjust the stent to a proper position. The doctor needs to not only observe the state of the blood vessel, but also observe the stent in the blood vessel. However, in the prior art, the state of the inside of the blood vessel in the photographed blood vessel image is not clear enough, and it is difficult to accurately observe the position and shape of the media in the blood vessel and the position and shape of the plaque in the blood vessel. In addition, the stent image obtained by photographing is an X-ray transmission image, and the stent in the photographed image is not clear enough. Therefore, even if the blood vessel image and the stent image obtained by photographing in the prior art are fused, because the blood vessel image itself cannot show the state of the inside of the blood vessel and the stent image itself is not clear enough, it is still difficult to accurately observe the actual position of the stent in the inside of the blood vessel, and the estimation of the position of the stent in the blood vessel is not accurate. SUMMARY

[0006] The present application provides a method for monitoring a blood vessel and a stent in the blood vessel, to solve the problems that the features in the inside of the blood vessel in the blood vessel image are not clear enough, the estimation of the position of the stent in the blood vessel is not accurate enough, and the position relationship between the stent and the media in the blood vessel is not clear enough and is inconvenient to observe.

[0007] In a first aspect, an embodiment of the present application discloses a method for monitoring a blood vessel and a stent in the blood vessel, comprising:

[0008] obtaining a target blood vessel fusion image of the target blood vessel at a preset angle;

[0009] The target blood vessel fusion image is an image obtained by fusing a coronary CT angiography image and an X-ray coronary angiography image of the target blood vessel. The target blood vessel fusion image contains the media features of the target blood vessel, and the media features include the media shape features and / or plaque component features of the target blood vessel.

[0010] obtaining stent transmission image graphs of the stent at different positions during the movement of the stent in the target blood vessel at the preset angle;

[0011] segmenting the stent transmission image graphs to obtain a stent segmentation image, wherein the stent segmentation image contains the shape information of the stent and the position relationship information between the stent and the target blood vessel.

[0012] fusing and displaying the target blood vessel fusion image and the stent segmentation image to obtain a multi-modality fusion image of the target blood vessel and the stent.

[0013] The multi-modality fusion image is used for monitoring the stent in the blood vessel.

[0014] The monitoring method of the blood vessel and the stent in the blood vessel fully considers the characteristics of the tunica media of the blood vessel, the characteristics of the tunica media include morphological characteristics of the tunica media and / or plaque component characteristics, and the obtained target blood vessel fusion image can accurately display the state in the blood vessel. In addition, the target blood vessel fusion image and the stent transmission image are collected at the same preset angle, the positions of the stent in the two images correspond to each other and the positions of the blood vessel correspond to each other, so that the target blood vessel fusion image and the stent transmission image are fused and displayed, and the blood vessel, the internal characteristics of the blood vessel, and the position of the stent in the blood vessel are more accurately displayed in the finally obtained multi-modal fusion image.

[0015] According to another specific embodiment of the present application, the stent is delivered into the target blood vessel by a guide wire to move the stent in the target blood vessel, and the stent segmentation image further contains morphological information of the guide wire, information of the position relationship between the guide wire and the stent and the target blood vessel, and information of the position relationship between the guide wire and the stent.

[0016] According to another specific embodiment of the present application, the coronary CT angiography image and the X-ray coronary angiography image are both collected at a first phase of the target blood vessel; preferably, the first phase is at 75% of the diastolic period or at the end of the diastolic period of the target blood vessel.

[0017] According to another specific embodiment of the present application, the target blood vessel fusion image of the target blood vessel is obtained at a preset angle, specifically including:

[0018] According to the coronary CT angiography image and the X-ray coronary angiography image, feature points are extracted to obtain a CT segmentation feature point image and an X-ray segmentation feature point image of the target blood vessel, respectively;

[0019] The CT segmentation feature point image and the X-ray segmentation feature point image are matched by a deep graph neural network model to obtain a target blood vessel centerline registration result;

[0020] According to the target blood vessel centerline registration result and the CT segmentation feature point image, the coronary CT angiography image and the X-ray coronary angiography image of the target blood vessel are fused by an image mapping method to obtain a target blood vessel fusion image;

[0021] The CT segmentation feature point image includes a three-dimensional blood vessel centerline and three-dimensional blood vessel feature points of the target blood vessel, and the three-dimensional blood vessel feature points contain radius information of the target blood vessel and / or bifurcation position information of the target blood vessel; the X-ray segmentation feature point image includes a two-dimensional blood vessel centerline and two-dimensional blood vessel feature points of the target, and the two-dimensional blood vessel feature points contain radius information of the target blood vessel and / or bifurcation position information of the target blood vessel.

[0022] According to another specific embodiment of the present application, feature point extraction is performed according to the coronary CT angiography image and the X-ray coronary angiography image, and CT segmentation feature point images and X-ray segmentation feature point images of the target blood vessel are obtained, specifically including:

[0023] The coronary CT angiography image and the X-ray coronary angiography image are respectively subjected to image segmentation, and CT angiography vessel segmentation result images and X-ray vessel segmentation result images of the target blood vessel are obtained;

[0024] The CT angiography vessel segmentation result images and the X-ray angiography vessel segmentation result images are respectively subjected to feature point extraction, and CT segmentation feature point images and X-ray segmentation feature point images of the target blood vessel are obtained;

[0025] The CT angiography vessel segmentation result images include the lumen morphology and the vascular media features of the target blood vessel; and the X-ray vessel segmentation result images include the lumen morphology of the target blood vessel.

[0026] According to another specific embodiment of the present application, the image mapping method is pixel mapping.

[0027] According to another specific embodiment of the present application, the blood vessel and stent monitoring method further includes:

[0028] The multi-modal fusion images corresponding to different positions of the current stent in the target blood vessel are sequentially displayed in the first display area, so as to display the state of the target blood vessel and the position of the stent in the target blood vessel in real time during the movement of the stent in the target blood vessel.

[0029] According to another specific embodiment of the present application, the stent transmission image maps at different positions of the stent during the movement of the stent in the target blood vessel are obtained at a preset angle, specifically including:

[0030] The stent transmission image maps at different positions of the stent during the movement of the stent in the target blood vessel are obtained at a preset frequency; or, the movement state of the stent is detected during the movement of the stent in the target blood vessel, and when it is detected that the stent has moved, the stent transmission image maps at different positions of the stent in the target blood vessel are obtained.

[0031] According to another specific embodiment of the present application, the stent transmission image map is segmented to obtain a stent segmentation image, specifically including: the stent transmission image map is segmented by an image segmentation model to obtain the stent segmentation image; preferably, the image segmentation model includes a preliminary segmentation network and a generative adversarial network, the preliminary segmentation network is used for segmenting the stent transmission image map to obtain a preliminary stent segmentation result image, and the generative adversarial network is used for repairing the preliminary stent segmentation result image to obtain the stent segmentation image.

[0032] According to another specific embodiment of the present application, the image segmentation model further comprises a convergence distribution module, which is used to connect the codec layer in the preliminary segmentation network and / or the codec layer in the generative adversarial network, so as to fuse the stent feature data in the stent transmission image.

[0033] According to another specific embodiment of the present application, the training of the image segmentation model is further included, wherein the loss function used in the training of the image segmentation model comprises a preliminary segmentation loss of the preliminary segmentation network, a generative network loss of the generative adversarial network and an adversarial loss of the generative adversarial network.

[0034] According to another specific embodiment of the present application, the target vessel fusion image and the stent segmentation image are fused and displayed to obtain a multi-modal fusion image of the target vessel and the stent, and the fusion display specifically comprises:

[0035] The target vessel fusion image and the stent segmentation image are superimposed to obtain the multi-modal fusion image, or the target vessel fusion image and the stent segmentation image are registered to obtain the multi-modal fusion image.

[0036] According to another specific embodiment of the present application, the target vessel fusion image and the stent segmentation image are registered to obtain the multi-modal fusion image, and the registration specifically comprises:

[0037] The target vessel fusion image and the stent segmentation image are registered by a rigid point cloud registration algorithm to obtain the multi-modal fusion image.

[0038] In a second aspect, the embodiments of the present application disclose a monitoring device for a blood vessel and a stent in the blood vessel, comprising:

[0039] a fusion image acquisition module, configured to acquire a target vessel fusion image of a target vessel at a preset angle;

[0040] The target vessel fusion image is an image obtained by fusing a CT angiography image and an X-ray coronary angiography image of the target vessel, and the target vessel fusion image contains a vascular media feature of the target vessel, wherein the vascular media feature comprises a media morphological feature and / or a plaque component feature of the target vessel.

[0041] a stent image acquisition module, configured to acquire stent transmission image maps of the stent at different positions in the process of moving the stent in the target vessel at the preset angle;

[0042] a stent segmentation module, configured to segment the stent transmission image maps to obtain a stent segmentation image, wherein the stent segmentation image contains morphological information of the stent and information about the positional relationship between the stent and the target vessel.

[0043] The image fusion module is configured to fuse and display the target blood vessel fusion image and the stent segmentation image to obtain a multi-modal fusion image of the target blood vessel and the stent.

[0044] The multi-modal fusion image is used to observe the stent in the blood vessel.

[0045] The technical scheme, the monitoring device for the blood vessel and the stent in the blood vessel can fully consider the characteristics of the tunica media of the blood vessel, the characteristics of the tunica media include the morphological characteristics of the tunica media and / or the plaque component characteristics, and the obtained target blood vessel fusion image can accurately display the state in the blood vessel. In addition, the target blood vessel fusion image and the stent transmission image are collected at the same preset angle, the positions of the stent and the blood vessel in the two images correspond to each other, the fusion display of the target blood vessel fusion image and the stent transmission image is facilitated, and the multi-modal fusion image finally obtained can more accurately display the blood vessel, the internal characteristics of the blood vessel, and the position of the stent in the blood vessel.

[0046] In a third aspect, an embodiment of the present application discloses an electronic device, comprising a processor and a memory, and the memory stores at least one instruction, and the at least one instruction is executed by the processor to implement the monitoring method for the blood vessel and the stent in the blood vessel in any of the preceding embodiments.

[0047] The technical scheme, the monitoring device for the blood vessel and the stent in the blood vessel can fully consider the characteristics of the tunica media of the blood vessel, the characteristics of the tunica media include the morphological characteristics of the tunica media and / or the plaque component characteristics, and the obtained target blood vessel fusion image can accurately display the state in the blood vessel. In addition, the target blood vessel fusion image and the stent transmission image are collected at the same preset angle, the positions of the stent and the blood vessel in the two images correspond to each other, the fusion display of the target blood vessel fusion image and the stent transmission image is facilitated, and the multi-modal fusion image finally obtained can more accurately display the blood vessel, the internal characteristics of the blood vessel, and the position of the stent in the blood vessel.

[0048] In a fourth aspect, an embodiment of the present application discloses a computer readable storage medium, and the computer readable storage medium stores at least one instruction, and the at least one instruction is executed to implement the monitoring method for the blood vessel and the stent in the blood vessel in any of the preceding embodiments.

[0049] The technical scheme is adopted, the computer readable storage medium fully considers the characteristics of the tunica media of the blood vessel, the characteristics of the tunica media include tunica media morphological characteristics and / or plaque component characteristics, and the obtained target blood vessel fusion image can accurately display the state in the blood vessel. In addition, by acquiring the target blood vessel fusion image and the stent transmission image at the same preset angle, the positions of the stent in the two images correspond to each other and the positions of the blood vessel correspond to each other, so that the target blood vessel fusion image and the stent transmission image are fused and displayed, and the obtained multi-modal fusion image is more accurate in displaying the blood vessel, the internal characteristics of the blood vessel, and the position of the stent in the blood vessel.

[0050] It should be noted that the blood vessel and stent monitoring method of the present application does not involve a surgical method, but is based on processing of medical images (including image fusion and real-time display to show animation effects), that is, the direct object is medical images such as CCTA images and XA images.

[0051] The blood vessel and stent monitoring device, electronic device and computer readable storage medium of the present application are also not related to surgical methods, but are related devices based on processing of medical images (including image fusion and real-time display to show animation effects), that is, the direct object is medical images such as CCTA images and XA images.

[0052] The present application has the following advantages:

[0053] The blood vessel and stent monitoring method and the blood vessel and stent monitoring device of the present application acquire the target blood vessel fusion image and the stent transmission image at the same preset angle, the target blood vessel fusion image includes the characteristics of the tunica media of the target blood vessel, the characteristics of the tunica media also include the tunica media morphological characteristics and / or plaque component characteristics of the target blood vessel, and the internal state of the blood vessel and the position of the stent in the blood vessel can be more clearly observed on the multi-modal fusion image obtained by fusing and displaying the target blood vessel fusion image and the stent transmission image.

[0054] The blood vessel and stent monitoring method and the blood vessel and stent monitoring device of the present application further select to acquire at the same phase (i.e., the same blood vessel motion state) of the target blood vessel, so as to ensure more accurate fusion of the target blood vessel fusion image and the stent transmission image.

[0055] In addition, when obtaining the target blood vessel fusion image, the characteristics of the tunica media are fully mapped and fused by using the feature point extraction and the deep graph neural network for feature point matching, so that the target blood vessel fusion image can more accurately display the characteristics in the blood vessel.

[0056] In addition, real-time animation display can be performed on the obtained multi-modal fusion image. During movement of the stent, the image of the stent is collected in real time, and the collected image of the stent is fused and displayed with the target blood vessel fusion image, so that the position movement of the stent in the blood vessel can be monitored in real time while monitoring the shape of the blood vessel and the internal condition information of the blood vessel. The multi-modal fusion image is displayed on the first display area, so that the real-time position movement of the stent in the target blood vessel can be observed more intuitively and clearly, and the real-time display can be performed on the first display area, and the animation effect is displayed on the first display area.

[0057] In addition, image segmentation is performed on the stent transmission image, the stent is segmented by using an image segmentation model, and a stent segmentation image is obtained, and the segmentation result is more accurate. BRIEF DESCRIPTION OF DRAWINGS

[0058] Fig. 1 shows a flowchart of a blood vessel and stent monitoring method in an embodiment of the present application;

[0059] Fig. 2 shows a flowchart of a blood vessel and stent monitoring method in an embodiment of the present application;

[0060] Fig. 3 shows a flowchart of a blood vessel and stent monitoring method in an embodiment of the present application;

[0061] Fig. 4 shows a flowchart of a blood vessel and stent monitoring method in an embodiment of the present application;

[0062] Fig. 5 shows a flowchart of a blood vessel and stent monitoring method in an embodiment of the present application;

[0063] Fig. 6 shows a flowchart of a blood vessel and stent monitoring method in an embodiment of the present application;

[0064] Fig. 7 shows a flowchart of a blood vessel and stent monitoring method in an embodiment of the present application;

[0065] Fig. 8 shows a flowchart of a blood vessel and stent monitoring method in an embodiment of the present application;

[0066] Fig. 9 shows a flowchart of a blood vessel and stent monitoring method in an embodiment of the present application;

[0067] Fig. 10 shows a flowchart of a blood vessel and stent monitoring method in an embodiment of the present application;

[0068] Fig. 11 shows a structural schematic diagram of a blood vessel and stent monitoring device in an embodiment of the present application;

[0069] Fig. 12 shows a structural schematic diagram of a blood vessel and stent monitoring device in another embodiment of the present application;

[0070] FIG. 13 shows a structural schematic diagram of an electronic device in an embodiment of the present application;

[0071] FIG. 14 shows a schematic diagram of image registration in an embodiment of the present application;

[0072] FIG. 15A shows a schematic diagram of a process of image fusion display to obtain a multi-modal fusion image in an embodiment of the present application;

[0073] FIG. 15B shows a schematic diagram of a stent transmission image in an embodiment of the present application;

[0074] FIG. 15C shows a schematic diagram of an X-ray coronary angiography image (XA image) in an embodiment of the present application;

[0075] FIG. 15D shows a schematic diagram of a coronary CT angiography image (CCTA image) in an embodiment of the present application;

[0076] FIG. 15E shows a schematic diagram of a multi-modal fusion image in an embodiment of the present application;

[0077] FIG. 16 shows a schematic diagram of a cardiac cycle time period acquisition window in an embodiment of the present application;

[0078] FIG. 17 shows a structural schematic diagram of an image segmentation model in an embodiment of the present application. DETAILED DESCRIPTION

[0079] The present application is described and explained with additional specificity and detail through the use of the accompanying drawings in which:

[0080] It should be noted that in the description of the application, similar reference numerals and letters in different drawings represent similar elements, so once an element is defined in one drawing, it need not be further defined and explained in subsequent drawings.

[0081] The terms "first", "second", and the like, are used only to distinguish descriptions, and cannot be understood as indicating or implying relative importance.

[0082] It should be noted that, in the description of the embodiments, unless otherwise explicitly specified and limited, the terms "set", "connected", "connection" should be understood in a broad sense, for example, can be fixedly connected, can be detachably connected, or integrally connected; can be mechanically connected, can be electrically connected; can be directly connected, can be indirectly connected through an intermediate medium, or can be internal communication of two elements. For those skilled in the art, the specific meanings of the above terms in the embodiments can be understood according to the specific circumstances.

[0083] In order to make the purpose, technical scheme and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0084] The inventors found through careful study of the prior art that in the prior art, when performing blood vessel image fusion matching, only the blood vessel lumens of the three-dimensional coronary CT angiography image and the two-dimensional X-ray coronary angiography image are matched, the information displayed by the obtained blood vessel fusion image is not complete, the internal media of the blood vessels are not clear enough, and the plaques inside the blood vessels are not clear enough. In actual PCI surgery, not only the blood vessel condition needs to be observed, but also the movement of the stent in the blood vessel needs to be observed, especially the positional relationship between the internal media of the blood vessels, the plaques and the stent. The prior art fails to solve the problem of how to accurately and clearly observe the blood vessels and the internal media of the blood vessels, and the positional condition of the stent inside the blood vessels on one image. The inventors hope to solve the above problems.

[0085] In a first aspect, with reference to FIG. 1, the present application provides a blood vessel and stent in blood vessel monitoring method, comprising the following steps:

[0086] S1: obtaining a target blood vessel fusion image of a target blood vessel at a preset angle.

[0087] The target blood vessel fusion image is an image obtained by fusing a coronary CT angiography image (CCTA image) and an X-ray coronary angiography image (XA image) of the target blood vessel. The target blood vessel fusion image contains the blood vessel media features of the target blood vessel. The lumen shape of the blood vessel and the blood vessel media features can be observed on the target blood vessel fusion image. The blood vessel media features include the media shape features of the target blood vessel, and / or the plaque component features. That is, the blood vessel media features can only include the media shape features of the target blood vessel, can only include the plaque component features, or can include both the media shape features and the plaque component features.

[0088] The preset angle refers to an angle for acquiring an X-ray coronary angiography image (XA image), and can be determined according to a position of a target blood vessel (for example, can be a position of a lesion segment of the target blood vessel). Exemplarily, for a left circumflex artery (LCX) blood vessel, the preset angle can be, for example, in a range of left anterior oblique (LAO) 10-25° and combined with a caudal position (CAU) 25-35°. For other types of blood vessels, a suitable preset angle can be arbitrarily selected according to actual conditions. It should be noted that when the preset angle is selected, the angle for acquiring the X-ray coronary angiography image (XA image) and the angle for acquiring the stent transmission image in the subsequent step need to be the same, that is, the same angle.

[0089] Further, the coronary CT angiography image (CCTA image) and the X-ray coronary angiography image (XA image) are both acquired at a first phase of the target blood vessel. Specifically, the CCTA image and the XA image are acquired at the same phase of the target blood vessel, at which time the vessel lumen morphology acquired by the CCTA image and the XA image is close and basically consistent, thereby increasing the accuracy of matching the two images. Referring to FIG. 16, in a clinic, CCTA image acquisition generally selects a specific heart cycle time period as an acquisition window according to the heart rate of a patient, so as to reduce imaging artifacts caused by heart motion. In FIG. 16, HR is a heart rate, HV is a heart rate variability, and Start and End represent start and end times on an electrocardiogram.

[0090] The meaning of each row in FIG. 16 is explained below. The first column is a heart rate (HR) and a heart rate variability (HV) level, and in the second column to the fourth column, start to end represent an acquisition time window, and % represents a relative range (i.e., a relative phase) of the acquisition time corresponding to an R-R interval. For example, if the R-R interval is 1000 ms, start 60% represents 600 ms from the first R wave. Correspondingly, ms represents an absolute time range (i.e., an absolute phase). When the heart rate is irregular, the R-R interval is obviously inconsistent, and using the relative phase can cause inconsistent data acquisition in different heart cycles, thereby causing reconstruction artifacts. At this time, using the absolute phase can be more accurate.

[0091] The first phase can be a diastolic phase of the target blood vessel or a systolic phase. Preferably, the first phase is a diastolic phase of the target blood vessel. At this time, since the target blood vessel is in a diastolic state, the collected image can more clearly show the lumen shape and the intima information of the target blood vessel. Since the intima information is displayed, the outer boundary of the plaque is the intima, and therefore clear plaque information can also be obtained. By selecting to collect the image in the diastolic phase, the negative impact of motion artifacts in the systolic phase on the image collection quality can be avoided. In addition, the blood vessel structure in the diastolic phase is more dispersed, and the blood vessel overlap is less, which is also more conducive to simplifying the subsequent fusion and registration operation of the collected coronary CT angiography image and the X-ray coronary angiography image, making the registration simpler and more accurate. More preferably, the diastolic phase 75% time can be usually selected for collection, which can make the lumen shape and the intima information of the target blood vessel more clearly displayed. Further, the end-diastolic phase can also be selected for image collection, which can ensure the high definition of the lumen shape and the intima information of the target blood vessel in the collected image, and at the same time, the operation of contrast and image collection at the time point of the end-diastolic phase is easier and more distinguishable.

[0092] S2: Obtain a stent transmission image of the stent at different positions in the target blood vessel during movement of the stent in the target blood vessel at a preset angle.

[0093] The stent transmission image is an X-ray transmission image and contains stent information. Further, the stent is conveyed into the target blood vessel by a guide wire to move the stent in the target blood vessel, and the stent transmission image can also contain information of the guide wire. The stent can be a stent balloon, for example. More specifically, the stent transmission image can contain information of a stent delivery system having the stent (e.g., a stent balloon) and the guide wire.

[0094] Further, when the stent transmission image is collected, the patient needs to be collected in the same projection body position as the X-ray coronary angiography image is obtained, and the C-arm machine used for collecting the image and the contrast bed position also need to be kept relatively constant to ensure that the angle (i.e., the preset angle in the foregoing step S1) when the stent transmission image and the X-ray coronary angiography image are collected is the same, the collected image is stable, and the accuracy of the real-time registration of the stent transmission image and the target blood vessel fusion image is improved, so that the medical staff can observe the accurate registration image of the stent and the blood vessel in real time during the movement of the stent.

[0095] S3: Segment the stent transmission image to obtain a stent segmentation image.

[0096] The stent segmentation image contains shape information of the stent and information of a position relationship between the stent and the target blood vessel. Further, the stent is conveyed into the target blood vessel through a guide wire to move the stent in the target blood vessel, and the stent segmentation image can further contain shape information of the guide wire, information of the position relationship between the guide wire and the stent and the target blood vessel, and information of the position relationship between the guide wire and the stent.

[0097] S4: The target blood vessel fusion image and the stent segmentation image are fused and displayed to obtain a multi-modal fusion image of the target blood vessel and the stent.

[0098] The multi-modal fusion image is used to monitor the condition of the stent in the blood vessel. For example, referring to FIGS. 15A-15E, FIG. 15A shows a schematic diagram of a process of image fusion and display to obtain a multi-modal fusion image. The three images on the left side of FIG. 15A are FIGS. 15B, 15C and 15D in sequence, and the image on the right side of FIG. 15A is FIG. 15E. FIG. 15B is a stent transmission image, which shows the condition of the stent in multiple states, and image segmentation can be performed on each state to obtain a stent segmentation image corresponding to each state. FIG. 15C is an X-ray coronary angiography image (XA image) of the target blood vessel. FIG. 15D is a coronary CT angiography image (CCTA image). FIG. 15E is a multi-modal fusion image. Specifically, in the process of fusion and display, the target blood vessel fusion image can be obtained by fusing FIGS. 15C and 15D, and the multi-modal fusion image of FIG. 15E can be obtained by fusing the target blood vessel fusion image and the stent segmentation image obtained by segmenting FIG. 15B. The multi-modal fusion image can display the guide wire, the stent balloon (i.e., the stent), the fused lumen of the target blood vessel, the CCTA plaque component and the CCTA plaque media of the target blood vessel from the CCTA image. The multi-modal fusion image integrates the plaque component of the blood vessel, the media information of the blood vessel and the stent information, which can display the fused blood vessel lumen information from the target blood vessel fusion image, the media information and the plaque information on the CT angiography image, and the condition of the stent and the position relationship between the stent and the blood vessel lumen and the blood vessel media.

[0099] By using the above technical solutions, the blood vessel and stent monitoring method fully considers the blood vessel media characteristics of the blood vessel, which include media shape characteristics and / or plaque component characteristics, and the obtained target blood vessel fusion image can accurately display the state in the blood vessel. In addition, by acquiring the target blood vessel fusion image and the stent transmission image at the same preset angle, the positions of the stent in the two images correspond to the positions of the blood vessel, which facilitates the fusion and display of the target blood vessel fusion image and the stent transmission image, and the blood vessel and the internal characteristics of the blood vessel, the position of the stent in the blood vessel are more accurately displayed in the obtained multi-modal fusion image.

[0100] The prior art adopts a method based on a convolutional neural network to segment the three-dimensional CCTA image to obtain blood vessel data, adopts a foreground-background separation method based on RPCA to segment the two-dimensional dynamic XA image to obtain blood vessel data, respectively constructs 3D and 2D blood vessel topology models, and uses a blood vessel 3D / 2D registration method to register the blood vessel data; for intraoperative real-time dynamic blood vessel data, a method based on distance transformation is used to quickly compensate the deformation of the 2D blood vessel; further according to the transformation of the 3D blood vessel to the XA imaging space, the 3D blood vessel model is projected to the XA imaging plane, and then an enhanced depth perception visualization method is used to realize the fusion of the 3D blood vessel and the 2D image. However, the blood vessel data of the technical solution of the prior art only considers the data of the blood vessel lumen, the image segmentation and data conversion processing involved in the registration process are complex, it is difficult to quickly process the intima and plaque information inside the blood vessel lumen at the same time, and the technical solution of the prior art can only realize the lumen matching of the 3D CCTA and the 2D XA. The matching process is complex, not fast enough, and the fusion image information obtained after matching is not comprehensive.

[0101] To solve the above problems, the inventors have improved the acquisition of the target blood vessel fusion image. Specifically, with reference to FIG. 2 and in combination with FIG. 14, the foregoing step S1 specifically includes the following steps:

[0102] S11: Feature point extraction is performed according to the coronary CT angiography image and the X-ray coronary angiography image to obtain a CT segmentation feature point image and an X-ray segmentation feature point image of the target blood vessel, respectively.

[0103] The CT segmentation feature point image includes a three-dimensional blood vessel centerline and three-dimensional blood vessel feature points of the target blood vessel, and the three-dimensional blood vessel feature points contain radius information of the target blood vessel and / or bifurcation position information of the target blood vessel. The X-ray segmentation feature point image includes a two-dimensional blood vessel centerline and two-dimensional blood vessel feature points of the target blood vessel, and the two-dimensional blood vessel feature points contain radius information of the target blood vessel and / or bifurcation position information of the target blood vessel.

[0104] The bifurcation position information of the target blood vessel includes specific bifurcation information of the target blood vessel. More specifically, when the target blood vessel is a main blood vessel, there can be a bifurcated blood vessel on the target blood vessel, and at this time, the bifurcation position information of the target blood vessel can include at least one of the following: a bifurcation position between the bifurcated blood vessel and the target blood vessel on the target blood vessel, a bifurcation angle, and a bifurcation morphology. When the target blood vessel is a bifurcated blood vessel on a main blood vessel, at this time, the bifurcation position information of the target blood vessel can include at least one of the following: a bifurcation position between the target blood vessel and the main blood vessel on the main blood vessel, a bifurcation angle, and a bifurcation morphology.

[0105] S12: performing feature point matching on the CT segmented feature point image and the X-ray segmented feature point image through a deep map neural network model to obtain a target blood vessel centerline registration result.

[0106] The deep map neural network model can be a graph neural network based on a self-attention mechanism and a cross-attention mechanism. The specific method of performing feature point matching through the deep map neural network model can refer to the specific content of the patent with the application number 202410169234.2.

[0107] S13: fusing the coronary CT angiography image and the X-ray coronary angiography image of the target blood vessel through an image mapping method according to the target blood vessel centerline registration result and the CT segmented feature point image to obtain a target blood vessel fusion image.

[0108] Specifically, using the target blood vessel centerline registration result, according to the relative positional relationship between the three-dimensional blood vessel centerline of the CT segmented feature point image and the media information (or the media information and the plaque information) of the target blood vessel, the media information (or the media information and the plaque information) and the X-ray segmented feature point image are established through the three-dimensional blood vessel centerline as a medium, and the coronary CT angiography image and the X-ray coronary angiography image of the target blood vessel are fused through the image mapping method, and finally the target blood vessel fusion image is obtained. Preferably, the image mapping method is pixel mapping.

[0109] The feature point matching process of the deep map neural network model is fast and accurate, and the three-dimensional blood vessel centerline of the CT segmented feature point image is taken as the reference, the lumen morphology and the media information (or the media information and the plaque information) are considered comprehensively, the coronary CT angiography image and the X-ray coronary angiography image are fused, the whole registration and fusion process is fast and accurate, and the target blood vessel fusion image obtained has comprehensive display information, which is conducive to identifying the plaque information and the spatial structure of the blood vessel in combination with the CT and XA influenced features during the movement of the stent (for example, during the movement of the stent in the operation).

[0110] Referring to FIG. 3, the foregoing step S11 specifically includes the following steps:

[0111] S111: performing image segmentation on the coronary CT angiography image and the X-ray coronary angiography image respectively to obtain a CT angiography vessel segmentation result image and an X-ray vessel segmentation result image of the target blood vessel.

[0112] S112: performing feature point extraction on the CT angiography vessel segmentation result image and the X-ray angiography vessel segmentation result image respectively to obtain a CT segmented feature point image and an X-ray segmented feature point image of the target blood vessel.

[0113] The CT contrast blood vessel segmentation result image includes the lumen shape of the target blood vessel and the characteristics of the tunica media of the blood vessel. The X-ray blood vessel segmentation result image includes the lumen shape of the target blood vessel. The lumen shape of the target blood vessel includes the diameter, radius, etc. of the blood vessel lumen. The characteristics of the tunica media include the morphological characteristics of the tunica media of the target blood vessel, and / or the characteristics of the plaque component. The morphological characteristics of the tunica media include the diameter, radius, etc. of the tunica media. The characteristics of the plaque component specifically include the diameter, radius, size, etc. of the plaque. The lumen shape and the morphological characteristics of the tunica media can determine the size of the plaque, and thus the plaque component can be distinguished by the size of the plaque to obtain the characteristics of the plaque component.

[0114] Further, the segmentation methods used for image segmentation of the coronary CT angiography image and the X-ray coronary angiography image are deep learning automatic segmentation algorithms, which can be, for example, a blood vessel automatic segmentation network based on a 2D / 3D Unet network architecture. After automatic segmentation, manual correction can be further performed to adjust and optimize the segmentation result. Preferably, the automatic segmentation algorithm can add a centerline accuracy constraint to the loss function involved in the blood vessel automatic segmentation network based on the 2D / 3D Unet network architecture, so that the segmentation result is more accurate and more conducive to subsequent image feature point matching. During the segmentation process, different plaque components on the coronary CT angiography image can be color-coded to obtain a more accurate CT contrast blood vessel segmentation result image.

[0115] Further, during the PCI operation, the doctor needs to observe the position of the stent in the blood vessel and the state of the blood vessel, and move the stent to adjust the stent to the appropriate position. More specifically, the doctor needs to observe two images, namely the blood vessel image and the stent transmission image, and estimate the position of the stent in the blood vessel by comparing the two images, so as to adjust the position of the stent. However, this estimation method is inaccurate and has low reliability. Moreover, since the stent transmission image is an X-ray transmission image, the stent transmission image itself is not clear enough, which makes the estimation of the position of the stent in the blood vessel even less accurate, and it is not convenient to compare the two images.

[0116] In the prior art, the paper "Implementing Coronary Computed Tomography Angiography in the Catheterization Laboratory" discloses a technical solution for simultaneously fusing 3D CCTA lumen and plaque information with 2D XA images using QAngioCT RE / 3D workbench software (Medis Medical Imaging System, the Netherlands), which synchronizes the direction of the 3D coronary artery tree with the projection of XA by tracking the movement of the C-arm machine in real time. The process first connects an external sensor (called an inertial measurement unit) to the C-arm machine, and then uses Raspberry Pi as an intermediary to synchronize the C-arm machine with the 3D visualization software in real time through the external sensor. This system simultaneously color-codes and visualizes different plaque components based on HU values. As mentioned earlier, during PCI surgery, not only is it necessary to observe the blood vessel image, but it is also necessary to observe the movement of the stent in the blood vessel. The technical solution in the paper does not solve the problems of inconvenience caused by comparing the blood vessel image with the stent transmission image during PCI surgery, inaccurate stent position estimation, and unclear images.

[0117] Referring to FIG. 4, to solve the above problems, in addition to steps S1, S2, S3, and S4 in the above embodiment, the blood vessel and stent monitoring method of the present application can further include the following steps:

[0118] S5: sequentially displaying the multi-modal fusion images corresponding to different positions of the current stent in the target blood vessel in the first display area to display the state of the target blood vessel and the position of the stent in the target blood vessel during the movement of the stent in the target blood vessel in real time.

[0119] Wherein, the position of the first display area is fixed, and is used to display the images collected during the surgery. By performing feature point matching on the target blood vessel fusion image and the corresponding stent segmentation image of each image frame obtained in the foregoing steps and obtaining the multi-modal fusion image, and then sequentially displaying the multi-modal fusion image in the first display area, medical personnel can observe the current position of the stent in the target blood vessel in real time.

[0120] With the above technical solution, the blood vessel and stent monitoring method of the present application can monitor the current position of the stent in the target blood vessel during the movement of the stent in real time, and can also monitor the shape of the blood vessel and the internal information of the blood vessel, and can further fuse the stent information and the blood vessel information on one image for display, which facilitates observation of the clearer blood vessel state and the position of the stent in the blood vessel, and can also display in real time in the first display area, and can also display an animation effect in the first display area.

[0121] Exemplarily, the above method can be used in the process of surgery, in the process of surgery, the doctor moves the stent, the stent transillumination image corresponding to the position of the stent at the moment is collected in real time, the stent segmentation image is obtained by segmentation, and the stent segmentation image is fused with the target blood vessel fusion image to obtain a multi-modal fusion image. Thus, the state of the blood vessel and the position of the stent in the blood vessel during the process of surgery can be displayed in real time on the first display area and on the same image, and can be presented in the form of animation. Not only can the blood vessel state, the stent state, and the position of the stent in the blood vessel be clearly displayed on one image, but also the inconvenience of the doctor looking at two images in the prior art is eliminated, and the accuracy of monitoring the actual state of the stent and the blood vessel is increased. The doctor no longer needs to estimate the position of the stent by looking at two images. The effect of animation enables the doctor to observe the dynamic movement of the stent in the blood vessel in real time during the process of surgery, and the observation is more accurate.

[0122] In the above embodiments, with reference to FIG. 5, the foregoing step S2 specifically includes the following steps:

[0123] S21: acquiring stent transillumination images of the stent at different positions in the target blood vessel during movement of the stent in the target blood vessel at a preset frequency.

[0124] The preset frequency can be determined according to actual conditions. The preset frequency can be in the range of 7.5-30 frames per second. By adjusting the preset frequency, an appropriate amount of data can be acquired, real-time acquisition of the stent transillumination image can be achieved, the accurate situation of the position of the stent and the target blood vessel at each moment can be observed, and the real-time dynamics of the stent and the target blood vessel can be mastered. The preset frequency can be in the range of 10-30 frames per second, and further can be in the range of 10-20 frames per second. Preferably, in the embodiment, the preset frequency can be 15 frames per second.

[0125] In other embodiments, with reference to FIG. 6, the foregoing step S2 can also include the following steps:

[0126] S22: detecting the movement state of the stent during movement of the stent in the target blood vessel, and when it is detected that the stent has moved, acquiring stent transillumination images of the stent at different positions in the target blood vessel.

[0127] When no movement of the stent is detected within a period of time, the stent transillumination image can not be collected, so that the amount of data in subsequent steps such as fusion and registration can be reduced, and the situation of the stent and the target blood vessel can be accurately displayed in the first display area. When it is detected that the stent has moved, the stent transillumination images of the stent at different positions in the target blood vessel during movement of the stent are collected, and subsequent steps such as fusion and registration are performed. The real-time dynamic image of the movement of the stent can be observed.

[0128] Referring to FIG. 7, the foregoing step S3 specifically comprises the following steps:

[0129] S31: segmenting the stent transmission image by the image segmentation model to obtain a stent segmentation image.

[0130] As mentioned before, the stent transmission image is an X-ray transmission image containing stent information. Further, the stent is conveyed into the target blood vessel by a guide wire to move the stent in the target blood vessel, and the stent transmission image can also contain information of the guide wire. The stent can be a stent balloon, for example. More specifically, the stent transmission image can contain information of a stent delivery system having the stent (e.g. stent balloon) and the guide wire. The image segmentation model in the present application can segment out the features related to the stent on the image, and can also segment out the features related to the guide wire.

[0131] The stent segmentation image segmented by the image segmentation model can contain the morphological information of the stent and the positional relationship information of the stent and the target blood vessel. Further, the stent segmentation image segmented by the image segmentation model can also contain the morphological information of the guide wire, the positional relationship information of the guide wire and the stent and the target blood vessel, and the positional relationship information between the guide wire and the stent.

[0132] FIG. 17 shows a structural schematic diagram of the image segmentation model.

[0133] The image segmentation model includes a preliminary segmentation network and a generative adversarial network. The image segmentation model adopts a cascaded structure and is mainly composed of two cascaded Unets, one of which is the preliminary segmentation network and the other is the generative adversarial network. The preliminary segmentation network is used to segment the stent transmission image to obtain a preliminary stent segmentation result image.

[0134] Referring to FIG. 17, when performing segmentation, the stent transmission image (i.e. in the form of original data) is first converted into the form of a tensor (i.e. a multi-dimensional array), and then input into the preliminary segmentation network for preliminary segmentation. The preliminary stent segmentation result image obtained after the preliminary segmentation is input into the generative adversarial network for further optimization, and the optimized result is converted from the form of a tensor into the form of original data, i.e. the stent segmentation image can be obtained.

[0135] Since the stent transmission image is an X-ray transmission image, and the guide wire of the stent has limited absorption of X-rays, there can be blurring and partial stent regions invisible in the stent transmission image, which can cause the segmented stent to be broken in the preliminary segmentation result of the preliminary segmentation network, and can have a negative impact on the subsequent fusion of the stent and the target blood vessel.

[0136] To solve the above problems, a generative adversarial network is introduced in the image segmentation model, and the generative adversarial network is used to repair the preliminary stent segmentation result image to obtain a stent segmentation image. The Unet size of the generative adversarial network is smaller. In addition, the generative adversarial network also adopts the patchGAN idea. Specifically, the discriminator of the generative adversarial network no longer discriminates the authenticity of the whole image, but discriminates the authenticity of multiple small patches (i.e. image blocks) in the image. This local discriminator can better focus on the detailed features of the image, so that the generated image has higher authenticity in the local area, so as to improve the ability of the network to repair the segmentation fracture. In addition, the use of local discriminators makes the training process more stable and easy to converge, reducing the risk of mode collapse.

[0137] Further, the traditional Unet adopts a skip connection to connect different levels (encoders and decoders) in the Unet, and the skip connection can only use the information between the connected levels, cannot use the information of other levels, and cannot realize efficient communication between multiple different levels. In this embodiment, the image segmentation model further includes a convergence distribution module, which is used to connect the encoding and decoding layers in the preliminary segmentation network and / or the encoding and decoding layers in the generative adversarial network, so as to fuse the stent feature data in the stent transmission image.

[0138] Referring to FIG. 17, specifically, the convergence distribution module connects the encoding and decoding layers inside the preliminary segmentation network, and the convergence distribution module can also connect the encoding and decoding layers inside the generator part of the generative adversarial network. The preliminary segmentation network and the generative adversarial network are not connected through the convergence distribution module. More specifically, the preliminary segmentation network is composed of an encoder, a decoder and a convergence distribution module. The generative adversarial network is composed of a generator and a discriminator, and the generator part is composed of an encoder, a decoder and a convergence distribution module. The discriminator part of the generative adversarial network can be the same as the discriminator in patchGAN.

[0139] In the preliminary segmentation network and the generator part of the generative adversarial network, the convergence distribution module is used instead of the skip connection between each layer of the encoder and the decoder. By converging the features of each layer of the encoder into a feature warehouse, performing multi-layer information fusion in the feature warehouse, and distributing the fused information to the decoder, the decoder can better utilize the features of each layer of the encoder, and the information interaction between the levels of the encoding and decoding structure can be improved. The convergence and distribution operation of the convergence distribution module can better combine the texture and semantic information of the image, and enhance the representation ability of the stent guide wire and other small structures.

[0140] Wherein, when training the image segmentation model, firstly, the generative adversarial network is frozen, that is, the generative adversarial network does not participate in training at first, and only the preliminary segmentation network is trained. The input of the preliminary segmentation network is the stent transmission image, and the output obtained is the preliminary stent segmentation result image. When training the preliminary segmentation network, the preliminary segmentation loss function used is the segmentation loss L first The segmentation loss is specifically L Dice and L CrossEntropy . Wherein, X1 is the preliminary segmentation result (that is, the preliminary stent segmentation result image segmented by the preliminary segmentation network), Y is the true label value (that is, the expected stent segmentation result corresponding to the stent transmission image), and δ is a smoothing coefficient, usually 10e-6. The specific formula is as follows: L first =L Dice +L CrossEntropy (Formula 1) L CrossEntropy =-[Y·log(X1)+(1-Y)·log(1-X1)] (Formula 3)

[0141] The preliminary segmentation network is trained according to the stent transmission image, the preliminary stent segmentation result image and the preliminary segmentation loss function to update the parameters of the preliminary segmentation network until the preliminary segmentation preset training condition is met. Wherein, the preliminary segmentation preset training condition may be, for example, that the training process reaches 1 / 3 (for example, it may be 1 / 3 of the preset training time, or 1 / 3 of the preset training iteration number), and when the training reaches 1 / 3, the generative adversarial network is unfrozen, and at this time it is considered that the preliminary segmentation network can segment out the approximate result.

[0142] After the above preliminary training of the preliminary segmentation network is completed, the generative adversarial network participates in the training, and at this time the preliminary segmentation network and the generative adversarial network are trained together. The input of the entire image segmentation model of the preliminary segmentation network and the generative adversarial network is the stent transmission image, and the output obtained is the stent segmentation image. At this time, the guide wire segmentation loss function used for training includes the loss function of the preliminary segmentation network and the generation network loss and the adversarial loss of the generative adversarial network, and the generation network loss includes the above segmentation loss. It should be noted that the generation network loss is only used to train the generator of the generative adversarial network, and the adversarial loss can be used to train the generator and the discriminator of the generative adversarial network. The generator and the discriminator of the generative adversarial network are trained simultaneously. Wherein, the generation network loss is specifically L Generator , and the adversarial loss is specifically L Discriminator , wherein Y is the true label value (that is, the expected stent segmentation result corresponding to the stent transmission image), X2 is the stent segmentation result (that is, the stent segmentation image, refer to FIG. 17) obtained by the generator of the generative adversarial network, D represents the discriminator of the generative adversarial network, and I sourceis the original angiography image (i.e., the stent transmission image). The specific formula is as follows: L Generator = L Dice + L CrossEntropy + L Den (Formula 4) L CrossEntropy = -[Y log(X2) + (1-Y) log(1-X2)] (Formula 6) L Gen = -log[1-D(X2, I source )] (Formula 7) L Discriminator = -[log[1-D(X2, I source )] + log[D(Y, I source )]] (Formula 8)

[0143] Wherein, with reference to FIG. 17, during the training process, the discriminator can be used to determine whether the result X2 generated by the generator is close to the real label value Y (i.e., the result of the stent segmentation expected to be obtained corresponding to the stent transmission image), so as to improve the segmentation performance. Specifically, the discriminator will generate a discrimination result, which is 0 or 1. When the discrimination result of the discriminator is 1, it means that X2 is relatively close to the real label value Y. When the discrimination result is 0, it means that the result X2 generated by the generator deviates from the real label value Y to a greater extent, and needs to be further trained to further optimize the result X2 generated by the generator.

[0144] The preliminary segmentation network and the generative adversarial network of the image segmentation model are trained according to the stent transmission image, the stent segmentation image and the guide wire segmentation loss function to update the parameters of the preliminary segmentation network and the generative adversarial network, until a preset training condition is met, at which time the obtained image segmentation model can segment a stent segmentation image with good quality. Wherein, the preset training condition is network convergence.

[0145] In the above embodiments, the trained image segmentation model can quickly and accurately segment a stent segmentation image according to a stent transmission image, so as to accelerate the fusion process of the stent and the target blood vessel images in the entire surgical process, which is conducive to realizing real-time fusion and real-time display of the multi-modal fusion image after fusion.

[0146] In the above embodiments, with reference to FIG. 8, the foregoing step S4 specifically includes the following steps:

[0147] S41: superimposing the target blood vessel fusion image and the stent segmentation image to obtain a multi-modal fusion image.

[0148] Specifically, since the target vessel fusion image and the stent segmentation image are both acquired at the same preset angle (i.e., the preset angle in the foregoing steps S1 and S2), and the patient is in the same projection position when the images are acquired, the positions of the target vessel on the target vessel fusion image and the positions of the stent on the stent segmentation image have been corresponded to each other, and the characteristics of the tunica media in the target vessel and the position of the stent in the target vessel can be observed accurately and clearly by simply superimposing and displaying on the same image space.

[0149] In other embodiments, referring to FIG. 9, the foregoing step S4 can further include the following steps:

[0150] S42: registering the target vessel fusion image and the stent segmentation image to obtain a multi-modal fusion image.

[0151] The multi-modal fusion image obtained by the registration method can more accurately display the internal position relationship between the vessel and the stent and reduce errors.

[0152] Referring to FIG. 10, the foregoing step S42 can specifically include the following steps:

[0153] S421: performing registration processing on the target vessel fusion image and the stent segmentation image by using a rigid point cloud registration algorithm to obtain a multi-modal fusion image.

[0154] Specifically, the target vessel fusion image is converted into target vessel fusion image point cloud data, the stent segmentation image is converted into stent segmentation image point cloud data, and then the rigid point cloud registration algorithm is used to register the target vessel fusion image point cloud data and the stent segmentation image point cloud data based on the center line of the vessel on the XA image, so as to reduce the large-scale rigid deformation caused by the respiratory motion of the human body and the phase mismatch. By using the rigid point cloud registration algorithm for registration, an accurate multi-modal fusion image after the stent and the vessel are fused can be obtained.

[0155] Further, the following steps can be performed:

[0156] The preliminary vessel-stent registration data corresponding to the accurate position relationship between the stent and the target vessel is obtained by processing the vessel-stent registration result obtained by the rigid point cloud registration algorithm by using a non-rigid registration algorithm, and the target vessel fusion image and the stent segmentation image are fused according to the accurate position relationship to obtain a multi-modal fusion image of the target vessel and the stent.

[0157] Specifically, the position of the current guide wire of the stent collected in the target blood vessel is accurately positioned by a non-rigid registration algorithm, the accurate positional relationship between the current stent and the target blood vessel is obtained, and then the blood vessel mask after rigid deformation is superimposed on the real-time stent segmentation image segmented from the real-time stent transmission image, so as to realize image fusion and finally obtain a multi-modal fusion image of the target blood vessel and the stent.

[0158] By combining the rigid point cloud registration algorithm and the non-rigid registration algorithm, the target blood vessel fusion image and the stent segmentation image are sequentially registered and fused, which is conducive to reducing the influence of respiratory motion and heartbeat to achieve higher registration accuracy. In addition, by the implementation mode of deep learning, the registration time can be significantly reduced while ensuring the accuracy, and real-time registration and fusion can be realized.

[0159] The present application fuses the three-dimensional CT angiography image (CCTA image) of the target blood vessel in each current state with the two-dimensional X-ray coronary angiography image (XA image) to obtain a target blood vessel fusion image during the movement of the stent (which can be during the operation). Based on the matching of the blood vessel center line (specifically the center line information of the CCTA image), the information of the blood vessel media (or the information of the blood vessel media and the plaque) displayed on the CCTA image is mapped to the matched XA image. Further, the blood vessel center line on the XA image is taken as a reference to fuse the stent transmission image and the target blood vessel fusion image, so as to obtain a multi-modal fusion image containing the target blood vessel lumen shape, media information (or media and plaque information), and the positioning and expansion state of the stent. Finally, the multi-modal fusion image is directly displayed, so as to realize real-time monitoring of the target blood vessel and the stent during the movement of the stent, facilitate real-time monitoring of the expansion state of the stent in the blood vessel by medical staff, and the whole process is simple and reliable, observation is more convenient and accurate, and animation effect makes the observation of the movement of the stent in the blood vessel more intuitive.

[0160] Exemplarily, in the subsequent application of the blood vessel and stent monitoring method in any of the above embodiments of the present application, PCI operation planning can also be assisted, including determining the necessity of plaque rotary grinding pretreatment, assisting in selecting the diameter and length of the stent, etc. The multi-modal fusion image can also be used to assist in stent positioning, guide the selection of appropriate stent release pressure, and monitor the stent expansion in real time. Through the mapping of the media information (or plaque and media information), it can be determined whether the stent reaches the optimal inflation and adhesion state, the possible reasons affecting the postoperative stent inflation or poor adhesion can be revealed, and the reasonable application of stent optimization technology can be guided.

[0161] In addition, from the perspective of health economics, the cost of CCTA examination is significantly lower than that of OCT or IVUS; from the perspective of convenience, CCTA belongs to non-invasive coronary imaging technology, does not increase additional intraoperative invasive operation and operation time, is safer than OCT / IVUS and other invasive coronary imaging, has high clinical penetration rate, and can ensure high safety and make more patients obtain inexpensive and reliable examination.

[0162] In a second aspect, referring to FIG. 11, the embodiment of the present application further discloses a blood vessel and stent monitoring device 1 for monitoring the target blood vessel and the stent during the movement of the stent by the blood vessel and stent monitoring method in any of the foregoing embodiments. The blood vessel and stent monitoring device 1 comprises a fusion image acquisition module 11, a stent image acquisition module 12, a stent segmentation module 13 and an image fusion module 14.

[0163] The fusion image acquisition module 11 is connected with the stent image acquisition module 12, the stent image acquisition module 12 is connected with the stent segmentation module 13, and the stent segmentation module 13 is connected with the image fusion module 14.

[0164] Continuing to refer to FIG. 11, the fusion image acquisition module 11 is configured to acquire a target blood vessel fusion image of the target blood vessel at a preset angle; wherein the target blood vessel fusion image is an image obtained by fusing a CT angiography image and an X-ray coronary angiography image of the target blood vessel. The target blood vessel fusion image contains a tunica media feature of the target blood vessel, and the tunica media feature includes a tunica media morphological feature and / or a plaque component feature of the target blood vessel. The preset angle is the preset angle in step S1 of each of the foregoing embodiments.

[0165] The stent image acquisition module 12 is configured to acquire a stent transmission image of the stent at different positions in the target blood vessel during the movement of the stent in the target blood vessel at the preset angle. The preset angle is the preset angle in step S1 of each of the foregoing embodiments.

[0166] The stent segmentation module 13 is configured to segment the stent transmission image to obtain a stent segmentation image; wherein the stent segmentation image contains morphological information of the stent and information about the positional relationship between the stent and the target blood vessel.

[0167] Further, the stent is conveyed into the target blood vessel by a guide wire to move the stent in the target blood vessel, and the stent transmission image can further contain information of the guide wire. The stent can be, for example, a stent balloon. More specifically, the stent transmission image can contain information of a stent delivery system having the stent (e.g. a stent balloon) and the guide wire.

[0168] The image fusion module 14 is configured to fuse and display the target blood vessel fusion image and the stent segmentation image to obtain a multi-modal fusion image of the target blood vessel and the stent. The multi-modal fusion image is used to observe the stent in the blood vessel.

[0169] By using the above technical solutions, the blood vessel and stent monitoring device 1 in the embodiment fully considers the characteristics of the blood vessel media, including the media shape characteristics and / or plaque component characteristics, and the obtained target blood vessel fusion image can accurately display the state in the blood vessel. In addition, by collecting the target blood vessel fusion image and the stent transmission image at the same preset angle, the positions of the stent and the blood vessel in the two images correspond to each other, which facilitates the fusion display of the target blood vessel fusion image and the stent transmission image, and the multi-modal fusion image obtained finally has more accurate display of the blood vessel, the internal characteristics of the blood vessel, and the position of the stent in the blood vessel.

[0170] Further, the fusion image acquisition module 11 can also be configured to perform the steps S11, S111, S112, S12 and S13 in the foregoing embodiments. The stent image acquisition module 12 can also be configured to perform the foregoing step S21 or the foregoing step S22. The stent segmentation module 13 can also be configured to perform the foregoing step S31. The image fusion module 14 can also be configured to perform the foregoing steps S41 or S42, and when the image fusion module 14 is configured to perform the step S42, it can also perform the step S421.

[0171] Referring to FIG. 12, in other embodiments, the blood vessel and stent monitoring device 1 further comprises a display module 15 on the basis of the fusion image acquisition module 11, the stent image acquisition module 12, the stent segmentation module 13 and the image fusion module 14 described above.

[0172] The display module 15 is configured to display the multi-modal fusion images corresponding to different positions of the current stent in the target blood vessel in the first display area in sequence to display the state of the target blood vessel and the position of the stent in the target blood vessel in real time during the movement of the stent in the target blood vessel.

[0173] By using the above technical solutions, not only the current position of the stent in the target blood vessel can be monitored in real time during the movement of the stent, but also the shape of the blood vessel and the internal information of the blood vessel can be monitored, and the stent information and the blood vessel information can be fused and displayed on one image, which facilitates the observation of the clearer blood vessel state and the position of the stent in the blood vessel, and the real-time display in the first display area and the animation effect in the first display area.

[0174] In a third aspect, referring to FIG. 13, the embodiments of the present application further disclose an electronic device 2 comprising a memory 21 and a processor 22, the memory 21 storing at least one instruction, which, when executed by the processor 22, implements the blood vessel and stent monitoring method in any of the preceding embodiments. The memory 21 may, for example, include a system memory, a fixed non-volatile storage medium, etc. The system memory may, for example, store an operating system, an application program, a Boot Loader, and other programs, etc.

[0175] In the present embodiment, the electronic device 2 fully considers the characteristics of the tunica media of the blood vessel, which include the morphological characteristics of the tunica media and / or the plaque component characteristics, and the obtained target blood vessel fusion image can accurately display the state inside the blood vessel. In addition, by acquiring the target blood vessel fusion image and the stent transmission image at the same preset angle, the positions of the stent and the blood vessel in the two images correspond to each other, which facilitates the fusion display of the target blood vessel fusion image and the stent transmission image, and the display of the blood vessel, the internal characteristics of the blood vessel, and the position of the stent inside the blood vessel in the final obtained multi-modal fusion image is more accurate.

[0176] In a fourth aspect, the embodiments of the present application further disclose a computer-readable storage medium, which stores at least one instruction, which, when executed, implements the blood vessel and stent monitoring method in any of the preceding embodiments.

[0177] With the above technical solution, the computer-readable storage medium fully considers the characteristics of the tunica media of the blood vessel, which include the morphological characteristics of the tunica media and / or the plaque component characteristics, and the obtained target blood vessel fusion image can accurately display the state inside the blood vessel. In addition, by acquiring the target blood vessel fusion image and the stent transmission image at the same preset angle, the positions of the stent and the blood vessel in the two images correspond to each other, which facilitates the fusion display of the target blood vessel fusion image and the stent transmission image, and the display of the blood vessel, the internal characteristics of the blood vessel, and the position of the stent inside the blood vessel in the final obtained multi-modal fusion image is more accurate.

[0178] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable non-transitory storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0179] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks or in conjunction with the flowchart blocks.

[0180] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks or in conjunction with the flowchart blocks.

[0181] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks or in conjunction with the flowchart blocks.

[0182] Although the present application has been described in terms of certain preferred embodiments, the skilled artisan will appreciate that various modifications, alterations, permutations, and substitutions can be made without departing from the spirit and scope of the application. Accordingly, the application is not limited by the foregoing description, but is only limited by the scope of the patent.

Claims

1. A method for monitoring blood vessels and stents within blood vessels, characterized in that, The method comprises the following steps: acquiring a target vessel fusion image of a target vessel at a preset angle; wherein the target vessel fusion image is an image obtained by fusing a coronary CT angiography image and an X-ray coronary angiography image of the target vessel; the target vessel fusion image contains a vessel media feature of the target vessel, and the vessel media feature includes a media morphological feature and / or a plaque component feature of the target vessel; acquiring a stent transmission image of the stent at different positions during movement of the stent in the target vessel at the preset angle; segmenting the stent transmission image to obtain a stent segmentation image; wherein the stent segmentation image contains morphological information of the stent and information about the position relationship between the stent and the target vessel; fusing and displaying the target vessel fusion image and the stent segmentation image to obtain a multi-modal fusion image of the target vessel and the stent; wherein the multi-modal fusion image is used for monitoring the condition of the stent in the vessel.

2. The method of claim 1, wherein the step of determining the location of the stent in the blood vessel comprises the steps of: determining the location of the stent in the blood vessel by using the magnetic field generated by the magnetic field generator. The stent is conveyed into the target vessel by a guide wire to move the stent in the target vessel, and the stent segmentation image further contains morphological information of the guide wire, information about the position relationship between the guide wire and the target vessel, and information about the position relationship between the guide wire and the stent.

3. The method of claim 1, wherein the step of determining the location of the stent in the blood vessel is performed by using a magnetic resonance imaging system. The coronary CT angiography image and the X-ray coronary angiography image are both acquired at a first phase of the target vessel; preferably, the first phase is 75% of diastole or end diastole of the target vessel.

4. The method for monitoring blood vessels and stents in blood vessels as described in claim 1, characterized in that, The method of acquiring a target vessel fusion image of a target vessel at a preset angle specifically comprises the following steps: extracting feature points from the coronary CT angiography image and the X-ray coronary angiography image to obtain a CT segmentation feature point image and an X-ray segmentation feature point image of the target vessel, respectively; performing feature point matching on the CT segmentation feature point image and the X-ray segmentation feature point image through a deep graph neural network model to obtain a target vessel centerline registration result; fusing the coronary CT angiography image and the X-ray coronary angiography image of the target vessel through an image mapping method according to the target vessel centerline registration result and the CT segmentation feature point image to obtain the target vessel fusion image; wherein the CT segmentation feature point image includes a three-dimensional vessel centerline and three-dimensional vessel feature points of the target vessel, and the three-dimensional vessel feature points contain radius information of the target vessel and / or bifurcation position information of the target vessel; the X-ray segmentation feature point image includes a two-dimensional vessel centerline and two-dimensional vessel feature points of the target vessel, and the two-dimensional vessel feature points contain radius information of the target vessel and / or bifurcation position information of the target vessel.

5. The method of claim 4, wherein the step of determining the location of the stent in the blood vessel is performed by using a magnetic resonance imaging system. The method of extracting feature points from the coronary CT angiography image and the X-ray coronary angiography image to obtain a CT segmentation feature point image and an X-ray segmentation feature point image of the target vessel, respectively, specifically comprises the following steps: The coronary CT angiography image and the X-ray coronary angiography image are respectively subjected to image segmentation to obtain a CT contrast vessel segmentation result image and an X-ray vessel segmentation result image of the target blood vessel respectively; The CT contrast vessel segmentation result image and the X-ray contrast vessel segmentation result image are respectively subjected to feature point extraction to obtain the CT segmentation feature point image and the X-ray segmentation feature point image of the target blood vessel; The CT contrast vessel segmentation result image includes the lumen morphology of the target blood vessel and the vascular media features; and the X-ray vessel segmentation result image includes the lumen morphology of the target blood vessel.

6. The method of claim 4, wherein the step of determining the location of the stent in the blood vessel is performed by using a magnetic resonance imaging system. The image mapping method is pixel mapping.

7. The method for monitoring blood vessels and stents in blood vessels as described in claim 1, characterized in that, The blood vessel and stent monitoring method further includes: The target blood vessel fusion image and the stent segmentation image are registered to obtain the multi-modal fusion image.

8. The method for monitoring blood vessels and stents in blood vessels as described in claim 1, characterized in that, The target blood vessel fusion image and the stent segmentation image are registered to obtain the multi-modal fusion image. The target blood vessel fusion image and the stent segmentation image are registered to obtain the multi-modal fusion image. ​ 9. The method for monitoring blood vessels and stents in blood vessels as described in claim 1, characterized in that, ​ ​ 10. The method for monitoring blood vessels and stents in blood vessels as described in claim 9, characterized in that, ​ 11. The method of claim 9, wherein the step of monitoring the blood vessel and the stent in the blood vessel comprises the step of: ​ ​ 12. The method for monitoring blood vessels and stents in blood vessels as described in claim 1, characterized in that, ​ ​ 13. The method of monitoring a blood vessel and a stent in a blood vessel according to claim 12, wherein, ​ The target vessel fusion image and the stent segmentation image are registered by a rigid point cloud registration algorithm to obtain the multi-modal fusion image.

14. A device for monitoring a blood vessel and a stent in the blood vessel, characterized by The method comprises the steps of: a fusion image acquisition module is configured to acquire a target vessel fusion image of a target vessel at a preset angle; wherein the target vessel fusion image is an image obtained by fusing a CT angiography image and an X-ray coronary angiography image of the target vessel; the target vessel fusion image contains a vascular media feature of the target vessel, and the vascular media feature includes a media morphological feature and / or a plaque component feature of the target vessel; a stent image acquisition module is configured to acquire stent transmission image graphs of a stent at different positions during movement of the stent in the target vessel at a preset angle; a stent segmentation module is configured to segment the stent transmission image graphs to obtain a stent segmentation image; wherein the stent segmentation image contains morphological information of the stent and information about the position relationship between the stent and the target vessel; an image fusion module is configured to fuse and display the target vessel fusion image and the stent segmentation image to obtain a multi-modal fusion image of the target vessel and the stent; wherein the multi-modal fusion image is used to observe the situation of the stent in the vessel.

15. An electronic device, comprising: The electronic device comprises a processor and a memory, and the memory stores at least one instruction, which, when executed by the processor, implements the method for monitoring a vessel and a stent in the vessel according to any one of claims 1-13.

16. A computer readable storage medium characterized by: The computer readable storage medium stores at least one instruction, which, when executed, implements the method for monitoring a vessel and a stent in the vessel according to any one of claims 1-13.

Images