A method, device, and readable storage medium for generating a coronary artery roadmap

Abstract

The present invention provides a method, an apparatus, and a readable storage medium for generating a coronary artery roadmap. The method for generating the coronary artery roadmap includes obtaining a dynamic fluoroscopic image of a coronary artery; obtaining a sequence of angiographic images of the coronary artery during a preset time period in which the heart performs periodic motion; matching an angiographic image in the sequence of angiographic images that is in phase with the fluoroscopic image; and fusing the fluoroscopic image with the angiographic image that is in phase to generate a dynamic coronary artery roadmap. The method can provide real-time image guidance, improve accuracy and surgical efficiency, and reduce the fluoroscopy time and the dosage of contrast agent.

Description

[0001] This application is a divisional application of a Chinese patent application with the application number 202210234976X, the application date of March 11, 2022, and the invention title of "A Method, Device, and Readable Storage Medium for Generating a Coronary Artery Roadmap". Technical Field

[0002] The present invention relates to image processing technology, and more specifically, to a method, device, and readable storage medium for generating a coronary artery roadmap. Background Art

[0003] The amount of contrast agent required for percutaneous coronary intervention (PCI) depends on multiple factors: Firstly and most importantly, it is the experience of the interventionalist. Secondly, it is the complexity of the anatomical structure. Thirdly, it is the complexity of the target vessel, target lesion, and surgical procedure. Currently, during PCI, when advancing a guide wire in the coronary artery, operations such as advancing and positioning a balloon catheter and related instruments all require repeated injection of contrast agent into the coronary artery for assistance or confirmation.

[0004] During PCI, especially in cases where there are many coronary branches, tortuous blood vessels, the patient is overweight, the quality of fluoroscopic images is poor, and there is a lot of overlap of blood vessel images, the operator's recognition of the target vessel and target lesion significantly decreases, the difficulty of interventional operations increases, and repeated injection of contrast agent is required for assistance or confirmation, which will inevitably lead to an increase in the amount of contrast agent used and an extension of the operation time. Currently known contrast agent-related complications include contrast agent-related nephropathy, contrast agent allergy, etc., which may endanger the patient's life in severe cases, and the occurrence of the above complications is positively correlated with the amount of contrast agent used. Traditional PCI surgeries require multiple uses of contrast agent to determine the position of the guide wire and blood vessels. Traditional roadmap production methods include calculation processes such as taking a mask, taking a target film, subtraction, and superposition, that is, they can only provide a guiding function for static blood vessels and are not applicable to the real-time dynamic PCI surgical environment where the heart is beating. Traditional dynamic coronary roadmaps require a large amount of fluoroscopy time and contrast agent dose. The inventor found that although the existing cardiac dynamic roadmaps have brought certain help to clinicians, because they only consider the cardiac cycle (ECG) and do not consider the influence of respiratory movement, and since the morphology of the cardiac blood vessels will be affected by respiratory movement, the accuracy of traditional cardiac dynamic roadmaps is not high and cannot meet the needs of accurate clinical judgment, resulting in a relatively low clinical usage frequency of the dynamic roadmap function. In addition, not only are traditional blood vessel segmentation methods slow, but the access of ECG signals places higher requirements on device compatibility. Summary of the Invention

[0005] The present invention is provided to solve the above problems existing in the prior art. There is a need for a method, apparatus, and readable storage medium for generating a coronary artery roadmap, which can reduce the usage amount of contrast agent and reduce the side effects of contrast agent on the human body. At the same time, it can use a deep learning network to segment a marked target including two motion characteristics of breathing and heartbeat, and obtain a more accurate dynamic coronary artery roadmap in an efficient manner to meet the real-time requirements.

[0006] According to the first aspect of the present invention, a method for generating a coronary artery roadmap includes obtaining a dynamic fluoroscopic image of the coronary artery; obtaining a sequence of angiographic images of the coronary artery in a preset time period during which the heart performs periodic motion; matching an angiographic image in the sequence of angiographic images that matches the phase of the fluoroscopic image; and fusing the fluoroscopic image with the angiographic image that matches the phase to generate a dynamic coronary artery roadmap.

[0007] According to the second aspect of the present invention, an apparatus for generating a coronary artery roadmap includes a processor configured to execute the method for generating a coronary artery roadmap according to various embodiments of the present invention.

[0008] According to the third aspect of the present invention, a computer-readable storage medium stores computer program instructions, and when the computer program instructions are run by a processor, the processor is caused to execute the method for generating a coronary artery roadmap according to various embodiments of the present invention.

[0009] Compared with the prior art, the beneficial effects of the embodiments of the present invention are as follows:

[0010] An angiographic image that matches the phase of the fluoroscopic image is matched in the sequence of angiographic images, and the matching accuracy is high. The fluoroscopic image is fused with the angiographic image that matches the phase to generate a highly accurate dynamic coronary artery roadmap. This not only helps to reduce the usage amount of contrast agent and reduce the harm of contrast agent to the human body, but also can meet the real-time requirements, provide automatic and real-time image guidance for PCI surgery, provide continuous and specific position feedback of the guide wire and blood vessel for doctors, and greatly reduce the risk brought by blind puncture.

[0011] The above general description and the following detailed description are only exemplary and explanatory, and are not intended to limit the claimed invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In the accompanying drawings, which are not necessarily drawn to scale, the same reference numerals may describe similar components in different views. Similar reference numerals with alphabetic suffixes or different alphabetic suffixes may represent different examples of similar components. The drawings generally illustrate various embodiments by way of example and not limitation, and are used in conjunction with the specification and the claims to explain the disclosed embodiments. Such embodiments are illustrative and exemplary and are not intended to be an exhaustive or exclusive embodiment of the method, apparatus, system, or non-transitory computer-readable medium having instructions for implementing the method.

[0013] Figure 1 A flowchart showing a method for generating a coronary artery roadmap according to an embodiment of the present invention;

[0014] Figure 2 An illustration showing a fluoroscopic image according to an embodiment of the present invention;

[0015] Figure 3 A flowchart showing a method for matching an angiographic image that matches the phase of a fluoroscopic image in a sequence of angiographic images according to an embodiment of the present invention;

[0016] Figure 4 An illustration showing an angiographic image obtained after image registration according to an embodiment of the present invention;

[0017] Figure 5 An illustration showing a coronary artery roadmap generated by the method for generating a coronary artery roadmap according to an embodiment of the present invention;

[0018] Figure 6 An illustration showing a system for performing the method for generating a coronary artery roadmap according to an embodiment of the present invention. Detailed Description of the Embodiments

[0019] To enable those skilled in the art to better understand the technical solutions of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and specific examples, but this is not a limitation of the present invention. For the various steps described herein, if there is no necessity for a sequential relationship between them, the order in which they are described as examples herein should not be considered a limitation, and those skilled in the art should know that they can be adjusted in order as long as the logic between them is not destroyed and the entire process cannot be implemented.

[0020] The "first", "second" and similar terms used in the present invention do not denote any order, quantity or importance, but are merely used to distinguish parts. Terms such as "comprising" or "including" mean that the elements before the term cover the elements listed after the term, and do not exclude the possibility of also covering other elements. Terms such as "upper", "lower", "left" and "right" are only used to represent relative positional relationships, and when the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0021] All terms used in the present invention (including technical terms or scientific terms) have the same meaning as understood by those of ordinary skill in the art to which the present invention pertains, unless otherwise specifically defined. It should also be understood that terms defined in general dictionaries should be interpreted as having meanings consistent with their meanings in the context of the relevant art, and should not be interpreted in an idealized or overly formal sense, unless specifically defined as such here. Technologies, methods and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, the said technologies, methods and devices should be regarded as part of the specification.

[0022] Figure 1 A flowchart showing a method for generating a coronary artery roadmap according to an embodiment of the present invention.

[0023] As shown in step S101, the method for generating the coronary artery roadmap includes obtaining a dynamic fluoroscopic image of the coronary artery. Among them, the dynamic fluoroscopic image can be a dynamic fluoroscopic image obtained in real time during the operation, or a dynamic fluoroscopic image extracted from an image database or a dynamic fluoroscopic image obtained by other means, and no specific limitation is made. Among them, the acquisition methods for fluoroscopic images or photographic images include, but are not limited to, direct acquisition through various imaging modalities, such as, but not limited to, medical contrast imaging technologies such as CT, MR, myocardial radionuclide scanning, spiral CT, positron emission tomography, X-ray imaging, fluorescence imaging and ultrasonic imaging, or reconstruction based on the original images obtained by the imaging device. For example, an X-ray imaging is centered on the estimated target position, and a fluoroscopic scan is performed to obtain a fluoroscopic image. Among them, the technical term "acquire" means any way of directly or indirectly obtaining, with or without additional image processing such as noise reduction, cropping, reconstruction, etc.

[0024] As shown in step S102, a sequence of angiography images of the coronary artery for a preset time period is obtained, during which the heart performs periodic motion. Among them, the angiography image is an image of the vascular structure, obtained through an angiography device such as a digital subtraction angiography (DSA) device. Digital subtraction angiography is to quickly inject a contrast agent containing an organic compound that is transparent under X-ray irradiation into the bloodstream, make the blood vessels visible under X-ray irradiation, photograph the development process of the blood vessel lumen, and from the development result, the blood flow order containing the contrast agent and the blood vessel filling condition can be seen, so as to understand the physiological and anatomical changes of the blood vessels.

[0025] Specifically, for example, a sequence of angiography images of the coronary artery for a preset time period is obtained under optimal contrast agent injection. Among them, the optimal contrast agent can refer to the image obtained after injecting the contrast agent into the region of interest of the coronary artery. At this time, the blood vessels are in a filled state and can continuously complete at least one cardiac motion cycle. The contrast agent can be added to the blood of the target object by arterial injection. During the image acquisition process, by using the characteristic that the contrast agent has strong attenuation for X-ray imaging, the vascular system of the target object is developed to obtain a gray-scale image of blood vessel filling. By observing the blood vessels in the image, it can assist in the diagnosis and treatment of vascular diseases.

[0026] The order of step S101 and step S102 is not limited. For example, a dynamic fluoroscopic image of the coronary artery can be obtained first, and then a sequence of angiography images of the coronary artery for a preset time period can be obtained; it can also be that a sequence of angiography images of the coronary artery for a preset time period is obtained first, and then a dynamic fluoroscopic image of the coronary artery is obtained; it can also be that a dynamic fluoroscopic image of the coronary artery and a sequence of angiography images of the coronary artery for a preset time period are obtained simultaneously.

[0027] As in step S103, an angiographic image that matches the phase of the fluoroscopic image is identified in the sequence of the angiographic images, so as to identify an angiographic image that has the best match with the current fluoroscopic image, i.e., an angiographic image that is at least in a matching cardiac cycle phase, for generating a dynamic coronary artery roadmap. Note that the matching phases at least include cardiac cycle phases, and may further include phases of the respiratory motion cycle. That is to say, at least the cardiac cycle phases match, or both the cardiac cycle and respiratory motion cycle phases may match. Matching can be understood as being exactly the same or having an acceptable deviation. When matching, a reasonable deviation is allowed between the phases of the two, of course, this deviation needs to be within a reasonable range to ensure the accuracy of the matching. Among them, the matching methods include, but are not limited to, feature-based image registration methods. For example, image features in two images are extracted, and the two images are registered based on the image features to obtain the corresponding relationship between homologous image points in the two images. Only by way of example, other methods capable of achieving registration may also be included. No specific limitation is made on the specific matching method, as long as an angiographic image that matches the phase of the fluoroscopic image can be identified in the sequence of the angiographic images based on the concept of this embodiment.

[0028] As in step S104, the fluoroscopic image is fused with the angiographic image that matches the phase to generate a dynamic coronary artery roadmap, so as to obtain a more accurate, dynamic, and real-time coronary artery roadmap, providing precise navigation for doctors during PCI surgery, improving the surgical efficiency, and shortening the surgical time. Based on this embodiment, without using more contrast agents, it plays a more definite and intuitive guiding role in wire operation, avoiding the surgical risks introduced by blind puncture. For example, when a patient has a heart disease, before performing a heart surgery, a doctor can inject a contrast agent into the patient's body in advance to keep the region of interest of the heart filled and obtain a sequence of angiographic images. During the surgery, the doctor can inject a small amount of contrast agent into the patient's body or even not inject the contrast agent again, identify an angiographic image that matches the phase of the intraoperative real-time fluoroscopic image in the sequence of the angiographic images, and fuse the fluoroscopic image with the angiographic image that matches the phase, then a real-time and precise coronary artery roadmap can be generated. This implementation manner is only an example and not a specific limitation on the method for generating a coronary artery roadmap. In addition, for this embodiment, no external device intervention is required, correspondingly improving the device compatibility.

[0029] In some embodiments, the dynamic fluoroscopic image of the coronary artery includes the intraoperative real-time fluoroscopic image of the coronary artery to adapt to the cardiac beating and real-time dynamic PCI surgical environment and meet the real-time requirement. During the surgery, along with the complex heartbeat process, a real-time dynamic fluoroscopic image of the coronary artery can be obtained, such as Figure 2As shown, for this dynamic fluoroscopic image, accurate registration with the phase-matched angiographic image can be rapidly achieved during the operation, and then a real-time dynamic coronary artery roadmap can be presented, enabling the doctor to directly observe the clear vascular path.

[0030] Furthermore, the preset time period includes at least one respiratory motion cycle, or the least common multiple cycle of the respiratory motion cycle and the cardiac cycle. The cardiac cycle can only reflect the heartbeat motion and does not include the states related to respiratory motion. The present invention finds that only considering the cardiac cycle and not considering the influence of respiratory motion cannot comprehensively reflect the cardiac motion cycle and cannot achieve accurate registration. This embodiment takes into account the influence of the respiratory motion cycle to distinguish the periodic motion of the heart from the cardiac cycle within the preset time period. The periodic motion of the heart within the preset time period can be one or more respiratory motion cycles to reflect the periodic motion of the heart, or can also be the least common multiple cycle of one or more respiratory motion cycles and the cardiac cycle to reflect the periodic motion of the heart. In this way, after the sequence of angiographic images obtained has experienced several cardiac contraction cycles and several respiratory motion cycles, the heart returns to its initial position (also referred to as a complete cardiac motion cycle in the following), and thus can comprehensively reflect the combined change process of the phase of the cardiac cycle together with the phase of the respiratory motion cycle. In this way, accurate registration with the real-time fluoroscopic image can be achieved at two phases. The smallest complete cardiac motion cycle is the least common multiple of the respiratory motion cycle and the cardiac cycle.

[0031] In some embodiments, the preset time period can exactly be the least common multiple cycle of the respiratory motion cycle and the cardiac cycle, so as to make the heart return to its initial position with the shortest duration. Specifically, the start time of this preset time period can be set as the start time of the first cardiac cycle and also the start time of the first respiratory motion cycle. Without being limited to this implementation manner, the preset time period can also be greater than the least common multiple cycle of the respiratory motion cycle and the cardiac cycle to ensure that the heart returns to its initial position at least once. For example, the preset time period is the least common multiple cycle and one respiratory motion cycle or one cardiac cycle, or other ways. In this way, the key features of both heartbeat motion and respiratory motion can be fully considered, which is beneficial to improving the registration accuracy, obtaining a dynamic real-time arterial coronary roadmap, providing automatic and real-time image guidance for PCI surgery, providing continuous and specific position feedback of the guide wire and blood vessel for the doctor, and reducing the risk brought by blind puncture.

[0032] In some embodiments, fusing the fluoroscopic image with the angiographic image that matches the phase to generate a dynamic coronary artery roadmap further includes presenting the segmented blood vessels of the coronary artery in the angiographic image that matches the phase on the coronary artery in the fluoroscopic image. After precise registration, an angiographic image that matches the phase of the fluoroscopic image is obtained. The angiographic image has the same phase as the current fluoroscopic image. For example, the extracted angiographic image can have the same cardiac cycle phase as the current fluoroscopic image, or have the same cardiac cycle phase together with the respiratory movement cycle phase to ensure precise registration at the two phases. By fusing the fluoroscopic image with the angiographic image that matches the phase, blood vessels that cannot be displayed clearly or have poor clarity in the fluoroscopic image are clearly presented synchronously.

[0033] Among them, the fusion refers to maximizing the extraction of their respective favorable information from the images to be fused through image processing, computer technology, etc., and finally synthesizing them into a high-quality image to improve the resolution of the original image. In some embodiments, image fusion can be divided into three levels from low to high: data-level fusion, feature-level fusion, and decision-level fusion. Data-level fusion, also known as pixel-level fusion, refers to the process of directly processing data to obtain a fused image, which is the basis for high-level image fusion. In this embodiment, no specific limitation is imposed on the specific fusion method.

[0034] Furthermore, fusing the fluoroscopic image with the angiographic image that matches the phase to generate a dynamic coronary artery roadmap may also include presenting the segmented blood vessels of the coronary artery in the angiographic image that matches the phase on the coronary artery in the fluoroscopic image, and the visual effects of the segmentation results at each location change according to their confidence levels, so as to facilitate the user to intuitively know the segmented blood vessels and their reliability, and reduce the workload. For example, during the operation, the doctor needs to concentrate fully on the operation, with a heavy workload. The system automatically identifies the confidence levels of the segmented blood vessels and presents different visual effects based on the high or low confidence levels. Specifically, when the confidence level of the segmented blood vessel is low, it indicates that the blood vessel may not be an actual blood vessel, and the system assigns it a light pink color. For blood vessels with a higher confidence level, the system assigns them a darker red color. Through the differences in the visual effects of the segmentation results, the doctor can quickly distinguish the positions and reliabilities of the blood vessels and make intraoperative responses quickly, reducing the workload and improving the work efficiency. This is only a specific embodiment and not a specific limitation method.

[0035] In some embodiments, the confidence level may characterize the level of uncertainty. In this way, the part where the certainty level exceeds the threshold can be presented to the doctor, and the part that needs to be presented to the doctor is visually presented to prompt the doctor which information needs to be focused on, and the information that does not need to be presented to the doctor is directly screened out. The doctor can tend to focus on the segmentation results with high confidence levels for detailed analysis. In contrast, the doctor can only use the remaining energy to refer to the segmentation results with low confidence levels, thereby achieving an optimal allocation of the workload. At the same time, it also provides the doctor with additional information for reference.

[0036] Among them, the specific relationship between the specific visual effect and the confidence level is not specifically limited. For example, when the confidence level is high, the system can assign a cool color visual effect or a warm color visual effect to facilitate distinction.

[0037] In some embodiments, as Figure 3 shown, matching the angiography image that matches the phase of the fluoroscopy image in the sequence of angiography images specifically includes:

[0038] Step S301 shows performing a first image segmentation on each angiography image in the sequence of angiography images to extract a first marker for tracking respiratory movement and a second marker for tracking heartbeat movement. By the first marker for tracking respiratory movement and the second marker for tracking heartbeat movement, an angiography sequence including at least one complete cardiac motion cycle (i.e., the least common multiple cycle of the respiratory movement cycle and the cardiac cycle) is obtained, providing key features for subsequent registration. The first marker for tracking respiratory movement is used to reflect the periodic change trajectory of respiratory movement, and the second marker for tracking heartbeat movement is used to reflect the periodic change trajectory of heartbeat movement. Image registration based on the first marker and the second marker is beneficial to improving the registration accuracy.

[0039] Step S302 shows performing a second image segmentation on the fluoroscopy image to extract the corresponding first marker and second marker. When performing the second image segmentation, the extracted first marker and second marker are the same as the first marker and the second marker in the above step S301, and are respectively used to reflect the periodic change trajectory of respiratory movement and the periodic change trajectory of heartbeat movement.

[0040] Step S303 shows that the fluoroscopic image is registered with the sequence of angiographic images by comparing the geometric correlation features of the first marker and the second marker respectively extracted by the first image segmentation and the second image segmentation and / or the relative position between the first marker and the second marker, so as to match an angiographic image in the sequence of angiographic images that is consistent with the phase of the fluoroscopic image. By comprehensively considering the geometric correlation features of the first marker and the second marker and / or the relative position between the first marker and the second marker, efficient and accurate registration can be achieved, avoiding registration errors, and making the registration speed suitable for the needs of dynamic registration with limited intraoperative time.

[0041] Specifically, during the actual cardiac motion, there are both heartbeat motions (i.e., the cardiac cycle reflects the heartbeat motion) and respiratory motions (i.e., the respiratory motion cycle reflects the respiratory motion). Affected by the respiratory motion, during matching, the entire position may rotate or change in position. By combining these two features of respiration and heartbeat to judge the motion cycle, higher registration accuracy can be achieved. In the specific registration process, different methods can be used for registration. For example, the geometric correlation features of the first marker extracted by the first image segmentation from the angiographic image sequence are matched with the geometric correlation features of the first marker extracted by the second image segmentation from the real-time fluoroscopic image, and the matching is performed by means of spatial transformation, structure matching, or pixel superposition, etc., to achieve registration based on the geometric correlation features of the first marker. After that, a similar method is used for registration based on the geometric correlation features of the second marker. Or, the geometric correlation features of the first marker extracted by the first image segmentation from the angiographic image sequence can be matched with the geometric correlation features of the first marker extracted by the second image segmentation from the real-time fluoroscopic image, and at the same time, the geometric correlation features of the second marker extracted by the first image segmentation from the angiographic image sequence are registered with the geometric correlation features of the second marker extracted by the second image segmentation from the real-time fluoroscopic image.

[0042] Or, it is also possible to only register the relative position during matching. For example, the relative positions of the first marker and the second marker extracted from the angiographic image are matched with the relative positions of the first marker and the second marker extracted from the fluoroscopic image, and after accurate matching, an angiographic image that is consistent with the phase of the fluoroscopic image is obtained.

[0043] Alternatively, during registration, by comprehensively considering the geometric correlation features and relative positions of different key features, the registration accuracy can be further improved based on these two aspects. For example, the first method can be to first match the geometric correlation features of the first and second markers extracted from the angiography image sequence with the first and second markers extracted from the fluoroscopy image. After the matching is completed, then match the relative positions between the first and second markers extracted from the angiography image sequence with the relative positions between the first and second markers extracted from the fluoroscopy image. Due to the complexity of cardiac motion, there may be deviations when registering based on geometric correlation features. By further registering based on relative positions, it is beneficial to exclude the cases where the registration based on geometric correlation features is not well done. The second method can be to perform the matching of geometric correlation features and the matching of relative positions simultaneously. By comprehensively considering these two factors, the registration accuracy is greatly improved.

[0044] In some embodiments, the first marker includes anatomic markers of the diaphragm and / or lungs, and the second marker includes a catheter fixed to the coronary ostium and / or the cardiac silhouette. Among them, the diaphragm is a feature closely related to respiratory motion and can be used to track the respiratory motion trajectory. The catheter is fixed at the coronary ostium and is used to track the cardiac motion trajectory. The anatomic markers of the lungs include, but are not limited to, the trachea and blood vessels. Both the diaphragm and the catheter have their respective geometric correlation features. At the same time, there is a relative position between the diaphragm and the catheter. During the respiratory and cardiac motions, the relative position between the diaphragm and the catheter changes. By comprehensively considering the diaphragm and the catheter for registration, not only the necessary image features are retained, but also the image range to be registered is reduced. Therefore, while improving the image registration efficiency, the image registration accuracy is also improved.

[0045] In some embodiments, registering the fluoroscopic image with the sequence of angiographic images by comparing the geometrically related features of the first marker and the second marker respectively extracted from the first image segmentation and the second image segmentation and / or the relative position between the first marker and the second marker specifically includes performing a first registration of the fluoroscopic image with the sequence of angiographic images by comparing the geometrically related features of the first marker extracted from the first image segmentation and the second image segmentation together with the relative position between the first marker and the second marker, so as to match a first angiographic image in the sequence of angiographic images that is in phase with the fluoroscopic image. Specifically, for example, if the first marker is the diaphragm and the second marker is a catheter fixed at the orifice of the coronary artery, during the matching process, the geometrically related features of the diaphragm can be compared together with the relative position between the diaphragm and the catheter. There are two specific ways for the comparison. For example, one way is to first compare the geometrically related features of the diaphragm, determine whether the geometrically related features of the diaphragm are registered, and then compare the relative position between the diaphragm and the catheter after registration to improve the registration accuracy; the other way is to compare the geometrically related features of the diaphragm and the relative position between the diaphragm and the catheter simultaneously, and match the geometrically related features and the relative position at the same time. Through this implementation method, a first angiographic image in the sequence of angiographic images that is in phase with the fluoroscopic image is obtained.

[0046] The registration method further includes performing a second registration of the fluoroscopic image with the sequence of angiographic images by comparing the geometrically related features of the second marker extracted from the first image segmentation and the second image segmentation together with the relative position between the first marker and the second marker, so as to match a second angiographic image in the sequence of angiographic images that is in phase with the fluoroscopic image. Specifically, for example, if the second marker is a catheter and the first marker is the diaphragm, during registration, the geometrically related features of the catheter are comprehensively registered together with the relative position between the diaphragm and the catheter. There are also two specific ways for the registration. For example, one way is to first compare the geometrically related features of the catheter, determine whether the geometrically related features of the catheter are registered, and then compare the relative position between the diaphragm and the catheter after registration to improve the registration accuracy; the other way is to compare the geometrically related features of the catheter and the relative position between the diaphragm and the catheter simultaneously, and match the geometrically related features and the relative position at the same time. Through this implementation method, a second angiographic image in the sequence of angiographic images that is in phase with the fluoroscopic image is obtained.

[0047] Due to the influence of respiratory movement, changes in position such as rotation may occur, which may cause deviations in registration based on a single key feature and its relative position. Therefore, after obtaining the first angiographic image and the second angiographic image, the embodiment of the present invention determines the degree of difference between the first angiographic image and the second angiographic image. When the degree of difference is lower than the threshold, the first angiographic image and / or the second angiographic image is used to obtain a finally matched angiographic image that is phase-consistent with the fluoroscopic image, so as to further improve the credibility of the registration result. Specifically, when the degree of difference between the first angiographic image and the second angiographic image is lower than the threshold, it indicates that the registration accuracy of obtaining the first angiographic image and the second angiographic image is relatively high and has a high credibility. A finally matched angiographic image that is phase-consistent with the fluoroscopic image can be obtained based on the first angiographic image and / or the second angiographic image. On the contrary, if the degree of difference between the first angiographic image and the second angiographic image is higher than the threshold, it indicates that the registration accuracy is poor and re-registration is required. Among them, the setting of the threshold can be based on actual needs. For example, it can be set when the system leaves the factory, or it can be a value manually set by the user based on the actual situation.

[0048] In another embodiment, the registration method further includes comparing the geometric correlation features of the first marker and the second marker respectively extracted by the first image segmentation and the second image segmentation, together with the relative position between the first marker and the second marker, and registering the fluoroscopic image with the sequence of angiographic images, so as to match an angiographic image in the sequence of angiographic images that is phase-consistent with the fluoroscopic image. The angiographic image obtained through this embodiment has higher accuracy. For example, the first marker is the diaphragm and the second marker is the catheter. During the registration process, if only the catheter or the diaphragm is considered, misregistration may occur. During cardiac movement, the shape of the entire coronary artery will be affected by respiratory movement and undergo some rotation and movement. If only the catheter is considered and respiration is not considered, there will be relatively large deviations at some phase points. Specifically, for example, when the current respiratory cycle moves downward from the highest point and upward from the lowest point, the heartbeat may have the same cycle, that is, the cardiac cycle is the same, but the respiratory cycle may be opposite, resulting in a large deviation in registration. Therefore, whether only geometric correlation features or only relative position are considered, there may be the same frame in different cycles, resulting in misregistration.

[0049] Based on the registration method described in this embodiment, by comprehensively considering the geometric correlation features of the first marker and the second marker together with the relative position for comparison and registration, high-precision registration can be ensured, and the situation of registration errors can be avoided. For example, based on the key features of the diaphragm and the catheter for registration, there are still two methods. One is to first register the geometric correlation features of the diaphragm and the catheter, and after the registration is correct, then compare the relative positions of the diaphragm and the catheter; the other is to compare the relative position relationship between the diaphragm and the catheter while registering the geometric correlation features of the diaphragm and the catheter, so as to avoid misregistration, improve the accuracy of registration, and ensure the registration quality. As Figure 4 shown, the angiography image obtained after image registration has the ability to accurately segment blood vessels, and the position of the blood vessels can be identified based on the angiography image. By fusing the fluoroscopy image with the angiography image that matches the phase, the dynamic coronary artery roadmap as Figure 5 described is obtained, which is convenient for doctors to intuitively distinguish the position of the blood vessels. The relative position between the diaphragm and the catheter can be the distance.

[0050] Among them, in each of the above embodiments, matching the angiography image that matches the phase of the fluoroscopy image in the sequence of angiography images can greatly reduce the computational load. For example, this can be completed in dozens of milliseconds to one hundred milliseconds, which is beneficial to obtaining a real-time coronary artery roadmap and is especially suitable for intraoperative real-time applications.

[0051] In some embodiments, the geometric correlation features of the first marker include the fitted curve, curvature, position, and shape of the first marker, and the geometric correlation features of the second marker include the corner, position, direction, curvature, shape, and width of the second marker to achieve precise registration regarding geometric correlation features.

[0052] In some embodiments, in the sequence of angiography images of the coronary artery in a preset time period, the concerned blood vessel segment remains in a filled state to obtain an accurate sequence of angiography images.

[0053] In some embodiments, the method for generating the dynamic coronary artery roadmap further includes performing the first image segmentation and the second image segmentation respectively using a learning network to obtain a first segmented image and a second segmented image. For example, the learning method segments features such as catheters, guidewires, blood vessels, and diaphragms, and the predicted blood vessel map is used to create a coronary artery roadmap, while the predicted catheter and diaphragm are used for image registration in step S103. For example, before performing segmentation extraction on medical images to obtain blood vessel images, the method further includes: obtaining training data, converting the images to a fixed size (such as 512*512), and performing normalization processing to convert the pixels to values between 0 and 1. Among them, the training data includes medical images with segmentation annotation information (blood vessels, guidewires, catheters, diaphragms). Image processing methods such as horizontal flipping, vertical flipping, random scaling, random brightness, random contrast, and random noise are used to enhance the training data, and the enhanced training data is used to learn and train the segmentation network model to obtain an image segmentation model.

[0054] The above-mentioned segmentation network model can be a segmentation network model such as ResUnet or transunet, which extracts network blood vessels and other features, greatly improving the feature extraction efficiency and being the fundamental guarantee for real-time navigation. By using training data of medical images with multiple segmentation annotation information (blood vessels, guidewires, catheters, diaphragms) to learn and train the segmentation network model, an image segmentation model can be obtained, thereby ensuring the segmentation accuracy and speed of the segmentation target using the obtained image segmentation model.

[0055] Among them, the second segmented image has the same size as the fluoroscopic image, and the intensity value of each pixel indicates the probability-related parameter of belonging to the segmentation target at that location. The first segmented image has the same size as each angiogram image, and the intensity value of each pixel indicates the probability-related parameter of belonging to the segmentation target at that location. The segmentation target at least includes blood vessels, the first marker, and the second marker. In the method for generating the deep learning model provided in the embodiment, performing segmentation extraction on medical images to obtain a segmented image includes: using the image segmentation model to segment the medical image to determine the probability value of each pixel on the medical image representing the segmentation target; extracting the pixels with probability values greater than the preset probability to obtain the predicted image of the segmentation target. A predicted image of the segmentation target can be obtained for each type of segmentation target, that is, the predicted image of the image segmentation model is multiple predicted images including the segmentation target (blood vessels, guidewires, catheters, diaphragms).

[0056] In the above embodiments, the calculation process of segmentation using the image segmentation model is actually that the image segmentation model predicts each pixel in the image to obtain the probability value of the pixel representing the segmentation target. When the predicted probability value of the pixel on the segmentation target is greater than the preset probability, it is determined that the pixel belongs to the segmentation target. By predicting one by one, finally all the pixels representing the segmentation target are obtained to complete the image segmentation and obtain the segmentation result.

[0057] Specifically, based on the first segmentation image, blood vessels, the first marker, and the second marker in the angiography image are extracted; based on the second segmentation image, blood vessels, the first marker, and the second marker in the fluoroscopy image are extracted for subsequent registration, which can greatly improve the registration accuracy.

[0058] In some embodiments, based on the first segmentation image, the confidence of each part of the extracted blood vessels is determined for display associated with each part of the segmented blood vessels. Optionally, in the method for generating a coronary artery roadmap provided in the embodiment, determining the confidence of each part of the blood vessels on the blood vessel image includes: determining the confidence of each part of the blood vessels on the blood vessel image based on the probability value of the pixel representing the segmentation target. Since the probability value of each pixel predicted by the image segmentation model is related to the confidence of the blood vessel segmentation accuracy. That is, based on the probability value of the pixel representing the blood vessel, the confidence of each part of the blood vessels on the blood vessel image can be determined. Based on the confidence of each part of the blood vessels, the pixel color of each part of the blood vessels on the blood vessel image is determined, thereby generating a coronary artery roadmap, which is convenient for doctors to intuitively judge the position of the blood vessels and improves the work efficiency of doctors.

[0059] Figure 6 A system for performing the method for generating a coronary artery path diagram according to an embodiment of the present invention is shown. In some embodiments, the coronary artery path diagram generating device 600 may be a dedicated intelligent device or a general intelligent device. For example, the coronary artery path diagram generating device 600 may be a computer customized for the task of generating a coronary artery path diagram, or a server in the cloud. For example, the coronary artery path diagram generating device 600 may be integrated into an image processing device.

[0060] As an example, in the coronary artery path diagram generating device 600, at least an interface 601 and a processor 603 are included. In some embodiments, a memory 602 may also be included.

[0061] In some embodiments, interface 601 is configured to receive a sequence of dynamic fluoroscopic images and / or angiographic images of a coronary artery acquired by an imaging device. For example, interface 601 can receive a sequence of dynamic fluoroscopic images and / or angiographic images of a coronary artery acquired by various imaging devices via a communication cable, a wireless local area network (WLAN), a wide area network (WAN), a wireless network (such as via radio waves, cellular or telecommunications networks, and / or a local or short-range wireless network (e.g., BluetoothTM)), or other communication methods.

[0062] In some embodiments, interface 601 can include an Integrated Services Digital Network (ISDN) card, a cable modem, a satellite modem, or a modem to provide a data communication connection. In such an implementation, interface 601 can send and receive electrical, electromagnetic, and / or optical signals via a direct communication link, which carry an analog / digital data stream representing various types of information. In still other embodiments, interface 601 can further include a Local Area Network (LAN) card (e.g., an Ethernet adapter) to provide a data communication connection to a compatible LAN. As an example, interface 601 can further include a network interface 6011 via which the coronary artery roadmap generation device 600 can be connected to a network (not shown), such as including but not limited to a local area network in a hospital or the Internet. The network can connect the coronary artery roadmap generation device 600 to external devices such as an image acquisition device (not shown), an image database 604, and an image data storage device 605. The image acquisition device can be any device that acquires an image of an object, such as an MRI imaging device, a CT imaging device, a myocardial scintigraphy, an ultrasound device, or other medical imaging devices for obtaining a sequence of dynamic fluoroscopic images and / or angiographic images of a patient's coronary artery.

[0063] In some embodiments, the coronary artery roadmap generation device 600 can additionally include at least one of an input / output 606 and an image display 607.

[0064] The processor 603 is configured to execute the method for generating a coronary artery roadmap according to various embodiments of the present invention, and is a processing device including one or more general-purpose processing devices (such as a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), etc.). More specifically, the processor 603 can be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a processor running other instruction sets, or a processor running a combination of instruction sets. The processor 603 can also be one or more dedicated processing devices, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a system on a chip (SoC), and so on. As those skilled in the art will understand, in some embodiments, the processor 603 can be a dedicated processor rather than a general-purpose processor. The processor 603 can include one or more known processing devices, such as Pentium TM , Core TM , Xeon TM or Itanium TM series microprocessors manufactured by Intel Corporation, Turion TM , Athlon TM , Sempron TM , Opteron TM , FX TM , Phenom TM series microprocessors manufactured by AMD Corporation, or any of various processors manufactured by Sun Microsystems. The processor 603 can also include a graphics processing unit, such as GeForce , Quadro , Tesla series GPUs manufactured by Nvidia Corporation, GMA, Iris TM series GPUs manufactured by Intel TM, or Radeon TM series GPUs manufactured by AMD Corporation. The processor 603 can also include an accelerated processing unit, such as the desktop A-4(6,8) series manufactured by AMD Corporation, Xeon Phi TMSeries. The disclosed embodiments are not limited to any type of processor or processor circuit that is otherwise configured to meet the following computing requirements: performing methods such as the method for generating a coronary artery roadmap according to the embodiments of the present invention. Additionally, the term "processor" or "image processor" may include more than one processor, for example, a multi-core design or multiple processors, each of which has a multi-core design. The processor 603 may execute a sequence of computer program instructions stored in the memory 602 to perform various operations, processes, and methods disclosed herein.

[0065] The processor 603 may be communicatively coupled to the memory 602 and configured to execute computer-executable instructions stored therein. The memory 602 may include read-only memory (ROM), flash memory, random access memory (RAM), such as dynamic random access memory (DRAM) like synchronous DRAM (SDRAM) or Rambus DRAM, static memory (e.g., flash memory, static random access memory), etc., on which computer-executable instructions are stored in any format. The computer program instructions may be accessed by the processor 603, read from the ROM or any other suitable storage location, and loaded into the RAM for execution by the processor 603. For example, the memory 602 may store one or more software applications. The software applications stored in the memory 602 may include, for example, an operating system (not shown) for a general computer system and a soft control device (not shown). Additionally, the memory 602 may store the entire software application or only a part of the software application to be executable by the processor 603. Moreover, the memory 602 may store multiple software modules for implementing the various steps described in the embodiments of the present invention. Further, the memory 602 may store data generated / cached during the execution of the computer program, such as medical images sent from an image acquisition device, an image database 604, an image data storage device 605, etc.

[0066] In some embodiments, the learning network for automatic segmentation may be stored in the memory 602. In other embodiments, the learning network for segmentation may be stored in a remote device, a separate database (such as the image database 604), or a distributed device.

[0067] The input / output 606 may be configured to allow the coronary artery roadmap generation device 600 to receive and / or send data. The input / output 606 may include one or more digital and / or analog communication devices that allow the coronary artery roadmap generation device 600 to communicate with a user or other machines and devices. For example, the input / output 606 may include a keyboard and a mouse that allow a user to provide input.

[0068] The network interface 6011 may include a network adapter, a cable connector, a serial connector, a USB connector, a parallel connector, a high-speed data transmission adapter such as an optical fiber, USB 3.0, Lightning, a wireless network adapter such as a WiFi adapter, a telecommunications (3G, 4G / LTE, etc.) adapter. The coronary artery roadmap generation device 600 may be connected to a network through the network interface 6011. The network may provide the functions of a local area network (LAN), a wireless network, a cloud computing environment (e.g., software as a service, platform as a service, infrastructure as a service, etc.), a client-server, a wide area network (WAN), and the like.

[0069] In addition to angiography images and dynamic coronary artery roadmaps, the image display 607 may also display other information, such as the position parameters, thickness, etc. of blood vessels. For example, the image display 607 may be an LCD, CRT, or LED display.

[0070] The present invention describes various operations or functions, which may be implemented as software code or instructions or defined as software code or instructions. Such content may be source code that can be directly executed or differential code (“incremental” or “patch” code) (“object” or “executable” form). The software code or instructions may be stored in a computer-readable storage medium and, when executed, may cause a machine to perform the described functions or operations, and include any mechanism for storing information in a form accessible to a machine (e.g., a computing device, an electronic system, etc.), such as a recordable or non-recordable medium (e.g., read-only memory (ROM), random access memory (RAM), disk storage media, optical storage media, flash devices, etc.).

[0071] The exemplary methods described in the present invention can be implemented, at least in part, by a machine or a computer. In some embodiments, a computer-readable storage medium stores computer program instructions, and when the computer program instructions are run by a processor, the processor is caused to execute the method for generating a coronary artery roadmap according to various embodiments of the present invention. The implementation of such a method may include software code, such as microcode, assembly language code, high-level language code, etc. Various software programming techniques can be used to create various programs or program modules. For example, a program part or a program module can be designed in or with the aid of Java, Python, C, C++, assembly language, or any known programming language. One or more of such software parts or modules can be integrated into a computer system and / or a computer-readable medium. Such software code can include computer-readable instructions for performing various methods. Such software code can form part of a computer program product or a computer program module. In addition, in an example, the software code can be tangibly stored on one or more volatile, non-transitory, or non-volatile tangible computer-readable media, such as during execution or at other times. Examples of such tangible computer-readable media can include, but are not limited to, hard disks, removable disks, removable optical disks (such as optical discs and digital video discs), magnetic tape cartridges, memory cards or memory sticks, random access memory (RAM), read-only memory (ROM), etc.

[0072] Various modifications and changes can be made to the methods and apparatuses of the present invention. In view of the description and practice of the disclosed systems and related methods, other embodiments can be derived by those skilled in the art. Each of the various claims of the present invention can be understood as an independent embodiment, and any combination between them is also used as an embodiment of the present invention, and these embodiments are considered to be included in the present invention.

[0073] The descriptions and examples are intended to be exemplary only, and the true scope is represented by the appended claims and their equivalents.

Claims

1. A method for generating a coronary artery roadmap, characterized in that Including: Obtaining a dynamic fluoroscopic image of the coronary artery, where the fluoroscopic image is obtained by directly acquiring through various imaging modalities or by reconstructing based on the original images acquired by an imaging device; Obtaining a sequence of angiographic images of the coronary artery for a preset time period during which the heart performs periodic motion, and the preset time period includes at least one respiratory motion cycle, or the least common multiple cycle of the respiratory motion cycle and the cardiac cycle; Matching an angiographic image that matches the phase of the fluoroscopic image in the sequence of angiographic images, specifically including: Performing a first image segmentation on each angiographic image in the sequence of angiographic images to extract a first marker for tracking respiratory motion and a second marker for tracking heartbeat motion; Performing a second image segmentation on the fluoroscopic image to extract the corresponding first marker and second marker; By comparing the first marker and the second marker respectively extracted by the first image segmentation and the second image segmentation, matching an angiographic image that matches the phase of the fluoroscopic image in the sequence of angiographic images; Fusing the fluoroscopic image with the angiographic image that matches the phase to generate a dynamic coronary artery roadmap during the operation, where on the coronary artery in the fluoroscopic image, the segmented blood vessels of the coronary artery in the angiographic image that matches the phase are presented.

2. The generation method according to claim 1, wherein The dynamic fluoroscopic image of the coronary artery includes a real-time intraoperative fluoroscopic image of the coronary artery.

3. The generation method according to claim 1, wherein Fusing the fluoroscopic image with the angiographic image that matches the phase to generate a dynamic coronary artery roadmap during the operation further includes: on the coronary artery in the fluoroscopic image, presenting the segmented blood vessels of the coronary artery in the angiographic image that matches the phase, and the visual effect of the segmentation results at each location changes according to their confidence levels.

4. The generation method according to any one of claims 1 to 3, characterized in that Matching an angiographic image that matches the phase of the fluoroscopic image in the sequence of angiographic images specifically includes: Registering the fluoroscopic image with the sequence of angiographic images by comparing the geometric correlation features of the first marker and the second marker respectively extracted by the first image segmentation and the second image segmentation and / or the relative position between the first marker and the second marker, so as to match an angiographic image that matches the phase of the fluoroscopic image in the sequence of angiographic images.

5. The generation method according to claim 4, wherein The first marker includes anatomic markers of the diaphragm and / or lungs, and the second marker includes a catheter fixed to the coronary ostium and / or the cardiac silhouette.

6. The generation method according to claim 4, characterized in that, Registering the fluoroscopic image with the sequence of angiographic images by comparing the geometric correlation features of the first marker and the second marker respectively extracted by the first image segmentation and the second image segmentation and / or the relative position between the first marker and the second marker specifically includes: A first registration of the fluoroscopic image and the sequence of angiographic images is performed by comparing the geometrically related features of the first marker extracted from the first image segmentation and the second image segmentation respectively, together with the relative position between the first marker and the second marker, so as to match a first angiographic image in the sequence of angiographic images that is in phase with the fluoroscopic image; A second registration of the fluoroscopic image and the sequence of angiographic images is performed by comparing the geometrically related features of the second marker extracted from the first image segmentation and the second image segmentation respectively, together with the relative position between the first marker and the second marker, so as to match a second angiographic image in the sequence of angiographic images that is in phase with the fluoroscopic image; Determine the degree of difference between the first angiographic image and the second angiographic image. When the degree of difference is lower than the threshold, use the first angiographic image and / or the second angiographic image to obtain a finally matched angiographic image that is in phase with the fluoroscopic image; Or Compare the geometrically related features of the first marker and the second marker extracted from the first image segmentation and the second image segmentation respectively, together with the relative position between the first marker and the second marker, and register the fluoroscopic image and the sequence of angiographic images, so as to match an angiographic image in the sequence of angiographic images that is in phase with the fluoroscopic image.

7. The generating method according to claim 6, wherein The geometrically related features of the first marker include at least one of the fitting curve, curvature, position and shape of the first marker, and the geometrically related features of the second marker include at least one of the corner, position, direction, curvature, shape and width of the second marker.

8. The generation method according to any one of claims 1-3, characterized in that, In the sequence of angiographic images of the preset time period of the coronary artery, the concerned vascular segment remains in a filled state.

9. The generating method according to claim 4, wherein It further includes: Respectively use a learning network to perform the first image segmentation and the second image segmentation to obtain a first segmentation image and a second segmentation image. Among them, the second segmentation image has the same size as the fluoroscopic image, and the intensity value of each pixel indicates the probability-related parameter of the pixel belonging to the segmentation target. The first segmentation image has the same size as each angiographic image, and the intensity value of each pixel indicates the probability-related parameter of the pixel belonging to the segmentation target. The segmentation target at least includes blood vessels, the first marker and the second marker; Based on the first segmentation image, extract blood vessels, the first marker and the second marker in the angiographic image; Based on the second segmentation image, extract blood vessels, the first marker and the second marker in the fluoroscopic image.

10. The generation method according to claim 9, characterized in that, It further includes: Based on the first segmentation image, determine the confidence of each part of the extracted blood vessels for display associated with each part of the segmented blood vessels.

11. A device for generating a coronary artery roadmap, the device comprising a processor configured to execute the method for generating a coronary artery roadmap according to any one of claims 1-10.

12. A computer-readable storage medium having computer program instructions stored thereon, the computer program instructions causing the processor to execute the method for generating a coronary artery roadmap according to any one of claims 1-10 when run by the processor.

Images