CTA and DSA image intelligent fusion method, device, equipment and medium

Abstract

The present disclosure relates to a method, device, equipment and medium for intelligent fusion of CTA and DSA images, the method comprising: obtaining a two-dimensional image and a three-dimensional image of a patient's coronary artery, the two-dimensional image being a digital subtraction angiography image, and the three-dimensional image being a computed tomography angiography reconstruction image; determining the projection angle of the two-dimensional image when the patient is in a preset position; performing position correction processing on the three-dimensional image to obtain a first image of the coronary artery when the patient is in the preset position; performing registration and fusion based on the two-dimensional image, the projection angle and the first image to obtain a two-dimensional fused image; and displaying the fused image. The present disclosure can display information on completely occluded lesions and plaque properties through fused images by registering and fusing the two-dimensional and three-dimensional images of the patient's coronary artery, assisting doctors in diagnosing coronary artery lesions and planning interventional pathways, improving the success rate of treatment, and improving patient prognosis.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a method, device, equipment and medium for intelligent fusion of CTA and DSA images. Background Art

[0002] Percutaneous coronary intervention (PCI) is a commonly used method for treating coronary heart disease. It requires the introduction of special catheters, guidewires and other precision instruments into the coronary arteries under the guidance of digital subtraction angiography (DSA) to diagnose and treat coronary artery lesions. However, DSA images are two-dimensional images that can only evaluate the stenosis of the lumen of non-occluded blood vessels, but cannot display completely occluded lesions, which increases the uncertainty of interventional treatment. In addition, DSA images cannot evaluate the nature of the plaque that causes stenosis. The nature of the plaque greatly affects the success rate of interventional treatment of coronary atherosclerotic lesions (especially chronic completely occluded lesions). Therefore, how to supplement more information on the basis of DSA images to improve the success rate of interventional treatment based on DSA images is a technical problem that needs to be solved. Summary of the invention

[0003] In order to solve the above technical problems, the present disclosure provides a method, device, equipment and medium for intelligent fusion of CTA and DSA images.

[0004] A first aspect of an embodiment of the present disclosure provides a method for intelligent fusion of CTA and DSA images, the method comprising:

[0005] Acquire a two-dimensional image and a three-dimensional image of the patient's coronary artery, wherein the two-dimensional image is a digital subtraction angiography image, and the three-dimensional image is a computed tomography angiography reconstruction image;

[0006] determining a projection angle of the two-dimensional image when the patient is in a preset position;

[0007] Performing body position correction processing on the three-dimensional image to obtain a first image of the coronary artery when the patient is in the preset body position;

[0008] Performing registration and fusion based on the two-dimensional image, the projection angle and the first image to obtain a two-dimensional fused image;

[0009] The fused image is displayed.

[0010] A second aspect of the embodiments of the present disclosure provides a device for intelligent fusion of CTA and DSA images, the device comprising:

[0011] An acquisition module, used to acquire a two-dimensional image and a three-dimensional image of the patient's coronary artery, wherein the two-dimensional image is a digital subtraction angiography image, and the three-dimensional image is a computed tomography angiography reconstruction image;

[0012] An angle determination module, used to determine the projection angle of the two-dimensional image when the patient is in a preset position;

[0013] A correction module, used for performing a posture correction process on the three-dimensional image to obtain a first image of the coronary artery when the patient is in the preset posture;

[0014] A registration and fusion module, used for performing registration and fusion based on the two-dimensional image, the projection angle and the first image to obtain a two-dimensional fused image;

[0015] A display module is used to display the fused image.

[0016] A third aspect of an embodiment of the present disclosure provides a computer device, including a memory and a processor, and a computer program, wherein the computer program is stored in the memory, and when the computer program is executed by the processor, the CTA and DSA image intelligent fusion method as described in the first aspect above is implemented.

[0017] A fourth aspect of the embodiments of the present disclosure provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the CTA and DSA image intelligent fusion method as described in the first aspect above is implemented.

[0018] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

[0019] In the CTA and DSA image intelligent fusion method, device, equipment and medium provided in the embodiments of the present disclosure, a two-dimensional image and a three-dimensional image of the patient's coronary artery are obtained, the two-dimensional image is a digital subtraction angiography image, and the three-dimensional image is a computed tomography angiography reconstruction image, the projection angle of the two-dimensional image is determined when the patient is in a preset position, the three-dimensional image is subjected to position correction processing to obtain a first image of the coronary artery when the patient is in the preset position, and registration and fusion are performed based on the two-dimensional image, the projection angle and the first image to obtain a two-dimensional fused image, which is displayed. The fused image can display completely occluded lesions and plaque property information, assisting doctors in diagnosing coronary artery lesions and pre-interventional path planning, thereby improving the success rate of interventional treatment and improving the prognosis of patients with coronary heart disease. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0022] Figure 1 is a flow chart of a method for intelligent fusion of CTA and DSA images provided by an embodiment of the present disclosure;

[0023] Figure 2 is a flow chart of a method for image registration and fusion provided by an embodiment of the present disclosure;

[0024] Figure 3 is a flow chart of a method for image registration provided by an embodiment of the present disclosure;

[0025] Figure 4 is a flow chart of another method for image registration provided by an embodiment of the present disclosure;

[0026] Figure 5 is a flow chart of an image correction method provided by an embodiment of the present disclosure;

[0027] Figure 6 It is a structural schematic diagram of a CTA and DSA image intelligent fusion device provided by an embodiment of the present disclosure;

[0028] Figure 7 It is a structural diagram of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0030] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0031] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0032] Figure 1 FIG. 1 is a flow chart of a method for intelligent fusion of CTA and DSA images provided by an embodiment of the present disclosure. The method can be executed by a device for intelligent fusion of CTA and DSA images. Figure 1 As shown, the CTA and DSA image intelligent fusion method provided in this embodiment includes the following steps:

[0033] S101. Acquire a two-dimensional image and a three-dimensional image of a patient's coronary artery, wherein the two-dimensional image is a digital subtraction angiography image, and the three-dimensional image is a computed tomography angiography reconstruction image.

[0034] The two-dimensional image in the disclosed embodiment can be understood as a DSA image obtained by digital subtraction angiography (DSA) technology, the basic principle of which is to digitally process two frames of X-ray images taken before and after the injection of contrast agent and input them into an image computer, and eliminate the bone and soft tissue images in the image through the subtraction, enhancement and re-imaging process to obtain a clear pure vascular image, which is convenient for doctors to diagnose or perform surgery.

[0035] The three-dimensional image in the embodiment of the present disclosure can be understood as angiography examination performed through computed tomography angiography (CTA) technology to obtain enhanced scanning tomographic images, and then computer image three-dimensional reconstruction technology is used to perform image processing to form a three-dimensional image that displays the three-dimensional morphology of blood vessels. The basic principle is to use computed tomography (CT) technology to introduce contrast agents to reduce the permeability of blood to X-rays, so that the blood vessels appear as high-density shadows on the CT film, thereby distinguishing the blood vessels from other tissues.

[0036] In the disclosed embodiment, although the two-dimensional DSA image can accurately assess the degree of vascular lumen stenosis, it cannot provide information on the nature of plaques, and cannot even display chronic total occlusion lesions, which makes it difficult to assess the difficulty of interventional treatment and plan the pre-interventional path. The three-dimensional CTA image can not only provide information on the vascular lumen and plaques at the same time, but also clearly display chronic total occlusion lesions, which is of great value in assisting the interventional treatment of coronary atherosclerotic lesions (especially chronic total occlusion lesions).

[0037] In the disclosed embodiment, the CTA and DSA image intelligent fusion device can obtain a two-dimensional image and a three-dimensional image of the patient's coronary artery, wherein the two-dimensional image is a DSA image and the three-dimensional image is a CTA image.

[0038] In an exemplary implementation of the disclosed embodiment, the CTA and DSA image intelligent fusion device can obtain a pre-shot three-dimensional image of the patient's coronary artery, as well as a real-time shot two-dimensional image.

[0039] S102: Determine a projection angle of the two-dimensional image when the patient is in a preset position.

[0040] The preset body position in the embodiment of the present disclosure may be understood as a pre-set shooting body position. For example, the preset body position may be a normal body position.

[0041] In the disclosed embodiment, after obtaining a two-dimensional image of the patient's coronary artery, the CTA and DSA image intelligent fusion device can determine the projection angle used to capture the two-dimensional image when the patient is in a preset position based on the two-dimensional image.

[0042] In an exemplary implementation of the disclosed embodiment, the CTA and DSA image intelligent fusion device can restore the shooting scene based on the two-dimensional image, simulate the position of the radiation source when the patient is in a preset position, and then obtain the projection angle of the two-dimensional image.

[0043] S103, performing posture correction processing on the three-dimensional image to obtain a first image of the coronary artery when the patient is in the preset posture.

[0044] In the disclosed embodiment, the CTA and DSA image intelligent fusion device can perform position correction processing on the three-dimensional image after obtaining the three-dimensional image of the patient's coronary artery, correct the original three-dimensional image into a three-dimensional image when the patient is in a preset position, and determine it as the first image.

[0045] In an exemplary implementation of the disclosed embodiment, the CTA and DSA image intelligent fusion device can identify the original three-dimensional image, obtain the human body structure corresponding to each position in the image, and then correct the three-dimensional image according to the pre-acquired relative position relationship between the structures when the human body is in a preset position, so that the relative position relationship between the structures in the corrected image is the same as the relative position relationship in the preset position, and the corrected three-dimensional image is determined as the first image.

[0046] S104: performing registration and fusion based on the two-dimensional image, the projection angle and the first image to obtain a two-dimensional fused image.

[0047] In the disclosed embodiment, the CTA and DSA image intelligent fusion device can determine the mapping relationship between the plane coordinates of each pixel point in the two-dimensional image and the spatial coordinates of each spatial point in the first image after obtaining the two-dimensional image, the projection angle of the two-dimensional image, and the first image of the coronary artery when the patient is in a preset position, and project the first image into the two-dimensional image based on the mapping relationship to obtain a two-dimensional fused image.

[0048] In an exemplary implementation of the disclosed embodiment, the CTA and DSA image intelligent fusion device can determine, based on the projection angle and the first image, the images of each section when the first image is projected from the projection angle, and align the images of each section with the two-dimensional image to obtain the image of the section corresponding to the two-dimensional image, and then fuse the two-dimensional image with the image of the corresponding section to obtain a two-dimensional fused image.

[0049] S105: Display the fused image.

[0050] In the disclosed embodiment, the CTA and DSA image intelligent fusion device can display the fused image through an image output device, such as a display screen.

[0051] In an exemplary implementation of the disclosed embodiment, the CTA and DSA image intelligent fusion device can add highlight annotation information of completely occluded lesions and plate property information in the fused image when displaying the fused image, and display the annotated fused image.

[0052] The disclosed embodiment acquires a two-dimensional image and a three-dimensional image of the patient's coronary arteries, wherein the two-dimensional image is a digital subtraction angiography image, and the three-dimensional image is a computed tomography angiography reconstruction image, determines the projection angle of the two-dimensional image when the patient is in a preset body position, performs body position correction processing on the three-dimensional image to obtain a first image of the coronary arteries when the patient is in the preset body position, performs registration and fusion based on the two-dimensional image, the projection angle and the first image to obtain a two-dimensional fused image, and displays the fused image. The fused image can display information on the properties of completely occluded lesions and plaques, thereby assisting doctors in diagnosing coronary artery lesions and planning the path before intervention, thereby improving the success rate of interventional treatment and the prognosis of patients with coronary heart disease.

[0053] Figure 2 is a flow chart of a method for image registration and fusion provided by an embodiment of the present disclosure, such as Figure 2 As shown, based on the above embodiment, the image registration and fusion can be performed by the following method.

[0054] S201. Register the first image based on the two-dimensional image and the projection angle to obtain a two-dimensional second image corresponding to the first image.

[0055] The second image in the embodiment of the present disclosure may be understood as an image matching the two-dimensional image obtained by projecting the first image from a projection angle.

[0056] In the disclosed embodiment, after obtaining the projection angle of the two-dimensional image, the CTA and DSA image intelligent fusion device can align the first image with the two-dimensional image based on the projection angle to obtain a two-dimensional second image corresponding to the first image.

[0057] In an exemplary implementation of the presently disclosed embodiment, the CTA and DSA image intelligent fusion device can project the first image from a projection angle to obtain a projected image, and then align the projected image with the two-dimensional image to obtain a two-dimensional second image corresponding to the first image.

[0058] S202: Fuse the two-dimensional image with the second image to obtain the fused image.

[0059] In the disclosed embodiment, the CTA and DSA image intelligent fusion device can fuse the two-dimensional image with the second image to obtain a fused image after obtaining the second image through registration.

[0060] In an exemplary implementation of the presently disclosed embodiment, the CTA and DSA image intelligent fusion device can identify the second image after obtaining the second image, determine the location of the completely occluded lesion in the second image, and the plaque property information contained in the second image, and add the information obtained from the second image to the corresponding position in the two-dimensional image to obtain a fused image.

[0061] The disclosed embodiment obtains a two-dimensional second image corresponding to the first image by registering the first image based on the two-dimensional image and the projection angle, and fuses the two-dimensional image with the second image to obtain the fused image. The registered second image can be fused with the two-dimensional image so that the fused image can more accurately reflect the information on the properties of completely occluded lesions and plaques, better assist doctors in diagnosis and path planning, and further improve the success rate of interventional treatment.

[0062] Figure 3 is a flow chart of a method for image registration provided by an embodiment of the present disclosure, such as Figure 3 As shown, based on the above embodiment, the image can be registered by the following method.

[0063] S301. Perform image segmentation on the two-dimensional image to obtain a first coronary artery segmentation result.

[0064] The coronary segmentation result in the embodiment of the present disclosure may be understood as an image obtained by separating the coronary image from the original image and annotating it, and the first coronary segmentation result may be understood as a coronary segmentation result obtained by segmenting the two-dimensional image.

[0065] In the disclosed embodiment, the CTA and DSA image intelligent fusion device can perform image segmentation processing on the two-dimensional image after obtaining the two-dimensional image of the patient's coronary artery, separate the coronary artery image from the original two-dimensional image, and mark the area where the coronary artery image is located, thereby obtaining a first coronary artery segmentation result containing marking information.

[0066] In an exemplary implementation of the presently disclosed embodiment, the CTA and DSA image intelligent fusion device can extract the grayscale features of the two-dimensional image, and based on the pre-acquired grayscale features of the coronary arteries, as well as the similarity and continuity of the grayscale features, identify the area where the coronary arteries are located from the two-dimensional image, and label it to obtain a first coronary artery segmentation result.

[0067] S302: Perform image segmentation on the three-dimensional image to obtain a second coronary artery segmentation result.

[0068] The second coronary artery segmentation result in the embodiment of the present disclosure may be understood as the coronary artery segmentation result obtained after performing image segmentation on the three-dimensional image.

[0069] In the disclosed embodiment, the CTA and DSA image intelligent fusion device can perform image segmentation processing on the three-dimensional image after obtaining the three-dimensional image of the patient's coronary artery, separate the coronary artery image from the original three-dimensional image, and mark the area where the coronary artery image is located to obtain a second coronary artery segmentation result containing the marking information.

[0070] In an exemplary implementation of the presently disclosed embodiment, the CTA and DSA image intelligent fusion device can slice the three-dimensional image, perform image segmentation according to the grayscale features for each slice image, obtain the coronary artery segmentation result corresponding to each slice image, and then fuse the coronary artery segmentation results to obtain a second coronary artery segmentation result corresponding to the three-dimensional image.

[0071] In another exemplary implementation of the disclosed embodiment, the CTA and DSA image intelligent fusion device can directly extract grayscale features in the three-dimensional image, identify the area where the coronary arteries are located from the three-dimensional image based on the extracted grayscale features, and mark it to obtain a second coronary artery segmentation result.

[0072] S303: Determine a transformation matrix between the three-dimensional image and the first image.

[0073] In the embodiment of the present disclosure, the CTA and DSA image intelligent fusion device can perform posture correction processing on the three-dimensional image to obtain a first image of the coronary artery when the patient is in a preset posture, and then determine the spatial coordinates of the same point before and after the posture correction based on the three-dimensional image and the first image, and then determine the conversion matrix for coordinate conversion between the three-dimensional image and the first image based on the spatial coordinates of multiple points before and after the posture correction.

[0074] S304. Register the first image based on the two-dimensional image, the first coronary artery segmentation result, the projection angle, the second coronary artery segmentation result, and the transformation matrix to obtain a two-dimensional second image corresponding to the first image.

[0075] In the disclosed embodiment, the CTA and DSA image intelligent fusion device can align the first image according to the two-dimensional image, the first coronary artery segmentation result, the projection angle, the second coronary artery segmentation result and the transformation matrix. Specifically, the plane coordinates of each pixel point in the area where the coronary artery is located in the two-dimensional image can be determined based on the first coronary artery segmentation result, and the spatial coordinates of each point in the area where the coronary artery is located in the first image can be determined based on the second coronary artery segmentation result and the transformation matrix. Then, the two-dimensional image and the first image are aligned based on the plane coordinates, the spatial coordinates and the projection angle to obtain a mapping relationship between the plane coordinates and the spatial coordinates. Then, based on the mapping relationship, the spatial points of each spatial coordinate corresponding to each plane coordinate are displayed in the two-dimensional second image to obtain the two-dimensional second image corresponding to the first image.

[0076] In an exemplary implementation of the presently disclosed embodiment, the CTA and DSA image intelligent fusion device can extract coronary artery features in the two-dimensional image and the three-dimensional image respectively based on the first coronary artery segmentation result and the second coronary artery segmentation result, and then perform feature matching based on the coronary artery features to obtain a mapping relationship between the coronary artery image in the two-dimensional image and the coronary artery image in the three-dimensional image, and then obtain a mapping relationship between the coronary artery image in the two-dimensional image and the coronary artery image in the first image in combination with the transformation matrix, and project the first image into the two-dimensional image based on the mapping relationship to achieve registration of the two-dimensional image with the first image.

[0077] The disclosed embodiment obtains a first coronary artery segmentation result by performing image segmentation on the two-dimensional image, obtains a second coronary artery segmentation result by performing image segmentation on the three-dimensional image, determines a transformation matrix between the three-dimensional image and the first image, and registers the first image based on the two-dimensional image, the first coronary artery segmentation result, the projection angle, the second coronary artery segmentation result and the transformation matrix to obtain a two-dimensional second image corresponding to the first image. The accuracy of the registration between the two-dimensional image and the first image can be further improved according to the coronary artery segmentation result, thereby improving the accuracy of doctors' diagnosis of coronary artery lesions and pre-interventional path planning based on fused images.

[0078] In some embodiments of the present disclosure, the CTA and DSA image intelligent fusion device can input the two-dimensional image into a pre-trained two-dimensional image segmentation model to obtain the first coronary artery segmentation result, and input the three-dimensional image into a pre-trained three-dimensional image segmentation model to obtain the second coronary artery segmentation result, wherein the two-dimensional image segmentation model and the three-dimensional image segmentation model are trained based on a deep learning algorithm.

[0079] Specifically, the CTA and DSA image intelligent fusion device can input the two-dimensional image into the two-dimensional image segmentation model obtained by training based on the deep learning algorithm, and input the three-dimensional image into the three-dimensional image segmentation model obtained by training based on the deep learning algorithm when performing image segmentation on the two-dimensional image. For example, the two-dimensional image segmentation model and the three-dimensional image segmentation model can be a convolutional neural network (CNN) model, a fully convolutional network (FCN) model, a Unet model improved from the FCN model, or other models, which are not limited here. When training the two-dimensional image segmentation model, multiple two-dimensional training images and image annotation results corresponding to each image can be collected in advance, and the two-dimensional training images can be input into the two-dimensional image segmentation model framework to obtain the output result, and then determine the deviation between the output result and the annotation result, calculate the loss value based on the pre-set loss function and update the gradient, repeat the process to optimize the model parameters to minimize the loss function, and finally obtain the trained two-dimensional image segmentation model, and the three-dimensional image segmentation model is trained using the three-dimensional training image and the corresponding image annotation results. Its training process is similar to that of the three-dimensional image segmentation model, which will not be repeated here.

[0080] Some embodiments of the present disclosure can obtain coronary artery segmentation results through an image segmentation model, thereby improving the accuracy of image segmentation processing, thereby further improving the accuracy of registration between the two-dimensional image and the first image.

[0081] Figure 4FIG. 1 is a flowchart of another method for image registration provided by an embodiment of the present disclosure. Figure 4 As shown, based on the above embodiment, the image can be registered by the following method.

[0082] S401. Project the first image from the projection angle to obtain a third image.

[0083] In the disclosed embodiment, the CTA and DSA image intelligent fusion device can project the first image from the projection angle after obtaining the projection angle to obtain a two-dimensional third image.

[0084] S402. Obtain a third coronary artery segmentation result corresponding to the first image based on the second coronary artery segmentation result and the transformation matrix.

[0085] In the disclosed embodiment, the CTA and DSA image intelligent fusion device can obtain the second coronary artery segmentation result corresponding to the three-dimensional image and the transformation matrix between the three-dimensional image and the first image, and then transform the second coronary artery segmentation result according to the transformation matrix to obtain the third coronary artery segmentation result corresponding to the first image.

[0086] S403: Project the third coronary artery segmentation result from the projection angle to obtain a fourth coronary artery segmentation result.

[0087] In the disclosed embodiment, the CTA and DSA image intelligent fusion device can project the third coronary artery segmentation result from a projection angle to obtain a two-dimensional fourth coronary artery segmentation result.

[0088] S404. Register the third image based on the two-dimensional image, the first coronary artery segmentation result, and the fourth coronary artery segmentation result to obtain the second image.

[0089] In the disclosed embodiment, the CTA and DSA image intelligent fusion device can align the third image according to the two-dimensional image, the first coronary artery segmentation result and the fourth coronary artery segmentation result. Specifically, the plane coordinates of each pixel point in the area where the coronary artery is located in the two-dimensional image can be determined based on the first coronary artery segmentation result, and the plane coordinates of each pixel point in the area where the coronary artery is located in the third image can be determined based on the fourth coronary artery segmentation result. The two-dimensional image and the third image are then aligned based on the plane coordinates in the two-dimensional image and the plane coordinates in the third image to obtain a mapping relationship between the plane coordinates in the two-dimensional image and the plane coordinates in the third image. Based on the mapping relationship, the third image is coordinate-converted to obtain the second image.

[0090] In another exemplary implementation of the disclosed embodiment, the CTA and DSA image intelligent fusion device may input the two-dimensional image, the first coronary artery segmentation result, the first image and the fourth coronary artery segmentation result into a pre-trained registration model to obtain a second image output by the registration model, wherein the registration model may be obtained by training based on a reinforcement learning algorithm. When training the registration model, the two-dimensional training image, the three-dimensional projection training image, the coronary artery segmentation results corresponding to each training image and a pre-set evaluation index are input into the model framework to obtain an output result, and iterative training is performed according to the output result until the model converges to finally obtain a trained registration model, wherein the three-dimensional projection training image is obtained by projecting the three-dimensional image in the same body position as the two-dimensional training image from the projection angle of the two-dimensional training image.

[0091] The disclosed embodiment obtains a third image by projecting the first image from the projection angle, obtains a third coronary artery segmentation result corresponding to the first image based on the second coronary artery segmentation result and the transformation matrix, projects the third coronary artery segmentation result from the projection angle to obtain a fourth coronary artery segmentation result, and registers the third image based on the two-dimensional image, the first coronary artery segmentation result and the fourth coronary artery segmentation result to obtain the second image. When performing image registration, the three-dimensional first image can be first projected into a two-dimensional image, and then the two two-dimensional images can be registered in combination with the coronary artery segmentation result, so as to further improve the accuracy of the registration, thereby improving the accuracy of the doctor's diagnosis of coronary artery lesions and pre-interventional path planning based on the fused image.

[0092] In some other embodiments of the present disclosure, the CTA and DSA image intelligent fusion device can input the two-dimensional image, the first coronary artery segmentation result, the projection angle, the second coronary artery segmentation result, the transformation matrix and the first image into a pre-trained image registration model to obtain the second image, and the image registration model is trained based on a reinforcement learning algorithm.

[0093] Specifically, when registering the two-dimensional image with the first image, the CTA and DSA image intelligent fusion device can input the two-dimensional image, the first coronary segmentation result, the projection angle, the second coronary segmentation result, the transformation matrix and the first image into the image registration model trained based on the reinforcement learning algorithm, and the image registration model transforms the second coronary segmentation result based on the transformation matrix to obtain the coronary segmentation result corresponding to the first image, and then registers the two-dimensional image with the first image in combination with other input content to obtain the second image. For example, the image registration model can be a Gaussian Process Regression (GPR) model, a Gaussian Mixed Model (GMM), or other models, which are not limited here. When training the image registration model, the two-dimensional training images and the three-dimensional training images, the projection angles of the two-dimensional training images, the coronary segmentation results corresponding to each training image, and the pre-set evaluation indicators can be input into the model framework to obtain the output results, and iterative training is performed according to the output results until the model converges, and finally a trained image registration model is obtained, wherein when shooting the three-dimensional training images and the two-dimensional training images, the photographed objects are in the same body position, and the accuracy of image registration can be further improved based on the image registration model.

[0094] In some other embodiments of the present disclosure, the CTA and DSA image intelligent fusion device can input the two-dimensional image into a pre-trained projection angle determination model to obtain the projection angle of the two-dimensional image when the patient is in a preset position, wherein the projection angle determination model is trained based on a deep learning algorithm.

[0095] Specifically, the CTA and DSA image intelligent fusion device can input the two-dimensional image into the projection angle determination model obtained by training based on the deep learning algorithm when determining the projection angle of the two-dimensional image. For example, the projection angle determination model can be a CNN model, an FCN model, a deep residual network (DRN) model, or other models, which are not limited here. When training the projection angle determination model, multiple two-dimensional training images taken from different projection angles for patients in a preset position and the shooting angles corresponding to each two-dimensional training image can be collected in advance, and the two-dimensional training image is input into the projection angle determination model framework to obtain the output projection angle, and then determine the deviation between the output projection angle and the actual shooting angle, calculate the loss value based on the pre-set loss function and update the gradient, repeat the process to optimize the model parameters to minimize the loss function, and finally obtain the trained projection angle determination model.

[0096] Some embodiments of the present disclosure can determine the accurate projection angle of a two-dimensional image through a projection angle determination model, thereby improving the accuracy of image registration based on the projection angle, and subsequently obtaining a more accurate fused image.

[0097] Figure 5 is a flow chart of an image correction method provided by an embodiment of the present disclosure, such as Figure 5 As shown, based on the above embodiment, image correction can be performed by the following method.

[0098] S501: extracting a first feature point from the three-dimensional image.

[0099] The feature points in the embodiments of the present disclosure may be understood as points in an image that can reflect local features, and the first feature points may be understood as feature points contained in a three-dimensional image.

[0100] In the disclosed embodiment, the CTA and DSA image intelligent fusion device can extract a first feature point reflecting a local feature of the three-dimensional image from the three-dimensional image after obtaining the three-dimensional image.

[0101] In an exemplary implementation of the presently disclosed embodiment, the CTA and DSA image intelligent fusion device can extract the first feature point in the three-dimensional image based on a scale-invariant feature transform (SIFT) extraction algorithm or a speeded up robust features (SURF) extraction algorithm, or can extract the first feature point based on other algorithms, which are not limited here.

[0102] S502: extracting a second feature point from a template image of the coronary artery in a preset body position that is acquired in advance.

[0103] In the disclosed embodiment, the CTA and DSA image intelligent fusion device can obtain a template image taken when the human body is in a preset position as a standard coronary artery image, and extract the second feature point from the template image. The specific extraction method is similar to S501 and will not be repeated here.

[0104] S503: Match the first feature point with the second feature point to obtain a corresponding relationship between the first feature point and the second feature point.

[0105] In the disclosed embodiment, the CTA and DSA image intelligent fusion device can calculate the descriptor of each feature point based on the spatial position of each feature point after obtaining the first feature point in the three-dimensional image and the second feature point in the template image, and then match the first feature point with the second feature point according to the descriptor of each feature point, and extract the corresponding relationship between the first feature point and the second feature point according to the matching result.

[0106] In an exemplary implementation of the disclosed embodiment, when the CTA and DSA image intelligent fusion device matches the first feature point with the second feature point, it can use the Fast Library for Approximate Nearest Neighbors (FLANN) feature matching algorithm, or it can use the Brute Force (BF) feature matching algorithm, or it can use other algorithms, which are not limited here.

[0107] S504: Convert the three-dimensional image based on the corresponding relationship to obtain a first image of the coronary artery when the patient is in the preset position.

[0108] In the disclosed embodiment, the CTA and DSA image intelligent fusion device can obtain the correspondence between the first feature point and the second feature point, and then convert the three-dimensional image according to the correspondence to obtain a first image of the coronary artery when the patient is in the preset position.

[0109] In an exemplary implementation of the presently disclosed embodiment, the CTA and DSA image intelligent fusion device can extract the conversion relationship from each spatial point in the three-dimensional image to each spatial point in the first image based on the correspondence between the first feature point and the second feature point, and then convert the three-dimensional image based on the conversion relationship to obtain the first image.

[0110] The disclosed embodiment extracts a first feature point from the three-dimensional image, extracts a second feature point from a template image of the coronary artery in a preset position acquired in advance, matches the first feature point with the second feature point to obtain a correspondence between the first feature point and the second feature point, converts the three-dimensional image based on the correspondence to obtain a first image of the coronary artery when the patient is in the preset position, and can convert the patient's position in the three-dimensional image to the preset position by matching the three-dimensional image with a standard template image to obtain a first image of the coronary artery when the patient is in the preset position. Therefore, when performing image registration, registration is performed based on the two-dimensional image and the first image after the position is unified, thereby improving the accuracy of registration.

[0111] Figure 6FIG. 1 is a schematic diagram of a CTA and DSA image intelligent fusion device provided by an embodiment of the present disclosure. Figure 6 As shown, the CTA and DSA image intelligent fusion device 600 includes: an acquisition module 610, an angle determination module 620, a correction module 630, a registration and fusion module 640, and a display module 650, wherein the acquisition module 610 is used to acquire a two-dimensional image and a three-dimensional image of the patient's coronary artery, the two-dimensional image is a digital subtraction angiography image, and the three-dimensional image is a computed tomography angiography reconstruction image; the angle determination module 620 is used to determine the projection angle of the two-dimensional image when the patient is in a preset position; the correction module 630 is used to perform position correction processing on the three-dimensional image to obtain a first image of the coronary artery when the patient is in the preset position; the registration and fusion module 640 is used to perform registration and fusion based on the two-dimensional image, the projection angle and the first image to obtain a two-dimensional fused image; the display module 650 is used to display the fused image.

[0112] Optionally, the registration and fusion module 640 includes: a registration unit, used to register the first image based on the two-dimensional image and the projection angle to obtain a two-dimensional second image corresponding to the first image; and a fusion unit, used to fuse the two-dimensional image with the second image to obtain the fused image.

[0113] Optionally, the CTA and DSA image intelligent fusion device 600 also includes: a first segmentation module, used to perform image segmentation on the two-dimensional image to obtain a first coronary artery segmentation result; a second segmentation module, used to perform image segmentation on the three-dimensional image to obtain a second coronary artery segmentation result; a matrix determination module, used to determine the transformation matrix between the three-dimensional image and the first image; and the registration unit, specifically used to register the first image based on the two-dimensional image, the first coronary artery segmentation result, the projection angle, the second coronary artery segmentation result and the transformation matrix to obtain a two-dimensional second image corresponding to the first image.

[0114] Optionally, the first segmentation module is specifically used to input the two-dimensional image into a pre-trained two-dimensional image segmentation model to obtain the first coronary artery segmentation result; the second segmentation module is specifically used to input the three-dimensional image into a pre-trained three-dimensional image segmentation model to obtain the second coronary artery segmentation result, wherein the two-dimensional image segmentation model and the three-dimensional image segmentation model are trained based on a deep learning algorithm.

[0115] Optionally, the registration unit includes: a first projection subunit, used to project the first image from the projection angle to obtain a third image; a conversion subunit, used to obtain a third coronary artery segmentation result corresponding to the first image based on the second coronary artery segmentation result and the conversion matrix; a second projection subunit, used to project the third coronary artery segmentation result from the projection angle to obtain a fourth coronary artery segmentation result; and a registration subunit, used to register the third image based on the two-dimensional image, the first coronary artery segmentation result and the fourth coronary artery segmentation result to obtain the second pass image.

[0116] Optionally, the registration unit is specifically used to input the two-dimensional image, the first coronary artery segmentation result, the projection angle, the second coronary artery segmentation result, the transformation matrix and the first image into a pre-trained image registration model to obtain the second image, and the image registration model is trained based on a reinforcement learning algorithm.

[0117] Optionally, the angle determination module 620 is specifically used to input the two-dimensional image into a pre-trained projection angle determination model to obtain the projection angle of the two-dimensional image when the patient is in a preset position, wherein the projection angle determination model is trained based on a deep learning algorithm.

[0118] Optionally, the correction module 630 includes: a first extraction unit, used to extract a first feature point from the three-dimensional image; a second extraction unit, used to extract a second feature point from a pre-acquired template image of the coronary artery in a preset position; a matching unit, used to match the first feature point with the second feature point to obtain a correspondence between the first feature point and the second feature point; and a conversion unit, used to convert the three-dimensional image based on the correspondence to obtain a first image of the coronary artery when the patient is in the preset position.

[0119] The CTA and DSA image intelligent fusion device provided in this embodiment can execute the method described in any of the above embodiments. Its execution method and beneficial effects are similar and will not be described in detail here.

[0120] Figure 7 It is a structural diagram of a computer device provided in an embodiment of the present disclosure.

[0121] like Figure 7 As shown, the computer device may include a processor 710 and a memory 720 storing computer program instructions.

[0122] Specifically, the processor 710 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0123] The memory 720 may include a large capacity memory for information or instructions. By way of example and not limitation, the memory 720 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive or a combination of two or more of these. Where appropriate, the memory 720 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 720 may be inside or outside the integrated gateway device. In a particular embodiment, the memory 720 is a non-volatile solid-state memory. In a particular embodiment, the memory 720 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (Electrically Erasable Programmable ROM, EEPROM), an electrically rewritable ROM (EAROM) or a flash memory, or a combination of two or more of these.

[0124] The processor 710 reads and executes the computer program instructions stored in the memory 720 to perform the steps of the CTA and DSA image intelligent fusion method provided in the embodiment of the present disclosure.

[0125] In one example, the computer device may further include a transceiver 730 and a bus 740. Figure 7 As shown, the processor 710, the memory 720 and the transceiver 730 are connected via a bus 740 and communicate with each other.

[0126] The bus 740 includes hardware, software, or both. For example, but not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a Memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, the bus 740 may include one or more buses. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.

[0127] The embodiments of the present disclosure further provide a computer-readable storage medium, which may store a computer program. When the computer program is executed by a processor, the processor implements the CTA and DSA image intelligent fusion method provided by the embodiments of the present disclosure.

[0128] The storage medium mentioned above may, for example, include a memory 720 of computer program instructions, and the instructions may be executed by the processor 710 of the CTA and DSA image intelligent fusion device to complete the CTA and DSA image intelligent fusion method provided by the embodiment of the present disclosure. Optionally, the storage medium may be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium may be a ROM, a random access memory (Random Access Memory, RAM), a compact disc read-only memory (Compact Disc ROM, CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc. The computer program mentioned above may be written in any combination of one or more programming languages ​​to perform the program code of the operation of the embodiment of the present disclosure, and the programming language includes an object-oriented programming language, such as Java, C++, etc., and also includes a conventional procedural programming language, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on the remote computing device, or entirely on the remote computing device or server.

[0129] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0130] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for intelligent fusion of CTA and DSA images, characterized in that: include: Acquire a two-dimensional image and a three-dimensional image of the patient's coronary artery, wherein the two-dimensional image is a digital subtraction angiography image, and the three-dimensional image is a computed tomography angiography reconstruction image; determining a projection angle of the two-dimensional image when the patient is in a preset position; Performing body position correction processing on the three-dimensional image to obtain a first image of the coronary artery when the patient is in the preset body position; Performing registration and fusion based on the two-dimensional image, the projection angle and the first image to obtain a two-dimensional fused image; Displaying the fused image; The step of performing posture correction processing on the three-dimensional image to obtain a first image of the coronary artery when the patient is in the preset posture includes: Extracting a first feature point from the three-dimensional image; Extracting a second feature point from a template image of the coronary artery in a preset position acquired in advance; Matching the first feature point with the second feature point to obtain a corresponding relationship between the first feature point and the second feature point; The three-dimensional image is converted based on the corresponding relationship to obtain a first image of the coronary artery when the patient is in the preset position.

2. The method according to claim 1, characterized in that The registering and fusing the two-dimensional image, the projection angle and the first image to obtain a two-dimensional fused image includes: Registering the first image based on the two-dimensional image and the projection angle to obtain a two-dimensional second image corresponding to the first image; The two-dimensional image is fused with the second image to obtain the fused image.

3. The method according to claim 2, characterized in that After acquiring the two-dimensional image and the three-dimensional image of the patient's coronary artery, the method further includes: Performing image segmentation on the two-dimensional image to obtain a first coronary artery segmentation result; Performing image segmentation on the three-dimensional image to obtain a second coronary artery segmentation result; Before registering the first image based on the two-dimensional image and the projection angle to obtain a two-dimensional second image corresponding to the first image, the method further includes: determining a transformation matrix between the three-dimensional image and the first image; The registering the first image based on the two-dimensional image and the projection angle to obtain a two-dimensional second image corresponding to the first image includes: The first image is registered based on the two-dimensional image, the first coronary artery segmentation result, the projection angle, the second coronary artery segmentation result and the transformation matrix to obtain a two-dimensional second image corresponding to the first image.

4. The method according to claim 3, characterized in that The performing image segmentation on the two-dimensional image to obtain a first coronary artery segmentation result includes: Inputting the two-dimensional image into a pre-trained two-dimensional image segmentation model to obtain the first coronary artery segmentation result; The performing image segmentation on the three-dimensional image to obtain a second coronary artery segmentation result includes: The three-dimensional image is input into a pre-trained three-dimensional image segmentation model to obtain the second coronary artery segmentation result, wherein the two-dimensional image segmentation model and the three-dimensional image segmentation model are trained based on a deep learning algorithm.

5. The method according to claim 3, characterized in that: The registering the first image based on the two-dimensional image, the first coronary artery segmentation result, the projection angle, the second coronary artery segmentation result and the transformation matrix to obtain a two-dimensional second image corresponding to the first image includes: Projecting the first image from the projection angle to obtain a third image; Based on the second coronary artery segmentation result and the transformation matrix, obtaining a third coronary artery segmentation result corresponding to the first image; Projecting the third coronary artery segmentation result from the projection angle to obtain a fourth coronary artery segmentation result; The third image is registered based on the two-dimensional image, the first coronary artery segmentation result, and the fourth coronary artery segmentation result to obtain the second image.

6. The method according to claim 3, characterized in that The registering the first image based on the two-dimensional image, the first coronary artery segmentation result, the projection angle, the second coronary artery segmentation result and the transformation matrix to obtain a two-dimensional second image corresponding to the first image includes: The two-dimensional image, the first coronary artery segmentation result, the projection angle, the second coronary artery segmentation result, the transformation matrix and the first image are input into a pre-trained image registration model to obtain the second image, and the image registration model is trained based on a reinforcement learning algorithm.

7. The method according to claim 1, characterized in that When determining that the patient is in a preset position, the projection angle of the two-dimensional image includes: The two-dimensional image is input into a pre-trained projection angle determination model to obtain a projection angle of the two-dimensional image when the patient is in a preset position, wherein the projection angle determination model is trained based on a deep learning algorithm.

8. A CTA and DSA image intelligent fusion device, characterized in that: include: An acquisition module, used to acquire a two-dimensional image and a three-dimensional image of the patient's coronary artery, wherein the two-dimensional image is a digital subtraction angiography image, and the three-dimensional image is a computed tomography angiography reconstruction image; An angle determination module, used to determine the projection angle of the two-dimensional image when the patient is in a preset position; A correction module, used for performing a posture correction process on the three-dimensional image to obtain a first image of the coronary artery when the patient is in the preset posture; A registration and fusion module, used for performing registration and fusion based on the two-dimensional image, the projection angle and the first image to obtain a two-dimensional fused image; A display module, used for displaying the fused image; The correction module comprises: A first extraction unit, used to extract a first feature point from the three-dimensional image; A second extraction unit, configured to extract a second feature point from a template image of the coronary artery in a preset position acquired in advance; a matching unit, configured to match the first feature point with the second feature point to obtain a corresponding relationship between the first feature point and the second feature point; A conversion unit is used to convert the three-dimensional image based on the corresponding relationship to obtain a first image of the coronary artery when the patient is in the preset position.

9. A computer device, characterized in that: include: Memory; processor; and a computer program; wherein the computer program is stored in the memory and is configured to be executed by the processor to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the CTA and DSA image intelligent fusion method according to any one of claims 1 to 7 is implemented.

Images