Image registration method, apparatus and electronic device

Abstract

The application relates to the technical field of image processing, in particular to an image registration method and device and electronic equipment, the method comprises the following steps: acquiring a two-dimensional blood vessel centerline image and a three-dimensional blood vessel centerline image comprising a blood vessel segment of interest; extracting feature points from the two-dimensional blood vessel centerline image to obtain a first feature point image; projecting the three-dimensional blood vessel centerline image to a two-dimensional plane, and extracting feature points from the obtained two-dimensional projection image to obtain a second feature point image; inputting the first feature point image and the second feature point image into a pre-trained deep map neural network model to perform feature point matching, and obtaining a feature point matching result. The method can improve the matching accuracy, matching speed and matching density of the blood vessel feature points, and further improve the speed and accuracy of the projection parameter optimization and non-rigid registration.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image registration method, apparatus, and electronic device. Background Technology

[0002] Currently, cardiovascular disease has become the leading cause of death among urban and rural residents in China, and the disease burden is increasingly heavy. In recent years, with the continuous advancement of surgical techniques and equipment, minimally invasive percutaneous coronary intervention (PCI) has become the most mature and widely used treatment method for cardiovascular disease. As image-guided imaging for PCI surgery, coronary angiography (CAG) is currently recognized as the gold standard for diagnosing coronary artery disease. CAG images can dynamically display the entire coronary artery tree in real time, assisting doctors in performing interventional procedures.

[0003] Because CAG images are essentially two-dimensional images generated from three-dimensional information projection, they inherently have certain limitations, including: 1. CAG images lack spatial information about blood vessels, showing signs of shortening and stacking. 2. CAG images lack information about vascular plaques; while they can accurately show vascular stenosis, they cannot indicate the type of plaque causing it. 3. CAG images cannot display distal information of vessels with chronic total occlusion (CTO) lesions. Therefore, CAG images are not a "perfect navigation" for PCI procedures.

[0004] Preoperative coronary computed tomography angiography (CTA) is a three-dimensional imaging technique that can indicate the nature of plaques through the magnitude of CT values. Therefore, the spatial structure of blood vessels and plaques can be presented to doctors more intuitively through three-dimensional rendering and color coding, helping doctors to make better surgical decisions.

[0005] Therefore, multimodal image registration and fusion of preoperative CTA and CAG images can achieve complementary advantages between the two modalities, optimizing the PCI procedure from preoperative planning to surgical navigation and decision-making, thereby improving the efficiency and success rate of PCI surgery. Thus, developing efficient and accurate multimodal image registration and fusion methods for CTA and CAG images can benefit patients, doctors, and other stakeholders, and has significant clinical value and importance.

[0006] However, existing registration methods are typically based on point features and employ iterative optimization algorithms for matching and registration, failing to utilize the rich topological structure of the coronary artery tree. Furthermore, these algorithms have long runtimes, which cannot meet the real-time requirements of surgery. Alternatively, registration is based on image grayscale features, resulting in poor registration accuracy and poor performance for complex vessels. Or, traditional algorithms are used for topological matching, but these algorithms are slow and also perform poorly for complex vessels. Summary of the Invention

[0007] To address the aforementioned problems in the prior art, the present invention aims to provide an image registration method, apparatus, and electronic device that can improve the matching accuracy, matching speed, and matching density of vascular feature points, thereby improving the speed and accuracy of projection parameter optimization and non-rigid registration.

[0008] To address the above problems, the present invention provides an image registration method, comprising:

[0009] Acquire two-dimensional and three-dimensional images of the vessel centerline, including the vessel segment of interest.

[0010] Feature points are extracted from the two-dimensional blood vessel centerline image to obtain a first feature point image;

[0011] The three-dimensional blood vessel centerline image is projected onto a two-dimensional plane, and feature points are extracted from the obtained two-dimensional projected image to obtain a second feature point image.

[0012] The first feature point image and the second feature point image are input into a pre-trained deep graph neural network model for feature point matching to obtain the feature point matching result.

[0013] The deep graph neural network model is trained using a training sample dataset that includes multiple image sample pairs. Each image sample pair includes a first feature point image sample, a second feature point image sample, and the feature point matching relationship between the two. The first feature point image sample is the feature point image corresponding to the synthetic blood vessel centerline image sample, and the second feature point image sample is the feature point image corresponding to the three-dimensional blood vessel centerline image sample. The synthetic blood vessel centerline image sample is obtained based on the three-dimensional blood vessel centerline image sample using a data deformation method that embeds anatomical prior knowledge.

[0014] Further, acquiring the two-dimensional and three-dimensional vessel centerline images, including the vessel segment of interest, includes:

[0015] Acquire coronary angiography images and coronary CT angiography images of the vessel segment of interest;

[0016] The coronary angiography image is segmented and the centerline is extracted to obtain the two-dimensional vessel centerline image;

[0017] The coronary CT angiography image is segmented and the centerline is extracted to obtain the three-dimensional vascular centerline image.

[0018] Further, the step of extracting feature points from the two-dimensional blood vessel centerline image to obtain a first feature point image includes:

[0019] The two-dimensional blood vessel centerline image is input into a pre-trained feature point extraction network model to extract feature points, thereby obtaining the first feature point image.

[0020] The step of projecting the three-dimensional blood vessel centerline image onto a two-dimensional plane and extracting feature points from the resulting two-dimensional projected image to obtain a second feature point image includes:

[0021] The three-dimensional blood vessel centerline image is projected onto a two-dimensional plane to obtain the two-dimensional projected image;

[0022] The two-dimensional projection image is input into the feature point extraction network model to extract feature points, thereby obtaining the second feature point image.

[0023] Furthermore, the method also includes:

[0024] Based on the feature point matching results, the projection parameters of the three-dimensional blood vessel centerline image projected onto the two-dimensional plane are corrected to obtain a rigid registration result between the two-dimensional blood vessel centerline image and the three-dimensional blood vessel centerline image.

[0025] Further, projecting the three-dimensional vascular centerline image onto a two-dimensional plane includes:

[0026] The three-dimensional vascular centerline image is projected onto a two-dimensional plane using the corrected projection parameters to obtain the two-dimensional projected image.

[0027] Furthermore, the method also includes:

[0028] Based on the feature point matching results, non-rigid registration is performed on the two-dimensional blood vessel centerline image and the three-dimensional blood vessel centerline image to obtain the corresponding non-rigid registration results.

[0029] Furthermore, the method also includes pre-establishing the deep graph neural network model, the process of establishing the deep graph neural network model including:

[0030] A training sample dataset is obtained, which includes multiple image sample pairs. Each image sample pair includes a first feature point image sample, a second feature point image sample, and the feature point matching relationship between the two. The first feature point image sample is the feature point image corresponding to the synthetic blood vessel centerline image sample, and the second feature point image sample is the feature point image corresponding to the three-dimensional blood vessel centerline image sample. The synthetic blood vessel centerline image sample is obtained based on the three-dimensional blood vessel centerline image sample and uses a data deformation method embedded with anatomical prior knowledge.

[0031] A deep graph neural network model is pre-constructed, wherein the deep graph neural network model is a deep graph neural network model based on self-attention and cross-attention;

[0032] Using the first feature point image sample and the second feature point image sample as input data, and the corresponding feature point matching relationship as supervision, the deep graph neural network model is trained to obtain a trained deep graph neural network model.

[0033] Furthermore, obtaining the training sample dataset includes:

[0034] Acquire multiple three-dimensional blood vessel centerline image samples;

[0035] Each of the three-dimensional blood vessel centerline image samples is modeled as a three-dimensional hinge structure, and the obtained three-dimensional hinge structure three-dimensional blood vessel centerline image samples are randomly deformed.

[0036] The deformed three-dimensional blood vessel centerline image sample is subjected to simulated C-arm projection to obtain the corresponding synthetic blood vessel centerline image sample.

[0037] Based on each of the three-dimensional blood vessel centerline image samples and their corresponding synthetic blood vessel centerline image samples, the corresponding first feature point image sample, second feature point image sample, and the feature point matching relationship between the two are determined.

[0038] Another aspect of the present invention provides an image registration apparatus, comprising:

[0039] The image acquisition module is used to acquire two-dimensional and three-dimensional images of the vessel centerline, including the vessel segment of interest.

[0040] The first feature point extraction module is used to extract feature points from the two-dimensional blood vessel centerline image to obtain a first feature point image.

[0041] The second feature point extraction module is used to project the three-dimensional blood vessel centerline image onto a two-dimensional plane, and extract feature points from the obtained two-dimensional projection image to obtain a second feature point image.

[0042] The feature point matching module is used to input the first feature point image and the second feature point image into a pre-trained deep graph neural network model to perform feature point matching and obtain the feature point matching result.

[0043] The deep graph neural network model is trained using a training sample dataset that includes multiple image sample pairs. Each image sample pair includes a first feature point image sample, a second feature point image sample, and the feature point matching relationship between the two. The first feature point image sample is the feature point image corresponding to the synthetic blood vessel centerline image sample, and the second feature point image sample is the feature point image corresponding to the three-dimensional blood vessel centerline image sample. The synthetic blood vessel centerline image sample is obtained based on the three-dimensional blood vessel centerline image sample using a data deformation method that embeds anatomical prior knowledge.

[0044] In another aspect, the present invention provides an electronic device including a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the image registration method as described above.

[0045] In another aspect, the present invention provides a computer-readable storage medium storing at least one instruction or at least one program, wherein the at least one instruction or the at least one program is loaded and executed by a processor to implement the image registration method as described above.

[0046] Due to the above technical solution, the present invention has the following beneficial effects:

[0047] According to the image registration method of the present invention, an image sample pair including a first feature point image sample, a second feature point image sample, and the feature point matching relationship between the two are generated by using a data deformation method embedded with anatomical prior knowledge. This is used to train a deep graph neural network model, and the deep graph neural network model is used to match blood vessel feature points. This method utilizes the topological features of blood vessels and has fast matching speed, good matching effect, and strong model generalization performance. Compared with traditional matching algorithms, this method can significantly improve the matching accuracy, matching speed, and matching density of blood vessel feature points.

[0048] Furthermore, since the deep graph neural network model can obtain denser feature matching points, using the feature matching points obtained by the deep graph neural network model for subsequent projection parameter optimization and non-rigid registration steps can greatly improve the speed and accuracy of projection parameter optimization and non-rigid registration. Attached Figure Description

[0049] To more clearly illustrate the technical solutions of the present invention, the accompanying drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0050] Figure 1 This is a schematic diagram of the implementation environment provided in one embodiment of the present invention;

[0051] Figure 2 This is a flowchart of an image registration method provided in one embodiment of the present invention;

[0052] Figure 3 This is a schematic diagram of a feature point extraction process provided in an embodiment of the present invention;

[0053] Figure 4 This is a schematic diagram of feature point matching results provided in one embodiment of the present invention;

[0054] Figure 5 This is a flowchart of an image registration method provided in another embodiment of the present invention;

[0055] Figure 6 This is a flowchart of an image registration method provided in another embodiment of the present invention;

[0056] Figure 7 This is a flowchart of a training method for a deep graph neural network model provided in an embodiment of the present invention;

[0057] Figure 8 This is a schematic diagram illustrating the process of generating image sample pairs according to an embodiment of the present invention;

[0058] Figure 9 This is a schematic diagram of a self-attention mechanism and a cross-attention mechanism provided in an embodiment of the present invention;

[0059] Figure 10 This is a schematic diagram of the structure of an image registration device provided in one embodiment of the present invention;

[0060] Figure 11 This is a schematic diagram of the structure of an electronic device provided in one embodiment of the present invention. Detailed Implementation

[0061] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0063] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and examples. First, the following concepts will be explained in the embodiments of the present invention:

[0064] The PnP (Perspective-n-Points) algorithm aims to solve for the pose of the camera coordinate system relative to the world coordinate system. It describes how to estimate the camera pose (i.e., solve for the rotation matrix R and translation vector t from the world coordinate system to the camera coordinate system) when the coordinates (relative to the world coordinate system) of n three-dimensional (3D) points and their pixel coordinates are known.

[0065] Deep Graph Matching (DGM) Algorithm: The DGM algorithm is a method for matching image feature points based on Graph Neural Network (GNN).

[0066] Percutaneous Coronary Intervention (PCI): PCI is a medical procedure and the most common type of angioplasty, which can be used to treat diseases such as myocardial infarction, heart disease, and coronary artery disease.

[0067] Chronic Total Occlusion (CTO): CTO is defined as a lesion caused by atherosclerotic stenosis of the coronary artery lumen, resulting in complete obstruction of forward blood flow, and the disease course exceeds three months. It accounts for about 15% to 20% of coronary artery lesions and is known as the "last barrier in the field of coronary interventional surgery".

[0068] Digital Imaging and Communications in Medicine (DCM / DICOM) Labels: DCM (DICOM) labels are an international standard for medical images and related information. They define a medical image format that meets clinical needs and can be used for data exchange.

[0069] Reference manual attached Figure 1 This illustrates a schematic diagram of an implementation environment provided by an embodiment of the present invention. Figure 1 As shown, the implementation environment may include at least one medical scanning device 110 and a computer device 120. The computer device 120 and each medical scanning device 110 may be directly or indirectly connected by wired or wireless communication. This embodiment of the invention does not limit this.

[0070] The medical scanning device 110 may be, but is not limited to, a CAG device or a CTA device, and the computer device 120 may be, but is not limited to, various servers, personal computers, laptops, smartphones, tablets, and portable wearable devices. The server may be an independent server or a server cluster or distributed system composed of multiple servers. It may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0071] In this embodiment of the invention, the medical scanning device 110 can acquire images of the vascular segment of interest to obtain corresponding two-dimensional and three-dimensional vascular images. The computer device 120 can acquire the two-dimensional and three-dimensional vascular images obtained by the medical scanning device 110 and perform registration using the image registration method provided in this embodiment of the invention. This improves the registration accuracy and speed of the two-dimensional and three-dimensional vascular images, achieving complementary advantages between the two modalities. In the treatment of coronary artery lesions, especially CTO lesions, this can help doctors make better surgical decisions and improve the efficiency and success rate of PCI surgery.

[0072] For example, the medical scanning device 110 can acquire images of the vessel segment of interest based on CAG and CTA technologies, obtaining CAG and CTA images of the vessel segment of interest. The computer device 120 can acquire the CAG and CTA images obtained by the medical scanning device 110 and perform registration using the image registration method provided in this embodiment of the invention, thereby optimizing the PCI surgical process from preoperative planning to surgical navigation and surgical decision-making, and improving the efficiency and success rate of PCI surgery.

[0073] It should be noted that, Figure 1 This is merely an example. Those skilled in the art will understand that, although... Figure 1 Only one medical scanning device 110 is shown in the figure, but this does not constitute a limitation on the embodiments of the present invention. It may include more or fewer medical scanning devices 110 than shown in the figure.

[0074] Reference manual attached Figure 2 This illustrates the flow of an image registration method provided by an embodiment of the present invention, which can be applied to... Figure 1 Among the computer devices 120, specific examples include... Figure 2 As shown, the method may include the following steps:

[0075] S210: Acquire a two-dimensional and a three-dimensional image of the vessel centerline, including the vessel segment of interest.

[0076] In this embodiment of the invention, the segment of interest can be a segment of a blood vessel that is abnormal relative to normal blood vessels, or it can be a vascular tree that includes a segment of a blood vessel that is abnormal relative to normal blood vessels. For example, it can be various types of coronary artery segments / vascular trees, or various types of cerebral blood vessels / vascular trees, etc. This embodiment of the invention does not impose specific limitations on the type of blood vessel for the segment of interest.

[0077] In this embodiment of the invention, the two-dimensional blood vessel centerline image can be obtained by performing blood vessel segmentation and centerline extraction operations on a two-dimensional blood vessel image including the blood vessel segment of interest. Similarly, the three-dimensional blood vessel centerline image can be obtained by performing blood vessel segmentation and centerline extraction operations on a three-dimensional blood vessel image including the blood vessel segment of interest. Furthermore, the radius information of the blood vessel can be fused into the grayscale values ​​of both the two-dimensional and three-dimensional blood vessel centerline images. In other words, the radius information of the current blood vessel position can be determined based on the grayscale values ​​of each pixel in the blood vessel centerline image.

[0078] In one possible embodiment, acquiring the two-dimensional and three-dimensional vessel centerline images including the vessel segment of interest may include: acquiring coronary angiography (CAG) and coronary CT angiography (CTA) images including the vessel segment of interest; performing vessel segmentation and centerline extraction on the coronary angiography image to obtain the two-dimensional vessel centerline image; and performing vessel segmentation and centerline extraction on the coronary CT angiography image to obtain the three-dimensional vessel centerline image.

[0079] Specifically, the CAG / CTA images can be acquired directly, or they can be selected from CAG / CTA images of a segment of blood vessel corresponding to the segment of interest. This embodiment of the invention does not impose any limitations on this. The image data can be sourced from directly imported files, or it can be obtained from other resource libraries through real-time configuration and connection. This embodiment of the invention also does not impose any limitations on this.

[0080] In practical applications, images of the vessel segment of interest can be acquired using CAG and CTA technologies respectively, resulting in CAG and CTA images of the vessel segment of interest. The detailed acquisition process of the CAG and CTA images will not be elaborated further in this embodiment of the invention.

[0081] Specifically, existing vessel segmentation methods can be used to segment the vessels in the CAG image and the CTA image respectively, obtaining corresponding two-dimensional and three-dimensional vessel segmentation images. Then, existing centerline extraction methods are used to extract the vessel centerlines based on the two-dimensional and three-dimensional vessel segmentation images respectively, thus obtaining the two-dimensional vessel centerline image corresponding to the CAG image and the three-dimensional vessel centerline image corresponding to the CTA image. The detailed process of vessel segmentation and vessel centerline extraction will not be elaborated further in this embodiment of the invention.

[0082] Specifically, when extracting the blood vessel centerline, the radius information of the blood vessels at various locations along the centerline can be determined and integrated into the grayscale value of the blood vessel centerline image. This allows for the capture of the radius information of blood vessels at different locations during subsequent feature point extraction and matching. For example, the radius values ​​of the blood vessels at various locations along the centerline can be standardized and used as the grayscale values ​​of the corresponding pixels in the blood vessel centerline image.

[0083] S220: Extract feature points from the two-dimensional blood vessel centerline image to obtain a first feature point image.

[0084] In this embodiment of the invention, existing feature point extraction methods can be used to directly extract feature points from the two-dimensional blood vessel centerline image, thereby obtaining a first feature point image corresponding to the two-dimensional blood vessel centerline image. The first feature point image may include multiple feature points on the two-dimensional blood vessel centerline and their corresponding feature descriptors, whereby the feature descriptors can be used to describe the feature information of the corresponding feature points.

[0085] In one possible embodiment, extracting feature points from the two-dimensional blood vessel centerline image to obtain a first feature point image may include: inputting the two-dimensional blood vessel centerline image into a pre-trained feature point extraction network model to extract feature points and obtain the first feature point image.

[0086] Specifically, multiple two-dimensional blood vessel centerline image samples can be used as input in advance, and a feature point set can be used as output. A feature point extraction network model can be trained using a deep learning feature extraction algorithm, and the trained feature point extraction network model can be deployed on a computer device. After acquiring the two-dimensional blood vessel centerline image, the trained feature point extraction network model can be directly used to process the two-dimensional blood vessel centerline image to obtain the first feature point image.

[0087] In this embodiment of the invention, since the radius information of the blood vessel is fused into the grayscale value of the blood vessel centerline image when extracting the blood vessel centerline, the feature point extraction network model can capture the radius information of blood vessels at different locations. For example, the radius information of the blood vessel can be incorporated into the feature descriptor.

[0088] It is understandable that by fusing the radius information of blood vessels into the grayscale values ​​of the blood vessel centerline image, the feature point extraction network model can capture the radius information of blood vessels at different locations, thereby extracting richer feature information, which helps to improve the matching speed and matching effect of subsequent feature point matching.

[0089] It should be noted that the embodiments of the present invention do not specifically limit the feature extraction algorithm. In practical applications, those skilled in the art can select according to actual needs. For example, the feature extraction algorithm may include, but is not limited to, the Scale-Invariant Feature Transform (SIFT) algorithm, the SuperPoint algorithm, and the ORB (Oriented FAST and Rotated BRIEF) algorithm.

[0090] By way of example, in conjunction with the appendix to the reference specification Figure 3 This illustrates a schematic diagram of a feature point extraction process provided in an embodiment of the present invention. Figure 3As shown, segmenting blood vessels in a CAG image yields a two-dimensional blood vessel segmentation image, as shown in Figure (a1). Extracting the blood vessel centerline yields a two-dimensional blood vessel centerline image, as shown in Figure (b1). Further feature point extraction yields the first feature point image, as shown in Figure (c1).

[0091] S230: Project the three-dimensional blood vessel centerline image onto a two-dimensional plane, and extract feature points from the obtained two-dimensional projected image to obtain a second feature point image.

[0092] In this embodiment of the invention, the three-dimensional blood vessel centerline image can first be projected onto a two-dimensional plane to obtain a two-dimensional projection image. Then, existing feature point extraction methods are used to directly extract feature points from the two-dimensional projection image to obtain a second feature point image corresponding to the three-dimensional blood vessel centerline image. The second feature point image may include multiple feature points on the three-dimensional blood vessel centerline and their corresponding feature descriptors. The feature descriptors can be used to describe the feature information of the corresponding feature points.

[0093] In one possible embodiment, the step of projecting the three-dimensional blood vessel centerline image onto a two-dimensional plane and extracting feature points from the resulting two-dimensional projected image to obtain a second feature point image may include: projecting the three-dimensional blood vessel centerline image onto a two-dimensional plane to obtain the two-dimensional projected image; and inputting the two-dimensional projected image into the feature point extraction network model to extract feature points to obtain the second feature point image.

[0094] Specifically, the three-dimensional vascular centerline image can be projected onto a two-dimensional plane using a simulated C-arm projection. Projection parameters related to C-arm projection can be obtained first, and then these parameters can be used to project the three-dimensional vascular centerline image. These projection parameters can be read from the DCM label of the CAG image.

[0095] Specifically, after obtaining the two-dimensional projection image, the trained feature point extraction network model can be directly used to process the two-dimensional projection image to obtain the second feature point image.

[0096] In this embodiment of the invention, since the radius information of the blood vessel is fused into the grayscale value of the blood vessel centerline image when extracting the blood vessel centerline, the feature point extraction network model can capture the radius information of blood vessels at different locations. For example, the radius information of the blood vessel can be incorporated into the feature descriptor.

[0097] It is understandable that by fusing the radius information of blood vessels into the grayscale values ​​of the blood vessel centerline image, the feature point extraction network model can capture the radius information of blood vessels at different locations, thereby extracting richer feature information, which helps to improve the matching speed and matching effect of subsequent feature point matching.

[0098] For example, such as Figure 3 As shown, segmenting blood vessels in a CTA image yields a three-dimensional segmented blood vessel image, as shown in Figure (a2). Extracting the vessel centerline and projecting it onto a two-dimensional plane using a simulated C-arm model results in a simulated two-dimensional projection image of the centerline, as shown in Figure (b2). Further feature point extraction yields the second feature point image, as shown in Figure (c2).

[0099] S240: Input the first feature point image and the second feature point image into a pre-trained deep graph neural network model to perform feature point matching and obtain the feature point matching result.

[0100] The deep graph neural network model is trained using a training sample dataset that includes multiple image sample pairs. Each image sample pair includes a first feature point image sample, a second feature point image sample, and the feature point matching relationship between the two. The first feature point image sample is the feature point image corresponding to the synthetic blood vessel centerline image sample, and the second feature point image sample is the feature point image corresponding to the three-dimensional blood vessel centerline image sample. The synthetic blood vessel centerline image sample is obtained based on the three-dimensional blood vessel centerline image sample using a data deformation method that embeds anatomical prior knowledge.

[0101] In this embodiment of the invention, multiple three-dimensional vascular centerline image samples can be acquired first, and data deformation based on each three-dimensional vascular centerline image sample involving anatomical prior knowledge embedding can be performed to generate multiple image sample pairs with feature point matching relationships. Based on the multiple image sample pairs, a deep graph neural network model is trained using the DGM algorithm, and the trained deep graph neural network model is deployed on a computer device. The training method of the deep graph neural network model will be described in detail later.

[0102] In this embodiment of the invention, after obtaining the first feature point image and the second feature point image, the trained deep graph neural network model can be directly used to process the first feature point image and the second feature point image to obtain the corresponding feature point matching result.

[0103] Specifically, the deep graph neural network model can be a graph neural network based on self-attention and cross-attention mechanisms. This model treats feature points in the first and second feature point images as nodes in the graph neural network, updating node information through multiple rounds of self-attention and cross-attention mechanisms. Each feature point generates a matching vector used to calculate the matching score between points. Finally, a matching score matrix is ​​calculated using Dual-Softmax to determine the feature point matching result between the first and second feature point images.

[0104] Specifically, in the scoring matrix, the problem of topological inconsistency in vascular feature point matching can be solved by setting redundant rows and columns (for example, some points in CTA images cannot be found in CAG images).

[0105] It is understandable that by constructing a deep graph neural network model based on self-attention and cross-attention, the model can capture the topological relationships between feature points by calculating self-attention and cross-attention between feature points. Dynamic, sparse, and soft-corresponding graph structures can be established for feature points within and between images, which facilitates the rapid acquisition of accurate and dense feature matching points, thereby further improving the matching accuracy, matching speed, and matching density of the model.

[0106] Specifically, the feature point matching result may include multiple pairs of feature matching points and their corresponding matching relationships in the first feature point image and the second feature point image. For example, [the following is an example:] Figure 3 The first feature point image shown in (c1) and the second feature point image shown in (c2) are input into the depth map neural network model to obtain the following: Figure 4 The feature point matching results are shown.

[0107] In summary, the image registration method according to embodiments of the present invention generates image sample pairs including a first feature point image sample, a second feature point image sample, and the feature point matching relationship between the two by utilizing a data deformation method that embeds anatomical prior knowledge, thereby training a deep graph neural network model. The deep graph neural network model is then used to match vascular feature points, taking advantage of the topological features of blood vessels. Furthermore, the matching speed is fast, the matching effect is good, and the model has strong generalization performance. Compared with traditional matching algorithms, this method can significantly improve the matching accuracy, matching speed, and matching density of vascular feature points.

[0108] In one possible embodiment, referencing the appendix to the specification... Figure 5 The method may further include the following steps:

[0109] S250: Based on the feature point matching results, the projection parameters of the three-dimensional blood vessel centerline image projected onto the two-dimensional plane are corrected to obtain a rigid registration result between the two-dimensional blood vessel centerline image and the three-dimensional blood vessel centerline image.

[0110] In this embodiment of the invention, after obtaining the feature point matching result, a projection parameter correction step can be performed based on the extracted feature matching points to correct the projection parameters of the three-dimensional blood vessel centerline image projected onto the two-dimensional plane, thereby obtaining a rigid registration result. For example, the projection parameters can be corrected using the PnP algorithm based on the extracted feature matching points to obtain a rigid registration result. The specific process of using the PnP algorithm for projection parameter correction will not be described in detail in this embodiment of the invention.

[0111] In one possible embodiment, after correcting the projection parameters, steps S230 to S250 can be repeated. In step S230, the three-dimensional blood vessel centerline image is reprojected onto a two-dimensional plane using the corrected projection parameters to obtain a new two-dimensional projected image, which in turn yields a new second feature point image. In step S240, a depth map neural network model is used to perform feature point matching between the first feature point image and the new second feature point image to obtain a new feature point matching result. In step S250, based on the new feature point matching result, the corrected projection parameters are corrected again to obtain a new rigid registration result.

[0112] It should be noted that in practical applications, the above process can be repeated multiple times (e.g., 2-3 times) to obtain more accurate rigid registration results. Since the algorithm runs quickly, repeated iterations will not affect the efficiency of image registration.

[0113] It is understandable that by using the feature point matching results based on the model output to optimize the projection parameters using the PnP algorithm, the optimization speed and effect of this algorithm are superior to traditional iterative projection parameter optimization algorithms, thus further improving the speed and accuracy of projection parameter optimization.

[0114] In one possible embodiment, referencing the appendix to the specification... Figure 6 The method may further include the following steps:

[0115] S260: Based on the feature point matching results, perform non-rigid registration on the two-dimensional blood vessel centerline image and the three-dimensional blood vessel centerline image to obtain the corresponding non-rigid registration results.

[0116] In this embodiment of the invention, the corrected projection parameters (i.e., rigid registration results) obtained in step S250 can be used as the initial registration result, and steps S230 and S240 can be repeated to obtain more accurate feature point matching results. Then, using the obtained more accurate feature point matching results, further non-rigid registration is performed to obtain non-rigid registration results, which are used as the final registration results.

[0117] In one possible embodiment, after obtaining the non-rigid registration result, steps S230, S240, and S260 can be repeated to obtain a more accurate rigid registration result. It should be noted that in practical applications, the above process can be repeated multiple times (e.g., 2-3 times) to improve the accuracy of the registration result. Since the algorithm runs quickly, multiple repetitions will not affect the efficiency of image registration.

[0118] It is understandable that by using feature matching points constructed based on topological relationships in global non-rigid registration, since the deep graph neural network model can obtain fairly dense and accurate feature matching points, the number of iterations in the registration process can be greatly reduced and the registration effect can be improved. The non-rigid registration process can be completed in a very short time, which greatly improves the speed and accuracy of non-rigid registration.

[0119] Reference manual attached Figure 7 This illustrates the flowchart of a training method for a deep graph neural network model provided in an embodiment of the present invention. Specifically, as shown... Figure 7 As shown, the method may include the following steps:

[0120] S710: Obtain a training sample dataset, which includes multiple image sample pairs. Each image sample pair includes a first feature point image sample, a second feature point image sample, and the feature point matching relationship between the two. The first feature point image sample is the feature point image corresponding to the synthetic blood vessel centerline image sample, and the second feature point image sample is the feature point image corresponding to the three-dimensional blood vessel centerline image sample. The synthetic blood vessel centerline image sample is obtained based on the three-dimensional blood vessel centerline image sample using a data deformation method embedded with anatomical prior knowledge.

[0121] In this embodiment of the invention, since a large amount of training sample data is required during the model training phase (especially for cases using the Transformer structure), and the point matching relationships of medical images are difficult to obtain due to a lack of data and significant non-rigid deformation, training sample data can be generated based on data deformation embedded with anatomical prior knowledge. Specifically, multiple three-dimensional vascular centerline image samples, including segments of interest, can be pre-acquired, and the data deformation method embedded with anatomical prior knowledge can be used to generate image sample pairs with feature point matching relationships corresponding to each of the three-dimensional vascular centerline image samples, thereby obtaining the training sample dataset including multiple image sample pairs. Specifically, the image sample pair may include a first feature point image sample corresponding to the synthetic vascular centerline image sample obtained using the data deformation method embedded with anatomical prior knowledge, a second feature point image sample corresponding to the three-dimensional vascular centerline image sample, and the feature point matching relationship between the two.

[0122] In one possible embodiment, acquiring the training sample dataset may include: acquiring multiple three-dimensional blood vessel centerline image samples; modeling each of the three-dimensional blood vessel centerline image samples as a three-dimensional hinge structure, and randomly deforming the obtained three-dimensional hinge structure three-dimensional blood vessel centerline image samples; performing simulated C-arm projection on the deformed three-dimensional blood vessel centerline image samples to obtain corresponding synthetic blood vessel centerline image samples; and determining corresponding first feature point image samples, second feature point image samples, and the feature point matching relationship between them based on each of the three-dimensional blood vessel centerline image samples and their corresponding synthetic blood vessel centerline image samples.

[0123] Specifically, the three-dimensional vessel centerline image sample can be obtained by performing vessel segmentation and centerline extraction operations on three-dimensional vessel image samples including the vessel segment of interest. For example, based on CTA technology, multiple CTA image samples including the vessel segment of interest from different objects can be acquired, and vessel segmentation can be performed on each of the CTA image samples to obtain the corresponding three-dimensional vessel segmentation image sample. Then, based on the three-dimensional vessel segmentation image sample, vessel centerline extraction can be performed to obtain the three-dimensional vessel centerline image sample corresponding to each of the CTA image samples.

[0124] It should be noted that the above implementation method of generating a training sample dataset using three-dimensional blood vessel centerline image samples including the blood vessel segment of interest is merely an example. In practical applications, it is not limited to image samples including the blood vessel segment of interest. That is, image samples including blood vessel segments different from the blood vessel segment of interest can also be used to train the deep graph neural network model, and the embodiments of the present invention do not impose specific limitations on this.

[0125] In practical applications, a corresponding deep graph neural network model can be trained for each type of blood vessel segment, or a deep graph neural network model applicable to all types of blood vessel segments can be trained. This embodiment of the invention does not impose any specific limitations on this.

[0126] Specifically, in the data deformation process of embedding anatomical prior knowledge, the 3D vascular centerlines in each of the three-dimensional vascular centerline image samples can first be modeled as 3D hinge structures. An angular offset field is then generated for each point of the 3D hinge structure vascular centerline image sample in spherical coordinates, generating realistic non-rigid vascular deformation to simulate cardiac and respiratory conditions. Then, by simulating the C-arm projection process, projection parameter errors between CTA and angiography due to patient positioning and equipment variations can be simulated using body position angles and randomly generated body position angle deviations. Optionally, topological inconsistencies between CTA and angiography can be simulated by randomly discarding blood vessels or randomly removing or adding noise points. Through the above methods, a synthetic vascular centerline image sample can be obtained, along with multiple pairs of feature matching points and their corresponding feature point matching relationships between the synthetic vascular centerline image sample and the three-dimensional vascular centerline image sample.

[0127] Specifically, the synthetic blood vessel centerline image sample and the three-dimensional blood vessel centerline image sample are projected onto a two-dimensional plane, and the generated multiple pairs of feature matching points and their corresponding feature point matching relationships are combined to obtain the first feature point image sample corresponding to the synthetic blood vessel centerline image sample, the second feature point image sample corresponding to the three-dimensional blood vessel centerline image sample, and the feature point matching relationship between the two, which are used for training the deep graph neural network model.

[0128] By way of example, in conjunction with the appendix to the reference specification Figure 8 This illustrates a schematic diagram of the process for generating image sample pairs according to an embodiment of the present invention. Figure 8 As shown, based on the three-dimensional blood vessel centerline image sample shown in Figure (1) a, the synthetic blood vessel centerline image sample shown in Figure (1) b can be obtained by deformation. Projecting the three-dimensional blood vessel centerline image sample onto a two-dimensional plane, the centerline image of the two-dimensional plane shown in Figure (2) a can be obtained; projecting the synthetic blood vessel centerline image sample onto a two-dimensional plane, the centerline image of the two-dimensional plane shown in Figure (2) b can be obtained. Combining the generated multiple pairs of feature matching points and their corresponding feature point matching relationships, the second feature point image sample corresponding to the three-dimensional blood vessel centerline image sample shown in Figure (3) a, the first feature point image sample corresponding to the synthetic blood vessel centerline image sample shown in Figure (3) b, and the feature point matching relationship between the two can be obtained.

[0129] It is understandable that by using data deformation methods embedded with anatomical prior knowledge to generate training sample data, the problem of difficulty in obtaining point matching relationships in medical images due to lack of data and large-scale non-rigid deformation can be solved, thereby improving the matching effect and generalization ability of the trained deep graph neural network model.

[0130] S720: A deep graph neural network model is pre-built, which is a deep graph neural network model based on self-attention and cross-attention.

[0131] In this embodiment of the invention, a deep graph neural network model can be pre-constructed based on the DGM algorithm. The construction process of the deep graph neural network model can refer to the existing technology, and will not be described in detail here.

[0132] Specifically, the deep graph neural network model can be a graph neural network based on self-attention and cross-attention mechanisms. This deep graph neural network model can utilize self-attention and cross-attention to construct dynamic, sparse, and soft-corresponding graph structures at feature points within and between images, respectively, to match vascular feature points using the topological features of blood vessels.

[0133] By way of example, in conjunction with the appendix to the reference specification Figure 9 This illustrates a schematic diagram of a self-attention mechanism and a cross-attention mechanism provided in an embodiment of the present invention. Figure 9 As shown, in shallow layers of the model (e.g., the first layer), self-attention can capture vascular information globally within an image, while cross-attention alternates between images globally to capture vascular information. In deeper layers (e.g., the twelfth layer), self-attention captures vascular information only locally within an image, and cross-attention alternates between images locally to capture vascular information. This local range can be the area containing feature points that roughly match the current feature point. Figure 9 This demonstrates that the image registration method provided in the embodiments of the present invention has good interpretability and is beneficial for clinical use.

[0134] It should be noted that the above implementation of using graph neural networks based on self-attention and cross-attention mechanisms as deep graph neural network models is only an example. The embodiments of the present invention do not impose specific restrictions on the structure of deep graph neural network models. In practical applications, those skilled in the art can determine the structure according to actual needs.

[0135] It is understandable that by constructing a deep graph neural network model based on self-attention and cross-attention, the model can capture the topological relationships between feature points by calculating self-attention and cross-attention between feature points. Dynamic, sparse, and soft-corresponding graph structures can be established for feature points within and between images, which facilitates the rapid acquisition of accurate and dense feature matching points, thereby further improving the matching accuracy, matching speed, and matching density of the model.

[0136] S730: Using the first feature point image sample and the second feature point image sample as input data, and the corresponding feature point matching relationship as supervision, the deep graph neural network model is trained to obtain a trained deep graph neural network model.

[0137] In this embodiment of the invention, each image sample pair in the training sample dataset can be used as input data, and the corresponding feature point matching relationship can be used as supervision to perform supervised training on the pre-constructed deep graph neural network model, so as to obtain a trained deep graph neural network model.

[0138] Specifically, the deep graph neural network model treats the feature matching points in the first and second feature point image samples as nodes in the graph neural network, and updates the node information through multiple rounds of self-attention and cross-attention mechanisms. Each feature matching point generates a matching vector used to calculate the matching score between points. Finally, a matching score matrix is ​​calculated using Dual-Softmax to determine the predicted matching result between the first and second feature point image samples. The predicted feature point matching result may include multiple pairs of predicted matching points in the first and second feature point image samples and their corresponding matching relationships.

[0139] Specifically, to address the matching difficulties caused by non-rigid deformation in medical image matching, the training process of the deep graph neural network model can utilize a focal loss function and a distance loss function to supervise the learning of the predicted matching results. Focal Loss assigns adaptive weights to easily matched and difficult-to-match points, allowing the network to focus more on the difficult-to-match points. Dist Loss, on the other hand, is obtained by calculating the Euclidean distance between the feature matching point and the predicted matching point at a given location.

[0140] It is understandable that the Dist Loss loss function, when the feature matching points are relatively dense, will give a smoother and continuous penalty for prediction errors of different degrees, thereby making up for the shortcomings of Focal Loss and improving the performance of the trained deep graph neural network model in non-rigid medical scenarios.

[0141] It should be noted that the specific details of the model training process can be found in existing technologies, and will not be repeated here in the embodiments of this invention. Other relevant content in the embodiments of this invention can be found in [reference needed]. Figures 2 to 6 The specific details of the method provided in the illustrated embodiment will not be repeated here.

[0142] In summary, by utilizing data deformation methods embedded with anatomical prior knowledge to generate image sample pairs including first feature point image samples, second feature point image samples, and the feature point matching relationship between them, a deep graph neural network model based on vascular topological features can be trained. This results in a deep graph neural network model with fast matching speed, good matching effect, and strong model generalization performance. Using the trained model for feature point matching significantly improves the accuracy, speed, and density of vascular feature point matching compared to traditional matching algorithms.

[0143] Furthermore, since the deep graph neural network model can obtain denser feature matching points, using the feature matching points obtained by the deep graph neural network model for subsequent projection parameter optimization and non-rigid registration steps can greatly improve the speed and accuracy of projection parameter optimization and non-rigid registration.

[0144] Reference manual attached Figure 10 This illustrates the structure of an image registration apparatus 1000 provided in one embodiment of the present invention. For example... Figure 10 As shown, the device 1000 may include:

[0145] The image acquisition module 1010 is used to acquire two-dimensional and three-dimensional images of the vessel centerline, including the vessel segment of interest.

[0146] The first feature point extraction module 1020 is used to extract feature points from the two-dimensional blood vessel centerline image to obtain a first feature point image.

[0147] The second feature point extraction module 1030 is used to project the three-dimensional blood vessel centerline image onto a two-dimensional plane, and extract feature points from the obtained two-dimensional projection image to obtain a second feature point image.

[0148] The feature point matching module 1040 is used to input the first feature point image and the second feature point image into a pre-trained deep graph neural network model to perform feature point matching and obtain the feature point matching result.

[0149] The deep graph neural network model is trained using a training sample dataset that includes multiple image sample pairs. Each image sample pair includes a two-dimensional blood vessel centerline image sample, a three-dimensional blood vessel centerline image sample, and the feature point matching relationship between the two. The two-dimensional blood vessel centerline image sample is obtained based on the three-dimensional blood vessel centerline image sample using a data deformation method that embeds anatomical prior knowledge.

[0150] In one possible embodiment, the device 1000 may further include:

[0151] A rigid registration module is used to correct the projection parameters of the three-dimensional blood vessel centerline image projected onto a two-dimensional plane based on the feature point matching results, so as to obtain a rigid registration result between the two-dimensional blood vessel centerline image and the three-dimensional blood vessel centerline image.

[0152] In one possible embodiment, the device 1000 may further include:

[0153] The non-rigid registration module is used to perform non-rigid registration on the two-dimensional blood vessel centerline image and the three-dimensional blood vessel centerline image based on the feature point matching results, so as to obtain the corresponding non-rigid registration results.

[0154] In one possible embodiment, the device 1000 may further include a model training module for pre-establishing the deep graph neural network model;

[0155] The model training module may include:

[0156] A sample data acquisition unit is used to acquire a training sample dataset, which includes multiple image sample pairs. Each image sample pair includes a first feature point image sample, a second feature point image sample, and a feature point matching relationship between the two. The first feature point image sample is a feature point image corresponding to a synthetic blood vessel centerline image sample, and the second feature point image sample is a feature point image corresponding to a three-dimensional blood vessel centerline image sample. The synthetic blood vessel centerline image sample is obtained based on the three-dimensional blood vessel centerline image sample using a data deformation method embedded with anatomical prior knowledge.

[0157] The model building unit is used to pre-build a deep graph neural network model, wherein the deep graph neural network model is a deep graph neural network model based on self-attention and cross-attention;

[0158] The model training unit is used to train the deep graph neural network model using the first feature point image sample and the second feature point image sample as input data and the corresponding feature point matching relationship as supervision, so as to obtain the trained deep graph neural network model.

[0159] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus provided in the above embodiments and the corresponding method embodiments belong to the same concept, and the specific implementation process can be found in the corresponding method embodiments, which will not be repeated here.

[0160] An embodiment of the present invention also provides an electronic device, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the image registration method provided in the above method embodiments or to implement the training method of the deep graph neural network model provided in the above method embodiments.

[0161] The memory can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, application programs required for the functions, etc.; the data storage area can store data created according to the use of the device, etc. In addition, the memory can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory can also include a memory controller to provide the processor with access to the memory.

[0162] In one specific embodiment Figure 11 This diagram illustrates a hardware structure of an electronic device for implementing the image registration method or the training method of a deep graph neural network model provided in the embodiments of the present invention. The electronic device can be a computer terminal, a mobile terminal, or other devices. The electronic device can also participate in or include the image registration apparatus provided in the embodiments of the present invention. Figure 11 As shown, the electronic device 1100 may include a memory 1110 of one or more computer-readable storage media, a processor 1120 of one or more processing cores, an input unit 1130, a display unit 1140, a radio frequency (RF) circuit 1150, a wireless fidelity (WiFi) module 1160, and a power supply 1170, etc. Those skilled in the art will understand that... Figure 11The electronic device structure shown does not constitute a limitation on electronic device 1100, and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0163] The memory 1110 can be used to store software programs and modules. The processor 1120 performs various functional applications and data processing by running or executing the software programs and modules stored in the memory 1110 and calling data stored in the memory 1110. The memory 1110 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function, etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 1110 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 1110 may also include a memory controller to provide the processor 1120 with access to the memory 1110.

[0164] The processor 1120 is the control center of the electronic device 1100. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 1110, and by calling data stored in the memory 1110, it performs various functions and processes data of the electronic device 1100, thereby providing overall monitoring of the electronic device 1100. The processor 1120 can be a central processing unit, or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0165] The input unit 1130 can be used to receive input numerical or character information, and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control. Specifically, the input unit 1130 may include a touch-sensitive surface 1131 and other input devices 1132. Specifically, the touch-sensitive surface 1131 may include, but is not limited to, a touchpad or a touch screen, and other input devices 1132 may include, but are not limited to, one or more of the following: a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, and a joystick.

[0166] The display unit 1140 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of electronic devices. These graphical user interfaces can be composed of graphics, text, icons, videos, and any combination thereof. The display unit 1140 may include a display panel 1141, which may optionally be configured as a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0167] The RF circuit 1150 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and hands it over to one or more processors 1120 for processing; additionally, it transmits uplink data to the base station. Typically, the RF circuit 1150 includes, but is not limited to, an antenna, at least one amplifier, a tuner, one or more oscillators, a Subscriber Identity Module (SIM) card, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc. Furthermore, the RF circuit 1150 can also communicate wirelessly with networks and other devices. The wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0168] WiFi is a short-range wireless transmission technology. Electronic device 1100, through WiFi module 1160, can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access. Although Figure 11 WiFi module 1160 is shown, but it is understood that it is not a necessary component of electronic device 1100 and can be omitted as needed without changing the nature of the invention.

[0169] The electronic device 1100 also includes a power supply 1170 (such as a battery) to power various components. Preferably, the power supply can be logically connected to the processor 1120 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 1170 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0170] It should be noted that, although not shown, the electronic device 1100 may also include a Bluetooth module, etc., which will not be described in detail here.

[0171] An embodiment of the present invention also provides a computer-readable storage medium, which can be disposed in an electronic device to store at least one instruction or at least one program related to implementing an image registration method or a training method for a deep graph neural network model. The at least one instruction or the at least one program is loaded and executed by the processor to implement the image registration method or the training method for a deep graph neural network model provided in the above method embodiment.

[0172] Optionally, in embodiments of the present invention, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0173] One embodiment of the present invention also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image registration method or the deep graph neural network model training method provided in the various optional implementations described above.

[0174] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0175] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0176] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0177] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An image registration method, characterized in that, include: Acquire two-dimensional and three-dimensional images of the vessel centerline, including the vessel segment of interest. Feature points are extracted from the two-dimensional blood vessel centerline image to obtain a first feature point image; The three-dimensional blood vessel centerline image is projected onto a two-dimensional plane, and feature points are extracted from the obtained two-dimensional projected image to obtain a second feature point image. The first feature point image and the second feature point image are input into a pre-trained deep graph neural network model for feature point matching to obtain the feature point matching result. The deep graph neural network model is trained using a training sample dataset that includes multiple image sample pairs. Each image sample pair includes a first feature point image sample, a second feature point image sample, and the feature point matching relationship between the two. The first feature point image sample is the feature point image corresponding to the synthetic blood vessel centerline image sample, and the second feature point image sample is the feature point image corresponding to the three-dimensional blood vessel centerline image sample. The synthetic blood vessel centerline image sample is obtained based on the three-dimensional blood vessel centerline image sample using a data deformation method that embeds anatomical prior knowledge. The deep graph neural network model is a deep graph neural network model based on self-attention and cross-attention; The training sample dataset is determined by synthetic blood vessel centerline image samples and three-dimensional blood vessel centerline image samples. The synthetic blood vessel centerline image samples are obtained by randomly deforming three-dimensional blood vessel centerline image samples modeled as three-dimensional hinge structures and then performing simulated C-arm projection.

2. The method according to claim 1, characterized in that, The acquisition of two-dimensional and three-dimensional vessel centerline images, including the vessel segment of interest, includes: Acquire coronary angiography images and coronary CT angiography images of the vessel segment of interest; The coronary angiography image is segmented and the centerline is extracted to obtain the two-dimensional vessel centerline image; The coronary CT angiography image is segmented and the centerline is extracted to obtain the three-dimensional vascular centerline image.

3. The method according to claim 1, characterized in that, The step of extracting feature points from the two-dimensional blood vessel centerline image to obtain a first feature point image includes: The two-dimensional blood vessel centerline image is input into a pre-trained feature point extraction network model to extract feature points, thereby obtaining the first feature point image. The step of projecting the three-dimensional blood vessel centerline image onto a two-dimensional plane and extracting feature points from the resulting two-dimensional projected image to obtain a second feature point image includes: The three-dimensional blood vessel centerline image is projected onto a two-dimensional plane to obtain the two-dimensional projected image; The two-dimensional projection image is input into the feature point extraction network model to extract feature points, thereby obtaining the second feature point image.

4. The method according to claim 1, characterized in that, The method further includes: Based on the feature point matching results, the projection parameters of the three-dimensional blood vessel centerline image projected onto the two-dimensional plane are corrected to obtain a rigid registration result between the two-dimensional blood vessel centerline image and the three-dimensional blood vessel centerline image.

5. The method according to claim 4, characterized in that, The step of projecting the three-dimensional vascular centerline image onto a two-dimensional plane includes: The three-dimensional blood vessel centerline image is projected onto a two-dimensional plane using the corrected projection parameters to obtain the two-dimensional projected image.

6. The method according to claim 5, characterized in that, The method further includes: Based on the feature point matching results, non-rigid registration is performed on the two-dimensional blood vessel centerline image and the three-dimensional blood vessel centerline image to obtain the corresponding non-rigid registration results.

7. The method according to claim 1, characterized in that, The method further includes pre-establishing the deep graph neural network model, the process of establishing the deep graph neural network model including: A training sample dataset is obtained, which includes multiple image sample pairs. Each image sample pair includes a first feature point image sample, a second feature point image sample, and the feature point matching relationship between the two. The first feature point image sample is the feature point image corresponding to the synthetic blood vessel centerline image sample, and the second feature point image sample is the feature point image corresponding to the three-dimensional blood vessel centerline image sample. The synthetic blood vessel centerline image sample is obtained based on the three-dimensional blood vessel centerline image sample and uses a data deformation method embedded with anatomical prior knowledge. A deep graph neural network model is pre-constructed, wherein the deep graph neural network model is a deep graph neural network model based on self-attention and cross-attention; Using the first feature point image sample and the second feature point image sample as input data, and the corresponding feature point matching relationship as supervision, the deep graph neural network model is trained to obtain a trained deep graph neural network model.

8. An image registration device, characterized in that, include: The image acquisition module is used to acquire two-dimensional and three-dimensional images of the vessel centerline, including the vessel segment of interest. The first feature point extraction module is used to extract feature points from the two-dimensional blood vessel centerline image to obtain a first feature point image. The second feature point extraction module is used to project the three-dimensional blood vessel centerline image onto a two-dimensional plane, and extract feature points from the obtained two-dimensional projection image to obtain a second feature point image. The feature point matching module is used to input the first feature point image and the second feature point image into a pre-trained deep graph neural network model to perform feature point matching and obtain the feature point matching result. The deep graph neural network model is trained using a training sample dataset that includes multiple image sample pairs. Each image sample pair includes a first feature point image sample, a second feature point image sample, and the feature point matching relationship between the two. The first feature point image sample is the feature point image corresponding to the synthetic blood vessel centerline image sample, and the second feature point image sample is the feature point image corresponding to the three-dimensional blood vessel centerline image sample. The synthetic blood vessel centerline image sample is obtained based on the three-dimensional blood vessel centerline image sample using a data deformation method that embeds anatomical prior knowledge. The deep graph neural network model is a deep graph neural network model based on self-attention and cross-attention; The training sample dataset is determined by synthetic blood vessel centerline image samples and three-dimensional blood vessel centerline image samples. The synthetic blood vessel centerline image samples are obtained by randomly deforming three-dimensional blood vessel centerline image samples modeled as three-dimensional hinge structures and then performing simulated C-arm projection.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the image registration method as described in any one of claims 1-7.

Images