Image registration method and apparatus, and electronic device

Abstract

The present invention relates to the technical field of image processing, and in particular to an image registration method and apparatus, and an electronic device. The method comprises: acquiring a two-dimensional blood vessel centerline image and a three-dimensional blood vessel centerline image comprising a blood vessel segment of interest; performing feature point extraction on the two-dimensional blood vessel centerline image to obtain a first feature point image; projecting the three-dimensional blood vessel centerline image to a two-dimensional plane, and performing feature point extraction on the obtained two-dimensional projection image to obtain a second feature point image; and inputting the first feature point image and the second feature point image into a pre-trained depth map neural network model for feature point matching to obtain a feature point matching result. The method of the present invention can improve the matching accuracy, the matching speed and the matching density of blood vessel feature points, thereby improving the speed and accuracy of projection parameter optimization and non-rigid registration.

Description

Image registration method, device and electronic equipment

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on February 6, 2024, with application number 202410169234.2 and application name “A method, device and electronic device for image registration”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present invention relates to the technical field of image processing, and in particular to an image registration method, device and electronic equipment. Background Art

[0003] Currently, cardiovascular disease has become the leading cause of death in both urban and rural Chinese residents, and the burden of cardiovascular disease is increasing. In recent years, with the continuous advancement of surgical techniques and instruments, minimally invasive percutaneous coronary intervention (PCI) has become the most technically mature and widely used treatment for cardiovascular disease. As an imaging guide for PCI procedures, coronary angiography (CAG) is currently recognized as the gold standard for diagnosing coronary heart disease. CAG images can dynamically display the entire coronary artery tree in real time, assisting physicians in performing interventional procedures.

[0004] Because CAG images are essentially two-dimensional images produced by projecting three-dimensional information, they have certain limitations, including: 1. CAG images lack spatial information about blood vessels, which can be shortened and stacked. 2. CAG images lack information about vascular plaques. While they can accurately display vascular stenosis, they cannot indicate the type of plaque causing the stenosis. 3. CAG images cannot display distal information about chronic total occlusion (CTO) diseased vessels. Therefore, CAG images are not a "perfect guide" for PCI surgery.

[0005] Preoperative coronary computed tomography angiography (CTA) is a three-dimensional imaging method that can indicate the nature of plaques through the size of the CT value. Therefore, the spatial structure of blood vessels and plaques can be presented to doctors more intuitively through three-dimensional rendering and color coding, helping doctors make better surgical decisions.

[0006] Therefore, the multimodal image registration and fusion of preoperative CTA and CAG images can complement the advantages of the two modalities, optimizing the PCI surgical process from preoperative planning to surgical navigation and decision-making, thereby improving the efficiency and success rate of PCI surgery. Therefore, developing efficient and accurate multimodal image registration and fusion methods for CTA and CAG images can benefit patients, physicians, and other parties, and has great clinical value and significance.

[0007] However, existing registration methods typically rely on point features and employ iterative optimization algorithms for matching and registration. These methods fail to leverage the rich topological structure of the coronary artery tree. Furthermore, these algorithms take a long time to run, failing to meet the real-time requirements of surgery. Alternatively, registration based on image grayscale features suffers from poor accuracy and ineffectiveness for complex vessels. Alternatively, traditional algorithms for topological matching are slow, resulting in poor matching performance for complex vessels. Summary of the Invention

[0008] In response to the above-mentioned problems in the prior art, the purpose of the present invention is to provide an image registration method, device and electronic equipment that can improve the matching accuracy, matching speed and matching density of vascular feature points, thereby improving the speed and accuracy of projection parameter optimization and non-rigid registration.

[0009] In order to solve the above problems, the present invention provides an image registration method, comprising:

[0010] acquiring a two-dimensional vascular centerline image and a three-dimensional vascular centerline image including a vascular segment of interest;

[0011] Extracting feature points from the two-dimensional blood vessel centerline image to obtain a first feature point image;

[0012] Projecting the three-dimensional blood vessel centerline image onto a two-dimensional plane, and extracting feature points from the obtained two-dimensional projection image to obtain a second feature point image;

[0013] Inputting the first feature point image and the second feature point image into a pre-trained deep graph neural network model for feature point matching to obtain a feature point matching result;

[0014] In which, the deep graph neural network model is trained using a training sample data set including multiple image sample pairs, each of the image sample pairs includes a first feature point image sample, a second feature point image sample and a feature point matching relationship between the two, the first feature point image sample is a feature point image corresponding to a synthetic blood vessel centerline image sample, the second feature point image sample is a feature point image corresponding to a three-dimensional blood vessel centerline image sample, and the synthetic blood vessel centerline image sample is obtained based on the three-dimensional blood vessel centerline image sample using a data deformation method embedded with anatomical prior knowledge.

[0015] Furthermore, acquiring a two-dimensional blood vessel centerline image and a three-dimensional blood vessel centerline image including a blood vessel segment of interest includes:

[0016] Acquire coronary angiography images and coronary CT angiography images including the vascular segment of interest;

[0017] performing blood vessel segmentation and centerline extraction on the coronary angiography image to obtain the two-dimensional blood vessel centerline image;

[0018] The coronary artery CT angiography image is subjected to vessel segmentation and centerline extraction to obtain the three-dimensional vessel centerline image.

[0019] Furthermore, extracting feature points from the two-dimensional blood vessel centerline image to obtain a first feature point image includes:

[0020] Inputting the two-dimensional blood vessel centerline image into a pre-trained feature point extraction network model to extract feature points to obtain the first feature point image;

[0021] The step of projecting the three-dimensional blood vessel centerline image onto a two-dimensional plane and extracting feature points from the obtained two-dimensional projection image to obtain a second feature point image includes:

[0022] Projecting the three-dimensional blood vessel centerline image onto a two-dimensional plane to obtain the two-dimensional projection image;

[0023] The two-dimensional projection image is input into the feature point extraction network model to extract feature points to obtain the second feature point image.

[0024] Furthermore, the method further comprises:

[0025] Based on the feature point matching result, projection parameters of the three-dimensional blood vessel centerline image projected onto a two-dimensional plane are corrected to obtain a rigid registration result between the two-dimensional blood vessel centerline image and the three-dimensional blood vessel centerline image.

[0026] Furthermore, projecting the three-dimensional blood vessel centerline image onto a two-dimensional plane includes:

[0027] The three-dimensional blood vessel centerline image is projected onto a two-dimensional plane using the corrected projection parameters to obtain the two-dimensional projection image.

[0028] Furthermore, the method further comprises:

[0029] Based on the feature point matching result, non-rigid registration is performed on the two-dimensional blood vessel centerline image and the three-dimensional blood vessel centerline image to obtain a corresponding non-rigid registration result.

[0030] Furthermore, the method further includes pre-establishing the deep graph neural network model, and the process of establishing the deep graph neural network model includes:

[0031] Acquire a training sample dataset, the training sample dataset comprising a plurality of image sample pairs, each of the image sample pairs comprising a first feature point image sample, a second feature point image sample, and a feature point matching relationship between the first feature point image sample and the second feature point image sample, wherein the first feature point image sample is a feature point image corresponding to a synthetic blood vessel centerline image sample, and the second feature point image sample is a feature point image corresponding to a three-dimensional blood vessel centerline image sample, wherein the synthetic blood vessel centerline image sample is obtained based on the three-dimensional blood vessel centerline image sample using a data deformation method embedded with anatomical prior knowledge;

[0032] Pre-building a deep graph neural network model, wherein the deep graph neural network model is a deep graph neural network model based on self-attention and cross-attention;

[0033] The first feature point image sample and the second feature point image sample are used as input data, and the corresponding feature point matching relationship is used as supervision to train the deep graph neural network model to obtain a trained deep graph neural network model.

[0034] Furthermore, the obtaining of the training sample data set includes:

[0035] Acquire multiple three-dimensional blood vessel centerline image samples;

[0036] Modeling each of the three-dimensional blood vessel centerline image samples as a three-dimensional hinge structure, and randomly deforming the obtained three-dimensional blood vessel centerline image samples of the three-dimensional hinge structure;

[0037] Performing simulated C-arm projection on the deformed three-dimensional blood vessel centerline image sample to obtain a corresponding synthetic blood vessel centerline image sample;

[0038] According to each of the three-dimensional blood vessel centerline image samples and its corresponding synthesized blood vessel centerline image sample, the corresponding first feature point image sample, the second feature point image sample and the feature point matching relationship between the two are determined.

[0039] Another aspect of the present invention provides an image registration device, comprising:

[0040] An image acquisition module, configured to acquire a two-dimensional vascular centerline image and a three-dimensional vascular centerline image including a vascular segment of interest;

[0041] a first feature point extraction module, configured to extract feature points from the two-dimensional blood vessel centerline image to obtain a first feature point image;

[0042] a second feature point extraction module, configured to project the three-dimensional blood vessel centerline image onto a two-dimensional plane, and extract feature points from the obtained two-dimensional projection image to obtain a second feature point image;

[0043] A feature point matching module, configured to input the first feature point image and the second feature point image into a pre-trained deep graph neural network model for feature point matching to obtain a feature point matching result;

[0044] In which, the deep graph neural network model is trained using a training sample data set including multiple image sample pairs, each of the image sample pairs includes a first feature point image sample, a second feature point image sample and a feature point matching relationship between the two, the first feature point image sample is a feature point image corresponding to a synthetic blood vessel centerline image sample, the second feature point image sample is a feature point image corresponding to a three-dimensional blood vessel centerline image sample, and the synthetic blood vessel centerline image sample is obtained based on the three-dimensional blood vessel centerline image sample using a data deformation method embedded with anatomical prior knowledge.

[0045] Another aspect of the present invention provides an electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the image registration method as described above.

[0046] Another aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by a processor to implement the image registration method as described above.

[0047] Due to the above technical solution, the present invention has the following beneficial effects:

[0048] According to the image registration method of an embodiment of the present invention, a data deformation method using anatomical prior knowledge embedding is used to generate image sample pairs including a first feature point image sample, a second feature point image sample, and a feature point matching relationship between the two, so as to train a deep graph neural network model, and the deep graph neural network model is used to match vascular feature points, thereby utilizing the topological features of the blood vessels, and having fast matching speed, good matching effect, and strong model generalization performance. Compared with traditional matching algorithms, this method can greatly improve the matching accuracy, matching speed, and matching density of vascular feature points.

[0049] Moreover, since the deep graph neural network model can be used to obtain denser feature matching points, the feature matching points obtained by the deep graph neural network model are used for subsequent projection parameter optimization and non-rigid registration steps, which can greatly improve the speed and accuracy of projection parameter optimization and non-rigid registration. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] To more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present invention, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0051] FIG1 is a schematic diagram of an implementation environment provided by an embodiment of the present invention;

[0052] FIG2 is a flow chart of an image registration method provided by one embodiment of the present invention;

[0053] FIG3 is a schematic diagram of a feature point extraction process provided by an embodiment of the present invention;

[0054] FIG4 is a schematic diagram of a feature point matching result provided by an embodiment of the present invention;

[0055] FIG5 is a flow chart of an image registration method provided by another embodiment of the present invention;

[0056] FIG6 is a flow chart of an image registration method provided by another embodiment of the present invention;

[0057] FIG7 is a flowchart of a method for training a deep graph neural network model according to an embodiment of the present invention;

[0058] FIG8 is a schematic diagram of a process for generating image sample pairs according to an embodiment of the present invention;

[0059] FIG9 is a schematic diagram of a self-attention mechanism and a cross-attention mechanism provided by one embodiment of the present invention;

[0060] FIG10 is a schematic structural diagram of an image registration device provided by one embodiment of the present invention;

[0061] FIG11 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0062] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0063] It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product, or device comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or devices.

[0064] In order to make the purpose, technical solutions and advantages disclosed in the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings and embodiments. First, the embodiments of the present invention explain the following concepts:

[0065] The PnP (Perspective-n-Points) algorithm determines the pose of the camera coordinate system relative to the world coordinate system. It describes how to estimate the camera pose (i.e., determining the rotation matrix R and translation vector t from the world coordinate system to the camera coordinate system) when the coordinates of n three-dimensional (3D) points (relative to the world coordinate system) and the pixel coordinates of these points are known.

[0066] Deep Graph Matching (DGM) algorithm: The DGM algorithm is a method for matching image feature points based on graph neural network (GNN).

[0067] Percutaneous Coronary Intervention (PCI) surgery: PCI surgery is a medical procedure and the most common angioplasty, which can be used to treat myocardial infarction, heart disease, coronary heart disease and other diseases.

[0068] Chronic Total Occlusion (CTO): CTO is defined as a lesion with a duration of more than three months due to stenosis of the coronary artery lumen resulting in complete blockage of forward blood flow. It accounts for approximately 15% to 20% of coronary artery lesions and is known as the "last barrier in the field of coronary artery interventional surgery."

[0069] Digital Imaging and Communications in Medicine (DCM / DICOM) tag: The DCM (DICOM) tag is an international standard for medical images and related information. It defines a medical image format that can be used for data exchange and that meets clinical needs.

[0070] Referring to FIG1 of the present specification, there is shown a schematic diagram of an implementation environment provided by one embodiment of the present invention. As shown in FIG1 , the implementation environment may include at least one medical scanning device 110 and a computer device 120. The computer device 120 and each medical scanning device 110 may be directly or indirectly connected via wired or wireless communication, although this embodiment of the present invention is not limited thereto.

[0071] Among them, the medical scanning device 110 can be but is not limited to a CAG device and a CTA device, etc., and the computer device 120 can be but is not limited to various servers, personal computers, laptops, smart phones, tablets and portable wearable devices. The server can be an independent server or a server cluster or distributed system composed of multiple servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs) and big data and artificial intelligence platforms.

[0072] In an embodiment of the present invention, the medical scanning device 110 can capture images of a vascular segment of interest to obtain corresponding two-dimensional and three-dimensional vascular images. The computer device 120 can acquire the two-dimensional and three-dimensional vascular images scanned by the medical scanning device 110 and register them using the image registration method provided in an embodiment of the present invention. This improves the registration accuracy and speed of the two-dimensional and three-dimensional vascular images, achieving complementary advantages between the two modalities. This can help physicians make better surgical decisions during the treatment of coronary artery lesions, particularly CTO lesions, and improve the efficiency and success rate of PCI procedures.

[0073] For example, the medical scanning device 110 can capture images of a vascular segment of interest based on CAG and CTA technologies, thereby obtaining a CAG image and a CTA image containing the vascular segment of interest. The computer device 120 can acquire the CAG image and the CTA image scanned by the medical scanning device 110 and register them using the image registration method provided in an embodiment of the present invention. This can optimize the PCI surgical process, from preoperative planning to surgical navigation and surgical decision-making, thereby improving the efficiency and success rate of PCI surgery.

[0074] It should be noted that FIG1 is merely an example. Those skilled in the art will appreciate that although FIG1 shows only one medical scanning device 110 , this does not limit the present invention, and the present invention may include more or fewer medical scanning devices 110 than shown.

[0075] Referring to FIG2 of the specification, which shows a process of an image registration method provided by an embodiment of the present invention, the method can be applied to the computer device 120 in FIG1 . Specifically, as shown in FIG2 , the method may include the following steps:

[0076] S210: Acquire a two-dimensional blood vessel centerline image and a three-dimensional blood vessel centerline image including a blood vessel segment of interest.

[0077] In the embodiments of the present invention, the vessel segment of interest may be a segment of a vessel that exhibits an abnormality relative to a normal vessel, or may be a vessel tree that includes the segment of a vessel that exhibits an abnormality relative to a normal vessel. For example, the vessel segment of interest may be a coronary artery segment / vessel tree of various types, or may be a cerebral vessel segment / vessel tree of various types. The embodiments of the present invention do not impose any specific limitation on the vessel type of the vessel segment of interest.

[0078] In an embodiment of the present invention, the two-dimensional vessel centerline image can be obtained by performing vessel segmentation and centerline extraction on a two-dimensional vessel image including a vessel segment of interest, and the three-dimensional vessel centerline image can be obtained by performing vessel segmentation and centerline extraction on a three-dimensional vessel image including a vessel segment of interest. The grayscale values ​​of the two-dimensional and three-dimensional vessel centerline images can also be fused with vessel radius information. In other words, the radius information of the current vessel location can be determined based on the grayscale value of each pixel in the vessel centerline image.

[0079] In a possible embodiment, acquiring a two-dimensional vascular centerline image and a three-dimensional vascular centerline image including a vascular segment of interest may include: acquiring a coronary angiography (CAG) image and a coronary CT angiography (CTA) image including the vascular segment of interest; performing vascular segmentation and centerline extraction on the coronary angiography image to obtain the two-dimensional vascular centerline image; and performing vascular segmentation and centerline extraction on the coronary CT angiography image to obtain the three-dimensional vascular centerline image.

[0080] Specifically, the CAG / CTA images can be directly acquired or selected from a CAG / CTA image of a blood vessel segment to determine the vascular segment of interest, and this embodiment of the present invention does not limit this. The image data can be directly imported from a related file or acquired from other resource libraries through a real-time configuration connection, and this embodiment of the present invention does not limit this either.

[0081] In practical applications, images of the vascular segment of interest can be acquired based on CAG technology and CTA technology, respectively, to obtain CAG images and CTA images including the vascular segment of interest. The detailed acquisition process of the CAG image and the CTA image is not further described in the embodiment of the present invention.

[0082] Specifically, existing vascular segmentation methods can be used to perform vascular segmentation on the CAG image and the CTA image, respectively, to obtain corresponding two-dimensional and three-dimensional vascular segmentation images. Existing centerline extraction methods can then be used to extract vascular centerlines based on the two-dimensional and three-dimensional vascular segmentation images, respectively, to obtain a two-dimensional vascular centerline image corresponding to the CAG image and a three-dimensional vascular centerline image corresponding to the CTA image. The detailed processes of vascular segmentation and vascular centerline extraction are not further described in this embodiment of the present invention.

[0083] Specifically, during vessel centerline extraction, the vessel radius information at each location along the vessel centerline can also be determined and integrated into the grayscale value of the vessel centerline image. This allows subsequent feature point extraction and matching to capture the radius information of vessels at different locations. For example, the vessel radius values ​​at each location along the vessel centerline can be normalized and used as the grayscale value of the corresponding pixel in the vessel centerline image.

[0084] S220: Extract feature points from the two-dimensional blood vessel centerline image to obtain a first feature point image.

[0085] In embodiments of the present invention, existing feature point extraction methods can be used to directly extract feature points from the two-dimensional blood vessel centerline image to obtain a first feature point image corresponding to the two-dimensional blood vessel centerline image. The first feature point image can include multiple feature points on the two-dimensional blood vessel centerline and their corresponding feature descriptors. The feature descriptors can be used to describe feature information of the corresponding feature points.

[0086] In a possible embodiment, extracting feature points from the two-dimensional blood vessel centerline image to obtain a first feature point image may include: inputting the two-dimensional blood vessel centerline image into a pre-trained feature point extraction network model to perform feature point extraction to obtain the first feature point image.

[0087] Specifically, a feature point extraction network model can be trained using a deep learning feature extraction algorithm using multiple two-dimensional vascular centerline image samples as input and a feature point set as output. The trained feature point extraction network model can then be deployed on a computer. After acquiring the two-dimensional vascular centerline image, the trained feature point extraction network model can be directly used to process the two-dimensional vascular centerline image to obtain the first feature point image.

[0088] In an embodiment of the present invention, since the radius information of the blood vessel is fused into the grayscale value of the blood vessel centerline image when extracting the blood vessel centerline, the feature point extraction network model can capture the radius information of the blood vessels at different positions. For example, the radius information of the blood vessel can be integrated into the feature descriptor.

[0089] It can be understood that by integrating the radius information of the blood vessels into the grayscale value of the blood vessel centerline image, the feature point extraction network model can capture the radius information of blood vessels at different positions, thereby extracting richer feature information, which helps to improve the matching speed and matching effect of subsequent feature point matching.

[0090] It should be noted that the embodiment of the present invention does not specifically limit the feature extraction algorithm. In practical applications, those skilled in the art can make a selection according to actual needs. For example, the feature extraction algorithm may include but is not limited to the Scale-Invariant Feature Transform (SIFT) algorithm, the SuperPoint algorithm, and the ORB (Oriented FAST and Rotated BRIEF) algorithm.

[0091] For example, referring to FIG3 of the reference specification, a schematic diagram of a feature point extraction process according to one embodiment of the present invention is shown. As shown in FIG3 , performing vessel segmentation on a CAG image yields a two-dimensional vessel segmentation image as shown in FIG3 (a1). Performing vessel centerline extraction on this image yields a two-dimensional vessel centerline image as shown in FIG3 (b1). Further feature point extraction yields a first feature point image as shown in FIG3 (c1).

[0092] S230: Projecting the three-dimensional blood vessel centerline image onto a two-dimensional plane, and extracting feature points from the obtained two-dimensional projection image to obtain a second feature point image.

[0093] In an embodiment of the present invention, the 3D blood vessel centerline image can be first projected onto a 2D plane to obtain a 2D projection image. Existing feature point extraction methods can then be used to directly extract feature points from the 2D projection image to obtain a second feature point image corresponding to the 3D blood vessel centerline image. The second feature point image can include multiple feature points on the 3D blood vessel centerline and their corresponding feature descriptors. The feature descriptors can be used to describe feature information of the corresponding feature points.

[0094] In a possible embodiment, projecting the three-dimensional blood vessel centerline image onto a two-dimensional plane and performing feature point extraction on the obtained two-dimensional projection image to obtain a second feature point image may include: projecting the three-dimensional blood vessel centerline image onto a two-dimensional plane to obtain the two-dimensional projection image; and inputting the two-dimensional projection image into the feature point extraction network model to perform feature point extraction to obtain the second feature point image.

[0095] Specifically, the three-dimensional vascular centerline image can be projected onto a two-dimensional plane by simulating C-arm projection. Projection parameters related to C-arm projection can be first obtained, and then the three-dimensional vascular centerline image can be projected using these projection parameters. The projection parameters related to C-arm projection can be read from the DCM tag of the CAG image.

[0096] Specifically, after obtaining the two-dimensional projection image, the trained feature point extraction network model can be directly used to process the two-dimensional projection image to obtain the second feature point image.

[0097] In an embodiment of the present invention, since the radius information of the blood vessel is fused into the grayscale value of the blood vessel centerline image when extracting the blood vessel centerline, the feature point extraction network model can capture the radius information of the blood vessels at different positions. For example, the radius information of the blood vessel can be integrated into the feature descriptor.

[0098] It can be understood that by integrating the radius information of the blood vessels into the grayscale value of the blood vessel centerline image, the feature point extraction network model can capture the radius information of blood vessels at different positions, thereby extracting richer feature information, which helps to improve the matching speed and matching effect of subsequent feature point matching.

[0099] For example, as shown in Figure 3, performing vessel segmentation on a CTA image can produce a 3D vessel segmentation image (a2). Extracting the vessel centerline and simulating C-arm projection onto a 2D plane can produce a simulated 2D projection image of the centerline (b2). Further feature point extraction can produce a second feature point image (c2).

[0100] S240: Input the first feature point image and the second feature point image into a pre-trained deep graph neural network model for feature point matching to obtain a feature point matching result.

[0101] In which, the deep graph neural network model is trained using a training sample data set including multiple image sample pairs, each of the image sample pairs includes a first feature point image sample, a second feature point image sample and a feature point matching relationship between the two, the first feature point image sample is a feature point image corresponding to a synthetic blood vessel centerline image sample, the second feature point image sample is a feature point image corresponding to a three-dimensional blood vessel centerline image sample, and the synthetic blood vessel centerline image sample is obtained based on the three-dimensional blood vessel centerline image sample using a data deformation method embedded with anatomical prior knowledge.

[0102] In an embodiment of the present invention, multiple 3D vascular centerline image samples can be acquired and, based on each 3D vascular centerline image sample, data deformation can be performed to embed anatomical prior knowledge, thereby generating multiple image sample pairs with matching feature points. Based on these multiple image sample pairs, a deep graph neural network model is trained using the DGM algorithm, and the trained deep graph neural network model is deployed on a computer device. The training method for the deep graph neural network model will be described in detail later.

[0103] In an embodiment of the present invention, after obtaining the first feature point image and the second feature point image, the first feature point image and the second feature point image can be directly processed using a trained deep graph neural network model to obtain corresponding feature point matching results.

[0104] Specifically, the deep graph neural network model can be a graph neural network based on the self-attention mechanism and the cross-attention mechanism. The deep graph neural network model can regard the feature points in the first feature point image and the second feature point image as nodes in the graph neural network, and update and generate the node information through multiple rounds of self-attention mechanism and cross-attention mechanism. A matching vector is generated for each feature point to calculate the matching score between the points. Finally, the matching score matrix is ​​calculated using Dual-Softmax to determine the feature point matching results of the first feature point image and the second feature point image.

[0105] Specifically, in the scoring matrix, the problem of topological inconsistency in vascular feature point matching (for example, some points in the CTA image cannot find corresponding points in the CAG image) can be solved by setting redundant rows and redundant columns.

[0106] It can be understood that by constructing a deep graph neural network model based on self-attention and cross-attention, the model is used to calculate self-attention and cross-attention between feature points, capture the topological relationship between feature points, and establish dynamic, sparse and soft corresponding graph structures for feature points within and between images, so as to facilitate the rapid acquisition of accurate and dense feature matching points, and further improve the matching accuracy, matching speed and matching density of the model.

[0107] Specifically, the feature point matching result may include multiple pairs of feature matching points in the first feature point image and the second feature point image and their corresponding matching relationships. For example, by inputting the first feature point image shown in (c1) and the second feature point image shown in (c2) in Figure 3 into the deep graph neural network model, a feature point matching result as shown in Figure 4 can be obtained.

[0108] In summary, according to the image registration method of an embodiment of the present invention, a data deformation method for embedding anatomical prior knowledge is used to generate an image sample pair including a first feature point image sample, a second feature point image sample, and a feature point matching relationship between the two, so as to train a deep graph neural network model, and use the deep graph neural network model to match vascular feature points, thereby utilizing the topological features of the blood vessels, and having a fast matching speed, good matching effect, and strong model generalization performance. Compared with traditional matching algorithms, this method can greatly improve the matching accuracy, matching speed, and matching density of vascular feature points.

[0109] In a possible embodiment, with reference to FIG5 of the specification, the method may further include the following steps:

[0110] S250: Based on the feature point matching result, correct the projection parameters of the three-dimensional blood vessel centerline image projected onto the two-dimensional plane to obtain a rigid registration result between the two-dimensional blood vessel centerline image and the three-dimensional blood vessel centerline image.

[0111] In embodiments of the present invention, after obtaining the feature point matching results, a projection parameter correction step can be performed based on the extracted feature matching points to correct the projection parameters of the three-dimensional vascular centerline image projected onto the two-dimensional plane, thereby obtaining a rigid registration result. For example, a PnP algorithm can be used to correct the projection parameters based on the extracted feature matching points to obtain a rigid registration result. The specific process of performing projection parameter correction using the PnP algorithm will not be further described in this embodiment of the present invention.

[0112] In one possible embodiment, after the projection parameters are corrected, steps S230 to S250 may be repeated. Furthermore, in step S230, the three-dimensional vascular centerline image is re-projected onto a two-dimensional plane using the corrected projection parameters to obtain a new two-dimensional projection image, thereby obtaining a new second feature point image. In step S240, feature point matching is performed on the first feature point image and the new second feature point image using a deep graph neural network model to obtain a new feature point matching result. In step S250, the corrected projection parameters are corrected again based on the new feature point matching result to obtain a new rigid registration result.

[0113] It should be noted that in actual applications, the above process can be repeated multiple times (for example, 2 to 3 times) to obtain a more accurate rigid registration result. Since the algorithm runs quickly, multiple repetitions do not affect the efficiency of image registration.

[0114] It can be understood that by using the PnP algorithm to optimize the projection parameters based on the feature point matching results output by the model, the optimization speed and optimization effect of this algorithm are better than the traditional iterative-based projection parameter optimization algorithm, so it can further improve the speed and accuracy of projection parameter optimization.

[0115] In a possible embodiment, with reference to FIG6 of the specification, the method may further include the following steps:

[0116] S260: Based on the feature point matching result, perform non-rigid registration on the two-dimensional blood vessel centerline image and the three-dimensional blood vessel centerline image to obtain a corresponding non-rigid registration result.

[0117] In this embodiment of the present invention, the corrected projection parameters obtained in step S250 (i.e., the rigid registration result) can be used as the initial registration result, and steps S230 and S240 can be repeated again to obtain a more accurate feature point matching result. The obtained more accurate feature point matching result can then be used to perform further non-rigid registration to obtain a non-rigid registration result as the final registration result.

[0118] In one possible embodiment, after obtaining the non-rigid registration result, steps S230, S240, and S260 may be repeated to obtain a more accurate rigid registration result. It should be noted that in actual applications, the above process may be repeated multiple times (e.g., 2 to 3 times) to improve the accuracy of the registration result. Because the algorithm runs quickly, multiple repetitions do not affect the efficiency of image registration.

[0119] It can be understood that by using feature matching points constructed based on topological relationships in global non-rigid registration, since the deep graph neural network model can obtain quite dense and accurate feature matching points, these feature matching points can be used to greatly reduce the number of iterations in the registration process and improve the registration effect. The non-rigid registration process can be completed in a very short time, greatly improving the speed and accuracy of non-rigid registration.

[0120] Referring to FIG7 of the specification, which shows the process of a training method for a deep graph neural network model provided by one embodiment of the present invention. Specifically, as shown in FIG7, the method may include the following steps:

[0121] S710: Obtain a training sample data set, wherein the training sample data set includes multiple image sample pairs, each of the image sample pairs includes a first feature point image sample, a second feature point image sample, and a feature point matching relationship between the first feature point image sample and the second feature point image sample, wherein the first feature point image sample is a feature point image corresponding to a synthetic blood vessel centerline image sample, and the second feature point image sample is a feature point image corresponding to a three-dimensional blood vessel centerline image sample, and the synthetic blood vessel centerline image sample is obtained based on the three-dimensional blood vessel centerline image sample using a data deformation method embedded with anatomical prior knowledge.

[0122] In an embodiment of the present invention, since a large amount of training sample data is required in the model training stage (especially for the case of using a Transformer structure), and the point matching relationship of medical images is difficult to obtain due to lack of data and the existence of large-scale non-rigid deformation, the training sample data can be generated based on data deformation embedded with anatomical prior knowledge. Specifically, a plurality of three-dimensional vascular centerline image samples including a vascular segment of interest can be obtained in advance, and a data deformation method embedded with anatomical prior knowledge can be used to generate the image sample pairs with feature point matching relationships corresponding to each of the three-dimensional vascular centerline image samples, thereby obtaining the training sample data set including the plurality of image sample pairs. Specifically, the image sample pairs can include a first feature point image sample corresponding to a synthetic vascular centerline image sample obtained by using the data deformation method embedded with anatomical prior knowledge, a second feature point image sample corresponding to the three-dimensional vascular centerline image sample, and a feature point matching relationship between the two.

[0123] In a possible embodiment, obtaining a training sample data set may include: obtaining multiple three-dimensional vascular centerline image samples; modeling each of the three-dimensional vascular centerline image samples as a three-dimensional hinge structure, and randomly deforming the obtained three-dimensional vascular centerline image samples of the three-dimensional hinge structure; performing simulated C-arm projection on the deformed three-dimensional vascular centerline image samples to obtain corresponding synthetic vascular centerline image samples; and determining, based on each of the three-dimensional vascular centerline image samples and its corresponding synthetic vascular centerline image sample, a corresponding first feature point image sample, a second feature point image sample, and a feature point matching relationship between the two.

[0124] Specifically, the 3D vascular centerline image samples can be obtained by performing vessel segmentation and centerline extraction on 3D vascular image samples including vascular segments of interest. For example, CTA image samples including vascular segments of interest can be acquired from multiple different subjects using CTA technology. Vessel segmentation can then be performed on each of the CTA image samples to obtain corresponding 3D vascular segmentation image samples. Vessel centerline extraction can then be performed on the 3D vascular segmentation image samples to obtain 3D vascular centerline image samples corresponding to each of the CTA image samples.

[0125] It should be noted that the above-described embodiment of using 3D vascular centerline image samples including a vessel segment of interest to generate a training sample dataset is merely an example. In practical applications, this is not limited to image samples including only vessel segments of interest. In other words, image samples including vessel segments other than the vessel segment of interest can also be used to train the deep graph neural network model, and this embodiment of the present invention imposes no specific limitation on this.

[0126] In practical applications, a corresponding deep graph neural network model can be trained for each type of vascular segment, or a deep graph neural network model applicable to all types of vascular segments can be trained. This embodiment of the present invention does not impose any specific restrictions on this.

[0127] Specifically, during data deformation to embed anatomical prior knowledge, the 3D vascular centerlines in each of the 3D vascular centerline image samples can be first modeled as 3D hinge structures. An angular offset field is then generated for each point in the 3D hinge structure in a spherical coordinate system. This allows for realistic non-rigid deformation of the vessels, thereby simulating conditions such as heartbeat and breathing. Subsequently, by simulating the C-arm projection process, the body position angle and randomly generated body position angle deviations can be used to simulate projection parameter errors between CTA and angiography due to patient position movement and equipment. Optionally, topological inconsistencies between CTA and angiography can be simulated by randomly discarding vessels or randomly removing or adding noise points. Through the above method, synthetic vascular centerline image samples can ultimately be obtained, as well as multiple pairs of feature matching points between the synthetic vascular centerline image samples and the 3D vascular centerline image samples, and their corresponding feature point matching relationships.

[0128] Specifically, the synthetic blood vessel centerline image sample and the three-dimensional blood vessel centerline image sample are projected onto a two-dimensional plane respectively, and the generated multiple pairs of feature matching points and their corresponding feature point matching relationships are combined to obtain the first feature point image sample corresponding to the synthetic blood vessel centerline image sample, the second feature point image sample corresponding to the three-dimensional blood vessel centerline image sample, and the feature point matching relationship between the two, which are used to train the deep graph neural network model.

[0129] For example, in conjunction with Figure 8 of the reference specification, a schematic diagram of the process of generating an image sample pair provided by an embodiment of the present invention is shown. As shown in Figure 8, based on the three-dimensional blood vessel centerline image sample shown in Figure (1) a, the synthetic blood vessel centerline image sample shown in Figure (1) b can be deformed. Projecting the three-dimensional blood vessel centerline image sample onto a two-dimensional plane can obtain the centerline image of the two-dimensional plane shown in Figure (2) a; projecting the synthetic blood vessel centerline image sample onto a two-dimensional plane can obtain the centerline image of the two-dimensional plane shown in Figure (2) b. Combining the generated multiple pairs of feature matching points and their corresponding feature point matching relationships, the second feature point image sample corresponding to the three-dimensional blood vessel centerline image sample shown in Figure (3) a, the first feature point image sample corresponding to the synthetic blood vessel centerline image sample shown in Figure (3) b, and the feature point matching relationship between the two can be obtained.

[0130] It can be understood that by using the data deformation method embedded with anatomical prior knowledge to generate training sample data, it is possible to solve the problem of difficulty in obtaining point matching relationships in medical images due to lack of data and the existence of large-scale non-rigid deformations, thereby improving the matching effect and generalization ability of the trained deep graph neural network model.

[0131] S720: Pre-construct a deep graph neural network model, where the deep graph neural network model is a deep graph neural network model based on self-attention and cross-attention.

[0132] In an embodiment of the present invention, a deep graph neural network model can be constructed in advance based on the DGM algorithm. The construction process of the deep graph neural network model can refer to the existing technology, and the embodiment of the present invention will not be repeated here.

[0133] Specifically, the deep graph neural network model can be a graph neural network based on self-attention and cross-attention mechanisms. The deep graph neural network model can use self-attention and cross-attention to construct dynamic, sparse, and soft-corresponding graph structures for feature points within and between images, respectively, to match vascular feature points using the topological characteristics of blood vessels.

[0134] Exemplarily, in conjunction with Figure 9 of the reference specification, a schematic diagram of the self-attention mechanism and the cross-attention mechanism provided by one embodiment of the present invention is shown. As shown in Figure 9, in the shallow layer of the model (for example, the first layer), self-attention can capture vascular information in a global range within the image, and cross-attention can alternately pay attention to the global range between images to capture vascular information. In the deep layer of the model (for example, the twelfth layer), self-attention only captures vascular information in a local range within the image, and cross-attention only alternates attention to the local range between images to capture vascular information. The local range can be the range where the feature point that roughly matches the current feature point is located. The above Figure 9 can show that the image registration method provided in the embodiment of the present invention has good interpretability and is conducive to clinical use.

[0135] It should be noted that the above-mentioned implementation method of using the graph neural network based on the self-attention mechanism and the cross-attention mechanism as the deep graph neural network model is only an example. The embodiment of the present invention does not impose specific restrictions on the structure of the deep graph neural network model. In actual applications, those skilled in the art can make determinations based on actual needs.

[0136] It can be understood that by constructing a deep graph neural network model based on self-attention and cross-attention, the model is used to calculate self-attention and cross-attention between feature points, capture the topological relationship between feature points, and establish dynamic, sparse and soft corresponding graph structures for feature points within and between images, so as to facilitate the rapid acquisition of accurate and dense feature matching points, and further improve the matching accuracy, matching speed and matching density of the model.

[0137] S730: Using the first feature point image sample and the second feature point image sample as input data and the corresponding feature point matching relationship as supervision, the deep graph neural network model is trained to obtain a trained deep graph neural network model.

[0138] In an embodiment of the present invention, each image sample pair in the training sample data set can be used as input data, and the corresponding feature point matching relationship can be used as supervision to perform supervised training on a pre-constructed deep graph neural network model to obtain a trained deep graph neural network model.

[0139] Specifically, the deep graph neural network model can regard the feature matching points in the first feature point image sample and the second feature point image sample as nodes in the graph neural network, and update and generate the node information through multiple rounds of self-attention mechanism and cross-attention mechanism. Each feature matching point will generate a matching vector for calculating the matching score between the points. Finally, the matching score matrix is ​​calculated using Dual-Softmax to determine the predicted matching results of the first feature point image sample and the second feature point image sample. The predicted feature point matching results may include multiple pairs of predicted matching points in the first feature point image sample and the second feature point image sample and their corresponding matching relationships.

[0140] Specifically, in order to solve the problem of matching difficulties caused by non-rigid deformation in the medical image matching process, during the training process of the deep graph neural network model, the focal loss function (Focal Loss) and the distance loss function (Dist Loss) can be used to supervise the learning of the predicted matching results. Among them, Focal Loss can make the network pay more attention to difficult-to-match points by assigning adaptive weights to easy-to-match points and difficult-to-match points. Dist Loss is obtained by calculating the Euclidean distance between the feature matching point and the predicted matching point at a certain position.

[0141] It can be understood that the Dist Loss loss function will give smoother and continuous penalties for different degrees of prediction errors when the feature matching points are relatively dense, thereby making up for the shortcomings of Focal Loss and improving the performance of the trained deep graph neural network model in non-rigid medical scenarios.

[0142] It should be noted that the specific content of the model training process can refer to the existing technology, and the embodiments of the present invention will not be repeated here. Other relevant contents in the embodiments of the present invention can refer to the specific content of the method provided in the embodiments shown in Figures 2 to 6, and the embodiments of the present invention will not be repeated here.

[0143] In summary, by leveraging data deformation methods embedded with anatomical prior knowledge to generate image sample pairs consisting of first feature point image samples, second feature point image samples, and the feature point matching relationship between the two, we can train a deep graph neural network model based on vascular topology features. This model can be trained to achieve fast matching speed, good matching results, and strong model generalization performance. Using the trained model for feature point matching can significantly improve the matching accuracy, matching speed, and matching density of vascular feature points compared to traditional matching algorithms.

[0144] Moreover, since the deep graph neural network model can be used to obtain denser feature matching points, the feature matching points obtained by the deep graph neural network model are used for subsequent projection parameter optimization and non-rigid registration steps, which can greatly improve the speed and accuracy of projection parameter optimization and non-rigid registration.

[0145] Referring to FIG10 of the specification, it shows the structure of an image registration device 1000 provided by an embodiment of the present invention. As shown in FIG10 , the device 1000 may include:

[0146] An image acquisition module 1010 is configured to acquire a two-dimensional blood vessel centerline image and a three-dimensional blood vessel centerline image including a blood vessel segment of interest;

[0147] A first feature point extraction module 1020 is configured to extract feature points from the two-dimensional blood vessel centerline image to obtain a first feature point image;

[0148] A second feature point extraction module 1030 is configured to project the three-dimensional blood vessel centerline image onto a two-dimensional plane and extract feature points from the obtained two-dimensional projection image to obtain a second feature point image;

[0149] A feature point matching module 1040 is configured to input the first feature point image and the second feature point image into a pre-trained deep graph neural network model for feature point matching to obtain a feature point matching result;

[0150] In which, the deep graph neural network model is trained using a training sample data set including multiple image sample pairs, each of the image sample pairs includes a two-dimensional vascular centerline image sample, a three-dimensional vascular centerline image sample and a feature point matching relationship between the two, and the two-dimensional vascular centerline image sample is based on the three-dimensional vascular centerline image sample and is obtained by using a data deformation method embedded with anatomical prior knowledge.

[0151] In a possible embodiment, the apparatus 1000 may further include:

[0152] A rigid registration module is used to correct the projection parameters of the three-dimensional blood vessel centerline image projected onto a two-dimensional plane based on the feature point matching result, so as to obtain a rigid registration result between the two-dimensional blood vessel centerline image and the three-dimensional blood vessel centerline image.

[0153] In a possible embodiment, the apparatus 1000 may further include:

[0154] The non-rigid registration module is used to perform non-rigid registration on the two-dimensional blood vessel centerline image and the three-dimensional blood vessel centerline image based on the feature point matching result to obtain a corresponding non-rigid registration result.

[0155] In a possible embodiment, the apparatus 1000 may further include a model training module for pre-establishing the deep graph neural network model;

[0156] The model training module may include:

[0157] a sample data acquisition unit, configured to acquire a training sample data set, the training sample data set comprising a plurality of image sample pairs, each of the image sample pairs comprising a first feature point image sample, a second feature point image sample, and a feature point matching relationship between the first feature point image sample and the second feature point image sample, wherein the first feature point image sample is a feature point image corresponding to a synthetic blood vessel centerline image sample, and the second feature point image sample is a feature point image corresponding to a three-dimensional blood vessel centerline image sample, wherein the synthetic blood vessel centerline image sample is obtained based on the three-dimensional blood vessel centerline image sample using a data deformation method embedded with anatomical prior knowledge;

[0158] A model construction unit, configured to pre-construct a deep graph neural network model, wherein the deep graph neural network model is a deep graph neural network model based on self-attention and cross-attention;

[0159] A model training unit is used to train the deep graph neural network model using the first feature point image sample and the second feature point image sample as input data and the corresponding feature point matching relationship as supervision to obtain a trained deep graph neural network model.

[0160] It should be noted that the devices provided in the above embodiments are only illustrated by the division of the above functional modules when implementing their functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the devices provided in the above embodiments and the corresponding method embodiments are based on the same concept. The specific implementation process is detailed in the corresponding method embodiments and will not be repeated here.

[0161] One embodiment of the present invention also provides an electronic device, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the image registration method provided in the above method embodiment or the training method of the deep graph neural network model provided in the above method embodiment.

[0162] The memory can be used to store software programs and modules, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store an operating system, application programs required for functions, etc.; the data storage area can store data created based on the use of the device, etc. In addition, the memory can include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory can also include a memory controller to provide the processor with access to the memory.

[0163] In a specific embodiment, FIG11 shows a schematic diagram of the hardware structure of an electronic device for implementing the image registration method or the training method of the deep graph neural network model provided by the embodiment of the present invention. The electronic device may be a computer terminal, a mobile terminal or other device. The electronic device may also participate in the formation of or include the image registration device provided by the embodiment of the present invention. As shown in FIG11 , the electronic device 1100 may include a memory 1110 of one or more computer-readable storage media, a processor 1120 of one or more processing cores, an input unit 1130, a display unit 1140, a radio frequency (RF) circuit 1150, a wireless fidelity (WiFi) module 1160, and a power supply 1170 and other components. Those skilled in the art will understand that the electronic device structure shown in FIG11 does not constitute a limitation on the electronic device 1100, and may include more or fewer components than shown in the figure, or a combination of certain components, or different component arrangements. Among them:

[0164] The memory 1110 can be used to store software programs and modules. The processor 1120 executes various functional applications and data processing by running or executing the software programs and modules stored in the memory 1110 and calling the data stored in the memory 1110. The memory 1110 may mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the electronic device, etc. In addition, the memory 1110 may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory 1110 may also include a memory controller to provide the processor 1120 with access to the memory 1110.

[0165] The processor 1120 is the control center of the electronic device 1100. It connects the various parts of the entire electronic device using various interfaces and lines. It executes or executes the software programs and / or modules stored in the memory 1110 and calls the data stored in the memory 1110 to perform various functions of the electronic device 1100 and process data, thereby monitoring the electronic device 1100 as a whole. The processor 1120 can be a central processing unit, or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits, etc.

[0166] (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0167] The input unit 1130 may be configured to receive input digital or character information and generate keyboard, mouse, joystick, optical, or trackball signal input related to user settings and function control. Specifically, the input unit 1130 may include a touch-sensitive surface 1131 and other input devices 1132. Specifically, the touch-sensitive surface 1131 may include, but is not limited to, a touchpad or a touch screen, and the other input devices 1132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control button, an on / off button, etc.), a trackball, a mouse, a joystick, and the like.

[0168] The display unit 1140 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the electronic device. These graphical user interfaces can be composed of graphics, text, icons, videos, or any combination thereof. The display unit 1140 may include a display panel 1141. Optionally, the display panel 1141 can be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0169] The RF circuit 1150 can be used to receive and send signals during information transmission or calls. In particular, after receiving downlink information from the base station, it is handed over to one or more processors 1120 for processing; in addition, uplink data is sent to the base station. Generally, the RF circuit 1150 includes but is not limited to an antenna, at least one amplifier, a tuner, one or more oscillators, a subscriber identity module (SIM) card, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF circuit 1150 can also communicate with the network and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile Communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0170] WiFi is a short-range wireless transmission technology. Electronic device 1100, through WiFi module 1160, can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband Internet access. Although FIG11 illustrates WiFi module 1160, it is understood that it is not a required component of electronic device 1100 and can be omitted as needed without changing the essence of the invention.

[0171] The electronic device 1100 also includes a power supply 1170 (e.g., a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 1120 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The power supply 1170 can also include one or more DC or AC power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0172] It should be noted that, although not shown, the electronic device 1100 may also include a Bluetooth module, etc., which will not be described in detail here.

[0173] One embodiment of the present invention also provides a computer-readable storage medium, which can be set in an electronic device to store at least one instruction or at least one program related to implementing an image registration method or a training method for a deep graph neural network model. The at least one instruction or the at least one program is loaded and executed by the processor to implement the image registration method or the training method for a deep graph neural network model provided by the above method embodiment.

[0174] Optionally, in an embodiment of the present invention, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store program codes.

[0175] One embodiment of the present invention further provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image registration method or deep graph neural network model training method provided in the various optional implementation examples described above.

[0176] It should be noted that the order in which the embodiments of the present invention are described above is for illustrative purposes only and does not represent the superiority or inferiority of the embodiments. The above description is of specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0177] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences from other embodiments. In particular, the device embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0178] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0179] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An image registration method, characterized in that: include: acquiring a two-dimensional vascular centerline image and a three-dimensional vascular centerline image including a vascular segment of interest; Extracting feature points from the two-dimensional blood vessel centerline image to obtain a first feature point image; Projecting the three-dimensional blood vessel centerline image onto a two-dimensional plane, and extracting feature points from the obtained two-dimensional projection image to obtain a second feature point image; Inputting the first feature point image and the second feature point image into a pre-trained deep graph neural network model for feature point matching to obtain a feature point matching result; In which, the deep graph neural network model is trained using a training sample data set including multiple image sample pairs, each of the image sample pairs includes a first feature point image sample, a second feature point image sample and a feature point matching relationship between the two, the first feature point image sample is a feature point image corresponding to a synthetic blood vessel centerline image sample, the second feature point image sample is a feature point image corresponding to a three-dimensional blood vessel centerline image sample, and the synthetic blood vessel centerline image sample is obtained based on the three-dimensional blood vessel centerline image sample using a data deformation method embedded with anatomical prior knowledge.

2. The method according to claim 1, characterized in that The acquiring of a two-dimensional vascular centerline image and a three-dimensional vascular centerline image of a vascular segment of interest includes: Acquiring coronary angiography images and coronary CT angiography images including the vascular segment of interest; performing blood vessel segmentation and centerline extraction on the coronary angiography image to obtain the two-dimensional blood vessel centerline image; The coronary artery CT angiography image is subjected to vessel segmentation and centerline extraction to obtain the three-dimensional vessel centerline image.

3. The method according to claim 1, characterized in that The extracting feature points from the two-dimensional blood vessel centerline image to obtain a first feature point image includes: Inputting the two-dimensional blood vessel centerline image into a pre-trained feature point extraction network model to extract feature points to obtain the first feature point image; The step of projecting the three-dimensional blood vessel centerline image onto a two-dimensional plane and extracting feature points from the obtained two-dimensional projection image to obtain a second feature point image includes: Projecting the three-dimensional blood vessel centerline image onto a two-dimensional plane to obtain the two-dimensional projection image; The two-dimensional projection image is input into the feature point extraction network model to extract feature points to obtain the second feature point image.

4. The method according to claim 1, wherein The method further comprises: Based on the feature point matching result, projection parameters of the three-dimensional blood vessel centerline image projected onto a two-dimensional plane are corrected to obtain a rigid registration result between the two-dimensional blood vessel centerline image and the three-dimensional blood vessel centerline image.

5. The method according to claim 4, characterized in that Projecting the three-dimensional blood vessel centerline image onto a two-dimensional plane includes: The three-dimensional blood vessel centerline image is projected onto a two-dimensional plane using the corrected projection parameters to obtain the two-dimensional projection image.

6. The method according to claim 5, characterized in that The method further comprises: Based on the feature point matching result, non-rigid registration is performed on the two-dimensional blood vessel centerline image and the three-dimensional blood vessel centerline image to obtain a corresponding non-rigid registration result.

7. The method according to claim 1, characterized in that The method further includes pre-establishing the deep graph neural network model, and the process of establishing the deep graph neural network model includes: Acquire a training sample dataset, the training sample dataset comprising a plurality of image sample pairs, each of the image sample pairs comprising a first feature point image sample, a second feature point image sample, and a feature point matching relationship between the first feature point image sample and the second feature point image sample, wherein the first feature point image sample is a feature point image corresponding to a synthetic blood vessel centerline image sample, and the second feature point image sample is a feature point image corresponding to a three-dimensional blood vessel centerline image sample, wherein the synthetic blood vessel centerline image sample is obtained based on the three-dimensional blood vessel centerline image sample using a data deformation method embedded with anatomical prior knowledge; Pre-building a deep graph neural network model, wherein the deep graph neural network model is a deep graph neural network model based on self-attention and cross-attention; The first feature point image sample and the second feature point image sample are used as input data, and the corresponding feature point matching relationship is used as supervision to train the deep graph neural network model to obtain a trained deep graph neural network model.

8. The method according to claim 7, characterized in that The obtaining of a training sample data set includes: Acquire multiple three-dimensional blood vessel centerline image samples; Modeling each of the three-dimensional blood vessel centerline image samples as a three-dimensional hinge structure, and randomly deforming the obtained three-dimensional blood vessel centerline image samples of the three-dimensional hinge structure; Performing simulated C-arm projection on the deformed three-dimensional blood vessel centerline image sample to obtain a corresponding synthetic blood vessel centerline image sample; According to each of the three-dimensional blood vessel centerline image samples and its corresponding synthesized blood vessel centerline image sample, the corresponding first feature point image sample, the second feature point image sample and the feature point matching relationship between the two are determined.

9. An image registration device, characterized in that: include: An image acquisition module, configured to acquire a two-dimensional vascular centerline image and a three-dimensional vascular centerline image including a vascular segment of interest; a first feature point extraction module, configured to extract feature points from the two-dimensional blood vessel centerline image to obtain a first feature point image; a second feature point extraction module, configured to project the three-dimensional blood vessel centerline image onto a two-dimensional plane, and extract feature points from the obtained two-dimensional projection image to obtain a second feature point image; A feature point matching module, configured to input the first feature point image and the second feature point image into a pre-trained deep graph neural network model for feature point matching to obtain a feature point matching result; In which, the deep graph neural network model is trained using a training sample data set including multiple image sample pairs, each of the image sample pairs includes a first feature point image sample, a second feature point image sample and a feature point matching relationship between the two, the first feature point image sample is a feature point image corresponding to a synthetic blood vessel centerline image sample, the second feature point image sample is a feature point image corresponding to a three-dimensional blood vessel centerline image sample, and the synthetic blood vessel centerline image sample is obtained based on the three-dimensional blood vessel centerline image sample using a data deformation method embedded with anatomical prior knowledge.

10. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the image registration method according to any one of claims 1 to 8.

Images