Blood vessel registration method, device, electronic device and storage medium

Abstract

The present invention discloses a blood vessel registration method, device, electronic device, and storage medium. The method includes: respectively obtaining a first blood vessel centerline image corresponding to a first blood vessel image to be registered and a second blood vessel centerline image corresponding to a second blood vessel image to be registered; determining a reconstructed projection image corresponding to the first blood vessel image to be registered, inputting the reconstructed projection image and the second blood vessel image to be registered into a pre-trained key point tracking model to obtain segmented key points; segmenting the first blood vessel centerline image and the second blood vessel centerline image based on the segmented key points to obtain a first segmented centerline image and a second segmented centerline image; and determining a blood vessel registration image based on the first segmented centerline image and the second segmented centerline image. The above technical solution predicts segmented key points through a key point tracking model, and then performs blood vessel segmentation based on the segmented key points with improved accuracy, and then performs registration, thereby improving registration accuracy.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a blood vessel registration method, device, electronic equipment and storage medium. Background Art

[0002] Percutaneous Coronary Intervention (PCI) is currently the most important and effective interventional treatment for coronary heart disease.

[0003] In complex PCI surgeries, doctors usually use intraoperative 2D angiographic images (such as DSA) with clear structure and dynamic real-time as guidance images. However, this modality image is acquired through the projection principle and therefore lacks three-dimensional depth information. Due to the tissue overlap caused by projection, it is difficult for doctors to make clear and intuitive decisions on intervention methods and treatment plans during surgery. Preoperative CTA just makes up for this shortcoming. Through three-dimensional reconstruction, it can present the three-dimensional spatial information of blood vessels intuitively and stereoscopically. Therefore, by fusing real-time 2D angiographic images and CTA images with spatial structural information to perform complex PCI surgeries, the uncertainty of using only 2D angiographic images for guidance can be greatly reduced, thereby improving the success rate of the surgery.

[0004] Current 3D / 2D vascular registration technologies mostly use the entire coronary tree as input to register the entire image. This global registration approach has strict requirements on the length of the vessels in the registered image (requiring vessel segments of the same length).

[0005] In the process of realizing the present invention, the inventors discovered that there are at least the following technical problems in the prior art: the prior art solution has the problem of low registration accuracy. Summary of the Invention

[0006] The present invention provides a blood vessel registration method, device, electronic equipment and storage medium to solve the problem of low blood vessel registration accuracy.

[0007] According to one aspect of the present invention, a blood vessel registration method is provided, comprising:

[0008] respectively acquiring a first blood vessel centerline image corresponding to the first blood vessel image to be registered and a second blood vessel centerline image corresponding to the second blood vessel image to be registered;

[0009] Determining a reconstructed projection image corresponding to the first vascular image to be registered, and inputting the reconstructed projection image corresponding to the first vascular image to be registered and the second vascular image to be registered into a pre-trained key point tracking model to obtain segmented key points;

[0010] Segmenting the first blood vessel centerline image and the second blood vessel centerline image based on the segmentation key points to obtain a first segmented centerline image and a second segmented centerline image;

[0011] A blood vessel registration image is determined based on the first segmented centerline image and the second segmented centerline image.

[0012] According to another aspect of the present invention, there is provided a blood vessel registration apparatus, comprising:

[0013] A centerline image acquisition module, configured to respectively acquire a first blood vessel centerline image corresponding to the first blood vessel image to be registered and a second blood vessel centerline image corresponding to the second blood vessel image to be registered;

[0014] A segmented key point prediction module is configured to determine a reconstructed projection image corresponding to the first vascular image to be registered, and input the reconstructed projection image corresponding to the first vascular image to be registered and the second vascular image to be registered into a pre-trained key point tracking model to obtain segmented key points;

[0015] a centerline image segmentation module, configured to segment the first blood vessel centerline image and the second blood vessel centerline image based on the segmentation key points to obtain a first segmented centerline image and a second segmented centerline image;

[0016] The blood vessel registration image determination module is configured to determine a blood vessel registration image based on the first segmented centerline image and the second segmented centerline image.

[0017] According to another aspect of the present invention, an electronic device is provided, comprising:

[0018] at least one processor; and

[0019] a memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores a computer program executable by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the blood vessel registration method according to any embodiment of the present invention.

[0021] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the blood vessel registration method according to any embodiment of the present invention when executed.

[0022] The vascular registration method provided by the present invention comprises the following steps: obtaining a first vascular centerline image corresponding to a first vascular image to be registered and a second vascular centerline image corresponding to a second vascular image to be registered; determining a reconstructed projection image corresponding to the first vascular image to be registered, inputting the reconstructed projection image corresponding to the first vascular image to be registered and the second vascular image to be registered into a pre-trained key point tracking model to obtain segmentation key points; segmenting the first and second vascular centerline images based on the segmentation key points to obtain first and second segmentation centerline images; and determining a vascular registration image based on the first and second segmentation centerline images. The above technical solution predicts segmentation key points using the key point tracking model, then segments the blood vessels based on the segmentation key points, and then performs registration, thereby improving registration accuracy.

[0023] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 This is a flow chart of a blood vessel registration method provided according to the first embodiment of the present invention;

[0026] Figure 2 A schematic diagram of a segmented registration process provided in the first embodiment of the present invention;

[0027] Figure 3 This is a flow chart of a blood vessel registration method provided according to the second embodiment of the present invention;

[0028] Figure 4 A schematic diagram of an image reconstruction process provided in the second embodiment of the present invention;

[0029] Figure 5 This is a flow chart of a blood vessel registration method provided according to the third embodiment of the present invention;

[0030] Figure 6 A network architecture diagram of a key point tracking model provided by an embodiment of the present invention;

[0031] Figure 7 is a structural diagram of a blood vessel registration device provided according to a fourth embodiment of the present invention;

[0032] Figure 8 3 is a schematic structural diagram of an electronic device for implementing the blood vessel registration method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0034] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] Example 1

[0036] Figure 1 This is a flow chart of a blood vessel registration method provided in the first embodiment of the present invention. This embodiment is applicable to the case where blood vessel images of different dimensions are automatically registered. The method can be performed by a blood vessel registration device. The blood vessel registration device can be implemented in the form of hardware and / or software. The blood vessel registration device can be configured in a computer terminal and / or terminal. Figure 1 As shown, the method includes:

[0037] S110 , respectively acquiring a first blood vessel centerline image corresponding to the first blood vessel image to be registered and a second blood vessel centerline image corresponding to the second blood vessel image to be registered.

[0038] In this embodiment, the first vascular image to be registered refers to the vascular image to be registered. Similarly, the second vascular image to be registered refers to the image to be registered with the first vascular image. The first vascular image to be registered or the second vascular image to be registered can be a real-time two-dimensional, three-dimensional, or multi-dimensional image, without limitation. It should be noted that the first vascular image to be registered and the second vascular image to be registered can be images of different dimensions. For example, the first vascular image to be registered can be an unprocessed three-dimensional vascular image, and the second vascular image to be registered can be an unprocessed two-dimensional vascular image. The number of the first vascular image to be registered and the second vascular image to be registered can be one or more, without limitation.

[0039] Illustratively, the first blood vessel image to be registered or the second blood vessel image to be registered may be a medical image containing blood vessels, such as computed tomography angiography (CTA) or digital subtraction angiography (DSA).

[0040] In this embodiment, the first vessel centerline image refers to a vessel centerline image obtained by performing morphological processing or other operations on the first vessel image to be registered. Optionally, the first vessel centerline image may be a projected centerline image. It is understood that when the first vessel image to be registered is a three-dimensional vessel image, the centerline of the three-dimensional vessel image may be reconstructed and projected to obtain a two-dimensional projected centerline image. Similarly, the second vessel centerline image refers to a vessel centerline image obtained by performing morphological processing or other operations on the second vessel image to be registered.

[0041] In some optional embodiments, respectively acquiring a first blood vessel centerline image corresponding to the first blood vessel image to be registered and a second blood vessel centerline image corresponding to the second blood vessel image to be registered includes: acquiring the blood vessel images to be registered, wherein the blood vessel images to be registered include the first blood vessel image to be registered and the second blood vessel image to be registered; segmenting the first blood vessel image to be registered to obtain a first blood vessel image, performing centerline extraction on the first blood vessel image to obtain a first blood vessel centerline image corresponding to the first blood vessel image to be registered; segmenting the second blood vessel image to be registered to obtain a second blood vessel image, performing centerline extraction on the second blood vessel image to obtain a second blood vessel centerline image corresponding to the second blood vessel image to be registered.

[0042] The first blood vessel image refers to the initial blood vessel segmentation image of the first blood vessel image to be registered. Similarly, the second blood vessel image refers to the initial blood vessel segmentation image of the second blood vessel image to be registered.

[0043] Exemplarily, the first vascular image to be registered may be a three-dimensional CTA preoperative image, and the second vascular image to be registered may be a two-dimensional DSA intraoperative image. The formats of the first vascular image to be registered and the second vascular image to be registered are not limited here, and may be, for example, DICOM format, etc. Image preprocessing may include, but is not limited to, a CTA image data segmentation module, a CTA image center line extraction module, a DSA image data segmentation module, and a DSA image center line extraction module. The CTA image data segmentation module includes: a training phase, in which a large amount of CTA data is annotated with blood vessels, and based on the annotation results, a three-dimensional (3D) segmentation model for blood vessel segmentation is trained. The network architecture of the three-dimensional segmentation model may be 3DU-net, V-net, etc.; an inference phase, in which the trained three-dimensional segmentation model is used to predict new CTA image data, and a three-dimensional initial segmentation image of the blood vessels is obtained through post-processing operations. The CTA image center line extraction module: a series of morphological operations are performed on the three-dimensional initial segmentation image of the blood vessels, and a smoothing algorithm is used to smooth the image to obtain a three-dimensional blood vessel centerline image. DSA Image Data Segmentation Module: During the training phase, vessels are annotated on a large amount of DSA data. Based on the annotated results, a two-dimensional (2D) segmentation model is trained for vessel segmentation. The network architecture of the 2D segmentation model can be 2DU-net, V-net, etc. During the inference phase, the trained 2D segmentation model is used to predict new DSA image data, and a 2D preliminary segmented vessel image is obtained through post-processing. DSA Image Centerline Extraction Module: A series of morphological operations are performed on the 2D preliminary segmented vessel image, and a smoothing algorithm is used to smooth the image to obtain a 2D vessel centerline image.

[0044] S120 , determining a reconstructed projection image corresponding to the first vascular image to be registered, and inputting the reconstructed projection image corresponding to the first vascular image to be registered and the second vascular image to be registered into a pre-trained key point tracking model to obtain segmented key points.

[0045] In this embodiment, the reconstructed projection image refers to a digitally reconstructed image of the first blood vessel image to be registered.

[0046] For example, the first image of the blood vessel to be registered can be digitally reconstructed and projected using a voxel-based projection algorithm, a maximum intensity projection (MIP) algorithm, or the like to obtain a reconstructed projection image. The MIP algorithm is a widely used projection algorithm for CT and MR images. By retaining the pixels with the highest density on a ray and projecting them onto a two-dimensional plane, the MIP algorithm can effectively display vascular stenosis, dilation, and pathology.

[0047] In this embodiment, the key point tracking model refers to a network model used to predict segment key points.

[0048] Specifically, the reconstructed projection image corresponding to the first vessel image to be registered and the second vessel image to be registered are used as input data to a pre-trained keypoint tracking model to obtain segmentation keypoints. The number of predicted segmentation keypoints can be one or more, and is not limited here. The segmentation keypoints can be used to segment the vessel centerline.

[0049] S130 . Segment the first blood vessel centerline image and the second blood vessel centerline image based on the segmentation key points to obtain a first segmented centerline image and a second segmented centerline image.

[0050] In this embodiment, the first segmented centerline image refers to the segmented blood vessel image of the first blood vessel centerline image. Similarly, the second segmented centerline image refers to the segmented blood vessel image of the second blood vessel centerline image.

[0051] It is understandable that the number of segments of the first segmented centerline image may vary according to the number of segmented key points. Similarly, the number of segments of the second segmented centerline image may also vary according to the number of segmented key points.

[0052] S140: Determine a blood vessel registration image based on the first segmented centerline image and the second segmented centerline image.

[0053] The blood vessel registration image refers to the registration result of the first blood vessel image to be registered and the second blood vessel image to be registered.

[0054] In some optional embodiments, determining the vascular registration image based on the first segmented centerline image and the second segmented centerline image includes: segmentally registering the first segmented centerline image and the second segmented centerline image corresponding to each vascular segment to obtain a registration image of each vascular segment; and fusing the registration images of each vascular segment to obtain a vascular registration image.

[0055] Exemplarily, the registration method of this embodiment may be a registration method based on deep learning, including but not limited to SyN, Demons and other registration methods. Figure 2 A schematic diagram of a segmented registration process provided by an embodiment of the present invention is shown in FIG. Figure 2As shown, centerline extraction and projection are performed on 3D CTA data to obtain a projection centerline image, which contains the projection centerline; centerline extraction is performed on 2D DSA data to obtain a DSA centerline image, which contains the DSA centerline; the projection centerline and the DSA centerline are then segmented based on the segmented key points predicted by the key point tracking model, where the segmented key points can contain coordinate information; further, segmented registration is performed (the above figure takes segmentation into two segments as an example); finally, the results of the segmented registration are fused into a complete centerline, i.e., a vascular registration image is obtained.

[0056] The vascular registration method provided by the present invention comprises the following steps: obtaining a first vascular centerline image corresponding to a first vascular image to be registered and a second vascular centerline image corresponding to a second vascular image to be registered; determining a reconstructed projection image corresponding to the first vascular image to be registered, inputting the reconstructed projection image corresponding to the first vascular image to be registered and the second vascular image to be registered into a pre-trained key point tracking model to obtain segmentation key points; segmenting the first and second vascular centerline images based on the segmentation key points to obtain first and second segmentation centerline images; and determining a vascular registration image based on the first and second segmentation centerline images. The above technical solution predicts segmentation key points using the key point tracking model, then segments the blood vessels based on the segmentation key points, and then performs registration, thereby improving registration accuracy.

[0057] Example 2

[0058] Figure 3 This is a flowchart of a blood vessel registration method provided in Example 2 of the present invention. The method of this embodiment can be combined with the various optional solutions in the blood vessel registration method provided in the above embodiments. The blood vessel registration method provided in this embodiment is further optimized. Optionally, the first blood vessel image to be registered is a three-dimensional blood vessel image, and determining the reconstructed projection image corresponding to the first blood vessel image to be registered includes: obtaining angle information of the second blood vessel image to be registered; rotating the three-dimensional blood vessel image based on the angle information of the second blood vessel image to be registered to obtain a rotated three-dimensional blood vessel image; resampling the rotated three-dimensional blood vessel image to obtain a resampled three-dimensional blood vessel image; and reconstructing and projecting the resampled three-dimensional blood vessel image to obtain a reconstructed projection image corresponding to the three-dimensional blood vessel image.

[0059] like Figure 3 As shown, the method includes:

[0060] S210 , respectively acquire a first blood vessel centerline image corresponding to the first blood vessel image to be registered and a second blood vessel centerline image corresponding to the second blood vessel image to be registered.

[0061] S220: Acquire angle information of the second blood vessel image to be registered.

[0062] The angle information may include a primary spatial angle and a secondary spatial angle.

[0063] Exemplarily, the second blood vessel image to be registered may be a digital subtraction angiography (DSA) image, and the corresponding angle information may be obtained from DICOM (Digital Imaging and Communications in Medicine) data of the DSA.

[0064] S230 : Rotate the three-dimensional blood vessel image based on the angle information of the second blood vessel image to be registered to obtain a rotated three-dimensional blood vessel image.

[0065] S240 : Resample the rotated three-dimensional blood vessel image to obtain a resampled three-dimensional blood vessel image.

[0066] S250 , reconstructing and projecting the resampled three-dimensional blood vessel image to obtain a reconstructed projection image corresponding to the three-dimensional blood vessel image.

[0067] For example, Figure 4 A schematic diagram of an image reconstruction process provided by an embodiment of the present invention. The three-dimensional vascular image may be a CTA image (1), and the CTA image is rotated along a primary spatial angle and a secondary spatial angle to obtain a rotated CTA image. The rotated CTA image is resampled to obtain a resampled CTA image (2). The resampled CTA image is digitally reconstructed and projected to obtain a two-dimensional projection image (3) corresponding to the CTA image, i.e., a two-dimensional reconstructed projection image.

[0068] S260 : Input the reconstructed projection image corresponding to the first blood vessel image to be registered and the second blood vessel image to be registered into a pre-trained key point tracking model to obtain segmented key points.

[0069] S270 , segment the first blood vessel centerline image and the second blood vessel centerline image based on the segmentation key points to obtain a first segmented centerline image and a second segmented centerline image.

[0070] S280: Determine a blood vessel registration image based on the first segmented centerline image and the second segmented centerline image.

[0071] In some optional embodiments, after obtaining a reconstructed projection image corresponding to a 3D vascular image, keypoint coordinates of the reconstructed projection image may also be obtained. Specifically, the keypoint coordinates of the reconstructed projection image may be directly obtained by projecting the 3D vascular image. The keypoint coordinates may be the coordinates of a branch point of the vascular vessel.

[0072] In some optional embodiments, the second blood vessel image to be registered can be a DSA image, and there can be multiple DSA images. Before inputting the key point tracking model, multiple DSA images can be spliced, so that this embodiment can process multiple DSA images at the same time, saving the time for key point positioning.

[0073] The vascular registration method provided by the present invention obtains angle information of a second vascular image to be registered; rotates the three-dimensional vascular image based on the angle information of the second vascular image to be registered to obtain a rotated three-dimensional vascular image; resamples the rotated three-dimensional vascular image to obtain a resampled three-dimensional vascular image; and reconstructs and projects the resampled three-dimensional vascular image to obtain a reconstructed projection image corresponding to the three-dimensional vascular image, thereby unifying the dimensions of the two vascular images and facilitating image registration.

[0074] Example 3

[0075] Figure 5 This is a flowchart of a blood vessel registration method provided in Example 3 of the present invention. The method of this embodiment can be combined with the various optional schemes in the blood vessel registration method provided in the above embodiments. The blood vessel registration method provided in this embodiment has been further optimized. Optionally, the key point tracking model includes a convolutional network model, a feature extraction module, and a key point convolutional layer; accordingly, the reconstructed projection image corresponding to the first blood vessel image to be registered and the second blood vessel image to be registered are input into the pre-trained key point tracking model to obtain segmented key points, including: inputting the reconstructed projection image corresponding to the first blood vessel image to be registered and the second blood vessel image to be registered into the convolutional network model to obtain a first feature map and a second feature map; inputting the key point coordinates of the reconstructed projection image and the first feature map into the feature extraction module to obtain a key point feature image; inputting the key point feature image and the second feature map into the key point convolutional layer to obtain segmented key points.

[0076] like Figure 5 As shown, the method includes:

[0077] S310 , respectively acquire a first blood vessel centerline image corresponding to the first blood vessel image to be registered and a second blood vessel centerline image corresponding to the second blood vessel image to be registered.

[0078] S320: Determine a reconstructed projection image corresponding to the first blood vessel image to be registered.

[0079] S330: Input the reconstructed projection image corresponding to the first blood vessel image to be registered and the second blood vessel image to be registered into a convolutional network model to obtain a first feature map and a second feature map.

[0080] In this embodiment, the keypoint tracking model includes a convolutional network model, a feature extraction module, and a keypoint convolution layer. The convolutional network model can be a three-dimensional convolutional model, such as a 3D-Unet model. The first feature map is the feature map corresponding to the reconstructed projection image, and the second feature map is the feature map corresponding to the second vascular image to be registered.

[0081] It should be noted that in this embodiment, a single convolutional network model can be used to learn the reconstructed projection image and the second vascular image to be registered to obtain the first feature map and the second feature map. Alternatively, two identical or different convolutional network models can be used to learn the reconstructed projection image and the second vascular image to be registered to obtain the first feature map and the second feature map. The method for obtaining the feature maps is not limited herein.

[0082] S340: Input the key point coordinates of the reconstructed projection image and the first feature map into a feature extraction module to obtain a key point feature image.

[0083] The feature extraction module is used to extract features from the first feature map based on the key point coordinates of the reconstructed projection image, thereby obtaining key point feature images corresponding to each key point coordinate. The number of key point feature images can be one or more, which is not limited here.

[0084] In some optional embodiments, the key point coordinates of the reconstructed projection image and the first feature map are input into a feature extraction module to obtain a key point feature image, including: obtaining features within the neighborhood of the key point coordinates of the reconstructed projection image on the first feature map through the feature extraction module, and using the features within the neighborhood of the key point coordinates of the reconstructed projection image as the key point feature image.

[0085] Exemplarily, with the key point coordinates as the center point, the features within a neighborhood with a preset radius k of the key point coordinates are used as the key point feature image, and the feature size can be (2k+1)×(2k+1).

[0086] S350: Input the key point feature image and the second feature map into a key point convolution layer to obtain segmented key points.

[0087] Specifically, the key point feature image is convolved on the second feature map to obtain a probability map, and the pixel with the largest probability value in the probability map is selected as the segmentation key point. In some embodiments, the pixel whose probability value is within a preset probability value range can also be selected as the segmentation key point.

[0088] S360 : Segment the first blood vessel centerline image and the second blood vessel centerline image based on the segmentation key points to obtain a first segmented centerline image and a second segmented centerline image.

[0089] S370: Determine a blood vessel registration image based on the first segmented centerline image and the second segmented centerline image.

[0090] In some optional embodiments, the training steps of the key point tracking model include: obtaining multiple groups of training samples, the training samples including the reconstructed projection image of the first sample image, the key point coordinates of the reconstructed projection image of the first sample image, the second sample image and the key point annotation information; training the key point tracking model to be trained based on the reconstructed projection image of the first sample image, the key point coordinates of the reconstructed projection image of the first sample image, the second sample image and the key point annotation information to obtain the key point tracking model.

[0091] Among them, the key point annotation information refers to the position information of the pre-annotated key points, which can be used as the gold standard.

[0092] For example, Figure 6 This is a network architecture diagram of a key point tracking model provided by an embodiment of the present invention. Taking the right coronary artery as an example, the first sample image can be 3D-CTA data, the reconstructed projection image can be a two-dimensional projection image, and the second sample image can be a DSA sequence image. The input of the entire key point tracking model is the two-dimensional projection image, the DSA sequence image, and the key point coordinates on the two-dimensional projection image. According to the key points of the 3D-CTA data, the corresponding key points of the DSA sequence image are annotated to obtain the gold standard of the DSA key points, which is recorded as Furthermore, a convolutional network model (such as 3D-Unet) is used to learn the input image (two-dimensional projection image I M and DSA sequence images I D ) features, and obtain the first feature map F M and the second feature map F D In the feature extraction module, according to the key point coordinates of the two-dimensional projection image (Take two key points as an example) in the first feature map F M Upsample the features of the corresponding position; specifically, in order to enrich the expression of features, set a neighborhood k and extract the features within the neighborhood of the key point coordinate k (the feature size is (2k+1)×(2k+1)), recorded as and In the keypoint convolution layer, and As a (2k+1)×(2k+1) size convolution kernel, the second feature map F D Perform convolution and get The specific calculation formula is as follows:

[0093]

[0094] Where W is a learnable parameter used to learn the weights of each feature in the neighborhood of the key point. ⊙ is a convolution operation, and i is the number of key points. and gold standard The loss between can be measured using mean square error or cross

[0095] Entropy loss. The keypoint tracking model is optimized through the back-propagation algorithm until the model converges.

[0096] The vascular registration method provided by the present invention involves inputting a reconstructed projection image corresponding to a first vascular image to be registered and a second vascular image to be registered into a convolutional network model to obtain a first feature map and a second feature map. The coordinates of key points in the reconstructed projection image and the first feature map are then input into a feature extraction module to obtain a key point feature image. The key point feature image and the second feature map are then input into a key point convolutional layer to obtain segmented key points. The key point tracking model of this embodiment can more accurately identify segmented key points in intraoperative images, improving their accuracy and ensuring subsequent segmented registration.

[0097] Example 4

[0098] Figure 7 This is a structural diagram of a blood vessel registration device provided by the fourth embodiment of the present invention. Figure 7 As shown, the device includes:

[0099] A centerline image acquisition module 410 is configured to respectively acquire a first blood vessel centerline image corresponding to the first blood vessel image to be registered and a second blood vessel centerline image corresponding to the second blood vessel image to be registered;

[0100] A segmented key point prediction module 420 is configured to determine a reconstructed projection image corresponding to the first vascular image to be registered, and input the reconstructed projection image corresponding to the first vascular image to be registered and the second vascular image to be registered into a pre-trained key point tracking model to obtain segmented key points;

[0101] a centerline image segmentation module 430 for segmenting the first blood vessel centerline image and the second blood vessel centerline image based on the segmentation key points to obtain a first segmented centerline image and a second segmented centerline image;

[0102] The blood vessel registration image determination module 440 is configured to determine a blood vessel registration image based on the first segmented centerline image and the second segmented centerline image.

[0103] The vascular registration device provided by the present invention obtains a first vascular centerline image corresponding to a first vascular image to be registered and a second vascular centerline image corresponding to a second vascular image to be registered; determines a reconstructed projection image corresponding to the first vascular image to be registered, inputs the reconstructed projection image corresponding to the first vascular image to be registered and the second vascular image to be registered into a pre-trained key point tracking model to obtain segmentation key points; segments the first and second vascular centerline images based on the segmentation key points to obtain first and second segmentation centerline images; and determines a vascular registration image based on the first and second segmentation centerline images. The above technical solution predicts segmentation key points using the key point tracking model, then segments the blood vessels based on the segmentation key points, and then performs registration, thereby improving registration accuracy.

[0104] In some optional implementations, the centerline image acquisition module 410 is specifically configured to:

[0105] Acquire a blood vessel image to be registered, wherein the blood vessel image to be registered includes a first blood vessel image to be registered and a second blood vessel image to be registered;

[0106] Segmenting the first blood vessel image to be registered to obtain a first blood vessel image, and extracting a centerline of the first blood vessel image to obtain a first blood vessel centerline image corresponding to the first blood vessel image to be registered;

[0107] The second blood vessel image to be registered is segmented to obtain a second blood vessel image, and a centerline is extracted from the second blood vessel image to obtain a second blood vessel centerline image corresponding to the second blood vessel image to be registered.

[0108] In some optional implementations, the first vascular image to be registered is a three-dimensional vascular image; the segmented key point prediction module 420 is specifically configured to:

[0109] Acquiring angle information of the second blood vessel image to be registered;

[0110] rotating the three-dimensional blood vessel image based on the angle information of the second blood vessel image to be registered to obtain a rotated three-dimensional blood vessel image;

[0111] resampling the rotated three-dimensional blood vessel image to obtain a resampled three-dimensional blood vessel image;

[0112] Reconstruct and project the resampled three-dimensional blood vessel image to obtain a reconstructed projection image corresponding to the three-dimensional blood vessel image.

[0113] In some optional embodiments, the keypoint tracking model includes a convolutional network model, a feature extraction module, and a keypoint convolution layer, and the segmented keypoint prediction module 420 includes:

[0114] a feature map determining unit, configured to input the reconstructed projection image corresponding to the first blood vessel image to be registered and the second blood vessel image to be registered into a convolutional network model to obtain a first feature map and a second feature map;

[0115] a key point feature image determining unit, configured to input the key point coordinates of the reconstructed projection image and the first feature map into a feature extraction module to obtain a key point feature image;

[0116] The segmentation key point determination unit is used to input the key point feature image and the second feature map into the key point convolution layer to obtain segmentation key points.

[0117] In some optional implementations, the key point feature image determination unit is specifically configured to:

[0118] The feature extraction module obtains features in the neighborhood of the key point coordinates of the reconstructed projection image on the first feature map, and uses the features in the neighborhood of the key point coordinates of the reconstructed projection image as a key point feature image.

[0119] In some optional embodiments, the blood vessel registration image determination module 440 is specifically configured to:

[0120] Performing segment-wise registration on the first segmented centerline image and the second segmented centerline image corresponding to each blood vessel segment to obtain a registered image of each blood vessel segment;

[0121] The registered images of the blood vessel segments are fused to obtain a registered blood vessel image.

[0122] In some optional embodiments, the blood vessel registration device comprises:

[0123] A training sample acquisition module, configured to acquire multiple sets of training samples, wherein the training samples include a reconstructed projection image of a first sample image, key point coordinates of the reconstructed projection image of the first sample image, a second sample image, and key point annotation information;

[0124] The key point tracking model training module is used to train the key point tracking model to be trained based on the reconstructed projection image of the first sample image, the key point coordinates of the reconstructed projection image of the first sample image, the second sample image and the key point annotation information to obtain a key point tracking model.

[0125] The blood vessel registration device provided in the embodiment of the present invention can execute the blood vessel registration method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0126] Example 5

[0127] Figure 8 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0128] like Figure 8 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0129] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0130] The processor 11 may be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a vascular registration method, which includes:

[0131] respectively acquiring a first blood vessel centerline image corresponding to the first blood vessel image to be registered and a second blood vessel centerline image corresponding to the second blood vessel image to be registered;

[0132] Determining a reconstructed projection image corresponding to the first vascular image to be registered, and inputting the reconstructed projection image corresponding to the first vascular image to be registered and the second vascular image to be registered into a pre-trained key point tracking model to obtain segmented key points;

[0133] Segmenting the first blood vessel centerline image and the second blood vessel centerline image based on the segmentation key points to obtain a first segmented centerline image and a second segmented centerline image;

[0134] A blood vessel registration image is determined based on the first segmented centerline image and the second segmented centerline image.

[0135] In some embodiments, the vessel registration method can be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the vessel registration method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the vessel registration method in any other suitable manner (e.g., via firmware).

[0136] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0137] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0138] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0139] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0140] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0141] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within a cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0142] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0143] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A blood vessel registration method, characterized in that: include: respectively acquiring a first blood vessel centerline image corresponding to the first blood vessel image to be registered and a second blood vessel centerline image corresponding to the second blood vessel image to be registered; Determining a reconstructed projection image corresponding to the first vascular image to be registered, and inputting the reconstructed projection image corresponding to the first vascular image to be registered and the second vascular image to be registered into a pre-trained key point tracking model to obtain segmented key points; Segmenting the first blood vessel centerline image and the second blood vessel centerline image based on the segmentation key points to obtain a first segmented centerline image and a second segmented centerline image; determining a blood vessel registration image based on the first segmented centerline image and the second segmented centerline image; The key point tracking model includes a convolutional network model, a feature extraction module and a key point convolution layer; Accordingly, the reconstructed projection image corresponding to the first to-be-registered blood vessel image and the second to-be-registered blood vessel image are input into a pre-trained key point tracking model to obtain segmented key points, including: Inputting the reconstructed projection image corresponding to the first blood vessel image to be registered and the second blood vessel image to be registered into a convolutional network model to obtain a first feature map and a second feature map; Acquire, by the feature extraction module, features within a neighborhood of key point coordinates of the reconstructed projection image on the first feature map, and use the features within the neighborhood of the key point coordinates of the reconstructed projection image as a key point feature image, wherein the key point coordinates of the reconstructed projection image are obtained by projecting a three-dimensional vascular image, and the key point coordinates are the coordinates of branch points of the blood vessels; The key point feature image and the second feature map are input into the key point convolution layer to obtain segmented key points.

2. The method according to claim 1, characterized in that The method of respectively acquiring a first blood vessel centerline image corresponding to the first blood vessel image to be registered and a second blood vessel centerline image corresponding to the second blood vessel image to be registered comprises: Acquire a blood vessel image to be registered, wherein the blood vessel image to be registered includes a first blood vessel image to be registered and a second blood vessel image to be registered; Segmenting the first blood vessel image to be registered to obtain a first blood vessel image, and extracting a centerline of the first blood vessel image to obtain a first blood vessel centerline image corresponding to the first blood vessel image to be registered; The second blood vessel image to be registered is segmented to obtain a second blood vessel image, and a centerline is extracted from the second blood vessel image to obtain a second blood vessel centerline image corresponding to the second blood vessel image to be registered.

3. The method according to claim 1, characterized in that The first blood vessel image to be registered is a three-dimensional blood vessel image; The determining of the reconstructed projection image corresponding to the first blood vessel image to be registered includes: Acquiring angle information of the second blood vessel image to be registered; rotating the three-dimensional blood vessel image based on the angle information of the second blood vessel image to be registered to obtain a rotated three-dimensional blood vessel image; resampling the rotated three-dimensional blood vessel image to obtain a resampled three-dimensional blood vessel image; Reconstruct and project the resampled three-dimensional blood vessel image to obtain a reconstructed projection image corresponding to the three-dimensional blood vessel image.

4. The method according to claim 1, wherein The determining of the blood vessel registration image based on the first segmented centerline image and the second segmented centerline image comprises: Performing segment-wise registration on the first segmented centerline image and the second segmented centerline image corresponding to each blood vessel segment to obtain a registered image of each blood vessel segment; The registered images of the blood vessel segments are fused to obtain a registered blood vessel image.

5. The method according to claim 1, wherein The training steps of the key point tracking model include: Acquire multiple sets of training samples, where the training samples include a reconstructed projection image of a first sample image, key point coordinates of the reconstructed projection image of the first sample image, a second sample image, and key point annotation information; The key point tracking model to be trained is trained based on the reconstructed projection image of the first sample image, the key point coordinates of the reconstructed projection image of the first sample image, the second sample image and the key point annotation information to obtain a key point tracking model.

6. A blood vessel registration device, characterized in that: include: A centerline image acquisition module, configured to respectively acquire a first blood vessel centerline image corresponding to the first blood vessel image to be registered and a second blood vessel centerline image corresponding to the second blood vessel image to be registered; A segmented key point prediction module is configured to determine a reconstructed projection image corresponding to the first vascular image to be registered, and input the reconstructed projection image corresponding to the first vascular image to be registered and the second vascular image to be registered into a pre-trained key point tracking model to obtain segmented key points; a centerline image segmentation module, configured to segment the first blood vessel centerline image and the second blood vessel centerline image based on the segmentation key points to obtain a first segmented centerline image and a second segmented centerline image; a blood vessel registration image determination module, configured to determine a blood vessel registration image based on the first segmented centerline image and the second segmented centerline image; The key point tracking model includes a convolutional network model, a feature extraction module and a key point convolution layer; Accordingly, the segment key point prediction module includes: a feature map determining unit, configured to input the reconstructed projection image corresponding to the first blood vessel image to be registered and the second blood vessel image to be registered into a convolutional network model to obtain a first feature map and a second feature map; a key point feature image determining unit, configured to obtain, on the first feature map, features within a neighborhood of key point coordinates of the reconstructed projection image using the feature extraction module, and use the features within the neighborhood of the key point coordinates of the reconstructed projection image as a key point feature image, wherein the key point coordinates of the reconstructed projection image are obtained by projecting a three-dimensional vascular image, and the key point coordinates are the coordinates of branch points of the vascular vessel; The segmentation key point determination unit is used to input the key point feature image and the second feature map into the key point convolution layer to obtain segmentation key points.

7. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the blood vessel registration method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the blood vessel registration method according to any one of claims 1 to 5 when executed.

Images