An image registration method, device and equipment based on unsupervised learning

Abstract

Embodiments of the present specification disclose a kind of image registration method, device and equipment based on unsupervised learning.The method comprises: obtaining three-dimensional image data to be processed;Edge feature extraction is carried out to the three-dimensional image data to be processed, and the multi-scale edge feature image of pyramid structure is obtained, the multi-scale edge feature image includes the multi-scale edge feature image of the image to be registered and the multi-scale edge feature image of the fixed image;The three-dimensional image data to be processed is registered using the preset rigid registration network, and the first registration image is obtained, the multi-scale edge feature image is registered using the preset rigid registration network, and the second registration image is obtained;Based on the first registration image, the second registration image and the fixed image, the first registration image is adjusted, and the new first registration image is obtained as the registration image of the three-dimensional image data to be processed, can be based on local registration details, the global registration of registration network is realized, the performance and registration accuracy of image registration are provided.

Description

Technical Field

[0001] This specification relates to the fields of medical imaging and artificial intelligence, and in particular to an image registration method, apparatus and device based on unsupervised learning. Background Technology

[0002] Medical image registration is an important technique for comprehensively analyzing different images of the same patient. Currently, medical image registration often employs manual annotation methods, which rely heavily on the physician's expertise and result in low efficiency and accuracy. Therefore, automatic registration is of great significance.

[0003] As a result, 3D image registration methods have developed rapidly, especially deep learning-based medical image registration. Existing medical image registration methods are mainly supervised and unsupervised. However, supervised registration methods suffer from difficulties in obtaining rigid transformation parameters and labeling relevant pixels, thus limiting their application in medicine. Unsupervised registration methods can reduce labeling costs and offer better registration speed, but their registration accuracy is not high.

[0004] Therefore, a new registration method is needed to improve the accuracy of unsupervised registration. Summary of the Invention

[0005] This specification provides an image registration method, apparatus, and device based on unsupervised learning to solve the following technical problem: the low registration accuracy of existing unsupervised registration methods.

[0006] To solve the above-mentioned technical problems, the embodiments in this specification are implemented as follows:

[0007] This specification provides an image registration method based on unsupervised learning, comprising:

[0008] Acquire the 3D image data to be processed, which includes the image data of the image to be registered and the image data of the fixed image.

[0009] Edge features are extracted from the three-dimensional image data to be processed to obtain a multi-scale edge feature image of the pyramid structure. The multi-scale edge feature image includes the multi-scale edge feature image of the image to be registered and the multi-scale edge feature image of the fixed image.

[0010] The three-dimensional image data to be processed is registered using a preset rigid registration network to obtain a first registered image, and the multi-scale edge feature image is registered using the preset rigid registration network to obtain a second registered image.

[0011] Based on the first registration image, the second registration image, and the fixed image, the first registration image is adjusted to obtain a new first registration image as the registration image of the three-dimensional image data to be processed.

[0012] This specification provides an image registration device based on unsupervised learning, comprising:

[0013] The acquisition module acquires the 3D image data to be processed, which includes the image data of the image to be registered and the image data of the fixed image.

[0014] The pyramid boundary module extracts edge features from the three-dimensional image data to be processed, and obtains a multi-scale edge feature image of the pyramid structure. The multi-scale edge feature image includes the multi-scale edge feature image of the image to be registered and the multi-scale edge feature image of the fixed image.

[0015] The first registration module uses a preset rigid registration network to register the three-dimensional image data to be processed to obtain a first registered image, and uses the preset rigid registration network to register the multi-scale edge feature image to obtain a second registered image.

[0016] The second registration module adjusts the first registration image based on the first registration image, the second registration image, and the fixed image to obtain a new first registration image as the registration image of the three-dimensional image data to be processed.

[0017] This specification also provides an electronic device, including:

[0018] At least one processor; and,

[0019] A memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to:

[0021] Acquire the 3D image data to be processed, which includes the image data of the image to be registered and the image data of the fixed image.

[0022] Edge features are extracted from the three-dimensional image data to be processed to obtain a multi-scale edge feature image of the pyramid structure. The multi-scale edge feature image includes the multi-scale edge feature image of the image to be registered and the multi-scale edge feature image of the fixed image.

[0023] The three-dimensional image data to be processed is registered using a preset rigid registration network to obtain a first registered image, and the multi-scale edge feature image is registered using the preset rigid registration network to obtain a second registered image.

[0024] Based on the first registration image, the second registration image, and the fixed image, the first registration image is adjusted to obtain a new first registration image as the registration image of the three-dimensional image data to be processed.

[0025] This specification describes an embodiment for acquiring 3D image data to be processed, which includes image data of an image to be registered and image data of a fixed image. Edge features are extracted from the 3D image data to be processed to obtain a multi-scale edge feature image of a pyramid structure. The multi-scale edge feature image includes the multi-scale edge feature image of the image to be registered and the multi-scale edge feature image of the fixed image. A preset rigid registration network is used to register the 3D image data to be processed to obtain a first registered image. The preset rigid registration network is then used to register the multi-scale edge feature image to obtain a second registered image. Based on the first registered image, the second registered image, and the fixed image, the first registered image is adjusted to obtain a new first registered image as the registered image of the 3D image data to be processed. This method enables global registration of the registration network based on local registration details, improving image registration performance and accuracy. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 A schematic diagram of a first registration method is provided for the embodiments of this specification;

[0028] Figure 2 A framework diagram of an image registration method based on unsupervised learning provided for embodiments of this specification;

[0029] Figure 3 This is a schematic diagram of an image registration device based on unsupervised learning provided in an embodiment of this specification. Detailed Implementation

[0030] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0031] Image registration aligns two different images in the same coordinate system by learning the nonlinear spatial correspondences between them, thus finding wide application in medical image processing. Examples include aligning medical images of an object taken at different times, or matching an image of an object to a predefined coordinate system (e.g., anatomical atlases). However, manual registration methods present significant challenges for many registration tasks. Therefore, automatic registration is of great importance.

[0032] Many algorithms for medical image registration have been proposed. Most of these algorithms define a transformable framework space and iteratively update parameters using a set of functions and metrics to find optimal parameter values, achieving good results on multiple datasets. However, traditional iterative registration methods are slow, especially when registering 3D images with numerous pixel features, significantly increasing the computational cost of iteration and making them unsuitable for clinical applications. Deep learning-based registration methods, on the other hand, are typically faster. They do not require repeated iterations before generating results, thus offering greater practical value.

[0033] Supervised registration methods face significant bottlenecks that hinder their application in real-world medical tasks. The primary reason is the difficulty in obtaining ground truth flow field values ​​(rigid transformation parameters). Even for medical experts, labeling paired related pixels for medical image registration is challenging. Unsupervised registration methods, however, utilize differentiable spatial transformers (STNs) to align tissue structures based on the similarity measured between deformed and stationary images. This eliminates reliance on annotation counts, reducing annotation costs—a crucial advantage for high-cost data like 3D image registration. Currently, research on rigid registration of medical images is limited, and open-source code is scarce. Existing methods rely on convolutional autoencoders to learn highly discriminative features. However, feature extraction may not be optimal for registration tasks, as it neglects global and discriminative information, leading to low accuracy. Furthermore, this approach is not end-to-end, as it depends on traditional methods for feature extraction to establish the mapping between the two images. This method uses a predefined parameter matrix to learn the error and obtain the optimal registration solution. It employs the Spatial Transformation Network (STN) proposed by Jaderberg et al., which allows the neural network to spatially transform feature maps and achieve complete discriminability. To ensure satisfactory performance of these frameworks, a good optimization metric must be defined. This can be a potential drawback, as different metrics have their advantages and disadvantages, and their applicability varies depending on the task. CNN regression directly estimates the transformation parameters and then trains the model on synthetic X-ray images. Although this method has achieved faster and relatively accurate registration success rates than traditional methods, its training is complex due to its hierarchical training and the use of six regressors instead of simultaneously estimating all parameters. Building on this limitation, AirNet constructs a network for medical image registration that utilizes a CNN to compute the transformation parameters between two 3D images. Unlike unsupervised learning methods that map images to scalar-valued metrics, AirNet learns 3D affine transformation parameters by directly predicting twelve-dimensional vectors, a completely different approach from previous hierarchical learning methods. This method effectively achieves rigid registration in 3D CT scans. Although unsupervised registration methods have made good progress and have advantages in registration speed and low annotation cost, their registration accuracy is significantly lower than that of traditional registration methods. Therefore, it is very important to improve the registration performance of unsupervised networks.

[0034] Based on this, the embodiments of this specification provide a registration method. Figure 1 A schematic diagram of a first registration method is provided for embodiments of this specification, such as... Figure 1 As shown, the method includes the following steps:

[0035] Step S101: Obtain the three-dimensional image data to be processed. The three-dimensional image data to be processed includes the image data of the image to be registered and the image data of the fixed image.

[0036] In the embodiments of this specification, the three-dimensional image data to be processed is three-dimensional medical image data, specifically, it can be CTP image data, MRI image data, etc.

[0037] In the embodiments of this specification, the three-dimensional image data are image data from different acquisition times of the same patient. In a specific embodiment, the acquisition time of the image data of the fixed image is earlier than the acquisition time of the image data of the image to be registered.

[0038] In one embodiment of this specification, after acquiring the 3D image data to be processed, it is necessary to preprocess the 3D image data, specifically by setting the window width and window level, normalizing, and other operations. The preprocessing of the 3D image data can be determined according to different task requirements.

[0039] Step S103: Extract edge features from the three-dimensional image data to be processed to obtain a multi-scale edge feature image of the pyramid structure. The multi-scale edge feature image includes the multi-scale edge feature image of the image to be registered and the multi-scale edge feature image of the fixed image.

[0040] In existing technologies, the impact of boundary features on the registration result is not considered during feature extraction in the registration process of 3D medical image data, thus affecting the accuracy of registration. To improve registration accuracy, edge features of the image data to be processed are extracted in the embodiments of this specification.

[0041] In the embodiments of this specification, the step of extracting edge features from the three-dimensional image data to be processed to obtain a multi-scale edge feature image of the pyramid structure specifically includes:

[0042] A first convolution operation is performed on the three-dimensional image data to be processed to obtain the feature map of the three-dimensional image data to be processed.

[0043] Edge features are extracted based on the feature map of the three-dimensional image data to be processed to obtain an extracted edge feature image, wherein the extracted edge feature image includes edge feature images of at least two sizes.

[0044] The extracted edge feature images are fused by addition or convolution to obtain a fused edge feature image.

[0045] The fused edge feature images are merged through a second convolution operation to obtain a multi-scale edge feature image of a pyramid structure.

[0046] In this embodiment of the specification, performing a first convolution operation on the three-dimensional image data to be processed to obtain a feature map of the three-dimensional image data to be processed specifically includes:

[0047] The three-dimensional image data to be processed is downsampled to obtain three-dimensional image data of different sizes;

[0048] The 3D image data of different sizes is processed by two convolution operations to obtain feature maps of each size corresponding to the 3D image data to be processed, and the feature maps of each size have the same dimension.

[0049] In the embodiments of this specification, the three-dimensional image data to be processed is downsampled to obtain three-dimensional image data of different sizes, and then edge features are extracted, thereby achieving accuracy and precision in edge feature extraction.

[0050] In one embodiment of this specification, based on the three-dimensional image data to be processed, four different sizes of three-dimensional image data can be obtained through three downsampling processes.

[0051] To further understand the methods provided in the embodiments of this specification, a brief introduction to downsampling will be given below. Downsampling is the process of reducing the size of an image. In the embodiments of this specification, the three-dimensional image data to be processed has a size of 512*512*512. After three downsampling processes, three-dimensional image data of 256*256*256, 128*128*128, and 64*62*64 can be obtained. That is, edge feature extraction can be performed based on the three-dimensional image data of 512*512*512, 256*256*256, 128*128*128, and 64*62*64. It should be noted that the size of the image data to be processed and the number of downsampling processes can be determined according to specific needs and do not constitute a specific limitation on this application.

[0052] In the embodiments of this specification, the step of extracting edge features from the three-dimensional image data to be processed to obtain a multi-scale edge feature image of the pyramid structure specifically includes:

[0053] A first convolution operation is performed on the three-dimensional image data to be processed to obtain the feature map of the three-dimensional image data to be processed.

[0054] Edge features are extracted based on the feature map of the three-dimensional image data to be processed to obtain an extracted edge feature image, wherein the extracted edge feature image includes edge feature images of at least two sizes.

[0055] The extracted edge feature images are fused by addition or convolution to obtain a fused edge feature image.

[0056] The fused edge feature images are merged through a second convolution operation to obtain a multi-scale edge feature image of a pyramid structure.

[0057] In this embodiment of the specification, performing a first convolution operation on the three-dimensional image data to be processed to obtain a feature map of the three-dimensional image data to be processed specifically includes:

[0058] The three-dimensional image data to be processed is downsampled to obtain three-dimensional image data of different sizes;

[0059] The 3D image data of different sizes is processed by two convolution operations to obtain feature maps of each size corresponding to the 3D image data to be processed, and the feature maps of each size have the same dimension.

[0060] In this embodiment of the specification, the method for edge feature extraction based on the feature map of the three-dimensional image data to be processed is as follows:

[0061] Ψ i =F i -avg(F i ), i = 1, 2, ..., L.

[0062] Where ψ represents the edge features at different scales, and F i Here, avg represents the feature maps of each size corresponding to the three-dimensional image data to be processed.

[0063] Since the dimensions of 3D image data of different sizes are different in the aforementioned steps, in order to achieve addition and fusion, it is necessary to first make the dimensions of 3D image data of different sizes consistent.

[0064] In this embodiment of the specification, after extracting edge features based on the feature map of the three-dimensional image data to be processed to obtain the extracted edge feature image, the following steps are further included:

[0065] The extracted edge feature images are upsampled so that all dimensions of the extracted edge feature images are the same.

[0066] In this embodiment of the specification, the extracted edge feature images are upsampled so that all the extracted edge feature images have the same dimensions. Specifically, the extracted edge feature images are upsampled so that the size of each edge feature image is the same as the size of the largest edge feature image. In this embodiment of the specification, upsampling can be performed using bilinear interpolation or deconvolution.

[0067] In this embodiment of the specification, the merging of the fused edge feature images through a second convolution operation to obtain a multi-scale edge feature image of a pyramid structure specifically includes:

[0068] The fused edge feature images are merged using a 1×1 convolution operation to obtain a multi-scale edge feature image with a pyramid structure.

[0069] Step S105: Register the three-dimensional image data to be processed using a preset rigid registration network to obtain a first registered image, and register the multi-scale edge feature image using the preset rigid registration network to obtain a second registered image.

[0070] In the embodiments described in this specification, the preset rigid registration network is an improved Voxelmorph rigid registration network (Voxelmorph6).

[0071] Step S107: Based on the first registration image, the second registration image, and the fixed image, adjust the first registration image to obtain a new first registration image as the registration image of the three-dimensional image data to be processed.

[0072] In the embodiments of this specification, the step of adjusting the first registration image based on the first registration image, the second registration image, and the fixed image to obtain a new first registration image as the registration image of the three-dimensional image data to be processed specifically includes:

[0073] The loss function of the first registered image and the fixed image is used as the first loss function, the loss function of the second registered image and the fixed image is used as the second loss function, and the loss function of the first registered image and the second registered image is used as the third loss function.

[0074] Based on the first loss function and the second loss function, the image to be registered is registered with the fixed image. Based on the third loss function, the registration effect of the first registered image and the second registered image is made consistent. The first registered image is adjusted to obtain a new first registered image as the registered image of the three-dimensional image data to be processed.

[0075] To further understand the image registration method based on unsupervised learning provided in the embodiments of this specification, the following will describe it in conjunction with specific embodiments. Figure 2 This is a framework diagram of an image registration method based on unsupervised learning, provided for embodiments of this specification. Figure 2 As shown, the input image I of the Pyramid edge module 1 After downsampling, three sizes of images I are obtained. 2 I 3 and I 4 Then, a convolution operation (Conv) is performed to generate corresponding feature maps F1, F2, F3, and F4. Edge features are extracted from feature maps F1, F2, F3, and F4 to obtain the edge feature image Ψ. 1 Ψ 2 Ψ 3 Ψ 4 Furthermore, by merging and fusing, multi-scale edge features δ are obtained. 2 δ 3 δ 4 Then, all edge features are convolved and merged to obtain multi-scale edge features, namely PMSE (pyramid multi-scale edge feature). Because multi-scale edge features δ 2 δ 3 δ 4 Since the dimensions of edge features differ, it is necessary to ensure that the dimensions of edge features are the same before performing convolutional merging. Specifically, upsampling can be used to allow smaller-scale edge features in a multi-scale edge feature model to be added and fused with larger-scale edge features. In a specific embodiment, the following method can be used to obtain this:

[0076] δ L =Ψ L +Ψ L-1

[0077] δ L-1 =Ψ L-1 +Ψ L-2

[0078] δ 2 =Ψ 2 +Ψ 1

[0079] In a specific embodiment, based on the aforementioned pyramid boundary module, two PMSE features can be obtained, corresponding to the image to be registered and the fixed image, respectively.

[0080] In the registration stage, the improved Voxelmorph rigid registration network (Voxelmorph6) is the basic rigid registration network. Using the image to be registered and the fixed image as input, a rigid registration result is obtained, namely the Wrap1 image (the first registration image), which is the global registration result. Simultaneously, another Voxelmorph6 network is trained using these two PMSE features to obtain another Wrap2 image (the second registration image), which is the local registration result. These two Voxelmorph6 networks have the same weights. Therefore, the overall network considers both global and local information, aiming to improve the performance of global rigid registration using local registration. Specifically, by calculating two loss functions, L1 and L2, the image to be registered is made closer to the fixed image, while the L3 loss function ensures that the registration effects of the local and global images are consistent. This allows the registration module to consider both local and global features simultaneously and utilize boundary features to improve the finer registration capability of the registration module. Here, L1 is the first registration image. With fixed image (I) f The mean squared error of ) is used as the first loss function, i.e. L2 is the second registration image. With fixed image (I) f The mean squared error of ) is used as the second loss function, i.e.

[0081] L3 is the first registered image. With the second registration image The mean squared error is used as the third loss function, i.e.

[0082] Finally, the Wrap1 image (the first registration image) is selected as the final registration output.

[0083] To further understand the registration method provided in the embodiments of this specification, the following will describe it in conjunction with specific embodiments. Taking CTP data as an example, a CTP dataset containing 393 subjects is constructed. In the first dataset, each subject has 14 CTP images, which is a time series of length 14. In the second dataset, each subject contains 18 CTP images, and the time series length is 18.

[0084] Furthermore, using the aforementioned dataset, a single-brain data generator and a random data generator were constructed. The single-brain data generator used the first CTP images acquired by all subjects as fixed images, and CTP images from other time points as images to be registered. Both the fixed and unregistered images came from the same subject. The random data generator randomly selected two images from all subject images, primarily for brain centerline extraction, a key technique in CTP analysis. The two generators each generated 5776 image pairs (one fixed image and one image to be registered), of which 3278 pairs were used as the training set, 820 pairs as the validation set, and 1687 pairs as the test set. To validate the pyramid boundary module, a comparative experiment was conducted based on the aforementioned dataset. The experimental results are shown in Table 1. The results indicate that Voxelmorph6 with the pyramid boundary module improved both MSE (mean squared error) and DICE similarity coefficient in both single-brain and random data registration, effectively demonstrating the role of the pyramid boundary module.

[0085] Table 1: Effectiveness Comparison Experiment of Boundary Modules in Pyramid Structures

[0086]

[0087]

[0088] The method provided in the embodiments of this specification can achieve global registration of the registration network based on local registration details, thereby improving the performance and accuracy of image registration.

[0089] The above describes in detail an image registration method based on unsupervised learning. Correspondingly, embodiments of this specification also provide an image registration apparatus based on unsupervised learning, such as... Figure 3 As shown. Figure 3 This is a schematic diagram of an image registration device based on unsupervised learning provided in an embodiment of this specification. The image registration device includes:

[0090] The acquisition module 301 acquires the three-dimensional image data to be processed, which includes the image data of the image to be registered and the image data of the fixed image.

[0091] The pyramid boundary module 303 extracts edge features from the three-dimensional image data to be processed to obtain a multi-scale edge feature image of the pyramid structure. The multi-scale edge feature image includes the multi-scale edge feature image of the image to be registered and the multi-scale edge feature image of the fixed image.

[0092] The first registration module 305 uses a preset rigid registration network to register the three-dimensional image data to be processed to obtain a first registered image, and uses the preset rigid registration network to register the multi-scale edge feature image to obtain a second registered image.

[0093] The second registration module 307 adjusts the first registration image based on the first registration image, the second registration image, and the fixed image to obtain a new first registration image as the registration image of the three-dimensional image data to be processed.

[0094] Furthermore, the step of extracting edge features from the three-dimensional image data to be processed to obtain a multi-scale edge feature image of the pyramid structure specifically includes:

[0095] A first convolution operation is performed on the three-dimensional image data to be processed to obtain the feature map of the three-dimensional image data to be processed.

[0096] Edge features are extracted based on the feature map of the three-dimensional image data to be processed to obtain an extracted edge feature image, wherein the extracted edge feature image includes edge feature images of at least two sizes.

[0097] The extracted edge feature images are fused by addition or convolution to obtain a fused edge feature image.

[0098] The fused edge feature images are merged through a second convolution operation to obtain a multi-scale edge feature image of a pyramid structure.

[0099] Further, the step of performing a first convolution operation on the three-dimensional image data to be processed to obtain a feature map of the three-dimensional image data to be processed specifically includes:

[0100] The three-dimensional image data to be processed is downsampled to obtain three-dimensional image data of different sizes;

[0101] The 3D image data of different sizes is processed by two convolution operations to obtain feature maps of each size corresponding to the 3D image data to be processed, and the feature maps of each size have the same dimension.

[0102] Furthermore, the method for edge feature extraction based on the feature map of the three-dimensional image data to be processed is as follows:

[0103] Ψ i =F i -avg(F i ), i = 1, 2, ..., L.

[0104] Where ψ represents the edge features at different scales, and F i Here, avg represents the feature maps of each size corresponding to the three-dimensional image data to be processed.

[0105] Furthermore, after extracting edge features from the feature map of the three-dimensional image data to be processed to obtain the extracted edge feature image, the process further includes the following steps:

[0106] The extracted edge feature images are upsampled so that all dimensions of the extracted edge feature images are the same.

[0107] Furthermore, the fused edge feature images are merged through a second convolution operation to obtain a multi-scale edge feature image of a pyramid structure, specifically including:

[0108] The fused edge feature images are merged using a 1×1 convolution operation to obtain a multi-scale edge feature image with a pyramid structure.

[0109] Furthermore, the preset rigid registration network is an improved Voxelmorph rigid registration network (Voxelmorph6).

[0110] Further, the step of adjusting the first registration image based on the first registration image, the second registration image, and the fixed image to obtain a new first registration image as the registration image for the three-dimensional image data to be processed specifically includes:

[0111] The loss function of the first registered image and the fixed image is used as the first loss function, the loss function of the second registered image and the fixed image is used as the second loss function, and the loss function of the first registered image and the second registered image is used as the third loss function.

[0112] Based on the first loss function and the second loss function, the image to be registered is registered with the fixed image. Based on the third loss function, the registration effect of the first registered image and the second registered image is made consistent. The first registered image is adjusted to obtain a new first registered image as the registered image of the three-dimensional image data to be processed.

[0113] This specification also provides an electronic device, including:

[0114] At least one processor; and,

[0115] A memory communicatively connected to the at least one processor; wherein,

[0116] The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to:

[0117] Acquire the 3D image data to be processed, which includes the image data of the image to be registered and the image data of the fixed image.

[0118] Edge features are extracted from the three-dimensional image data to be processed to obtain a multi-scale edge feature image of the pyramid structure. The multi-scale edge feature image includes the multi-scale edge feature image of the image to be registered and the multi-scale edge feature image of the fixed image.

[0119] The three-dimensional image data to be processed is registered using a preset rigid registration network to obtain a first registered image, and the multi-scale edge feature image is registered using the preset rigid registration network to obtain a second registered image.

[0120] Based on the first registration image, the second registration image, and the fixed image, the first registration image is adjusted to obtain a new first registration image as the registration image of the three-dimensional image data to be processed.

[0121] Furthermore, the step of extracting edge features from the three-dimensional image data to be processed to obtain a multi-scale edge feature image of the pyramid structure specifically includes:

[0122] A first convolution operation is performed on the three-dimensional image data to be processed to obtain the feature map of the three-dimensional image data to be processed.

[0123] Edge features are extracted based on the feature map of the three-dimensional image data to be processed to obtain an extracted edge feature image, wherein the extracted edge feature image includes edge feature images of at least two sizes.

[0124] The extracted edge feature images are fused by addition or convolution to obtain a fused edge feature image.

[0125] The fused edge feature images are merged through a second convolution operation to obtain a multi-scale edge feature image of a pyramid structure.

[0126] Further, the step of performing a first convolution operation on the three-dimensional image data to be processed to obtain a feature map of the three-dimensional image data to be processed specifically includes:

[0127] The three-dimensional image data to be processed is downsampled to obtain three-dimensional image data of different sizes;

[0128] The 3D image data of different sizes is processed by two convolution operations to obtain feature maps of each size corresponding to the 3D image data to be processed, and the feature maps of each size have the same dimension.

[0129] Furthermore, the method for edge feature extraction based on the feature map of the three-dimensional image data to be processed is as follows:

[0130] Ψ i =F i -avg(F i ), i = 1, 2, ..., L.

[0131] Where ψ represents the edge features at different scales, and F i Here, avg represents the feature maps of each size corresponding to the three-dimensional image data to be processed.

[0132] Furthermore, after extracting edge features from the feature map of the three-dimensional image data to be processed to obtain the extracted edge feature image, the process further includes the following steps:

[0133] The extracted edge feature images are upsampled so that all dimensions of the extracted edge feature images are the same.

[0134] Furthermore, the fused edge feature images are merged through a second convolution operation to obtain a multi-scale edge feature image of a pyramid structure, specifically including:

[0135] The fused edge feature images are merged using a 1×1 convolution operation to obtain a multi-scale edge feature image with a pyramid structure.

[0136] Furthermore, the preset rigid registration network is an improved Voxelmorph rigid registration network (Voxelmorph6).

[0137] Further, the step of adjusting the first registration image based on the first registration image, the second registration image, and the fixed image to obtain a new first registration image as the registration image for the three-dimensional image data to be processed specifically includes:

[0138] The loss function of the first registered image and the fixed image is used as the first loss function, the loss function of the second registered image and the fixed image is used as the second loss function, and the loss function of the first registered image and the second registered image is used as the third loss function.

[0139] Based on the first loss function and the second loss function, the image to be registered is registered with the fixed image. Based on the third loss function, the registration effect of the first registered image and the second registered image is made consistent. The first registered image is adjusted to obtain a new first registered image as the registered image of the three-dimensional image data to be processed.

[0140] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0141] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for apparatus, electronic devices, and non-volatile computer storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0142] The apparatus, electronic device, and non-volatile computer storage medium and method provided in the embodiments of this specification are corresponding. Therefore, the apparatus, electronic device, and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, electronic device, and non-volatile computer storage medium will not be repeated here.

[0143] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using a hardware physical module. For example, a Programmable Logic Device (PLD) (e.g., a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program a digital system themselves to "integrate" it onto a PLD, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0144] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91 SAM, Microchip PIC1 8F26K20, and Silicon Labs C805 1F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0145] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0146] For ease of description, the above apparatus is described by dividing it into various functional units. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in one or more software and / or hardware.

[0147] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0148] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data-optimized device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data-optimized device, produce a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0149] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data optimization device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0150] These computer program instructions can also be loaded onto a computer or other programmable data optimization device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0151] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0152] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0153] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0154] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0155] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside on local and remote computer storage media, including storage devices.

[0156] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0157] The above description is merely an embodiment of this specification and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application.

Claims

1. An image registration method based on unsupervised learning, characterized in that, The method includes: Acquire the 3D image data to be processed, which includes the image data of the image to be registered and the image data of the fixed image. The method of extracting edge features from the 3D image data to be processed to obtain a multi-scale edge feature image with a pyramid structure specifically includes: performing a first convolution operation on the 3D image data to be processed to obtain a feature map of the 3D image data to be processed; extracting edge features based on the feature map of the 3D image data to be processed to obtain an extracted edge feature image, wherein the extracted edge feature image includes edge feature images of at least two sizes; fusing the extracted edge feature images by addition or convolution to obtain a fused edge feature image; and merging the fused edge feature images by a second convolution operation to obtain a multi-scale edge feature image with a pyramid structure; wherein the multi-scale edge feature image includes the multi-scale edge feature image of the image to be registered and the multi-scale edge feature image of the fixed image. The three-dimensional image data to be processed is registered using a preset rigid registration network to obtain a first registered image. The multi-scale edge feature image is registered using the preset rigid registration network to obtain a second registered image. The preset rigid registration network is an improved Voxelmorph6. Based on the first registered image, the second registered image, and the fixed image, the first registered image is adjusted to obtain a new first registered image as the registered image of the three-dimensional image data to be processed. Specifically, this includes: using the loss function of the first registered image and the fixed image as a first loss function, the loss function of the second registered image and the fixed image as a second loss function, and the loss function of the first registered image and the second registered image as a third loss function; based on the first loss function and the second loss function, so that the image to be registered is registered with the fixed image; and based on the third loss function, so that the registration effect of the first registered image and the second registered image is consistent, the first registered image is adjusted to obtain a new first registered image as the registered image of the three-dimensional image data to be processed.

2. The image registration method based on unsupervised learning as described in claim 1, characterized in that, The step of performing a first convolution operation on the three-dimensional image data to be processed to obtain a feature map of the three-dimensional image data to be processed specifically includes: The three-dimensional image data to be processed is downsampled to obtain three-dimensional image data of different sizes; The 3D image data of different sizes is processed by two convolution operations to obtain feature maps of each size corresponding to the 3D image data to be processed, and the feature maps of each size have the same dimension.

3. The image registration method based on unsupervised learning as described in claim 1, characterized in that, The method for edge feature extraction based on the feature map of the three-dimensional image data to be processed is as follows: Where Ψi represents the edge features at different scales, Fi represents the feature maps of each size corresponding to the three-dimensional image data to be processed, avg represents the global average value, i represents the index, and L represents the total number of pyramid levels.

4. The image registration method based on unsupervised learning as described in claim 1, characterized in that, After extracting edge features from the feature map of the three-dimensional image data to be processed to obtain the extracted edge feature image, the process further includes the following steps: The extracted edge feature images are upsampled so that all dimensions of the extracted edge feature images are the same.

5. The image registration method based on unsupervised learning as described in claim 1, characterized in that, The fused edge feature images are then merged using a second convolution operation to obtain a multi-scale edge feature image of a pyramid structure, specifically including: The fused edge feature images are merged using a 1×1 convolution operation to obtain a multi-scale edge feature image with a pyramid structure.

6. An image registration device based on unsupervised learning, characterized in that, The device includes: The acquisition module acquires the 3D image data to be processed, which includes the image data of the image to be registered and the image data of the fixed image. The pyramid boundary module extracts edge features from the 3D image data to be processed, obtaining a multi-scale edge feature image of the pyramid structure. Specifically, this includes: performing a first convolution operation on the 3D image data to be processed to obtain a feature map of the 3D image data; extracting edge features based on the feature map of the 3D image data to obtain an extracted edge feature image, wherein the extracted edge feature image includes edge feature images of at least two sizes; fusing the extracted edge feature images using methods such as addition and fusion or convolution to obtain a fused edge feature image; and merging the fused edge feature images through a second convolution operation to obtain a multi-scale edge feature image of the pyramid structure. The multi-scale edge feature image includes the multi-scale edge feature image of the image to be registered and the multi-scale edge feature image of the fixed image. The first registration module uses a preset rigid registration network to register the three-dimensional image data to be processed to obtain a first registered image, and uses the preset rigid registration network to register the multi-scale edge feature image to obtain a second registered image. The preset rigid registration network is an improved Voxelmorph6. The second registration module, based on the first registration image, the second registration image, and the fixed image, adjusts the first registration image to obtain a new first registration image as the registration image for the three-dimensional image data to be processed. Specifically, this includes: using the loss function of the first registration image and the fixed image as a first loss function, the loss function of the second registration image and the fixed image as a second loss function, and the loss function of the first registration image and the second registration image as a third loss function; based on the first loss function and the second loss function, to ensure that the image to be registered is registered with the fixed image, and based on the third loss function, to ensure that the registration effect of the first registration image and the second registration image is consistent, adjusting the first registration image to obtain a new first registration image as the registration image for the three-dimensional image data to be processed.

7. An electronic device, comprising: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to: Acquire the 3D image data to be processed, which includes the image data of the image to be registered and the image data of the fixed image. The method of extracting edge features from the 3D image data to be processed to obtain a multi-scale edge feature image with a pyramid structure specifically includes: performing a first convolution operation on the 3D image data to be processed to obtain a feature map of the 3D image data to be processed; extracting edge features based on the feature map of the 3D image data to be processed to obtain an extracted edge feature image, wherein the extracted edge feature image includes edge feature images of at least two sizes; fusing the extracted edge feature images by addition or convolution to obtain a fused edge feature image; and merging the fused edge feature images by a second convolution operation to obtain a multi-scale edge feature image with a pyramid structure; wherein the multi-scale edge feature image includes the multi-scale edge feature image of the image to be registered and the multi-scale edge feature image of the fixed image. The three-dimensional image data to be processed is registered using a preset rigid registration network to obtain a first registered image. The multi-scale edge feature image is registered using the preset rigid registration network to obtain a second registered image. The preset rigid registration network is an improved Voxelmorph6. Based on the first registered image, the second registered image, and the fixed image, the first registered image is adjusted to obtain a new first registered image as the registered image of the three-dimensional image data to be processed. Specifically, this includes: using the loss function of the first registered image and the fixed image as a first loss function, the loss function of the second registered image and the fixed image as a second loss function, and the loss function of the first registered image and the second registered image as a third loss function; based on the first loss function and the second loss function, so that the image to be registered is registered with the fixed image; and based on the third loss function, so that the registration effect of the first registered image and the second registered image is consistent, the first registered image is adjusted to obtain a new first registered image as the registered image of the three-dimensional image data to be processed.

Images