Image registration method, device, equipment and storage medium

Abstract

Embodiments of the present disclosure provide an image registration method, apparatus, device, and computer-readable storage medium. The method provided by the embodiments of the present disclosure extracts feature maps of the image pair to be registered at multiple scales, and converts the displacement vector of each voxel in the moving image of the image pair to be registered at each scale into a combination of multiple displacement basis vectors determined based on the feature vector corresponding to the voxel in the corresponding feature map, so as to learn the displacement field from the paired feature maps, and generate the final displacement field based on the displacement fields at all scales, so as to make full use of the image feature information at different scales to achieve efficient and accurate image registration. The method of the embodiments of the present disclosure can learn the displacement field from coarse to fine based on different scales, so as to make full use of the potential information of different representation subspaces by fusing the displacement fields at all scales without introducing dense continuous network calculations.

Description

Technical Field

[0001] The present disclosure relates to the fields of artificial intelligence and image processing, and more specifically, to an image registration method, apparatus, device and storage medium. Background Art

[0002] Image registration, as the basis of digital image processing, is a process that uses different images as a premise to perform spatial transformation (such as translation, rotation or scale transformation) on two or more images of the same area acquired at different times, of different types or by different imaging devices to make them consistent. Its purpose is to find the optimal spatial transformation (i.e., the optimal deformation field or displacement field). Image registration technology can realize functions such as image transformation and feature point extraction and matching, and has important applications in the field of medical image processing. The purpose of medical image registration is to find an optimal spatial transformation that best aligns the basic anatomical structure, thereby helping doctors to more easily observe and analyze the growth and changes of organs and lesions from multiple angles, making clinical diagnosis results more reliable and accurate.

[0003] Traditional non-learning-based image registration methods define the registration task as an optimization problem with various constraints to combine desired properties such as inverse consistency, symmetry, and topological order preservation. However, these methods require iterative intensive calculations for each pair of registered images, which makes optimization slow and difficult. In recent years, more and more studies have begun to use deep learning methods to deal with image registration problems, achieving results that are superior to traditional algorithms. However, existing registration methods based on unsupervised learning cannot achieve the desired registration performance due to the limited ability of convolutional neural networks to capture spatial relationships.

[0004] Therefore, an efficient and accurate image registration method is needed. Summary of the invention

[0005] In order to solve the above problems, the present disclosure converts the displacement vector at each voxel in the image into a displacement field that is a combination of multiple displacement basis vectors determined based on image features, so as to more effectively learn the spatial relationship from the image features, thereby achieving accurate image registration.

[0006] Embodiments of the present disclosure provide an image registration method, apparatus, device, and computer-readable storage medium.

[0007] An embodiment of the present disclosure provides an image registration method, comprising: acquiring a fixed image and a moving image to be registered; obtaining respective feature maps of the fixed image and the moving image at multiple scales; for each scale in the multiple scales, determining a displacement field at the scale from respective feature maps of the fixed image and the moving image at the scale, the displacement field being used to indicate a mapping from the moving image to the fixed image at the scale; and generating a final displacement field based on the multiple displacement fields determined at the multiple scales, and applying the final displacement field to the moving image, so that the moving image to which the final displacement field is applied is registered with the fixed image; wherein, in the displacement field, the displacement vector of each voxel in the moving image to the corresponding voxel in the fixed image is represented as a combination of multiple displacement basis vectors, wherein the multiple displacement basis vectors are obtained based on feature vectors corresponding to the voxels in the feature map of the moving image at the scale.

[0008] An embodiment of the present disclosure provides an image registration device, comprising: an image acquisition module, configured to acquire a fixed image and a moving image to be registered; a feature map extraction module, configured to obtain respective feature maps of the fixed image and the moving image at multiple scales; a displacement field determination module, configured to determine, for each scale in the multiple scales, a displacement field at the scale from respective feature maps of the fixed image and the moving image at the scale, the displacement field being used to indicate a mapping from the moving image to the fixed image at the scale; and a displacement field synthesis module, configured to generate a final displacement field based on a plurality of displacement fields determined at the multiple scales, and apply the final displacement field to the moving image, so that the moving image to which the final displacement field is applied is registered with the fixed image; wherein, in the displacement field, a displacement vector from each voxel in the moving image to a corresponding voxel in the fixed image is represented as a combination of a plurality of displacement basis vectors, wherein the plurality of displacement basis vectors are obtained based on feature vectors corresponding to the voxels in the feature map of the moving image at the scale.

[0009] An embodiment of the present disclosure provides an image registration device, comprising: one or more processors; and one or more memories, wherein a computer executable program is stored in the one or more memories, and when the computer executable program is executed by the processor, the image registration method as described above is performed.

[0010] An embodiment of the present disclosure provides a computer-readable storage medium having computer-executable instructions stored thereon. When the instructions are executed by a processor, the instructions are used to implement the image registration method as described above.

[0011] The embodiments of the present disclosure provide a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the image registration method according to the embodiments of the present disclosure.

[0012] Compared with the traditional image registration method, the method provided by the embodiment of the present disclosure greatly improves the registration speed and registration accuracy by adopting an image registration model based on unsupervised neural network learning, while alleviating the problem of scarce annotation information of image data sets. In addition, compared with the existing registration method based on unsupervised learning, the method provided by the embodiment of the present disclosure improves the interpretability of the registration network and reduces the amount of calculation required for registration by promoting the mapping from image representation to spatial relationship instead of directly converting image representation into spatial correspondence.

[0013] The method provided by the embodiment of the present disclosure extracts feature maps of the image pair to be registered at multiple scales, converts the displacement vector of each voxel in the moving image of the image pair to be registered at each scale into a combination of multiple displacement basis vectors determined based on the feature vector corresponding to the voxel in the corresponding feature map, so as to learn the displacement field from the paired feature maps, and generate the final displacement field based on the displacement fields at all scales, so as to make full use of the image feature information at different scales to achieve efficient and accurate image registration. The method of the embodiment of the present disclosure can learn the displacement field from coarse to fine based on different scales, so as to make full use of the potential information of different representation subspaces by fusing the displacement fields at all scales without introducing dense continuous network calculations. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some exemplary embodiments of the present disclosure, and a person of ordinary skill in the art can obtain other drawings based on these drawings without creative work.

[0015] Figure 1 is a schematic diagram showing a scenario of processing a registration request from a user terminal according to an embodiment of the present disclosure;

[0016] Figure 2A is a flowchart illustrating an image registration method according to an embodiment of the present disclosure;

[0017] Figure 2B is a schematic diagram showing a multi-scale registration network of an image registration method according to an embodiment of the present disclosure;

[0018] Figure 2C is a schematic diagram illustrating the application of a multi-scale registration network according to an embodiment of the present disclosure;

[0019] Figure 3A is a flow chart showing determination of a displacement field from a feature map of a pair of images to be registered according to an embodiment of the present disclosure;

[0020] Figure 3B is a schematic diagram showing determination of a displacement field from a feature map of a pair of images to be registered according to an embodiment of the present disclosure;

[0021] Figure 3C is a schematic flow chart showing a method of determining a displacement field from a feature map of a pair of images to be registered according to an embodiment of the present disclosure;

[0022] Figure 4A is a schematic diagram showing cascade fusion of multiple displacement fields determined at multiple scales according to an embodiment of the present disclosure;

[0023] Figure 4B is a flow chart showing each level of fusion in cascade fusion according to an embodiment of the present disclosure;

[0024] Figure 4C is a schematic diagram showing the first level fusion in the cascade fusion according to an embodiment of the present disclosure;

[0025] Figure 5 is a schematic diagram showing image registration results under different methods according to an embodiment of the present disclosure and visualization results of deformation fields under different scales;

[0026] Figure 6 is a schematic diagram showing the application of the image registration method according to an embodiment of the present disclosure;

[0027] Figure 7 is a schematic diagram showing an image registration device according to an embodiment of the present disclosure;

[0028] Figure 8 A schematic diagram of an image registration device according to an embodiment of the present disclosure is shown;

[0029] Fig. 9 A schematic diagram illustrating the architecture of an exemplary computing device according to an embodiment of the present disclosure; and

[0030] Fig.10 A schematic diagram of a storage medium according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the present disclosure more obvious, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0032] In this specification and the accompanying drawings, substantially the same or similar steps and elements are represented by the same or similar reference numerals, and repeated descriptions of these steps and elements will be omitted. At the same time, in the description of the present disclosure, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance or ranking.

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present disclosure pertains. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0034] To facilitate description of the present disclosure, concepts related to the present disclosure are introduced below.

[0035] The image registration method disclosed in the present invention may be based on artificial intelligence (AI). Artificial intelligence is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. For example, for an image registration method based on artificial intelligence, it can achieve image registration by extracting potential features from the image pairs to be registered and determining the displacement field of the image based on these features in a manner similar to how humans recognize the association between the image pairs to be registered by the naked eye and align the image pairs to be registered based on their association. Artificial intelligence enables the image registration method disclosed in the present invention to have the function of quickly and accurately extracting various potential features in the image for use in establishing a spatial mapping between voxel pairs of the image pairs to be registered.

[0036] The image registration method disclosed in the present invention can be based on computer vision (CV) technology. Computer vision is a science that studies how to make machines "see". To put it more concretely, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Therefore, based on computer vision technology, the image registration method disclosed in the present invention can extract feature information from the input image, and achieve accurate image registration by effectively learning spatial relationships from image features. For example, the image registration method disclosed in the present invention can be based on deep learning technology. Deep learning technology has outstanding performance in the field of medical image analysis (such as organ or tumor segmentation, lesion detection and disease diagnosis, etc.) with its powerful learning ability and feature extraction ability. In recent years, some researchers have carried out medical image registration research based on deep learning. These studies generally include: combining deep networks with traditional registration algorithms, direct registration based on supervised learning models, and direct registration based on unsupervised learning models. For image registration methods that combine deep networks with traditional registration algorithms, although these deep networks have improved the efficiency of registration to a certain extent, they are still within the iterative optimization framework of traditional image registration. In addition, the feature extraction process of the deep network is separated from the image registration task, so it cannot be guaranteed that the features extracted by the network are the most descriptive for the registration problem. In the registration model research based on supervised learning, the parameters of the registration network are trained by using the difference loss between the gold standard deformation and the predicted deformation as the objective function, but the performance of the supervised registration model is usually limited by the accuracy of the supervised label information (i.e., the gold standard deformation field), and for clinical medical images, it is very difficult and time-consuming to obtain the gold standard deformation field in the training data. The direct registration method based on the unsupervised learning model uses the deformation field output by the registration network to perform spatial transformation on the image to be registered to generate a deformed image, and then trains the registration network by maximizing the similarity between the deformed image and the reference image, avoiding the need for a gold standard deformation field and realizing an end-to-end unsupervised registration model. Therefore, in the embodiments of the present disclosure, image registration can be realized by training an unsupervised registration network based on an unsupervised learning method.

[0037] Optionally, the image registration method disclosed in the present invention can be based on a neural network. A neural network (such as a feedforward network) is a mechanism for learning to simulate the hierarchical processing of information by the human brain visual system. By constructing a model with multiple hidden layers, different weights are assigned to the information in the input image, and multi-level and multi-angle image features are automatically extracted to better understand the input image. A convolutional neural network (CNN) is a type of feedforward neural network that includes convolution calculations and has a deep structure. It is one of the representative algorithms of deep learning. A convolutional neural network has representation learning capabilities and can perform shift-invariant classification on input information according to its hierarchical structure. The image registration method disclosed in the present invention can learn all the parameters in the registration network based on neural network training for subsequent direct use of the registration network without optimizing the registration metric function each time the registration is performed.

[0038] Optionally, the image registration method disclosed in the present invention can also be based on the attention mechanism. The essence of the attention mechanism originates from the human visual mechanism and belongs to the brain signal processing mechanism unique to human vision, that is, human attention. The attention mechanism only averages the learned features and has certain limitations. The multi-head attention mechanism is a special variation of the attention mechanism. It uses a function to calculate similarity and comprehensively learns features from different aspects of the model through multiple transformations. It is equivalent to multiple single attentions learning features of different subspaces and then splicing these features together through a linear transformation combination. Multi-head attention uses multiple queries to calculate and select multiple information from the input information in parallel, where each attention focuses on different parts of the input information. In the embodiment of the present invention, the multi-head attention mechanism can be used to help the registration model assign different weights to each input feature and extract more critical information, thereby obtaining multiple displacement basis vectors and their weights used to describe the spatial transformation relationship of the image from the feature vector of the input image, and automatically learning the hidden relationship between the features of these images by fully considering the association between the images to be registered, so as to improve the accuracy of image registration.

[0039] In summary, the solutions provided by the embodiments of the present disclosure involve technologies such as artificial intelligence, unsupervised image registration, and multi-head attention mechanism. The embodiments of the present disclosure will be further described below in conjunction with the accompanying drawings.

[0040] Figure 1 It is a schematic diagram of a scenario showing processing of a registration request from a user terminal according to an embodiment of the present disclosure.

[0041] exist Figure 1 In the application, a user can initiate an image registration request through his user terminal, for example, by uploading a pair of images to be registered through a specific interface on his user terminal. Then the user terminal can transmit these image data to the server of the application through the network (or directly).

[0042] Optionally, the user terminal may specifically include a smart phone, a tablet computer, a laptop, a vehicle-mounted terminal, a wearable device, and the like. The user terminal may also be a client that installs a browser or various applications (including system applications and third-party applications). The network may be an Internet of Things based on the Internet and / or telecommunications network, which may be a wired network or a wireless network, for example, it may be an electronic network that can realize information exchange functions such as a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a cellular data communication network, etc. The server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0043] like Figure 1 As shown, the server can perform image registration processing in real time based on the received user data (e.g., data of the image pair to be registered). Subsequently, the server can return the processed image registration results (e.g., the registered image pair and corresponding result data) to the user terminal through the network and display them as a response to the user's registration request.

[0044] Nowadays, image registration is a key and difficult task in the field of image processing research. It mainly studies the alignment and fusion algorithms of images obtained under different conditions for the same object. In daily life, images obtained for the same object often come from different acquisition devices, taken at different times, and from different perspectives. Therefore, these images need to be registered. Specifically, for two paired images in a set of image data, registration is used to find the spatial mapping relationship between the two images, so as to match the points at the same position in the two images one by one, and then facilitate subsequent recognition, segmentation, fusion and other operations. In various fields such as computer vision, medical image processing, and material mechanics, image registration algorithms have wide applications and far-reaching impacts. For example, in recent years, medical imaging technology has developed rapidly, gradually developing from static to dynamic imaging, from morphology to function, and from plane images to three-dimensional stereo images. Because human organs are easily deformed and human anatomical structures grow and change at different times, doctors often need to observe different medical images by naked eye for registration, which is difficult for doctors who lack spatial imagination and subjective medical experience. The purpose of medical image registration is to find an optimal spatial transformation and best align the basic anatomical structure, so as to help doctors more easily observe and analyze the growth and changes of organs and lesions from multiple angles, making clinical diagnosis more reliable and accurate. Medical image registration converts the coordinates of one image (moving image) to another image (fixed image) so that the corresponding positions of the two images match and obtain a registered image.

[0045] According to the different methods that the registration algorithm relies on, the registration method can be divided into traditional registration method and learning-based registration method. The traditional non-learning-based registration method is based on mathematical iterative optimization. It first defines a similarity index (for example, L2 norm), defines the registration task as an optimization problem with various constraints, and combines the desired characteristics (such as inverse consistency, symmetry, and topological order preservation). By continuously iteratively optimizing the parametric transformation or non-parametric transformation, the similarity between the registered moving image and the fixed image is maximized. However, such a registration method requires iterative intensive calculations for each new registration image pair, and its optimization speed is slow and difficult.

[0046] Compared with traditional registration methods, medical image registration methods based on deep learning have great advantages and potential. Therefore, more and more researchers are studying this method, and many related works have been published in recent years. Learning-based registration methods refer to registration methods trained by neural networks. In this method, parameters are shared. First, a large amount of data is used to train the model, and then the trained model is used to register new images. With the increasing application of deep learning in the research of medical image analysis, it has achieved quite good results in organ segmentation, lesion detection and classification tasks. More and more studies have begun to use deep learning methods to deal with image registration problems and have achieved results that are better than traditional algorithms. Among them, unsupervised neural networks greatly improve the registration speed and accuracy in the process of image registration, and at the same time alleviate the problem of scarce annotation information in image data sets. In recent registration methods based on unsupervised learning, the unsupervised medical image registration model (VoxelMorph) is usually used as a benchmark, and the semantic segmentation network (U-Net) architecture is used as the main body to learn the dense nonlinear correspondence between input image pairs. However, these registration methods based on unsupervised learning cannot achieve the desired registration performance due to the limited ability of convolutional neural networks to capture spatial relationships. Therefore, there are also registration methods that propose to use attention mechanisms to better represent spatial relationships, but because they focus too much on the long-range dependencies between images, they ignore the locality of registration, that is, in the registration task, it is necessary to find corresponding voxels in local areas.

[0047] Based on this, the present disclosure provides an image registration method, which converts the displacement vector at each voxel in the image into a displacement field that is a combination of multiple displacement basis vectors determined based on image features, so as to more effectively learn the spatial relationship from the image features, thereby achieving accurate image registration.

[0048] Compared with the traditional image registration method, the method provided by the embodiment of the present disclosure greatly improves the registration speed and registration accuracy by adopting an image registration model based on unsupervised neural network learning, while alleviating the problem of scarce annotation information of image data sets. In addition, compared with the existing registration method based on unsupervised learning, the method provided by the embodiment of the present disclosure improves the interpretability of the registration network and reduces the amount of calculation required for registration by promoting the mapping from image representation to spatial relationship instead of directly converting image representation into spatial correspondence.

[0049] The method provided by the embodiment of the present disclosure extracts feature maps of the image pair to be registered at multiple scales, converts the displacement vector of each voxel in the moving image of the image pair to be registered at each scale into a combination of multiple displacement basis vectors determined based on the feature vector corresponding to the voxel in the corresponding feature map, so as to learn the displacement field from the paired feature maps, and generate the final displacement field based on the displacement fields at all scales, so as to make full use of the image feature information at different scales to achieve efficient and accurate image registration. The method of the embodiment of the present disclosure can learn the displacement field from coarse to fine based on different scales, so as to make full use of the potential information of different representation subspaces by fusing the displacement fields at all scales without introducing dense continuous network calculations.

[0050] Figure 2A is a flow chart illustrating an image registration method 200 according to an embodiment of the present disclosure. Figure 2B is a schematic diagram showing a multi-scale registration network of an image registration method according to an embodiment of the present disclosure. Figure 2C is a schematic diagram illustrating the application of a multi-scale registration network according to an embodiment of the present disclosure.

[0051] In step 201 , a fixed image and a moving image to be registered may be acquired.

[0052] As mentioned above, the purpose of image registration is to spatially transform the coordinates of each voxel (corresponding to a three-dimensional image, for example, alternatively, a pixel in a two-dimensional image) in the moving image in the image pair to be registered so as to align with the coordinates of the corresponding voxel in the fixed image, that is, to determine the displacement vector of each voxel in the moving image, thereby determining the displacement field of the moving image. Voxel refers to a volume element (Volume Pixel), which is conceptually similar to the smallest unit of a two-dimensional image space, a pixel, and is the smallest unit of a three-dimensional image space. Voxels are usually used in fields such as three-dimensional imaging and medical imaging, while pixels are used in image data of two-dimensional computer images.

[0053] For the above-mentioned scenarios such as medical image registration, considering the development of imaging technology from planar images to three-dimensional stereo images, the processing of images also needs to adapt to the obtained three-dimensional images. Therefore, in the following text of this disclosure, the image registration processing scenario for three-dimensional images is mainly described, but this does not mean that the scope of application of the image registration method of this disclosure is limited. The image registration method of this disclosure can also be applied to other higher or lower dimensional image processing. The following operations on three-dimensional images are only used as examples and not as limitations.

[0054] Optionally, in an embodiment of the present disclosure, for a three-dimensional medical image registration scenario (where the moving image and the fixed image are three-dimensional medical images), the acquired pair of images to be registered may be represented as: Where M is the moving image, and F is the fixed image, D, W and H represent the depth, width and height of the moving image and the fixed image respectively. Therefore, the image registration process can be expressed as: for each voxel in the moving image M, find the most similar voxel in the fixed image F, and the mapping function between the image pairs to be registered can usually be expressed by the deformation field φ formed by the displacement vector between each pair of voxels.

[0055] Therefore, the image registration method disclosed in the present invention aims to determine the deformation field of the moving image M By using a spatial transformation network, the moving image M is mapped to the fixed image F according to the determined deformation field φ, so that the deformed moving image It can be registered to the fixed image F, where the dimension 3 of the deformation field φ represents the three-dimensional space dimension, which corresponds to the three-dimensional space transformation of each voxel in the moving image M. Specifically, the deformation field φ can be expressed as Id+u, where Id represents the identity transformation for the moving image M, and u can represent the displacement field, which is composed of the displacement vectors between each pair of voxels in the image pair {M, F} to be registered. For example, for a two-dimensional image of size [W, H], the displacement field size is [W, H, 2], and for a three-dimensional image of size [D, W, H], the displacement field size is [D, W, H, 3], where the displacement field can represent the displacement of each voxel / pixel in each direction (e.g., x, y, z axis). The spatial transformation network can generate a normalized sampling network based on the displacement field, and the moving image is sampled by using the network to obtain the registered moving image.

[0056] In step 202, feature maps of the fixed image and the moving image at multiple scales may be obtained.

[0057] Considering that for the same image, different image information can be obtained from its feature maps at different scales. According to an embodiment of the present disclosure, obtaining the feature maps of the fixed image and the moving image at multiple scales may include: using an encoder based on a convolutional neural network, extracting feature maps at different scales through different convolution blocks in the encoder, so as to obtain the feature maps of the fixed image and the moving image at multiple scales.

[0058] Convolutional neural networks extract target features through layer-by-layer abstraction. The shallow network feature map has high resolution, small receptive field, strong geometric information representation ability, and weak semantic representation ability, which is suitable for processing small targets; the deep network has a large receptive field and strong semantic information representation ability, but the feature map has low resolution and weak geometric information representation ability, which is suitable for processing large targets. The low-level detail feature map captures rich spatial information and highlights the boundaries of organs; while the high-level semantic feature map contains position information and locates the location of the organ. Therefore, the moving image M and the fixed image F can be sent to the same encoder (shared weights) based on the convolutional neural network for feature extraction, so that the encoder can extract a sequence of intermediate feature maps at different scales through different convolution blocks. That is, feature maps at multiple scales are obtained. For example, Figure 2B The case of four scales (the number of scales L = 4) is shown in FIG, where B represents the number of image pairs to be registered in the batch, such as Figure 2B As shown in the figure, the encoder can extract feature maps at four scales based on the moving image M and the fixed image F through different convolution blocks (the number of convolution kernels is C1, C2, C3 and C4 respectively) And for a fixed image F, Among them, for the same scale l, the dimensions of the feature maps of the moving image and the fixed image are the same. For example, for the scale l = 3, and The dimensions are B*D3*W3*H3*C3.

[0059] It should be understood that the present disclosure only uses the case of four scales as an example to describe the image registration process, but this is not a limitation on the number of scales. The image registration method of the present disclosure can also use other numbers of scales to achieve image registration.

[0060] According to an embodiment of the present disclosure, the size of the feature map of the fixed image or the moving image at a specific scale may be the size of the fixed image or the moving image reduced in accordance with a corresponding ratio. Optionally, in order to increase the receptive field, the size of the feature map at scale l may be reduced in proportion to 1 / 2 of the size of the original image. l For example, for the above case of L=4, the size of the feature map of the moving image at scale l=3 can be 1 / 8 of the size of the original moving image, that is, D3*W3*H3=(D*W*H) / 8. Of course, the size reduction of the feature map can also be based on other methods. The above-mentioned proportional reduction method is only used as an example and not a limitation in this disclosure.

[0061] Therefore, multiple displacement fields can be learned from paired feature maps of fixed and moving images at multiple scales. And the displacement fields learned at all scales are fused to combine the characteristics of low-level detail feature maps and high-level semantic feature maps to obtain a more accurate displacement field.

[0062] In step 203, for each scale of the multiple scales, a displacement field at the scale can be determined from respective feature maps of the fixed image and the moving image at the scale, wherein the displacement field is used to indicate a mapping from the moving image to the fixed image at the scale.

[0063] As described above, a displacement field may be determined for a pair of feature maps at each scale, wherein the displacement field is composed of displacement vectors between pairs of voxels between the pair of images to be registered. According to an embodiment of the present disclosure, in the displacement field, the displacement vector from each voxel in the moving image to the corresponding voxel in the fixed image may be represented as a combination of multiple displacement basis vectors, wherein the multiple displacement basis vectors may be obtained based on feature vectors corresponding to the voxels in the feature map of the moving image at the scale.

[0064] Different from the method of directly converting image representation into spatial correspondence, in order to promote the mapping from image representation to spatial relationship and improve the interpretability of spatial transformation in the image registration network, in the image registration method disclosed in the present invention, for example Figure 2B The displacement field determination part shown in the figure decomposes the displacement vector between two voxels into a combination representation of multiple displacement basis vectors according to linear algebra theory, thereby avoiding complex and poorly interpretable spatial correspondence direct conversion. According to an embodiment of the present disclosure, determining the displacement field from the feature maps of the fixed image and the moving image at the scale may include: Figure 3A Steps shown.

[0065] Figure 3A Detailed description is a flow chart showing a method for determining a displacement field from a feature map of a pair of images to be registered according to an embodiment of the present disclosure. Figure 3B is a schematic diagram showing how to determine a displacement field from a feature map of a pair of images to be registered according to an embodiment of the present disclosure. Figure 3C It is a schematic flowchart showing the determination of a displacement field from a feature map of a pair of images to be registered according to an embodiment of the present disclosure.

[0066] like Figure 3A As shown, in step 2031, for each feature vector in the feature map of the moving image at the scale, a plurality of displacement basis vectors may be generated based on the feature vector.

[0067] Optionally, for the feature map of the moving image and the feature map of the fixed image at the same scale, the combination of the above multiple displacement basis vectors can be expressed by Figure 3BThe two independent branches of the multi-head attention mechanism are shown to be obtained. Specifically, according to an embodiment of the present disclosure, the multiple displacement basis vectors can be generated based on the feature vector using K attention heads according to the multi-head attention mechanism, wherein N displacement basis vectors can be generated using each attention head, wherein K and N can be positive integers.

[0068] like Figure 3B As shown, using each attention head to generate N displacement basis vectors can correspond to Figure 3B The upper branch in the figure operates on the feature map of the moving image. In this branch, linear projection can be used to transform each feature vector in the feature map of the moving image into N (for example, Figure 3B 5) displacement basis vectors, each of which can be composed of three elements, which can correspond to displacements in three spatial directions, namely, displacements on the x, y, and z axes. Therefore, in order to obtain information from different representation subspaces, the image registration method disclosed in the present invention is based on a multi-head attention mechanism, using K attention heads for each feature vector, and the number of displacement basis vectors for each attention head is N, thereby generating K×N displacement basis vectors for each voxel. Specifically, for the feature map of the moving image M at scale l It can be regarded as a C3-dimensional feature vector including B*D3*W3*H3. For each feature vector, in each attention head, a fully connected layer outputs an N×3-dimensional vector, which can be regarded as N three-dimensional displacement basis vectors. Therefore, through K attention heads, K×N displacement basis vectors can be extracted for each feature vector. Among them, this branch only uses the feature map of the moving image to generate the displacement basis vector, which is sufficient to extract the most representative displacement basis vector to represent the most likely deformation direction of each voxel.

[0069] In step 2032, a weight of each displacement basis vector in the plurality of displacement basis vectors may be determined for the feature vector based on respective feature maps of the fixed image and the moving image at the scale.

[0070] like Figure 3B As shown, the weight of each displacement basis vector in the plurality of displacement basis vectors may correspond to Figure 3B The lower half of the branch in the K attention heads operates on the feature maps of the fixed image and the moving image. According to an embodiment of the present disclosure, for each of the K attention heads, the weights of the N displacement basis vectors generated by the attention head are determined by the attention head for the feature vector based on the concatenation of the feature maps of the fixed image and the moving image at the scale.

[0071] Optionally, in this branch, by splicing the moving image and the fixed image (shown as C in the figure) feature map Linear projection is performed to learn the attention weight of each of the N displacement basis vectors corresponding to each attention head. Specifically, in this branch, the similarity between the voxels of the moving image and the fixed image can be learned through a linear layer and a normalized exponential function (softmax function), thereby determining the attention weight of the N displacement basis vectors (for example, Figure 3B In , the attention weights corresponding to the above five displacement basis vectors are 0.2, 0.2, 0.4, 0.1 and 0.1 respectively).

[0072] It should be noted that both linear projections are performed independently for each pair of feature vectors, resulting in fewer parameters than standard CNN or Transformer blocks. In addition, since the potential information of the nearby voxels of a voxel has been integrated into the feature vector corresponding to the voxel through the CNN-based encoder, the similarity measure is no longer limited to voxels at the same position, and the receptive field expands as the resolution of the feature map decreases.

[0073] Therefore, based on the above two independent branches, the displacement vector of the voxel corresponding to the feature vector can be determined. In step 2033, for each feature vector in the feature map of the moving image at the scale, the displacement field at the scale can be determined based on a plurality of displacement basis vectors corresponding to the feature vector and their weights.

[0074] By determining the direction in which each voxel is most likely to deform based on the feature vector in the moving image, and determining the relative displacement in each direction in which deformation is most likely to occur based on the similarity between the moving image and the fixed image, the displacement vector of the corresponding voxel can be determined based on these directions and the relative displacement in these directions.

[0075] According to an embodiment of the present disclosure, determining the displacement field at the scale based on multiple displacement basis vectors corresponding to the feature vector and their weights may include: for the K attention heads, determining the displacement field at the scale based on the average value of the corresponding products of the N displacement basis vectors generated by each attention head and the determined weights of the N displacement basis vectors; wherein the displacement vector from each voxel in the moving image to the corresponding voxel in the fixed image may be represented as a weighted sum of the multiple displacement basis vectors.

[0076] Optionally, the combined representation of the above-mentioned multiple displacement basis vectors can be a weighted sum of multiple displacement basis vectors, wherein the weights of these displacement basis vectors are determined by normalizing the attention weights of all displacement basis vectors for the same feature vector.

[0077] As mentioned above, each displacement vector in the displacement field at scale l (l=1...L) (where k represents the number of the displacement vector in the displacement field) can be obtained by taking the average of the sum of the products of the attention weights of all attention heads and the corresponding displacement basis vectors. For example, It can be expressed as:

[0078]

[0079]

[0080] in, is the i-th displacement basis vector in the j-th attention head, w i，j is the attention weight of the displacement basis vector, [·] represents the concatenation operation, and fc1 and fc2 represent the mapping functions of the two branches respectively. Therefore, by determining each displacement vector in the displacement field The displacement field u at scale l can be determined l (l=1...L).

[0081] In this disclosure, reference is made to FIG. 3A to FIG. 3C The described process of determining the displacement field at each scale uses a non-global multi-head attention mechanism, which respects the locality of the registration while having fewer parameters than solutions using standard CNN or Transformer blocks.

[0082] Figure 3C The process of determining the displacement field u based on the feature maps of the moving image and the fixed image at the scale l is shown in the form of a flow chart for the dimensions of the feature maps of the moving image and the fixed image. l process. Figure 3C As shown in the flowchart, the two process flows in the flowchart correspond to the above two independent branches. For the process flow on the left, the feature map of the moving image is converted into K N×3-dimensional displacement basis vectors through linear projection, so the dimension of its feature map is increased from B*D l *W l *H l *C l Change to B*D l *W l *H l *(K*N*3), while in the process flow on the right, the feature maps of the moving image and the fixed image are spliced, and the dimension of the spliced ​​feature map is B*D l *W l *H l *2C l, the spliced ​​feature map is converted into the attention weights of the above K N×3-dimensional displacement basis vectors after the above linear projection, so the spliced ​​feature map is changed to B*D l *W l *H l *(K*N*1). Therefore, by reshaping the results of the two process flows (such as transposition transformation, etc.), the displacement field in the K attention heads can be obtained, whose dimension is B*D l *W l *H l *K*3. Therefore, by averaging the displacement field in all attention heads as described above, the displacement field u at scale l can be determined l , whose dimension is B*3*D l *W l *H l .

[0083] In step 204, a final displacement field may be generated based on the multiple displacement fields determined at the multiple scales, and the final displacement field may be applied to the moving image, so that the moving image to which the final displacement field is applied is registered with the fixed image.

[0084] After determining multiple displacement fields for feature maps at multiple scales, these displacement fields at different scales cannot be directly used as the final registration results, because the fine-scale displacement field has a limited receptive field and cannot represent large deformations, while the coarse-scale displacement field is too rough to provide accurate voxel-level displacement vectors for the original image. Therefore, in the image registration method disclosed in the present invention, a refinement fusion network can be used to fuse these displacement fields to combine the characteristics of low-level detail feature maps and high-level semantic feature maps to obtain a more accurate displacement field.

[0085] According to an embodiment of the present disclosure, generating a final displacement field based on multiple displacement fields determined at the multiple scales may include: cascading and fusing the multiple displacement fields according to the scale sizes to convert the multiple displacement fields at the multiple scales into a final displacement field at the original size of the moving image.

[0086] In the image registration method disclosed in the present invention, the following method can be used: Figure 4A The cascaded fusion network shown is used to generate a final displacement field based on multiple displacement fields. The cascaded fusion network repeatedly applies fusion blocks and convolution heads to convert the displacement fields at L scales into the final displacement field at the original resolution.

[0087] Figure 4A is a schematic diagram showing the cascade fusion of multiple displacement fields determined at multiple scales according to an embodiment of the present disclosure. Figure 4B is a flowchart illustrating each level of fusion in cascade fusion according to an embodiment of the present disclosure. Figure 4C FIG. 1 is a schematic diagram showing the first level fusion in the cascade fusion according to an embodiment of the present disclosure.

[0088] Optionally, for the case of L=4, cascade fusion can be performed in order of scale l from large to small, and the cascade fusion includes L-1 level fusion, wherein the output feature map of the fusion block of the lth level fusion is denoted as g l For the l-th level fusion, we can use the output g of the previous fusion l+1 and the displacement field corresponding to this level of fusion (expressed as B*3*D l *W l *H l ) and feature maps of moving and fixed images and The fusion result g of this level of fusion is generated by the fusion block l .like Figure 4A As shown in the figure, for the case of L = 4, the displacement field determined at the four scales is The final displacement field B*3*D*W*H can be obtained through three levels of fusion. For the L-1 level fusion, its input includes the displacement field corresponding to this level of fusion and the feature maps of the moving image and the fixed image. and In addition, there is no input of the upper fusion, but it also includes the feature extraction of the displacement field at scale L g L .

[0089] According to an embodiment of the present disclosure, cascading and fusing the multiple displacement fields according to the scale may include: Figure 4B Steps 2041-2044 are shown.

[0090] In step 2041, in each level of fusion in the cascade fusion, the feature map with a specific number of channels output by the upper level fusion may be upsampled to obtain a first feature map.

[0091] Alternatively, if Figure 4C As shown in (a), the feature map g output by the upper fusion can be l+1 Upsampling is performed to obtain the first feature map. For example, the size of the feature map at scale l provided as an example in the present disclosure is proportionally reduced to 1 / 2 of the size of the original image. l , the output g of the upper fusion can be l+1 Perform bilinear interpolation upsampling, and the dimension of the first feature map obtained is B*3*D l *W l *H l It should be understood that other data processing methods that can achieve the same effect can also be applied to the method of the present disclosure, and the upsampling method is only used as an example rather than a limitation in the present disclosure.

[0092] In step 2042, feature extraction may be performed on the displacement field corresponding to the level of fusion to convert the displacement field into a second feature map having the specific number of channels.

[0093] Optionally, the displacement field u on scale l can be transformed into l Converted into a feature map h with a fixed number of channels C (for example, C = 128, which can be set according to actual needs) l , i.e., the second feature map. Similarly, the feature extraction g of the displacement field at scale L in the L-1th level fusion above is L It can also be obtained in this way. As an example, the predetermined convolution block may include three convolution layers (for example, the number of channels is 16, 64 and 128 respectively, and the scale of each convolution layer is 1), and may be followed by group normalization and activation function (for example, ReLU activation function). It should be noted that convolution blocks of different scales do not share weights, and their convolution kernel sizes can be set to (5, 5, 5), (5, 5, 3), (5, 3, 3), (3, 3, 3) from large to small scales. It should be understood that the specific parameters of the predetermined convolution block can be set according to actual needs, and the above-mentioned specific parameter settings are only used as examples and not limitations in this disclosure.

[0094] In step 2043, the first feature map and the second feature map may be added and concatenated with the feature maps of the fixed image and the moving image at the scale corresponding to the fusion at this level.

[0095] Optionally, the result of the previous fusion can be combined with the displacement field information and feature map corresponding to the current fusion to generate a fusion result. l+1 Added to the feature map h extracted from the displacement field corresponding to the current fusion l (The dimension of the result of the addition is still B*3*D l *W l *H l ). Afterwards, the result of the addition can be combined with the latent representation and The result of splicing can be expressed as Here, upsample(·) represents upsampling (·). Therefore, by applying the following Figure 4C The convolution block shown in (b) extracts feature information and can provide the image information g of this level of fusion l .

[0096] In step 2044, feature extraction may be performed on the concatenated result to generate a feature map with the specific number of channels as the output of this level of fusion.

[0097] like Figure 4C As shown in (b), the last component of the fusion block can be another convolution block, which can increase the number of channels (C+2C) of the output high-order potential representation. l ) is reduced to a fixed number of channels C. The convolution block can have the same structure and parameters as the predetermined convolution block described above, which will not be described in detail here, but it should be understood that other methods that can achieve the same purpose or effect can also be applied to the image registration method disclosed in the present invention.

[0098] According to an embodiment of the present disclosure, cascading and fusing the multiple displacement fields according to scale sizes may also include: for a feature map with the specific number of channels output by the first level fusion in the cascade fusion, performing feature extraction on the feature map to generate the final displacement field.

[0099] As mentioned above, for the case of L=4, through the cascade fusion of L-1 fusion blocks, the final displacement field with 3 channels needs to be output. Optionally, the feature map g output by the first level fusion can be 1 A similar convolutional head is applied (e.g., a convolutional head containing three decoding blocks, each decoding block having two consecutive structures including a convolutional layer and an activation function (e.g., a ReLU activation function)) to generate a final displacement field with dimensions B*3*D*W*H.

[0100] In the image registration method disclosed in the present invention, by adopting the reference FIG. 4A to FIG. 4C The described refinement fusion network fuses the intermediate displacement fields determined at multiple scales to generate a final displacement field, so as to provide high-resolution large deformation mapping capability by combining the displacement field information and the feature maps at multiple scales based on the characteristics of the low-level detail feature map and the high-level semantic feature map.

[0101] Therefore, the image registration method described above with reference to steps 201 to 204 can be expressed as follows: Figure 2C The schematic flow chart shown in FIG. 1 shows a schematic flow chart showing ...

[0102] In the image registration method disclosed in the present invention, the training of the multi-scale registration network can be based on the optimization of the registration loss function. Among them, by optimizing the registration loss function based on the training data in the multi-scale registration network, the optimal parameters of the multi-scale registration network can be determined. The determined optimal parameters can be directly applied in subsequent image registration tasks without re-optimizing the calculation for real-time tasks each time. The parameters involved in operations such as convolution in the image registration method disclosed in the present invention can be predetermined based on the optimization of the registration loss function.

[0103] Therefore, according to an embodiment of the present disclosure, the generation of the final displacement field can be based on the optimization of the registration loss function, and the registration loss function may include a first term for measuring the similarity between the moving image and the fixed image after applying the final displacement field, and a second term for imposing a penalty on the displacement field to penalize the local spatial changes of the moving image.

[0104] Optionally, the image registration method of the present disclosure may use most registration loss functions, such as the following registration loss function:

[0105]

[0106] Among them, λ is the regularization trade-off parameter.

[0107] Among them, the first The similarity between the deformed moving image and the fixed image is measured by local normalized cross-correlation, which can be expressed as:

[0108]

[0109] Among them, Ω is the entire voxel set, and P represents the neighboring voxels of voxel p.

[0110] If you use and represents the expected value of the neighboring voxels of the fixed image and the deformed moving image at voxel p, then ΔF(q) and Can be defined as and In order to eliminate the influence of absolute gray value, the structural differences between the image pairs to be registered can be better reflected.

[0111] In the embodiments of the present disclosure, the second It can be a regularization applied to the deformation field to penalize local spatial changes in the moving image, which can be expressed as:

[0112]

[0113] in, Represents the approximate spatial gradient of the displacement vector between two adjacent voxels.

[0114] Optionally, u(p (x+1) , p y , p z )-u(p x , p y , p z ) is calculated And apply similar operations to obtain and Therefore, by constraining the size of the displacement as above, it can be ensured that the moving image will not produce too much distortion.

[0115] As described above, the image registration method of the present disclosure provides direct guidance for the displacement field determination and feature extractor at each scale by imposing a penalty on the displacement field at each scale.

[0116] It should be noted that in order to compute the objective function, the displacement field determined at each scale needs to be upsampled to the original size to deform the moving image.

[0117] Therefore, the total registration loss function used in the image registration method of the present disclosure can be expressed as:

[0118]

[0119] Among them, β l represents the intermediate registration loss function at scale l The weight of .

[0120] As mentioned above, an auxiliary loss is imposed on the output determined by the displacement field at each scale, such as Figure 2B As shown, additional guidance can be provided to fully utilize the representation subspace at each scale to avoid excessive distortion in moving images.

[0121] Figure 5 It is a schematic diagram showing the image registration results under different methods according to an embodiment of the present disclosure and the visualization results of the deformation field under different scales.

[0122] Figure 5 The application results of the image registration method disclosed in the present invention in the three-dimensional medical image registration scenario are shown, wherein Figure 5 (a) shows the comparison of medical image registration results using different methods, and Figure 5 (b) shows the visualization results of the deformation field at different scales using the image registration method disclosed in the present invention.

[0123] Figure 5The image registration results in (a) and (b) are the registration results of magnetic resonance imaging (MRI) images of the brain, where curves of different depths represent the boundaries of different brain structures (for example, the two closed curves symmetrically distributed at the bottom of the MRI image correspond to the boundaries of the caudate nucleus). Figure 5 As shown in (a), compared with other image registration methods (such as SyN, NiftyReg, VoxelMorph, etc.), the image registration method in the present disclosure can better retain the detailed information of the medical image, and the deformed moving image (marked as "the present disclosure" in the figure) can better match the segmentation annotations of different brain structures.

[0124] Figure 5 The different colors of the deformation field in (a) and (b) represent the degree of deformation, that is, the displacement of the voxel point, where the darker the color, the greater the displacement, and the direction of the curve change in the deformation field indicates the displacement direction of the voxel point. Figure 5 As shown in (b), as the scale increases from small to large, the obtained deformation field gradually becomes blurred and the resolution gradually decreases. Through the image registration method disclosed in the present invention, a more accurate displacement field can be obtained by combining the characteristics of the low-level detail feature map and the high-level semantic feature map.

[0125] Figure 6 is a schematic diagram illustrating the application of the image registration method according to an embodiment of the present disclosure.

[0126] like Figure 6 As shown, the image registration method disclosed in the present invention can be directly applied as a model. Figure 6 In (a), the user can upload pairs of images to be registered (for example, on front end A), and after receiving the image pairs to be registered, the backend can use the model trained based on the image registration method of the present invention to directly register the image pairs (back end) and output the registration results (for example, on front end B).

[0127] like Figure 6 As shown in (b), the image registration method of the present invention can also be deployed through the intelligent open laboratory platform. Users can directly use the image registration method and output the registered image by dragging relevant modules (e.g., image registration module) into the display interface.

[0128] Figure 7 is a schematic diagram showing an image registration apparatus 700 according to an embodiment of the present disclosure.

[0129] According to an embodiment of the present disclosure, the image registration apparatus 700 may include an image acquisition module 701 , a feature map extraction module 702 , a displacement field determination module 703 and a displacement field synthesis module 704 .

[0130] The image acquisition module 701 may be configured to acquire a fixed image and a moving image to be registered. Optionally, the image acquisition module 701 may perform the operations described above with reference to step 201 .

[0131] For example, considering the development of imaging technology from planar images to three-dimensional stereo images, the fixed images and moving images to be registered can be three-dimensional images. In the present disclosure, the image registration processing scenario for three-dimensional images is mainly described, but this does not mean that the scope of application of the image registration method of the present disclosure is limited. The image registration method of the present disclosure can also be applied to other higher or lower dimensional image processing. The following operations on three-dimensional images are only used as examples and not as limitations.

[0132] The feature map extraction module 702 may be configured to obtain feature maps of the fixed image and the moving image at multiple scales. Optionally, the feature map extraction module 702 may perform the operations described above with reference to step 202 .

[0133] Optionally, the moving image and the fixed image can be fed into the same encoder (with shared weights) based on a convolutional neural network for feature extraction, so that the encoder can extract a sequence of intermediate feature maps at different scales through different convolution blocks, that is, obtain feature maps at multiple scales.

[0134] Therefore, multiple displacement fields can be learned from paired feature maps of fixed images and moving images at multiple scales, and the displacement fields learned at all scales are fused to obtain a more accurate displacement field by combining the characteristics of low-level detail feature maps and high-level semantic feature maps.

[0135] The displacement field determination module 703 may be configured to determine, for each scale in the plurality of scales, a displacement field at the scale from the feature maps of the fixed image and the moving image at the scale, wherein the displacement field is used to indicate a mapping from the moving image to the fixed image at the scale. Optionally, the displacement field determination module 703 may perform the operations described above with reference to step 203.

[0136] Optionally, a displacement field can be determined for the paired feature maps at each scale, wherein the displacement field is composed of displacement vectors between voxel pairs between the pair of images to be registered. Different from the method of directly converting the image representation into spatial correspondence, in order to facilitate the mapping from the image representation to the spatial relationship and improve the interpretability of the spatial transformation in the image registration network, in the image registration method disclosed in the present invention, according to the theory of linear algebra, the displacement vector between two voxels is decomposed into a combination representation of multiple displacement basis vectors, thereby avoiding complex and poorly interpretable direct conversion of spatial correspondence.

[0137] Optionally, for the feature map of the moving image and the feature map of the fixed image at the same scale, the combination of the above multiple displacement basis vectors can be expressed by Figure 3B As shown, it is obtained by using two independent branches of a multi-head attention mechanism. In the first branch, linear projection can be used to convert each feature vector in the feature map of the moving image into N displacement basis vectors, where each displacement basis vector can be composed of three elements, which can correspond to displacements in three spatial directions, namely, displacements on the x, y, and z axes. Therefore, in order to obtain information from different representation subspaces, the image registration method disclosed in the present invention is based on a multi-head attention mechanism, using K attention heads for each feature vector, and the number of displacement basis vectors for each attention head is N, thereby generating K×N displacement basis vectors for each voxel. This branch only uses the feature map of the moving image to generate the displacement basis vector, which is sufficient to extract the most representative displacement basis vector to represent the most likely deformation direction of each voxel.

[0138] In the second branch, the attention weight of each of the N displacement basis vectors corresponding to each attention head is learned by linearly projecting the concatenated feature maps of the moving image and the fixed image.

[0139] Therefore, based on the above two independent branches, the direction in which each voxel is most likely to be deformed is determined based on the feature vector in the moving image, and the relative displacement in each direction in which deformation is most likely to occur is determined based on the similarity between the moving image and the fixed image. The displacement vectors of the corresponding voxels can be determined based on these directions and the relative displacement in these directions.

[0140] The displacement field synthesis module 704 may be configured to generate a final displacement field based on the multiple displacement fields determined at the multiple scales, and apply the final displacement field to the moving image so that the moving image to which the final displacement field is applied is aligned with the fixed image. Optionally, the displacement field synthesis module 704 may perform the operations described above with reference to step 204.

[0141] After determining multiple displacement fields for feature maps at multiple scales, these displacement fields at different scales cannot be directly used as the final registration results, because the fine-scale displacement field has a limited receptive field and cannot represent large deformations, while the coarse-scale displacement field is too rough to provide accurate voxel-level displacement vectors for the original image. FIG. 4A to FIG. 4C The described refinement fusion network fuses the intermediate displacement fields determined at multiple scales to generate a final displacement field, so as to provide high-resolution large deformation mapping capability by combining the displacement field information and the feature maps at multiple scales based on the characteristics of the low-level detail feature map and the high-level semantic feature map.

[0142] Therefore, by inputting the image pair to be registered into the multi-scale registration network of the present invention, the deformation field of the moving image in the image pair to be registered can be obtained. Therefore, by applying the obtained deformation field to the moving image, the final registered image can be output.

[0143] In the image registration method disclosed in the present invention, the training of the multi-scale registration network can be based on the optimization of the registration loss function. Among them, the optimal parameters of the multi-scale registration network can be determined by optimizing the registration loss function based on the training data in the multi-scale registration network. The determined optimal parameters can be directly applied in subsequent image registration tasks without re-optimizing the calculation for real-time tasks each time. The parameters involved in operations such as convolution in the image registration method disclosed in the present invention can be predetermined based on the optimization of the registration loss function. Optionally, the registration loss function adopted by the image registration method disclosed in the present invention may include regularization applied to the deformation field to penalize local spatial changes in the moving image, so that the image registration method disclosed in the present invention can provide direct guidance for the displacement field determination and feature extractor at each scale by applying a penalty to the displacement field at each scale, thereby avoiding excessive distortion in the moving image.

[0144] According to yet another aspect of the present disclosure, an image registration device is provided. Figure 8 A schematic diagram of an image registration device 2000 according to an embodiment of the present disclosure is shown.

[0145] like Figure 8 As shown, the image registration device 2000 may include one or more processors 2010 and one or more memories 2020. The memory 2020 stores a computer readable code, and when the computer readable code is run by the one or more processors 2010, the image registration method described above may be executed.

[0146] The processor in the embodiments of the present disclosure may be an integrated circuit chip having signal processing capabilities. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc., and may be an X86 architecture or an ARM architecture.

[0147] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the embodiments of the present disclosure are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0148] For example, the method or device according to the embodiment of the present disclosure may also be implemented by Fig. 9 The architecture of the computing device 3000 shown in FIG. Fig. 9 As shown, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage device in the computing device 3000, such as the ROM 3030 or the hard disk 3070, may store various data or files used for processing and / or communication of the image registration method provided by the present disclosure and program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, Figure 8 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Fig. 9 One or more components of a computing device are shown.

[0149] According to yet another aspect of the present disclosure, a computer-readable storage medium is provided. Fig.10 A schematic diagram 4000 of a storage medium according to the present disclosure is shown.

[0150] like Fig.10As shown, the computer storage medium 4020 stores computer readable instructions 4010. When the computer readable instructions 4010 are executed by the processor, the image registration method according to the embodiment of the present disclosure described with reference to the above figures can be executed. The computer readable storage medium in the embodiment of the present disclosure can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous connection dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM). It should be noted that memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory. It should be noted that memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0151] The embodiments of the present disclosure also provide a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device performs the image registration method according to the embodiments of the present disclosure.

[0152] Embodiments of the present disclosure provide an image registration method, apparatus, device, and computer-readable storage medium.

[0153] Compared with the traditional image registration method, the method provided by the embodiment of the present disclosure greatly improves the registration speed and registration accuracy by adopting an image registration model based on unsupervised neural network learning, while alleviating the problem of scarce annotation information of image data sets. In addition, compared with the existing registration method based on unsupervised learning, the method provided by the embodiment of the present disclosure improves the interpretability of the registration network and reduces the amount of calculation required for registration by promoting the mapping from image representation to spatial relationship instead of directly converting image representation into spatial correspondence.

[0154] The method provided by the embodiment of the present disclosure extracts feature maps of the image pair to be registered at multiple scales, converts the displacement vector of each voxel in the moving image of the image pair to be registered at each scale into a combination of multiple displacement basis vectors determined based on the feature vector corresponding to the voxel in the corresponding feature map, so as to learn the displacement field from the paired feature maps, and generate the final displacement field based on the displacement fields at all scales, so as to make full use of the image feature information at different scales to achieve efficient and accurate image registration. The method of the embodiment of the present disclosure can learn the displacement field from coarse to fine based on different scales, so as to make full use of the potential information of different representation subspaces by fusing the displacement fields at all scales without introducing dense continuous network calculations.

[0155] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of the code contains at least one executable instruction for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0156] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the embodiments of the present disclosure are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0157] The exemplary embodiments of the present disclosure described in detail above are merely illustrative and not restrictive. It should be understood by those skilled in the art that various modifications and combinations may be made to these embodiments or their features without departing from the principles and spirit of the present disclosure, and such modifications should fall within the scope of the present disclosure.

Claims

1. A deep learning-based image registration method, comprising: Acquire a fixed image and a moving image to be registered; Obtaining feature maps of the fixed image and the moving image at multiple scales; For each scale of the plurality of scales, determining a displacement field at the scale from respective feature maps of the fixed image and the moving image at the scale, the displacement field being used to indicate a mapping from the moving image to the fixed image at the scale, wherein in the displacement field, a displacement vector from each voxel in the moving image to a corresponding voxel in the fixed image is represented as a combination of a plurality of displacement basis vectors, wherein the plurality of displacement basis vectors are obtained based on feature vectors corresponding to the voxels in the feature map of the moving image at the scale; as well as A final displacement field is generated based on the multiple displacement fields determined at the multiple scales, and the final displacement field is applied to the moving image so that the moving image to which the final displacement field is applied is registered with the fixed image.

2. The method of claim 1, wherein: Determining the displacement field from the feature maps of the fixed image and the moving image at the scale comprises: For each feature vector in the feature map of the moving image at the scale, generating a plurality of displacement basis vectors based on the feature vector; Based on the feature maps of the fixed image and the moving image at the scale, respectively, determining a weight of each displacement basis vector in the plurality of displacement basis vectors for the feature vector; and For each feature vector in the feature map of the moving image at the scale, a displacement field at the scale is determined based on a plurality of displacement basis vectors corresponding to the feature vector and weights thereof.

3. The method of claim 2, wherein: The plurality of displacement basis vectors are generated based on the feature vector using K attention heads according to a multi-head attention mechanism, wherein N displacement basis vectors are generated using each attention head; For each of the K attention heads, the weights of the N displacement basis vectors generated by the attention head are determined for the feature vector by the attention head based on the concatenation of the respective feature maps of the fixed image and the moving image at the scale.

4. The method of claim 3, wherein: Determining the displacement field at the scale based on a plurality of displacement basis vectors corresponding to the feature vector and their weights includes: For the K attention heads, determining a displacement field at the scale based on an average of corresponding products of N displacement basis vectors generated by each attention head and the determined weights of the N displacement basis vectors; The displacement vector of each voxel in the moving image to the corresponding voxel in the fixed image is represented as a weighted sum of the multiple displacement basis vectors.

5. The method of claim 1, wherein: Obtaining feature maps of the fixed image and the moving image at multiple scales respectively includes: Using an encoder based on a convolutional neural network, extracting feature maps at different scales through different convolution blocks in the encoder to obtain feature maps of the fixed image and the moving image at multiple scales; The size of the feature map of the fixed image or the moving image at a specific scale is the size of the fixed image or the moving image after being reduced according to a corresponding ratio.

6. The method of claim 5, wherein: Generating a final displacement field based on the multiple displacement fields determined at the multiple scales comprises: The multiple displacement fields are cascaded and fused according to the scales, so as to convert the multiple displacement fields at the multiple scales into a final displacement field at the original size of the moving image.

7. The method of claim 6, wherein: The cascading fusion of the multiple displacement fields according to the scales includes: In each level of fusion in the cascade fusion, upsampling the feature map with a specific number of channels output by the upper level fusion to obtain a first feature map; Performing feature extraction on the displacement field corresponding to the level of fusion to convert the displacement field into a second feature map having the specific number of channels; Adding the first feature map to the second feature map, and concatenating the first feature map with the feature maps of the fixed image and the moving image at the scale corresponding to the fusion level; and Feature extraction is performed on the concatenated result to generate a feature map with the specific number of channels as the output of the fusion at this level.

8. The method of claim 7, wherein: The cascading fusion of the multiple displacement fields according to the scales also includes: For the feature map with the specific number of channels output by the first level fusion in the cascade fusion, feature extraction is performed on the feature map to generate the final displacement field.

9. The method of claim 1, wherein: The generation of the final displacement field is based on the optimization of a registration loss function, which includes a first term for measuring the similarity between the moving image and the fixed image after applying the final displacement field, and a second term for penalizing the displacement field to penalize local spatial variations of the moving image.

10. An image registration device based on deep learning, comprising: An image acquisition module, configured to acquire a fixed image and a moving image to be registered; A feature map extraction module is configured to obtain feature maps of the fixed image and the moving image at multiple scales; a displacement field determination module configured to determine, for each scale of the plurality of scales, a displacement field at the scale from respective feature maps of the fixed image and the moving image at the scale, wherein the displacement field is used to indicate a mapping from the moving image to the fixed image at the scale; as well as a displacement field synthesis module, configured to generate a final displacement field based on the multiple displacement fields determined at the multiple scales, and apply the final displacement field to the moving image, so that the moving image to which the final displacement field is applied is registered with the fixed image; In which, in the displacement field, the displacement vector of each voxel in the moving image to the corresponding voxel in the fixed image is represented as a combination of multiple displacement basis vectors, wherein the multiple displacement basis vectors are obtained based on the feature vectors corresponding to the voxels in the feature map of the moving image at the scale.

11. The device of claim 10, wherein: Determining the displacement field from the feature maps of the fixed image and the moving image at the scale comprises: For each feature vector in the feature map of the moving image at the scale, generating a plurality of displacement basis vectors based on the feature vector; Based on the feature maps of the fixed image and the moving image at the scale, respectively, determining a weight of each displacement basis vector in the plurality of displacement basis vectors for the feature vector; and For each feature vector in the feature map of the moving image at the scale, a displacement field at the scale is determined based on a plurality of displacement basis vectors corresponding to the feature vector and weights thereof.

12. The device of claim 10, wherein: Generating a final displacement field based on the multiple displacement fields determined at the multiple scales comprises: Cascadingly fusing the multiple displacement fields according to the scales, so as to convert the multiple displacement fields at the multiple scales into a final displacement field at the original size of the moving image; The cascading fusion of the multiple displacement fields according to the scales includes: In each level of fusion in the cascade fusion, upsampling the feature map with a specific number of channels output by the upper level fusion to obtain a first feature map; Performing feature extraction on the displacement field corresponding to the level of fusion to convert the displacement field into a second feature map having the specific number of channels; Adding the first feature map to the second feature map, and concatenating the first feature map with the feature maps of the fixed image and the moving image at the scale corresponding to the fusion level; and Feature extraction is performed on the concatenated result to generate a feature map with the specific number of channels as the output of the fusion at this level.

13. An image registration device, comprising: one or more processors; as well as One or more memories, wherein a computer executable program is stored, and when the computer executable program is executed by the processor, the method of any one of claims 1-9 is performed.

14. A computer program product stored on a computer-readable storage medium and comprising computer instructions which, when executed by a processor, cause a computer device to perform the method of any one of claims 1 to 9.

15. A computer-readable storage medium having computer-executable instructions stored thereon, wherein the instructions are used to implement the method according to any one of claims 1 to 9 when executed by a processor.

Images