Image registration method and device, storage medium and electronic equipment

Abstract

The invention provides an image registration method and device, a computer storage medium and electronic equipment, and relates to the technical field of image processing. Comprising the following steps: determining a first feature map of a to-be-registered image under multiple different resolutions and a second feature map of a reference template image under multiple different resolutions; similar feature matching is carried out on the first sub-feature map of the first resolution and the second sub-feature map of the first resolution to obtain a first deformation field; according to the first deformation field, deforming the first sub-feature map of the second resolution to obtain a third sub-feature map, and carrying out similar feature matching on the third sub-feature map and the second sub-feature map of the second resolution to obtain a residual deformation field; and determining a second deformation field through the first deformation field and the residual deformation field until a target deformation field with the highest resolution is obtained, and performing deformation registration on the to-be-registered image according to the target deformation field to obtain a registered target image. The calculation complexity of deformable image registration can be reduced, and the calculation efficiency can be improved.

Description

Background Art

[0002] The goal of deformable image registration is to find the optimal displacement vector field (deformation field) between the fixed image and the floating image, and the floating image is deformed by the estimated deformation field so that the anatomical structure of the deformed floating image should be aligned with the fixed image. The related technical solutions mainly use convolutional neural networks (CNN) to regress the displacement vector field, which has high computational complexity and slow calculation speed, thus affecting the image registration performance.

[0003] Therefore, there is an urgent need for an image registration method with low computational complexity and fast calculation speed. Summary of the invention

[0004] The present disclosure provides an image registration method, an image registration device, a computer storage medium and an electronic device, thereby reducing the computational complexity of deformable image registration and improving the computational efficiency of deformable image registration.

[0005] In a first aspect, an embodiment of the present disclosure provides an image registration method, the method comprising: acquiring an image to be registered and a reference template image, and determining a first feature map of the image to be registered at multiple different resolutions, and a second feature map of the reference template image at multiple different resolutions; for a first sub-feature map of a first resolution and a second sub-feature map of the first resolution in multiple image resolutions, performing similar feature matching on the first sub-feature map of the first resolution and the second sub-feature map of the first resolution to obtain a first deformation field; performing deformation registration on the first sub-feature map of the second resolution according to the first deformation field to obtain a third sub-feature map, and performing similar feature matching on the third sub-feature map and the second sub-feature map of the second resolution to obtain a residual deformation field; the first resolution is smaller than the second resolution; based on the first deformation field and the residual deformation field, determining the second deformation field until a target deformation field at the highest resolution is obtained, so as to perform deformation registration on the image to be registered according to the target deformation field to obtain a registered target image.

[0006] In an optional embodiment of the present disclosure, determining a first feature map of an image to be registered at multiple different resolutions and a second feature map of a reference template image at multiple different resolutions includes: inputting the image to be registered and the reference template image into a convolutional network, the convolutional network comprising at least n first convolutional modules and n second convolutional modules; performing feature extraction on the image to be registered based on the superposition of j first convolutional modules among the n first convolutional modules, to obtain first feature maps at n different resolutions; and, performing feature extraction on the reference template image based on the superposition of j second convolutional modules among the n second convolutional modules, to obtain second feature maps at n different resolutions; wherein the number of j first convolutional modules / second convolutional modules is less than or equal to the number of n first convolutional modules / second convolutional modules.

[0007] In an optional embodiment of the present disclosure, the weight values ​​of the convolution modules in the same superposition layer among the n first convolution modules and the n second convolution modules are the same.

[0008] In an optional embodiment of the present disclosure, similar feature matching is performed on the third sub-feature map and the second sub-feature map of the second resolution to obtain a residual deformation field, including: for each target voxel in the second sub-feature map of the second resolution, determining the feature correlation between the target voxel and the adjacent voxels in the third sub-feature map to obtain a correlation weight parameter of each voxel; and determining the residual deformation field of each voxel based on the correlation weight parameter.

[0009] In an optional embodiment of the present disclosure, determining the residual deformation field of each voxel based on the correlation weight parameter includes: determining the residual deformation field of each voxel based on the formula:

[0010]

[0011] Among them, ψ p is the residual deformation field of each target voxel p; the softmax function is a normalized exponential function; C pd G is the correlation weight parameter between the adjacent d voxels in the third sub-feature map for each target voxel p; pd is the relative position coordinate vector relative to each target voxel p; c is the feature dimension of the second sub-feature map.

[0012] In an optional embodiment of the present disclosure, the method further includes: if the target voxel is a voxel at a boundary position in a second sub-feature map of a second resolution, performing voxel filling on the third sub-feature map to determine the feature correlation between the target voxel at the boundary position and the adjacent voxels filled with voxels in the third sub-feature map, and obtaining the correlation weight parameters of each voxel.

[0013] In an optional embodiment of the present disclosure, the method further includes: determining a similarity loss parameter between the target image and the reference template image, and determining a displacement offset loss parameter of the target deformation field in space; superimposing the similarity loss parameter and the displacement offset loss parameter to obtain a target loss parameter between the target image and the reference template image; and adjusting the parameters for similar feature matching based on the target loss parameter.

[0014] In a second aspect, an embodiment of the present disclosure provides an image registration device, which includes: a feature map determination module, configured to execute acquisition of an image to be registered and a reference template image, and determine a first feature map of the image to be registered at multiple different resolutions, and a second feature map of the reference template image at multiple different resolutions; a feature matching module, configured to execute similar feature matching of the first sub-feature map of the first resolution and the second sub-feature map of the first resolution among multiple image resolutions to obtain a first deformation field; a first deformation registration module, configured to execute deformation registration of the first sub-feature map of the second resolution according to the first deformation field to obtain a third sub-feature map, and similar feature matching of the third sub-feature map and the second sub-feature map of the second resolution to obtain a residual deformation field; the first resolution is smaller than the second resolution; and a second deformation registration module, configured to execute determination of the second deformation field based on the first deformation field and the residual deformation field until a target deformation field at the highest resolution is obtained, so as to perform deformation registration on the image to be registered according to the target deformation field to obtain a registered target image.

[0015] In an optional embodiment of the present disclosure, the feature map determination module is specifically used to input the image to be registered and the reference template image into a convolutional network, wherein the convolutional network includes at least n first convolution modules and n second convolution modules; based on the superposition of j first convolution modules in the n first convolution modules, feature extraction is performed on the image to be registered to obtain n first feature maps at different resolutions; and, based on the superposition of j second convolution modules in the n second convolution modules, feature extraction is performed on the reference template image to obtain n second feature maps at different resolutions; wherein the number of j first convolution modules / second convolution modules is less than or equal to the number of n first convolution modules / second convolution modules.

[0016] In an optional embodiment of the present disclosure, the weight values ​​of the convolution modules in the same superposition layer among the n first convolution modules and the n second convolution modules are the same.

[0017] In an optional embodiment of the present disclosure, the first deformation registration module is specifically used to determine, for each target voxel in the second sub-feature map of the second resolution, the feature correlation between the target voxel and the adjacent voxels in the third sub-feature map, and obtain the correlation weight parameters of each voxel; based on the correlation weight parameters, determine the residual deformation field of each voxel.

[0018] In an optional embodiment of the present disclosure, the first deformation registration module is specifically used to determine the residual deformation field of each voxel based on the formula:

[0019]

[0020] Among them, ψp is the residual deformation field of each target voxel p; the softmax function is a normalized exponential function; C pd G is the correlation weight parameter between the adjacent d voxels in the third sub-feature map for each target voxel p; pd is the relative position coordinate vector relative to each target voxel p; c is the feature dimension of the second sub-feature map.

[0021] In an optional embodiment of the present disclosure, the device also includes a voxel filling module, which is used to perform voxel filling on the third sub-feature map when the target voxel is a voxel at a boundary position in the second sub-feature map of the second resolution, so as to determine the feature correlation between the target voxel at the boundary position and the adjacent voxel filled in the third sub-feature map, and obtain the correlation weight parameter of each voxel.

[0022] In an optional embodiment of the present disclosure, the device also includes a loss determination module and a loss adjustment module, wherein the loss determination module is used to determine the similarity loss parameters between the target image and the reference template image, and determine the displacement offset loss parameters of the target deformation field in space; superimpose the similarity loss parameters and the displacement offset loss parameters to obtain the target loss parameters between the target image and the reference template image; the loss adjustment module is used to adjust the parameters of similar feature matching based on the target loss parameters.

[0023] In a third aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which implements the above image registration method when executed by a processor.

[0024] In a fourth aspect, an embodiment of the present disclosure provides an electronic device, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the above image registration method by executing the executable instructions.

[0025] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which is executed by a processor to implement the above image registration method.

[0026] The technical solution disclosed in this disclosure has the following beneficial effects:

[0027] The above-mentioned image registration method obtains the image to be registered and the reference template image, and determines the first feature map of the image to be registered at multiple different resolutions, and the second feature map of the reference template image at multiple different resolutions; for the first sub-feature map of the first resolution and the second sub-feature map of the first resolution in multiple image resolutions, similar feature matching is performed on the first sub-feature map of the first resolution and the second sub-feature map of the first resolution to obtain a first deformation field; the first sub-feature map of the second resolution is deformed and registered according to the first deformation field to obtain a third sub-feature map, and similar feature matching is performed on the third sub-feature map and the second sub-feature map of the second resolution to obtain a residual deformation field; the first resolution is smaller than the second resolution; based on the first deformation field and the residual deformation field, the second deformation field is determined until the target deformation field at the highest resolution is obtained, so as to deform and register the image to be registered according to the target deformation field to obtain the registered target image.

[0028] On the one hand, this method separates the feature extraction process from the feature matching process, thereby more effectively utilizing the feature extraction capability and the feature correlation recognition capability of the attention mechanism, and is more efficient in calculating the dense correspondence between the voxels of the deformable image to be registered and the fixed reference template image, thereby improving the computational efficiency and registration performance of deformable image registration. On the other hand, this method determines the feature maps of the image to be registered and the reference template image at multiple different resolutions, respectively, so as to gradually align the overall structure from the low-resolution feature map to the high-resolution feature map, that is, in the low-resolution feature map, a large displacement is made between the image to be registered and the reference template image, and a rough alignment of the overall structure is established. As the resolution of the feature map continues to increase, the registration network achieves fine anatomical structure alignment, thereby improving the accuracy of image registration.

[0029] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0031] Figure 1 A schematic diagram showing an architecture of an image registration system according to the exemplary embodiment;

[0032] Figure 2 A flowchart of an image registration method in this exemplary embodiment is schematically shown;

[0033] Figure 3A A schematic diagram schematically shows a method of using a plurality of convolution modules of different numbers to stack and obtain a plurality of first feature maps of different resolutions for an image to be registered in this exemplary embodiment;

[0034] Figure 3B A schematic diagram schematically shows a method of using a plurality of convolution modules of different numbers to stack and obtain a plurality of second feature maps of different resolutions for a reference template image in this exemplary embodiment;

[0035] Figure 4 A flowchart of a method for similar feature matching in this exemplary embodiment is schematically shown;

[0036] Figure 5 A schematic diagram schematically shows a comparison of an existing global attention mechanism, a neighborhood attention mechanism, and an expanded attention mechanism in this exemplary embodiment;

[0037] Figure 6 A schematic diagram of an overall image registration network structure in this exemplary embodiment is schematically shown;

[0038] Figure 7 A schematic diagram of a DMM network structure in feature matching in this exemplary embodiment is shown schematically;

[0039] Figure 8 A schematic diagram of the structure of an image registration device in this exemplary embodiment is shown;

[0040] Fig. 9 The structure diagram of an electronic device in this exemplary embodiment is schematically shown. DETAILED DESCRIPTION

[0041] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, exemplary embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the exemplary embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0042] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0043] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the steps. For example, some steps may be decomposed, while some steps may be combined or partially combined, so the actual execution order may change according to the actual situation.

[0044] In order to help those skilled in the art better understand the technical solution of the present disclosure, the relevant contents involved in the technical solution of the present disclosure are introduced below.

[0045] 1) Receptive field: In a convolutional neural network, it determines the size of the region of the input layer that corresponds to an element in the output of a certain layer.

[0046] 2) Deformable image registration (DIR): It is a medical image registration method that aims to establish a dense nonlinear correspondence between a pair of images. Its core goal is to align the floating image (MovingImage) to the fixed image (Fixed Image) through nonlinear transformation, thereby maximizing the similarity between the two images; this registration process involves complex spatial transformations and can handle subtle deformations and irregular shape changes in the image.

[0047] 3) Attention mechanism: In deep learning, the attention mechanism is widely used in computer vision, natural language processing and other fields. The core idea is that the network should attach different importance to different parts of the input data. The network needs to assign different weights to different input parts in order to better capture the important information in the input data, and this weight assignment process is the attention mechanism.

[0048] Among them, the global attention mechanism, also known as the self-attention mechanism or the multi-head attention mechanism, is a way to associate and weight all positions in the input sequence. It calculates the degree of association between each position and all other positions and assigns a weight value accordingly. The local attention mechanism, also known as the window attention mechanism or the convolutional attention mechanism, is a way to divide the input sequence into segments with a fixed window or convolution kernel size and associate and weight each segment. Unlike the global attention mechanism, the local attention mechanism only considers a local area of ​​the input sequence and weights the information in that area.

[0049] 4) Displacement vector field (DVF): describes the displacement change of each pixel in the image during the registration process. The deformation field DVF is a two-dimensional or three-dimensional vector field, in which each vector represents the displacement direction and distance of each pixel in the image. Through the deformation field DVF, one image can be transformed to the spatial position of another image, thereby achieving image registration.

[0050] 5) Voxel: The basic component of a three-dimensional image, analogous to the pixel of a two-dimensional image.

[0051] In the relevant technical background, deformable image registration (DIR) is often used in the field of medical image analysis. Specifically, given a pair of three-dimensional medical images (including: a fixed image to be registered (Target Image) and a floating image / moving image (Moving Image) corresponding to the image to be registered), image registration is used to provide a pixel-by-pixel correspondence between the fixed image and the moving image, thereby facilitating a variety of applications such as disease tracking, surgical guidance, clinical diagnosis, attribute transfer, statistical shape analysis, and treatment effect evaluation. The goal of deformable image registration is to find the optimal displacement vector field (also called deformation field (Displacement vector field, DVF)) between the fixed image and the moving image. The floating image is deformed by the estimated deformation field so that the anatomical structure of the deformed moving image is aligned pixel by pixel with the fixed image.

[0052] In traditional technical solutions, it mainly relies on a large number of iterative optimizations to determine the spatial transformation parameters of the DVF, which will lead to slow calculation speed and low efficiency. However, with the rapid development of deep learning, image registration technology based on deep learning has become an alternative to the above-mentioned traditional optimization-based iterative method, with faster calculation speed. Among them, a related technical solution uses a convolutional neural network (CNN) to regress the displacement vector field (i.e., deformation field). It usually uses the network result of U-Net as the backbone architecture, and the convolution layer needs to extract the deep features of the input image and convert the extracted deep features into a dense displacement field.

[0053] However, the above CNN-based image registration method, on the one hand, has limited ability to understand the spatial correlation between features and capture feature dependencies, which easily restricts the system performance. On the other hand, the above CNN-based image registration scheme does not fully consider the relationship between cross-image attention and deformation inference, and does not distinguish the process of feature extraction and displacement vector calculation, which makes the system understand the coordinate system based on regression. Third, in the process of feature matching, it is necessary to establish a dense correspondence between the voxels in the moving image and the voxels in the fixed image, and for each voxel in the fixed image, it is necessary to perform feature matching on the voxels in the moving image search box within a certain search range to find the voxel with the highest feature similarity as the corresponding target voxel. The scope of existing image attention can be divided into global attention and neighborhood attention mechanisms. Since three-dimensional images are usually composed of millions or even tens of millions of voxels, directly applying global attention to high-resolution three-dimensional feature maps will bring huge computational costs, so the neighborhood attention mechanism is more feasible. For example, in the feature matching link, Im2Grid and ModeT usually use a 3×3×3 neighborhood attention range to search for matching features. However, even if the neighborhood attention mechanism is applied to high-resolution three-dimensional feature maps, it will incur huge computational costs, and as the search range continues to increase, the corresponding computational complexity shows an O(n^3) growth trend.

[0054] Therefore, there is an urgent need for an image registration method with low computational complexity and fast computational speed to effectively improve the system performance of deformable image registration.

[0055] In view of the above problems, the exemplary embodiments of the present disclosure propose an image registration method, which can be applied to any application field that requires image registration, such as the field of medical image analysis, the field of aerospace images, and any other field that performs image registration. The method separates the process of feature extraction from the process of feature matching, thereby more effectively utilizing the feature extraction capability and the feature correlation recognition capability of the attention mechanism, and is more efficient in calculating the dense correspondence direction between the voxels of the deformable image to be registered and the fixed reference template image, thereby improving the registration performance. In addition, the method determines the feature maps of the image to be registered and the reference template image at multiple different resolutions, respectively, so as to gradually align the overall structure from the low-resolution feature map to the high-resolution feature map, that is, in the low-resolution feature map, a large displacement is made between the image to be registered and the reference template image, and a rough alignment of the overall structure is established. With the continuous improvement of the resolution of the feature map, the registration network realizes fine anatomical structure alignment, thereby improving the accuracy of image registration.

[0056] For ease of understanding, the following will be combined Figure 1 , an image registration method and device proposed in the present disclosure are applied to Figure 1 The system architecture of the exemplary application environment shown is exemplarily described.

[0057] Figure 1 The following schematically shows an architecture diagram of an image registration system according to the exemplary embodiment; Figure 1 As shown, in this application scenario, a terminal device 101 and a server 102 may be included.

[0058] The terminal device 101 can be, for example, a mobile phone, a tablet computer, a laptop computer, a desktop computer, a smart TV, a smart vehicle-mounted device, a smart wearable device, a smart TV, and an aircraft, and any other device involved in image registration. The terminal device 101 can be installed with a target application, and the target application can have functions such as displaying the image to be registered and the reference template image, and displaying the target image after image registration. The application involved in the embodiment of the present application can be a software client, or a client such as a web page or a small program. The server 102 is a server corresponding to the software or a web page, a small program, etc., and does not limit the specific type of the client.

[0059] The server 102 may be a background server of the target application, used to provide corresponding background services for it, such as image registration services, etc. It may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, i.e., content delivery networks (CDNs), and big data and artificial intelligence platforms, but is not limited thereto.

[0060] It should be noted that the image registration method in the embodiment of the present application can be executed separately by the terminal device 101 or the server 102, or it can be executed jointly by the server 102 and the terminal device 101. When executed separately by the terminal device 101 or the server 102, the model training and the image registration process can be implemented separately by the terminal device 101 or the server 102. In actual application, specific configuration can be performed according to the situation, and this application is not specifically limited here. Among them, the server 102 and the terminal device 101 can both include one or more processors, memories, and interactive I / O interfaces, etc. In addition, the server 102 can also be configured with a database, which can be used to store model parameters obtained by training, etc. Among them, the memory of the server 102 and the terminal device 101 can also store the program instructions required to be executed in the image registration method provided in the embodiment of the present application, and these program instructions can be used to implement the image registration process provided in the embodiment of the present application when executed by the processor.

[0061] It should be noted that when the image registration method provided in the embodiment of the present application is executed by the server 102 or the terminal device 101 alone, the above-mentioned application scenario may also only include a single device of the server 102 or the terminal device 101, or the server 102 and the terminal device 101 may be considered to be the same device. Of course, in actual application, when the image registration method provided in the embodiment of the present application is jointly executed by the server 102 and the terminal device 101, the server 102 and the terminal device 101 may also be the same device, that is, the server 102 and the terminal device 101 may be different functional modules of the same device, or virtual devices virtualized from the same physical device.

[0062] In the embodiment of the present application, the terminal device 101 and the server 102 can be directly or indirectly connected to each other through one or more networks 103. The network 103 can be a wired network or a wireless network. For example, the wireless network can be a mobile cellular network or a wireless fidelity (Wireless-Fidelity, WIFI) network. Of course, it can also be other possible networks, which are not limited in the embodiment of the present application. It should be noted that Figure 1 The figure is only an example. In fact, the number of terminal devices and servers is not limited and is not specifically limited in the embodiments of the present application.

[0063] For example, in an exemplary embodiment, the server 102 obtains an image to be registered and a reference template image, and determines a first feature map of the image to be registered at multiple different resolutions, and a second feature map of the reference template image at multiple different resolutions; for a first sub-feature map of the first resolution and a second sub-feature map of the first resolution in multiple image resolutions, similar feature matching is performed on the first sub-feature map of the first resolution and the second sub-feature map of the first resolution to obtain a first deformation field; deformation registration is performed on the first sub-feature map of the second resolution according to the first deformation field to obtain a third sub-feature map, and similar feature matching is performed on the third sub-feature map and the second sub-feature map of the second resolution to obtain a residual deformation field; the first resolution is smaller than the second resolution; based on the first deformation field and the residual deformation field, the second deformation field is determined until a target deformation field at the highest resolution is obtained, so as to deform and register the image to be registered according to the target deformation field to obtain a registered target image, and then the registered target image is sent to the terminal device 101 for display.

[0064] It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of the present application, and the implementation methods of the present application are not limited in this regard.

[0065] After understanding the application scenarios described above, the method provided by the exemplary embodiment of the present application will be described below with reference to the accompanying drawings. It should be noted that the following method can be executed by the above-mentioned terminal device or server, or can be executed jointly by the terminal device and the server. Here, the server execution is specifically illustrated as an example. Figure 2 A flowchart schematically shows an image registration method in this exemplary embodiment, see Figure 2 The image registration method provided by the embodiment of the present disclosure includes the following steps S201 to S204:

[0066] Step S201: Acquire an image to be registered and a reference template image, and determine a first feature map of the image to be registered at multiple different resolutions, and a second feature map of the reference template image at multiple different resolutions.

[0067] Step S202: For a first sub-feature map of a first resolution and a second sub-feature map of the first resolution among multiple image resolutions, similar feature matching is performed on the first sub-feature map of the first resolution and the second sub-feature map of the first resolution to obtain a first deformation field.

[0068] Step S203: deform and register the first sub-feature map of the second resolution according to the first deformation field to obtain a third sub-feature map, and perform similar feature matching on the third sub-feature map and the second sub-feature map of the second resolution to obtain a residual deformation field.

[0069] The first resolution is smaller than the second resolution.

[0070] Step S204: determine a second deformation field based on the first deformation field and the residual deformation field until a target deformation field at the highest resolution is obtained, and perform deformation registration on the image to be registered according to the target deformation field to obtain a registered target image.

[0071] exist Figure 2In the provided technical solution, an image to be registered and a reference template image are obtained, and a first feature map of the image to be registered at multiple different resolutions and a second feature map of the reference template image at multiple different resolutions are determined; for a first sub-feature map of the first resolution and a second sub-feature map of the first resolution in multiple image resolutions, similar feature matching is performed on the first sub-feature map of the first resolution and the second sub-feature map of the first resolution to obtain a first deformation field; deformation registration is performed on the first sub-feature map of the second resolution according to the first deformation field to obtain a third sub-feature map, and similar feature matching is performed on the third sub-feature map and the second sub-feature map of the second resolution to obtain a residual deformation field; the first resolution is smaller than the second resolution; based on the first deformation field and the residual deformation field, the second deformation field is determined until a target deformation field at the highest resolution is obtained, so that deformation registration is performed on the image to be registered according to the target deformation field to obtain a registered target image. On the one hand, this method separates the feature extraction process from the feature matching process, thereby more effectively utilizing the feature extraction capability and the feature correlation recognition capability of the attention mechanism, and is more efficient in calculating the dense correspondence between the voxels of the deformable image to be registered and the fixed reference template image, thereby improving the computational efficiency and registration performance of deformable image registration. On the other hand, this method determines the feature maps of the image to be registered and the reference template image at multiple different resolutions, respectively, so as to gradually align the overall structure from the low-resolution feature map to the high-resolution feature map, that is, in the low-resolution feature map, a large displacement is made between the image to be registered and the reference template image, and a rough alignment of the overall structure is established. As the resolution of the feature map continues to increase, the registration network achieves fine anatomical structure alignment, thereby improving the accuracy of image registration.

[0072] The following will be combined with specific embodiments to Figure 2 The specific implementation methods of each step in the embodiment shown are described in detail:

[0073] In step S201, an image to be registered and a reference template image are acquired, and a first feature map of the image to be registered at multiple different resolutions and a second feature map of the reference template image at multiple different resolutions are determined.

[0074] The image to be registered is a deformable floating image / moving image (Moving Image), and the reference template image can be an immutable fixed image (Target Image). The reference template image is used as a standard so that the image to be registered with a variable line can be registered with reference to the reference template image.

[0075] Exemplarily, after obtaining a deformable image to be registered and a non-deformable reference template image, multi-resolution feature extraction can be performed on the image to be registered and the reference template image respectively, so as to obtain first feature maps at multiple different resolutions and second feature maps at multiple different resolutions respectively.

[0076] The following will be combined with an embodiment to exemplarily illustrate the process of determining the first feature map of the image to be registered at multiple different resolutions and the second feature map of the reference template image at multiple different resolutions.

[0077] In an optional embodiment of the present disclosure, the image to be registered and the reference template image are input into a convolutional network, and the convolutional network includes at least n first convolutional modules and n second convolutional modules; based on the superposition of j first convolutional modules among the n first convolutional modules, feature extraction is performed on the image to be registered to obtain n first feature maps at different resolutions; and, based on the superposition of j second convolutional modules among the n second convolutional modules, feature extraction is performed on the reference template image to obtain n second feature maps at different resolutions.

[0078] Among them, the number of j first convolution modules / second convolution modules is less than or equal to the number of n first convolution modules / second convolution modules, that is, j can be: j=1,2,...,n. And the number of first convolution modules is the same as the number of second convolution modules. For example, if the number of first convolution modules is 5, the number of second convolution modules is also 5.

[0079] For example, based on n convolution modules, by stacking different numbers of convolution modules, feature maps with different resolutions can be obtained. Figure 3A and Figure 3B An exemplary description may be given.

[0080] Take the image to be registered as an example, Figure 3A A schematic diagram schematically shows a method of using a plurality of convolution modules of different numbers to stack and obtain a plurality of first feature maps of different resolutions for an image to be registered in this exemplary embodiment; Figure 3AAs shown, the target image registration model includes five first convolution modules, namely, convolution module a1, convolution module a2, convolution module a3, convolution module a4 and convolution module a5. The image to be registered is first input to the convolution module a1 of the first convolution layer, that is, only the convolution module a1 is superimposed, and the first feature map with a resolution of A1 can be obtained; the first feature map with a resolution of A1 is then input to the convolution module a2, that is, the convolution module a1 of the first convolution layer and the convolution module a1 of the second convolution layer are superimposed, and the first feature map with a resolution of A2 can be obtained; and so on, the first feature maps with resolutions of A3 and A4 can be obtained, until the convolution modules a1, convolution modules a2, convolution modules a3, convolution modules a4 and convolution modules a5 of five convolution layers are finally superimposed, and the first feature map with a resolution of A5 can be obtained.

[0081] That is, for the image to be registered, by superimposing different numbers of convolution modules to extract its features, multiple first feature maps with different resolutions can be obtained. And as the number of superimposed convolution modules increases, the resolution of the corresponding first feature map becomes lower. Figure 3A As shown, resolution A1>resolution A2>resolution A3>resolution A4>resolution A5.

[0082] Correspondingly, for the reference template image, Figure 3B Schematically illustrating a schematic diagram of using a plurality of convolution modules of different numbers to stack and obtain a plurality of second feature maps of different resolutions for a reference template image in this exemplary embodiment; Figure 3B As shown, the target image registration model includes five second convolution modules, namely, convolution module b1, convolution module b2, convolution module b3, convolution module b4 and convolution module b5. The reference template image is first input to the convolution module b1 of the first convolution layer (i.e., the first superposition layer), that is, only the convolution module b1 is superimposed, and the second feature map with a resolution of B1 can be obtained; the first feature map with a resolution of B1 is then input to the convolution module b2, that is, the convolution module b1 of the first convolution layer and the convolution module b1 of the second convolution layer are superimposed, and the second feature map with a resolution of B2 can be obtained; and so on, the second feature maps with resolutions of B3 and B4 can be obtained, until the convolution modules b1, convolution modules b2, convolution modules b3, convolution modules b4 and convolution modules b5 of the five convolution layers are finally superimposed, and the second feature map with a resolution of B5 can be obtained.

[0083] That is, for the reference template image, by superimposing different numbers of convolution modules to extract its features, multiple second feature maps with different resolutions can be obtained. And as the number of superimposed convolution modules increases, the resolution of the corresponding second feature map becomes lower. Figure 3B As shown, resolution B1>resolution B2>resolution B3>resolution B4>resolution B5.

[0084] It should be explained that the first convolution module and the second convolution module can be the same convolution module or different convolution modules. When the first convolution module and the second convolution module are the same convolution module, convolution module a1, convolution module a2, convolution module a3, convolution module a4 and convolution module a5 are convolution module b1, convolution module b2, convolution module b3, convolution module b4 and convolution module b5 respectively.

[0085] In this embodiment, by superimposing different numbers of convolution modules to extract features from the image, the feature extraction capability of convolution can be fully utilized to subsequently calculate the dense correspondence between the voxels of the image to be registered and the reference template image, thereby improving the registration performance.

[0086] In an optional embodiment of the present disclosure, the weight values ​​of the convolution modules in the same superposition layer among the n first convolution modules and the n second convolution modules are the same.

[0087] On the basis of the above embodiment, in the process of feature extraction for the image to be registered and the reference template image shown in the above embodiment, the weight values ​​of the convolution modules in the same superposition layer among the n first convolution modules and the n second convolution modules can be made the same. The following will be described in combination with two different situations:

[0088] Embodiment 1: In order to make the resolution of the first feature map and the second feature map outputted in the same superposition layer equal when extracting features from the image to be registered and the reference template image, and to save resources, the first convolution module and the second convolution module can be configured to be the same convolution module. For example, the following is used for both the image to be registered and the reference template image: Figure 3A The convolution structure of convolution module a1, convolution module a2, convolution module a3, convolution module a4 and convolution module a5 is used for feature extraction, and the convolution modules corresponding to each stacking layer in the convolution structure are set unchanged. For example, the first stacking layer is all convolution module a1, the second stacking layer is all convolution module a2, ..., and the fifth stacking layer is all convolution module a5. At this time, since the convolution modules in the same stacking layer are the same, the weight values ​​of the convolution modules in the same stacking layer can be made the same.

[0089] Embodiment 2: When the first convolution module and the second convolution module are different convolution modules, for example, in order to make the resolution of the first feature map and the second feature map outputted in the same superposition layer equal when extracting features for the image to be registered and the reference template image, the weight values ​​of the convolution modules in the same superposition layer can be made the same. For example, Figure 3A The convolutional module a1 shown is Figure 3BThe convolution module b1 shown is in the same superposition layer, and the weight values ​​configured for the convolution modules a1 and b1 are the same, so that the resolution A1 based on the output of the convolution module a1 is equivalent to the resolution B1 based on the output of the convolution module b1 (i.e., the same or similar), so as to facilitate subsequent image deformation registration. Similarly, the weight values ​​configured for the convolution modules a2 and b2, ..., and a5 and b5 in the same superposition layer are the same. It should be explained that the weight values ​​configured for the convolution modules in different superposition layers are usually different.

[0090] It should be understood that for the above single convolutional modules, each convolutional module contains an average pooling operation (except for the feature map at the lowest resolution) and two submodules. Each submodule includes a 3D convolution layer, an instance normalization layer, and a leaky ReLU activation function.

[0091] In step S202, for a first sub-feature map of a first resolution and a second sub-feature map of the first resolution among multiple image resolutions, similar feature matching is performed on the first sub-feature map of the first resolution and the second sub-feature map of the first resolution to obtain a first deformation field.

[0092] Among them, for the lower first resolution, similar feature matching is performed on the first sub-feature map of the first resolution and the second sub-feature map of the first resolution, so as to process the large displacement between the image to be registered and the reference template image in the low-resolution pyramid level, thereby preferentially establishing a rough alignment of the overall structure.

[0093] Exemplarily, the first deformation field may be obtained by performing similar feature matching on the first sub-feature map of the first resolution and the second sub-feature map of the first resolution.

[0094] For example, Figure 3A , Figure 3B Assume that the image to be registered is I m , the reference template image is I f , the first feature maps of multiple different resolutions extracted in step S201 and multiple second feature maps of different resolutions Among them, i∈{1,2,3,4,5}.

[0095] Taking the lowest resolution as the first resolution as an example, and Similar feature matching is performed respectively and the corresponding target voxels (voxels corresponding to the features with the highest similarity) are identified, thereby generating an initial lowest resolution DVF estimate, namely the first deformation field ψ1.

[0096] With respect to the process of performing similar feature matching on the first sub-feature map of the first resolution and the second sub-feature map of the first resolution to obtain the first deformation field in step S202, the first sub-feature map of the first resolution and the second sub-feature map of the first resolution may be linearized and normalized respectively to obtain the processed first sub-feature map of the first resolution and the second sub-feature map of the first resolution; for each voxel in the processed first sub-feature map of the first resolution, the feature of the voxel is compared with the feature of the adjacent voxel in the processed second sub-feature map of the first resolution to obtain a correlation weight parameter, and then the first deformation field is calculated based on the correlation weight parameter.

[0097] Combination Figure 3A , Figure 3B The lowest resolution and Take similar feature matching as an example, first and Linearization and normalization (LN) processing is performed respectively to obtain the processed first sub-feature map Q1 of the first resolution and the second sub-feature map K1 of the first resolution. By traversing each voxel p in Q1, the feature correlation between Q1 and K1 is compared, specifically, the feature correlation between a voxel in Q1 and a voxel in the adjacent position of a voxel in K1 is compared, so as to obtain a correlation weight parameter, and then the first deformation field is calculated based on the correlation weight parameter. The execution process of this process can refer to the following implementation steps of performing similar feature matching on the third sub-feature map and the second sub-feature map of the second resolution in step S203 to obtain the residual deformation field, which will not be described in detail here.

[0098] In step S203, deformation registration is performed on the first sub-feature map of the second resolution according to the first deformation field to obtain a third sub-feature map, and similar feature matching is performed on the third sub-feature map and the second sub-feature map of the second resolution to obtain a residual deformation field.

[0099] The first resolution is smaller than the second resolution. The first resolution among the plurality of image resolutions is a resolution lower than the second resolution. For example, the first resolution is Figure 3A , Figure 3B The medium resolution range is A5, and the resolution range is B5, the second resolution can be Figure 3A , Figure 3B The medium resolution range is A4, and the resolution range is B4.

[0100] Exemplarily, based on the first deformation field corresponding to the first resolution of the above embodiment, the first sub-feature map of higher resolution (i.e., second resolution) can be deformed and registered according to the first deformation field, so as to roughly align the overall structure with reference to the reference template image to obtain the third sub-feature map. Continuing to refer to the above embodiment, according to the first deformation field ψ1, the first sub-feature map of the second resolution (e.g., A2) is deformed and registered. Perform deformation registration to obtain the third sub-feature map Then the third sub-feature map and the second sub-feature map of the second resolution (e.g. B2) Similar feature matching is performed to obtain the corresponding residual deformation field △ψ2.

[0101] After the first sub-feature map of the second resolution is deformed and registered based on the first deformation field to obtain the third sub-feature map, the third sub-feature map can be matched with the second sub-feature map of the second resolution for similar features to obtain the residual deformation field. Figure 4 The above process is described in detail.

[0102] Figure 4 A flowchart of a similar feature matching method in this exemplary embodiment is schematically shown. Figure 4 In an optional embodiment of the present disclosure, the process of performing similarity matching between the third sub-feature map and the second sub-feature map of the second resolution in the above step S203 to obtain the residual deformation field includes the following steps S401-S402:

[0103] Step S401: for each target voxel in the second sub-feature map of the second resolution, determine the feature correlation between the target voxel and the adjacent voxels in the third sub-feature map, and obtain the correlation weight parameter of each voxel.

[0104] Step S402: Determine the residual deformation field of each voxel based on the correlation weight parameter.

[0105] First, in the pre-trained first image registration model, linearization and normalization processing can be performed through a linear layer and a layer normalization, respectively, so as to obtain a processed third sub-feature map and a second sub-feature map of a second resolution.

[0106] Continue with the third sub-feature map and the second sub-feature map of the second resolution (e.g. B2) Taking similar feature matching as an example, linearization and normalization processing can be performed based on the following formula (1) to obtain the processed third sub-feature map Q and the processed second sub-feature map K of the second resolution:

[0107]

[0108] In formula (1), LN function is the normalization processing function, and Line function is the linearization processing function; is a second sub-feature map of the reference template image at the second resolution before being processed; It is the third sub-feature map of the image to be registered after deformation registration based on the first deformation field and before processing.

[0109] Then, traverse each target voxel p in the processed third sub-feature map, i.e., Q p ∈R l×c And the processed third sub-feature map K pd ∈R n×c The feature correlation between them is used to obtain the correlation weight parameter C of each voxel. pd Among them, Q p is the target voxel p of the processed third sub-feature map Q, the rows and columns corresponding to Q are l and c respectively, K pd is the voxel at the position corresponding to the offset d of the adjacent position of the second sub-feature map of the second resolution after processing at the position where the target voxel p is located; n and c are K pd The corresponding rows and columns.

[0110] In an optional embodiment of the present disclosure, if the target voxel is a voxel at a boundary position in a second sub-feature map of a second resolution, the third sub-feature map is voxel-filled to determine the feature correlation between the target voxel at the boundary position and the adjacent voxels filled with voxels in the third sub-feature map, and obtain the correlation weight parameters of each voxel.

[0111] Exemplarily, when the target voxel is a voxel at a boundary position in the second sub-feature map of the second resolution, the adjacent voxels of the target voxel are empty in the third sub-feature map. At this time, the third sub-feature map can be voxel-filled, such as zero-filled, to perform feature matching on voxels at all coordinate positions in the second sub-feature map of the second resolution to obtain the correlation weight parameters of each voxel.

[0112] The correlation weight parameter C of each voxel pd The calculation process is shown in the following formula (2):

[0113]

[0114] Furthermore, based on the above steps, the correlation weight parameter C of each voxel is obtained. pd After that, the residual deformation field of each voxel can be determined.

[0115] In an optional embodiment of the present disclosure, the correlation weight parameter C may be selected to determine the voxel position in K with the target pixel p of Q having the highest similarity. pdThe position of the maximum value in the residual deformation field ψ p That is, the following formula (3) is used:

[0116] ψ p = argmax(C pd ) Formula (3)

[0117] In the above embodiment, since the argmax operation is not differentiable, it is not possible to simply and directly determine the voxel position in K that has the highest similarity to the target pixel p in Q. For this, a differentiable softmax operation can be used to calculate the weighted average of the two-dimensional coordinates as the voxel position with the highest similarity.

[0118] In an optional embodiment of the present disclosure, the residual deformation field of each voxel is determined based on formula (4):

[0119]

[0120] In formula (4), ψ p is the residual deformation field of each target voxel p; the softmax function is a normalized exponential function; C pd ∈R n×3 is the correlation weight parameter between the adjacent d voxels in the second sub-feature map of the second resolution after processing for each target voxel p, where pd represents the expanded neighboring position; G pd ∈R n×3 is the relative position coordinate vector relative to each target voxel p; c is the feature dimension of the second sub-feature map.

[0121] Among them, the expanded neighboring position pd(j)∈N d (p), where N d (p) is defined as shown in the following formula (5):

[0122]

[0123] In formula (5), (x′, y′, z′) is the coordinate value of the voxel.

[0124] In this embodiment, the coefficient sampling mechanism of attention can be implemented by expanding neighboring positions, thereby increasing the receptive field of the image while having linear computational complexity. Compared with the existing global attention mechanism or neighborhood attention mechanism, the computational burden can be effectively reduced.

[0125] The following will be combined Figure 5 (a)- Figure 5 (e) Comparison between the existing global attention mechanism or neighborhood attention mechanism and the expanded attention mechanism under the expanded neighboring position d provided by the embodiments of the present disclosure.

[0126] Figure 5 (a) is a schematic diagram of correlation feature matching under the existing global attention mechanism; Figure 5 (b) is a schematic diagram of correlation feature matching under the existing neighborhood attention mechanism; Figure 5 (c)- Figure 5 (e) Schematic diagram of correlation feature matching of the expanded attention at different expansion rates d provided in an embodiment of the present disclosure.

[0127] Depend on Figure 5 (a) Figure 5 (b) It can be seen that the existing global attention mechanism and neighborhood attention mechanism use a smaller local search window for feature matching, which has the technical problems of small receptive field and high complexity. Figure 5 (c)- Figure 5 The expanded attention mechanism shown in (e) has a larger receptive field and a larger local search window compared to the above-mentioned prior art, thereby reducing the computational complexity.

[0128] In step S204, a second deformation field is determined based on the first deformation field and the residual deformation field until a target deformation field at the highest resolution is obtained, so as to perform deformation registration on the image to be registered according to the target deformation field to obtain a registered target image.

[0129] For example, based on the first deformation field and the residual deformation field, the second deformation field can be determined until the highest resolution (eg Figure 3A , Figure 3B The target deformation field ψ5 of the resolution A1 and B1 in the image is thus treated according to the target deformation field ψ5. m Perform deformation registration to obtain the registered target image The specific formula is as follows (6):

[0130]

[0131] In an optional embodiment of the present disclosure, after obtaining the target image after image registration, a loss value can be calculated to optimize and adjust the parameters in the image matching process based on the loss value.

[0132] In an optional embodiment of the present disclosure, a similarity loss parameter between a target image and a reference template image is determined, as well as a displacement offset loss parameter of a target deformation field in space is determined; the similarity loss parameter and the displacement offset loss parameter are superimposed to obtain a target loss parameter between the target image and the reference template image; and based on the target loss parameter, parameters for similar feature matching are adjusted.

[0133] Exemplarily, the loss function consists of two parts: the similarity loss parameter L during image alignmentalign , and the deformation field (DVF) smoothing regularization term (i.e., displacement offset loss parameter) L reg , then the calculation process of the target loss parameter L is shown in the following formula (7):

[0134] L=L align +L reg Formula (7)

[0135] In formula (7), L align Used to measure the reference template image I f and the deformed target image The difference between them. Using image similarity loss, denoted as Its L align The corresponding formula is as follows:

[0136]

[0137] In formula (8), α align is the weight of the similarity loss function.

[0138] The above displacement loss parameter, that is, the smoothing regularization term L reg , which enforces the smoothness of the multiresolution DVF by penalizing large differences in displacement vectors between neighboring voxels. This is done by a Frobenius norm-based regularizer that applies i , i∈{1,2,3,4,5} is constrained by the spatial gradient:

[0139]

[0140] In formula (9), α reg is the weight coefficient used to adjust the regularization strength at different resolutions; is the calculation of spatial gradient; ψ i is the spatial gradient is estimated by ψ i It is realized by numerical difference along the x, y, and z axes; L is introduced reg It can ensure the spatial consistency and rationality of the estimated deformation field DVF and avoid abrupt or unrealistic deformation of the anatomical structure.

[0141] The following will refer to Figure 6 The entire image registration process of the image registration method according to the exemplary embodiment of the present disclosure is described in detail.

[0142] Reference Figure 6As shown, the feature extraction process adopts the superposition of 5 convolution modules to obtain the first feature map and the second feature map of 5 different resolutions. The feature matching (i.e., feature similarity matching) process adopts the image registration model structure formed by the dilated matching module (DMM) based on the dilated attention as an example.

[0143] First, the floating image, i.e. the image to be registered And, the fixed image is the reference template image Input into 5 convolutional blocks (CB) respectively to obtain the image to be registered I m The first feature map corresponding to 5 different resolutions And the reference template image I f Corresponding to the second feature map of 5 different resolutions Among them, i∈{1,2,3,4,5}.

[0144] It needs to be explained that the first feature maps of 5 different resolutions middle, The image resolution is the lowest. The image resolution is the highest; accordingly, the second feature maps of 5 different resolutions middle, The image resolution is the lowest. The image has the highest resolution.

[0145] Then, starting from the lowest resolution feature map (i=1), a coarse-to-fine deformation field inference is performed to treat the registration image I m A gradual transition is made from coarse image registration to more refined image registration, thereby improving the accuracy of image registration.

[0146] Specifically, the smallest resolution and The feature maps are respectively input into the dilated attention matching module (DMM module) for similarity feature matching, and the voxel position with the highest similarity is determined to obtain the corresponding deformation field ψ1.

[0147] Then use a feature map with a higher resolution than the minimum resolution above and The deformation field ψ1 obtained above is modified and adjusted. Specifically, the deformation field ψ1 is upsampled in the second stacking layer, and the deformation field ψ1 is used to adjust the feature map Perform deformation registration and generate

[0148] Then, and The residual deformation field △ψ2 is input into the dilated attention matching module (DMM module) for similarity feature matching to obtain the residual deformation field △ψ2 of the deformation field. Finally, the residual deformation field △ψ2 is combined with the deformation field ψ1 of the first layer to obtain the deformation field ψ2 of the second superimposed layer.

[0149] By analogy, the high-resolution deformation field is updated based on the low-resolution deformation field until the highest resolution corresponding The final deformation field ψ5 is used to treat the registered image I based on the final deformation field ψ5. m Perform deformation registration to obtain the registered target image. It can be seen that:

[0150]

[0151] In formula (10), UP is the upsampling layer.

[0152] For the above-mentioned dilated attention matching module (hereinafter referred to as dilated matching module, DMM), the following will be combined with Figure 7 Provide detailed explanation.

[0153] Figure 7 A schematic diagram of a DMM network structure in a feature matching in this exemplary embodiment is shown schematically; Figure 7 As shown, based on Figure 5 (c)- Figure 5 The dilated attention pair shown in (e) and Perform an extended attention search to find features with similarity matching; first, the feature map is linearized and normalized through a linear layer and layer normalization to obtain Q and K, respectively. Then, for The target feature of each target voxel in By expanding attention Search, that is, obtain the corresponding correlation weight parameter C through matrix multiplication pd .

[0154] In order to determine the voxel position with the highest similarity to the target voxel, a softmax function can be used for differentiable processing to obtain the deformation field ψ of the target voxel at each layer based on the above formula (4): i (p) to obtain the deformation field ψ of each voxel in each layer i So that the registration image I can be treated by the deformation field ψ5 corresponding to the highest resolution m Perform deformation registration to obtain the registered target image.

[0155] Reference Figure 7As shown, in an optional embodiment of the present disclosure, in order to further optimize the expansion matching module DMM and enhance the retention of the topology structure, an aggregation module can also be introduced into each DMM module.

[0156] The aggregation module can be composed of a 3×3×3 convolutional layer and several scaling and squaring layers. The convolutional layer infers the displacement vector by considering the neighboring displacement vectors, while the scaling and squaring layers are used to ensure the differential homeomorphism of DIR.

[0157] In order to implement the above-mentioned image registration method, an image registration device is provided in one embodiment of the present disclosure. Figure 8 The schematic diagram of the architecture of the image registration device is schematically shown.

[0158] The image registration device 800 includes a feature map determination module 801 , a feature matching module 802 , a first deformation registration module 803 and a second deformation registration module 804 .

[0159] The feature map determination module 801 is configured to execute acquisition of the image to be registered and the reference template image, and determine a first feature map of the image to be registered at multiple different resolutions, and a second feature map of the reference template image at multiple different resolutions;

[0160] The feature matching module 802 is configured to perform similar feature matching on the first sub-feature map of the first resolution and the second sub-feature map of the first resolution to obtain a first deformation field;

[0161] The first deformation registration module 803 is configured to perform deformation registration on the first sub-feature map of the second resolution according to the first deformation field to obtain a third sub-feature map, and perform similarity matching on the third sub-feature map and the second sub-feature map of the second resolution to obtain a residual deformation field; the first resolution is smaller than the second resolution;

[0162] The second deformation registration module 804 is configured to determine the second deformation field based on the first deformation field and the residual deformation field until a target deformation field at the highest resolution is obtained, so as to perform deformation registration on the image to be registered according to the target deformation field to obtain a registered target image.

[0163] In an optional embodiment of the present disclosure, the feature map determination module 801 is specifically used to input the image to be registered and the reference template image into a convolutional network, wherein the convolutional network includes at least n first convolution modules and n second convolution modules; based on the superposition of j first convolution modules in the n first convolution modules, feature extraction is performed on the image to be registered to obtain n first feature maps at different resolutions; and, based on the superposition of j second convolution modules in the n second convolution modules, feature extraction is performed on the reference template image to obtain n second feature maps at different resolutions; wherein the number of j first convolution modules / second convolution modules is less than or equal to the number of n first convolution modules / second convolution modules.

[0164] In an optional embodiment of the present disclosure, the weight values ​​of the convolution modules in the same superposition layer among the n first convolution modules and the n second convolution modules are the same.

[0165] In an optional embodiment of the present disclosure, the first deformation registration module 803 is specifically used to determine, for each target voxel in the second sub-feature map of the second resolution, the feature correlation between the target voxel and the adjacent voxels in the third sub-feature map, and obtain the correlation weight parameters of each voxel; based on the correlation weight parameters, determine the residual deformation field of each voxel.

[0166] In an optional embodiment of the present disclosure, the first deformation registration module 803 is specifically configured to determine the residual deformation field of each voxel based on the formula:

[0167]

[0168] Among them, ψ p is the residual deformation field of each target voxel p; the softmax function is a normalized exponential function; C pd G is the correlation weight parameter between the adjacent d voxels in the third sub-feature map for each target voxel p; pd is the relative position coordinate vector relative to each target voxel p; c is the feature dimension of the second sub-feature map.

[0169] In an optional embodiment of the present disclosure, the device also includes a voxel filling module, which is used to perform voxel filling on the third sub-feature map when the target voxel is a voxel at a boundary position in the second sub-feature map of the second resolution, so as to determine the feature correlation between the target voxel at the boundary position and the adjacent voxel filled in the third sub-feature map, and obtain the correlation weight parameter of each voxel.

[0170] In an optional embodiment of the present disclosure, the device also includes a loss determination module and a loss adjustment module, wherein the loss determination module is used to determine the similarity loss parameters between the target image and the reference template image, and determine the displacement offset loss parameters of the target deformation field in space; superimpose the similarity loss parameters and the displacement offset loss parameters to obtain the target loss parameters between the target image and the reference template image; the loss adjustment module is used to adjust the parameters of similar feature matching based on the target loss parameters.

[0171] The image registration device 800 provided in the embodiment of the present disclosure can execute the technical solution of the image registration method in any of the above-mentioned embodiments. Its implementation principle and beneficial effects are similar to those of the image registration method. Please refer to the implementation principle and beneficial effects of the image registration method, and no further details will be given here.

[0172] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present invention may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of the present specification.

[0173] The program product for implementing the above method according to an embodiment of the present invention may adopt a portable compact disc read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device.

[0174] The program product may adopt any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0175] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0176] The program code contained on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, radio frequency (RF), etc., or any suitable combination of the foregoing.

[0177] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet Service Provider).

[0178] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0179] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0180] Refer to the following Fig. 9 An electronic device 900 according to this embodiment of the present invention will be described. Fig. 9 The electronic device 900 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0181] like Fig. 9 As shown, the electronic device 900 is in the form of a general computing device. The components of the electronic device 900 may include, but are not limited to: the at least one processing unit 910, the at least one storage unit 920, a bus 930 connecting different system components (including the storage unit 920 and the processing unit 910), and a display unit 940.

[0182] The storage unit stores program codes, which can be executed by the processing unit 910, so that the processing unit 910 performs the steps of various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification. For example, the processing unit 910 can perform the following steps: Figure 2 Steps S201 to S204 shown in FIG.

[0183] The storage unit 920 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 9201 and / or a cache storage unit 9202 , and may further include a read-only storage unit (ROM) 9203 .

[0184] The storage unit 920 may also include a program / utility 9204 having a set (at least one) of program modules 9205, such program modules 9205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0185] Bus 930 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0186] The electronic device 900 may also communicate with one or more external devices 1000 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 900, and / or communicate with any device that enables the electronic device 900 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 950. In addition, the electronic device 900 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 960. As shown, the network adapter 960 communicates with other modules of the electronic device 900 via a bus 930. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, disk arrays (Redundant Arrays of Independent Disks, RAID) systems, tape drives, and data backup storage systems.

[0187] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0188] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0189] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0190] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0191] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image registration method, characterized in that: include: Acquire an image to be registered and a reference template image, and determine a first feature map of the image to be registered at multiple different resolutions, and a second feature map of the reference template image at multiple different resolutions; For a first sub-feature map of a first resolution and a second sub-feature map of the first resolution, performing similar feature matching on the first sub-feature map of the first resolution and the second sub-feature map of the first resolution to obtain a first deformation field; Performing deformation registration on the first sub-feature map of the second resolution according to the first deformation field to obtain a third sub-feature map, and performing similarity matching on the third sub-feature map and the second sub-feature map of the second resolution to obtain a residual deformation field; the first resolution is smaller than the second resolution; Based on the first deformation field and the residual deformation field, a second deformation field is determined until a target deformation field at the highest resolution is obtained, so as to perform deformation registration on the image to be registered according to the target deformation field to obtain a registered target image.

2. The method according to claim 1, characterized in that The determining of the first feature map of the image to be registered at a plurality of different resolutions and the second feature map of the reference template image at a plurality of different resolutions comprises: Inputting the image to be registered and the reference template image into a convolutional network, wherein the convolutional network comprises at least n first convolutional modules and n second convolutional modules; According to the superposition of j first convolution modules in the n first convolution modules, feature extraction is performed on the image to be registered to obtain n first feature maps at different resolutions; and according to the superposition of j second convolution modules in the n second convolution modules, feature extraction is performed on the reference template image to obtain n second feature maps at different resolutions; Among them, the number of the j first convolution modules / second convolution modules is less than or equal to the number of the n first convolution modules / second convolution modules.

3. The method according to claim 2, characterized in that The weight values ​​of the convolution modules in the same superposition layer among the n first convolution modules and the n second convolution modules are the same.

4. The method according to claim 1, characterized in that: The performing similarity feature matching on the third sub-feature map and the second sub-feature map of the second resolution to obtain a residual deformation field includes: For each target voxel in the second sub-feature map of the second resolution, determining a feature correlation between the target voxel and an adjacent voxel in the third sub-feature map, and obtaining a correlation weight parameter of each voxel; Based on the correlation weight parameter, a residual deformation field of each voxel is determined.

5. The method according to claim 4, characterized in that The determining of the residual deformation field of each voxel based on the correlation weight parameter comprises: The residual deformation field of each voxel is determined based on the formula: Among them, ψ p is the residual deformation field of each target voxel p; the softmax function is a normalized exponential function; C pd is a correlation weight parameter between d adjacent voxels in the second sub-feature map of the second resolution for each target voxel p; pd is the relative position coordinate vector relative to each target voxel p; c is the feature dimension of the second sub-feature map.

6. The method according to claim 4, characterized in that The method further comprises: If the target voxel is a voxel at a boundary position in the second sub-feature map of the second resolution, voxel filling is performed on the third sub-feature map to determine the feature correlation between the target voxel at the boundary position and the adjacent voxels filled with voxels in the third sub-feature map, and obtain the correlation weight parameter of each voxel.

7. The method according to claim 1, characterized in that The method further comprises: Determining a similarity loss parameter between the target image and the reference template image, and determining a displacement offset loss parameter of the target deformation field in space; Superimposing the similarity loss parameter and the displacement offset loss parameter to obtain a target loss parameter between the target image and the reference template image; Based on the target loss parameter, the parameters of the similar feature matching are adjusted.

8. An image registration device, characterized in that: The device comprises: A feature map determination module is configured to acquire an image to be registered and a reference template image, and determine a first feature map of the image to be registered at a plurality of different resolutions, and a second feature map of the reference template image at a plurality of different resolutions; a feature matching module configured to perform similar feature matching on a first sub-feature map of a first resolution and a second sub-feature map of the first resolution to obtain a first deformation field; a first deformation registration module configured to perform deformation registration on the first sub-feature map of the second resolution according to the first deformation field to obtain a third sub-feature map, and perform similarity feature matching on the third sub-feature map and the second sub-feature map of the second resolution to obtain a residual deformation field; the first resolution is smaller than the second resolution; The second deformation registration module is configured to determine a second deformation field based on the first deformation field and the residual deformation field until a target deformation field at the highest resolution is obtained, so as to perform deformation registration on the image to be registered according to the target deformation field to obtain a registered target image.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image registration method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to perform the image registration method according to any one of claims 1 to 7 by executing the executable instructions.

Images