Deep learning-based image registration method

Abstract

A deep learning-based image registration method. The method comprises: S1, screening for a reference image and a plurality of images to be registered, and selecting a plurality of candidate points on each image to be registered; S2, cropping each image to be registered and the reference image to obtain a plurality of image tile pairs; S3, constructing a deep homography estimation model, and inputting each image tile pair into a trained deep homography estimation model to acquire a homography estimation matrix corresponding to each pair of image tiles; S4, using the homography estimation matrix to calculate the positions, on the reference image, of the plurality of candidate points on each image to be registered, so as to obtain a plurality of control point pairs; and S5, on the basis of the plurality of control point pairs, constructing a triangular mesh and using affine transformation to perform correction, so as to achieve image registration. The method is applicable to image registration in low-texture, weak-texture, and single-texture scenarios, and uses the deep homography estimation model and affine transformation to implement registration between remote sensing images, thereby improving registration accuracy and efficiency.

Description

A deep learning-based image registration method TECHNICAL FIELD

[0001] The present application relates to the technical field of image preprocessing, in particular to a deep learning-based image registration method. BACKGROUND

[0002] Image registration is a technology for realizing image geometric calibration, which obtains the mapping relationship between the reference image and the image to be registered by calculating the homography matrix. In image registration, the homography matrix plays a crucial role. By finding corresponding feature points in multiple images and calculating the homography matrix corresponding to these points, image registration and reconstruction can be achieved.

[0003] Traditional image registration methods mainly include pixel-based methods and feature-based methods. In the feature-based image registration method, when using a feature point detection operator to detect images, a large number of trivial and clustered feature points often appear. The homography matrix calculated based on these feature points will over-describe the local area of the image, resulting in low registration quality. However, this method is not suitable for low-texture images. Currently, there are few studies on image registration methods for low-texture, weak-texture, and single-texture scenes. In the pixel-based image registration method, the pixel values of the pixel points in the image are directly used to estimate the transformation relationship between the images. First, the homography matrix between the images is initialized. Then, the homography matrix is used to perform image transformation on each image, and the pixel value error of the pixel points in the transformed image is calculated. Finally, optimization techniques are used to minimize the error function to achieve image registration. This method has good results for low-texture images, but it is less robust to scale, brightness changes, and rotation images.

[0004] Homography matrix estimation is a key step in image registration. With the rapid development of deep learning technology, deep learning has also been applied to image registration. The basic idea is to learn a homography matrix estimation model that can be used for image registration, so that the estimated homography matrix of the image pair to be registered can be used to achieve image registration of the image pair to be registered. Currently, in supervised learning-based image registration methods, sample labeling is required, which is costly and often difficult to ensure the accuracy of the labeling. SUMMARY

[0005] To solve the above technical problems, the present application provides a deep learning-based image registration method, which is suitable for image registration in low-texture, weak-texture, and single-texture scenes. This method uses a deep homography estimation model and affine transformation to achieve image registration. At the same time, the training of the deep homography estimation model does not depend on manually labeled labels, reducing the cost of sample labeling and improving registration accuracy and efficiency.

[0006] The application provides a deep learning-based image registration method, which comprises the following steps:

[0007] S1: selecting a reference image and a plurality of to-be-registered images from an image database, and selecting a plurality of candidate points on each to-be-registered image;

[0008] S2: sequentially performing cutting processing on each to-be-registered image and the reference image based on a preset size to obtain a plurality of pairs of image tiles; each pair of image tiles comprises one reference image tile and one to-be-registered image tile;

[0009] S3: constructing a deep homography estimation model, training the deep homography estimation model based on a training data set, and obtaining a trained deep homography estimation model;

[0010] S4: sequentially inputting each pair of image tiles into the trained deep homography estimation model to obtain a homography estimation matrix corresponding to each pair of image tiles;

[0011] S5: sequentially obtaining the positions of the plurality of candidate points on the reference image on each to-be-registered image by using the homography estimation matrix to obtain a plurality of control point pairs;

[0012] S6: constructing a triangular network based on the plurality of control point pairs, rectifying each triangular network by using affine transformation, and realizing the registration between each to-be-registered image and the reference image.

[0013] Specifically, the deep homography estimation model in step S3 comprises a feature extraction network, a feature enhancement network, a feature fusion network and a homography estimation network; the feature enhancement network comprises a self-attention mechanism unit and a cross-attention mechanism unit; wherein the feature extraction network is connected with the feature enhancement network, the feature enhancement network is connected with the feature fusion network, and the feature fusion network is connected with the homography estimation network.

[0014] Specifically, step S4 further comprises:

[0015] S41: sequentially inputting each pair of image tiles into the feature extraction network in the trained deep homography estimation model to extract a first feature map of the reference image tile and a first feature map of the to-be-registered image tile, respectively;

[0016] S42: calculating a reference position code of the first feature map of the reference image tile and a to-be-registered position code of the first feature map of the to-be-registered image tile, respectively; and calculating a multi-head attention matrix of the first feature map of the reference image tile and the first feature map of the to-be-registered image tile;

[0017] S43 inputs the first feature map of the reference image tile and the corresponding reference position code thereof, the first feature map of the image tile to be registered and the corresponding to-be-registered position code thereof into a feature enhancement network, and calculates the second feature map of the reference image tile and the second feature map of the image tile to be registered in each pair of image tile pairs through a self-attention mechanism unit after the self-structure information of the reference image tile is enhanced;

[0018] S44 calculates the third feature map of the reference image tile and the third feature map of the image tile to be registered in each pair of image tile pairs through a cross-structure information enhancement mechanism unit after the cross-structure information of the reference image tile is enhanced;

[0019] S45 inputs the third feature map of the reference image tile and the third feature map of the image tile to be registered into a feature fusion network for feature fusion, to obtain a fourth feature map;

[0020] S46 converts the fourth feature map into a feature vector group and inputs the feature vector group into a homography estimation network, to predict the offset of the four corner points of the image tile to be registered relative to the four corner points of the reference image tile in each pair of image tile pairs, and to obtain a homography estimation matrix corresponding to each pair of image tile pairs.

[0021] Specifically, the method for calculating the multi-head attention matrix in step S42 is as follows:

[0022] The calculation formula of the dot product attention Attention(Q, K, V) is as follows:

[0023] wherein Q, K and V represent the Queue, Key and Value matrices of the dot product attention, d k is the dimension of the Q and K matrices;

[0024] The calculation formula of each attention head head i is as follows: i head i Q i i K i i V )

[0025] wherein W i Q i i K i i V are trainable parameters, representing the matrix applied to obtain Q, K and V by the i-th attention head;

[0026] The calculation formula of the multi-head attention matrix MultiHead(Q, K, V) is: MultiHead(Q, K, V) = Concat(head1,.., headH)WQ n )W 0

[0027] Wherein, W 0 represents a trainable weight matrix for weighting the multi-head attention result.

[0028] Specifically, the method for calculating the second feature map of the reference image tile and the second feature map of the image tile to be registered in each pair of image tile pairs after self-structure information enhancement through the self-attention mechanism unit in step S43 is:

[0029] For the reference image tile and the image tile to be registered in each pair of image tile pairs, the corresponding self-structure enhanced feature information is calculated through the following formula respectively, to obtain the second feature map of the reference image tile and the second feature map of the image tile to be registered after self-structure information enhancement; the calculation formula of the self-structure enhanced feature is: X self = X + MultiHead(X + P X , X + P X , X)

[0030] Wherein, X self is the self-structure enhanced feature of the first feature map of the reference image tile or the first feature map of the image tile to be registered, X is the input first feature map of the reference image tile or the first feature map of the image tile to be registered, and P x is the position encoding of the input first feature map of the reference image tile or the first feature map of the image tile to be registered.

[0031] Specifically, the method for calculating the third feature map of the reference image tile and the third feature map of the image tile to be registered after cross-structure information enhancement through the cross-attention mechanism unit in each pair of image tile pairs in step S44 is:

[0032] For the second feature map of the reference image tile and the second feature map of the image tile to be registered after self-structure information enhancement in each pair of image tile pairs, the corresponding cross-structure enhanced feature information is calculated through the following formula respectively, to obtain the third feature map of the reference image tile and the third feature map of the image tile to be registered after cross-structure information enhancement; the cross-structure enhanced feature information is obtained by adding the cross-structure enhanced feature information residual in the interactive structure enhancement network and the cross-structure enhanced feature information residual, and the specific calculation formula is:

[0033] Wherein is the cross-structure enhanced feature information residual, and the calculation formula is:

[0034] wherein X q is a self-structure enhanced feature of a first feature map of the image tile to be registered, X kv is a self-structure enhanced feature of a first feature map of the reference image tile, P q is a reference position encoding of the first feature map of the reference image tile. kv

[0035] Specifically, the network structure of the feature extraction network in step S3 is ResNet-50; the specific network structure includes: a first convolutional layer, a first BN layer, an activation function layer, a maximum pooling layer, a second convolutional layer, a second BN layer, a third convolutional layer, a third BN layer, a fourth convolutional layer, a fourth BN layer, a fifth convolutional layer, and a fifth BN layer; the number of convolutional kernels of the first to fifth convolutional layers is set to 64, 64, 128, 256, and 512, respectively, and the size of the convolutional kernel is set to 7, 3, 3, 3, and 3, respectively; the step length of the first, second, and third convolutional layers is set to 2, and the step length of the fourth and fifth convolutional layers is set to 1; the kernel void ratio in the fourth and fifth convolutional layers is set to 2 and 4, respectively; the size of the kernel of the maximum pooling layer is set to 3*3, and the step length is set to 2; the first to fifth BN layers all use the batch normalization function, the activation function layer uses the linear rectifier function, and the maximum pooling layer uses the region maximum value pooling function.

[0036] Specifically, the method for training the deep homography estimation model based on the training data set in step S3 includes:

[0037] S31 obtains a plurality of samples in the MS-COCO public data set, selects the middle region of each sample, and cuts out a plurality of reference sample tiles with a resolution of 256*256;

[0038] S32 randomly moves the coordinates of the four corner points of each reference sample tile by the same value in sequence to obtain a plurality of sample tiles to be registered corresponding thereto; the plurality of reference sample tiles and the plurality of sample tiles to be registered corresponding thereto serve as a training data set, and the training data set includes a plurality of pairs of training data, one pair of training data including one reference sample tile and one sample tile to be registered corresponding thereto;

[0039] S33 inputs the plurality of pairs of training data in the training data set into the deep homography estimation model in sequence for iterative training, uses the homography estimation matrix output in the training process to spatially transform the sample tile to be registered on the reference sample tile, and respectively calculates the errors of the corresponding horizontal and vertical coordinates of the four corner points in the sample tile to be registered after spatial transformation and the four corner points in the reference sample tile.

[0040] ​S34 constructs a loss function of the depth homography estimation model, updates the model according to the error until the loss function reaches convergence.

[0041] Compared with the prior art, the application has the following beneficial effects:

[0042] (1) The application combines homography estimation matrix transformation and affine transformation to realize image registration, enhances the feature matching process through global and local features, and improves the accuracy of image registration; the application predicts the offset of four corner points of a to-be-registered image tile relative to four corner points of a reference image tile by constructing a depth homography estimation model, and then obtains a homography matrix; a plurality of control point pairs are obtained by using the homography estimation matrix, and affine transformation is used to realize registration between the to-be-registered image and the reference image; the problem of poor global matching of remote sensing images is solved, and the method is suitable for image registration in low-texture, weak-texture and single-texture scenes;

[0043] (2) The depth homography estimation model constructed by the application includes a feature enhancement network, the self-attention mechanism unit and the cross-attention mechanism unit are introduced into the feature enhancement network, the self-attention mechanism improves the self-attention relationship between different pixel points in the same tile, the cross-attention mechanism establishes a correlation between different pixel points in different tiles; and the features of the two tiles are fused to reduce the loss of the features and better capture the matching between different features; and the global receptive field of the Transformer is used to establish dense matching between the to-be-registered image tile and the reference image tile at the pixel level, so that dense matching can be realized only by a small amount of feature points, and the accuracy and robustness of image registration are improved;

[0044] (3) The method uses the homography estimation matrix for spatial transformation, takes the difference between the changed sample tile and the pixels in the reference sample tile as the loss function of the depth homography estimation model, does not depend on manually labeled labels in the training process, reduces the cost of sample labeling, is an unsupervised homography estimation method, and thus improves the registration accuracy and efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0046] Fig. 1 is a flowchart of the image registration method based on deep learning in the embodiment of the application;

[0047] Fig. 2 is a technical framework diagram of the image registration method based on deep learning in the embodiment of the application;

[0048] Fig. 3 is a structural schematic diagram of a feature extraction network in a deep homography estimation model in an embodiment of the present application;

[0049] Fig. 4 is a structural schematic diagram of a feature enhancement network in a deep homography estimation model in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art belong to the scope of protection of the present application.

[0051] Please refer to Fig. 1, which is a flow chart of an image registration method based on deep learning in an embodiment of the present application. The embodiment of the present application provides an image registration method based on deep learning, which comprises the following steps:

[0052] S1, screening a reference image and multiple images to be registered from an image database, and selecting multiple candidate points on each image to be registered;

[0053] S2, cutting each image to be registered and the reference image based on a preset size in turn to obtain multiple pairs of image tiles; each pair of image tiles comprises one reference image tile and one image to be registered tile.

[0054] In the embodiment of the present application, 120 pixel points are selected as candidate points on the image to be registered according to the 2m resolution from the top-left to the bottom-right area with the same horizontal axis interval, so as to ensure that the candidate points are uniformly distributed at different positions on the image to be registered. Then, a 256*256-sized image tile is cut out near each candidate point, and multiple corresponding 256*256-sized reference image tiles are cut out in the reference image, to form multiple pairs of image tiles.

[0055] S3, constructing a deep homography estimation model, training the deep homography estimation model based on a training data set, and obtaining a trained deep homography estimation model.

[0056] In the embodiment of the present application, the deep homography estimation model in step S3 comprises a feature extraction network, a feature enhancement network, a feature fusion network and a homography estimation network; the feature enhancement network comprises a self-attention mechanism unit and a cross-attention mechanism unit; wherein the feature extraction network is connected with the feature enhancement network, the feature enhancement network is connected with the feature fusion network, and the feature fusion network is connected with the homography estimation network.

[0057] The method for training the deep homography estimation model based on the training data set in step S3 in the embodiment of the application comprises:

[0058] S31 obtains a plurality of samples in the MS-COCO public data set, selects a middle region of each sample, and cuts out a plurality of reference sample tiles with a resolution of 256*256.

[0059] S32 randomly moves the coordinates of the four corner points of each reference sample tile by the same value in sequence to obtain a plurality of to-be-registered sample tiles corresponding thereto; the plurality of reference sample tiles and the plurality of to-be-registered sample tiles corresponding thereto serve as a training data set, and the training data set comprises a plurality of pairs of training data, and each pair of training data comprises one reference sample tile and one to-be-registered sample tile corresponding thereto.

[0060] In the embodiment of the application, the MS-COCO public data set is used to obtain a plurality of samples subjected to rigid transformation mainly in the form of displacement transformation, a middle region of each sample is selected, and a plurality of reference sample tiles with a resolution of 256*256 are cut out; the coordinates of the four corner points of the reference sample tiles are randomly moved by the same value to obtain to-be-registered sample tiles corresponding thereto; wherein the maximum displacement in the x direction and the y direction is 75% of the side length of the reference sample tile, and a total of 50,000 pairs of training data sets of a low-overlap scene with a minimum overlap rate of 10% are obtained.

[0061] S33 inputs the plurality of pairs of training data in the training data set into the deep homography estimation model in sequence for iterative training, uses the homography estimation matrix output in the training process to spatially transform the to-be-registered sample tiles on the reference sample tiles, and respectively calculates the errors of the corresponding horizontal and vertical coordinates of the four corner points in the to-be-registered sample tiles and the four corner points in the reference sample tiles after spatial transformation.

[0062] S34 constructs a loss function of the deep homography estimation model, updates the model according to the errors, and stops until the loss function converges.

[0063] In the embodiment of the application, the loss function of the deep homography estimation model is as follows:

[0064] wherein L represents the error of the corresponding horizontal and vertical coordinates of the corner points, N is 4, represents the i-th corner point of the to-be-registered sample tile, represents the i-th corner point of the reference sample tile.

[0065] S4 inputs each pair of image tiles into the trained deep homography estimation model in sequence to obtain a homography estimation matrix corresponding to each pair of image tiles.

[0066] Please refer to FIG. 2, which is a technical framework diagram of the image registration method based on deep learning in the embodiment of the present application; in the embodiment of the present application, step S4 further comprises:

[0067] S41 input each pair of image tiles into the feature extraction network in the trained deep homography estimation model in turn, and extract the first feature map of the reference image tile and the first feature map of the image tile to be registered respectively.

[0068] In the embodiment of the present application, the feature extraction network is constructed by the ResNet-50 network to extract the first feature map F1' of the reference image tile and the first feature map of the image tile to be registered in each pair of image tiles in turn. Please refer to FIG. 3, which is a structural schematic diagram of the feature extraction network in the deep homography estimation model in the embodiment of the present application; the network structure of the feature extraction network in step S3 is ResNet-50; the number of convolution kernels of the first to fifth convolution layers is set to 64, 64, 128, 256, and 512 in turn, the size of the convolution kernel is set to 7, 3, 3, 3, and 3 in turn, the step length of the first, second, and third convolution layers is set to 2, the step length of the fourth and fifth convolution layers is set to 1, the hole rate of the convolution kernel in the fourth and fifth convolution layers is set to 2 and 4, the size of the kernel of the maximum pooling layer is set to 3x3, and the step length is set to 2; the first to fifth BN layers all adopt the batch normalization function, the activation function layer adopts the linear rectifier function, and the maximum pooling layer adopts the region maximum pooling function.

[0069] In the embodiment of the present application, the above standardization function is as follows:

[0070] Wherein, y represents the batch normalization function, γ L represents the scaling parameter of the Lth BN layer in the trained deep twin convolutional neural network, x L,l represents the lth input into the Lth BN layer in the trained deep twin convolutional neural network, L=1, 2, 3, 4, 5; l=1, 2, 3, …, m; m represents the number of elements in the input set, ∑ represents the summation operation, ε L , β L respectively represent the batch offset parameter and the overall offset parameter of the Lth BN layer in the trained deep twin convolutional neural network.

[0071] The region maximum pooling function is as follows:

[0072] Wherein, h represents the region maximum pooling function, max represents the maximum value operation, the coordinate system takes the top left vertex of the input matrix as the origin, the horizontal right is the positive direction of the horizontal axis, and the vertical downward is the positive direction of the vertical axis, in the coordinate system, represents the maximum pooling function input matrix located at the horizontal coordinate Nx wherein N represents the ordinate of the element value at the point, represents a symbol, i and j respectively represent the abscissa and ordinate of the top left corner of the pooling region in the input matrix of the max-pooling function, and k represents the size of the pooling kernel. y wherein N represents the ordinate of the element value at the point, represents a symbol, i and j respectively represent the abscissa and ordinate of the top left corner of the pooling region in the input matrix of the max-pooling function, and k represents the size of the pooling kernel.

[0073] S42 calculates the reference position encoding of the first feature map of the reference image tile and the to-be-registered position encoding of the first feature map of the to-be-registered image tile respectively, and calculates the multi-head attention matrix of the first feature map of the reference image tile and the first feature map of the to-be-registered image tile.

[0074] In the embodiment of the application, the method for calculating the multi-head attention matrix in step S42 is as follows:

[0075] The calculation formula of the dot product attention Attention(Q, K, V) is as follows:

[0076] wherein Q, K, and V respectively represent the Queue, Key, and Value matrices of the dot product attention, d k is the dimension of the Q and K matrices;

[0077] The calculation formula of each attention head head i is as follows: head i = Attention(QW i Q , KW i K , VW i V )

[0078] wherein W i Q , W i K , and W i V are trainable parameters, and represent the i-th attention head applied to the matrices used to obtain Q, K, and V;

[0079] The calculation formula of the multi-head attention matrix MultiHead(Q, K, V) is as follows: MultiHead(Q, K, V) = Concat(head1,.., head n )W 0

[0080] wherein W 0 represents a trainable weight matrix, and is used to weight the multi-head attention result.

[0081] The values of Q, K and V are obtained by multiplying the eigenvectors by the initialized weights, Q and K are used to measure the similarity between each pixel point in V to determine the matching information between the features, the input of the self-attention mechanism unit is the feature of a single image tile, and the input of the cross-attention mechanism is the feature of two corresponding image tiles, so that the feature of the strong texture region is gradually given to the weak texture region after the feature enhancement module.

[0082] S43 inputs the first feature map of the reference image tile and the corresponding reference position code thereof, the first feature map of the image tile to be registered and the corresponding to-be-registered position code thereof into a feature enhancement network, and calculates the second feature map of the reference image tile and the second feature map of the image tile to be registered in each pair of image tile pairs after self-structure information enhancement through a self-attention mechanism unit.

[0083] In the embodiment of the application, the method for calculating the second feature map of the reference image tile and the second feature map of the image tile to be registered in each pair of image tile pairs after self-structure information enhancement through a self-attention mechanism unit in step S43 is as follows:

[0084] For the reference image tile and the image tile to be registered in each pair of image tile pairs, the corresponding self-structure enhanced feature information is calculated through the following formula respectively, so as to obtain the second feature map of the reference image tile and the second feature map of the image tile to be registered after self-structure information enhancement; the calculation formula of the self-structure enhanced feature is as follows: X self = X + MultiHead (X + P X ,X + P X ,X)

[0085] Wherein X self is the self-structure enhanced feature of the first feature map of the reference image tile or the first feature map of the image tile to be registered, X is the input first feature map of the reference image tile or the first feature map of the image tile to be registered, P x is the position code of the input first feature map of the reference image tile or the first feature map of the image tile to be registered.

[0086] S44 calculates the third feature map of the reference image tile and the third feature map of the image tile to be registered after cross-structure information enhancement through a cross-attention mechanism unit.

[0087] In the embodiment of the application, the method for calculating the third feature map of the reference image tile and the third feature map of the image tile to be registered after cross-structure information enhancement through a cross-attention mechanism unit in step S44 is as follows:

[0088] For each pair of the second feature map of the reference image tile after the self-structure information enhancement and the second feature map of the image tile to be registered in each image tile pair, the corresponding cross-structure enhanced feature information is calculated respectively by the following formula, to obtain the third feature map of the reference image tile after the cross-structure information enhancement and the third feature map of the image tile to be registered; the cross-structure enhanced feature information is obtained by adding the cross-structure enhanced feature information residual and the feedforward network FFN in the cross-structure enhanced network, and the specific calculation formula is:

[0089] Wherein is the cross-structure enhanced feature information residual, and the calculation formula is:

[0090] Wherein X q is the self-structure enhanced feature of the first feature map of the image tile to be registered, X kv is the self-structure enhanced feature of the first feature map of the reference image tile, P q is the registration position code of the first feature map of the image tile to be registered, P kv is the reference position code of the first feature map of the reference image tile.

[0091] Please refer to FIG. 4, which is a structural schematic diagram of the feature enhancement network in the deep homography estimation model in the embodiment of the application; the feature enhancement network is composed of two self-attention mechanism units and two cross-attention mechanism units; by enhancing the image tile features by using the self-structure information enhancement and the cross-structure information enhancement, the cross-attention mechanism is used to effectively guide the alignment of the two modal features.

[0092] In the embodiment of the application, the cross-attention mechanism introduces the position code to maintain the position of the input sequence elements, takes the first feature map of the reference image tile and the first feature map of the image tile to be registered as the input of the feature enhancement network, takes the input first feature map as a two-dimensional patch grid and learns two sets of embeddings, named x embedding and y embedding, uses each embedding for one axis, and connects the x embedding and the y embedding to obtain the final position code P x of the input feature. And by means of the self-attention mechanism, the attention of the features in the modal itself and between different modes is obtained, the image tile features are enhanced by using the attention, and the features between different modes can be processed more effectively.

[0093] S45 inputs the third feature map of the reference image tile and the third feature map of the image tile to be registered into the feature fusion network to perform feature fusion, to obtain a fourth feature map;

[0094] S46 converts the fourth feature map into a feature vector group and inputs into a homography estimation network to predict the offset of four corner points of the image tile to be registered in each pair of image tile pairs relative to four corner points on the reference image tile, to obtain a homography estimation matrix corresponding to each pair of image tile.

[0095] In the embodiment of the application, the correspondence from the image tile to be registered to the reference image tile is found by calculating the homography matrix between the two images. This 3x3 matrix contains rotation, translation and scaling information. Through the homography matrix H, any pixel point in a to-be-registered image tile can be mapped to the corresponding pixel point in the reference image tile, that is, the pixel point [u, w] in the to-be-registered image is mapped to the pixel point [u', w'] in the reference.

[0096] wherein H 11 , H 21 ,..., H 33 are 9 parameters of the homography matrix H. The problem of finding 9 parameters is converted into the problem of finding 8 parameters. Four-point parameterization provides an intuitive way to describe the homographic transformation, which focuses on the position changes of specific corner points in the image. This parameterization simplifies certain optimization problems because it avoids directly dealing with complex matrix operations and instead focuses on the concept of corner point displacement, which is more intuitive. Δu1, Δu2,..., Δw4 are the offsets of the four corner points of the image tile to be registered in each pair of image tile pairs relative to the four corner points on the reference image tile.

[0097] Once the displacement of the four corner points is known, H 4point can be easily converted into H matrix， The conversion can be achieved using the normalized direct linear transformation algorithm. The homographic transformation is defined based on the position changes of the four corner points in the image tile to be registered. The position difference (i.e., offset) of each corner point between the image tile to be registered and the reference image tile is used to represent the homography. These offsets can be organized into an "corner point offset" matrix with 8 degrees of freedom. There is a one-to-one correspondence between the "corner point offset" matrix and the traditional 3x3 matrix representation of homography. That is, given a "corner point offset" matrix, a corresponding 3x3 homography matrix H matrix can be uniquely determined.

[0098] S5 uses the homography estimation matrix to sequentially find the positions of multiple candidate points on each to-be-registered image in the reference image, to obtain multiple control point pairs;

[0099] S6 constructs a triangulation network based on the control points, and rectifies each triangulation network by using an affine transformation to realize the registration between each image to be registered and the reference image.

[0100] In the embodiment of the present application, a dense image triangulation network is constructed based on the control points, and each triangular facet is rectified by using an affine transformation. The affine transformation uses the following expression:

[0101] wherein a1, a2, a3, a4, a5, a6 are six parameters of the affine transformation, a1, a2, a3, a4 are rotation parameters, and a5, a6 are translation parameters, is the u and w coordinates before the affine transformation, is the u' and w' coordinates after the affine transformation.

[0102] The six coefficients in the above expression are calculated according to the three vertices of the triangle, and then the triangle on the image to be registered is rectified to the triangle on the reference image according to the polynomial, thereby completing the accurate registration between the image to be registered and the reference image.

[0103] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt a computer program product in the form of being implemented on one or more computer usable storage media containing computer usable program codes (including but not limited to disk storage and optical storage, etc.).

[0104] The present application is described with reference to flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices produce a device that implements the functions specified in one or more flows in the flowcharts and / or one or more blocks in the block diagrams.

[0105] These computer program instructions can also be stored in a computer readable storage medium that can direct the computer or other programmable data processing devices to work in a specific manner, so that the instructions stored in the computer readable storage medium produce a manufactured product including instruction devices that implement the functions specified in one or more flows in the flowcharts and / or one or more blocks in the block diagrams.

[0106] These computer program instructions can also be loaded into computer or other programmable data processing devices, so that a series of operation steps are performed on the computer or other programmable data processing devices to generate computer-implemented processes, thus the instructions executed on the computer or other programmable data processing devices provide steps for implementing the functions specified in one or more flows of the flowchart and / or one or more blocks of the block diagram.

[0107] Obviously, various modifications and changes can be made to the present application by those skilled in the art without departing from the spirit and scope of the present application. Thus, it is intended that the present application embrace all such modifications and changes and, accordingly, the application is not to be limited by the foregoing description.

Claims

1. A deep learning-based image registration method, characterized by, The method comprises: S1, screening a reference image and a plurality of to-be-registered images in an image database, and selecting a plurality of candidate points on each to-be-registered image; S2, sequentially performing cutting processing on each to-be-registered image and the reference image based on a preset size to obtain a plurality of pairs of image tiles; each pair of image tiles comprises one reference image tile and one to-be-registered image tile; S3, constructing a deep homography estimation model, training the deep homography estimation model based on a training data set to obtain a trained deep homography estimation model; S4, sequentially inputting each pair of image tiles into the trained deep homography estimation model to obtain a homography estimation matrix corresponding to each pair of image tiles; S5, sequentially obtaining the positions of the plurality of candidate points on the reference image on each to-be-registered image by using the homography estimation matrix to obtain a plurality of control point pairs; S6, constructing a triangular network based on the plurality of control point pairs, rectifying each triangular network by using affine transformation to realize registration between each to-be-registered image and the reference image; In step S3, the deep homography estimation model comprises a feature extraction network, a feature enhancement network, a feature fusion network and a homography estimation network; the feature enhancement network comprises a self-attention mechanism unit and a cross-attention mechanism unit; the feature extraction network is connected with the feature enhancement network, the feature enhancement network is connected with the feature fusion network, and the feature fusion network is connected with the homography estimation network.

2. The method of claim 1, wherein, Step S4 further comprises: S41, sequentially inputting each pair of image tiles into the feature extraction network in the trained deep homography estimation model to extract a first feature map of the reference image tile and a first feature map of the to-be-registered image tile, respectively; S42, calculating a reference position encoding of the first feature map of the reference image tile and a to-be-registered position encoding of the first feature map of the to-be-registered image tile, respectively; and calculating a multi-head attention matrix of the first feature map of the reference image tile and the first feature map of the to-be-registered image tile; S43, inputting the first feature map of the reference image tile and the corresponding reference position encoding, the first feature map of the to-be-registered image tile and the corresponding to-be-registered position encoding into the feature enhancement network, and calculating a second feature map of the reference image tile and a second feature map of the to-be-registered image tile in each pair of image tiles after self-structure information enhancement through the self-attention mechanism unit; S44, calculating a third feature map of the reference image tile and a third feature map of the to-be-registered image tile in each pair of image tiles after cross-structure information enhancement through the cross-attention mechanism unit; S45, inputting the third feature map of the reference image tile and the third feature map of the to-be-registered image tile into the feature fusion network for feature fusion to obtain a fourth feature map; S46, converting the fourth feature map into a feature vector group and inputting the feature vector group into the homography estimation network to predict the offset of four corner points of the to-be-registered image tile relative to four corner points of the reference image tile in each pair of image tiles to obtain a homography estimation matrix corresponding to each pair of image tiles.

3. The method of claim 2, wherein, In step S42, the method for calculating the multi-head attention matrix is: The calculation formula of dot product attention Attention(Q, K, V) is: wherein Q, K, V represent Queue, Key, Value matrix of dot-product attention respectively, d k is the dimension of Q, K matrix; Each attention head head i The calculation formula is: head i = Attention(QW i Q ,KW i K ,VW i V ) where W i Q , W i K , W i V are trainable parameters, representing the i-th attention head applied to the matrices resulting in Q, K, V. The calculation formula of the multi-head attention matrix MultiHead(Q, K, V) is: MultiHead(Q, K, V) = Concat(head1,.., head n )W 0 where W 0 represents a trainable weight matrix used to weight the multi-head attention results.

4. The method of claim 3, wherein, The method for calculating the second feature map of the reference image tile and the second feature map of the image tile to be registered in each pair of image tile pairs after self-structure information enhancement in step S43 is as follows: For the reference image tile and the image tile to be registered in each pair of image tile pairs, the corresponding self-structure enhanced feature information is calculated by the following formula, to obtain the second feature map of the reference image tile and the second feature map of the image tile to be registered after self-structure information enhancement; and the calculation formula of the self-structure enhanced feature is as follows: X self = X + MultiHead(X + P X , X + P X , X) wherein X self is a self-structure enhanced feature of the first feature map of the reference image tile or the first feature map of the image tile to be registered, X is the input first feature map of the reference image tile or the first feature map of the image tile to be registered, P x is a positional encoding of the input first feature map of the reference image tile or the first feature map of the image tile to be registered.

5. The method of claim 4, wherein, The method for calculating the third feature map of the reference image tile and the third feature map of the image tile to be registered after cross-structure information enhancement in each pair of image tile pairs in step S44 is as follows: For each pair of the second feature map of the reference image tile and the second feature map of the image tile to be registered after the self-structure information enhancement, the corresponding cross-structure enhanced feature information is calculated by the following formula to obtain the third feature map of the reference image tile and the third feature map of the image tile to be registered after the cross-structure information enhancement; the cross-structure enhanced feature information is obtained by adding the cross-structure enhanced feature information residual to the feedforward network (FFN) in the cross-structure enhanced network, and the specific calculation formula is as follows: wherein The enhanced feature information residual for the interaction structure is calculated according to the following formula: wherein X q is a self-structure enhanced feature of a first feature map of the image tile to be registered, X kv is a self-structure enhanced feature of a first feature map of the reference image tile, P q is a position encoding of the first feature map of the image tile to be registered, P kv is a reference position encoding of the first feature map of the reference image tile.

6. The method of claim 1, wherein, The network structure of the feature extraction network in step S3 is ResNet-50; the specific network structure includes: a first convolutional layer, a first BN layer, an activation function layer, a maximum pooling layer, a second convolutional layer, a second BN layer, a third convolutional layer, a third BN layer, a fourth convolutional layer, a fourth BN layer, a fifth convolutional layer, and a fifth BN layer; the number of convolutional kernels of the first to fifth convolutional layers is set to 64, 64, 128, 256, and 512, respectively, and the size of the convolutional kernel is set to 7, 3, 3, 3, and 3, respectively; the step length of the first, second, and third convolutional layers is set to 2, and the step length of the fourth and fifth convolutional layers is set to 1; the kernel void ratio in the fourth and fifth convolutional layers is set to 2 and 4, respectively; the size of the kernel of the maximum pooling layer is set to 3*3, and the step length is set to 2; the first to fifth BN layers all adopt the batch normalization function, the activation function layer adopts the linear rectifier function, and the maximum pooling layer adopts the region maximum pooling function.

7. The method of claim 2, wherein, The method for training the deep homography estimation model based on the training data set in step S3 includes: S31 obtains a plurality of samples in the MS-COCO public data set, selects the middle region of each sample, and cuts out a plurality of reference sample tiles with a resolution of 256*256; S32 randomly moves the coordinates of the four corner points of each reference sample tile by the same value in sequence to obtain a plurality of image tiles to be registered; the plurality of reference sample tiles and the plurality of image tiles to be registered correspondingly are taken as a training data set, and the training data set includes a plurality of pairs of training data, one pair of training data including one reference sample tile and one image tile to be registered; S33 inputs the plurality of pairs of training data in the training data set into the deep homography estimation model in sequence for iterative training, uses the homography estimation matrix output in the training process to spatially transform the image tile to be registered on the reference sample tile, and calculates the error of the corresponding horizontal and vertical coordinates of the four corner points in the spatially transformed image tile to be registered and the four corner points in the reference sample tile, respectively; S34 constructs a loss function of the deep homography estimation model, updates the model according to the error, and stops until the loss function converges.

Images