A multispectral image registration method based on multi-scale deep feature map fusion

Abstract

The application discloses a multispectral image registration method based on multiscale deep feature map fusion, which can be used for registering spectral images between different wave bands. The method first acquires consistent feature maps of a target image and a source image by using a multiscale consistent feature map mapper, performs self-correlation feature modification on the multiscale consistent feature maps by using a transformer structure to obtain multiscale matching information feature maps, and then processes the multiscale matching information feature maps based on a self-attention mechanism to output registration parameter residuals. Under the supervision of an image registration true value and consistent feature map content consistency loss function, the network is iteratively run until a preset iteration number is reached, and finally, registration parameters are obtained. Experimental results and analysis on a multispectral dataset show that the multispectral image registration method based on multiscale deep feature map fusion effectively improves the registration accuracy of multispectral images.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a multispectral image registration method based on multiscale deep feature map fusion. BACKGROUND

[0002] Multispectral image registration has important applications in remote sensing, earth observation, agriculture, environmental monitoring, etc. Multispectral images are usually composed of multiple bands of images, each band corresponding to a different spectral range, capturing different ground information. However, there may be spatial shifts, deformations, and sensor noise between different bands, so multispectral image registration is a key step to obtain consistent and accurate information.

[0003] Traditional multispectral image registration methods usually rely on hand-designed feature extraction and matching algorithms, such as SIFT (Scale Invariant Feature Transform), ORB (Oriented Fast and Rotated Brief), SURF (Speeded Up Robust Features), etc. Then use these features to estimate geometric transformations such as translation, rotation, scaling, etc. to achieve spatial alignment of multispectral images. To avoid the problem of inconsistent features or gradient reversal on multispectral images caused by directly using the statistics of gradients in features such as SIFT, LGHD (A feature descriptor for matching across non-linear intensity variations) uses a log-Gabor filter to construct a histogram for each scale response. The log-Gabor filter histogram of different scales and different directions is constructed into the final descriptor. Unlike LGHD, RIFT (Radiation-Variation Insensitive Feature Transform) first detects feature points in multispectral images and divides the local area of each feature point into multiple sub-windows; calculates the histogram of pixels in each sub-window through a normalization strategy to eliminate the influence of radiation variation; finally, the normalized histograms are spliced into feature descriptors to achieve multispectral image matching.

[0004] However, traditional methods may have certain limitations in handling deformation, nonlinear transformation, and modal differences in multispectral images. In recent years, machine learning-based methods have made significant progress in the field of multispectral image registration. Machine learning-based multispectral image registration methods overcome the limitations of traditional methods in deformation modeling, modal differences, and other aspects. By directly learning features and transformation models from data, these methods can handle more complex registration tasks, such as large-scale deformation, nonlinear transformation, and other aspects. In addition, machine learning methods have the advantages of strong adaptability and automatic learning of features, without the need for manual design of complex feature extraction processes. Related methods include SCB (Structure Consistency Boosting) and PCNet (Phase Congruency Network). SCB uses a transformation of the Generalized Gaussian Distribution (GGD) function with 3 trainable parameters to enhance the consistent intrinsic edge structure in multispectral images. PCNet contains 2 trainable layers and an improved learnable Gabor kernel. Due to strong prior knowledge guidance, PCNet only needs to be trained once on a small amount of multispectral data to achieve good registration results on multiple multispectral image datasets. The above methods have improved performance for multispectral image registration with large deformation, but still cannot achieve high-precision multispectral image registration. SUMMARY

[0005] To overcome the deficiencies in the prior art described above, the present application provides a multispectral image registration method based on multiscale deep feature map mapping fusion, which can be used to accurately register multispectral channel images and achieve advanced registration performance.

[0006] The present application adopts the following technical solutions:

[0007] A multispectral image registration method based on multiscale deep feature map fusion, multispectral data usually contains more than 3 spectral images, for non-registered multispectral data, taking one of the wavebands as the target image, and taking the other waveband images as the source images, aligning the source images one by one to the target image during registration; aligning the source images to the target image, specifically including the following steps:

[0008] 1) First, obtain multiscale consistent feature maps of two images respectively;

[0009] 2) Then, concatenate the obtained multiscale consistent feature maps of the target image and the source image in the channel dimension, and process them using a deformable convolution layer to output matching information feature maps at each scale; fuse the multiscale matching information feature maps based on a self-attention mechanism to output registration parameter residuals;

[0010] 3) update the current prediction registration parameters using the registration parameter residual, use the current prediction registration parameters to perform deformation correction on the multi-scale consistent feature maps of the source image, replace the previous multi-scale consistent feature maps of the source image with the corrected multi-scale consistent feature maps of the source image, continue to perform step 2), and output the next registration parameter residual;

[0011] 4) stop when step 3) is performed to a preset number of iterations, and output the final prediction registration parameters;

[0012] 5) perform deformation correction on the source image based on the finally obtained prediction registration parameters, thereby realizing multi-spectral image registration.

[0013] In the above technical solution, further, in step 1), the multi-scale consistent feature maps of the two images are obtained by using a multi-scale consistent feature map mapper to obtain the multi-scale consistent feature maps of the two images in two paths, and the specific process is as follows:

[0014] The target image and the source image are processed in two paths, and the two processing methods are the same, specifically: first, a plurality of deep feature extraction units are used to process the image to obtain a plurality of scale (spatial size) feature maps; then, for each scale feature map, a 1x1 convolution mapping is used to reduce the dimension, and the channel dimension is reduced to 1 to obtain a multi-scale consistent feature map; the deep feature extraction unit includes a residual block and a max pooling layer.

[0015] Further, after the obtained multi-scale consistent feature maps of the target image and the source image are concatenated in the channel dimension, a deformable convolution layer is used for processing, and a scale matching information feature map is output, and the specific process can be represented as:

[0016] For the scale consistent feature maps of the input target image and source image, after concatenating them in the channel dimension, a deformable convolutional neural network is used for processing, guiding the deformable convolutional neural network to implicitly learn the matching information, and outputting a scale matching information feature map. Since the absolute amount of offset of different scales is different, different weight deformable convolutional neural networks are used for parallel processing. If it is the first iteration, the scale consistent feature maps of the target image and the scale consistent feature maps of the source image are processed; otherwise, the scale consistent feature maps of the target image and the scale consistent feature maps of the source image after deformation correction are processed.

[0017] Further, in step 2), the multi-scale matching information feature maps are fused based on the self-attention mechanism, and the specific process is as follows:

[0018] First, the self-attention mechanism of the transformer is used to calculate the scale matching information feature maps self-attention mechanism captures long-range self-correlation in matching information, and its self-correlation calculation is shown in formula (1),

[0019]

[0020] where Q, K, and V are query matrix, key matrix, and value matrix, respectively, and the attention range is the entire feature map. To better fuse spatial position information at different scales, position encoding is added at each scale, and it is set as a learnable vector. Each coordinate position in the matching information feature map at each scale is adaptively encoded. The learnable position encoding of each scale feature map is a feature map with the same spatial size and channel number, denoted as The position encoding information is added to the scale matching information feature map in an additive manner, as shown in formula (2),

[0021]

[0022] Then, for the multi-scale matching information feature map modified by self-attention Taking a selected scale as the reference scale, for each scale matching information feature map, if its scale spatial resolution is greater than the reference scale, it is further down-sampled by max-pooling + convolution processing, otherwise it is up-sampled by bilinear interpolation, and finally a multi-scale matching information feature map with the same spatial resolution is obtained.

[0023] Finally, for the multi-scale matching information feature map with the same spatial resolution, a deformable convolutional neural network is used to fuse the matching information feature maps at different scales. First, the matching information feature maps at different scales are concatenated in the channel dimension, then a deformable convolutional neural network is used to reduce the dimension of the concatenated matching information feature map in the channel dimension, and the fusion process can be represented as formula (3),

[0024]

[0025] where J n is the fused matching information feature map, concatenate represents channel concatenation, DCN represents deformable convolutional neural network, and fuse represents channel dimension reduction fusion operation.

[0026] Further, the registration parameter residual is output in step 2), and the specific process is as follows:

[0027] Based on the fused matching information feature map J n , the registration parameter is predicted, specifically: for J nGlobal information fusion is performed to output a current registration parameter residual. Specifically, the fused matching information feature map J is first processed by two convolutional layers n , and then down-sampled by multiple residual basic units, and finally processed by a convolutional layer, a group normalization layer and a convolutional layer in sequence to obtain the registration parameter residual. The residual basic unit includes a convolutional layer, a group normalization layer, a parametric rectified linear unit (PReLU) and a max-pooling layer. The group normalization method is suitable for the iterative prediction framework of the network.

[0028] Further, in step 3), the current predicted registration parameter is updated using the registration parameter residual, and the multi-scale consistent feature map of the source image is corrected by deformation using the current predicted registration parameter. Specifically,

[0029] The current predicted registration parameter is obtained by matrix multiplication of the last iteration predicted registration parameter and the current iteration registration parameter residual. When the first iteration, since there is no last iteration predicted registration parameter, the last iteration registration parameter is set to an identity transformation, i.e. no transformation.

[0030] The multi-scale consistent feature map of the source image is corrected by deformation using the current predicted registration parameter at different scale channels, and the correction process is as formula (4):

[0031]

[0032] wherein, is the corrected l-scale source image consistent feature map, represents coordinate mapping, represents the predicted registration parameter of k-1 iteration, and l represents the l-scale.

[0033] A multi-spectral registration model for performing the multi-spectral image registration method based on multi-scale deep feature map fusion, the training process of the registration model includes two network supervisions based on image registration ground truth and consistent feature map content consistency, wherein the network supervision of the image registration ground truth is shown as formula (5),

[0034]

[0035] wherein, K represents the total number of iterations, k ranges from 1 to K, and a is a value greater than 0 and less than 1, used to adjust the weight of each iteration number;

[0036] The network supervision based on consistent feature map content consistency is shown as formula (6),

[0037]

[0038] where P represents a perceptual loss function, L represents a maximum scale of a consistent feature map, for the l-th scale source image consistent feature map after correction, represents a consistent feature map under the true value deformation, and β is a value greater than 0 and less than 1, used to adjust the weight of each iteration.

[0039] The final loss function is shown in equation (7), and the accuracy of the multispectral registration model is ensured based on the loss function:

[0040] S = S1 + S2, (7) The beneficial effects of the present application are:

[0041] Multispectral image processing of different wavebands is a complex and important task. The present application achieves better image matching and fusion through a series of innovative steps. First, for multispectral images of different wavebands, multiscale consistent feature maps of the two images are obtained. Second, the concatenated feature maps are processed using a deformable convolutional layer, which can adapt to features of different scales and shapes, thereby better capturing matching information between images. Finally, by fusing the matching information feature maps of each scale, the network can learn how to better understand and utilize these information, thereby achieving high-precision multispectral image registration. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 Flowchart of the multispectral image registration method implemented by the present application;

[0043] Figure 2 Multiscale consistent feature map mapping method implemented by the present application;

[0044] Figure 3 Matching information implicit learning process of the scale deformable convolutional neural network implemented by the present application;

[0045] Figure 4 Multiscale matching information feature map fusion process implemented by the present application;

[0046] Figure 5 Registration parameter decoding process based on global information fusion implemented by the present application. DETAILED DESCRIPTION

[0047] The specific embodiments of the present application will be further described below in conjunction with the accompanying drawings

[0048] As Figure 1As shown, the multispectral image registration method based on multiscale deep feature map fusion of the application takes the intermediate channel of the multispectral image as the reference image, takes the remaining channel images as the source images, all channel source images and the reference image form a to-be-registered image pair respectively, and performs registration in channels with the reference image as the standard; for one pair of reference image and source image, the registration method comprises the following steps:

[0049] (1) First, the multiscale consistent feature map of the two images is obtained by using the multiscale consistent feature map mapper. Specifically, this step does not participate in the network iteration process and is only processed once;

[0050] (2) Then, at each scale, the obtained target image and source image multiscale consistent feature maps are concatenated in the channel dimension and then processed by using the deformable convolution layer, the multiscale parallel processing is performed, and the matching information feature map at each scale is output; the multiscale matching information feature map is fused based on the self-attention mechanism, and the registration parameter residual is output;

[0051] (3) The current predicted registration parameter is updated by using the registration parameter residual, the multiscale consistent feature map of the source image is corrected by using the current predicted registration parameter, the corrected multiscale consistent feature map of the source image is replaced by the previous multiscale consistent feature map of the source image, and step (2) is continuously executed, and the next registration parameter residual is output;

[0052] (4) Stop when step (3) is executed to a preset iteration number, and output the final predicted registration parameter;

[0053] (5) The source image is corrected based on the final obtained predicted registration parameter, so as to realize the registration of the multispectral image.

[0054] The registration model for executing the above method, in the training process, uses the loss function based on the image registration true value and the consistency of the consistent feature map content to supervise, specifically, the AdamW stochastic gradient descent algorithm is used to update the network parameters, and the optimal network parameters are obtained.

[0055] Figure 2 The multispectral image multiscale consistent feature map mapping method implemented by the application is shown. In order to realize the mapping of consistent feature maps of different scales, 3x3 residual blocks are used as basic units, and a maximum pooling is connected after each residual block to realize the scaling of the feature map; for different scale feature maps, a 1x1 convolution layer is used to project the dimension of the feature map, so as to output the multiscale consistent feature map. Preferably, the step length of the maximum pooling used in the method is 2, the number of basic units is 3, and the total scale is 3.

[0056] Figure 3The matching information implicit learning process of the multi-scale deformable convolutional neural network implemented by the present application is shown. The multi-scale consistent feature maps are processed in parallel using the multi-scale deformable convolutional neural network, and the multi-scale matching information feature maps are output in parallel. To facilitate the network to implicitly learn the matching information, the multi-scale consistent feature maps are concatenated in the channel dimension and then processed using the deformable convolutional neural network, which facilitates the perception of multiple scale networks to image deformation. The deformable convolutional neural network is used to deal with the offset between the two channel consistent feature maps to be registered.

[0057] Figure 4 The multi-scale matching information feature map fusion process implemented by the present application is shown. Different scale features are first modified using a transformer with position encoding, and the attention mechanism is autocorrelation attention. The autocorrelation attention calculation method is shown in formula (1), and the transformer with position encoding calculation method is shown in formula (2). Preferably, in combination with the iterative inference process of the entire neural network, the number of layers of the transformer is set to 1 layer to reduce the total calculation amount. Then, taking a certain scale feature size as a reference, the sizes of all feature maps are unified: for each scale matching information feature map, if its scale space resolution is greater than the reference scale, down-sampling is performed using convolution + maximum pooling operation, otherwise up-sampling is performed using bilinear interpolation operation. Preferably, the reference scale is selected as the second of the three scales. Finally, the feature map fusion is realized using the deformable convolutional neural network, and the fused feature map J n .

[0058] Figure 5 The registration parameter decoding process based on global information fusion implemented by the present application is shown. For the input fused feature map J n , it is first processed using two convolutional layers, then using convolution + group normalization + parameter rectified linear unit (PReLU) + maximum pooling as the basic unit, the feature map is continuously down-sampled using the unit, and finally a 3x2 registration parameter residual is output.

[0059] Table 1 shows the registration effect of the present method, and the error is described by ACE (average corner error), it can be seen that the present method has high multispectral image registration accuracy.

[0060] Table 1

[0061]

[0062] The above is only a specific embodiment of the present application, which cannot limit the scope of the present application. Equivalent changes made by general technical personnel in the technical field according to the present application, and changes well known to those skilled in the art, should still fall within the scope of the present application.

Claims

1. A multispectral image registration method based on multi-scale deep feature map fusion, characterized in that, For non-registered multispectral images, one of the bands is taken as the target image, and the other bands are taken as source images, which are aligned to the target image one by one during registration; the specific steps of aligning the source image to the target image are as follows: 1) First, obtain multiscale consistent feature maps of two images respectively; 2) Then, after concatenating the obtained multiscale consistent feature maps of the target image and the source image in the channel dimension, process them using a deformable convolution layer to output multiscale matching information feature maps; fuse the multiscale matching information feature maps based on the self-attention mechanism to output a registration parameter residual; 3) Update the current predicted registration parameter using the registration parameter residual, deform and correct the multiscale consistent feature maps of the source image using the current predicted registration parameter, replace the previous multiscale consistent feature map of the source image with the corrected multiscale consistent feature map of the source image, and continue to execute step 2) to output the next registration parameter residual; 4) Stop when step 3) is executed to a preset number of iterations, and output the final predicted registration parameter; 5) Deform and correct the source image based on the final predicted registration parameter to achieve multispectral image registration; In step 1), the multiscale consistent feature maps of the two images are obtained by using a multiscale consistent feature map mapper in two paths, and the specific process is as follows: The target image and the source image are processed in two separate steps, using the same method for both. Specifically, first, multiple deep feature extraction units are used to process the image, obtaining feature maps at multiple scales; then, for each scale feature map, [the following steps are applied]... Convolutional mapping reduces dimensionality, decreasing the channel dimension to... A multi-scale consistent feature map is obtained; the deep feature extraction unit includes a residual block and a max pooling layer. In step 3), the current predicted registration parameter is updated using the registration parameter residual, and the multiscale consistent feature maps of the source image are deformed and corrected using the current predicted registration parameter, which is specifically: Multiply the last iteration predicted registration parameter and the current iteration registration parameter residual by a matrix to obtain the current predicted registration parameter. When the first iteration, since there is no last iteration predicted registration parameter, set the last iteration registration parameter to be an identity transformation, i.e. no transformation; Utilizing current predicted registration parameters on multi-scale consistent feature maps of source images Synchronous morphing correction is performed on different scale channels, and the correction process is as formula (4): wherein, for the corrected post scale source image consistent feature map, denotes a coordinate mapping, represent predicted registration parameters of the next iteration, denotes the scale.

2. The multispectral image registration method based on multi-scale deep feature map fusion according to claim 1, characterized in that, In step 2), the multiscale matching information feature maps are fused based on the self-attention mechanism, and the specific steps are as follows: First, the self-attention mechanism of the transformer is used to calculate the matching information feature maps at each scale. … The relevant information, among which Let n represent the scale number and n represent the multispectral channels; the calculation of the self-attention mechanism is given by formula (1): wherein, The query matrix, the key matrix, and the value matrix are sequentially arranged, and the attention range thereof is the entire feature map. Each coordinate position in the scale matching information feature map is adaptively encoded, and the scale feature maps are added to the scale matching information feature map in an additive manner. …, The position encoding information of the … is added to the scale matching information feature map in an additive manner, and the specific formula (2) is as follows: Then, for the self-attention modified multi-scale matching information feature map … A baseline scale is determined. For the matching information feature maps at each scale, if the scale spatial resolution is greater than the baseline scale, it is further downsampled by processing through max pooling layers and convolutional layers; otherwise, it is upsampled by bilinear interpolation, and finally, multi-scale matching information feature maps with the same spatial resolution are obtained. Finally, for multiscale matching information feature maps with the same spatial resolution, use a deformable convolution layer to fuse information of matching information feature maps of different scales; first, concatenate the multiscale matching information feature maps in the channel dimension, then use a deformable convolution layer to reduce the dimension of the concatenated matching information feature maps in the channel dimension, and the fusion process is as formula (3): wherein, is the fused matching information feature map, represents channel concatenation, represents a deformable convolution layer, represents a channel dimension reduction fusion operation.

3. The multispectral image registration method based on multi-scale deep feature map fusion according to claim 2, characterized in that, In step 2), the registration parameter residual is output, and the specific process is as follows: The matched information feature map after fusion The registration parameter prediction is performed, specifically, the matched information feature map after fusion is processed by two convolution layers Global information fusion is performed, thereby outputting the current registration parameter residual; specifically, first, the matched information feature map after fusion is processed by two convolution layers , then is down-sampled by multiple residual basic units, and finally is processed by a convolution layer, a group normalization layer and a convolution layer in sequence, to obtain the registration parameter residual; the residual basic unit comprises a convolution layer, a group normalization layer, a parametric rectified linear unit (PReLU) and a max-pooling layer.

4. The multispectral image registration method based on multi-scale deep feature map fusion according to any one of claims 1-3, characterized in that, The training process of the registration model includes network supervision based on image registration true value and consistent feature map content consistency; The network supervision based on image registration true value is shown in formula (5): wherein, denotes the total number of iterations, the value of k ranges from 1 to , is a value greater than 0 and less than 1, used to adjust the weight of the weighting of each iteration number; The network supervision based on consistent feature map content consistency is shown in formula (6): wherein, represents a perceptual loss function, is the corrected 1th iteration scale source image consistent feature map, represents a consistent feature map under the true value deformation, is a value greater than 0 and less than 1, used to adjust the weight of each iteration. The loss function of the final model training process is shown in formula (7):

Images