Multi-source remote sensing optical image registration and fusion method based on space deformation field
Through the multi-source remote sensing optical image registration and fusion method based on spatial deformation field, the channel attention mechanism and cross attention guidance module are used to solve the problem of inaccurate feature representation and small receptive field in hyperspectral image fusion, achieving high efficiency image fusion and precise hyperspectral image reconstruction.
Patent Information
- Application Number
- CN202410821737.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-24
- Publication Date
- 2024-11-12
AI Technical Summary
The existing hyperspectral image fusion method cannot accurately characterize the fusion characteristics, resulting in distortion of the result, the convolutional neural network receptive field cannot use the global information of the image, and the unregistered image fusion performance is degraded.
Using multi-source remote sensing optical image registration and fusion method based on spatial deformation field, multi-scale null spectral features are extracted by constructing the channel attention mechanism of the null spectral feature pyramid module, and the cross-attention-guided spatial feature alignment module calculates the spatial relationship matrix. Generate deformation fields and use convolutional neural networks for feature fusion and reconstruction.
The fusion accuracy of multimodal remote sensing optical image under unregistered conditions is significantly improved, the spatial resolution of hyperspectral images is improved, the global image information can be used more effectively, feature expression capabilities are enhanced, and image reconstruction accuracy is improved.
Smart Images

Figure CN118941600A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to but is not limited to the field of remote sensing image processing technology, and in particular relates to a multi-source remote sensing optical image registration and fusion method and system based on spatial deformation field. Background Art
[0002] The fusion of hyperspectral images and multispectral images is an image processing technology that improves the spatial resolution of hyperspectral images. With the popularization of remote sensing satellite images, hyperspectral images have hundreds of continuous spectral bands and have been applied to remote sensing tasks such as change detection, classification, and semantic segmentation. Due to the limitations of the imaging system, the spatial resolution of hyperspectral images is low. In recent years, research on hyperspectral image fusion has emerged and received widespread attention. Depending on the high spatial resolution auxiliary image used, hyperspectral image fusion can be divided into hyperspectral and panchromatic image fusion, hyperspectral and multispectral image fusion, etc., which has become an effective means to improve the quality of hyperspectral images.
[0003] Many scholars have proposed some classic methods for the task of hyperspectral fusion. These basic methods can be divided into four categories: component replacement, multi-resolution analysis, Bayesian method and matrix decomposition. The component replacement-based method first decomposes the low-spatial-resolution hyperspectral image into spectral components and spatial components, then replaces the spatial components with auxiliary images with higher spatial resolution and converts them back to the original vector space; the multi-resolution analysis method injects spatial features into low-resolution hyperspectral images through spatial filters. These filters can reduce the low-frequency components of the image and retain high-frequency information; the Bayesian method predicts the required image by fitting the posterior distribution of the hyperspectral image and the multispectral image. The Bayesian-based method considers the problem comprehensively from the mathematical model it constructs, but it is usually computationally intensive and requires certain prior conditions; matrix decomposition decomposes the image into two matrices, one of which contains all the types of ground material in the image, called the endmember matrix, and the other matrix represents the concentration of the ground material in the image, called the abundance matrix. Matrix decomposition uses the abundance matrix of the multispectral image and the endmember matrix of the hyperspectral image to reconstruct the image.
[0004] In recent years, with the rapid development of deep learning (DL), a series of fusion networks for hyperspectral images have emerged, such as spectral fidelity global interactive hyperspectral multispectral cross-modal fusion methods and remote sensing image fusion methods based on convolutional neural networks. These models show very excellent fusion performance under the condition of strict data registration. However, most of the existing methods are based on convolutional neural networks. When processing images, the receptive field of convolutional neural networks is limited and the global context information of the image cannot be utilized, which limits the further improvement of network performance.
[0005] At the same time, in actual situations, due to the influence of factors such as the imaging platform, imaging perspective, and atmospheric turbulence, it is difficult to perform fine alignment of images. The performance of existing methods is severely degraded when the data is not aligned. How to achieve image alignment in the fusion process is a more realistic problem.
[0006] In view of the above analysis, the technical problems that need to be solved urgently in the prior art are:
[0007] (1) Traditional methods are based on mathematical model calculations and cannot accurately characterize fusion features, which can easily lead to distorted results;
[0008] (2) The convolutional neural network-based method has a small receptive field and cannot utilize the global information of the image;
[0009] (3) Existing methods are only applicable to registered pairs of multispectral images and hyperspectral images, and there is no effective solution for unregistered images. Summary of the invention
[0010] In view of the problems existing in the prior art, the present invention provides a multi-source remote sensing optical image registration and fusion method based on spatial deformation field.
[0011] The present invention is implemented as follows: a multi-source remote sensing optical image registration and fusion method based on a spatial deformation field, characterized in that the multi-source remote sensing optical image registration and fusion method based on a spatial deformation field specifically includes:
[0012] S1: Input high spatial resolution multispectral image Y M and low spatial resolution hyperspectral image Y H , construct a spatial-spectral feature pyramid module based on the channel attention mechanism to extract multi-scale spatial-spectral features of the two image branches;
[0013] S2: A cross-attention guided spatial feature alignment module is proposed. The multi-scale features of different branches are used to calculate the spatial relationship matrix in a cross-attention manner to generate a deformation field.
[0014] S3: Use the deformation field to shift the position of the multi-scale features of the hyperspectral image and align them with the multi-scale features of the multispectral image in spatial position;
[0015] S4: The multi-scale features of the two registered modalities are cascaded in the channel direction, and the features of the same scale are fused using a convolutional neural network to generate multi-scale features with spatial details;
[0016] S5: Establish a cross-scale information transmission module. Features at multiple scales share information through a convolutional network, complete the interaction of feature information across scales, and gradually enhance the expressiveness of features.
[0017] S6: Information fusion of enhanced multi-scale features and reconstruction of high spatial resolution hyperspectral images through convolutional neural networks;
[0018] S7: Perform supervised training on the constructed network model to obtain network parameters suitable for the model.
[0019] Further, the specific process of S1 is as follows:
[0020] The normalized image is passed through a linear projection layer to generate three matrices Q, K, and V. The dimensions of the three matrices are reshaped to C×HW, where C represents the channel dimension and HW represents stretching the spatial dimension into a one-dimensional vector. Q and K T After multiplication, the similarity matrix with dimension C×C is obtained through Softmax activation function. The similarity matrix represents the autocorrelation relationship between each channel and spectral vector. Then the similarity matrix is multiplied with V to obtain the intermediate feature F′. After F′ passes through the feedforward layer of the gating mechanism and the dimension reconstruction convolution layer, the dimension is scaled and the feature size is reshaped to i represents the size level of the scale; the process can be expressed by the following formula:
[0021]
[0022] Among them, the hyperspectral image feature is expressed as Where H, W represent the height and width of the target hyperspectral image, N represents the number of spectra in the hyperspectral image, n represents the number of spectra in the multispectral image, and i represents the scale level. represents the feature with scale level i+1, i.e., the output of the feature extractor at the upper level of the pyramid; W q ,W k ,W v represents the projection matrix; Q H , K H , V H Respectively represent the Q, K, V matrices generated by the hyperspectral image, Denotes the matrix K H The transpose of represents the scaling factor; Represents the hyperspectral intermediate features of the output of the attention layer; Conv() represents the convolution operation; ReLU() represents the activation function. The feedforward layer of the gating mechanism divides the same matrix into two branches. Each branch is convolved separately and then multiplied. One of the branches is activated by the activation function. The gating mechanism can select valid data to pass to the subsequent network.
[0023] For the multispectral image branch, the process can be expressed by the following formula:
[0024]
[0025] Among them, the multispectral image features can be expressed as represents the feature with scale level i-1, i.e., the output of the feature extractor at the upper level of the pyramid; Q M , K M , V M Respectively represent the Q, K, V matrices generated by the hyperspectral image, Denotes the matrix K M The transpose of Represents the hyperspectral intermediate features output by the attention layer.
[0026] In addition, to avoid the loss of feature information, residual connections are added to the outputs of the attention layer and feedforward layer of the two branches respectively.
[0027] Further, the S2 is specifically:
[0028] First, the features of the two modalities are vectorized and embedded. The features of the multispectral image are projected as Q, and the features of the hyperspectral image are projected as K and V. The cross attention is used to embed Q and K. T Do matrix product, calculate the spatial relationship matrix, cascade the spatial relationship matrix and the matrix V of the hyperspectral image projection to be registered, and send it to the neural network to learn the spatial difference to generate the deformation field. This process can be specifically expressed by the following formula:
[0029]
[0030] Spa=QK T
[0031] D i =f D (concat(Spa,V))
[0032] Among them, Spa represents Q and K T The spatial relationship matrix obtained by multiplication, K T represents the transpose of matrix K; f D () represents the solution of deformation field D i The present invention adopts U-net to f D () is used for modeling. In addition, because Spa represents spatial relationships, the dimensions of matrices Q and K are The spatial relationship can be modeled by multiplying the last two dimensions of the matrix.
[0033] Furthermore, the S3 uses the deformation field to shift the position of the multi-scale features of the hyperspectral image, and aligns them with the multi-scale features of the multispectral image in spatial position, and the formula is:
[0034]
[0035] D i The dimension is The first two dimensions represent the spatial dimensions, and the last dimension is 2, which represents the horizontal and vertical pixel displacements. i Acting on the original features You can get a hyperspectral image that is spatially aligned with the multispectral image.
[0036] Furthermore, S4 cascades the multi-scale features of the two registered modalities in the channel direction, and uses a convolutional neural network to fuse features of the same scale to generate multi-scale features with spatial details. The formula is:
[0037]
[0038] Among them, X i It represents the feature of fusing two image information.
[0039] Further, the S5 is specifically:
[0040] Small-scale features are transferred to large-scale and same-scale features, the smallest-scale features no longer transfer information forward, and large-scale features do not transfer information to small-scale features; the formula for this process is:
[0041] X (i)(j) =f R (X (i)(j-1) ,X (i-1)(j-1) ,…,X (3-j)(j-1) ),j∈[0,3],i∈[0,3-j];
[0042] Among them, X (i)(j) represents the feature node with scale i in the jth layer, f R () represents the convolutional neural network for interactive aggregation of feature information, X (i)(j-1) ,X (i-1)(j-1) ,…,X (j-1)(j-1) All feature nodes with scale levels from i to 3-j, which represent the output of the j-1 layer, i.e., the previous layer module, are fed into the aggregation network.
[0043] Furthermore, the S6 performs information fusion on the enhanced multi-scale features and reconstructs a high-spatial-resolution hyperspectral image through a convolutional neural network, and the formula is:
[0044]
[0045] X (N)(0) Indicates the node with scale 0, i.e. the maximum scale, in the last layer of the multi-scale feature information transmission structure. represents a hyperspectral image with high spatial resolution reconstructed by the network, and R() represents a convolutional neural network used for image reconstruction.
[0046] Further, in S7, supervised training is performed on the constructed network model to obtain network parameters suitable for the model, specifically:
[0047] (7a) The multispectral image Y with the reference image X M and the hyperspectral image Y H The dataset is loaded into the network model to be trained, and the predicted high spatial resolution hyperspectral image is output.
[0048] (7b) Using the following L1 reconstruction loss function, calculate the predicted image The loss function between and the reference image X is:
[0049]
[0050] Where || ||1 represents the 1-norm of the matrix;
[0051] (7c) Use the stochastic gradient descent method to train the network parameters until the network converges and save the optimal network parameters to complete the training of Y H With Y M The Adam optimizer with a learning rate of 1e-3 is used for registration and fusion. The learning rate decreases to 0.1 of itself every 100 epochs, and the learning process is completed after 300 epochs.
[0052] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:
[0053] First, the present invention designs a feature extraction structure based on channel attention Transformer, which makes full use of the global information of hyperspectral images, and the structure constitutes a spatial spectrum feature pyramid module, outputs multi-scale features, and characterizes image features of different sizes. It can characterize the autocorrelation relationship between each channel and spectral vector of the hyperspectral image, effectively extract the global context information of the image, and compared with the traditional self-attention, the vector length is reduced from HW to C, which greatly reduces the computational complexity;
[0054] The present invention innovatively proposes a cross-attention guided spatial feature alignment module, which models the spatial relationship between two modal images through a spatial attention mechanism and uses the spatial relationship matrix to learn the deformation field to achieve the effect of feature registration.
[0055] The present invention designs a cross-scale information transfer structure, which makes full use of the feature information of the registration at different scales, can form a more powerful feature representation, and produce a better hyperspectral image reconstruction effect.
[0056] The present invention describes a multi-source remote sensing optical image registration and fusion method based on spatial deformation field. The method uses the spatial spectrum feature pyramid module to extract multi-scale features, and then realizes feature alignment by calculating the spatial relationship matrix and solving the deformation field. Finally, the convolutional neural network is used to perform feature fusion and reconstruct the hyperspectral image.
[0057] Second, as auxiliary evidence of the inventiveness of the claims of the present invention, it is also reflected in the following important aspects:
[0058] (1) The multi-source remote sensing optical image registration and fusion method based on spatial deformation field proposed in the present invention can significantly improve the fusion accuracy of remote sensing optical images of different modalities under unregistered conditions and improve the spatial resolution of hyperspectral images. This provides an important means of improving image quality in the fields of environmental monitoring, agricultural resource management, and military reconnaissance, and has important application value. By improving the spatial resolution of hyperspectral images, the present invention helps to obtain more accurate and detailed ground object information, thereby improving the scientificity and effectiveness of related decisions, and bringing significant economic benefits and social value.
[0059] (2) The present invention solves the problem of inconsistent distribution of multi-source remote sensing data in different scenarios, especially the problem of non-rigid deformation of data. Previous remote sensing image fusion methods only focused on how to make full use of the complementary information of each image, ignoring the problem of ground object distortion caused by the imaging process. The present invention utilizes the spatial information of multimodal data and the deformation field to achieve global feature alignment, filling the gap in multi-source remote sensing optical image registration and fusion solutions at home and abroad, and providing a more effective and realistic solution for remote sensing image fusion.
[0060] (3) The present invention adopts a cross-attention guided spatial feature alignment module as the core, which makes full use of the spatial information in optical images of different modalities, making the generated deformation field more accurate and solving the influence of the complex degradation process on the texture of the ground objects faced in hyperspectral imaging. At the same time, the feature extraction and feature reconstruction structure designed by the present invention makes full use of the global spectral features, improving the reconstruction accuracy and pan-scanning ability of hyperspectral images. BRIEF DESCRIPTION OF THE DRAWINGS
[0061] Figure 1 It is a flow chart of a multi-source remote sensing optical image registration and fusion method based on a spatial deformation field provided by an embodiment of the present invention;
[0062] Figure 2is a schematic diagram of a Transformer structure based on channel attention provided by an embodiment of the present invention;
[0063] Figure 3 is a schematic diagram of the structure of a spatial feature alignment module based on an attention mechanism provided in an embodiment of the present invention;
[0064] Figure 4 It is an experimental visual effect diagram based on a pseudo-color image provided by an embodiment of the present invention;
[0065] Figure 5 It is an experimental visual effect diagram based on the residual graph provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0066] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0067] like Figure 1 As shown, an embodiment of the present invention provides a multi-source remote sensing optical image registration and fusion method based on a spatial deformation field, and the method specifically includes:
[0068] S1: Input high spatial resolution multispectral image Y M and low spatial resolution hyperspectral image Y H , construct a spatial-spectral feature pyramid module based on the channel attention mechanism to extract multi-scale spatial-spectral features of the two image branches;
[0069] S2: A cross-attention guided spatial feature alignment module is proposed. The multi-scale features of different branches are used to calculate the spatial relationship matrix in a cross-attention manner to generate a deformation field.
[0070] S3: Use the deformation field to shift the position of the multi-scale features of the hyperspectral image and align them with the multi-scale features of the multispectral image in spatial position;
[0071] S4: The multi-scale features of the two registered modalities are cascaded in the channel direction, and the features of the same scale are fused using a convolutional neural network to generate multi-scale features with spatial details;
[0072] S5: Establish a cross-scale information transmission module. Features at multiple scales share information through a convolutional network, complete the interaction of feature information across scales, and gradually enhance the expressiveness of features.
[0073] S6: Information fusion of enhanced multi-scale features and reconstruction of high spatial resolution hyperspectral images through convolutional neural networks;
[0074] S7: Perform supervised training on the constructed network model to obtain network parameters suitable for the model.
[0075] S1, input high spatial resolution multispectral image Y M and low spatial resolution hyperspectral image Y H , construct a spatial-spectral feature pyramid module based on the channel attention mechanism to extract multi-scale spatial-spectral features of the two image branches. Figure 2 As shown, the structure of each level of the Laplace pyramid is as follows:
[0076] The specific structure of each level of the Laplacian pyramid is a channel attention-based Transformer feature extraction structure, which consists of a normalization layer, a linear projection layer, a channel attention layer, a normalization layer, a feedforward layer, and a dimension reconstruction convolution layer. The specific process is as follows:
[0077] The normalized image is passed through a linear projection layer to generate three matrices Q, K, and V. The dimensions of the three matrices are reshaped to C×HW, where C represents the channel dimension and HW represents stretching the spatial dimension into a one-dimensional vector. Q and K T After multiplication, the similarity matrix with dimension C×C is obtained through Softmax activation function. The similarity matrix represents the autocorrelation relationship between each channel and spectral vector. Then the similarity matrix is multiplied with V to obtain the intermediate feature F′. After F′ passes through the feedforward layer of the gating mechanism and the dimension reconstruction convolution layer, the dimension is scaled and the feature size is reshaped to i represents the size level of the scale; the process can be expressed by the following formula:
[0078]
[0079] Among them, the hyperspectral image features can be expressed as Where H, W represent the height and width of the target hyperspectral image, N represents the number of spectra in the hyperspectral image, n represents the number of spectra in the multispectral image, and i represents the scale level. represents the feature with scale level i+1, i.e., the output of the feature extractor at the upper level of the pyramid; W q ,W k ,W v represents the projection matrix; Q H , K H , V H Respectively represent the Q, K, V matrices generated by the hyperspectral image, Denotes the matrix K H The transpose of represents the scaling factor; Represents the hyperspectral intermediate features of the output of the attention layer; Conv() represents the convolution operation; ReLU() represents the activation function. The feedforward layer of the gating mechanism divides the same matrix into two branches. Each branch is convolved separately and then multiplied. One of the branches is activated by the activation function. The gating mechanism can select valid data to pass to the subsequent network.
[0080] For the multispectral image branch, the process can be expressed by the following formula:
[0081]
[0082] Among them, the multispectral image features can be expressed as represents the feature with scale level i-1, i.e., the output of the feature extractor at the upper level of the pyramid; Q M , K M , V M Respectively represent the Q, K, V matrices generated by the hyperspectral image, Denotes the matrix K M The transpose of Represents the hyperspectral intermediate features output by the attention layer.
[0083] In addition, to avoid the loss of feature information, residual connections are added to the outputs of the attention layer and feedforward layer of the two branches respectively.
[0084] The S2, as Figure 3 As shown, specifically:
[0085] First, the features of the two modalities are vectorized and embedded. The features of the multispectral image are projected as Q, and the features of the hyperspectral image are projected as K and V. The cross attention is used to embed Q and K. T Do matrix product, calculate the spatial relationship matrix, cascade the spatial relationship matrix and the matrix V of the hyperspectral image projection to be registered, and send it to the neural network to learn the spatial difference to generate the deformation field. This process can be specifically expressed by the following formula:
[0086]
[0087] Spa=QK T
[0088] D i =f D (concat(Spa,V))
[0089] Among them, Spa represents Q and K T The spatial relationship matrix obtained by multiplication, K T represents the transpose of matrix K; f D () represents the solution of deformation field D iThe present invention adopts U-net to f D () is used for modeling. In addition, because Spa represents spatial relationships, the dimensions of matrices Q and K are The spatial relationship can be modeled by multiplying the last two dimensions of the matrix.
[0090] The S3 uses the deformation field to shift the position of the multi-scale features of the hyperspectral image and aligns them with the multi-scale features of the multispectral image in spatial position. The formula is:
[0091]
[0092] D i The dimension is The first two dimensions represent the spatial dimensions, and the last dimension is 2, which represents the horizontal and vertical pixel displacements. i Acting on the original features You can get a hyperspectral image that is spatially aligned with the multispectral image.
[0093] S4 cascades the multi-scale features of the two registered modalities in the channel direction, and uses a convolutional neural network to fuse features of the same scale to generate multi-scale features with spatial details. The formula is:
[0094]
[0095] Among them, X i It represents the feature of fusing two image information.
[0096] S5, establishing a cross-scale information transmission module, features of multiple scales share information through a convolutional network, complete the interaction of feature information across scales, and gradually enhance the expression ability of features, specifically:
[0097] Small-scale features are transferred to large-scale and same-scale features, the smallest-scale features no longer transfer information forward, and large-scale features do not transfer information to small-scale features; the formula for this process is:
[0098] X (i)(j) =f R (X (i)(j-1) ,X (i-1)(j-1) ,…,X (3-j)(j-1) ),j∈[0,3],i∈[0,3-j]
[0099] Among them, X (i)(j) represents the feature node with scale i in the jth layer, f R () represents the convolutional neural network for interactive aggregation of feature information, X (i)(j-1) ,X (i-1)(j-1) ,…,X (j-1)(j-1)All feature nodes with scale levels from i to 3-j, which represent the output of the j-1 layer, i.e., the previous layer module, are fed into the aggregation network.
[0100] S6, performs information fusion on the enhanced multi-scale features and reconstructs a high-spatial-resolution hyperspectral image through a convolutional neural network, and its formula is:
[0101]
[0102] X (N)(0) Indicates the node with scale 0, i.e. the maximum scale, in the last layer of the multi-scale feature information transmission structure. represents a hyperspectral image with high spatial resolution reconstructed by the network, and R() represents a convolutional neural network used for image reconstruction.
[0103] S7, performs supervised training on the constructed network model to obtain network parameters suitable for the model, specifically:
[0104] (7a) The multispectral image Y with the reference image X M and the hyperspectral image Y H The dataset is loaded into the network model to be trained, and the predicted high spatial resolution hyperspectral image is output.
[0105] (7b) Using the following L1 reconstruction loss function, calculate the predicted image The loss function between and the reference image X is:
[0106]
[0107] Where ||||1 represents the 1-norm of the matrix. The loss function Loss can quantitatively reflect the difference between the prediction result and the reference image, and the optimal network model can be obtained by minimizing Loss.
[0108] (7c) Use the stochastic gradient descent method to train the network parameters until the network converges and save the optimal network parameters to complete the training of Y H With Y M The Adam optimizer with a learning rate of 1e-3 is used, and the learning rate decreases to 0.1 of itself every 100 epochs, and the learning process is completed after 300 epochs.
[0109] 1. Specific application fields or related products of the present invention.
[0110] The present invention realizes a multi-source remote sensing optical image registration and fusion method based on spatial deformation field, which is used for the fusion of hyperspectral image and multispectral image. The present invention designs and constructs a spatial spectral feature pyramid module based on channel attention mechanism and a spatial feature alignment module guided by cross attention, which makes full use of the global information of the image, solves the problem of image misregistration, and improves the effect of image fusion.
[0111] In terms of the specific implementation plan, you need to first build a deep neural network model, which should include a module for multi-scale feature extraction of the input image, as well as a module for aligning and fusing the extracted features. Then, you need to prepare a dataset containing hyperspectral images and corresponding high-resolution multispectral images for training the model. Through supervised training, the parameters of the model can be optimized so that it can better complete the task. After the training is completed, the trained model can be used to fuse and reconstruct new hyperspectral images and multispectral images.
[0112] The present invention is an excellent multi-source remote sensing image enhancement method, which can be used as a data enhancement method for many remote sensing applications. Its specific application can be reflected in improving the data accuracy of downstream tasks, such as change detection, target detection, semantic segmentation, multimodal image classification, etc. It is of great significance in agricultural production, urban planning, geological resource protection, forest fire warning, etc.
[0113] 2. Relevant evidence of the technical effects obtained by the embodiments of the present invention.
[0114] 1. Simulation Experiment Conditions
[0115] The hardware platform for the simulation experiment of the present invention is: Nvidia GeForce RTX 3090, Intel(R) Core(TM) i9-10900X CPU@3.70GHz
[0116] The software platforms for the simulation experiment of the present invention are: Linux18.06 operating system, python3.7 and pytorch1.12.
[0117] The hyperspectral image used in the simulation experiment of the present invention adopts the Harvard data set, and a commercial hyperspectral camera (Nuance FX, CRI Inc.) is used to collect 50 outdoor and indoor image databases under sunlight. The camera uses an integrated liquid crystal tunable filter, which can continuously tune the filter through a series of 31 narrow bands to obtain a hyperspectral image. The resolution of each image is about 10nm per bandwidth, and is centered on 10nm from 420nm to 720nm. Among the 50 Harvard images, each image size is 1040×1392, and an area of 960×960 is intercepted as the experimental object.
[0118] 2. Evaluation indicators
[0119] In addition to qualitative observation of subjective results, quantitative evaluation should also be performed to measure the performance of image fusion methods. This experiment uses four objective indicators, peak signal-to-noise ratio (PSNR), spectral angle mapping (SAM), correlation coefficient (CC), and root mean square error (RMSE), for quantitative evaluation. The higher the PSNR and CC, the better. The ideal value of PSNR is infinity, the ideal value of CC is 1, the smaller the SAM and RMSE, the better. The ideal values of SAM and RMSE are both 0.
[0120] 3. Comparison of algorithms and results analysis
[0121] In order to verify the effectiveness of the present invention, four widely used remote sensing image fusion methods were selected, including CNMF, NonRegSRNet, SFIM, and HyperTransformer. The Harvard dataset was trained and tested respectively to obtain the fusion results of the final test set.
[0122] Table 1 Objective evaluation results
[0123]
[0124] The fusion results obtained by different methods are as follows Figure 4 , Figure 5 As shown. (a)-(f) are respectively the reference image, the fusion result of the present invention, the CNMF fusion result, the NonRegSRNet fusion result, the SFIM fusion result, and the HyperTransformer fusion result. It can be seen from the figures that CNMF and the present invention have the best performance, but CNMF has a large amount of calculation, NonRegSRNet has a large spectral distortion, SFIM has a large distortion of spatial information, and HyperTransformer has the worst performance. The reason is that the HyperTransformer mentioned above is a method proposed for the registered data set and cannot handle the situation where the data is not registered.
[0125] The objective evaluation of different methods is shown in Table 1, and the bold font indicates the best indicator. According to Table 1, the CC of the two traditional methods of CNMF and SFIM is lower than 0.99, and the performance of the other three indicators in the two methods is different, which indicates that there is at least one distortion in the fusion result of the traditional method. NonregSRNet and HyperTransformer are both deep learning methods. NonRegSRNet is proposed for misaligned images, and its various indicators perform well, while HyperTransformer has the worst performance. The fusion result of the present invention performs best in all indicators.
[0126] The feature extractor used in the present invention is compared with the traditional self-attention mechanism, proving that the Transformer structure based on the channel attention mechanism designed in the present invention is more suitable for hyperspectral images. In order to make the experiment more convincing, this experiment uses an efficient version of Transformer - the multi-head self-attention mechanism, and the experimental results are as follows:
[0127] Table 2 Effect of feature extractor
[0128]
[0129]
[0130] In order to reflect the effectiveness of the spatial feature alignment module used in the present invention, this experiment removed this module for verification. The test results are as follows:
[0131] Table 3 Effect of spatial feature alignment module
[0132]
[0133] In the embodiment of the present invention, the spatial feature alignment module only generates a deformation field acting on the original data, that is, it only changes the position distribution of the original data, and does not change the value. Figure 4 , it can be concluded that this module can correct the distribution differences between data, thereby obtaining better fusion results. Figure 4 It can be clearly seen from the other methods that the fusion results of other methods still have large spatial distortion.
[0134] In order to prove the effectiveness of the cross-scale information transfer structure used in the present invention, this experiment uses the traditional U-net structure to reconstruct the registered features. The experimental results are as follows:
[0135] Table 3 Effect of cross-scale information transfer structure
[0136]
[0137] Experiments have shown that the cross-scale information transfer structure used in the present invention can better utilize multi-scale information and overcome the problem of information loss in single-scale reconstruction.
[0138] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. It can be understood by a person of ordinary skill in the art that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.
[0139] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.
Claims
1. A multi-source remote sensing optical image registration and fusion method based on spatial deformation field, characterized in that: The method specifically includes: S1: Input high spatial resolution multispectral image Y M and low spatial resolution hyperspectral image Y H , construct a spatial-spectral feature pyramid module based on the channel attention mechanism to extract multi-scale spatial-spectral features of the two image branches; S2: A cross-attention guided spatial feature alignment module is proposed. The multi-scale features of different branches are used to calculate the spatial relationship matrix in a cross-attention manner to generate a deformation field. S3: Use the deformation field to shift the position of the multi-scale features of the hyperspectral image and align them with the multi-scale features of the multispectral image in spatial position; S4: The multi-scale features of the two registered modalities are cascaded in the channel direction, and the features of the same scale are fused using a convolutional neural network to generate multi-scale features with spatial details; S5: Establish a cross-scale information transmission module. Features at multiple scales share information through a convolutional network, complete the interaction of feature information across scales, and gradually enhance the expressiveness of features. S6: Information fusion of enhanced multi-scale features and reconstruction of high spatial resolution hyperspectral images through convolutional neural networks; S7: Perform supervised training on the constructed network model to obtain network parameters suitable for the model.
2. The multi-source remote sensing optical image registration and fusion method based on spatial deformation field as claimed in claim 1, characterized in that: The specific process of S1 is as follows: The normalized image is passed through a linear projection layer to generate three matrices Q, K, and V. The dimensions of the three matrices are reshaped to C×HW, where C represents the channel dimension and HW represents stretching the spatial dimension into a one-dimensional vector. Q and K T After multiplication, the similarity matrix with dimension C×C is obtained through Softmax activation function. The similarity matrix represents the autocorrelation relationship between each channel and spectral vector. Then the similarity matrix is multiplied with V to obtain the intermediate feature F′. After F′ passes through the feedforward layer of the gating mechanism and the dimension reconstruction convolution layer, the dimension is scaled and the feature size is reshaped to i represents the scale level; in this process, the hyperspectral image is gradually collected, and the multispectral image is gradually collected. For the hyperspectral image branch, this process can be expressed by the following formula: Among them, the hyperspectral image features can be expressed as Where H, W represent the height and width of the target hyperspectral image, N represents the number of spectra in the hyperspectral image, n represents the number of spectra in the multispectral image, and i represents the scale level. represents the feature with scale level i+1, i.e., the output of the feature extractor at the upper level of the pyramid; W q ,W k ,W v represents the projection matrix; Q H , K H , V H Respectively represent the Q, K, V matrices generated by the hyperspectral image, Denotes the matrix K H The transpose of represents the scaling factor; Represents the hyperspectral intermediate features of the output of the attention layer; Conv() represents the convolution operation; ReLU() represents the activation function. The feedforward layer of the gating mechanism divides the same matrix into two branches. Each branch is convolved separately and then multiplied. One of the branches is activated by the activation function. The gating mechanism can select valid data to pass to the subsequent network. For the multispectral image branch, the process can be expressed by the following formula: Among them, the multispectral image features can be expressed as represents the feature with scale level i-1, i.e., the output of the feature extractor at the upper level of the pyramid; Q M , K M , V M Respectively represent the Q, K, V matrices generated by the hyperspectral image, Denotes the matrix K M The transpose of Represents the hyperspectral intermediate features output by the attention layer. In addition, to avoid the loss of feature information, residual connections are added to the outputs of the attention layer and feedforward layer of the two branches respectively.
3. The multi-source remote sensing optical image registration and fusion method based on spatial deformation field as claimed in claim 1, characterized in that: The S2 is specifically: First, the features of the two modalities are vectorized and embedded. The features of the multispectral image are projected as Q, and the features of the hyperspectral image are projected as K and V. The cross attention is used to embed Q and K. T Do matrix product, calculate the spatial relationship matrix, cascade the spatial relationship matrix and the matrix V of the hyperspectral image projection to be registered, and send it to the neural network to learn the spatial difference to generate the deformation field. This process can be specifically expressed by the following formula: Spa=QK T D i =f D (concat(Spa,V)) Among them, Spa represents Q and K T The spatial relationship matrix obtained by multiplication, K T represents the transpose of matrix K; f D () represents the solution of deformation field D i The present invention adopts U-net to f D () is used for modeling. In addition, because Spa represents spatial relationships, the dimensions of matrices Q and K are The spatial relationship can be modeled by multiplying the last two dimensions of the matrix.
4. The multi-source remote sensing optical image registration and fusion method based on spatial deformation field as claimed in claim 1, characterized in that: The S3 uses the deformation field to shift the position of the multi-scale features of the hyperspectral image and aligns them with the multi-scale features of the multispectral image in spatial position. The formula is: D i The dimension is The first two dimensions represent the spatial dimensions, and the last dimension is 2, which represents the horizontal and vertical pixel displacements. i Acting on the original features You can get a hyperspectral image that is spatially aligned with the multispectral image.
5. The multi-source remote sensing optical image registration and fusion method based on spatial deformation field as claimed in claim 1, characterized in that: S4 cascades the multi-scale features of the two registered modalities in the channel direction, and uses a convolutional neural network to fuse features of the same scale to generate multi-scale features with spatial details. The formula is: Among them, X i It represents the feature of fusing two image information.
6. The multi-source remote sensing optical image registration and fusion method based on spatial deformation field as claimed in claim 1, characterized in that: The S5 is specifically: Small-scale features are transferred to large-scale and same-scale features, the smallest-scale features no longer transfer information forward, and large-scale features do not transfer information to small-scale features; the formula for this process is: X (i)(j) =f R (X (i)(j-1) ,X (i-1)(j-1) ,…,X (3-j)(j-1) ),j∈[0,3],i∈[0,3-j] Among them, X (i)(j) represents the feature node with scale i in the jth layer, f R () represents the convolutional neural network for interactive aggregation of feature information, X (i)(j-1) ,X (i-1)(j-1) ,…,X (j-1)(j-1) All feature nodes with scale levels from i to 3-j, which represent the output of the j-1 layer, i.e., the previous layer module, are fed into the aggregation network.
7. The multi-source remote sensing optical image registration and fusion method based on spatial deformation field as claimed in claim 1, characterized in that: S6, performs information fusion on the enhanced multi-scale features and reconstructs a high-spatial-resolution hyperspectral image through a convolutional neural network, and its formula is: X (N)(0) Indicates the node with scale 0, i.e. the maximum scale, in the last layer of the multi-scale feature information transmission structure. represents a hyperspectral image with high spatial resolution reconstructed by the network, and R() represents a convolutional neural network used for image reconstruction.
8. The multi-source remote sensing optical image registration and fusion method based on spatial deformation field as claimed in claim 1, characterized in that: S7, performs supervised training on the constructed network model to obtain network parameters suitable for the model, specifically: (7a) The multispectral image Y with the reference image X M and the hyperspectral image Y H The dataset is loaded into the network model to be trained, and the predicted high spatial resolution hyperspectral image is output. (7b) Using the following L1 reconstruction loss function, calculate the predicted image The loss function between and the reference image X is: Where || ||1 represents the 1-norm of the matrix, and Loss represents the loss function; (7c) Use the stochastic gradient descent method to train the network parameters until the network converges and save the optimal network parameters to complete the training of Y H With Y M The Adam optimizer with a learning rate of 1e-3 is used for registration and fusion. The learning rate decreases to 0.1 of itself every 100 epochs, and the learning process is completed after 300 epochs.
Citation Information
Cited By
Video processing method, electronic device, storage medium, chip system and computer program product
CN119211464A
Video processing method, electronic device, storage medium, chip system and computer program product
CN119211464B
Promoter-enhancer interaction prediction method based on DNA sequence
CN120072055A
A method for predicting promoter-enhancer interaction based on DNA sequence
CN120072055B
Remote sensing image registration method, system and equipment based on enhanced feature fusion
CN120339351A
