Multi-scale spatial spectrum information interaction-based multispectral and hyperspectral image registration and fusion joint optimization method

By using a multi-scale spatial-spectral information interaction method, combined with interactive registration and spectral correction, high-quality fusion of hyperspectral images is achieved, solving the problem of image quality degradation caused by non-registration in existing technologies. This method is applicable to fields such as environmental monitoring and military reconnaissance.

CN118887104APending Publication Date: 2024-11-01XIDIAN UNIV

Patent Information

Application Number
CN202411024080.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-11-01

AI Technical Summary

Technical Problem

Most existing hyperspectral image fusion methods neglect the effective use of global and local spatial spectral information and rely heavily on precise registration, resulting in a decline in the quality and accuracy of the fused image and an inability to effectively cope with complex deformation scenarios.

Method used

A joint optimization method for multispectral and hyperspectral image registration and fusion based on multi-scale spatial-spectral information interaction is adopted. By utilizing multi-scale spatial-spectral features at multiple levels and combining interactive registration module, spatial transformation module and spectral correction and information fusion module, joint optimization of registration and fusion is achieved.

Benefits of technology

It improves the reconstruction quality of hyperspectral images, can flexibly cope with complex deformation scenes, and generate fused images with high spatial and hyperspectral resolution, which are suitable for fields such as environmental monitoring, agricultural resource surveys and military reconnaissance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118887104A_ABST
    Figure CN118887104A_ABST
Patent Text Reader

Abstract

The invention discloses a multispectral and hyperspectral image registration and fusion joint optimization method based on multiscale spatial spectrum information interaction. The method comprises the following steps: S101, inputting a multispectral image and a hyperspectral image which are not registered, and respectively extracting multiscale spatial-spectral features of the two images by constructing a feature encoder; s102, forming a registration-fusion joint optimization component, and promoting the network to refine registration and fusion results step by step by using multi-scale spatial spectrum features; s103, generating a deformation field used for non-rigid registration; s104, enabling spatial feature transformation of the hyperspectral image to be aligned with spatial features of the multispectral image at pixel positions; s105, performing high-fidelity spectrum correction and detail enhancement on a registration result; s106, reconstructing a high-resolution hyperspectral image; and S107, building a network model, and carrying out supervised training on the network to obtain network parameters suitable for the model. According to the method, the quality of the reconstructed image is improved by adopting a registration and fusion combined optimization strategy, and the method has the characteristic of being capable of flexibly coping with various complex deformation scenes.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image processing, and in particular relates to a multispectral and hyperspectral image registration and fusion joint optimization method based on multi-scale spatial-spectral information interaction. Background Art

[0002] Compared with natural images, the fine spectral information of hyperspectral images can more accurately reflect the characteristics of different ground objects. However, in the design of optical remote sensing systems, due to the limitations of factors such as equipment and incident energy, the acquired hyperspectral images often cannot have both high spectral resolution and high spatial resolution, which limits their application in the fields of ground object classification, semantic segmentation, change detection and anomaly detection. Image fusion technology aims to integrate complementary information from different source images. One of its research branches, hyperspectral image fusion technology, usually fuses panchromatic or multispectral images with high spatial resolution with hyperspectral images, and finally obtains a fused image with both high spatial resolution and high spectral resolution. Since hyperspectral image fusion technology can effectively improve the quality, details and information content of images, it has a very broad research prospect.

[0003] In the past two decades, many scholars have proposed some classic methods for hyperspectral image fusion tasks, which can be roughly divided into four categories: component replacement, multi-resolution analysis, Bayesian estimation and matrix decomposition. The component replacement method first converts the hyperspectral image to other domains to separate the spectral and spatial information, then uses the multispectral image to replace the spatial information part of the hyperspectral image, and finally performs an inverse transformation to obtain the fused image. The multi-resolution analysis method first decomposes the multispectral image and the upsampled hyperspectral image into multiple scales, and then injects spatial information into the hyperspectral image according to rules at different scales. Both the Bayesian estimation method and the matrix decomposition method assume that the hyperspectral image and the multispectral image are modeled as degraded images and linear combinations of ideal high-resolution hyperspectral images, respectively, and then restore the ideal high-resolution hyperspectral image by estimating the solution to the inverse problem.

[0004] However, all of the above four methods have some defects. Although the principle of the component replacement method is simple, it will cause serious spectral distortion; the multi-resolution analysis method can maintain strong spectral consistency, but it is easy to introduce artifacts when injecting spatial information, and the detailed information is seriously lost; the Bayesian estimation and matrix decomposition methods can retain spectral and spatial information to the greatest extent, but they are highly dependent on prior information and have high algorithm complexity.

[0005] In recent years, deep learning has made significant progress in the field of remote sensing image processing, and many hyperspectral image fusion methods based on auto encoders, convolutional neural networks (CNNs) and generative adversarial networks (GANs) have emerged. These methods can achieve very excellent performance due to their powerful feature extraction and expression capabilities, but most of them only focus on the fusion process and rely heavily on the precise registration relationship between input images. However, in most cases, due to factors such as terrain undulations and atmospheric turbulence, it is impossible to achieve strict spatial registration between images captured by different sensors, which greatly weakens the diversity of hyperspectral image fusion networks in practical applications.

[0006] In view of the above analysis, the technical problems that need to be solved urgently in the prior art are:

[0007] (1) Most existing methods ignore the effective use of global and local spatial spectral information, which leads to a decrease in the overall quality and accuracy of the fused image, affecting its effect and performance in practical applications.

[0008] (2) Although many deep learning-based hyperspectral image fusion methods have achieved good results, they are usually based only on accurately registered source images or consider registration and fusion as a two-stage task, which means that the fusion must tolerate the error accumulation caused by improper registration. Summary of the invention

[0009] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a multispectral and hyperspectral image registration and fusion joint optimization method based on multi-scale spatial-spectral information interaction. The method adopts a strategy of joint optimization of registration and fusion to improve the quality of reconstructed images, and has the characteristics of being able to flexibly cope with various complex deformation scenes.

[0010] In order to achieve the above object, the technical solution adopted by the present invention is:

[0011] The multispectral and hyperspectral image registration and fusion joint optimization method based on multi-scale spatial-spectral information interaction includes the following steps:

[0012] S101: inputting an unregistered multispectral image and a hyperspectral image, constructing a feature encoder on two independent branches to extract multi-scale spatial-spectral features of the two images respectively;

[0013] S102: Designing an interactive registration module, a spatial transformation module, and a spectral correction and information fusion module to jointly constitute a registration-fusion joint optimization component, and utilizing the multi-scale spatial-spectral features to promote the network to gradually refine the registration and fusion results;

[0014] S103: the interactive registration module globally and adaptively learns the association and difference between the spatial features of the multispectral and hyperspectral images, and generates a deformation field for non-rigid registration;

[0015] S104: performing spatial transformation on the hyperspectral image spatial features based on the deformation field so as to align the hyperspectral image spatial features with the multispectral image spatial features in terms of pixel positions;

[0016] S105: The spectral correction and information fusion module performs spectral correlation embedding and spatial information injection on two branches to achieve high-fidelity spectral correction and detail enhancement of the registration results;

[0017] S106: construct a decoder to restore the output features of the registration-fusion joint optimization component at the final level to the original space and reconstruct a high-resolution hyperspectral image;

[0018] S107: Use the feature encoder, decoder, and multiple registration-fusion joint optimization components to jointly build a network model, perform supervised training on the network, and obtain network parameters suitable for the model.

[0019] As a further technical solution of the present invention, step S101 is specifically as follows:

[0020] The feature encoders on both branches are built based on convolutional neural networks, which are f The spatial features extracted from The spatial resolution of spatial features gradually transitions from the finest level n to the coarsest level 1, where the feature map size of level i is 1 / 2 of that of level (i+1). This process is expressed by the following formula:

[0021]

[0022] in represents the combination of 3×3 convolutional layer and LeakyReLU layer, j represents AvgPooling(·) represents the global average pooling layer, from the hyperspectral image H m The features extracted from in represents spatial features, and the rest are spectral features. The resolution of spectral features gradually transitions from the coarsest level 1 to the finest level n, where the feature map size of the (i+1) level is twice that of the i level. This process is expressed by the following formula:

[0023]

[0024] in represents the combination of a 1×1 convolutional layer and a LeakyReLU layer, and TransConv(·) represents a transposed convolutional layer.

[0025] As a further technical solution of the present invention, step S102 is specifically as follows:

[0026] The registration-fusion joint optimization component is composed of an interactive registration module, a spatial transformation and spectral correction and information fusion module. This component is distributed in multiple levels for iteration, making full use of the spatial-spectral features at the corresponding level to reconstruct high-resolution hyperspectral images step by step from global to local.

[0027] Let the output of the (i-1)th registration-fusion joint optimization component be the rough result, and the i-th (i≠1) registration-fusion joint optimization component will use the output of the (i-1)th component and the spatial features of the multispectral image and spectral characteristics of hyperspectral images To achieve further optimization, the above process can be expressed as:

[0028]

[0029] in, represents the registration-fusion joint optimization component at level i, represents the output of the (i-1)th level registration-fusion joint optimization component, represents the output features of the i-th level registration-fusion joint optimization component.

[0030] As a further technical solution of the present invention, step S103 is specifically as follows:

[0031] The input feature map of the interactive registration module is and in represents the multispectral image features extracted at level i, represents the output of the (i-1)th level registration-fusion joint optimization component.

[0032] First, use a linear layer to and The mapping results in two different sets of query vectors Q (Query), key vectors K (Key) and value vectors V (Value), defined as and In order to achieve information interaction, a cross-attention mechanism is designed to generate feature representations containing spatial correlations; specifically, a cross-attention mechanism is used on two branches based on and The feature vector in learns two spatial attention maps:

[0033]

[0034] in is the scaling factor, Reflects for The degree of relevance, Reflects right They all help to focus on the differences and association information in the image in the subsequent generation of the deformation field. Since the generation of the deformation field needs to ensure that the semantic information of the input image pair and the learned spatial association information are consistent in position, it is necessary to and The corresponding and Connect and obtain Z (i) Will have a richer feature representation, the above process is expressed as:

[0035]

[0036] Where Z (i) It not only contains the input feature information, but also introduces the spatial attention map at the corresponding position, which will prompt more attention to certain areas in the subsequent generation of spatial transformation parameters, and these areas can well reflect the association and difference between features. Then, Z (i) Input is an encoder-decoder with skip connections; both the encoder and decoder are composed of stacked convolutional layers, batch normalization layers, and LeakyReLU layers, and the convolutional layer stride of the encoder is set to 2 to gradually reduce the size of the feature map, and the transposed convolutional layer is set in the decoder to gradually restore the size of the feature map. The final output is a spatial transformation parameter matrix U of size H×W×2. φ is generated by distorting the standard grid based on the spatial transformation parameter matrix U, which is expressed as:

[0037] φ=Id+U

[0038] Where Id represents the identity transformation describing the pixel coordinate position, and φ is the output deformation field.

[0039] As a further technical solution of the present invention, the step S104 is specifically as follows:

[0040] The deformation field φ is a grid matrix containing spatial transformation parameters. The hyperspectral image features are spatially transformed using φ to achieve alignment of the hyperspectral image features with the multispectral image features at the pixel position. For the registration result obtained at level i, Each pixel point p can be Find the corresponding pixel p′=p+U(p), where U(p) is a displacement such that and corresponding to similar positions;

[0041] It should be noted that the p′ obtained from the above calculation may be a decimal, but There are values ​​only at positions where the pixel coordinates are integers, so we choose to perform bilinear interpolation on the four pixel values ​​with integer coordinates around p' to obtain the value of p', and then assign the value of p' to p to complete the spatial transformation. The above steps are expressed as:

[0042]

[0043] Where Z(p′) represents the neighborhood pixels of p′.

[0044] As a further technical solution of the present invention, step S105 is specifically as follows:

[0045] The input feature map of the spectral correction and spatial information fusion module is the registration result obtained at the i-th level Spectral characteristics of hyperspectral images and multispectral image spatial features

[0046] First, an adaptive average pooling layer is used to Converted into a one-dimensional vector, the one-dimensional vector is considered to be the global spectral feature representation corresponding to each channel, and then multiple fully connected layers are used to adaptively learn the nonlinear relationship between the one-dimensional vectors to further analyze the dependency between spectra. The above process can be expressed as:

[0047]

[0048] W ( i ) represents the one-dimensional spectral feature representation obtained after the fully connected layer, f FC (·) represents a fully connected layer, and AdaPooling (·) represents an adaptive average pooling layer. (i) To achieve spectral recalibration, adaptive instance normalization (AdaIN) is used to globally adjust the feature map channel by channel. First, W (i) Normalize, and obtain the scaling and translation coefficients after a learnable affine transformation, and then use the obtained coefficients to Perform channel-by-channel recalibration, which is defined as:

[0049] ys1,i,yb1,i=faffine(Norm(W ( i ) ))

[0050]

[0051] where y s1,iand b1,i denote the learnable scaling and translation coefficients, respectively. represents the output of the AdaIN layer, f affine (·) represents a learnable affine transformation, Norm(·) represents normalization, and Respectively The mean and variance of .

[0052] In the spatial information fusion branch, The frequency block and the spatial block are input to extract the global and local information of the features respectively, and then integrated into the spectral correction branch, and the information is integrated through the subsequent convolution and activation function layers. The process is defined as:

[0053]

[0054]

[0055] Among them, f spa (·) represents the spatial information fusion branch, f conv,3 (·) denotes the convolutional layer and the activation function layer; specifically, the spatial block is composed of cascaded residual blocks, and the frequency block is composed of Based on Fourier transform, it is decomposed into phase A and amplitude P components, and then two convolutional layers are used to restore the two components respectively. and Finally, the inverse Fourier transform is used to output the global information.

[0056] As a further technical solution of the present invention, step S106 is specifically as follows:

[0057] The decoder is similar to the feature encoder and is also built based on a convolutional neural network. The decoder is used to restore the output features of the n-th level registration-fusion joint optimization component to the original space. The formula is:

[0058]

[0059] where f rec (·) represents the decoder for image reconstruction, represents the output features of the n-th level registration-fusion optimization component, Represents the reconstructed high-resolution hyperspectral image.

[0060] As a further technical solution of the present invention, step S107 is specifically as follows:

[0061] The unregistered multispectral image M f and the hyperspectral image H m The dataset is loaded into the network model to be trained, and the predicted high-resolution hyperspectral image is output. The loss is calculated using the L1 reconstruction loss function, expressed as:

[0062]

[0063] where f Net (·) represents the constructed network model, represents the reconstructed nth high-resolution hyperspectral image, represents the nth reference image, N represents the number of images in a training batch, Θ ​​represents the parameter set of the model. During the training process, the network parameters are updated based on the Adam optimizer until the network converges. The batchsize is set to 4, the total number of epochs is set to 400, the learning rate is initialized to 0.001, and the learning rate is decayed by 10 times at epochs of 50, 150, and 300. After the training, the optimal network parameters are saved.

[0064] Beneficial effects of the present invention:

[0065] First, the present invention integrates registration and fusion into one component, so that it can gradually refine the registration and fusion results at multiple levels using multi-scale spatial-spectral information to produce better high-resolution hyperspectral image reconstruction effects.

[0066] The present invention proposes an interactive registration module, which generates a deformation field based on spatial correlation and dissimilarity captured at a long distance to repair non-rigid pixel offsets and achieve spatial registration.

[0067] The present invention designs a spectral correction and spatial information fusion module, which fully utilizes the spatial spectrum features on the two branches to perform spectral correlation embedding and spatial information injection, thereby achieving high-fidelity spectral correction and detail enhancement of the registration results.

[0068] Second, (1) the expected benefits and commercial value of the technical solution of the present invention after transformation are:

[0069] The multispectral and hyperspectral image registration and fusion joint optimization method based on multi-scale spatial-spectral information interaction proposed in the present invention solves the problem of unsatisfactory fusion results caused by the misregistration of source images in existing methods. The fused image obtained by the present invention has both high spatial and high spectral resolution and can be widely used in environmental monitoring, agricultural resource surveys, military reconnaissance and other fields. These fields have an urgent need for high-quality image data. After applying this technology, the effect and accuracy of various applications can be significantly improved, promoting technological progress and business development in various industries.

[0070] (2) The technical solution of the present invention fills the technical gap in the industry at home and abroad:

[0071] The present invention fills the technical gap of achieving high-precision fusion under the condition of unregistered multispectral and hyperspectral images. Existing fusion methods rely heavily on accurately registered images or regard registration and fusion as two independent stages, resulting in low-quality fusion results due to the accumulation of registration errors. The present invention integrates registration and fusion into one component, and progressively realizes the joint optimization of registration and fusion at multiple levels using multi-scale spatial-spectral features in an information interactive manner, achieving high-quality hyperspectral image reconstruction, and bringing new breakthroughs to the field of hyperspectral image processing and application, with important scientific research value and broad application prospects.

[0072] (3) The technical solution of the present invention solves the technical problems that people have been eager to solve but have never been able to solve successfully:

[0073] The present invention utilizes multi-scale spatial-spectral features to realize the joint optimization of registration and fusion of multispectral and hyperspectral images for the first time in an information interactive manner, effectively solving two core problems in the long-standing hyperspectral image fusion processing: high-precision fusion of unregistered multispectral and hyperspectral images. Most existing methods ignore the effective use of global and local spatial-spectral information, and rely heavily on accurately registered source images or regard registration and fusion as two-stage tasks. Fusion must tolerate the accumulation of errors caused by improper registration. The present invention effectively solves the above-mentioned problems by introducing a progressive registration and fusion joint optimization strategy, realizes the high-precision fusion of unregistered multispectral and hyperspectral images, and greatly improves the quality and practicality of the fused images.

[0074] (4) The technical solution of the present invention overcomes technical prejudice:

[0075] The multispectral and hyperspectral image registration and fusion joint optimization method based on multi-scale spatial-spectral information interaction proposed in the present invention breaks the strict reliance on registration accuracy in the existing multispectral and hyperspectral image fusion process and overcomes the inherent bias in the prior art. The prior art usually regards registration and fusion as two independent steps, which easily leads to cumulative errors. The present invention utilizes multi-scale spatial-spectral information, realizes the joint optimization of registration and fusion through interactive registration module, spatial transformation, and spectral correction and spatial information fusion module, and significantly improves the quality and accuracy of the fused image. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 It is a flow chart of a multispectral and hyperspectral image registration and fusion joint optimization method based on multi-scale spatial-spectral information interaction provided by an embodiment of the present invention.

[0077] Figure 2 It is a network structure diagram of a multispectral and hyperspectral image registration and fusion joint optimization method based on multi-scale spatial-spectral information interaction provided by an embodiment of the present invention.

[0078] Figure 3 It is a schematic diagram of the structure of an interactive registration module provided in an embodiment of the present invention.

[0079] Figure 4 It is a schematic diagram of the structure of the spectrum correction and information fusion module provided in an embodiment of the present invention.

[0080] Figure 5 : The results of the comparison method and the proposed method provided in the embodiment of the present invention are as follows: (a) is the result of the HySure method; (b) is the result of the CNMF method; (c) is the result of the MoG-DCN method; (d) is the result of the Integrated method; (e) is the result of the u 2 (f) is the result diagram of the MDN method; (g) is the result diagram of the present invention; (h) is the reference image. DETAILED DESCRIPTION

[0081] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0082] In view of the problems existing in the prior art, the present invention provides a multispectral and hyperspectral image registration and fusion joint optimization method based on multi-scale spatial-spectral information interaction. The present invention is described in detail below with reference to the accompanying drawings.

[0083] like Figure 1 As shown, the multi-spectral and hyperspectral image registration and fusion joint optimization method with multi-scale spatial-spectral information interaction provided by the embodiment of the present invention includes the following steps:

[0084] S101: Input an unregistered multispectral image and a hyperspectral image, and construct a feature encoder on two independent branches to extract multi-scale spatial-spectral features of the two images respectively;

[0085] S102: Design interactive registration module, spatial transformation and spectral correction and information fusion module to form a registration-fusion joint optimization component, and use multi-scale spatial-spectral features to promote the network to gradually refine the registration and fusion results;

[0086] S103: The interactive registration module adaptively learns the associations and differences between feature regions globally and generates a deformation field for non-rigid registration;

[0087] S104: performing spatial transformation on the hyperspectral image feature based on the deformation field so as to align the hyperspectral image feature with the multispectral image feature in terms of pixel position;

[0088] S105: The spectral correction and information fusion module performs spectral correlation embedding and spatial information injection on two branches to achieve high-fidelity spectral correction and detail enhancement of the registration results;

[0089] S106: construct a decoder, restore the features to the original space, and reconstruct the high-resolution hyperspectral image;

[0090] S107: Perform supervised training on the constructed network model to obtain network parameters suitable for the model.

[0091] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0092] like Figure 1 As shown, the multi-spectral and hyperspectral image registration and fusion joint optimization method with multi-scale spatial-spectral information interaction provided by the embodiment of the present invention has the following implementation process:

[0093] (1) An unregistered multispectral image and a hyperspectral image are input, and a feature encoder is constructed on two independent branches to extract the multi-scale spatial-spectral features of the two images respectively.

[0094] (1a) Construct a feature encoder to extract the spatial features of multispectral images, expressed as Their spatial resolution gradually transitions from the finest level n to the coarsest level 1, where the feature map size of level i is 1 / 2 of that of level (i+1). This process can be expressed by the following formula:

[0095]

[0096] in represents the combination of 3×3 convolutional layer and LeakyReLU layer, j represents , AvgPooling(·) represents the global average pooling layer.

[0097] (1b) Construct a feature encoder to extract the spatial spectrum features of the hyperspectral image, expressed as in represents spatial features, and the rest are spectral features. Their spatial resolution gradually transitions from the coarsest level 1 to the finest level n, where the feature map size of the (i+1) level is twice that of the i level. This process can be expressed by the following formula:

[0098]

[0099]

[0100] in represents the combination of a 1×1 convolutional layer and a LeakyReLU layer, and TransConv(·) represents a transposed convolutional layer.

[0101] (2) Figure 2As shown in the figure, the core component of the network structure is the registration-fusion joint optimization component, which is composed of an interactive registration module, a spatial transformation and spectral correction and information fusion module. This component uses the spatial-spectral features at the corresponding level to reconstruct a high-resolution hyperspectral image from the global to the local. This step promotes the network to achieve joint optimization of registration and fusion step by step to flexibly cope with deformations of different degrees.

[0102] The registration-fusion joint optimization components are distributed in multiple levels. The components make full use of the spatial-spectral features of multispectral and hyperspectral images to achieve joint optimization of registration and fusion. The output feature maps of the registration-fusion joint optimization components at each level will gradually transition from low resolution to high resolution. The output of the (i-1)th registration-fusion joint optimization component is a rough result. The i-th (i≠1) registration-fusion joint optimization component will use the output of the (i-1)th module and the spatial features of the multispectral image. and spectral characteristics of hyperspectral images To achieve further optimization. The above process can be expressed as:

[0103]

[0104] in, represents the registration-fusion joint optimization module at level i, and Represent the input and output features of the i-th level module respectively.

[0105] (3) Figure 3 As shown in the figure, the interactive registration module globally and adaptively learns the associations and differences between feature regions to generate a deformation field for non-rigid registration. The module first finds the spatial correlation between input features through the designed cross-attention mechanism, and then warps the standard grid based on the spatial transformation parameters output by the encoder-decoder to obtain the deformation field. This step provides an accurate deformation field for the subsequent spatial transformation part to repair non-rigid pixel offsets.

[0106] Specifically:

[0107] (3a) Input feature map and in represents the multispectral image features extracted at level i, represents the output of the (i-1)th level registration-fusion joint optimization module. Using a linear layer and The mapping results in two different sets of query vectors Q (Query), key vectors K (Key) and value vectors V (Value), defined as and

[0108] (3b) In order to achieve information interaction, a cross-attention mechanism is designed to generate feature representations that contain spatial correlations. Specifically, a cross-attention mechanism is used on the two branches based on and The feature vector in learns two spatial attention maps:

[0109]

[0110] in is the scaling factor, Reflects for The degree of relevance, Reflects right The correlation between the input image pairs and the learned spatial association information is necessary to ensure that the semantic information of the input image pairs and the learned spatial association information are consistent in position. and The corresponding and Connect and obtain Z (i) Will have a richer feature representation. The above process can be expressed as:

[0111]

[0112] Where Z (i) It not only contains the input feature information, but also introduces a spatial attention map at the corresponding position. This will encourage more focus on certain areas in the subsequent generation of spatial transformation parameters, and these areas can well reflect the association and difference between features.

[0113] (3c) Z (i) Input is an encoder-decoder with skip connections, and the output is a spatial transformation parameter matrix U of size H×W×2. φ is generated by distorting the standard grid based on the spatial transformation parameter matrix U, expressed as:

[0114] φ=Id+U

[0115] Where Id represents the identity transformation describing the pixel coordinate position, and φ is the output deformation field.

[0116] (4) The hyperspectral image features are spatially transformed based on the deformation field so that they are aligned with the multispectral image features in terms of pixel position. This step repairs the non-rigid pixel offset of the hyperspectral image features and achieves spatial registration.

[0117] Specifically:

[0118] Since the deformation field φ is just a grid matrix containing spatial transformation parameters, if we want to align the hyperspectral image features with the multispectral image features at the pixel position, we need to use φ to perform spatial transformation on the hyperspectral image features. Each pixel point p can be Find the corresponding pixel p′=p+U(p), where U(p) is a displacement such that and Corresponding to similar positions. It should be noted that the p′ obtained from the above calculation may be a decimal, but There are values ​​only at positions where the pixel coordinates are integers, so we choose to perform bilinear interpolation on the four pixel values ​​with integer coordinates around p' to obtain the value of p', and then assign the value of p' to p to complete the spatial transformation. The above steps can be expressed as:

[0119]

[0120] Where Z(p′) represents the neighborhood pixels of p′.

[0121] (5) Figure 4 As shown in the figure, the spectral correction and information fusion module performs spectral correlation embedding and spatial information injection on two branches to achieve high-fidelity spectral correction and detail enhancement of the registration results. Specifically:

[0122] (5a) The input feature map is the registration result obtained at the i-th level Spectral characteristics of hyperspectral images and multispectral image spatial features Using an adaptive average pooling layer It is converted into a one-dimensional vector, which can be considered as the global spectral feature representation corresponding to each channel. Then, multiple fully connected layers are used to adaptively learn the nonlinear relationship between the one-dimensional vectors to further analyze the dependency between spectra. The above process can be expressed as:

[0123]

[0124] W ( i ) represents the one-dimensional spectral feature representation obtained after the fully connected layer, f FC (·) represents a fully connected layer, and AdaPooling(·) represents an adaptive average pooling layer.

[0125] (5b) In order to utilize W (i) To achieve spectral recalibration, adaptive instance normalization (AdaIN) is used to globally adjust the feature map channel by channel. (i)Normalized, after a learnable affine transformation, the scaling and translation coefficients are obtained. Then the obtained coefficients are used to Perform channel-by-channel recalibration. The process can be defined as:

[0126] ys1,i,yb1,i=faffine(Norm(W ( i ) ))

[0127]

[0128] where y s1,i and b1,i denote the learnable scaling and translation coefficients, respectively. represents the output of the AdaIN layer, f affine (·) represents a learnable affine transformation, Norm(·) represents normalization, and Respectively The mean and variance of .

[0129] (5c) In the spatial information fusion branch, The frequency block and the spatial block are input respectively to extract the global and local information of the features, and then integrated into the spectral correction branch, and the information integration is realized through the subsequent convolution and activation function layers. The process can be defined as:

[0130]

[0131] Among them, f spa (·) represents the spatial information fusion branch, f conv,3 (·) represents the convolutional layer and activation function layer. Specifically, the spatial block is composed of cascaded residual blocks. The frequency block is Based on Fourier transform, it is decomposed into phase A and amplitude P components, and then two convolutional layers are used to restore the two components respectively. and Finally, the inverse Fourier transform is used to output the global information.

[0132] (6) Construct a decoder to restore the output features of the registration-fusion joint optimization component at the final level to the original space and reconstruct a high-resolution hyperspectral image. This step maps the features from the high-dimensional space to the original low-dimensional space to obtain a high-fidelity high-resolution hyperspectral image.

[0133] Specifically:

[0134] The decoder is similar to the feature encoder and is also built based on a convolutional neural network. The decoder is used to restore the output features of the n-th level registration-fusion joint optimization component to the original space. The formula is:

[0135]

[0136] where f rec (·) represents the decoder for image reconstruction, represents the output features of the n-th level registration-fusion optimization component, Represents the reconstructed high-resolution hyperspectral image.

[0137] (7) Perform supervised training on the constructed network model to obtain network parameters suitable for the model, specifically:

[0138] (7a) The unregistered multispectral image M f and the hyperspectral image H m The dataset is loaded into the network model to be trained, and the predicted high-resolution hyperspectral image is output.

[0139] (7b) The loss is calculated using the L1 reconstruction loss function, expressed as:

[0140]

[0141] where f Net (·) represents the constructed network model, represents the reconstructed nth high-resolution hyperspectral image, represents the nth reference image, N represents the number of images in a training batch, and Θ represents the parameter set of the model.

[0142] (7c) During the training process, the network parameters are updated based on the Adam optimizer until the network converges. The batch size is set to 4, the total number of epochs is set to 400, the learning rate is initialized to 0.001, and the learning rate is decayed by 10 times at epochs 50, 150, and 300. After the training is completed, the optimal network parameters are saved.

[0143] The technical effects of the present invention are described in detail below in conjunction with simulation experiments:

[0144] 1. Simulation Experiment Conditions

[0145] The hardware platform of the simulation experiment of the present invention is: Nvidia GeForce RTX 3090, Intel(R) Core(TM) i9-10900X CPU@3.70GHz.

[0146] The software platforms for the simulation experiment of the present invention are: Linux18.06 operating system, python3.7 and pytorch1.12.

[0147] The hyperspectral image used in the simulation experiment of the present invention adopts the Chikusei dataset. The dataset was acquired by the HeadwallHyperspec-VNIR-C imaging sensor in Chikusei, Ibaraki, Japan in 2014. The size of the original hyperspectral image is 2304×2048×128, the spatial resolution is 2.5m, and it covers 128 bands from 363nm to 1018nm. The dataset is divided into 36 non-overlapping image blocks, 27 of which are used as training sets and the remaining 9 are used as test sets. In order to expand the number of samples, each image block is further divided into 8 image blocks of size 256×256×128 with a fixed step size.

[0148] The acquired high-resolution hyperspectral image is used as the reference image, and a low-resolution hyperspectral image and a high-resolution multispectral image are generated according to the Wald protocol. For the low-resolution hyperspectral image, the reference image is blurred using an isotropic Gaussian blur kernel and then reduced to 1 / 4 of the original spatial size. The high-resolution multispectral image is obtained by using the blue-green-red-near-infrared spectral response function on the reference image. In order to obtain unregistered multispectral and hyperspectral image pairs, elastic deformation is used for the low-resolution hyperspectral image in the experiment to simulate the non-rigid deformation that may occur during image acquisition in reality.

[0149] 2. Evaluation indicators

[0150] Five widely used spatial and spectral evaluation indicators are selected for quantitative evaluation, which are peak signal-to-noise ratio (PSNR), correlation coefficient (CC), spectral angle mapping (SAM), root mean square error (RMSE) and structural similarity index (SSIM). The higher the PSNR, CC and SSIM, the lower the SAM, ERGAS and RMSE, indicating that the quality of the fused image is higher. These indicators are widely used in the field of multispectral and hyperspectral image fusion, which helps to comprehensively evaluate the network model from an objective level.

[0151] 3. Experimental content and results analysis

[0152] In order to verify the effectiveness of the present invention, six widely used remote sensing image fusion methods are selected, including CNMF, HySure, MoG-DCN, Integrated, NonRegSRNet and u 2 MDN, respectively, trains and tests the Chikusei dataset to obtain the fusion results of the final test set. CNMF, HySure and Integrated are classic methods, and the rest are methods based on deep learning.

[0153] Table 1 Comparison of performance indicators of the present invention and the comparative algorithm

[0154]

[0155] A comprehensive subjective and objective evaluation of the fusion results of the proposed method and the comparison method was conducted on the Chikusei dataset with elastic deformation. Table 1 shows the objective evaluation results of the fusion. The bold font indicates the best indicator. Figure 5 The output fusion image and its corresponding residual map are shown. (a)-(g) represent the fusion images of HySure, CNMF, MoG-DCN, Integrated, and u 2 The results obtained by MDN, NonRegSRNet and the method of the present invention, (g) represents the reference image. The residual image is calculated based on MSE, and the display range is set to 0-0.1. The darker the color and the less the residual image, the higher the fusion quality.

[0156] From Table 1 and Figure 5 It can be seen that, except for MoG-DCN and the method of the present invention, the other algorithms all perform poorly and will produce relatively serious fusion errors. MoG-DCN is based on the idea of ​​model guidance and can achieve relatively excellent fusion effects, but there is still a slight gap compared with the algorithm of the present application. In general, the present invention performs best in all indicators and can obtain the best visual effect visualization image and the darkest color residual map, which proves its effectiveness.

[0157] The technical effects of the present invention are described in detail below in conjunction with experiments.

[0158] In order to prove the effectiveness of the multi-level iterative optimization of the registration-fusion joint optimization component designed by the present invention, this experiment also designs a network model that only uses the registration-fusion joint optimization component at a single scale. The experimental results are as follows:

[0159] Table 2 Comparison of performance indicators of the present invention and its variants having only a single-stage structure

[0160]

[0161] Keep other settings unchanged and optimize the parameters, and the results are shown in Table 2. Compared with the single-level structure, the present invention can achieve better performance. This is because when the low-resolution hyperspectral image undergoes a large-scale deformation, the multi-level iteration method can not only maintain global consistency in the process of reconstructing the high-resolution hyperspectral image, but also accurately adjust its local details.

[0162] In order to prove the effectiveness of the interactive registration module, this experiment designed a variant, that is, to delete the information interaction part of the interactive registration module, and directly generate the deformation field after connecting the input image features along the channel dimension. The experimental results are as follows:

[0163] Table 3 Comparison of performance indicators of the present invention and its variants removing information interaction

[0164]

[0165] In the embodiment of the present invention, the interactive registration module globally pays attention to the spatial correlation between the feature regions of the input image in an information interactive manner, so that the difference and correlation between the images are focused on when generating the deformation field. It can be intuitively seen from Table 3 that the present invention proves that the interactive registration module is effective in promoting the generation of a more refined deformation field.

[0166] In order to prove the effectiveness of the spectrum correction and information fusion module, this experiment designs two variants for research: removing the spatial information fusion branch and removing the spectrum correction branch. The experimental results are as follows:

[0167] Table 4 Performance comparison of the present invention and its variants with the spatial information fusion / spectral correction branch removed

[0168]

[0169] It can be seen from Table 4 that the removal of the spatial information fusion branch and the removal of the spectrum correction branch have poor performance due to the lack of spatial information injection and spectrum correction respectively. Experiments have shown that the module designed in the present invention is sufficient to achieve good spectrum correction and spatial information fusion.

[0170] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When the use is implemented in whole or in part in the form of a computer program product, the computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL) or wireless (e.g., infrared, wireless, microwave, etc.) mode) to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk SolidState Disk (SSD)), etc.

[0171] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. Joint optimization method of multispectral and hyperspectral image registration and fusion based on multi-scale spatial-spectral information interaction, It is characterized in that The steps include: S101: inputting an unregistered multispectral image and a hyperspectral image, constructing a feature encoder on two independent branches to extract multi-scale spatial-spectral features of the two images respectively; S102: Designing an interactive registration module, a spatial transformation module, and a spectral correction and information fusion module to jointly constitute a registration-fusion joint optimization component, and utilizing the multi-scale spatial-spectral features to promote the network to gradually refine the registration and fusion results; S103: the interactive registration module globally and adaptively learns the association and difference between the spatial features of the multispectral and hyperspectral images, and generates a deformation field for non-rigid registration; S104: performing spatial transformation on the hyperspectral image spatial features based on the deformation field so as to align the hyperspectral image spatial features with the multispectral image spatial features in terms of pixel positions; S105: The spectral correction and information fusion module performs spectral correlation embedding and spatial information injection on two branches to achieve high-fidelity spectral correction and detail enhancement of the registration results; S106: construct a decoder to restore the output features of the registration-fusion joint optimization component at the final level to the original space and reconstruct a high-resolution hyperspectral image; S107: Use the feature encoder, decoder, and multiple registration-fusion joint optimization components to jointly build a network model, perform supervised training on the network, and obtain network parameters suitable for the model.

2. The multispectral and hyperspectral image registration and fusion joint optimization method based on multi-scale spatial-spectral information interaction according to claim 1, It is characterized in that The step S101 is specifically as follows: The feature encoders on both branches are built based on convolutional neural networks, which are f The spatial features extracted from The spatial resolution of spatial features gradually transitions from the finest level n to the coarsest level 1, where the feature map size of level i is 1 / 2 of that of level (i+1). This process is expressed by the following formula: in represents the combination of 3×3 convolutional layer and LeakyReLU layer, j represents AvgPooling(·) represents the global average pooling layer, from the hyperspectral image H m The features extracted from in represents spatial features, and the rest are spectral features. The resolution of spectral features gradually transitions from the coarsest level 1 to the finest level n, where the feature map size of the (i+1) level is twice that of the i level. This process is expressed by the following formula: in represents the combination of a 1×1 convolutional layer and a LeakyReLU layer, and TransConv(·) represents a transposed convolutional layer.

3. The multispectral and hyperspectral image registration and fusion joint optimization method based on multi-scale spatial-spectral information interaction according to claim 2, It is characterized in that The step S102 is specifically as follows: The registration-fusion joint optimization component is composed of an interactive registration module, a spatial transformation and spectral correction and information fusion module. This component is distributed in multiple levels for iteration, making full use of the spatial-spectral features at the corresponding level to reconstruct high-resolution hyperspectral images step by step from global to local. Let the output of the (i-1)th registration-fusion joint optimization component be the rough result, and the i-th (i≠1) registration-fusion joint optimization component will use the output of the (i-1)th component and the spatial features of the multispectral image and spectral characteristics of hyperspectral images To achieve further optimization, the above process can be expressed as: in, represents the registration-fusion joint optimization component at level i, represents the output of the (i-1)th level registration-fusion joint optimization component, represents the output features of the i-th level registration-fusion joint optimization component.

4. The multispectral and hyperspectral image registration and fusion joint optimization method based on multi-scale spatial-spectral information interaction according to claim 3, It is characterized in that The step S103 is specifically as follows: The input feature map of the interactive registration module is and in represents the multispectral image features extracted at level i, represents the output of the (i-1)th level registration-fusion joint optimization component; First, use a linear layer to and The mapping results in two different sets of query vectors Q (Query), key vectors K (Key) and value vectors V (Value), defined as and Design a cross-attention mechanism to generate feature representations containing spatial correlations; adopt a cross-attention form on two branches based on and The feature vector in learns two spatial attention maps: in is the scaling factor, Reflects for The degree of relevance, Reflects right The degree of relevance will and The corresponding and Connect and obtain Z (i) , the above process is expressed as: Where Z (i) It contains both the input feature information and the spatial attention map at the corresponding position. Then, Z (i) Input encoder-decoder with skip connections; Both the encoder and decoder are composed of a stack of convolutional layers, batch normalization layers, and LeakyReLU layers. The final output is a spatial transformation parameter matrix U of size H×W×2. φ is generated by distorting the standard grid based on the spatial transformation parameter matrix U, which is expressed as: φ=Id+U Where Id represents the identity transformation describing the pixel coordinate position, and φ is the output deformation field.

5. According to claim 4, the multispectral and hyperspectral image registration and fusion joint optimization method based on multi-scale spatial-spectral information interaction, It is characterized in that The step S104 is specifically as follows: The deformation field φ is a grid matrix containing spatial transformation parameters. The hyperspectral image features are spatially transformed using φ to achieve alignment of the hyperspectral image features with the multispectral image features at the pixel position. For the registration result obtained at level i, Each pixel point p can be Find the corresponding pixel p′=p+U(p), where U(p) is a displacement such that and corresponding to similar positions; Select the four pixel values ​​with integer coordinates around p' to perform bilinear interpolation to obtain the value of p', and then assign the value of p' to p to complete the spatial transformation. The above steps are expressed as: Where Z(p′) represents the neighborhood pixels of p′.

6. The multispectral and hyperspectral image registration and fusion joint optimization method based on multi-scale spatial-spectral information interaction according to claim 5, It is characterized in that The step S105 is specifically as follows: The input feature map of the spectral correction and spatial information fusion module is the registration result obtained at the i-th level Spectral characteristics of hyperspectral images and multispectral image spatial features First, an adaptive average pooling layer is used to It is converted into a one-dimensional vector, and then multiple fully connected layers are used to adaptively learn the nonlinear relationship between the one-dimensional vectors to further analyze the dependency between the spectra. The above process can be expressed as: W ( i ) represents the one-dimensional spectral feature representation obtained after the fully connected layer, f FC (·) represents the fully connected layer, and AdaPooling(·) represents the adaptive average pooling layer. First, W (i) Normalize, and obtain the scaling and translation coefficients after a learnable affine transformation, and then use the obtained coefficients to Perform channel-by-channel recalibration, which is defined as: ys 1 ,i,yb 1 ,i=faffine(Norm(W ( i ) )) where y s1,i and b1,i denote the learnable scaling and translation coefficients, respectively. represents the output of the AdaIN layer, f affine (·) represents a learnable affine transformation, Norm(·) represents normalization, and Respectively The mean and variance of .

7. The multispectral and hyperspectral image registration and fusion joint optimization method based on multi-scale spatial-spectral information interaction according to claim 6, It is characterized in that In the spatial information fusion branch, The frequency block and the spatial block are input to extract the global and local information of the features respectively, and then integrated into the spectral correction branch, and the information is integrated through the subsequent convolution and activation function layers. The process is defined as: Among them, f spa (·) represents the spatial information fusion branch, f conv,3 (·) denotes the convolutional layer and the activation function layer; specifically, the spatial block is composed of cascaded residual blocks, and the frequency block is composed of Based on Fourier transform, it is decomposed into phase A and amplitude P components, and then two convolutional layers are used to restore the two components respectively. and Finally, the inverse Fourier transform is used to output the global information.

8. The multispectral and hyperspectral image registration and fusion joint optimization method based on multi-scale spatial-spectral information interaction according to claim 7, It is characterized in that The step S106 is specifically as follows: The decoder is similar to the feature encoder and is also built based on a convolutional neural network. The decoder is used to restore the output features of the n-th level registration-fusion joint optimization component to the original space. The formula is: where f rec (·) represents the decoder for image reconstruction, represents the output features of the n-th level registration-fusion optimization component, Represents the reconstructed high-resolution hyperspectral image.

9. The multispectral and hyperspectral image registration and fusion joint optimization method based on multi-scale spatial-spectral information interaction according to claim 8, It is characterized in that The step S107 is specifically as follows: The unregistered multispectral image M f and the hyperspectral image H m The dataset is loaded into the network model to be trained, and the predicted high-resolution hyperspectral image is output. The loss is calculated using the L1 reconstruction loss function, expressed as: where f Net (·) represents the constructed network model, represents the reconstructed nth high-resolution hyperspectral image, Represents the nth reference image, N represents the number of images in a training batch, Θ ​​represents the parameter set of the model. During the training process, the network parameters are updated based on the Adam optimizer until the network converges. After the training is completed, the optimal network parameters are saved.

Citation Information

Patent Citations

  • Global interaction hyperspectral multispectral cross-modal fusion method with spectral fidelity

    CN117911830A

Cited By

  • Deep mine intelligent geological sensing method and system based on image spectrum technology

    CN119206512A

  • Characteristic pattern generation and collaborative optimization method for near infrared spectrum modeling

    CN119415934A

  • Space-frequency double-domain implicit guide sampling remote sensing image fusion method based on KAN

    CN119515706A

  • Network model training method for hyperspectral image enhancement

    CN119809938A

  • A method for training network models for hyperspectral image enhancement

    CN119809938B