A high-multispectral image fusion method based on deep kalman filter

Abstract

The application belongs to the technical field of image processing, and relates to a hyperspectral image fusion method based on deep Kalman filtering, which comprises the following steps: constructing a training sample, creating a hyperspectral image fusion network based on deep Kalman filtering, training the network, using the trained network model, taking the preprocessed hyperspectral image and multispectral image as joint input, and generating a fusion result image with high spatial resolution and high spectral information through the nonlinear mapping relationship of the network. The network of the application is composed of an encoder module, an implicit feature fusion module, an optical flow estimation module, a Kalman gain estimation module and a Kalman recursive fusion module. The application explicitly models the spectral dimension dynamic evolution relationship between bands through Kalman filtering, thereby improving the expression consistency of the fusion image in the spectral dimension. The method can effectively enhance the spatial detail reconstruction capability, and significantly reduce the model parameter size and redundant calculation cost while ensuring the accuracy of the space-spectrum modeling.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a hyperspectral image fusion method based on deep Kalman filtering. Background Technology

[0002] Hyperspectral remote sensing (HSI) images, with their rich spectral information, have significant advantages in fine-grained classification of ground features. However, due to their generally low spatial resolution, although resolution can be improved by enhancing imaging detector performance or extending focal length, these solutions are difficult to implement in the short term due to limitations in chip manufacturing processes and the size and weight of optical systems. Multispectral (MSI) images, while having limited spectral resolution, possess higher spatial resolution, enabling them to more effectively characterize the spatial structure and morphological features of ground features. Therefore, the spatial information of multispectral images can be used as a supplement to enhance the spatial details of hyperspectral images. Hyperspectral and multispectral remote sensing image fusion (HMIF) can improve the spatial resolution of hyperspectral images at the data level without changing sensor hardware. The fusion result retains both the spectral discrimination capability of hyperspectral images and the spatial resolution capability of multispectral images.

[0003] In existing technologies, HMIF (High-Intensity Mitigation) methods are divided into two categories: model-based methods and deep learning methods. Model-based methods, including panchromatic sharpening, matrix and tensor decomposition, achieve fusion based on predefined knowledge or specific rules. However, these predefined priors have limited ability to capture nonlinearities during the fusion process, often resulting in fused image quality that fails to meet practical requirements, especially when faced with significant differences in spatial resolution. Deep learning methods, through end-to-end strategies, can more sensitively perceive nonlinear mappings during the fusion process, typically achieving superior fusion quality compared to traditional methods. However, current fusion methods often focus on network structure innovation, neglecting the dynamic evolution characteristics of the spectral dimension. In fact, strong correlations exist between hyperspectral bands that continuously change with wavelength; without physical modeling, this limits the fusion effect. Furthermore, existing methods mostly perform feature processing in the discrete domain, which is inconsistent with the inherently continuous representation of the real world, and their perception of fine spatial granularity needs improvement. Finally, while current algorithms have achieved fusion accuracy sufficient for practical applications, their high computational cost makes them unsuitable for deployment on edge devices.

[0004] Kalman filtering (KML) is an efficient recursive state estimation method widely used for optimal state inference in dynamic systems. Through a prediction-update mechanism, it iteratively estimates the current system state using the system state transition model and observational information under the Gaussian noise assumption. It offers advantages such as not requiring complete historical data and low computational cost. If each band of the HSI is considered as a dimension of the system state, the HMIF problem can be formalized as a state estimation task based on a spectral sequence. By recursively modeling the states between bands, KML naturally utilizes the inherent correlation of spectral variations with wavelength, thereby improving the representational power and fusion accuracy of spectral features. Furthermore, implicit neural representation (INR) constructs a mapping function from continuous coordinate space to signal values, enabling discrete image signals to be modeled in the continuous domain, thus finely depicting the spatial structure and texture details of the image. Introducing INR into the fusion framework helps enhance the spatial resolution and continuity of the reconstructed image, achieving a more refined fusion effect.

[0005] Based on this, in order to address the problems of insufficient utilization of spectral correlation and high computational complexity in the HMIF process, a novel fusion framework—Deep Kalman Filter Fusion Network (DKFNet)—is proposed to effectively solve the above problems. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of the prior art by providing a hyperspectral and multispectral image fusion method based on deep Kalman filtering, so as to solve the problem of insufficient spectral information representation in fused images.

[0007] The objective of this invention is achieved through the following technical solution:

[0008] A hyperspectral image fusion method based on deep Kalman filtering includes the following steps:

[0009] S1. Construct training samples:

[0010] Gaussian blur is applied to the original hyperspectral and multispectral images to simulate the spatial degradation of the imaging system; then, spatial downsampling is performed through bilinear interpolation to generate degraded hyperspectral and multispectral image samples for model training; the original undegraded high-resolution hyperspectral image is used as the fusion target and as the supervision label during the training phase.

[0011] S2. Construct a hyperspectral image fusion network model based on deep Kalman filtering:

[0012] By jointly modeling the optical flow field and latent states, a fusion network model is constructed, consisting of an encoder module, an implicit feature fusion module, an optical flow estimation module, a Kalman gain estimation module, and a Kalman recursive fusion module. The current state is conceptualized as information corresponding to the current band, and optical flow information is used as a guiding condition for spectral correlation. The optical flow is calculated through a convolutional neural network to capture pixel-level motion relationships between adjacent bands. The formula for the fusion network model is as follows:

[0013] (1)

[0014] in, This represents the optical flow information from the previous band to the current band. For the feature fusion operation of the design, For state transition model, For the current input depth control model, This is process noise;

[0015] S3. Conduct network training:

[0016] The Adam optimizer is used to update the model parameters constructed in step S2. The network is gradually optimized through multiple rounds of iteration to obtain a network model with accurate fusion capabilities.

[0017] S4. Using the network model trained in step S3, the preprocessed hyperspectral image and multispectral image are used as joint inputs. Through the nonlinear mapping relationship of the network, a fused image with both high spatial resolution and hyperspectral information is generated.

[0018] Further, step S1 specifically includes the following steps:

[0019] S11. Spatial filtering of hyperspectral and multispectral images is performed using a Gaussian kernel of size 5×5 to simulate the point spread function effect of the sensor and reproduce the imaging degradation process.

[0020] S12. Spatial downsampling is performed on the filtered image using bilinear interpolation to generate degraded hyperspectral image samples and multispectral image samples required for training as input data.

[0021] S13. Unsampling, undegraded, high-resolution raw hyperspectral images are used as reference images and as supervisory labels for network training.

[0022] Further, in step S2, the encoder module is used to perform shallow feature extraction on the hyperspectral and multispectral images and map them to a unified latent representation space; the implicit feature fusion module can achieve cross-modal feature alignment in the continuous spectral domain and enhance the spatial-spectral interaction between bands; the optical flow estimation module can explicitly estimate the optical flow field between adjacent bands of the hyperspectral image to capture subtle trends in spectral variation; the Kalman gain estimation module models the state covariance through a multilayer perceptron and generates dynamic spectral fusion weights by a lightweight convolution module; the Kalman recursive fusion module completes state prediction and update for each band based on the Kalman filtering strategy, gradually generating a fused image with high spatial-spectral fidelity.

[0023] Furthermore, in step S2, the encoder module employs two parallel shallow convolutional structures to extract and encode features from the hyperspectral image and the multispectral image respectively, and maps them to a unified latent feature space. The formula corresponding to the encoder module is expressed as follows:

[0024] (2)

[0025] (3)

[0026] in, and These represent the features of the hyperspectral and multispectral images of the latent space, respectively. For hyperspectral images, For multispectral images, For convolution operations, both of the above convolutions are... The kernel dimension is set to .

[0027] Furthermore, in step S2, the implicit feature fusion module uses the low-resolution features of the hyperspectral image extracted by the encoder. High-resolution features of multispectral images and two-dimensional normalized coordinate field Using coordinate-aware implicit neural networks as input, feature prediction and fusion are achieved for high-resolution features of multispectral images. It is used as a guiding feature to introduce high-resolution structural information.

[0028] Furthermore, the implicit feature fusion module first constructs a continuous two-dimensional coordinate grid in the high-resolution space, and then performs interpolation sampling on the hyperspectral image feature map in four directions: upper left, upper right, lower left, and lower right, to obtain the neighborhood response of each query location; simultaneously, it calculates the relative coordinate offset between each location and its sampling center. This is used to enhance the model's ability to perceive spatial structure.

[0029] Furthermore, during the fusion stage, local sampling features of the hyperspectral image are... Multispectral image guided features and relative coordinate encoding The three information sources are concatenated into a fused query vector, which is then input into a multilayer perceptron for nonlinear mapping to generate fused features.

[0030] (4)

[0031] in, For the first The fused input vector for each query location; This represents the spatial offset of the location relative to the center of its sampling neighborhood; the module is denoted from four different directions. The above query process is executed separately to generate four sets of candidate fusion results. Simultaneously, the fusion weight score for each direction is estimated. And generate fusion weights using softmax. The final fusion output is as follows:

[0032] (5)

[0033] (6)

[0034] For the first The position in the Fusion output in each direction, The fusion direction weights are derived from the softmax function.

[0035] Further, in step S2, the optical flow estimation module takes the hyperspectral image as input, first upsamples the hyperspectral image using bilinear methods, and then uses two lightweight convolutional layers to dynamically model it across wavebands, extracting implicit optical flow information between continuous wavebands. Its mathematical expression is as follows:

[0036] (7)

[0037] in, This is a bilinear upsampling operation. The output optical flow tensor provides the interspectral motion information required for state prediction in the subsequent Kalman recursion module.

[0038] Further, in step S2, the Kalman gain estimation module first concatenates the hyperspectral image features extracted by the encoder with the multispectral image features along the channel dimension to construct a cross-modal joint feature representation; then, the fused features are input into a multilayer perceptron to model the information uncertainty between different modalities; this uncertainty representation reflects the confidence level of the current band in the fusion process and provides a basis for judgment for gain estimation; next, the module uses two cascaded convolutional layers to further extract and compress the uncertainty features, and finally outputs the Kalman gain tensor, expressed by the following formula:

[0039] (8)

[0040] (9)

[0041] in, For uncertainty, To estimate the Kalman gain, the output structure ensures that each interspectral state recursion step corresponds to an independent fusion weight, thereby enabling dynamic control at the band level.

[0042] Furthermore, in step S2, the input to the Kalman recursive fusion module is... , and implicit fusion features Composition, in which, Used to model the state transition trends between adjacent bands in the spectrum. This is used to adjust the fusion ratio of observed and predicted values ​​in state updates; it is an implicit fusion feature. This represents the observation state of each band predicted by the implicit neural network. The recursive process uses the fused features of the first band as the initial anchor state. And it performs two phases step by step on the spectral dimension: state prediction and state update.

[0043] State prediction phase: in the first stage Band, based on optical flow characteristics of the previous state Perform translation prediction to obtain the current prior state estimate. ,Right now:

[0044] (10)

[0045] in, Represents a standard spatial coordinate grid. Bilinear interpolation was achieved on continuous coordinates after offset;

[0046] State update phase: based on the Kalman gain of the current band. , integrate prior predicted states With observation status The updated state estimate is obtained. :

[0047] (11)

[0048] in, The fusion weights, obtained through learning, determine the degree of confidence in the current state regarding observations and predictions. The recursive process continues until all bands are fused. Finally, the complete fused image is output.

[0049] Compared with the prior art, the beneficial effects of the present invention are:

[0050] 1. This invention introduces a recursive modeling mechanism into the HMIF task for the first time, and improves the consistency of the fused image in the spectral dimension by explicitly modeling the dynamic evolution relationship between bands through Kalman filtering;

[0051] 2. DKFNet, which incorporates implicit neural representations, can achieve high-precision alignment and fusion of multimodal features in a continuous spatial domain, effectively enhancing the ability to reconstruct spatial details;

[0052] 3. DKFNet introduces a weight sharing mechanism in the band-by-band recursion process, which significantly reduces the model parameter size and redundant computational overhead while ensuring the accuracy of spatial spectrum modeling.

[0053] 4. In this invention, the encoder in the network structure performs shallow feature extraction on HSI and MSI and maps them to a unified latent representation space; the IFF module achieves cross-modal feature alignment in the continuous spectral domain, enhancing the spatial-spectral interaction between bands; the OFE module explicitly estimates the optical flow field between adjacent bands of the hyperspectral image to capture subtle trends in spectral variation; the KGE module models the state covariance through a multilayer perceptron and generates dynamic spectral fusion weights by a lightweight convolution module; finally, the KRF module completes state prediction and update for each band based on the Kalman filtering strategy, gradually generating a fused image with high spatial-spectral fidelity. Attached Figure Description

[0054] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 A flowchart illustrating the steps of a hyperspectral image fusion method based on deep Kalman filtering provided by this invention;

[0056] Figure 2 A network structure diagram of a hyperspectral image fusion method based on deep Kalman filtering provided by this invention;

[0057] Figure 3 This is a comparison chart of the fusion results of different HMIF algorithms in this invention. Detailed Implementation

[0058] The present invention will be further described below with reference to embodiments:

[0059] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0060] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0061] This invention proposes a novel fusion framework—Deep Kalman Filter Fusion Network (DKFNet)—which mainly consists of an encoder module, an implicit feature fusion module, an optical flow estimation module, a Kalman gain estimation module, and a Kalman recursive fusion module. These modules work together to efficiently model the cross-modal relationship between HSI and MSI and the dynamic evolution process between bands. First, the encoder module extracts shallow features from HSI and MSI respectively and maps them to a unified latent feature space, providing a consistent feature foundation for subsequent fusion. Next, the implicit feature fusion module introduces INR and, through a continuous coordinate query mechanism, collaboratively models and finely fuses the features of HSI and MSI in a high-dimensional latent space, thereby generating band latent states with continuity-spectral consistency. The optical flow estimation module, based on the differences between HSI band sequences, learns the temporal variation patterns of their spectral dimensions, thus modeling the dynamic correlation between bands. To achieve dynamic modeling and adaptive filtering, the Kalman gain estimation module learns the optimal weight coefficients, i.e., the Kalman gain, during the inter-spectral state update process based on the fused latent features, providing a global control mechanism for spectral recursive fusion. Building upon this, the Kalman recursive fusion module uses the latent state of the first fused band as the initial anchor point, combining the inter-spectral predicted state from optical flow estimation, the observed state of the current band, and the estimated Kalman gain to perform state prediction and correction band by band, achieving recursive modeling of inter-spectral dynamic information and joint fusion of multi-source data. Finally, a fused image with high spectral accuracy and spatial consistency is obtained through a recursive approach.

[0062] like Figure 1 As shown, the hyperspectral image fusion method based on deep Kalman filtering of the present invention includes the following steps:

[0063] S1. Construct training samples:

[0064] Since ideal fusion reference images are lacking in real-world scenarios, this invention follows the Wald protocol principle in designing the training sample construction process. Specifically, firstly, Gaussian blur is applied to the original hyperspectral image (HSI) to simulate the spatial degradation of the imaging system; then, spatial downsampling is achieved through bilinear interpolation to generate degraded hyperspectral image samples for model training. Simultaneously, the same processing steps are performed on the multispectral image (MSI) to obtain the corresponding training input samples. The original, undegraded high-resolution HSI image serves as the fusion target and is used as a supervision label during the training phase.

[0065] To meet the training sample construction requirements of the Wald protocol, the original HSI and MSI images need to be spatially downsampled to generate low-spatial-resolution HSI and high-spatial-resolution MSI images for training. Specifically, firstly, a 5×5 Gaussian kernel is used to spatially filter the HSI and MSI images to simulate the point spread function effect of the sensor, thereby more realistically reproducing the imaging degradation process. Subsequently, a spatial downsampling operation is performed on the filtered image using bilinear interpolation to generate the input data required for training. The unsampled original HSI image is used as a reference image, serving as the supervision label for network training.

[0066] S2. Construct a hyperspectral image fusion network model based on deep Kalman filtering:

[0067] This fusion network model mainly consists of five functional sub-modules: encoder module, implicit feature fusion (IFF) module, optical flow estimation (OFE) module, Kalman gain estimation (KGE) module, and Kalman recursive fusion (KRF) module. These modules work together to efficiently model the cross-modal relationship between HSI and MSI and the dynamic evolution process between wavebands.

[0068] The encoder module is used to extract shallow features from HSI and MSI and map them to a unified latent representation space. The IFF module can achieve cross-modal feature alignment in the continuous spectral domain and enhance the spatial-spectral interaction between bands. The OFE module can explicitly estimate the optical flow field between adjacent bands of the hyperspectral image to capture subtle trends in spectral variation. The KGE module models the state covariance through a multilayer perceptron and generates dynamic spectral fusion weights by a lightweight convolution module. The KRF module completes state prediction and update for each band based on a Kalman filtering strategy, gradually generating a fused image with high spatial-spectral fidelity.

[0069] Regarding the deep Kalman filter model:

[0070] Traditional Kalman filtering, as a classic state estimation method, is theoretically rooted in linear algebra and hidden Markov models. In this model framework, the current state of the system is determined by the previous state and the current input, and its state equation is:

[0071]

[0072] in, This is the current state. As a key state transition model This is the current input in the deep Kalman filter network. It can be parameterized using deep neural networks and transformed into a powerful nonlinear state transition function. This is shaped by deep networks. It can accurately capture the complex dynamic characteristics of spectral data as it evolves across different spectral bands, and compared to traditional linear models, it expands the ability to describe nonlinear changes. As the current input, the deep control model can efficiently map external input information to the state space, and more accurately reflect the impact of the input on the system state. It is process noise.

[0073] This invention innovatively extends the traditional Kalman filter model, introducing it into the fusion domain. It conceptualizes the current state in the traditional model as information corresponding to the current spectral band. Simultaneously, it incorporates optical flow, a crucial motion information, as a guiding condition for spectral correlation. Optical flow is calculated using a convolutional neural network, accurately capturing pixel-level motion relationships between adjacent spectral bands. Based on this, the fusion model can be redefined as:

[0074] (1)

[0075] in, This represents the optical flow information from the previous band to the current band. This involves feature fusion operations for the design. Potential correlations between spectra are uncovered through joint modeling of the optical flow field and hidden states.

[0076] In this invention, the encoder module employs two parallel shallow convolutional structures to extract and encode features from HSI and MSI respectively, mapping them to a unified latent feature space. Unlike the deep encoders (such as EDSR) commonly used in previous multimodal fusion methods, this invention simplifies the encoder structure, aiming to significantly reduce model complexity and training overhead while maintaining intermodal expressiveness. This design strategy is based on two main considerations: firstly, HSI and MSI have inherent differences in the number of bands, resolution, and statistical distribution, making independent branches necessary for modal feature extraction; secondly, the HMIF task focuses more on the modeling ability of subsequent modules for interspectral dynamics and modal complementarity, rather than introducing a large amount of redundant computation in the encoding stage. Therefore, using a lightweight convolutional encoder not only maintains the modal independence of HSI and MSI but also provides sufficient and efficient initial feature representations for subsequent implicit fusion and Kalman recursion modules. The corresponding formula for the encoder module is as follows:

[0077] (2)

[0078] (3)

[0079] in, and Let HSI and MSI represent the characteristics of the latent space, respectively. For HSI, For MSI, For convolution operations, in this invention, both of the above convolutions are... The kernel dimension is set to .

[0080] The implicit feature fusion module aims to achieve cross-modal feature alignment and fine-grained fusion between HSI and MSI in a continuous spatial domain. This module utilizes the low-resolution HSI features extracted by the encoder. MSI high-resolution features and two-dimensional normalized coordinate field Using coordinate-aware implicit neural networks as input, feature prediction and fusion are achieved. Specifically, the module first constructs a continuous two-dimensional coordinate grid in high-resolution space, and then uses the `grid_sample` operation to interpolate and sample the HSI feature map in four directions (top left, top right, bottom left, and bottom right) to obtain the neighborhood response for each query location. Simultaneously, the relative coordinate offset between each location and its sampling center is calculated. This is used to enhance the model's ability to perceive spatial structure. It is worth noting that MSI high-resolution features... These features are used as guiding features to introduce high-resolution structural information to compensate for the shortcomings of HSI features in detailed modeling. During the fusion stage, the three information sources—HSI local sampling features—are integrated. MSI guiding features and relative coordinate encoding The vectors are concatenated into a fused query vector and then fed into a multilayer perceptron (MLP) for nonlinear mapping to generate fused features.

[0081] (4)

[0082] in, For the first The fused input vector for each query location; This represents the spatial offset of the location relative to the center of its sampling neighborhood. To improve the robustness and accuracy of the fusion results, the module operates from four different directions (denoted as...). The above query process is executed separately to generate four sets of candidate fusion results. Simultaneously, the fusion weight score is estimated for each direction. And generate fusion weights using softmax. The final fusion output is as follows:

[0083] (5)

[0084] (6)

[0085] For the first The position in the Fusion output in each direction, The fusion direction weights are derived from the softmax function. This module, based on the INR continuous coordinate query mechanism, alleviates the limitation of fixed grid sampling in traditional convolution methods, enabling the generation of high-quality fused features at arbitrary spatial locations. Furthermore, the introduction of MSI to compensate for missing structural information and the further improvement of prediction stability and accuracy through multi-directional weighted fusion are key components in this invention for achieving cross-modal feature alignment and interspectral continuity modeling.

[0086] The optical flow estimation module aims to uncover the dynamic evolution of HSI across spectral bands. Since HSI exhibits high continuity and sequence across different bands, its spectral variations can be analogized to the motion characteristics between time-series frames. Based on this, this module uses HSI as input. First, it upsamples the HSI using bilinear methods. Then, it uses two lightweight convolutional layers to dynamically model the HSI across spectral bands, extracting the implicit "optical flow" information between continuous bands. The mathematical expression is as follows:

[0087] (7)

[0088] in, This is a bilinear upsampling operation. The output optical flow tensor provides the inter-spectral motion information required for state prediction in the subsequent Kalman recursion module. Compared to traditional explicit optical flow estimation methods, this module directly models the inter-spectral displacement trend based solely on the internal structure of the spectral domain, avoiding additional inter-frame supervision information dependencies and exhibiting strong generalization and physical consistency.

[0089] The Kalman Gain Estimation Module aims to provide an adaptive fusion weight adjustment mechanism for the inter-spectral state recursion process. This module predicts the Kalman gain for each band using a learning-based method to estimate the optimal fusion weight between the current observation and the predicted state. Specifically, the module first concatenates the HSI and MSI features extracted by the encoder along the channel dimension to construct a cross-modal joint feature representation. Subsequently, the fused features are input into a multilayer perceptron (MLP) to model the uncertainty between different modalities. This uncertainty reflects the confidence level of the current band in the fusion process and provides a criterion for gain estimation. Next, the module uses two cascaded convolutional layers to further extract and compress the uncertainty features, finally outputting the Kalman gain tensor. The formula is as follows:

[0090] (8)

[0091] (9)

[0092] in, For uncertainty, To estimate the Kalman gain, the output structure ensures that each interspectral state recursion step corresponds to an independent fusion weight, thereby enabling dynamic control at the band level.

[0093] The Kalman Recursive Fusion Module is the core component of this method for realizing spectral dynamic modeling. It aims to simulate the evolution process between HSI bands and generate the final fused image band by band. The design idea of ​​this module comes from the state estimation mechanism of Kalman filtering and combines optical flow sensing and implicit fusion results to construct a structured and interpretable spectral recursive framework. The input of this module consists of three parts: (1) Used to model the state transition trend between adjacent bands in the spectrum, (2) (3) Implicit fusion features , representing the observation state of each band predicted by the implicit neural network. The recursive process uses the fused features of the first band as the initial anchor state. And it will be implemented in two phases across the spectral dimensions:

[0094] State prediction phase: in the first stage Band, based on optical flow characteristics of the previous state Perform translation prediction to obtain the current prior state estimate. ,Right now

[0095] (10)

[0096] in, Represents a standard spatial coordinate grid. This operation achieves bilinear interpolation on continuous coordinates after the offset. It effectively models inter-spectral state changes as positional offsets in continuous space, thus providing a structurally sound state prior for spectral dimension recursion.

[0097] State update phase: based on the Kalman gain of the current band. , integrate prior predicted states With observation status The updated state estimate is obtained. :

[0098] (11)

[0099] in, The fusion weights, obtained through learning, determine the degree of confidence in the current state regarding observations and predictions. The recursive process continues until all bands are fused. The module ultimately outputs a complete fused image. Through a progressive propagation and update process between spectral bands, it maintains spectral consistency across bands while comprehensively modeling spatial structure, inter-spectral variations, and modal differences, generating a fused image. Compared to traditional integrated fusion methods, recursive structures have stronger physical interpretability and dynamic modeling capabilities, making them particularly suitable for modeling interspectral correlations in high-dimensional data.

[0100] S3. Conduct network training:

[0101] The Adam optimizer is used to update the model parameters, and the network is gradually optimized through multiple iterations to obtain a model with accurate fusion capabilities.

[0102] S4. Using the trained network model, the preprocessed hyperspectral image and multispectral image are used as joint inputs. Through the nonlinear mapping relationship of the network, a fused image with both high spatial resolution and hyperspectral information is generated.

[0103] Example 1: A hyperspectral / multispectral image fusion method based on deep Kalman filtering, comprising the following steps:

[0104] S1. Construct training samples:

[0105] Since ideal fusion reference images are lacking in real-world scenarios, this invention follows the Wald protocol principle in designing the training sample construction process. Specifically, Gaussian blur is first applied to the original hyperspectral image (HSI) to simulate spatial degradation of the imaging system. Then, spatial downsampling is achieved through bilinear interpolation to generate a degraded version of the hyperspectral image for model training. Simultaneously, the same processing steps are performed on the multispectral image (MSI) to obtain the corresponding training input. The original, undegraded high-resolution HSI image serves as the fusion target and is used as a supervision label during the training phase.

[0106] S2. A hyperspectral and multispectral image fusion network based on deep Kalman filtering is created. This network mainly consists of five functional sub-modules: an encoder module, an implicit feature fusion module, an optical flow estimation module, a Kalman gain estimation module, and a Kalman recursive fusion module. These modules work together to efficiently model the cross-modal relationship between HSI and MSI and the dynamic evolution process between bands.

[0107] S3. Conduct network training:

[0108] The Adam optimizer is used to update the model parameters, and the network is gradually optimized through multiple iterations to obtain a model with accurate fusion capabilities.

[0109] S4. Using the trained network model, the preprocessed hyperspectral image and multispectral image are used as joint inputs. Through the nonlinear mapping relationship of the network, a fused image with both high spatial resolution and hyperspectral information is generated.

[0110] To demonstrate the effectiveness of this invention, fusion experiments were conducted using images captured by the ROSIS sensor with spatial resolution differences of 4x, 8x, and 16x, and compared with current novel fusion methods. The results are presented in... Figure 3 . Figure 3The first row represents the fused image, the second row represents the SAM error heatmap, the third row represents the MRAE error heatmap, and the fourth row represents the difference map. The SAM error heatmap reflects the spectral fidelity of the fused image, the MRAE error heatmap reflects the global consistency of the fused image, and the difference map measures the residuals in individual bands. Darker colors indicate better fusion results. It is clear that the present invention achieves the best results. Quantitative evaluations of its fusion at 4x, 8x, and 16x resolution are shown in Table 1. It should be noted that smaller RMSE, ERGAS, and SAM values ​​indicate better fusion quality, meaning it is closer to the true state; higher PSNR values ​​indicate better fusion performance. It can be seen that the present invention outperforms the compared methods at 4x, 8x, and 16x spatial resolution, indicating that the present invention can effectively preserve the spatial-spectral characteristics of the fused image. Furthermore, a complexity comparison analysis was conducted with current deep learning algorithms, as shown in Table 2. Here, Model Size represents the model's memory, Flops represents the number of floating-point operations, and Param represents the number of learned parameters. It can be seen that the present invention performs optimally in terms of both memory usage and the number of parameters, further demonstrating the superiority and lightweight nature of the present invention.

[0111] Table 1. Quantitative evaluation of the fusion experiment between the present invention and the comparison algorithm.

[0112]

[0113] Table 2. Comparison of Complexity

[0114]

[0115] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A hyperspectral image fusion method based on deep Kalman filtering, characterized in that, Includes the following steps: S1. Construct training samples: Gaussian blur is applied to the original hyperspectral and multispectral images respectively to simulate the spatial degradation of the imaging system; then spatial downsampling is achieved through bilinear interpolation to generate degraded hyperspectral and multispectral image samples for model training. The original, undegraded, high-resolution hyperspectral image is used as the fusion target and as a supervisory label during the training phase. S2. Construct a hyperspectral image fusion network model based on deep Kalman filtering: By jointly modeling the optical flow field and latent states, a fusion network model is constructed, consisting of an encoder module, an implicit feature fusion module, an optical flow estimation module, a Kalman gain estimation module, and a Kalman recursive fusion module. The current state is conceptualized as information corresponding to the current band, and optical flow information is used as a guiding condition for spectral correlation. The optical flow is calculated through a convolutional neural network to capture pixel-level motion relationships between adjacent bands. The formula for the fusion network model is as follows: ， in, This represents the optical flow information from the previous band to the current band. For the feature fusion operation of the design, For state transition model, For the current input depth control model, This is process noise; S3. Conduct network training: The Adam optimizer is used to update the model parameters constructed in step S2. The network is gradually optimized through multiple rounds of iteration to obtain a network model with accurate fusion capabilities. S4. Using the network model trained in step S3, the preprocessed hyperspectral image and multispectral image are used as joint inputs. Through the nonlinear mapping relationship of the network, a fused image with both high spatial resolution and hyperspectral information is generated.

2. The hyperspectral image fusion method based on deep Kalman filtering according to claim 1, characterized in that, Step S1 specifically includes the following steps: S11. Spatial filtering of hyperspectral and multispectral images is performed using a Gaussian kernel of size 5×5 to simulate the point spread function effect of the sensor and reproduce the imaging degradation process. S12. Spatial downsampling is performed on the filtered image using bilinear interpolation to generate degraded hyperspectral image samples and multispectral image samples required for training as input data. S13. Unsampling, undegraded, high-resolution raw hyperspectral images are used as reference images and as supervisory labels for network training.

3. The hyperspectral image fusion method based on deep Kalman filtering according to claim 1, characterized in that: In step S2, the encoder module is used to perform shallow feature extraction on hyperspectral and multispectral images and map them to a unified latent representation space; the implicit feature fusion module can achieve cross-modal feature alignment in the continuous spectral domain and enhance the spatial-spectral interaction between bands. The optical flow estimation module can explicitly estimate the optical flow field between adjacent bands of a hyperspectral image to capture subtle interspectral variations. The Kalman gain estimation module models the state covariance using a multilayer perceptron and generates dynamic spectral fusion weights using a lightweight convolution module. The Kalman recursive fusion module completes state prediction and updates for each band based on a Kalman filtering strategy, gradually generating a fused image with high spatial-spectral fidelity.

4. The hyperspectral image fusion method based on deep Kalman filtering according to claim 1, characterized in that: In step S2, the encoder module employs two parallel shallow convolutional structures to extract and encode features from the hyperspectral and multispectral images, respectively, and maps them to a unified latent feature space. The formula corresponding to the encoder module is as follows: ， in, and These represent the features of the hyperspectral and multispectral images of the latent space, respectively. For hyperspectral images, For multispectral images, For convolution operations, both of the above convolutions are... The kernel dimension is set to .

5. The hyperspectral image fusion method based on deep Kalman filtering according to claim 1, characterized in that: In step S2, the implicit feature fusion module uses the low-resolution features of the hyperspectral image extracted by the encoder. High-resolution features of multispectral images and two-dimensional normalized coordinate field Using coordinate-aware implicit neural networks as input, feature prediction and fusion are achieved for high-resolution features of multispectral images. It is used as a guiding feature to introduce high-resolution structural information.

6. The hyperspectral image fusion method based on deep Kalman filtering according to claim 5, characterized in that: The implicit feature fusion module first constructs a continuous two-dimensional coordinate grid in high-resolution space, and then performs interpolation sampling on the hyperspectral image feature map in four directions: upper left, upper right, lower left, and lower right, to obtain the neighborhood response of each query location; simultaneously, it calculates the relative coordinate offset between each location and its sampling center. This is used to enhance the model's ability to perceive spatial structure.

7. The hyperspectral image fusion method based on deep Kalman filtering according to claim 6, characterized in that: During the fusion stage, local sampling features of the hyperspectral image are used. Multispectral image-guided features and relative coordinate encoding The three information sources are concatenated into a fused query vector, which is then input into a multilayer perceptron for nonlinear mapping to generate fused features. ， in, For the first The fused input vector for each query location; This represents the spatial offset of the location relative to the center of its sampling neighborhood; the module is denoted from four different directions. The above query process is executed separately to generate four sets of candidate fusion results. Simultaneously, the fusion weight score for each direction is estimated. And generate fusion weights using softmax. The final fusion output is as follows: ， For the first The position in the Fusion output in all directions, The fusion direction weights are derived from the softmax function.

8. The hyperspectral image fusion method based on deep Kalman filtering according to claim 1, characterized in that: In step S2, the optical flow estimation module takes the hyperspectral image as input, first upsamples the hyperspectral image using bilinear methods, and then uses two lightweight convolutional layers to dynamically model the image across wavelength bands, extracting implicit optical flow information between continuous wavelength bands. Its mathematical expression is as follows: ， in, This is an upsampling operation based on bilinearity. The output optical flow tensor provides the interspectral motion information required for state prediction in the subsequent Kalman recursion module.

9. The hyperspectral image fusion method based on deep Kalman filtering according to claim 1, characterized in that: In step S2, the Kalman gain estimation module first concatenates the hyperspectral image features extracted by the encoder with the multispectral image features along the channel dimension to construct a cross-modal joint feature representation. Then, the fused features are input into a multilayer perceptron to model the information uncertainty between different modalities. This uncertainty representation reflects the confidence level of the current band during the fusion process and provides a basis for gain estimation. Next, the module uses two cascaded convolutional layers to further extract and compress the uncertainty features. Finally, it outputs the Kalman gain tensor, expressed by the following formula: ， in, For uncertainty, To estimate the Kalman gain, the output structure ensures that each interspectral state recursion step corresponds to an independent fusion weight, thereby enabling dynamic control at the band level.

10. The hyperspectral image fusion method based on deep Kalman filtering according to claim 1, characterized in that: In step S2, the input of the Kalman recursive fusion module is... , and implicit fusion features Composition, in which, Used to model the state transition trends between adjacent bands in the spectrum. This is used to adjust the fusion ratio of observed and predicted values ​​in state updates; it is an implicit fusion feature. This represents the observation state of each band predicted by the implicit neural network. The recursive process uses the fused features of the first band as the initial anchor state. And it performs two phases step by step on the spectral dimension: state prediction and state update. State prediction phase: in the first stage Band, through optical flow characteristics of the previous state Perform translation prediction to obtain the current prior state estimate. ,Right now: ， in, Represents a standard spatial coordinate grid. Bilinear interpolation was achieved on continuous coordinates after offset; State update phase: based on the Kalman gain of the current band. , integrate prior predicted states With observation status The updated state estimate is obtained. : ， in, The fusion weights, obtained through learning, determine the degree of confidence in the current state regarding observations and predictions. The recursive process continues until all bands are fused. Finally, the complete fused image is output.

Images