High-multispectral image fusion method based on deep Kalman filtering

Abstract

The invention belongs to the technical field of image processing, and relates to a deep Kalman filtering-based high-multispectral image fusion method, which comprises the steps of constructing a training sample, creating a deep Kalman filtering-based high-multispectral image fusion network, carrying out network training, and carrying out high-multispectral image fusion by utilizing a trained network model. And taking the preprocessed hyperspectral image and multispectral image as joint input, and generating a fusion result image with both high spatial resolution and hyperspectral information through a nonlinear mapping relation of a network. The network is composed of an encoder module, an implicit feature fusion module, an optical flow estimation module, a Kalman gain estimation module and a Kalman recursion fusion module. According to the method, the spectral dimension dynamic evolution relation between wavebands is explicitly modeled through Kalman filtering, and the expression consistency of the fused image in the spectral dimension is improved; according to the method, the spatial detail reconstruction capability can be effectively enhanced; and the model parameter scale and the redundancy calculation overhead are remarkably reduced while the spatial spectrum modeling precision is ensured.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a hyperspectral-multispectral image fusion method based on deep Kalman filtering. BACKGROUND

[0002] Hyperspectral remote sensing image (HSI) has a significant advantage in fine classification of ground objects due to its rich spectral information. However, due to its generally low spatial resolution, although the resolution can be improved by enhancing the performance of the imaging detector or prolonging the focal length, such solutions are difficult to achieve low-cost applications in the short term due to limitations of chip manufacturing processes and factors such as the volume and weight of the optical system. Although multispectral image (MSI) has limited spectral resolution, it has higher spatial resolution and can more effectively depict the spatial structure and morphological characteristics of ground objects. Therefore, the spatial information of multispectral image can be used as a supplement to achieve spatial detail enhancement of hyperspectral image. Hyperspectral and multispectral remote sensing image fusion (HMIF) does not need to change the sensor hardware, but can improve the spatial resolution of the hyperspectral image at the data level, and the fusion result not only retains the spectral discrimination ability of the hyperspectral image, but also has the spatial resolution ability of the multispectral image.

[0003] In the prior art, HMIF is divided into two categories: model-based methods and deep learning methods. Among them, the model-based method includes panchromatic sharpening, matrix and tensor decomposition, etc., which realizes fusion according to the human-defined knowledge priori or specific rules. However, the human-defined priori has limited ability to capture the nonlinearity in the fusion process, so that the quality of the fused image often cannot meet the actual demand, especially when facing a large difference in spatial resolution. The deep learning method can more sensitively perceive the non-linear mapping in the fusion process through an end-to-end strategy, and the fusion quality is usually better than that of the traditional method. However, current fusion methods often focus on the innovation of network structure, and often ignore the dynamic evolution characteristics of the spectral dimension. In fact, there is a strong correlation between the hyperspectral bands that changes continuously with the wavelength, and if it is not physically modeled, the fusion effect will be limited. In addition, most existing methods process features in discrete domains, which is inconsistent with the continuous representation of the real world, and the perception of spatial fine granularity needs to be improved. Finally, although the current algorithms have reached the requirements of practical applications in terms of fusion accuracy, their computational overhead is high, which makes it difficult to adapt to the deployment needs of edge devices.

[0004] Kalman filter is an efficient recursive state estimation method, which is widely used in optimal state inference of dynamic systems. Through the prediction-update mechanism, it iteratively estimates the current system state under the assumption of Gaussian noise using the system state transition model and observation information, and has the advantages of not needing to save complete historical data and low computational overhead. If each band of HSI is regarded as a dimension of the system state, the HMIF problem can be formalized as a state estimation task based on spectral sequence. By recursively modeling the state between bands, Kalman filter can naturally utilize the inherent correlation of the spectrum with wavelength, thereby improving the representation ability and fusion accuracy of spectral features. In addition, implicit neural representation (INR) enables discrete image signals to be modeled in continuous domain by constructing a mapping function from continuous coordinate space to signal values, thereby finely capturing the spatial structure and texture details of images. The introduction of INR in the fusion framework helps to enhance the spatial resolution and continuity of the reconstructed image, achieving a more delicate fusion effect.

[0005] Based on this, in order to solve the problems of insufficient utilization of spectral correlation and high computational complexity in the HMIF process, a novel fusion framework, Deep Kalman Filter Fusion Network (DKFNet), is developed to effectively solve the above problems. SUMMARY

[0006] The present application aims to solve the problem of insufficient representation of spectral information in the fused image by providing a high-multispectral image fusion method based on deep Kalman filter.

[0007] The purpose of the present application is achieved by the following technical solutions: A high-multispectral image fusion method based on deep Kalman filter, comprising the following steps: S1, constructing training samples: Apply Gaussian blur to the original hyperspectral image and multispectral image respectively to simulate the spatial degradation of the imaging system; then perform spatial down-sampling through bilinear interpolation to generate degraded hyperspectral image samples and multispectral image samples for model training; the original high-resolution hyperspectral image without degradation is used as the fusion target and as the supervision label in the training stage; S2, constructing a high-multispectral image fusion network model based on deep Kalman filter: By jointly modeling the optical flow field and the hidden state, a fusion network model composed of an encoder module, an implicit feature fusion module, an optical flow estimation module, a Kalman gain estimation module, and a Kalman recursion fusion module is constructed. The current state is conceptualized as information corresponding to the current waveband, and the optical flow information is combined as a spectral correlation guide condition. The optical flow is calculated through a convolutional neural network to capture the pixel-level motion relationship between adjacent wavebands. The fusion network model is as follows: (1) wherein, represents the optical flow information from the previous waveband to the current waveband, is a designed feature fusion operation, is a state transformation model, is a current input depth control model, is process noise; S3, network training: The model parameters constructed in step S2 are updated using the Adam optimizer, and the network is gradually optimized through multiple iterations, thereby obtaining a network model with accurate fusion capability. S4, using the network model trained in step S3, the preprocessed hyperspectral image and multispectral image are used as joint input, and through the nonlinear mapping relationship of the network, a fusion result image with high spatial resolution and high spectral information is generated.

[0008] Further, step S1 specifically includes the following steps: S11, a Gaussian kernel with a size of 5x5 is used to perform spatial filtering on the hyperspectral image and the multispectral image to simulate the point spread function effect of the sensor and reproduce the imaging degradation process; S12, the filtered images are spatially down-sampled by a bilinear interpolation method to generate degraded version hyperspectral image samples and multispectral image samples required for training as input data; S13, the non-degraded high-resolution original hyperspectral image without down-sampling is used as a reference image as a supervision label for network training.

[0009] Further, in step S2, the encoder module is configured to perform shallow feature extraction on the hyperspectral image and the multispectral image and map them to a unified latent representation space; the implicit feature fusion module is capable of achieving cross-modal feature alignment in the continuous spectral domain and enhancing the spectral-spatial interaction between bands; the optical flow estimation module is capable of explicitly estimating the optical flow field between adjacent bands of the hyperspectral image to capture the trend of the slight change between spectra; the Kalman gain estimation module models the state covariance by using a multilayer perceptron and generates spectral-dimension dynamic fusion weights by using a lightweight convolutional module; and the Kalman recursive fusion module completes the state prediction and update of each band based on the Kalman filtering strategy to gradually generate a fused image with high spectral fidelity.

[0010] Further, in step S2, the encoder module adopts two parallel shallow convolutional structures to perform feature extraction and encoding on the hyperspectral image and the multispectral image respectively, and map them to a unified latent feature space. The formula corresponding to the encoder module is as follows: (2) (3) wherein, and represent the hyperspectral image and multispectral image features in the latent space respectively, is the hyperspectral image, is the multispectral image, is a convolution operation, and the two convolutions are both , and the kernel dimension is set to .

[0011] Further, in step S2, the implicit feature fusion module takes the low-resolution hyperspectral image feature extracted by the encoder, the high-resolution multispectral image feature , and the two-dimensional normalized coordinate field as inputs, and realizes feature prediction and fusion by using a coordinate-aware implicit neural network. The high-resolution multispectral image feature is used as a guide feature to introduce high-resolution structural information.

[0012] Further, the implicit feature fusion module first constructs a continuous two-dimensional coordinate grid in the high-resolution space, and then performs interpolation sampling on the hyperspectral image feature map in the left-up, right-up, left-down and right-down directions to obtain the neighborhood response of each query position. Meanwhile, the relative coordinate offset between each position and its sampling center is calculated to enhance the spatial structure perception ability of the model.

[0013] Further, in the fusion stage, the local sampling feature , multispectral image guided features and relative coordinate encoding Three information sources are spliced into a fusion query vector and input into a multi-layer perception for nonlinear mapping to generate a fusion feature: (4) wherein, is the fusion input vector of the i-th query position; represents the spatial offset of the position relative to the center of its sampling neighborhood; the module performs the above query process from four different directions, denoted as , to generate four groups of candidate fusion results ; at the same time, the fusion weight score of each direction is estimated, and the fusion weight is generated through softmax, and the final fusion output is as follows: (5) (6) is the fusion output of the i-th position in the j-th direction, is the fusion direction weight obtained by the softmax function. Further, in step S2, the optical flow estimation module takes the hyperspectral image as input, first upsamples the hyperspectral image using bilinear, and then uses two lightweight convolutional layers to dynamically model the cross-band direction and extract implicit optical flow information between consecutive bands, which is mathematically expressed as:

[0014] (7) wherein, is the bilinear-based upsampling operation, is the output optical flow tensor, which provides the inter-spectral motion information required for state prediction for the subsequent Kalman recursion module. Further, in step S2, the Kalman gain estimation module first splices the hyperspectral image features extracted by the encoder and the multispectral image features in the channel dimension to construct a cross-modal joint feature representation; then the fused features are input into a multi-layer perception for modeling the information uncertainty between different modalities; this uncertainty representation reflects the trust degree of the current band in the fusion process and provides a basis for gain estimation; then, the module uses two convolutional layers in series to further extract and compress the uncertainty features, and finally outputs the Kalman gain tensor, which is mathematically expressed as:

[0015] ​​​ (8) (9) wherein, is the uncertainty, is the estimated Kalman gain, the output structure ensures that each inter-spectral state recursion step corresponds to an independent fusion weight, thereby achieving dynamic regulation at the waveband level.

[0016] Further, in step S2, the input of the Kalman recursion fusion module is composed of , and implicit fusion features , wherein, is used to model the state transition trend between adjacent wavebands in the spectral dimension, is used to adjust the fusion proportion of the observation value and the predicted value in the state update, the implicit fusion feature represents the observed state of each waveband predicted by the implicit neural network, the recursion process takes the fusion feature of the first waveband as the initial anchor state , and performs state prediction and state update in two stages in the spectral dimension: State prediction stage: in the first waveband, the previous state is translated and predicted by the optical flow feature to obtain the current prior state estimate , that is: (10) wherein, represents the standard space coordinate grid, achieves bilinear interpolation on the offset continuous coordinates; State update stage: according to the Kalman gain of the current waveband, the prior predicted state is fused with the observed state to obtain the updated state estimate : (11) wherein, is the fusion weight obtained by learning, which determines the trust degree of the current state to the observation and the prediction, and the recursion process continues until all wavebands are fused , and finally, the complete fused image is output.

[0017] Compared with the prior art, the present application has the following beneficial effects: 1. The present application first introduces a recursive modeling mechanism system into the HMIF task, explicitly models the spectral dimension dynamic evolution relationship between wavebands through Kalman filtering, and improves the expression consistency of the fused image in the spectral dimension; 2、DKFNet fusion implicit neural representation, can realize high-precision alignment and fusion of multi-modal features in continuous spatial domain, effectively enhance the spatial detail reconstruction ability; 3、DKFNet introduces a weight sharing mechanism in the process of recursive wave band by wave band, which significantly reduces the model parameter size and redundant calculation overhead while ensuring the accuracy of spatial-spectral modeling; 4、In the present application, the encoder in the network structure extracts shallow features of HSI and MSI and maps them to a unified latent representation space; the IFF module realizes cross-modal feature alignment in the continuous spectral domain, enhancing the spatial-spectral interaction between wave bands; the OFE module explicitly estimates the optical flow field between adjacent wave bands of hyperspectral images to capture the trend of small spectral changes; the KGE module models the state covariance through a multilayer perceptron and generates spectral dimension dynamic fusion weights through a lightweight convolution module; finally, the KRF module completes the state prediction and update of each wave band based on the Kalman filtering strategy, gradually generating a fused image with high spatial-spectral fidelity. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments, and it should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0019] Figure 1 A step flow chart of a high-multispectral image fusion method based on deep Kalman filtering provided by the present application; Figure 2 A network structure diagram of a high-multispectral image fusion method based on deep Kalman filtering provided by the present application; Figure 3 A fusion result comparison diagram of different HMIF algorithms in the present application. DETAILED DESCRIPTION

[0020] The present application will be further described below in conjunction with the embodiments: The present application will be further described below in conjunction with the embodiments:

[0021] It should be noted that similar reference numerals and letters refer to like items in the accompanying drawings, and once an item is defined in one drawing, it need not be further defined and explained in subsequent drawings. Meanwhile, in the description of the present application, the terms "first", "second", and the like are used only to distinguish descriptions, and cannot be understood as indicating or implying relative importance.

[0022] The present application proposes a new type of fusion framework-Deep Kalman Filter Fusion Network (DKFNet), which mainly consists of an encoder module, an implicit feature fusion module, an optical flow estimation module, a Kalman gain estimation module and a Kalman recursive fusion module. Each module cooperates to efficiently model the cross-modal relationship between HSI and MSI and the dynamic evolution process between bands. First, the encoder module extracts shallow features from HSI and MSI respectively and maps them to a unified latent feature space, providing a consistent feature basis for subsequent fusion. Then, the implicit feature fusion module introduces INR and uses the query mechanism of continuous coordinates to model and finely fuse the features of HSI and MSI in the high-dimensional latent space, thereby generating band latent states with continuous spectral consistency. The optical flow estimation module learns the temporal variation law in the spectral dimension based on the differences between HSI band sequences, thereby modeling the dynamic correlation between bands. In order to realize dynamic modeling and adaptive filtering, the Kalman gain estimation module learns the optimal weight coefficient, i.e. Kalman Gain, in the spectral state update process based on the fused latent features, providing a global regulation mechanism for spectral recursive fusion. On this basis, the Kalman recursive fusion module takes the latent state of the first fused band as the initial anchor point, combines the spectral prediction state estimated by the optical flow estimation, the observation state of the current band, and the estimated Kalman gain, and performs state prediction and correction for each band, realizing recursive modeling of spectral dynamic information and joint fusion of multi-source data. Finally, the fused image with high spectral accuracy and spatial consistency is obtained through recursive method.

[0023] As shown in Figure 1 The hyperspectral-multispectral image fusion method based on deep Kalman filter of the present application includes the following steps: S1, constructing training samples: Due to the lack of ideal fusion reference images in real scenes, the application follows the Wald protocol principle to design the construction process of training samples. Specifically, first, Gaussian blur is applied to the original hyperspectral image (HSI) to simulate the spatial degradation of the imaging system; then, spatial down-sampling is realized by bilinear interpolation to generate degraded hyperspectral image samples for model training. At the same time, the same processing steps are performed on the multispectral image (MSI) to obtain the corresponding training input required samples. The original high-resolution HSI image is used as the fusion target and as the supervision label in the training stage.

[0024] To meet the training sample construction requirements of the Wald protocol, spatial down-sampling processing is required on the original HSI and MSI to generate low spatial resolution HSI and high spatial resolution MSI for training. Specifically, first, a 5x5 Gaussian kernel is used for spatial filtering of HSI and MSI to simulate the point spread function effect of the sensor, thereby more realistically reproducing the imaging degradation process. Subsequently, spatial down-sampling operation is performed on the filtered image by bilinear interpolation method to generate the input data required for training. The original HSI image without down-sampling is used as the reference image as the supervision label for network training.

[0025] S2, construct a hyperspectral-multispectral image fusion network model based on deep Kalman filter: The fusion network model mainly consists of five functional sub-modules: an encoder module, an implicit feature fusion (IFF) module, an optical flow estimation (OFE) module, a Kalman gain estimation (KGE) module, and a Kalman recursive fusion (KRF) module. Each module cooperates to efficiently model the cross-modal relationship between HSI and MSI and the dynamic evolution process between bands.

[0026] The encoder module is used to extract shallow features from HSI and MSI and map them to a unified latent representation space; the IFF module can align cross-modal features in the continuous spectral domain and enhance the spatial-spectral interaction between bands; the OFE module can explicitly estimate the optical flow field between adjacent bands of the hyperspectral image to capture the trend of small spectral changes; the KGE module models the state covariance through a multilayer perceptron and generates spectral dimension dynamic fusion weights through a lightweight convolution module; and the KRF module completes the state prediction and update of each band based on the Kalman filtering strategy to gradually generate a fused image with high spatial-spectral fidelity.

[0027] Regarding the deep Kalman filter model: As a classic state estimation method, traditional Kalman filtering has its theoretical foundation deeply rooted in linear algebra and implicit Markov models. In this model system, the current state of the system is determined by the previous state and the current input, and its state equation is:

[0028] where, is the current state, as the key state transition model, is the current input, in the deep Kalman filter network, can be parameterized by a deep neural network, converting into a powerful nonlinear state transition function. This state transition function shaped by the deep network , can accurately capture the complex dynamic characteristics of spectral data when evolving in the wavelength dimension, and compared with traditional linear models, it expands the description ability of nonlinear changes, as the current input, the deep control model can efficiently map external input information to the state space, and more accurately reflect the influence of input on the system state, is the process noise.

[0029] The present application innovatively expands the traditional Kalman filter model, introduces it into the fusion field, and conceptualizes the current state in the traditional model as information corresponding to the current wavelength. At the same time, combined with the important motion information of optical flow as the spectral correlation guide condition. The optical flow is calculated through a convolutional neural network to accurately capture the pixel-level motion relationship between adjacent wavebands. Based on this, the fusion model can be redefined as: (1) where, indicates the optical flow information from the previous waveband to the current waveband, is the designed feature fusion operation. By jointly modeling the optical flow field and the hidden state, the potential correlation between spectra is excavated.

[0030] In the present application, the encoder module adopts two parallel shallow convolutional structures to extract and encode the features of HSI and MSI respectively, and map them to a unified latent feature space. Unlike the deep encoder (such as EDSR) commonly used in previous multi-modal fusion methods, the present application simplifies the encoder structure, with the purpose of significantly reducing the model complexity and training overhead on the premise of ensuring the expression ability between modalities. This design strategy is mainly based on two considerations: on the one hand, HSI and MSI have natural differences in the number of wavebands, resolution and statistical distribution, so it is necessary to use independent branches to extract modal features; on the other hand, the HMIF task pays more attention to the modeling ability of the subsequent modules for spectral inter-dynamic and modal complementarity, rather than introducing a large amount of redundant calculation in the encoding stage. Therefore, using a lightweight convolutional encoder not only maintains the modal independence of HSI and MSI, but also provides sufficient and efficient initial feature representation for the subsequent implicit fusion and Kalman recursion modules. The formula corresponding to the encoder module is as follows: (2) (3) where, and denote the HSI and MSI features in latent space, respectively, is the HSI, is the MSI, is the convolution operation, in the present application, both of the above convolutions are , and the kernel dimension is set to .

[0031] The implicit feature fusion module (Implicit Feature Fusion) aims to achieve cross-modal feature alignment and fine-grained fusion between HSI and MSI in the continuous spatial domain. This module takes the HSI low-resolution features extracted by the encoder, MSI high-resolution features, and a two-dimensional normalized coordinate field as input, and realizes feature prediction and fusion through a coordinate-aware implicit neural network. Specifically, the module first constructs a continuous two-dimensional coordinate grid in the high-resolution space, and uses the grid_sample operation to interpolate and sample the HSI feature map in four directions (top-left, top-right, bottom-left, bottom-right) to obtain the neighborhood response of each query position. At the same time, the relative coordinate offset between each position and its sampling center is calculated to enhance the model's perception of spatial structure. It is worth noting that the MSI high-resolution features are used as guide features to introduce high-resolution structural information to make up for the lack of HSI features in detail modeling. In the fusion stage, the three information sources, i.e., the HSI local sampling features , the MSI guide features , and the relative coordinate encoding are concatenated into a fusion query vector and input into a multi-layer perceptron (MLP) for nonlinear mapping to generate the fusion features: (4) where, is the fusion input vector of the th query position; denotes the spatial offset of the position relative to the center of its sampling neighborhood. In order to improve the robustness and accuracy of the fusion results, the module performs the above query process from four different directions, denoted as , to generate four groups of candidate fusion results . At the same time, the fusion weight score is estimated for each direction, and the fusion weight The final fusion output is as follows: (5) (6) For the first The position in the Fusion output in all directions, The fusion direction weights are derived from the softmax function. This module, based on the INR continuous coordinate query mechanism, alleviates the limitation of fixed grid sampling in traditional convolution methods, enabling the generation of high-quality fused features at arbitrary spatial locations. Furthermore, the introduction of MSI to compensate for missing structural information and the further improvement of prediction stability and accuracy through multi-directional weighted fusion are key components in this invention for achieving cross-modal feature alignment and interspectral continuity modeling.

[0032] The optical flow estimation module aims to uncover the dynamic evolution of HSI across spectral bands. Since HSI exhibits high continuity and sequence across different bands, its spectral variations can be analogized to the motion characteristics between time-series frames. Based on this, this module uses HSI as input. First, it upsamples the HSI using bilinear methods. Then, it uses two lightweight convolutional layers to dynamically model the HSI across spectral bands, extracting the implicit "optical flow" information between continuous bands. The mathematical expression is as follows: (7) in, This is an upsampling operation based on bilinearity. The output optical flow tensor provides the inter-spectral motion information required for state prediction in the subsequent Kalman recursion module. Compared to traditional explicit optical flow estimation methods, this module directly models the inter-spectral displacement trend based solely on the internal structure of the spectral domain, avoiding additional inter-frame supervision information dependencies and exhibiting strong generalization and physical consistency.

[0033] The Kalman Gain Estimation Module aims to provide an adaptive fusion weight regulation mechanism for the inter-spectral state recursive process. This module predicts the Kalman gain of each band through a learning method to achieve the estimation of the optimal fusion proportion between the current observation and the predicted state. Specifically, the module first concatenates the HSI features and MSI features extracted by the encoder in the channel dimension to construct a cross-modal joint feature representation. Then, the fusion features are input into a multi-layer perception (MLP) to model the information uncertainty between different modalities. This uncertainty representation reflects the trustworthiness of the current band in the fusion process and provides a basis for gain estimation. Next, the module uses two convolutional layers in series to further extract and compress the uncertainty features, and finally outputs the Kalman gain tensor. The formula is as follows: (8) (9) where, is the uncertainty, is the estimated Kalman gain. The output structure ensures that each inter-spectral state recursive step corresponds to an independent fusion weight, thereby achieving dynamic regulation at the band level.

[0034] The Kalman Recursive Fusion Module is the core component of the method for implementing spectral dimension dynamic modeling, aiming to simulate the evolution process between HSI bands and generate the final fusion image band by band. The design idea of this module is derived from the state estimation mechanism of Kalman filtering, and combines optical flow perception and implicit fusion results to build a structured and interpretable spectral dimension recursive framework. The input of this module consists of three parts: (1) used to model the state transition trend between adjacent bands in the spectrum, (2) used to adjust the fusion proportion of the observation and the prediction in the state update; (3) implicit fusion features representing the observation state of each band predicted by the implicit neural network. The recursive process takes the fusion features of the first band as the initial anchor state and performs two stages step by step in the spectral dimension: State prediction stage: at the band, the last state is translated and predicted through the optical flow feature to obtain the current prior state estimate , i.e. (10) where, denotes the standard spatial coordinate grid, Bilinear interpolation on the shifted continuous coordinates is implemented. This operation effectively models the inter-spectral state changes as a position shift in the continuous space, providing a structurally sound state prior for the spectral dimension recursion.

[0035] State update stage: according to the Kalman gain of the current band , the prior predicted state and the observation state are fused to obtain the updated state estimate : (11) where, is the fusion weight obtained by learning, which determines the degree of trust in the current state observation and prediction, and the recursion process continues until all bands are fused , and the complete fused image is finally output. Through the step-by-step propagation and update process between spectral bands, this module realizes the comprehensive modeling of spatial structure, inter-spectral changes and modal differences while maintaining spectral consistency between bands, generating a fused image . Compared with traditional integrated fusion methods, the recursive structure has stronger physical interpretability and dynamic modeling capability, especially suitable for modeling inter-spectral correlations in high-dimensional data.

[0036] S3, network training: The Adam optimizer is used to update the model parameters, and the network is gradually optimized through multiple iterations, so as to obtain a model with accurate fusion capability.

[0037] S4, using the trained network model, the preprocessed hyperspectral image and multispectral image are used as joint input, and through the nonlinear mapping relationship of the network, a fused image with high spatial resolution and high spectral information is generated.

[0038] Embodiment 1: a hyperspectral / multispectral image fusion method based on deep Kalman filtering, comprising the following steps: S1, construct training samples: Since there is a lack of ideal fusion reference images in real scenes, the present application follows the Wald protocol principle to design the construction process of training samples. Specifically, first, apply Gaussian blur to the original hyperspectral image (HSI) to simulate the spatial degradation of the imaging system, and then perform spatial down-sampling through bilinear interpolation to generate a degraded version of the hyperspectral image for model training. At the same time, the same processing steps are performed on the multispectral image (MSI) to obtain the corresponding training input. The original high-resolution HSI image, which has not been degraded, is used as the fusion target and as the supervision label in the training stage.

[0039] S2, a high-multispectral image fusion network based on a deep Kalman filter is created, which mainly consists of five functional sub-modules: an encoder module, an implicit feature fusion module, an optical flow estimation module, a Kalman gain estimation module, and a Kalman recursive fusion module. Each module cooperates to efficiently model the cross-modal relationship between HSI and MSI and the dynamic evolution process between bands.

[0040] S3, network training is performed: The Adam optimizer is used to update the model parameters, and the network is gradually optimized through multiple iterations, so as to obtain a model with accurate fusion capability.

[0041] S4, using the trained network model, the preprocessed hyperspectral image and multispectral image are used as joint input, and through the nonlinear mapping relationship of the network, a fused image with high spatial resolution and high spectral information is generated.

[0042] In order to prove the effect of the present application, the images taken by ROSIS sensor are used for fusion experiments with 4 times, 8 times and 16 times spatial resolution difference, and the new fusion method at the present stage is compared. The results are shown in Figure 3 . Figure 3 The first row is the fused image, the second row is the SAM error heat map, the third row is the MRAE error heat map, and the fourth row is the difference image. Among them, the SAM error heat map reflects the fidelity of the fused image in the spectrum, the MRAE error heat map reflects the degree of global consistency of the fused image, and the difference image measures the residual error of individual bands. The darker the color, the better the fusion result. It can be seen that the present application has the best effect. The quantitative evaluation of 4 times, 8 times and 16 times fusion is shown in Table 1. It should be noted that the smaller the RMSE, ERGAS and SAM, the better the fusion quality, that is, the closer to the true state, and the higher the PSNR value, the better the fusion performance. It can be seen that the present application is superior to the compared method in 4 times, 8 times and 16 times spatial resolution fusion, which shows that the present application can effectively preserve the space-spectral characteristics of the fused image. And complexity comparison analysis is carried out with the deep algorithm at the present stage, as shown in Table 2, wherein Model Size represents the memory of the model, FLOPS represents the number of floating point operations, and Param represents the amount of learned parameters. It can be seen that the present application has the best performance in memory occupation and parameter amount, which again shows that the present application has certain superiority and light weight degree.

[0043] Table 1 Quantitative evaluation of fusion experiment of the present application and the compared algorithm

[0044] Table 2 Complexity comparison

[0045] Note that the above merely describes the preferred embodiments of the present application and the principles of the technology applied. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, modifications and substitutions can be made by those skilled in the art without departing from the scope of the present application. Therefore, although the present application has been described in detail through the above embodiments, the present application is not limited to the above embodiments, and can include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the claims.

Claims

1. A hyperspectral image fusion method based on deep Kalman filtering, characterized in that, Includes the following steps: S1. Construct training samples: Gaussian blur is applied to the original hyperspectral and multispectral images respectively to simulate the spatial degradation of the imaging system; then spatial downsampling is achieved through bilinear interpolation to generate degraded hyperspectral and multispectral image samples for model training. The original, undegraded, high-resolution hyperspectral image is used as the fusion target and as a supervisory label during the training phase. S2. Construct a hyperspectral image fusion network model based on deep Kalman filtering: By jointly modeling the optical flow field and latent states, a fusion network model is constructed, consisting of an encoder module, an implicit feature fusion module, an optical flow estimation module, a Kalman gain estimation module, and a Kalman recursive fusion module. The current state is conceptualized as information corresponding to the current band, and optical flow information is used as a guiding condition for spectral correlation. The optical flow is calculated through a convolutional neural network to capture pixel-level motion relationships between adjacent bands. The formula for the fusion network model is as follows: ， in, This represents the optical flow information from the previous band to the current band. For the feature fusion operation of the design, For state transition model, For the current input depth control model, This is process noise; S3. Conduct network training: The Adam optimizer is used to update the model parameters constructed in step S2. The network is gradually optimized through multiple rounds of iteration to obtain a network model with accurate fusion capabilities. S4. Using the network model trained in step S3, the preprocessed hyperspectral image and multispectral image are used as joint inputs. Through the nonlinear mapping relationship of the network, a fused image with both high spatial resolution and hyperspectral information is generated.

2. The hyperspectral image fusion method based on deep Kalman filtering according to claim 1, characterized in that, Step S1 specifically includes the following steps: S11. Spatial filtering of hyperspectral and multispectral images is performed using a Gaussian kernel of size 5×5 to simulate the point spread function effect of the sensor and reproduce the imaging degradation process. S12. Spatial downsampling is performed on the filtered image using bilinear interpolation to generate degraded hyperspectral image samples and multispectral image samples required for training as input data. S13. Unsampling, undegraded, high-resolution raw hyperspectral images are used as reference images and as supervisory labels for network training.

3. The hyperspectral image fusion method based on deep Kalman filtering according to claim 1, characterized in that: In step S2, the encoder module is used to perform shallow feature extraction on hyperspectral and multispectral images and map them to a unified latent representation space; the implicit feature fusion module can achieve cross-modal feature alignment in the continuous spectral domain and enhance the spatial-spectral interaction between bands. The optical flow estimation module can explicitly estimate the optical flow field between adjacent bands of a hyperspectral image to capture subtle interspectral variations. The Kalman gain estimation module models the state covariance using a multilayer perceptron and generates dynamic spectral fusion weights using a lightweight convolution module. The Kalman recursive fusion module completes state prediction and updates for each band based on a Kalman filtering strategy, gradually generating a fused image with high spatial-spectral fidelity.

4. The hyperspectral image fusion method based on deep Kalman filtering according to claim 1, characterized in that: In step S2, the encoder module employs two parallel shallow convolutional structures to extract and encode features from the hyperspectral and multispectral images, respectively, and maps them to a unified latent feature space. The formula corresponding to the encoder module is as follows: ， in, and These represent the features of the hyperspectral and multispectral images of the latent space, respectively. For hyperspectral images, For multispectral images, For convolution operations, both of the above convolutions are... The kernel dimension is set to .

5. The hyperspectral image fusion method based on deep Kalman filtering according to claim 1, characterized in that: In step S2, the implicit feature fusion module uses the low-resolution features of the hyperspectral image extracted by the encoder. High-resolution features of multispectral images and two-dimensional normalized coordinate field Using coordinate-aware implicit neural networks as input, feature prediction and fusion are achieved for high-resolution features of multispectral images. It is used as a guiding feature to introduce high-resolution structural information.

6. The hyperspectral image fusion method based on deep Kalman filtering according to claim 5, characterized in that: The implicit feature fusion module first constructs a continuous two-dimensional coordinate grid in high-resolution space, and then performs interpolation sampling on the hyperspectral image feature map in four directions: upper left, upper right, lower left, and lower right, to obtain the neighborhood response of each query location; simultaneously, it calculates the relative coordinate offset between each location and its sampling center. This is used to enhance the model's ability to perceive spatial structures.

7. The hyperspectral image fusion method based on deep Kalman filtering according to claim 6, characterized in that: During the fusion stage, local sampling features of the hyperspectral image are used. Multispectral image-guided features and relative coordinate encoding The three information sources are concatenated into a fused query vector, which is then input into a multilayer perceptron for nonlinear mapping to generate fused features. , in, For the first The fused input vector for each query location; This represents the spatial offset of the location relative to the center of its sampling neighborhood; the module is denoted from four different directions. The above query process is executed separately to generate four sets of candidate fusion results. Simultaneously, the fusion weight score for each direction is estimated. And generate fusion weights using softmax. The final fusion output is as follows: ， For the first The position in the Fusion output in all directions, The fusion direction weights are derived from the softmax function.

8. The hyperspectral image fusion method based on deep Kalman filtering according to claim 1, characterized in that: In step S2, the optical flow estimation module takes the hyperspectral image as input, first upsamples the hyperspectral image using bilinear methods, and then uses two lightweight convolutional layers to dynamically model the image across wavelengths, extracting implicit optical flow information between continuous wavelengths. Its mathematical expression is as follows: ， in, This is an upsampling operation based on bilinearity. The output optical flow tensor provides the interspectral motion information required for state prediction in the subsequent Kalman recursion module.

9. The hyperspectral image fusion method based on deep Kalman filtering according to claim 1, characterized in that: In step S2, the Kalman gain estimation module first concatenates the hyperspectral image features extracted by the encoder with the multispectral image features along the channel dimension to construct a cross-modal joint feature representation. Then, the fused features are input into a multilayer perceptron to model the information uncertainty between different modalities. This uncertainty representation reflects the confidence level of the current band during the fusion process and provides a basis for gain estimation. Next, the module uses two cascaded convolutional layers to further extract and compress the uncertainty features. Finally, it outputs the Kalman gain tensor, expressed by the following formula: ， in, For uncertainty, To estimate the Kalman gain, the output structure ensures that each interspectral state recursion step corresponds to an independent fusion weight, thereby enabling dynamic control at the band level.

10. The hyperspectral image fusion method based on deep Kalman filtering according to claim 1, characterized in that: In step S2, the input of the Kalman recursive fusion module is... , and implicit fusion features Composition, in which, Used to model the state transition trends between adjacent bands in the spectrum. This is used to adjust the fusion ratio of observed and predicted values ​​in state updates; it is an implicit fusion feature. This represents the observation state of each band predicted by the implicit neural network. The recursive process uses the fused features of the first band as the initial anchor state. And it performs two phases step by step on the spectral dimension: state prediction and state update. State prediction phase: in the first stage Band, based on optical flow characteristics of the previous state Perform translation prediction to obtain the current prior state estimate. ,Right now: ， in, Represents a standard spatial coordinate grid. Bilinear interpolation was achieved on continuous coordinates after offset; State update phase: based on the Kalman gain of the current band. , integrate prior predicted states With observation status The updated state estimate is obtained. : ， in, The fusion weights, obtained through learning, determine the degree of confidence in the current state regarding observations and predictions. The recursive process continues until all bands are fused. Finally, the complete fused image is output.

Images