A hyperspectral image reconstruction method and system based on spatial-spectral fusion

Abstract

A hyperspectral image reconstruction method based on spatial-spectral fusion comprises the following steps: extracting a deep feature map of Bayer data according to a Bayer format spatial position through an adaptive spatial convolution module, constructing a spatial mode mask and generating an adaptive kernel weight; constructing a parallel spatial network branch and a spectral network branch model, respectively performing local feature enhancement in the spatial dimension and the spectral dimension, and adding to obtain a spatial-spectral feature map; constructing a spatial-spectral fusion attention module, and adopting a multi-dimensional cross-spectral self-attention unit and a spatial rectangular self-attention unit to fuse the feature map; and finally outputting a hyperspectral image through channel mapping. The present application can directly use Bayer data to reconstruct a hyperspectral image, reduce errors introduced by demosaicing processing, effectively balance spatial details and spectral fidelity through global and local feature collaborative optimization, and improve the robustness and accuracy of the model.

Description

Technical Field

[0001] This invention relates to a hyperspectral image reconstruction method and system based on spatial spectral fusion, belonging to the fields of computer vision and deep learning technology. Background Technology

[0002] Hyperspectral image reconstruction is one of the core tasks in computational spectral imaging. Its core objective is to recover the complete spectral information and spatial details of a scene from limited observational data (such as multispectral images and compressed sensing measurements). This technology, with its high resolution in the spectral dimension, is widely used in precision agriculture, environmental monitoring, medical diagnosis, and remote sensing.

[0003] With the breakthroughs in deep learning for image restoration tasks, data-driven hyperspectral image reconstruction methods, which do not rely on explicit physical modeling but instead utilize datasets to learn the nonlinear mapping relationship between known images and target hyperspectral images, demonstrate good generalization ability and spectral fidelity. As imaging technology continues to develop, some researchers are attempting hyperspectral image reconstruction based on known RGB images. The advantages are that RGB cameras are widely used in both civilian and industrial applications, their hardware manufacturing cost is far lower than that of hyperspectral cameras, and data acquisition can be completed without requiring specialized operational skills. Furthermore, RGB cameras offer fast imaging speeds and have no special requirements for the shooting environment, enabling rapid data acquisition in diverse scenarios, including indoor and outdoor environments, and dynamic / static scenes.

[0004] While the aforementioned methods provide rich sample support for training deep learning models to some extent, helping to improve the model's generalization ability to different scenes, objects, and lighting conditions, these methods use RGB images obtained from Bayer data acquired by RGB cameras after demosaicing as network input to obtain hyperspectral images, without considering the impact of the demosaicing process on the quality of the reconstructed hyperspectral image. To address this, some researchers directly utilize the fusion of high-resolution Bayer data and low-resolution hyperspectral images to achieve hyperspectral image super-resolution, thereby reducing the error introduced by the demosaicing process before acquiring the RGB image, which affects the quality of the reconstructed high-resolution hyperspectral image. However, these methods neglect the spatial and spectral complementarity, which is crucial for feature learning in spectral reconstruction, limiting the model's performance.

[0005] Therefore, the key to solving the above problems lies in how to construct a hyperspectral image reconstruction method based on Bayer data, while combining spatial and spectral features. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of existing technologies by creatively proposing a hyperspectral image reconstruction method and system based on spatial spectral fusion. This invention not only directly reconstructs hyperspectral images using Bayer data, reducing errors introduced by demosaicing, but also takes into account both the spatial and spectral features obtained from the model network, improving the robustness and accuracy of the hyperspectral image reconstruction model.

[0007] The present invention is achieved using the following technical solution.

[0008] A hyperspectral image reconstruction method based on spatial spectral fusion includes the following steps:

[0009] Step 1: Extract the depth feature map of Bayer data based on the spatial location of the Bayer format using the adaptive spatial convolution module.

[0010] Specifically, the size is obtained as Bayer data, in which Indicates the height of the data. Indicates the width of the data. This represents the three channels of Bayer data. Different color channels in Bayer data have spatial location information differences; therefore, an adaptive spatial convolution module is used to convert the above-mentioned dimensions... Bayer data input, output size The depth feature map, where This refers to the number of channels in the depth feature map. The construction and convolution operation of the adaptive spatial convolution module specifically include:

[0011] First, construct a mask representing the pixel arrangement pattern of the Bayer data space. The specific steps are as follows: First, initialize a mask with a size of... A three-dimensional zero matrix is ​​used to carry the spatial pattern encoding of Bayer data; then, values ​​are assigned to the zero matrix based on the pixel arrangement rules of the Bayer array; finally, the assigned matrix is ​​converted into a tensor form, and the tensor is expanded in dimension to obtain a dimension of... Spatial pattern mask This mask fully represents the pixel arrangement pattern information of each spatial location of the Bayer data.

[0012] Then, use the weight generation function. Spatial pattern mask After performing feature extraction and dimension mapping to obtain the weight tensor, a dimension reshaping operation is performed to obtain a tensor of size [value missing]. Adaptive spatial kernel weights ,in Let be the number of convolutional groups. Each element of this weight tensor corresponds to an adaptive spatial kernel parameter for each spatial location in the Bayer data. This process can be represented as:

[0013]

[0014] in, This indicates a dimension reshaping operation.

[0015] Finally, Bayer data is input into the channel kernel to perform channel convolution operations, resulting in channel convolution feature maps. This process extracts the dependencies between channels. Feature unrolling is then performed on the channel convolutional feature maps. Extract each spatial location Local neighborhood feature windows are generated, and each window is flattened into a one-dimensional vector. The resulting expanded feature tensor is then reshaped to match the dimensions with the adaptive spatial kernel weights. Finally, the adaptive spatial kernel weights are... Element-wise multiplication with the expanded feature tensor is performed to achieve spatial adaptive weighting and dimensional reshaping, resulting in a size of Spatial adaptive spatial depth feature map This process can be formalized as follows:

[0016]

[0017] in, The symbol ⊙ indicates a dimension reshaping operation, and ⊙ indicates element-wise multiplication.

[0018] Step 2: Construct parallel spatial network branch and spectral network branch models, perform local feature enhancement in the spatial and spectral dimensions respectively to obtain spatial feature maps and spectral feature maps, and add them together to obtain a spatial-spectral feature map.

[0019] Step 2.1: Use spatial network branches to enhance local spatial features.

[0020] The spatial network branches include a shallow spatial feature extraction module, a deep spatial feature extraction module, and a low-frequency information preservation module. During shallow spatial feature extraction, the adaptive spatial convolution module from step 1 is used to extract features from the deep feature map obtained in step 1, resulting in a shallow spatial feature map.

[0021] Subsequently, during deep spatial feature extraction, N spatial residual groups are cascaded together. The shallow spatial feature map is input into this cascaded structure, and the deep spatial feature map is output. Each spatial residual group... The structure consists of M residual spatial attention blocks connected in series, with a short skip connection at the end of each spatial attention block to add the input features of the spatial residual group to the output features of the M residual spatial attention blocks element-wise, achieving residual fusion of local features. The operation of the i-th spatial residual group can be expressed as:

[0022]

[0023] in, This represents the j-th residual space attention block. hour, This is a shallow spatial feature map.

[0024] The residual spatial attention block is used to capture spatial dependencies between feature maps. Internally, it adaptively adjusts the feature weights at different spatial locations through a spatial attention mechanism, strengthening the representation of key spatial information. Spatial attention weights. The calculation can be simplified to:

[0025]

[0026] in, Indicates channel splicing. It is a convolutional layer. For input features, It is the Sigmoid activation function. and These are average pooling and max pooling operations, respectively.

[0027] Finally, low-frequency information is preserved. A long jumper connection is set up to directly add the original input depth feature map and spatial feature map element-wise to obtain the spatial feature map. This preserves the original low-frequency information of the original input features and avoids the loss of low-frequency information during the deep feature extraction process.

[0028] Step 2.2: Use spectral network branches to enhance local spectral features.

[0029] The spectral network branch includes a shallow spectral feature extraction module and a deep spectral feature extraction module. In the shallow spectral feature extraction, a single convolutional layer is used to extract features from the deep feature map obtained in step 1. The convolutional kernel size of the convolutional layer is 3×3, and the stride is 1, resulting in the shallow spectral feature map.

[0030] Subsequently, during deep spectral feature extraction, N spectral residual groups are connected in series. The shallow spectral feature map is input into this series structure, and the spectral feature map is output. Where N is the same as the number of spatial residual groups in the spatial network branches, and each spectral residual group... The attention blocks are connected in series by P channels. The operation of the i-th spectral residual group is:

[0031]

[0032] in, This represents the j-th residual space attention block. hour, This is a shallow spatial feature map.

[0033] The spatial attention block uses a global average pooling layer to learn the weight coefficients for each feature channel. By employing a channel attention mechanism to enhance the representation of key spectral information and suppress redundant information, the spectral details of hyperspectral images can be efficiently recovered. Weighting coefficients The calculation process can be expressed as:

[0034]

[0035] in, Indicates global average pooling. and For the weights of the fully connected layer, It is the ReLU activation function. This is the Sigmoid activation function.

[0036] Step 2.3: Spatial-spectral feature map fusion generation.

[0037] The spatial feature map obtained in step 2.1 Compared with the spectral feature map obtained in step 2.2 The fusion is performed using a residual addition strategy, specifically expressed as follows:

[0038] in, This results in a fused spatial-spectral feature map. The addition operation is performed element-wise at the corresponding pixel position and channel dimension of the feature map, achieving deep coupling between spatial and spectral information.

[0039] Step 3: Construct a spatial-spectral fusion attention module to fuse spatial-spectral feature maps.

[0040] Specifically, a spatial-spectral fusion attention module is constructed, which adopts a fusion strategy of collaborative modeling of multi-dimensional spectral features and spatial features, and combines cross-spectral self-attention mechanism and spatial rectangle self-attention mechanism to achieve accurate alignment and fusion of features. The module consists of three parts: multi-dimensional cross-spectral self-attention unit (step 3.1), spatial rectangle self-attention unit (step 3.2), and feature fusion unit (step 3.3).

[0041] Step 3.1: Perform multi-dimensional cross-spectral self-attention units.

[0042] Multi-dimensional cross-spectral self-attention units are used to capture global contextual information with linear complexity. The core is the application of self-attention mechanisms across channel dimensions rather than spatial dimensions, generating a global attention map by calculating the cross-covariance between channels. First, feature preprocessing and local context enhancement are performed. The spatial-spectral feature map obtained in step 2... After performing layer normalization to obtain a normalized tensor, the spatial-spectral feature map obtained in step 2 is processed by a combination of 1×1 point convolution and 3×3 depth convolution. Perform feature projection and local context encoding to generate query Q, key K, and value V tensors, where the specific expression for Q is:

[0043]

[0044] in, () indicates that a 1×1 point convolution is used to aggregate pixel-level cross-channel context information. This indicates that a 3×3 depthwise convolution is used to encode channel-level spatial local context information.

[0045] Next, Q and K are dimensionally reshaped to create an attention map generated by the dot product interaction of Q and K. Simultaneously, a multi-head attention mechanism is employed to increase the number of channels. The system is divided into M independent attention heads, and multiple independent attention maps are learned in parallel. Finally, the multi-head features are fused by splicing and linear projection to obtain the output of the multi-dimensional cross-spectral self-attention unit.

[0046] Step 3.2: Perform spatial rectangular self-attention units.

[0047] Spatial rectangular self-attention units are used to enhance the model's perception of spatial regions, achieved through vertical-horizontal bi-branch rectangular self-attention and spectral shuffling operations. First, the spatial-spectral feature map obtained in step 2 is processed... Perform spectral dimension segmentation, segmenting into and Two sub-feature tensors, with size After that, Perform horizontal multi-head self-attention rectangle partitioning, dividing the data into N non-overlapping horizontal rectangular sub-blocks, where each sub-block has a size of [missing value]. For each horizontal rectangular sub-block, a query Q, key K, and value V are generated using a projection matrix, and a learnable positional encoding parameter P is embedded, resulting in the expression for each horizontal rectangular sub-block:

[0048]

[0049] in, For feature dimensions P is the location encoding matrix, used to supplement spatial location information. Next, feature aggregation is performed, concatenating the attention outputs of all horizontal rectangular sub-blocks to obtain the feature tensor. .

[0050] right The operation procedure for vertical multi-head self-attention is the same as described above, and the output feature tensor is... Finally, spectrum shuffling and branch fusion are performed. and Perform a spectrum shuffling operation, which interleaves and recombines the channel dimensions of the two sub-feature tensors to expand the overall receptive field of the module; then concatenate the channel dimensions of the shuffled feature tensors to obtain the output of the spatial rectangular self-attention unit.

[0051] Step 3.3: Perform feature fusion unit.

[0052] The output features of the multi-dimensional cross-spectral self-attention unit obtained in step 3.1 are element-wise added and fused with the output features of the spatial rectangular self-attention unit obtained in step 3.2 to obtain preliminary fused features. A 1×1 convolutional layer is then used to transform the channel dimension of the preliminary fused features, increasing the number of feature channels from... Mapped to the number of target channels in a hyperspectral image The final output size is Hyperspectral images.

[0053] Based on the above method, this invention further proposes a hyperspectral image reconstruction system based on spatial spectral fusion, comprising an adaptive spatial feature extraction subsystem, a spatial-spectral dual-branch feature enhancement subsystem, a spatial-spectral fusion reconstruction subsystem, and a supervised optimization and result evaluation subsystem. The connections between these components are as follows:

[0054] The output of the adaptive spatial feature extraction subsystem is connected to the input of the spatial-spectral bi-branch feature enhancement subsystem; the output of the spatial-spectral bi-branch feature enhancement subsystem is connected to the input of the spatial-spectral fusion reconstruction subsystem; and the output of the spatial-spectral fusion reconstruction subsystem is connected to the input of the supervised optimization and result evaluation subsystem.

[0055] Beneficial effects

[0056] Compared with the prior art, the present invention has the following advantages:

[0057] 1. This invention can directly reconstruct hyperspectral images based on Bayer data without modifying existing hardware systems or de-mosaicing the hardware output. It is applicable to Bayer data output from various RGB imaging devices such as consumer cameras and smartphones, and has a wide range of application scenarios.

[0058] 2. This invention designs independent spatial and spectral network branches to extract spatial and spectral features respectively, ensuring both spatial detail clarity and spectral information fidelity. By differentiating and optimizing the network structure and feature extraction mechanisms of the two types of branches, targeted separation and extraction of spatial and spectral features are achieved.

[0059] 3. This invention innovatively designs a spatial-spectral fusion attention network with global-local feature collaborative optimization as its core objective. Through a refined modular architecture design, it simultaneously achieves deep modeling of global context information and accurate perception of local spatial regions. Ultimately, it achieves the dual optimization goals of spatial detail recovery and spectral information fidelity through an efficient feature fusion mechanism. Attached Figure Description

[0060] Figure 1 This is a flowchart of the method of the present invention.

[0061] Figure 2 This is a schematic diagram of the neural network structure described in the method of the present invention.

[0062] Figure 3 This is a schematic diagram of the system composition of the present invention. Detailed Implementation

[0063] This embodiment discloses a hyperspectral image reconstruction method based on spatial spectral fusion. To better illustrate the purpose and advantages of the present invention, the method will be further described below with reference to the accompanying drawings and examples.

[0064] Example

[0065] A hyperspectral image reconstruction method based on spatial spectral fusion includes the following steps:

[0066] Step 1: Extract the depth feature map of Bayer data based on the spatial location of the Bayer format using the adaptive spatial convolution module.

[0067] First, construct a mask representing the pixel arrangement pattern in the Bayer data space, and initialize a mask with a size of... A three-dimensional zero matrix is ​​used to carry the spatial pattern encoding of Bayer data. Then, the spatial region division rules of the feature map are defined: even-numbered row regions are sets of rows with row indices starting from 0 and a step size of 2, and odd-numbered row regions are sets of rows with row indices starting from 1 and a step size of 2; even-numbered column regions are sets of columns with column indices starting from 0 and a step size of 2, and odd-numbered column regions are sets of columns with column indices starting from 1 and a step size of 2. Next, the zero matrix is ​​assigned values ​​based on the pixel arrangement rules of the Bayer array: even-numbered rows and even-numbered columns of the first channel are assigned a value of 1; even-numbered rows and odd-numbered columns of the second channel are assigned a value of 2; odd-numbered rows and even-numbered columns of the third channel are assigned a value of 3; and odd-numbered rows and odd-numbered columns of the second channel are assigned a value of 2. The assigned matrix is ​​converted into tensor form, and the tensor is dimensionally expanded to obtain the spatial pattern mask, which fully represents the pixel arrangement pattern information of each spatial location in the Bayer data. Subsequently, the spatial pattern mask is subjected to feature extraction and dimension mapping using a weight generation function to obtain adaptive spatial kernel weights. Each element of this weight tensor corresponds to the spatial adaptive kernel parameters of each spatial location in the Bayer data.

[0068] Finally, Bayer data is input into the channel kernel to perform channel convolution operations, resulting in channel convolution feature maps, which are then expanded. The adaptive spatial kernel weights are multiplied element-wise with the expanded feature tensor to complete spatial adaptive weighting and dimensional reshaping, yielding the size. The depth feature map of the spatial adaptive space.

[0069] Step 2: Construct parallel spatial network branch and spectral network branch models, perform local feature enhancement in the spatial and spectral dimensions respectively to obtain spatial feature maps and spectral feature maps, and add them together to obtain a spatial-spectral feature map.

[0070] Step 2.1: Use spatial network branches for local spatial feature enhancement. In this embodiment, the size of the input spatial feature map is... The spatial network branch parameters are set with three spatial residual groups, each containing five residual spatial attention blocks. Each residual spatial attention block consists of a 3×3 convolutional layer, batch normalization, LeakyReLU activation, a spatial attention submodule, and a 1×1 convolutional layer. The 3×3 convolutional layer has a stride of 1 and maintains 64 output channels, used to extract local spatial neighborhood features. Batch normalization normalizes the convolutional output features, accelerating network training convergence and avoiding gradient vanishing. The LeakyReLU activation function retains more low-response feature information. The spatial attention submodule uses a structure of global average pooling, 1×1 convolution, LeakyReLU, 1×1 convolution, and Sigmoid activation to generate a spatial attention weight map. Pixel-by-pixel weights are assigned to the feature map, strengthening the feature responses of key spatial regions such as edges and textures, and suppressing background redundancy. The 1×1 convolutional layer is used for feature dimension calibration, maintaining the number of output feature channels consistent with the input. Simultaneously, a short skip connection is set at the end of each spatial residual group to achieve residual fusion of local features and alleviate the gradient decay problem in deep network training. After the three spatial residual groups are processed in series, a spatial feature map with enhanced local spatial features is output. This feature map has clear spatial detail representation and complete structural information.

[0071] Step 2.2: Extract spectral features using spectral network branches. In this embodiment, the size of the input spectral feature map is... The spectral network branch parameters are set to three spectral residual groups, each containing five channel attention blocks. Each channel attention block consists of a 3×3 convolutional layer, batch normalization, LeakyReLU activation, a channel attention submodule, and a 1×1 convolutional layer. The channel attention submodule employs a structure of global average pooling, 1×1 convolution, LeakyReLU, 1×1 convolution, and Sigmoid activation to generate a channel attention weight map. Channel-by-channel weights are assigned to the feature map, strengthening the responses of key channels related to the spectral characteristics of the substance and suppressing redundant channel information. A short skip connection is added at the end of each spectral residual group to achieve residual fusion of spectral features. After the three spectral residual groups are concatenated, a locally enhanced spectral feature map is output, which accurately represents the spectral dependencies between channels and the spectral response characteristics of the substance.

[0072] Step 2.3: Spatial-Spectral Feature Map Fusion Generation. The spatial feature map obtained in Step 2.1 is added to and fused with the spectral feature map obtained in Step 2.2 to obtain the spatial-spectral feature map.

[0073] Step 3: Construct a spatial-spectral fusion attention module to fuse spatial-spectral feature maps. A fusion strategy of co-modeling multi-dimensional spectral features and spatial features is adopted, and cross-spectral self-attention mechanism and spatial rectangle self-attention mechanism are combined to achieve accurate alignment and fusion of features.

[0074] Step 3.1: Perform multi-dimensional cross-spectral self-attention units. The size of the input spatial-spectral feature map is... Multi-dimensional cross-spectral self-attention unit parameter settings: number of attention heads Both 1×1 point convolution and 3×3 depth convolution have 64 output channels, resulting in the output features of multi-dimensional cross-spectral self-attention units.

[0075] Step 3.2: Perform spatial rectangular self-attention units. The size of the input spatial-spectral feature map is... Spatial rectangular self-attention unit parameter settings: the number of channels after spectral segmentation is 32; the horizontal rectangular sub-block size... Vertical rectangular sub-block size The attention outputs of all horizontal and vertical rectangular sub-blocks are concatenated to obtain horizontal and vertical feature tensors. Finally, a spectrum shuffling and branch fusion operation is performed. The spectrum shuffling operation interleaves and recombines the channel dimensions of the two sub-feature tensors to expand the overall receptive field of the module. The shuffled feature tensors are then concatenated by channel dimensions to obtain the output features of the spatial rectangular self-attention unit.

[0076] Step 3.3: Perform feature fusion unit. The output features of the multi-dimensional cross-spectral self-attention unit obtained in Step 3.1 are element-wise added and fused with the output features of the spatial rectangular self-attention unit obtained in Step 3.2 to obtain preliminary fused features. A 1×1 convolutional layer is then used to transform the channel dimension of the preliminary fused features, increasing the number of feature channels from... Mapped to the target number of hyperspectral images, the final output size is Hyperspectral images.

[0077] Figure 1 This is the overall flowchart of the method of the present invention. The hyperspectral image reconstruction process of the present invention includes three core steps: (1) Depth feature extraction: Input Bayer data and extract depth feature maps through an adaptive spatial convolution module; (2) Dual-branch feature enhancement: Parallel spatial-spectral dual-branch networks extract spatial feature maps and spectral feature maps respectively, and add them together to obtain a spatial-spectral feature map; (3) Fusion and output: The spatial-spectral fusion module fuses the feature maps and outputs a hyperspectral image.

[0078] Figure 2 This is a schematic diagram of the neural network structure of the present invention. The overall architecture of the neural network includes: Input layer: receiving... Bayer data; Adaptive spatial convolution module: containing three sub-units: spatial pattern mask construction, adaptive kernel weight generation, and spatial adaptive convolution; Parallel dual-branch network: spatial network branch and spectral network branch; Spatial-spectral fusion attention module: containing multi-dimensional cross-spectral self-attention unit, spatial rectangular self-attention unit, and feature fusion unit; Output layer: output Hyperspectral images.

[0079] Figure 3 A schematic diagram of a hyperspectral image reconstruction system based on spatial-spectral fusion provided in this application embodiment includes an adaptive spatial feature extraction subsystem 10, a spatial-spectral dual-branch feature enhancement subsystem 20, a spatial-spectral fusion reconstruction subsystem 30, and a supervised optimization and result evaluation subsystem 40.

[0080] Among them, the adaptive spatial feature extraction subsystem 10 is used to directly receive the original Bayer format data. Through the adaptive spatial convolution module, it extracts the depth feature map according to the spatial pixel arrangement pattern unique to the Bayer array, avoiding the errors introduced by the traditional demosaic process and preserving accurate original spatial and spectral information for subsequent processing.

[0081] The spatial-spectral dual-branch feature enhancement subsystem 20 includes a parallel spatial network branch and a spectral network branch. The spatial network branch enhances local spatial features through residual spatial attention blocks, strengthening the representation of key spatial structures; the spectral network branch enhances local spectral features through channel attention blocks, focusing on the recovery of key spectral channels. The outputs of the two branches are fused using a residual addition strategy to generate a feature map that couples spatial and spectral information.

[0082] The core of the spatial-spectral fusion reconstruction subsystem is the spatial-spectral fusion attention module. This module captures the global spectral context through multi-dimensional cross-spectral self-attention units, enhances the perception of different spatial regions through spatial rectangular self-attention units, and finally fuses the two outputs and maps them through convolutional layers to reconstruct a high-resolution, high-fidelity hyperspectral image.

[0083] The supervised optimization and result evaluation subsystem 40 is used to construct and calculate the reconstruction loss function to optimize the parameters of the entire network in an end-to-end manner. Simultaneously, this system integrates multiple image quality evaluation metrics such as PSNR, SSIM, and SAM to provide an objective and comprehensive performance evaluation of the reconstructed hyperspectral images.

[0084] The connections between the above-mentioned components are as follows:

[0085] The output of the adaptive spatial feature extraction subsystem is connected to the input of the spatial-spectral bi-branch feature enhancement subsystem; the output of the spatial-spectral bi-branch feature enhancement subsystem is connected to the input of the spatial-spectral fusion reconstruction subsystem; and the output of the spatial-spectral fusion reconstruction subsystem is connected to the input of the supervised optimization and result evaluation subsystem.

[0086] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A hyperspectral image reconstruction method based on spatial spectral fusion, characterized in that, Includes the following steps: Step 1: Extract the depth feature map of Bayer data based on the spatial location of the Bayer format using the adaptive spatial convolution module; Step 2: Construct parallel spatial network branch and spectral network branch models, perform local feature enhancement in the spatial and spectral dimensions respectively, and obtain spatial feature maps and spectral feature maps. Add them together to obtain a spatial-spectral feature map. Step 3: Construct a spatial-spectral fusion attention module, adopting a fusion strategy of collaborative modeling of multi-dimensional spectral features and spatial features, and combining multi-dimensional cross-spectral self-attention units and spatial rectangular self-attention units to fuse spatial-spectral feature maps; Step 4: Perform channel mapping on the fused features to output a hyperspectral image.

2. The method as described in claim 1, characterized in that, The construction and convolution operation of the adaptive spatial convolution module in step 1 include: constructing a mask representing the spatial pixel arrangement pattern of the Bayer data to obtain a spatial pattern mask; using a weight generation function to extract features and map dimensions of the spatial pattern mask to obtain a weight tensor, and then performing a dimension reshaping operation to obtain adaptive spatial kernel weights; inputting the Bayer data into the channel kernel to perform channel convolution operations to obtain channel convolution feature maps, and performing a feature unrolling operation; multiplying the unrolled feature tensor element-wise with the adaptive spatial kernel weights to complete spatial adaptive weighting and dimension reshaping to obtain a depth feature map.

3. The method as described in claim 1, characterized in that, The spatial network branch in step 2 includes a shallow spatial feature extraction module, a deep spatial feature extraction module, and a low-frequency information preservation module. The deep spatial feature extraction module consists of N spatial residual groups connected in series. Each spatial residual group includes M residual spatial attention blocks connected in series and short-skip connections. The residual spatial attention blocks adaptively adjust the feature weights for different spatial locations using a spatial attention mechanism. The low-frequency information preservation module uses long-skip connections to add the original input deep feature map element-wise with the feature map processed by the spatial residual groups to obtain the spatial feature map. The spectral network branch in step 2 includes a shallow spectral feature extraction module and a deep spectral feature extraction module. The deep spectral feature extraction module consists of N spectral residual groups connected in series. Each spectral residual group consists of P channel attention blocks connected in series and short-skip connections. The channel attention blocks enhance the representation of key spectral information through a channel attention mechanism. The output spectral feature map is fused with the spatial feature map using a residual addition strategy to obtain a spatial-spectral feature map.

4. The method as described in claim 1, characterized in that, The spatial-spectral fusion attention module in step 3 includes a multi-dimensional cross-spectral self-attention unit, a spatial rectangular self-attention unit, and a feature fusion unit. The multi-dimensional cross-spectral self-attention unit captures global contextual information with linear complexity, generating a global attention map by calculating the cross-covariance between channels. The spatial rectangular self-attention unit enhances the model's perception of spatial regions, achieved through vertical-horizontal bi-branch rectangular self-attention and spectral shuffling operations. The feature fusion unit element-wise adds and fuses the output features of the multi-dimensional cross-spectral self-attention unit and the spatial rectangular self-attention unit to obtain preliminary fused features. Finally, a 1×1 convolutional layer transforms the channel dimensions of the preliminary fused features, mapping the number of feature channels to the target number of channels in the hyperspectral image.

5. A spatial spectral fusion hyperspectral image reconstruction system based on Bayer data, characterized in that, include: Adaptive Spatial Convolution Module: Used to construct spatial pattern masks and generate adaptive kernel weights, and extract depth feature maps based on Bayer format spatial locations; Parallel dual-branch network module: includes parallel spatial network branches and spectral network branches, which are used to extract spatial features and spectral features respectively, and generate a spatial-spectral feature map by adding and fusing them; Spatial-spectral fusion attention module: includes multi-dimensional cross-spectral self-attention units and spatial rectangular self-attention units, used for the collaborative fusion and output of global and local features of spatial-spectral feature maps.

Images