Hyperspectral imaging method and system based on dual-sensor fusion and storage medium

Abstract

The application provides a hyperspectral imaging method and system based on double-sensor fusion, and a storage medium. The method comprises the following steps: S101, acquiring corresponding RGB data and multispectral RAW data under the same target scene; S102, simultaneously performing space spectrum conversion and multi-channel reconstruction on the RGB data and the multispectral RAW data, to obtain a three-channel RGB image corresponding to the RGB data and a multi-channel multispectral image corresponding to the multispectral RAW data; S103, performing alignment and fusion processing on the three-channel RGB image and the multi-channel multispectral image to obtain a fusion image; and S104, performing spectral channel super-resolution processing on the fusion image to output a hyperspectral image of the target scene. The application can support hyperspectral imaging with high time, high space and high spectral resolution, and is convenient for small devices to implement, has low deployment cost, and has certain universality.

Description

Technical Field

[0001] This invention relates to the field of hyperspectral imaging technology, and in particular to a hyperspectral imaging method, system and storage medium based on dual-sensor fusion. Background Technology

[0002] Hyperspectral imaging technology, as an advanced detection technology integrating optics, spectroscopy, and image processing, can simultaneously acquire spatial geometric information and spectral feature information of a target scene, and has irreplaceable application value in fields such as precision agriculture, environmental monitoring, biomedicine, and industrial quality inspection. The core performance indicators of this technology focus on spatial resolution, temporal resolution, and spectral resolution. At the same time, in scenarios such as portable detection devices and UAV-mounted systems, the demand for miniaturization of devices is becoming increasingly urgent, requiring high-performance imaging within a limited volume.

[0003] Currently, mainstream hyperspectral imaging methods are mainly divided into two categories: the first is the push-broom imaging method based on grating spectral dispersion. This method decomposes the incident light into spectral information of different wavelengths through a grating and completes spatial dimension scanning by combining push-broom motion. Although it can effectively ensure high spatial and spectral resolution and meet the needs of fine spectral analysis, the push-broom process requires a certain amount of time to accumulate, resulting in a significant reduction in temporal resolution, making it difficult to adapt to dynamic target detection scenarios. The second is the area scanning imaging method based on quantum dot spectral chip spectral dispersion. This method uses the spectral selectivity of quantum dot materials to achieve spectral dispersion and directly completes two-dimensional spatial imaging with an area array detector. It can simultaneously ensure spatial and temporal resolution and is suitable for dynamic scene monitoring. However, the number of spectral channels of quantum dot spectral chips is limited, and the spectral bandwidth is wide, resulting in a significant sacrifice in spectral resolution, which cannot meet the needs of high-precision material identification.

[0004] In summary, existing hyperspectral imaging devices generally suffer from a technical bottleneck in achieving both high spatial resolution, high temporal resolution, and high spectral resolution simultaneously in their implementation. This severely limits the application of hyperspectral imaging technology in portable and dynamic detection scenarios. Summary of the Invention

[0005] This invention provides a hyperspectral imaging method, system, and storage medium based on dual-sensor fusion, aiming to solve the problem that existing hyperspectral imaging schemes cannot simultaneously achieve spatial, temporal, and spectral resolution.

[0006] To address the aforementioned technical problems, in a first aspect, the present invention provides a hyperspectral imaging method based on dual-sensor fusion, comprising the following steps:

[0007] S101. Collect corresponding RGB data and multispectral RAW data respectively under the same target scene;

[0008] S102. Simultaneously perform spatial-spectral conversion and multi-channel reconstruction on the RGB data and the multispectral RAW data to obtain a three-channel RGB image corresponding to the RGB image and a multi-channel multispectral image corresponding to the multispectral image, respectively.

[0009] S103. Align and fuse the three-channel RGB image and the multi-channel multispectral image to obtain a fused image;

[0010] S104. Perform spectral channel super-resolution processing on the fused image to output a hyperspectral image of the target scene.

[0011] Furthermore, in step S101, which involves acquiring corresponding RGB data and multispectral RAW data under the same target scene, the acquisition of RGB data is achieved by a first sensing unit containing a broadband pass-through filter that allows 400nm-900nm incident light to pass through and a Bayer color filter array. The acquisition of multispectral RAW data is achieved by a second sensing unit containing a broadband pass-through filter that allows 400nm-900nm incident light to pass through and a multispectral mosaic array filter.

[0012] Furthermore, the multispectral mosaic array filter comprises multiple mosaic macrochannels, each of which contains four microchannels, the four microchannels being:

[0013] Narrow-bandpass channel with center wavelength in the range of 400nm-550nm;

[0014] Narrow-bandpass channel with center wavelength in the range of 550nm-700nm;

[0015] Narrow-bandpass channel with center wavelength in the 700nm-900nm range;

[0016] Full-color channel, or wide channel with an upward-opening arc or near-arc shape.

[0017] Furthermore, step S102 is implemented based on a preset image reconstruction network model, which includes an input unit with two entry points, a channel splitting unit, a shared weight encoder, a multispectral decoding and reconstruction unit, and an RGB decoding and reconstruction unit. Step S102 includes the following sub-steps:

[0018] The RGB data and the multispectral RAW data are simultaneously input into the preset image reconstruction network model through the input unit;

[0019] The pixel in the RGB data and the multispectral RAW data is rearranged by the channel splitting unit so that the neighboring pixels in the RGB data and the multispectral RAW data are arranged in the channel dimension, thereby obtaining the first processed RGB image and the first processed multispectral image respectively.

[0020] The shared weight encoder is used to perform downsampling feature learning on the first processed RGB image and the first processed multispectral image to obtain a second processed RGB image with RGB features and a second processed multispectral image with multispectral features, respectively.

[0021] The second processed multispectral image is upsampled by the multispectral decoding and reconstruction unit. During upsampling, the multispectral features learned by the shared weight encoder at the same scale are stitched together in the channel dimension. Each upsampling layer doubles the dimension of the image feature space and halves the number of channels. After multiple upsampling layers, a convolutional layer is applied to map the number of channels to be the same as that of the multispectral image to obtain the multichannel multispectral image.

[0022] The second processed RGB image is upsampled by the RGB decoding and reconstruction unit. During upsampling, the RGB features learned by the shared weight encoder at the same scale are stitched together in the channel dimension. After multiple upsampling layers, a convolutional layer is applied to map the number of channels to be the same as that of the RGB image, resulting in the three-channel RGB image. The cross-attention mechanism based on Transformer self-attention is used to perform color correction on the RGB features according to the multispectral features during the stitching process.

[0023] Furthermore, step S103 is implemented based on a preset alignment and fusion network model, which includes a guiding branch coding unit, a reconstructing branch coding unit, a decoding unit, a cross-modal feature fusion unit, and a channel integration unit. Step S103 includes the following sub-steps:

[0024] The guided branch coding unit performs downsampling feature learning on the three-channel RGB image to obtain a third-processed RGB image with underlying RGB features;

[0025] The multi-channel multispectral image is downsampled and its features are learned by the reconstruction branch coding unit to obtain a third-processed multispectral image with underlying multispectral features.

[0026] The resolution of the third-processed multispectral image is gradually increased by the decoding unit using a layer-by-layer upsampling method, wherein:

[0027] Before the first upsampling, the resolution of the third-processed multispectral image is first increased by interpolation. In subsequent upsampling, the cross-modal feature fusion unit introduces the third-processed RGB image with the same resolution as the current third-processed multispectral image in each upsampling. The displacement prediction sub-network included in the cross-modal feature fusion unit obtains the deformation field of the third-processed RGB image with the same resolution as the current third-processed multispectral image. Then, the current third-processed multispectral image is pixel-aligned according to the deformation field, and the third-processed RGB image is stitched together in the channel dimension and then halved in dimension by the convolutional layer to obtain the fourth-processed multispectral image.

[0028] The channel integration unit maps the fourth processed multispectral image dimension obtained by the decoding unit to be the same as the multichannel multispectral image, completes the splicing with the RGB image in the channel dimension, and outputs it as the fused image.

[0029] Furthermore, step S104 is implemented based on a preset spectral reconstruction network model, which includes a feature extraction unit, a deep fusion unit, and a reconstruction unit. Step S104 includes the following sub-steps:

[0030] The feature extraction unit performs low-level spatial feature extraction on the fused image and expands the number of feature channels to obtain a first processed fused image.

[0031] The deep fusion unit extracts local and global features from the first processed fused image to obtain the second processed fused image.

[0032] The reconstruction unit converts the number of feature channels of the second processed fused image to a preset number of hyperspectral image bands through interpolation mapping to obtain a third processed fused image, and outputs the third processed fused image as the hyperspectral image.

[0033] Secondly, the present invention also provides a hyperspectral imaging system based on dual-sensor fusion, comprising:

[0034] The acquisition module is used to acquire corresponding RGB data and multispectral RAW data respectively in the same target scene;

[0035] The multi-channel reconstruction module is used to simultaneously perform spatial-spectral conversion and multi-channel reconstruction on the RGB data and the multispectral RAW data to obtain a three-channel RGB image corresponding to the RGB data and a multi-channel multispectral image corresponding to the multispectral RAW data, respectively.

[0036] The alignment and fusion module is used to perform alignment and fusion processing on the three-channel RGB image and the multi-channel multispectral image to obtain a fused image;

[0037] The feature reconstruction module is used to perform spectral channel super-resolution processing on the fused image and output a hyperspectral image of the target scene.

[0038] Furthermore, the acquisition module includes a first sensing unit and a second sensing unit. The acquisition of RGB data is achieved by the first sensing unit, which includes a broadband pass-through filter that allows 400nm-900nm incident light to pass through and a Bayer color filter array. The acquisition of multispectral RAW data is achieved by the second sensing unit, which includes a broadband pass-through filter that allows 400nm-900nm incident light to pass through and a multispectral mosaic array filter.

[0039] The multispectral mosaic array filter comprises multiple mosaic macrochannels, each of which contains four microchannels, the four microchannels being:

[0040] Narrow-bandpass channel with center wavelength in the range of 400nm-550nm;

[0041] Narrow-bandpass channel with center wavelength in the range of 550nm-700nm;

[0042] Narrow-bandpass channel with center wavelength in the 700nm-900nm range;

[0043] Full-color channel, or wide channel with an upward-opening arc or near-arc shape.

[0044] Thirdly, the present invention also provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the hyperspectral imaging method based on dual-sensor fusion as described in any of the above embodiments.

[0045] Fourthly, the present invention also provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the hyperspectral imaging method based on dual-sensor fusion as described in any of the above embodiments.

[0046] The beneficial effects achieved by this invention are that it proposes a hyperspectral imaging method based on dual-sensor fusion. This method uses RGB and multispectral imaging as a benchmark, and through multi-channel reconstruction and fast inference of the network model, it can support hyperspectral imaging with high temporal, high spatial and high spectral resolution. Moreover, the entire method can be easily implemented with small devices and has low deployment cost, making the method of this invention have a certain degree of versatility. Attached Figure Description

[0047] The present invention will now be described in detail with reference to the accompanying drawings. The above and other aspects of the present invention will become clearer and more readily understood through the detailed description following the accompanying drawings. In the drawings:

[0048] Figure 1 This is a flowchart of the steps of the hyperspectral imaging method based on dual-sensor fusion provided in the embodiments of the present invention;

[0049] Figure 2 This is a schematic diagram of the filter array of the first sensing unit provided in an embodiment of the present invention;

[0050] Figure 3 This is a schematic diagram of the filter array of the second sensing unit provided in an embodiment of the present invention;

[0051] Figure 4 This is a schematic diagram of the structure of a hyperspectral imaging system based on dual-sensor fusion provided in an embodiment of the present invention;

[0052] Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0054] Example 1

[0055] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the steps of a hyperspectral imaging method based on dual-sensor fusion provided in this invention. The hyperspectral imaging method based on dual-sensor fusion includes the following steps:

[0056] S101. Collect corresponding RGB data and multispectral RAW data respectively under the same target scene.

[0057] Specifically, in step S101, which involves acquiring corresponding RGB data and multispectral RAW data under the same target scene, the acquisition of RGB RAW data is achieved by a first sensing unit containing a broadband pass-through filter that allows 400nm-900nm incident light to pass through and a Bayer color filter array. The acquisition of multispectral RAW data is achieved by a second sensing unit containing a broadband pass-through filter that allows 400nm-900nm incident light to pass through and a multispectral mosaic array filter.

[0058] In this embodiment of the invention, two different sensing units simultaneously acquire images of the same target scene. In possible implementations, the imaging device running the method proposed in this embodiment is equipped with both sensing units. The first sensing unit, comprising a broadband pass-through filter allowing 400nm-900nm incident light to pass through and a Bayer color filter array, is used to achieve imaging that balances high spatial and temporal resolution. The second sensing unit, comprising a broadband pass-through filter allowing 400nm-900nm incident light to pass through and a multispectral mosaic array filter, is used to achieve imaging that balances spectral resolution. Therefore, the images obtained by these two different sensing units have different optical characteristics. During implementation, a complete sensing unit should also include a lens, microlenses, a photosensitive chip, and corresponding structural components.

[0059] Furthermore, the multispectral mosaic array filter comprises multiple mosaic macrochannels, each of which contains four microchannels, the four microchannels being:

[0060] Narrow-bandpass channel with center wavelength in the range of 400nm-550nm;

[0061] Narrow-bandpass channel with center wavelength in the range of 550nm-700nm;

[0062] Narrow-bandpass channel with center wavelength in the 700nm-900nm range;

[0063] Full-color channel, or wide channel with an upward-opening arc or near-arc shape.

[0064] For example, the filter array of the first sensing unit is as follows: Figure 2 As shown, the filter array of the second sensing unit is as follows: Figure 3 As shown, Figure 3 Channels 4, 5, and 6 are narrowband pass filters, and channel 7 is a full-color channel. During implementation, each microchannel corresponds to n square pixels on the photosensitive chip, where n ranges from 1 to 5. Each bandpass channel can be implemented using multilayer coatings, metasurfaces, Fabry-Perot cavities, or other methods.

[0065] The purpose of employing narrow bandpasses in the 400nm-550nm and 550nm-700nm range is to achieve better hyperspectral reconstruction results in the visible light range, in collaboration with the first sensing unit of RGB imaging, through the design of ideal bandpass curves. This is for applications such as accurate color temperature detection and color correction. The purpose of employing a narrow bandpass in the 700nm-900nm range is to support accurate detection of the near-infrared spectrum. The purpose of employing a full-color channel is to acquire complete spectral information in the 400nm-900nm band. The purpose of employing upward-opening arc-shaped or near-arc-shaped channels is to avoid overexposure in certain bands while acquiring relatively complete spectral information in the 400nm-900nm band. Among these, employing three precise bandpass channels improves hyperspectral reconstruction performance compared to existing methods based on RGB and wide-spectral channels.

[0066] S102. Simultaneously perform spatial-spectral conversion and multi-channel reconstruction on the RGB data and the multispectral RAW data to obtain a three-channel RGB image corresponding to the RGB data and a multi-channel multispectral image corresponding to the multispectral RAW data, respectively.

[0067] Understandably, when the sensing unit outputs data, both RGB and multispectral RAW data appear as raw single-channel RAW images. However, each pixel in an RGB image actually contains data from three channels: R, G, and B. Based on the structure of the second sensing unit in the aforementioned embodiment, each pixel in a multispectral image also contains data from four microchannels. Therefore, the data from different channels in the RAW data imaged by the sensing unit are dispersed across different spatial dimensions. Since images are acquired through different sensing units, the data from RGB and multispectral images are not initially correlated. Therefore, a method is needed to combine the channel data of the two images to achieve hyperspectral imaging.

[0068] Specifically, step S102 is implemented based on a preset image reconstruction network model, and in order to clarify the changes in image data in subsequent embodiments, the data tensor is defined here in this embodiment of the invention:

[0069] B: Batch Size, which represents the number of images processed in a single batch during deep learning;

[0070] H, W: The height and width of the RGB data collected by the first sensing unit;

[0071] h, w: The height and width of the multispectral RAW data collected by the second sensing unit; due to the physical differences in the installation position and angle of different sensing units, in this embodiment of the invention, h is less than H and w is less than W;

[0072] C: The number of channels in the feature map when the intermediate layers of the network perform feature processing;

[0073] D: Within the operational scope of the Transformer attention mechanism, the dimension of the feature vector after linear projection;

[0074] λ: The total number of bands (i.e., spectral channels) contained in the final generated hyperspectral image.

[0075] The preset image reconstruction network model includes an input unit with two entry points, a channel splitting unit, a shared weight encoder, a multispectral decoding and reconstruction unit, and an RGB decoding and reconstruction unit. Step S102 includes the following sub-steps:

[0076] S1021. The RGB image and the multispectral image are simultaneously input into the preset image reconstruction network model through the input unit.

[0077] According to the above definition, the input shape of the preset image reconstruction network model is [B, 1, H, W] for RGB images and [B, 1, h, w] for multispectral images.

[0078] S1022. The pixels in the RGB data and the multispectral RAW data are rearranged by the channel splitting unit so that the neighboring pixels in the RGB data and the multispectral RAW data are arranged in the channel dimension, thereby obtaining the first processed RGB image and the first processed multispectral image respectively.

[0079] Channel splitting refers to rearranging the 2×2 neighborhood pixel units in each image, migrating the pixel information from the spatial dimension to the channel dimension. After processing, the original data shape of the RGB image changes from [B, 1, H, W] to [B, 4, H / 2, W / 2], and the original data shape of the multispectral image changes from [B, 1, h, w] to [B, 4, h / 2, w / 2]. Through step S1022, while preserving information, the spatial resolution of both data streams is halved, preparing for subsequent feature extraction.

[0080] S1023. The first processed RGB image and the first processed multispectral image are downsampled and learned by the shared weight encoder to obtain a second processed RGB image with RGB features and a second processed multispectral image with multispectral features, respectively.

[0081] Specifically, the shared weight encoder includes three downsampling stages, each containing a residual block and a convolutional layer with a stride of 2, aiming to gradually reduce spatial resolution, expand the receptive field, and extract the low-level spatial features common to the scene. The output RGB feature shape is [B, C, H / 16, W / 16], and the multispectral feature shape is [B, C, h / 16, w / 16].

[0082] S1024. The second processed multispectral image is upsampled by the multispectral decoding and reconstruction unit. During upsampling, the multispectral features of the same scale learned by the shared weight encoder are stitched together in the channel dimension. Each upsampling layer doubles the dimension of the image feature space and halves the number of channels. After multiple upsampling layers, a convolutional layer is applied to map the number of channels to be the same as that of the multispectral image to obtain the multi-channel multispectral image.

[0083] The multispectral decoding and reconstruction unit described in this embodiment of the invention is implemented based on the standard U-Net architecture decoder. It uses three upsampling stages, each of which uses a transposed convolution with a stride of 2 to amplify the resolution by a factor of 2 and halve the number of channels to gradually restore the spatial resolution. Each upsampling stage uses skip connections to concatenate the upsampled feature map with the multispectral features of the corresponding layer of the encoder in step S1023 in the channel dimension to supplement high-resolution shallow details. After concatenation, the information is fused through convolutional blocks containing GeLU activation function and layer normalization, and finally the concatenated intermediate multispectral feature map data is output with the shape [B, C_final, h / 2, w / 2].

[0084] Next, a structure consisting of a 1×1 convolutional layer, a GeLU activation function, and layer normalization is needed to reconstruct the feature map, mapping the number of channels from C_final to 4, and finally obtaining a multi-channel multispectral image with a data shape of [B, 4, h / 2, w / 2].

[0085] S1025. The second processed RGB image is upsampled by the RGB decoding and reconstruction unit. During upsampling, the RGB features learned by the shared weight encoder at the same scale are stitched together in the channel dimension. After multi-layer upsampling, a convolutional layer is applied to map the number of channels to be the same as the RGB image to obtain the three-channel RGB image. Among them, the cross-attention mechanism based on Transformer self-attention is used to make the RGB features color corrected according to the multispectral features during the stitching process.

[0086] The implementation architecture of step S1025 is basically the same as that of step S1024, but a new cross-attention mechanism based on Transformer self-attention is added. The purpose is to enable the model to learn purer color information from multispectral features when calculating RGB features. This mechanism enables the network to learn a scene-adaptive, non-linear color correction transformation.

[0087] Specifically, through three upsampling stages, each stage uses a transposed convolution with a stride of 2 to magnify the resolution by a factor of 2 and halve the number of channels, gradually restoring the spatial resolution. Each upsampling stage uses skip connections to concatenate the upsampled feature map with the RGB features of the corresponding layer of the encoder in step S1023 in the channel dimension, supplementing high-resolution shallow details. After concatenation in each upsampling stage, the currently concatenated RGB features are used as the "Query", and the multispectral features of the same stage in step S1024 are used as the "Key" and "Value". Through a cross-attention mechanism, the RGB features borrow the pure spectral information of the multispectral array to dynamically correct the color. After correction, the information is fused through convolutional blocks to output an intermediate RGB feature map with the shape [B, C_final, H / 2, W / 2].

[0088] Next, a reconstruction is required using a structure consisting of a 1×1 convolutional layer, a GeLU activation function, and layer normalization to map the number of channels in the feature map from C_final to 3, ultimately resulting in a three-channel RGB image with a data shape of [B, 3, H / 2, W / 2].

[0089] In the implementation of this invention, the preset image reconstruction network model adopts a composite loss function that combines pixel-level L1 loss and VGG perceptual loss to ensure that the output three-channel RGB image and multi-channel multispectral image meet pixel accuracy and have a natural visual effect.

[0090] S103. Align and fuse the three-channel RGB image and the multi-channel multispectral image to obtain a fused image.

[0091] Specifically, in step S102, a three-channel RGB image (high resolution, data shape [B, 3, H / 2, W / 2], where 3 is the RGB three-channel) and a multi-channel multispectral image (low resolution, data shape [B, 4, h / 2, w / 2], where 4 is the multispectral four-channel) are obtained. The initial spatial resolutions of the two are inconsistent (H / W is much larger than h / w), and the pixels are not aligned. Therefore, it is necessary to unify the size and match the position before performing hyperspectral reconstruction.

[0092] Step S103 is implemented based on a preset alignment and fusion network model, which includes a guiding branch coding unit, a reconstructing branch coding unit, a decoding unit, a cross-modal feature fusion unit, and a channel integration unit. Step S103 includes the following sub-steps:

[0093] S1031. The three-channel RGB image is downsampled and its features are learned by the guiding branch coding unit to obtain a third processed RGB image with underlying RGB features.

[0094] During implementation, the branch coding unit processes the high-resolution three-channel RGB image [B, 3, H / 2, W / 2] through three downsampling layers, each containing downsampling convolution and three residual blocks, extracting detailed features such as structure and texture of the image, and finally outputting a third-processed RGB image with the underlying RGB features, with the shape [B, C, H / 16, W / 16], where C is the number of feature channels, and the resolution is reduced to 1 / 8 of the original RGB image.

[0095] S1032. The multi-channel multispectral image is downsampled and learned by the reconstruction branch coding unit to obtain a third-processed multispectral image with underlying multispectral features.

[0096] The reconstructed branch coding unit processes the low-resolution multi-channel multispectral image [B, 4, h / 2, w / 2]. It also extracts the low-level spectral features unique to multispectral images through three downsampling layers, and finally outputs a third-processed multispectral image with low-level multispectral features, with a shape of [B, C, h / 16, w / 16], and the resolution is reduced to 1 / 8 of the original multispectral image.

[0097] S1033. The resolution of the third-processed multispectral image is gradually increased by the decoding unit using a layer-by-layer upsampling method, wherein:

[0098] Before the first upsampling, the resolution of the third-processed multispectral image is first increased using an interpolation algorithm. In subsequent upsampling, resolution increase is not required. The cross-modal feature fusion unit introduces a third-processed RGB image with the same resolution as the current resolution-increased third-processed multispectral image during each upsampling. The displacement prediction sub-network included in the cross-modal feature fusion unit obtains the deformation field of the third-processed RGB image with the same resolution as the current third-processed multispectral image. Then, the current third-processed multispectral image is pixel-aligned according to the deformation field. After alignment, it is concatenated with the third-processed RGB image in the channel dimension and passed through a convolutional layer to halve the dimension, resulting in the fourth-processed multispectral image.

[0099] Specifically, the multispectral feature map [B, C, h / 16, w / 16] is input into the decoding unit. Through layer-by-layer upsampling (using transposed convolution with a stride of 2, increasing the resolution by a factor of 2 and adjusting the number of channels each time), the resolution of the original multispectral image is gradually restored. After each layer of upsampling, the cross-modal feature fusion unit is activated, introducing a third-processed RGB image from the guiding branch coding unit in step S1031, which is from the same stage as the current third-processed multispectral image, to complete alignment and preliminary fusion. This step can be specifically represented as the following process:

[0100] Uniform feature size: Since the resolution of the third-processed RGB image and the multispectral feature map are still inconsistent during the first upsampling, bilinear interpolation is first performed on the third-processed multispectral image to enlarge it to the same size as the current scale of the third-processed RGB image [B, C, H / 16, W / 16]. It is not necessary to perform bilinear interpolation on the third-processed multispectral image again during subsequent upsampling.

[0101] Predicting the deformation field: The third-processed multispectral image and the third-processed RGB image of uniform size are fed into the displacement prediction sub-network. The displacement prediction sub-network contains three 3*3 convolutional blocks and one 2-channel 1*1 output convolutional layer. This sub-network is used to analyze the spatial structure difference between the two types of features and outputs the 2-channel deformation field [B, 2, H_s, W_s] of the third-processed multispectral image compared with the third-processed RGB image.

[0102] Spatial Transform Network (STN) Aligns Multispectral Features: The third-processed multispectral image and the deformation field are input into the spatial transform network together.

[0103] First, an irregular sampling grid is generated by creating a regular coordinate grid and superimposing a deformation field. This grid is used to determine which position in the original multispectral feature map each pixel in the output aligned feature map needs to be sampled from.

[0104] Then, using a bilinear interpolation algorithm, the third-processed multispectral image is resampled in a differentiable manner to generate a fourth-processed multispectral image [B, C_ms, H_s, W_s] that is perfectly aligned with the pixels of the third-processed RGB image.

[0105] S1034. The channel integration unit performs channel dimension mapping on the fourth processed multispectral image obtained by the decoding unit, completes the splicing with the three-channel RGB image, and outputs it as the fused image.

[0106] After aligning the multispectral low-level features with the RGB low-level features in step S1033, the fusion operation continues: the aligned fourth-processed multispectral image and the third-processed RGB image of the same scale are stitched together again in the channel dimension.

[0107] Input a 1×1 convolutional layer to perform information integration and channel dimensionality reduction on the concatenated features, allowing the network to autonomously learn how to optimally combine the detailed features of RGB and the spectral features of multispectral images. The alignment and fusion process is repeated in each layer of the decoding unit until the resolution of the multispectral feature map is restored to be consistent with the original high-resolution RGB image. At this point, the feature map shape is [B, C_ms, H / 2, W2].

[0108] Finally, the final fused feature map is processed by a convolutional block to output a high-resolution multispectral image [B, 4, H / 2, W / 2] with the same size as the high-resolution RGB image and strictly aligned pixels. This image is then combined with the three-channel RGB image [B, 3, H / 2, W / 2] obtained in step S102 to output the fused image with 7 channels.

[0109] S104. Perform spectral channel super-resolution processing on the fused image to output a hyperspectral image of the target scene.

[0110] A high-resolution 4-channel multispectral fused image was obtained through step S103. Combined with the 3-channel RGB image obtained in step S102, the sizes of the two images have been unified. At this point, multispectral reconstruction processing of the fused image [B, 7, H / 2, W / 2] of all 7 channels of data was performed through step S104.

[0111] Specifically, step S104 is implemented based on a preset spectral reconstruction network model, which includes a feature extraction unit, a deep fusion unit, and a reconstruction unit. Step S104 includes the following sub-steps:

[0112] S1041. The feature extraction unit performs low-level spatial feature extraction on the fused image and expands the number of feature channels to obtain a first processed fused image.

[0113] The feature extraction unit is mainly composed of a CNN convolutional neural network. During implementation, it captures basic spatial details of the image, such as edges and textures, through one 3×3 convolutional block and three residual blocks.

[0114] By using a 1×1 convolutional block, the features of the original 7 channels are expanded to the channel width C required for internal processing within the network, and the first processed fused image [B, C, H / 2, W / 2] is output, that is, the number of channels changes from 7 to C, while the spatial resolution remains unchanged.

[0115] S1042. The first processed fused image is processed by the deep fusion unit to extract local and global features to obtain the second processed fused image.

[0116] The deep fusion unit includes a local spatial module and a global spectral Transformer module, which are used for local feature extraction and global feature extraction, respectively.

[0117] The local spatial module uses a standard residual block (containing 3 2D convolutions, layer normalization, and GeLU activation) to perform spatial dimension depth extraction on the current feature map, further optimizing local details such as edges and textures, and outputting a local spatial feature map [B, C, H / 2, W / 2] with the same size as the input.

[0118] The global spectral Transformer module first reshapes the feature map [B, C, H / 2, W / 2], flattening the spatial dimension (H / 2 × W / 2) into a sequence. Each pixel corresponds to one "token" in the sequence, with a sequence length of H / 2 × W / 2 and a feature dimension of C, resulting in a shape of [B, H / 2 × W / 2, C]. Then, the data is fed into a standard Transformer encoder layer (containing multi-head self-attention and a feedforward network), where the spectral features of each pixel are correlated with the spectral features of all pixels in the entire image to calculate association weights, learning global spectral patterns. After processing, the sequence is reshaped back into the image grid, outputting a global spectral feature map of [B, C, H / 2, W / 2].

[0119] Finally, the local spatial feature map and the global spectral feature map are fused by element-wise addition to output the second processed fused image [B, C, H / 2, W / 2].

[0120] S1043. The reconstruction unit converts the number of feature channels of the second processed fused image to the preset number of hyperspectral image bands through a 1×1 convolutional layer to obtain a third processed fused image, and outputs the third processed fused image as the hyperspectral image.

[0121] As defined in the above embodiments, the preset number of hyperspectral image bands is λ. The reconstruction unit uses at least one 2D convolutional layer as its core, mapping the number of channels in the feature map from the C interpolation within the network to the preset number of hyperspectral image bands λ. For example, if the target is a hyperspectral image of 400-700nm with a stride of 10nm, then λ=31. Step S1043 realizes the conversion from C feature channels to λ spectral bands, outputting a third-processed fusion image with shape [B, λ, H / 2, W / 2], and outputting it as a hyperspectral image with high spatial resolution and high spectral resolution to complete the hyperspectral reconstruction process.

[0122] The beneficial effects achieved by this invention are that it proposes a hyperspectral imaging method based on dual-sensor fusion. This method uses RGB and multispectral imaging as a benchmark, and through multi-channel reconstruction and fast inference of the network model, it can support hyperspectral imaging with high temporal, high spatial and high spectral resolution. Moreover, the entire method can be easily implemented with small devices and has low deployment cost, making the method of this invention have a certain degree of versatility.

[0123] Example 2

[0124] This invention also provides a hyperspectral imaging system 200 based on dual-sensor fusion, please refer to... Figure 4 , Figure 4 This is a schematic diagram of the structure of a hyperspectral imaging system based on dual-sensor fusion provided in an embodiment of the present invention, which includes:

[0125] Acquisition module 201 is used to acquire corresponding RGB data and multispectral RAW data respectively in the same target scene;

[0126] The multi-channel reconstruction module 202 is used to simultaneously perform spatial-spectral conversion and multi-channel reconstruction on the RGB data and the multispectral RAW data to obtain a three-channel RGB image corresponding to the RGB data and a multi-channel multispectral image corresponding to the multispectral RAW data, respectively.

[0127] The alignment and fusion module 203 is used to perform alignment and fusion processing on the three-channel RGB image and the multi-channel multispectral image to obtain a fused image;

[0128] The feature reconstruction module 204 is used to perform spectral channel super-resolution processing on the fused image and output a hyperspectral image of the target scene.

[0129] Furthermore, the acquisition module 201 includes a first sensing unit and a second sensing unit. The acquisition of the RGB image is achieved by the first sensing unit, which includes a broadband pass-through filter that allows 400nm-900nm incident light to pass through and a Bayer color filter array. The acquisition of the multispectral image is achieved by the second sensing unit, which includes a broadband pass-through filter that allows 400nm-900nm incident light to pass through and a multispectral mosaic array filter.

[0130] The multispectral mosaic array filter comprises multiple mosaic macrochannels, each of which contains four microchannels, the four microchannels being:

[0131] Narrow-bandpass channel with center wavelength in the range of 400nm-550nm;

[0132] Narrow-bandpass channel with center wavelength in the range of 550nm-700nm;

[0133] Narrow-bandpass channel with center wavelength in the 700nm-900nm range;

[0134] Full-color channel, or wide channel with an upward-opening arc or near-arc shape.

[0135] The hyperspectral imaging system 200 based on dual-sensor fusion can implement the steps in the hyperspectral imaging method based on dual-sensor fusion as described in the above embodiments, and can achieve the same technical effect. Referring to the description in the above embodiments, it will not be repeated here.

[0136] Example 3

[0137] This invention also provides a computer device, please refer to... Figure 5 , Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. The computer device 300 includes: a memory 302, a processor 301, and a computer program stored in the memory 302 and executable on the processor 301.

[0138] The processor 301 calls the computer program stored in the memory 302 to execute the steps in the hyperspectral imaging method based on dual-sensor fusion provided in this embodiment of the invention. Please refer to... Figure 1 Specifically, it includes the following steps:

[0139] S101. Collect corresponding RGB data and multispectral RAW data respectively under the same target scene.

[0140] In step S101, which involves acquiring corresponding RGB data and multispectral RAW data under the same target scene, the acquisition of RGB data is achieved by a first sensing unit containing a broadband pass-through filter that allows 400nm-900nm incident light to pass through and a Bayer color filter array. The acquisition of multispectral RAW data is achieved by a second sensing unit containing a broadband pass-through filter that allows 400nm-900nm incident light to pass through and a multispectral mosaic array filter.

[0141] The multispectral mosaic array filter comprises multiple mosaic macrochannels, each of which contains four microchannels, the four microchannels being:

[0142] Narrow-bandpass channel with center wavelength in the range of 400nm-550nm;

[0143] Narrow-bandpass channel with center wavelength in the range of 550nm-700nm;

[0144] Narrow-bandpass channel with center wavelength in the 700nm-900nm range;

[0145] Full-color channel, or wide channel with an upward-opening arc or near-arc shape.

[0146] S102. Simultaneously perform spatial-spectral conversion and multi-channel reconstruction on the RGB data and the multispectral RAW data to obtain a three-channel RGB image corresponding to the RGB data and a multi-channel multispectral image corresponding to the multispectral RAW data, respectively.

[0147] Step S102 is implemented based on a preset image reconstruction network model, which includes an input unit with two entry points, a channel splitting unit, a shared weight encoder, a multispectral decoding and reconstruction unit, and an RGB decoding and reconstruction unit. Step S102 includes the following sub-steps:

[0148] The RGB data and the multispectral RAW data are simultaneously input into the preset image reconstruction network model through the input unit;

[0149] The pixel in the RGB data and the multispectral RAW data is rearranged by the channel splitting unit so that the neighboring pixels in the RGB data and the multispectral RAW data are arranged in the channel dimension, thereby obtaining the first processed RGB image and the first processed multispectral image respectively.

[0150] The shared weight encoder is used to perform downsampling feature learning on the first processed RGB image and the first processed multispectral image to obtain a second processed RGB image with RGB features and a second processed multispectral image with multispectral features, respectively.

[0151] The second processed multispectral image is upsampled by the multispectral decoding and reconstruction unit. During upsampling, the multispectral features learned by the shared weight encoder at the same scale are stitched together in the channel dimension. Each upsampling layer doubles the dimension of the image feature space and halves the number of channels. After multiple upsampling layers, a convolutional layer is applied to map the number of channels to be the same as that of the multispectral image to obtain the multichannel multispectral image.

[0152] The second processed RGB image is upsampled by the RGB decoding and reconstruction unit. During upsampling, the RGB features learned by the shared weight encoder at the same scale are stitched together in the channel dimension. After multiple upsampling layers, a convolutional layer is applied to map the number of channels to be the same as that of the RGB image, resulting in the three-channel RGB image. The cross-attention mechanism based on Transformer self-attention is used to perform color correction on the RGB features according to the multispectral features during the stitching process.

[0153] S103. Align and fuse the three-channel RGB image and the multi-channel multispectral image to obtain a fused image.

[0154] Step S103 is implemented based on a preset alignment and fusion network model, which includes a guiding branch coding unit, a reconstructing branch coding unit, a decoding unit, a cross-modal feature fusion unit, and a channel integration unit. Step S103 includes the following sub-steps:

[0155] The guided branch coding unit performs downsampling feature learning on the three-channel RGB image to obtain a third-processed RGB image with underlying RGB features;

[0156] The multi-channel multispectral image is downsampled and its features are learned by the reconstruction branch coding unit to obtain a third-processed multispectral image with underlying multispectral features.

[0157] The resolution of the third-processed multispectral image is gradually increased by the decoding unit using a layer-by-layer upsampling method, wherein:

[0158] Before the first upsampling, the resolution of the third-processed multispectral image is first increased by interpolation. In subsequent upsampling, the cross-modal feature fusion unit introduces the third-processed RGB image with the same resolution as the current third-processed multispectral image in each upsampling. The displacement prediction sub-network included in the cross-modal feature fusion unit obtains the deformation field of the third-processed RGB image with the same resolution as the current third-processed multispectral image. Then, the current third-processed multispectral image is pixel-aligned according to the deformation field, and it is concatenated with the third-processed RGB image in the channel dimension and the dimension is halved by the convolutional layer to obtain the fourth-processed multispectral image.

[0159] The channel integration unit maps the fourth processed multispectral image obtained by the decoding unit to the same dimension as the multi-channel multispectral image, completing the stitching with the three-channel RGB image in the channel dimension, and outputting it as the fused image.

[0160] S104. Perform spectral channel super-resolution processing on the fused image to output a hyperspectral image of the target scene.

[0161] Step S104 is implemented based on a preset spectral reconstruction network model, which includes a feature extraction unit, a deep fusion unit, and a reconstruction unit. Step S104 includes the following sub-steps:

[0162] The feature extraction unit performs low-level spatial feature extraction on the fused image and expands the number of feature channels to obtain a first processed fused image.

[0163] The deep fusion unit extracts local and global features from the first processed fused image to obtain the second processed fused image.

[0164] The reconstruction unit converts the number of feature channels of the second processed fused image to a preset number of hyperspectral image bands through interpolation mapping to obtain a third processed fused image, and outputs the third processed fused image as the hyperspectral image.

[0165] The computer device 300 provided in this embodiment of the invention can implement the steps in the hyperspectral imaging method based on dual-sensor fusion as described in the above embodiments, and can achieve the same technical effect. Referring to the description in the above embodiments, it will not be repeated here.

[0166] Example 4

[0167] This invention also provides a storage medium storing a computer program. When executed by a processor, the computer program implements the various processes and steps of the hyperspectral imaging method based on dual-sensor fusion provided in this invention and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0168] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by hardware related to computer programs or instructions. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0169] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0170] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0171] The embodiments of the present invention have been described above with reference to the accompanying drawings. The disclosed embodiments are merely preferred embodiments of the present invention. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many equivalent changes in form under the guidance of the present invention without departing from the spirit and scope of the claims. All such changes are within the protection scope of the present invention.

Claims

1. A dual-sensor fusion based hyperspectral imaging method, characterized in that, The method comprises the following steps: S101, acquiring corresponding RGB data and multispectral RAW data under the same target scene respectively; S102, simultaneously performing hyperspectral conversion and multi-channel reconstruction on the RGB data and the multispectral RAW data to obtain a three-channel RGB image corresponding to the RGB data and a multi-channel multispectral image corresponding to the multispectral RAW data respectively; S103, performing alignment and fusion processing on the three-channel RGB image and the multi-channel multispectral image to obtain a fusion image; S104, performing spectral channel super-resolution processing on the fusion image to output a hyperspectral image of the target scene; wherein, step S102 is implemented based on a preset image reconstruction network model, the preset image reconstruction network model comprises an input unit with two entrances, a channel splitting unit, a shared weight encoder, a multispectral decoding and reconstruction unit and an RGB decoding and reconstruction unit, and step S102 comprises the following substeps: inputting the RGB data and the multispectral RAW data into the preset image reconstruction network model simultaneously through the input unit; rearranging the pixels in the RGB data and the multispectral RAW data respectively through the channel splitting unit to arrange the neighborhood pixels in the RGB data and the multispectral RAW data to the channel dimension to obtain a first processed RGB image and a first processed multispectral image respectively; performing down-sampling feature learning on the first processed RGB image and the first processed multispectral image through the shared weight encoder to obtain a second processed RGB image with RGB features and a second processed multispectral image with multispectral features respectively; performing up-sampling processing on the second processed multispectral image through the multispectral decoding and reconstruction unit, wherein the multispectral features of the same scale learned by the shared weight encoder are spliced in the channel dimension during up-sampling, each layer of up-sampling doubles the image feature space dimension and halves the channel number, and after multi-layer up-sampling, a convolution layer is applied to map the channel number to the same number as the channel number of the multispectral RAW data to obtain the multi-channel multispectral image; performing up-sampling processing on the second processed RGB image through the RGB decoding and reconstruction unit, wherein the RGB features of the same scale learned by the shared weight encoder are spliced in the channel dimension during up-sampling, and after multi-layer up-sampling, a convolution layer is applied to map the channel number to the same number as the channel number of the RGB data to obtain the three-channel RGB image; wherein, the cross-attention mechanism based on the Transformer self-attention is used to make the RGB features perform color correction according to the multispectral features during the splicing process.

2. The dual-sensor fusion based hyperspectral imaging method of claim 1, wherein, In the step S101 of collecting corresponding RGB data and multi-spectral RAW data in the same target scene, the RGB data is collected by a first sensing unit comprising a wideband pass filter which is transparent to incident light of 400-900 nm and a Bayer color filter array, and the multi-spectral RAW data is collected by a second sensing unit comprising a wideband pass filter which is transparent to incident light of 400-900 nm and a multi-spectral mosaic array filter.

3. The dual-sensor fusion based hyperspectral imaging method of claim 2, wherein, The multi-spectral mosaic array filter comprises a plurality of mosaic macro-channels, each of which comprises four micro-channels, and the four micro-channels are respectively: a narrowband pass channel with a center wavelength of 400-550 nm; a narrowband pass channel with a center wavelength of 550-700 nm; a narrowband pass channel with a center wavelength of 700-900 nm; a panchromatic channel, or a wide channel in the form of an open upward circular arc or a nearly circular arc.

4. The dual-sensor fusion based hyperspectral imaging method of claim 1, wherein, Step S103 is implemented based on a preset alignment fusion network model, which includes a guide branch encoding unit, a reconstruction branch encoding unit, a decoding unit, a cross-modal feature fusion unit and a channel integration unit, and step S103 includes the following sub-steps: The third processing RGB image with bottom-layer RGB features is obtained by performing down-sampling feature learning on the three-channel RGB image through the guide branch encoding unit; The third processing multi-spectral image with bottom-layer multi-spectral features is obtained by performing down-sampling feature learning on the multi-channel multi-spectral image through the reconstruction branch encoding unit; The resolution of the third processing multi-spectral image is gradually increased by using a layer-by-layer up-sampling method through the decoding unit, wherein: Before the first up-sampling, the third processing multi-spectral image is first processed by a resolution magnification method, and in subsequent up-sampling, the third processing RGB image with the same resolution as the current third processing multi-spectral image is introduced at each up-sampling through the cross-modal feature fusion unit, and a deformation field of the third processing RGB image with the same resolution as the current third processing multi-spectral image is obtained through a displacement prediction sub-network included in the cross-modal feature fusion unit, then the current third processing multi-spectral image is aligned according to the deformation field, and the third processing RGB image is spliced in the channel dimension and processed by a convolution layer to reduce the dimension by half, to obtain a fourth processing multi-spectral image; The fourth processing multi-spectral image obtained by the decoding unit is mapped to the same dimension as the multi-channel multi-spectral image through the channel integration unit, the splicing in the channel dimension with the three-channel RGB image is completed, and the spliced image data is output as the fusion image.

5. The dual-sensor fusion based hyperspectral imaging method of claim 1, wherein, Step S104 is implemented based on a preset spectral reconstruction network model, which includes a feature extraction unit, a deep fusion unit and a reconstruction unit, and step S104 includes the following sub-steps: The feature extraction unit is configured to perform low-level spatial feature extraction on the fused image, and expand the number of feature channels to obtain a first processed fused image; The deep fusion unit is configured to perform local feature extraction and global feature extraction on the first processed fused image to obtain a second processed fused image; The reconstruction unit is configured to convert the number of feature channels of the second processed fused image to a preset number of bands of the hyperspectral image by interpolation mapping to obtain a third processed fused image, and output the third processed fused image as the hyperspectral image.

6. A dual-sensor fusion based hyperspectral imaging system implementing the dual-sensor fusion based hyperspectral imaging method according to any one of claims 1-5, characterized in that, Comprise: The acquisition module is configured to acquire corresponding RGB data and multispectral RAW data under the same target scene; The multi-channel reconstruction module is configured to simultaneously perform spatial-spectral conversion and multi-channel reconstruction on the RGB data and the multispectral RAW data to obtain a three-channel RGB image corresponding to the RGB data and a multi-channel multispectral image corresponding to the multispectral RAW data, respectively; The alignment fusion module is configured to perform alignment fusion processing on the three-channel RGB image and the multi-channel multispectral image to obtain a fused image; The feature reconstruction module is configured to perform spectral channel interpolation and reconstruction processing on the fused image to output a hyperspectral image of the target scene; The multi-channel reconstruction module is implemented based on a preset image reconstruction network model, and the preset image reconstruction network model comprises an input unit with two entrances, a channel splitting unit, a shared weight encoder, a multispectral decoding reconstruction unit and an RGB decoding reconstruction unit. The input unit is configured to simultaneously input the RGB data and the multispectral RAW data into the preset image reconstruction network model; The channel splitting unit is configured to rearrange the pixels in the RGB data and the multispectral RAW data, respectively, so that the neighborhood pixels in the RGB data and the multispectral RAW data are arranged in the channel dimension to obtain a first processed RGB image and a first processed multispectral image, respectively; The shared weight encoder is configured to perform down-sampling feature learning on the first processed RGB image and the first processed multispectral image to obtain a second processed RGB image with RGB features and a second processed multispectral image with multispectral features, respectively; The multispectral decoding reconstruction unit is configured to perform up-sampling processing on the second processed multispectral image, and when up-sampling, the multispectral features of the same scale learned by the shared weight encoder are spliced in the channel dimension, each layer of up-sampling doubles the spatial dimension of the image features and halves the number of channels, and after multi-layer up-sampling, a convolution layer is applied to map the number of channels to the same number of channels as the multispectral RAW data to obtain the multi-channel multispectral image; The second processed RGB image is up-sampled by the RGB decoding reconstruction unit, and the RGB features of the same size learned by the shared weight encoder are spliced in the channel dimension during up-sampling. After multi-layer up-sampling, a convolution layer is applied to map the number of channels to the same number of channels as the RGB data, to obtain the three-channel RGB image. The RGB features are color corrected according to the multi-spectral features during the splicing process by using the cross-attention mechanism based on the Transformer self-attention.

7. The dual-sensor fusion based hyperspectral imaging system of claim 6, wherein, The acquisition module includes a first sensing unit and a second sensing unit. The RGB data is acquired by the first sensing unit containing a wideband pass filter that can transmit incident light of 400-900 nm. The multi-spectral RAW data is acquired by the second sensing unit containing a wideband pass filter that can transmit incident light of 400-900 nm and a multi-spectral mosaic array filter. The multi-spectral mosaic array filter includes a plurality of macro-mosaic channels. Each macro-mosaic channel includes four micro-channels, and the four micro-channels are: a narrowband pass channel with a center wavelength of 400-550 nm; a narrowband pass channel with a center wavelength of 550-700 nm; a narrowband pass channel with a center wavelength of 700-900 nm; a panchromatic channel, or a wide channel in the form of an open upward circular arc or a nearly circular arc.

8. A computer device, comprising: It includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the dual-sensor fusion-based hyperspectral imaging method according to any one of claims 1-5.

9. A storage medium, characterized by The storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the dual-sensor fusion-based hyperspectral imaging method according to any one of claims 1-5.

Images