A hyperspectral image imaging method, device, system and storage medium
By employing decoupling and migration strategies, the problem of balancing spatial and spectral resolution in hyperspectral imaging was solved, achieving high-quality hyperspectral image reconstruction that restored spatial details while maintaining spectral accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HUAZHONG UNIV OF SCI & TECH
- Filing Date
- 2026-06-23
- Publication Date
- 2026-07-24
AI Technical Summary
Existing hyperspectral imaging technologies struggle to balance spatial and spectral resolution, resulting in fused images that fail to simultaneously meet the requirements of clear spatial details and accurate spectral information, thus limiting their high-precision applications in complex scenarios.
A decoupling-then-transfer strategy is adopted. Low spatial resolution hyperspectral images and high spatial resolution multispectral images are acquired simultaneously, and spectral and spatial knowledge are extracted separately. Feature enhancement is performed using spectral knowledge transfer models and spatial knowledge transfer models. Finally, spatial spectral features are fused to construct a shared space to achieve feature alignment and fusion.
It achieves high-quality imaging with high spatial resolution and high spectral fidelity, overcomes the problem of feature interaction difficulties caused by modal differences in traditional fusion technology, restores high-frequency spatial details and maintains spectral consistency.
Smart Images
Figure CN122454346A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of spectral imaging technology, and more specifically, relates to an imaging method, apparatus, system and storage medium for hyperspectral images. Background Technology
[0002] Hyperspectral imaging technology can simultaneously capture two-dimensional spatial information and continuous spectral information of a target, providing key data support for the precise identification of material composition, and has important application value in fields such as remote sensing, precision agriculture, environmental monitoring and medical diagnosis.
[0003] However, due to the inherent trade-off between spatial and spectral resolution in optical system design, existing single-mode hyperspectral imaging systems struggle to balance spatial and spectral resolution: acquiring rich spectral information often requires sacrificing spatial resolution, resulting in blurred spatial details and lost textures in hyperspectral images; conversely, pursuing high spatial resolution compromises the integrity of spectral dimensions. This limitation hinders the ideal performance of hyperspectral images in tasks demanding high spatial detail, such as fine-grained ground feature classification and small target detection.
[0004] To overcome this physical limitation, hyperspectral and multispectral image fusion techniques have emerged. This technology aims to enhance the spatial details of low-spatial-resolution hyperspectral images using high-spatial-resolution multispectral images, thereby reconstructing an ideal image that combines both high spatial and hyperspectral resolution. Currently, deep learning-based methods are gradually becoming the mainstream fusion technique. These methods typically employ channel stitching to directly merge bimodal features, or utilize simple attention mechanisms for weighted fusion.
[0005] However, the fused images generated by existing fusion technologies cannot simultaneously meet the requirements of clear spatial details and accurate spectral information, which limits their high-precision application in complex scenarios. Summary of the Invention
[0006] In view of the above-mentioned defects or improvement needs of the prior art, this application provides an imaging method, apparatus, system and storage medium for hyperspectral images, the purpose of which is to solve the technical problem that the fused images generated by the existing fusion technology cannot simultaneously meet the requirements of clear spatial details and accurate spectral information.
[0007] To achieve the above objectives, according to one aspect of this application, an imaging method for hyperspectral images is provided, comprising: S1: Simultaneously acquire a hyperspectral image X with a first spatial resolution and a multispectral image Y with a second spatial resolution; the first spatial resolution is lower than the second spatial resolution. S2: Downsample the hyperspectral image X and the multispectral image Y respectively to obtain the first multispectral image Q1 and the second multispectral image Q2 with the first spatial resolution; S3: Input the second multispectral image Q2 into the spectral knowledge extraction model for spectral knowledge decoupling extraction to obtain a hyperspectral image X' with a first spatial resolution; input the first multispectral image Q1 into the spatial knowledge extraction model for spatial knowledge decoupling extraction to obtain a multispectral image Y' with a second spatial resolution; S4: Input the first spatial resolution hyperspectral image X' into a spatial knowledge transfer model with parameters consistent with the spatial knowledge extraction model to obtain a first hyperspectral image M with enhanced spatial features; input the second spatial resolution multispectral image Y' into a spectral knowledge transfer model with parameters consistent with the spectral knowledge extraction model to obtain a second hyperspectral image N with enhanced spectral features; S5: Perform spatial spectral feature fusion and image decoding on the first hyperspectral image M and the second hyperspectral image N to obtain a hyperspectral image. .
[0008] Further, S2 includes: performing spectral downsampling on the hyperspectral image X using a downsampling model, blurring and spatially downsampling the hyperspectral image Y; mapping the two downsampling results to the same shared space Q, thereby obtaining the first multispectral image Q1 and the second multispectral image Q2; The shared space Q is used to uniformly map the two downsampling results into a multispectral image with a first spatial resolution. Then, a spatial spectral feature distribution consistency constraint is introduced to achieve feature alignment of multispectral images under different modalities in the shared space.
[0009] Furthermore, the loss function of the downsampling model Represented as: ; in, Indicates spectral degradation loss. Represents structural similarity loss. Indicates loss of color consistency. Indicates the first equilibrium parameter. This is the second equilibrium parameter.
[0010] Furthermore, during training, the spatial knowledge extraction model takes the set corresponding to the first multispectral image Q1 as input, the set corresponding to the second spatial resolution multispectral image Y as supervision, and the set corresponding to the second spatial resolution multispectral image Y' as output, and trains until the model converges.
[0011] Furthermore, during training, the spectral knowledge extraction model takes the set corresponding to the second multispectral image Q2 as input, the set corresponding to the first spatial resolution hyperspectral image X as supervision, and the set corresponding to the first spatial resolution hyperspectral image X' as output, and trains until the model converges.
[0012] Furthermore, the training process of the spatial knowledge extraction model includes: An image super-resolution network is used as a teacher model, and the first multispectral image Q1 is input into the teacher model. The second spatial resolution multispectral image Y is used as a supervision signal to fine-tune the teacher model so that it adapts to the distribution characteristics of the current data. A student model is constructed using a multi-scale pixel feature aggregation network, and the parameters of the student model are initialized. The first multispectral image Q1 is input into the fine-tuned teacher model and the initialized student model respectively, and the intermediate feature layer output of the teacher model and the intermediate feature layer output of the student model are obtained accordingly. Based on the output of the intermediate feature layer of the teacher model, the similarity matrix between feature points of the teacher model is calculated, and a teacher manifold relationship graph is constructed; based on the output of the intermediate feature layer of the student model, the similarity matrix between feature points of the student model is calculated, and a student manifold relationship graph is constructed. The difference between the teacher manifold graph and the student manifold graph is calculated to obtain the manifold distillation loss; the parameters of the student model are updated by backpropagation using the manifold distillation loss, constraining the student model to imitate the spatial structure knowledge of the teacher model; until the student model converges, the converged student model is used as the spatial knowledge extraction model.
[0013] Furthermore, the multi-scale pixel feature aggregation network is used to: perform channel adjustment on the input image and extract shallow spatial features. ; to the shallow spatial features Multi-scale aggregated features are obtained by sequentially using a multi-pixel feature aggregation model. ; for the multi-scale aggregation features By resetting the image pixels, a spatial super-resolution image is obtained; The pixel feature aggregation model is used to: divide the input features into image blocks in the spatial dimension, with the feature block size gradually increasing with the model depth, and extract multi-scale local spatial information; calculate the feature similarity between feature blocks, select the most topologically relevant source feature blocks for each target feature block to construct a sparse cross-attention matrix, and use cross-attention to perform non-local information interaction between the corresponding target feature block and its corresponding source feature blocks to obtain non-global context features. Self-attention computation is performed within each feature block to extract high-frequency detail features in the local space. High-frequency detail features Non-global context features Adaptive weighted fusion is performed to generate multi-scale aggregated features. .
[0014] According to another aspect of this application, an imaging apparatus for hyperspectral images is provided, comprising: The acquisition module is used to simultaneously acquire a hyperspectral image X with a first spatial resolution and a multispectral image Y with a second spatial resolution; the first spatial resolution is lower than the second spatial resolution. The sampling module is used to downsample the hyperspectral image X and the multispectral image Y respectively to obtain the first multispectral image Q1 and the second multispectral image Q2 with the first spatial resolution. The extraction module is used to input the second multispectral image Q2 into the spectral knowledge extraction model for spectral knowledge decoupling extraction to obtain a hyperspectral image X' with a first spatial resolution; and to input the first multispectral image Q1 into the spatial knowledge extraction model for spatial knowledge decoupling extraction to obtain a multispectral image Y' with a second spatial resolution. The enhancement module is used to input the hyperspectral image X' with the first spatial resolution into a spatial knowledge transfer model with parameters consistent with the spatial knowledge extraction model to obtain a first hyperspectral image M with enhanced spatial features; and to input the multispectral image Y' with the second spatial resolution into a spectral knowledge transfer model with parameters consistent with the spectral knowledge extraction model to obtain a second hyperspectral image N with enhanced spectral features. The fusion module is used to perform spatial spectral feature fusion and image decoding on the first hyperspectral image M and the second hyperspectral image N to obtain a hyperspectral image. .
[0015] According to another aspect of this application, an imaging system for hyperspectral images is provided, comprising: A hyperspectral camera for capturing hyperspectral images X at first spatial resolution; Multispectral camera, used to capture multispectral images with second spatial resolution. A processor, connected to the hyperspectral camera and the multispectral camera, is used to execute a computer program to implement the steps of the hyperspectral image imaging method.
[0016] According to another aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the hyperspectral image imaging method.
[0017] In summary, compared with the prior art, the above-described technical solutions conceived in this application can achieve the following beneficial effects: (1) This application provides an imaging method for hyperspectral images, which performs spectral downsampling on hyperspectral image X and spatial downsampling on multispectral image Y; the purpose is to project onto a low-resolution multispectral space to reduce differences, thereby reducing the difficulty of subsequent knowledge extraction; subsequently, the second multispectral image Q2 is input into the spectral knowledge extraction model for spectral knowledge decoupling extraction, and the first multispectral image Q1 is input into the spatial knowledge extraction model for spatial knowledge decoupling extraction. Furthermore, the hyperspectral image X' and the multispectral image Y' are respectively input into the spatial knowledge transfer model and the spectral knowledge transfer model to realize the projection of spectral knowledge into the high spatial resolution domain and the injection of spatial knowledge into the hyperspectral domain; finally, the two hyperspectral images are reconstructed in complementary fusion. This application effectively overcomes the problem of feature interaction difficulties caused by large differences in spatial and spectral modes in traditional direct fusion methods by adopting the strategy of "decoupling first, then migrating, and then fusing". It uses hyperspectral data to provide accurate spectral fingerprints to correct spectral distortion in the spatial super-resolution process, and uses multispectral data to provide rich spatial textures to repair detail blurring in the spectral super-resolution process. Therefore, it finally obtains high-quality imaging with both high spatial clarity and high spectral fidelity.
[0018] (2) This scheme achieves independent decoupling and bidirectional transfer of spectral and spatial knowledge between low spatial resolution hyperspectral images and high spatial resolution multispectral images by constructing a shared space; by explicitly modeling and reducing the feature distribution differences between the two modes, it overcomes the problem of feature interaction difficulties caused by modal heterogeneity in traditional direct fusion strategies, thereby significantly restoring spatial high-frequency details while effectively maintaining spectral consistency, and finally obtaining a high-quality fused image with both high spatial resolution and high spectral fidelity.
[0019] (3) The loss function of the downsampling model described in this scheme Considering the synergistic relationship between spectral degradation consistency, spatial structure preservation, and joint spatial-spectral constraints, it is possible to simultaneously constrain the degradation consistency of spectral information and spatial structure information during downsampling, ultimately achieving the construction of spatial-spectral consistency for shared low-resolution multispectral data.
[0020] (4) During training, the spatial knowledge extraction model described in this scheme takes the set corresponding to the first multispectral image Q1 as input, the set corresponding to the second spatial resolution multispectral image Y as supervision, and the set corresponding to the second spatial resolution multispectral image Y' as output, and trains until the model converges. This design takes into account the mapping relationship between low-resolution shared spatial data and high-resolution spatial structure, and can use the spatial texture information in the high spatial resolution multispectral image to restore the spatial details of the low-resolution input, and finally realize the independent decoupling of spatial structure knowledge and spatial enhancement feature learning.
[0021] (5) During training, the spectral knowledge extraction model described in this scheme takes the set corresponding to the second multispectral image Q2 as input, the set corresponding to the hyperspectral image X as supervision, and the set corresponding to the hyperspectral image X' with the first spatial resolution as output, and trains until the model converges. This design, considering the spectral mapping relationship between low-resolution multispectral data and hyperspectral data, can recover continuous spectral details from limited band information, and ultimately achieve independent decoupling of fine spectral knowledge and reconstruction of spectral features.
[0022] (6) This scheme obtains a spatial knowledge extraction model based on a hierarchical pixel aggregation network architecture of knowledge distillation. This design takes into account the spatial structure knowledge transfer relationship between the teacher model and the student model, which can ensure high-quality spatial feature expression ability while reducing the number of model parameters and computational complexity, and finally realize the construction of a lightweight spatial knowledge extraction model.
[0023] (7) This scheme will incorporate the aforementioned shallow spatial features Multi-scale aggregated features are obtained by sequentially using a multi-pixel feature aggregation model. This design, taking into account the complementary relationship between spatial texture information at different scales, can simultaneously extract local high-frequency detail features and non-local global context features, ultimately achieving spatial structural detail enhancement and multi-scale spatial feature aggregation. Attached Figure Description
[0024] Figure 1 This is a flowchart of the steps of an imaging method for hyperspectral images according to an embodiment of this application.
[0025] Figure 2 This is a schematic diagram illustrating the degradation mechanism of a hyperspectral image imaging method in one embodiment of this application.
[0026] Figure 3 This is a schematic diagram of the overall framework of a hyperspectral image imaging method according to an embodiment of this application.
[0027] Figure 4 This is a schematic diagram of constructing a low-resolution multispectral image (shared space) in one embodiment of this application.
[0028] Figure 5 This is a schematic diagram of a lightweight multi-scale pixel feature aggregation network structure in one embodiment of this application.
[0029] Figure 6 This is a schematic diagram of the structure of an imaging system for hyperspectral images according to an embodiment of this application. Detailed Implementation
[0030] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description, in conjunction with the accompanying drawings and embodiments, further illustrates this application. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. Furthermore, the technical features involved in the various embodiments described below can be combined with each other as long as they do not conflict with each other.
[0031] Example 1 This embodiment provides an imaging method for hyperspectral images, such as... Figure 1 As shown, it includes: S1-S5. S1, Simultaneously acquire low spatial resolution hyperspectral images. and high spatial resolution multispectral images The first spatial resolution in this application is lower than the second spatial resolution. Hereinafter, the lower spatial resolution will be described as low spatial resolution, and the higher spatial resolution as high spatial resolution. It should be noted that the first spatial resolution generally ranges from meter-level to ten-meter-level spatial resolution, while the second spatial resolution generally ranges from sub-meter-level spatial resolution.
[0032] S2, For low spatial resolution hyperspectral images Low spatial resolution multispectral images are obtained by spectral downsampling. For high spatial resolution multispectral images Spatial blurring and downsampling are performed to obtain low spatial resolution multispectral images. S3. Input the second multispectral image Q2 into the spectral knowledge extraction model for spectral knowledge decoupling and extraction to obtain a low spatial resolution hyperspectral image X'; input the first multispectral image Q1 into the spatial knowledge extraction model for spatial knowledge decoupling and extraction to obtain a high spatial resolution multispectral image Y'. S4. Using the low spatial resolution hyperspectral image... The input spatial knowledge transfer model, which has the same structure and reuses the spatial knowledge extraction model in S3, yields a high spatial resolution hyperspectral image with enhanced spatial features. High spatial resolution multispectral images The input spectral knowledge transfer model, which has the same structure and reuses the spectral knowledge extraction model in S3, yields a high spatial resolution hyperspectral image with enhanced spectral features. S5. Achieve bidirectional transfer of spatial-spectral features through weighted reusability; using high spatial resolution hyperspectral images M and Input a complementary fusion model, perform spatial spectral feature fusion and image decoding, and output the final high spatial resolution hyperspectral image. .
[0033] In this embodiment, the spatial knowledge transfer model and the spectral knowledge transfer model in S4 perform weight freezing and parameter reuse on the spatial super-resolution model and the spectral super-resolution model trained in S3, and their structures are completely consistent with those in S3.
[0034] In S5 of this embodiment, the high spatial resolution hyperspectral image M and the hyperspectral image The complementary fusion model is used to perform spatial spectral feature fusion and image decoding. The complementary fusion model includes channel stitching operations and a fusion structure.
[0035] High spatial resolution multispectral images and low spatial resolution hyperspectral images From high spatial resolution hyperspectral images The degenerate observation model matrix is in the following form: ; ； This includes spatial fuzzy kernels and downsampling operations. In traditional observation models, spatial fuzzy kernels are generally modeled using anisotropic Gaussian fuzzy kernels, and downsampling operations are generally performed by sampling at equal intervals. This represents the spectral response function. Figure 2 High spatial resolution multispectral images obtained from observation models and low spatial resolution hyperspectral images To high spatial resolution hyperspectral images A schematic diagram of the degradation process. Spatial degradation operators for high spatial resolution hyperspectral images. Each band undergoes the same degree of blurring and downsampling, without disrupting the continuity of the spectral dimensions; the spectral degradation operator is effective for high spatial resolution hyperspectral images. Each pixel corresponds to several bands that are linearly combined, without loss of spatial detail. That is, spatial degradation and spectral degradation are independent and do not affect each other. Based on this consideration, the degradation process can be simulated for high spatial resolution multispectral images. and low spatial resolution hyperspectral images Spatial-spectral feature alignment is performed. In this embodiment, S1 involves simultaneously acquiring low-spatial-resolution hyperspectral images. and high spatial resolution multispectral images This provides a foundation for utilizing the fine spatial details of multispectral images as a priori. In this embodiment, S2 uses a neural network to simulate the differences in spectral response and spatial blurring effects during the imaging process. Figure 3 This is a schematic diagram of the overall framework of a hyperspectral image imaging method according to an embodiment of this application.
[0036] As an optional implementation, S2 includes: using a downsampling model to perform spectral downsampling on a low spatial resolution hyperspectral image X, and blurring and spatially downsampling on a high spatial resolution multispectral image Y; mapping the two downsampling results to the same shared space Q to obtain a first multispectral image Q1 and a second multispectral image Q2; the shared space Q is used to uniformly map the two downsampling results to a multispectral image with the first spatial resolution, and then introducing a spatial spectral feature distribution consistency constraint to achieve feature alignment of multispectral images under different modes in the shared space.
[0037] In this embodiment, the low spatial resolution hyperspectral image is targeted. The spectral downsampling process is simulated using convolutional layers; this is applied to high spatial resolution multispectral images. For the spatial fuzziness process, a parameterized matrix is used for fuzzy kernel estimation; the reduction in spatial resolution is uniformly handled using a downsampling operator.
[0038] Specifically, the spectral downsampling function is implemented through a 1x1 convolutional layer, whose weight matrix is configured to simulate the spectral response function of a camera, given an input low spatial resolution hyperspectral image. , Represents the number of spectral channels, and the weights of the convolutional layer are... , The number of multispectral bands is adjusted by... The size is adjustable to accommodate differences in spectral response between different sensors. Low spatial resolution multispectral images obtained through spectral downsampling. It is represented as: , Sigmoid(·) is a linear activation function.
[0039] Furthermore, spatial fuzzy kernel estimation is achieved by constructing a parameterized Gaussian kernel matrix, and downsampling uses a strided convolution operator with a step size greater than 1. The input high spatial resolution multispectral image is Parameterized fuzzy kernel matrix The low spatial resolution multispectral image obtained after spatial blurring and downsampling It is represented as: , It is the product of Hadama.
[0040] As an optional implementation, the downsampling model includes a spectral downsampling model and a spatial fuzzy kernel estimation model. The spectral downsampling model and the spatial fuzzy kernel estimation model are jointly trained, and the loss function, which incorporates a consistency constraint on the spatial-spectral feature distribution, is defined as: ; ; ; ; Indicates spectral degradation loss. The norm is set to 1 to ensure rapid network convergence; Using structural similarity loss to measure the structural consistency after degradation avoids the network learning an overly smooth Gaussian blur kernel, which would lead to structural distortion. This is the first balancing parameter, used to prevent the spectral downsampling model and the spatial fuzzy kernel estimation model from getting trapped in local optima. This indicates the loss of color consistency, preventing overfitting of the spectral downsampling model. This is the second balancing parameter, which gradually increases with each training epoch to avoid non-convergence in the early stages and color distortion in the later stages. Using the Hypern dataset as an example, low spatial resolution multispectral images are obtained through training. and like Figure 4 As shown.
[0041] As an optional implementation, the spatial knowledge extraction model is trained by using the set corresponding to the first multispectral image Q1 as input, the set corresponding to the high spatial resolution multispectral image Y as supervision, and the set corresponding to the high spatial resolution multispectral image Y' as output, until the model converges. Alternatively, the spectral knowledge extraction model is trained by using the set corresponding to the second multispectral image Q2 as input, the set corresponding to the low spatial resolution hyperspectral image X as supervision, and the set corresponding to the low spatial resolution hyperspectral image X' as output, until the model converges.
[0042] Specifically, using low-resolution multispectral images With low-resolution hyperspectral images Supervised pairs are constructed, and a spectral mapping function from the multispectral domain to the hyperspectral domain is learned through a spectral knowledge extraction model. Low-resolution multispectral images are utilized. With high-resolution multispectral images Supervised pairs are constructed, and spatial structure functions from low-resolution spatial domain to high-resolution spatial domain are learned through spatial knowledge extraction models.
[0043] As an optional implementation method, the training process of the spatial knowledge extraction model includes the following steps.
[0044] 1) Obtain the pre-trained image super-resolution network as the teacher model, and use the aforementioned low spatial resolution multispectral images... Input into the teacher model; 2) Utilizing high spatial resolution multispectral images As a supervisory signal, the teacher model is fine-tuned and trained to adapt it to the distribution characteristics of the current data; 3) Construct a lightweight multi-scale pixel feature aggregation network as a student model and initialize the parameters of the student model; 4) Convert low spatial resolution multispectral images Simultaneously, the inputs are fed into the fine-tuned teacher model and the initialized student model to obtain the intermediate feature layer outputs of the teacher model and the student model, respectively. 5) Based on the output of the intermediate feature layer of the teacher model, calculate the similarity matrix between the feature points of the teacher model and construct the teacher manifold relationship graph; 6) Based on the output of the intermediate feature layer of the student model, calculate the similarity matrix between feature points of the student model and construct the student manifold relationship graph; 7) Calculate the difference between the teacher manifold diagram and the student manifold diagram to obtain the manifold distillation loss; 8) Use the manifold relationship distillation loss backpropagation to update the parameters of the student model, constraining the student model to imitate the spatial structure knowledge of the teacher model; 9) Repeat the steps of feature extraction, manifold construction and parameter update until the student model converges, and obtain the final student model with spatial super-resolution capability.
[0045] As an optional implementation, the student model is used to: perform channel adjustment on the input first multispectral image Q1 and extract shallow spatial features. ; Shallow spatial features Multi-scale aggregated features are obtained by sequentially using a multi-pixel feature aggregation model. ; for multi-scale aggregation features The image pixels are reset to obtain a spatial super-resolution image.
[0046] A lightweight, multi-scale pixel feature aggregation network is constructed as the student model, and the network structure is as follows: Figure 5 As shown. First, the input low-resolution multispectral image... Perform channel adjustments and extract shallow spatial features The shallow feature extraction model uses 3 3. Convolution; Shallow Spatial Features Multi-scale aggregated features are obtained sequentially through an n-pixel feature aggregation model. Multi-scale aggregation features Spatial super-resolution images are obtained through an image pixel reset model. The image pixel reset model uses... 1. Convolution.
[0047] Furthermore, let's take the first pixel feature aggregation model as an example: For input features Image blocks are divided spatially, with the feature block size gradually increasing with model depth to extract multi-scale local spatial information. The formula for calculating the block size is: , The base block size (e.g., 4*4). The scaling factor (usually taken as the scaling factor) ), This represents the total network depth. The input features are divided into... A set of non-overlapping blocks Each block Subsequently, the feature blocks are calculated. The feature similarity between the features is used to select the most relevant topological structure for each target feature block. For each source feature block, a sparse cross-attention matrix is constructed. Attention is applied only when the target feature block corresponds to its source feature block. Nonlocal information interaction is achieved between individual source feature blocks using cross-attention, extracting nonlocal contextual features. .
[0048] Specifically, the formula for calculating feature similarity is: ; Feature mapping factor (·) Use a single-layer connection layer. For each target feature block, select the layer with the most relevant topological structure. Individual feature blocks: ; Construct a sparse cross-attention matrix, and only apply it to the target feature block and its corresponding... Nonlocal information interaction is achieved between individual source feature blocks using cross-attention, extracting nonlocal contextual features. : ; ; ; Subsequently, self-attention computation is performed within each local feature block to extract high-frequency detail features in the local space. .
[0049] Specifically, local window partitioning is performed, dividing each feature block into... Size of local window: ; Self-attention calculation is performed for each local window: ; ; Use positional coding to enhance high-frequency information: ; .
[0050] Subsequently, high-frequency detail features in local space With non-local context features Adaptive weighted fusion is performed to dynamically balance spatial detail recovery and spectral consistency preservation, generating the final multi-scale aggregated features. .
[0051] Specifically, the feature channels are first spliced and transformed, and the calculation formula is as follows: ; ; The weights are learned adaptively through global average pooling and fully connected layers, and the calculation formula is as follows: ; ; ; Where σ is the Sigmoid function and δ is the ReLU activation function.
[0052] Finally, dynamic weighting of non-local and local features is performed: .
[0053] Example 2 This embodiment provides an imaging device for hyperspectral images, including: an acquisition module, a sampling module, an extraction module, an enhancement module, and a fusion module.
[0054] The acquisition module is used to simultaneously acquire a hyperspectral image X with a first spatial resolution and a multispectral image Y with a second spatial resolution; the first spatial resolution is lower than the second spatial resolution.
[0055] The sampling module is used to downsample the hyperspectral image X and the multispectral image Y respectively, so as to obtain the first multispectral image Q1 and the second multispectral image Q2 with the first spatial resolution.
[0056] The extraction module is used to input the second multispectral image Q2 into the spectral knowledge extraction model for spectral knowledge decoupling extraction to obtain the first spatial resolution hyperspectral image X'; and to input the first multispectral image Q1 into the spatial knowledge extraction model for spatial knowledge decoupling extraction to obtain the second spatial resolution multispectral image Y'.
[0057] The enhancement module is used to input the first spatial resolution hyperspectral image X' into a spatial knowledge transfer model with parameters consistent with the spatial knowledge extraction model to obtain a first hyperspectral image M with enhanced spatial features; and to input the second spatial resolution multispectral image Y' into a spectral knowledge transfer model with parameters consistent with the spectral knowledge extraction model to obtain a second hyperspectral image N with enhanced spectral features.
[0058] The fusion module is used to perform spatial spectral feature fusion and image decoding on the first hyperspectral image M and the second hyperspectral image N to obtain a hyperspectral image. .
[0059] Example 5 This embodiment provides an imaging system for hyperspectral images, including: A hyperspectral camera for capturing hyperspectral images X at first spatial resolution; A multispectral camera for capturing multispectral images Y at a second spatial resolution; A processor, connected to a hyperspectral camera and a multispectral camera; used to execute computer programs to implement the steps of the above-described hyperspectral image imaging method.
[0060] Specifically, it may also include a beam splitter, a first eyepiece, a second eyepiece, filters, etc., such as Figure 6 As shown, the beam splitter separates external light into two parts: one part passes through a first eyepiece and is input to a hyperspectral camera, while the other part passes through a filter and a second eyepiece and is input to a multispectral camera. Both the hyperspectral and multispectral cameras are connected to a microprocessor. The microprocessor is programmed or configured to execute the steps of Example 1.
[0061] Example 4 This embodiment provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the hyperspectral image imaging method described above.
[0062] Specifically, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.
[0063] Example 5 This embodiment provides a computer program product, including a computer program or instructions, wherein when the computer program or instructions are executed by a processor, they implement the steps of the above-described hyperspectral image imaging method.
[0064] The computer program product or computer program includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions to implement the steps of the hyperspectral image imaging method described above.
[0065] The technical features of the embodiments described above can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. It should be noted that the terms "in one embodiment," "for example," and "as in another example" in this application are intended to illustrate the application and are not intended to limit the application.
[0066] The embodiments described above are merely examples of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application.
Claims
1. An imaging method for hyperspectral images, characterized in that, include: S1: Simultaneously acquire the hyperspectral image X with the first spatial resolution and the multispectral image Y with the second spatial resolution; The first spatial resolution is lower than the second spatial resolution; S2: Downsample the hyperspectral image X and the multispectral image Y respectively to obtain the first multispectral image Q1 and the second multispectral image Q2 with the first spatial resolution; S3: Input the second multispectral image Q2 into the spectral knowledge extraction model for spectral knowledge decoupling extraction to obtain a hyperspectral image X' with a first spatial resolution; input the first multispectral image Q1 into the spatial knowledge extraction model for spatial knowledge decoupling extraction to obtain a multispectral image Y' with a second spatial resolution; S4: Input the hyperspectral image X' with the first spatial resolution into a spatial knowledge transfer model with parameters consistent with the spatial knowledge extraction model to obtain a first hyperspectral image M with enhanced spatial features; The second spatial resolution multispectral image Y' is input into a spectral knowledge transfer model with parameters consistent with the spectral knowledge extraction model to obtain a second hyperspectral image N with enhanced spectral features. S5: Perform spatial spectral feature fusion and image decoding on the first hyperspectral image M and the second hyperspectral image N to obtain a hyperspectral image. .
2. The hyperspectral image imaging method as described in claim 1, characterized in that, S2 includes: The hyperspectral image X is spectrally downsampled using a downsampling model, and the multispectral image Y is blurred and spatially downsampled. The two downsampling results are mapped to the same shared space Q to obtain the first multispectral image Q1 and the second multispectral image Q2; The shared space Q is used to uniformly map the two downsampling results into a multispectral image with a first spatial resolution. Then, a spatial spectral feature distribution consistency constraint is introduced to achieve feature alignment of multispectral images under different modalities in the shared space.
3. The hyperspectral image imaging method as described in claim 2, characterized in that, The loss function of the downsampling model Represented as: ； in, Indicates spectral degradation loss, Represents structural similarity loss. Indicates loss of color consistency. Indicates the first equilibrium parameter. This is the second equilibrium parameter.
4. The hyperspectral image imaging method as described in claim 1, characterized in that, During training, the spatial knowledge extraction model takes the set corresponding to the first multispectral image Q1 as input, the set corresponding to the multispectral image Y as supervision, and the set corresponding to the multispectral image Y' with the second spatial resolution as output, and trains until the model converges.
5. The hyperspectral image imaging method as described in claim 4, characterized in that, During training, the spectral knowledge extraction model takes the set corresponding to the second multispectral image Q2 as input, the set corresponding to the hyperspectral image X as supervision, and the set corresponding to the hyperspectral image X' with the first spatial resolution as output, and trains until the model converges.
6. The hyperspectral image imaging method as described in claim 4, characterized in that, The training process of the spatial knowledge extraction model includes: An image super-resolution network is used as the teacher model, and the first multispectral image Q1 is input into the teacher model; the multispectral image Y is used as the supervision signal to fine-tune the teacher model so that it adapts to the distribution characteristics of the current data; a student model is constructed using a multi-scale pixel feature aggregation network, and the parameters of the student model are initialized. The first multispectral image Q1 is input into the fine-tuned teacher model and the initialized student model respectively, and the intermediate feature layer output of the teacher model and the intermediate feature layer output of the student model are obtained accordingly. Based on the output of the intermediate feature layer of the teacher model, the similarity matrix between feature points of the teacher model is calculated, and a teacher manifold relationship graph is constructed; based on the output of the intermediate feature layer of the student model, the similarity matrix between feature points of the student model is calculated, and a student manifold relationship graph is constructed. The difference between the teacher manifold graph and the student manifold graph is calculated to obtain the manifold distillation loss; the parameters of the student model are updated by backpropagation using the manifold distillation loss, constraining the student model to imitate the spatial structure knowledge of the teacher model; until the student model converges, the converged student model is used as the spatial knowledge extraction model.
7. The hyperspectral image imaging method as described in claim 6, characterized in that, The multi-scale pixel feature aggregation network is used for: Channel adjustment of the input image and extraction of shallow spatial features. ; The shallow spatial features Multi-scale aggregated features are obtained by sequentially using a multi-pixel feature aggregation model. ; For the multi-scale aggregation features By resetting the image pixels, a spatial super-resolution image is obtained; The pixel feature aggregation model is used to: divide the input features into image blocks in the spatial dimension, with the feature block size gradually increasing with the model depth, and extract multi-scale local spatial information; calculate the feature similarity between feature blocks, select the most topologically relevant source feature blocks for each target feature block to construct a sparse cross-attention matrix, and use cross-attention to perform non-local information interaction between the corresponding target feature block and its corresponding source feature blocks to obtain non-global context features. Self-attention computation is performed within each feature block to extract high-frequency detail features in the local space. High-frequency detail features Non-global context features Adaptive weighted fusion is performed to generate multi-scale aggregated features. .
8. An imaging device for hyperspectral images, characterized in that, include: The acquisition module is used to simultaneously acquire a hyperspectral image X with a first spatial resolution and a multispectral image Y with a second spatial resolution; The first spatial resolution is lower than the second spatial resolution; The sampling module is used to downsample the hyperspectral image X and the multispectral image Y respectively to obtain the first multispectral image Q1 and the second multispectral image Q2 with the first spatial resolution. The extraction module is used to input the second multispectral image Q2 into the spectral knowledge extraction model for spectral knowledge decoupling extraction to obtain a hyperspectral image X' with a first spatial resolution; and to input the first multispectral image Q1 into the spatial knowledge extraction model for spatial knowledge decoupling extraction to obtain a multispectral image Y' with a second spatial resolution. The enhancement module is used to input the hyperspectral image X' with the first spatial resolution into a spatial knowledge transfer model with parameters consistent with the spatial knowledge extraction model, so as to obtain a first hyperspectral image M with enhanced spatial features; The second spatial resolution multispectral image Y' is input into a spectral knowledge transfer model with parameters consistent with the spectral knowledge extraction model to obtain a second hyperspectral image N with enhanced spectral features. The fusion module is used to perform spatial spectral feature fusion and image decoding on the first hyperspectral image M and the second hyperspectral image N to obtain a hyperspectral image. .
9. An imaging system for hyperspectral images, characterized in that, include: A hyperspectral camera for capturing hyperspectral images X at first spatial resolution; Multispectral camera, used to capture multispectral images with second spatial resolution Y A processor, connected to the hyperspectral camera and the multispectral camera, for executing a computer program to implement the steps of the imaging method for the hyperspectral image according to any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the imaging method for the hyperspectral image according to any one of claims 1 to 7.