Hyperspectral and multispectral fusion reconstruction method, controller, device and medium
By constructing a spectral and spatial encoder, calculating the hybrid spectral response matrix, and introducing a shared latent space alignment mechanism, the problems of spectral drift and spatial artifacts in the fusion of hyperspectral and multispectral data are solved, achieving efficient cross-modal semantic alignment and stable fusion reconstruction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHUHAI ORBIT SATELLITE BIG DATA CO LTD
- Filing Date
- 2026-04-23
- Publication Date
- 2026-08-04
AI Technical Summary
Existing technologies suffer from problems such as spectral drift, spatial artifacts, cross-modal representation instability, and gradient conflicts in the fusion of hyperspectral and multispectral data, making it difficult to achieve a balance between spatial and spectral resolution.
By constructing a spectral encoder and a spatial encoder, a mixed spectral response matrix is calculated, a shared alignment mechanism for spectral latent space and spatial latent information is established, spatial information is modeled using an adaptive point spread function, and cross-modal semantic alignment is constrained by a consistency loss function.
Without relying on a strict degradation model, this study improves the fusion effect of hyperspectral and multispectral images, enhances cross-modal semantic alignment, reduces training instability caused by feature coupling, and constructs a well-structured and scalable fusion reconstruction framework.
Smart Images
Figure CN122089586B_ABST
Abstract
Description
Technical Field
[0001] This application relates to, but is not limited to, the field of spectral reconstruction technology, and particularly to a hyperspectral and multispectral fusion reconstruction method, controller, device, and medium. Background Technology
[0002] Hyperspectral data, with its continuous band design, exhibits unique advantages and value in the field of remote sensing. It has since been widely applied in areas such as agricultural monitoring, horticultural conservation, disaster monitoring, mineral exploration, vegetation classification, and change detection, demonstrating enormous application potential. However, due to hardware limitations, the spatial resolution of hyperspectral data is typically low, making it difficult to meet the needs of refined observation applications and severely restricting its application and promotion.
[0003] Multispectral imagery boasts high spatial resolution but suffers from limited spectral dimensions, making it difficult to precisely characterize the spectral properties of ground features. Fusion of multispectral and hyperspectral data can enhance spatial resolution while maintaining spectral consistency, becoming an effective means of addressing the challenge of balancing spatial and spectral resolution. However, current techniques typically employ multispectral imagery-assisted hyperspectral imagery reconstruction, primarily including degradation model-based fusion methods and deep learning-based end-to-end reconstruction methods. These methods still face the following challenges in real-world remote sensing applications:
[0004] (1) Idealized degenerate models or synthetic data result in insufficient generalization ability of the model under real hyperspectral and multispectral data conditions, and are prone to spectral drift or spatial artifacts.
[0005] (2) Features of different modalities are usually implicitly fused within the network, lacking explicit semantic consistency constraints, and cross-modal representations are unstable;
[0006] (3) Spatial information and spectral information are highly coupled, and gradient conflicts are likely to occur during network training, making it difficult to achieve stable convergence. Summary of the Invention
[0007] This application provides a method, controller, device, and medium for hyperspectral and multispectral fusion reconstruction, which can improve the fusion effect of spatial information and spectral information.
[0008] In a first aspect, embodiments of this application provide a hyperspectral and multispectral fusion reconstruction method, including:
[0009] Acquire raw hyperspectral data and raw multispectral data;
[0010] According to the preset super-resolution ratio, the same region of data is extracted from the original hyperspectral data and the original multispectral data to obtain the extracted data;
[0011] Resampling is performed on the extracted data to obtain resampled hyperspectral data and resampled multispectral data;
[0012] Calculate the mixed spectral response matrix based on the resampled hyperspectral data and the resampled multispectral data;
[0013] Based on the hybrid spectral response matrix, a spectral encoder is constructed to extract the spectral latent space information of the resampled hyperspectral data and the resampled multispectral data;
[0014] A spatial encoder is constructed to extract the spatial latent information of the resampled hyperspectral data and the resampled multispectral data;
[0015] A shared latent space alignment mechanism is constructed based on the spectral latent space information and the spatial latent information.
[0016] In some embodiments, the step of extracting data from the same region of the original hyperspectral data and the original multispectral data according to a preset super-resolution factor to obtain the extracted data includes:
[0017] Geometric fine correction is performed on the original hyperspectral data and the original multispectral data;
[0018] The original hyperspectral data and the original multispectral data after geometric correction are used to extract data from the same region according to the preset super-resolution factor to obtain the extracted data.
[0019] In some embodiments, the observation value of the a-th band in the resampled multispectral data is represented as:
[0020] ;
[0021] Where a is a positive integer, This represents the spectral response function of the a-th band in the resampled multispectral data. Represents continuous hyperspectral reflectance under discrete wavelength sampling conditions. Represented as:
[0022] ;
[0023] Where b is a positive integer, The number of bands in the resampled hyperspectral data. This represents the b-th discrete band value of the resampled hyperspectral data. Let the response function of the b-th band of the resampled hyperspectral data be:
[0024] ;
[0025] in The first resampled hyperspectral data represents the... The first band of the resampled multispectral data The contribution weight of each band.
[0026] In some embodiments, the hybrid spectral response matrix is represented as:
[0027] ;
[0028] in, This represents the weighted overlap integral of the spectral response function of hyperspectral and multispectral sensors in the wavelength dimension. The spectral response function of the multispectral sensor is represented by the function at the 1st... The summation result over each wavelength dimension; This indicates row division. It is a positive integer.
[0029] In some embodiments, constructing a spectral encoder based on the mixed spectral response matrix to extract the spectral latent space information of the resampled hyperspectral data and the resampled multispectral data includes:
[0030] The hybrid spectral response matrix is used as the base matrix, and a trainable perturbation matrix with consistent dimensions is constructed based on the base matrix;
[0031] Based on the fundamental matrix and the trainable perturbation matrix, the joint spectral response matrix is obtained;
[0032] Applying a nonlinear activation function to the joint spectral response matrix yields a nonnegative spectral response matrix:
[0033] The nonnegative spectral response matrix is normalized so that the sum of the weights corresponding to each multispectral band is 1, resulting in a normalized spectral mapping matrix:
[0034] Based on the normalized spectral mapping matrix, the resampled hyperspectral data is downsampled to obtain spectral downsampling results;
[0035] Based on the hybrid spectral response matrix, the Moore-Penrose pseudo-inverse is obtained to yield the initial spectral mapping inverse matrix;
[0036] A 1×1 convolution operator is initialized using the inverse matrix of the initial spectral mapping, and the resampled multispectral data is linearly mapped to obtain the residual mapping result.
[0037] The residual mapping result is input into the residual learning module to correct the mapping process from the resampled multispectral data to the resampled hyperspectral data;
[0038] By using another 1×1 convolution operator, the corrected features are mapped to the hyperspectral dimension to obtain the spectral upsampling result.
[0039] In some embodiments, constructing a spatial encoder to extract spatial latent information from the resampled hyperspectral data and the resampled multispectral data includes:
[0040] Based on the adaptive point spread function, a trainable PSF convolution operator is constructed to model the spatial degradation process of the imaging system in an end-to-end learning framework, and to perform spatial downsampling on the resampled hyperspectral data.
[0041] Based on the adaptive point spread function, a corresponding inverse point spread function module is constructed using the transposed convolution operator, and the inverse point spread function module is used to perform spatial upsampling on the resampled multispectral data.
[0042] In some embodiments, constructing a shared latent space alignment mechanism based on the spectral latent space information and the spatial latent information includes:
[0043] The spectral encoder and spatial encoder are used to map the resampled hyperspectral data and the resampled multispectral data to different spatial modes, and the mapping results are aligned with the original hyperspectral data and the original multispectral data respectively to construct spatial reconstruction loss and spectral reconstruction loss.
[0044] In the process of spatial mapping and spectral mapping, intermediate latent feature representations are extracted respectively, and the similarity between the two in the common feature space is constrained by the consistency loss function to obtain the latent space consistency loss;
[0045] The spatial reconstruction loss, the spectral reconstruction loss, and the latent space consistency loss are combined to construct a joint training objective function with mutual constraints.
[0046] The spatial network and the spectral network are jointly optimized end-to-end based on the joint training objective function.
[0047] Secondly, embodiments of this application also provide a controller, including: at least one processor and a memory for communicatively connecting to the at least one processor; the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the hyperspectral and multispectral fusion reconstruction method described in the first aspect embodiment.
[0048] Thirdly, embodiments of this application also provide an electronic device, including the controller described in the second aspect embodiment.
[0049] Fourthly, embodiments of this application also provide a computer-readable storage medium storing computer-executable instructions for performing the hyperspectral and multispectral fusion reconstruction method as described in the first aspect.
[0050] The hyperspectral and multispectral fusion reconstruction method, controller, electronic device, and medium according to the embodiments of this application have at least the following beneficial effects: they achieve fusion modeling of real hyperspectral and multispectral images without relying on strict degradation models; they improve cross-modal semantic alignment capabilities by explicitly constructing a shared latent space and applying consistency constraints; they reduce training instability caused by feature coupling by decoupling spatial and spectral modeling; and they construct a fusion reconstruction framework with a clear structure and strong scalability, which is convenient for engineering implementation and application promotion.
[0051] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description
[0052] Figure 1 This is a flowchart of the steps of a hyperspectral and multispectral fusion reconstruction method provided in one embodiment of this application;
[0053] Figure 2 This is a schematic diagram illustrating the execution process of a hyperspectral and multispectral fusion reconstruction method provided in another embodiment of this application;
[0054] Figure 3 This is a schematic diagram of raw multispectral data provided in another embodiment of this application;
[0055] Figure 4 This is a schematic diagram of the raw hyperspectral data provided in another embodiment of this application;
[0056] Figure 5 This is a schematic diagram of a spectral response function provided in another embodiment of this application;
[0057] Figure 6 This is a schematic diagram of a spectral downsampling encoder provided in another embodiment of this application;
[0058] Figure 7 This is a schematic diagram of a spectral upsampling encoder provided in another embodiment of this application;
[0059] Figure 8 This is a schematic diagram of a spatial downsampling encoder provided in another embodiment of this application;
[0060] Figure 9 This is a schematic diagram of a spatial upsampling encoder provided in another embodiment of this application;
[0061] Figure 10 This is a schematic diagram illustrating the principle of constructing a shared latent space alignment mechanism provided in another embodiment of this application;
[0062] Figure 11 This is a schematic diagram comparing the original hyperspectral data and the fusion result provided in another embodiment of this application;
[0063] Figure 12 This is a schematic diagram comparing the original multispectral data and the fusion result provided in another embodiment of this application;
[0064] Figure 13 This application provides another embodiment of the spectral reflectance characteristics of homonymous land features between hyperspectral data before and after fusion;
[0065] Figure 14 This is a schematic diagram of the controller provided in another embodiment of this application. Detailed Implementation
[0066] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.
[0067] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.
[0068] The terms "first," "second," "third," and "fourth," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.
[0069] In this invention, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.
[0070] This application proposes a hyperspectral and multispectral fusion reconstruction method, controller, electronic device, and medium. It achieves fusion modeling of real hyperspectral and multispectral images without relying on a strict degradation model. By explicitly constructing a shared latent space and applying consistency constraints, it improves cross-modal semantic alignment capabilities. Through spatial and spectral decoupling modeling, it reduces training instability caused by feature coupling. It constructs a clear and scalable fusion reconstruction framework, facilitating engineering implementation and application promotion.
[0071] The embodiments of this application will be further described below with reference to the accompanying drawings.
[0072] Firstly, embodiments of this application propose a hyperspectral and multispectral fusion reconstruction method, such as... Figure 1 and Figure 2 As shown, the method includes the following steps:
[0073] Step S100: Acquire raw hyperspectral data and raw multispectral data;
[0074] The raw hyperspectral data and raw multispectral data were acquired from satellite imagery of a specific satellite constellation. Multimodal data was obtained from the satellite imagery, and the raw hyperspectral data is represented as follows: The original multispectral data is represented as , The number of bands in the original hyperspectral data is represented by h, the number of spatial rows in the original hyperspectral data is represented by w, and the number of spatial columns in the original hyperspectral data is represented by w. Here, H represents the number of bands in the original multispectral data, W represents the number of spatial rows in the original multispectral data, and H represents the number of spatial columns in the original multispectral data. Figure 3 The image shown is an example of raw multispectral data for a certain region. Figure 4 The image shows exemplary raw hyperspectral data from the same region. Examples of specific parameters for the raw multispectral data and raw hyperspectral data are shown in Table 1.
[0075] Table 1
[0076]
[0077] Step S200: Extract data from the same region of the original hyperspectral data and the original multispectral data according to the preset super-resolution ratio to obtain the extracted data;
[0078] Since the spatial scale and geographical extent of the original hyperspectral data and the original multispectral data may differ, it is necessary to unify them to facilitate subsequent processing. Because the original hyperspectral data has high spatial resolution and many bands, while the original multispectral data has low spatial resolution and fewer bands, it is necessary to use super-resolution to magnify the low-resolution original multispectral data, generating an image with the same spatial resolution as the original hyperspectral data. This ensures consistency between the two, accurately corresponding to the same ground area, and facilitates subsequent analysis.
[0079] Step S300: Resample the extracted data to obtain resampled hyperspectral data and resampled multispectral data;
[0080] Resampled hyperspectral data is represented as Where, l represents the wavelength sampling point of the resampled hyperspectral data, and the resampled multispectral data is represented as... , where m represents the wavelength sampling point of the resampled multispectral data. Resampling the truncated data is mainly to ensure that the wavelength sampling points and sampling intervals of the resampled hyperspectral data and the resampled multispectral data are consistent through linear interpolation techniques. The wavelength sampling point B = max(l, m).
[0081] Step S400: Calculate the mixed spectral response matrix based on the resampled hyperspectral data and the resampled multispectral data;
[0082] After resampling is completed, the mixed spectral response matrix is calculated based on the spectral data obtained from the resampling, thereby establishing a physically consistent mapping relationship between the hyperspectral image and the multispectral image, and introducing a linear degradation model based on the sensor spectral response function (SRF).
[0083] Step S500: Based on the hybrid spectral response matrix, construct a spectral encoder to extract the spectral latent space information of the resampled hyperspectral data and the resampled multispectral data;
[0084] Step S600: Construct a spatial encoder to extract the spatial latent information of the resampled hyperspectral data and the resampled multispectral data;
[0085] By constructing a spectral encoder and a spatial encoder, spectral latent spatial information and spatial latent information are extracted respectively, making the spectral information and spatial information independent, which facilitates the subsequent establishment of a mechanism for the fusion of spectral information and spatial information.
[0086] Step S700: Construct a shared latent space alignment mechanism based on spectral latent space information and spatial latent information.
[0087] By constructing a shared latent space alignment mechanism, spectral and spatial information are fused, which greatly improves the spatial resolution of the fused hyperspectral data, and the spectral reflectance characteristics remain relatively stable, achieving the expected goal.
[0088] The hyperspectral and multispectral fusion reconstruction method according to the embodiments of this application realizes the fusion modeling of real hyperspectral and multispectral images without relying on a strict degradation model; it improves cross-modal semantic alignment capability by explicitly constructing a shared latent space and applying consistency constraints; it reduces training instability caused by feature coupling by decoupling spatial and spectral modeling; and it constructs a fusion reconstruction framework with a clear structure and strong scalability, which is convenient for engineering implementation and application promotion.
[0089] In some embodiments of this application, step S200 above, which involves extracting data from the same region of the original hyperspectral data and the original multispectral data according to a preset super-resolution factor, to obtain the extracted data, specifically includes the following two steps:
[0090] Step S210: Perform geometric correction on the original hyperspectral data and the original multispectral data;
[0091] Step S220: For the original hyperspectral data and original multispectral data after geometric correction, extract data from the same region according to the preset super-resolution factor to obtain the extracted data.
[0092] By performing geometrical fine correction on the raw hyperspectral and multispectral data, geometric distortions (such as image stretching, offset, and rotation) caused by factors like sensor attitude, Earth curvature, terrain undulation, and atmospheric refraction are eliminated. This accurately maps image pixels to the real geographic coordinate system, ensuring complete alignment of pixels of the same ground feature in the raw hyperspectral and multispectral data. Then, data from the same region is cropped from the geometrically corrected raw hyperspectral and multispectral data according to a preset super-resolution factor, resulting in perfectly corresponding spatial locations between the two datasets. It should be noted that the specific value of the super-resolution factor is set according to the actual situation; in this example, the super-resolution factor can be rounded down to [value missing]. .
[0093] In some embodiments of this application, when calculating the mixed spectral response matrix, the observed value of the a-th band in the resampled multispectral data is represented as a weighted integral of the continuous spectral reflectance in the wavelength domain:
[0094] (1)
[0095] Where a is a positive integer, Let represent the spectral response function of the a-th band in the resampled multispectral data. Represents continuous hyperspectral reflectance under discrete wavelength sampling conditions. This can be approximated as a linear combination of bands in the resampled hyperspectral data:
[0096] (2)
[0097] Where b is a positive integer, The number of bands for resampling hyperspectral data. This represents the b-th discrete band value of the resampled hyperspectral data. Let represent the response function of the b-th band of the resampled hyperspectral data. Substituting equation (2) into equation (1) and discretizing it, we can obtain the linear relationship between the multispectral observations and the hyperspectral band values:
[0098] (3)
[0099] in, This indicates the first resampled hyperspectral data. The first band of the resampled multispectral data The contribution weights of each band are determined. Then, the mixed spectral response matrix from resampled hyperspectral data to resampled multispectral data is calculated. Defined as:
[0100] (4)
[0101] in, This represents the weighted overlap integral of the spectral response function of hyperspectral and multispectral sensors in the wavelength dimension. The spectral response function of a multispectral sensor is represented at the 1st... The summation result over each wavelength dimension; This indicates row division, i.e., division by row into matrices. The Divide the row by the corresponding normalization factor , It is a positive integer. For example... Figure 4 The diagram shows the spectral response functions, where the dashed line represents the spectral response function of the multispectral sensor and the solid line represents the spectral response function of the hyperspectral sensor.
[0102] In some embodiments of this application, step S500 above—constructing a spectral encoder based on the hybrid spectral response matrix to extract the spectral latent space information of resampled hyperspectral data and resampled multispectral data—includes the following nine sub-steps:
[0103] Step S510: Use the mixed spectral response matrix as the base matrix, and construct a trainable perturbation matrix with consistent dimensions based on the base matrix;
[0104] Step S520: Obtain the joint spectral response matrix based on the fundamental matrix and the trainable perturbation matrix;
[0105] Step S530: Apply a nonlinear activation function to the joint spectral response matrix to obtain a nonnegative spectral response matrix:
[0106] Step S540: Normalize the non-negative spectral response matrix so that the sum of the weights corresponding to each multispectral band is 1, thus obtaining the normalized spectral mapping matrix:
[0107] Step S550: Based on the normalized spectral mapping matrix, downsample the resampled hyperspectral data to obtain the spectral downsampling result;
[0108] Step S560: Based on the mixed spectral response matrix, calculate the Moore-Penrose pseudo-inverse to obtain the initial spectral mapping inverse matrix;
[0109] Step S570: Initialize a 1×1 convolution operator using the inverse matrix of the initial spectral mapping, perform linear mapping on the resampled multispectral data, and obtain the residual mapping result;
[0110] Step S580: Input the residual mapping result into the residual learning module to correct the mapping process;
[0111] Step S590: Map the corrected features to the hyperspectral dimension using another 1×1 convolution operator to obtain the hyperspectral reconstruction result.
[0112] It should be noted that constructing a spectral encoder includes both spectral downsampling and spectral upsampling processes. The structure of the spectral downsampling encoder is as follows: Figure 6 As shown, resampled hyperspectral data is input into SpeDnet (Spectral Downsampling Network) to extract a latent space representation dominated by spectral information. First, the obtained mixed spectral response matrix... As the base matrix, and constructing a trainable perturbation matrix with consistent dimensions, it is defined as follows:
[0113] ;
[0114] The trainable perturbation matrix is used as a learnable parameter to compensate for response deviations caused by sensor drift, environmental changes, etc., during actual imaging. The joint spectral response matrix can then be defined as:
[0115] ;
[0116] For the joint spectral response matrix Apply a nonlinear activation function, and use The function yields the nonnegative spectral response matrix:
[0117] ;
[0118] This is to ensure that each spectral response coefficient satisfies the physical non-negativity constraint. Then, for... Normalization is performed so that the sum of the weights corresponding to each multispectral band is 1, resulting in the normalized spectral mapping matrix:
[0119] ;
[0120] This represents the contribution weight of the b-th band of the resampled hyperspectral data to the a-th multispectral band of the resampled multispectral data. Spectral downsampling can then be expressed as:
[0121] ;
[0122] This indicates resampling of hyperspectral data. This indicates the result obtained after resampling hyperspectral data and then performing spectral downsampling.
[0123] The structure of the spectral upsampling encoder is as follows Figure 7 As shown, resampled multispectral data is input into SpeUnet (Spectral Upsampling Network) to extract a latent space representation dominated by spectral information. Similarly, based on the spectral response matrix, the Moore-Penrose pseudoinverse is calculated to obtain the initial spectral mapping inverse matrix:
[0124] ;
[0125] Then use Initialize a 1×1 convolution operator to perform a linear mapping on the resampled multispectral data:
[0126] ;
[0127] in, This indicates resampled multispectral data. The result represents the linear mapping, and * indicates a 1×1 convolution operation, which performs channel mapping in the spectral dimension while preserving spatial information. Then, the residual mapping result is input into the residual learning module to further refine the multispectral-to-hyperspectral mapping process, thereby improving the ability to model complex spectral relationships. Finally, a second 1×1 convolution operator maps the residual-compensated features to the hyperspectral dimension, yielding the hyperspectral reconstruction result.
[0128] Furthermore, in some embodiments of this application, step S600 above: constructing a spatial encoder to extract spatial latent information from resampled hyperspectral data and resampled multispectral data, includes the following two steps:
[0129] Step S610: Based on the adaptive point spread function, a trainable PSF convolution operator is constructed to model the spatial degradation process of the imaging system in the end-to-end learning framework, and spatial downsampling is performed on the resampled hyperspectral data.
[0130] Step S620: Based on the adaptive point spread function, construct the corresponding inverse point spread function module using the transposed convolution operator, and use the inverse point spread function module to perform spatial upsampling on the resampled hyperspectral data.
[0131] Similarly, the construction of a spatial encoder also includes spatial downsampling and spatial upsampling processes. During spatial downsampling, the Spatial Downsampling Network (SpaDnet) is as follows: Figure 8 As shown, based on the adaptive point spread function (PSF), a trainable PSF convolution operator is constructed to model the spatial degradation process of the imaging system within an end-to-end learning framework, thereby obtaining low-resolution observation results that conform to the real imaging mechanism. This scheme does not require pre-defined PSF priors and can adaptively learn the spatial degradation characteristics of the real imaging system through a data-driven approach. It utilizes a depthwise separable convolutional structure to achieve consistent modeling of spatial degradation across channels, avoiding inter-channel interference. It can be seamlessly embedded into an end-to-end deep learning framework, improving the consistency between spatial downsampling results and real low-resolution observations. The degradation model is as follows:
[0132]
[0133] in, This represents the input high spatial resolution image. This indicates the output low spatial resolution image. This represents a two-dimensional convolution operation.
[0134] When performing spatial upsampling, the Spatial Upsampling Network (SpaUnet) is as follows: Figure 9 As shown, the obtained PSF is used as a priori condition, and the transposed convolution operator is used to construct the corresponding inverse point spread function module. The convolution kernel is obtained by flipping or transposing the PSF while keeping the parameters fixed. For each channel of the input low-resolution image, a transposed convolution operation is performed through the inverse point spread function module to achieve spatial scale expansion:
[0135] .
[0136] After constructing the spectral encoder and spatial encoder, a shared latent space alignment mechanism is built based on the spectral latent space information and the spatial latent information, which specifically includes the following four steps:
[0137] Step S710: Map resampled hyperspectral data and resampled multispectral data to different spatial modes using a spectral encoder and a spatial encoder, and align the mapping results with the original hyperspectral data and the original multispectral data respectively to construct hyperspectral reconstruction loss and multispectral reconstruction loss.
[0138] Step S720: In the process of spatial mapping and spectral mapping, intermediate latent feature representations are extracted respectively, and the similarity between the two in the common feature space is constrained by the consistency loss function to obtain the latent space consistency loss;
[0139] Step S730: Combine the hyperspectral reconstruction loss, multispectral reconstruction loss, and latent space consistency loss to construct a joint training objective function with mutual constraints;
[0140] Step S740: Perform end-to-end joint optimization of the spatial network and the spectral network based on the joint training objective function.
[0141] like Figure 10 As shown, a spectral and spatial mapping network is constructed to map hyperspectral and multispectral data to different spatial modes, aligning the reconstructed results with the corresponding original observations, and constructing a reconstruction loss:
[0142] ;
[0143] ;
[0144] in, Indicates the loss in hyperspectral reconstruction. This represents the directly reconstructed hyperspectral prediction value. This represents the actual hyperspectral observation values. Indicates direct reconstruction losses. This represents the hyperspectral prediction value derived from the closed-loop regression. This represents the closed-loop consistency loss in hyperspectral imaging. Indicates the multispectral reconstruction loss. This represents the multispectral prediction value obtained through direct dimensionality reduction. Represents true multispectral observations. This indicates the direct loss due to dimensionality reduction. This represents the multispectral prediction value derived from the closed-loop regression. This represents the closed-loop consistency loss in multispectral imaging.
[0145] In the spatial mapping and spectral mapping processes, intermediate latent feature representations are extracted respectively. and This is used to describe the common structural information of multimodal images, and the similarity between them in the common feature space is constrained by the consistency loss function:
[0146] ;
[0147] Finally, the hyperspectral reconstruction loss, multispectral reconstruction loss, and latent space consistency loss are jointly combined to construct a joint training objective function with mutual constraints:
[0148] ;
[0149] Based on the above joint loss, end-to-end joint optimization of the spatial and spectral networks is performed. This method achieves stable fusion of multimodal images under conditions without high-resolution labels by using spatial-spectral dual-network collaborative modeling and latent space consistency constraints, thereby improving the physical consistency and reconstruction accuracy of the fusion results. Figure 11-13 This demonstrates the fusion and reconstruction effect of this application on real data pairs. Figure 11 The comparison between the original hyperspectral data and the fusion result is shown. Figure 12 It shows a comparison between the original multispectral data and the fused results. Figure 13 The study demonstrated the spectral reflectance characteristics of homologous ground features in the hyperspectral data before and after fusion. The spatial resolution of the fused hyperspectral data was greatly improved, and the spectral reflectance characteristics remained relatively stable, achieving the expected goal.
[0150] Based on the hyperspectral and multispectral fusion reconstruction method proposed in this application, and addressing the problems of existing fusion methods such as reliance on idealized degradation models, insufficient cross-modal semantic alignment capabilities, and poor stability under real-world data conditions, this application constructs a hyperspectral-multispectral fusion reconstruction framework based on shared latent space alignment without relying on strict degradation assumptions. This method establishes spectral mixing relationships using the spectral response functions of real hyperspectral and multispectral sensors. It achieves decoupling modeling of spatial and spectral information by constructing spatial encoders and spectral encoders separately, and introduces explicit consistency constraints at the latent space level to achieve stable cross-modal semantic alignment. Simultaneously, it effectively alleviates the training instability problem caused by feature coupling through closed-loop reconstruction and multi-loss co-optimization. Experimental results show that this application has good reconstruction accuracy, stability, and interpretability under real remote sensing data conditions, and is suitable for hyperspectral-multispectral full-spectrum fusion reconstruction and related remote sensing applications.
[0151] Secondly, such as Figure 14 As shown, the present invention also provides a controller, comprising:
[0152] The processor 101 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.
[0153] The memory 102 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 102 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 102 and is called and executed by the processor 101 to execute the hyperspectral and multispectral fusion reconstruction method of the embodiments of this application.
[0154] Input / output interface 103 is used to implement information input and output;
[0155] The communication interface 104 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).
[0156] Bus 105 transmits information between various components of the device (e.g., processor 101, memory 102, input / output interface 103, and communication interface 104);
[0157] The processor 101, memory 102, input / output interface 103 and communication interface 104 are connected to each other within the device via bus 105.
[0158] Thirdly, embodiments of this application also provide an electronic device, including the controller described in the second aspect embodiment.
[0159] Fourthly, embodiments of this application also provide a storage medium, which is a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described hyperspectral and multispectral fusion reconstruction method.
[0160] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof. The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate, and may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0161] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically include computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.
[0162] The above provides a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.
Claims
1. A method for hyperspectral and multispectral fusion reconstruction, characterized in that, include: Acquire raw hyperspectral data and raw multispectral data; According to the preset super-resolution ratio, the same region of data is extracted from the original hyperspectral data and the original multispectral data to obtain the extracted data; Resampling is performed on the extracted data to obtain resampled hyperspectral data and resampled multispectral data; Calculate the mixed spectral response matrix based on the resampled hyperspectral data and the resampled multispectral data; Based on the hybrid spectral response matrix, a spectral encoder is constructed to extract the spectral latent space information of the resampled hyperspectral data and the resampled multispectral data; A spatial encoder is constructed to extract the spatial latent information of the resampled hyperspectral data and the resampled multispectral data; Based on the spectral latent space information and the spatial latent information, a shared latent space alignment mechanism is constructed; The step of extracting data from the same region of the original hyperspectral data and the original multispectral data according to a preset super-resolution factor to obtain the extracted data includes: Geometric fine correction is performed on the original hyperspectral data and the original multispectral data; The original hyperspectral data and the original multispectral data after geometric fine correction are used to extract data from the same region according to the preset super-resolution factor to obtain the extracted data. The step of constructing a spectral encoder based on the hybrid spectral response matrix to extract the spectral latent space information of the resampled hyperspectral data and the resampled multispectral data includes: The hybrid spectral response matrix is used as the base matrix, and a trainable perturbation matrix with consistent dimensions is constructed based on the base matrix; Based on the fundamental matrix and the trainable perturbation matrix, the joint spectral response matrix is obtained; Applying a nonlinear activation function to the joint spectral response matrix yields a nonnegative spectral response matrix: The nonnegative spectral response matrix is normalized so that the sum of the weights corresponding to each multispectral band is 1, resulting in a normalized spectral mapping matrix: Based on the normalized spectral mapping matrix, the resampled hyperspectral data is downsampled to obtain spectral downsampling results; Based on the hybrid spectral response matrix, the Moore-Penrose pseudo-inverse is obtained to yield the initial spectral mapping inverse matrix; A 1×1 convolution operator is initialized using the inverse matrix of the initial spectral mapping, and the resampled multispectral data is linearly mapped to obtain the residual mapping result. The residual mapping result is input into the residual learning module to correct the mapping process from the resampled multispectral data to the resampled hyperspectral data; By using another 1×1 convolution operator, the corrected features are mapped to the hyperspectral dimension to obtain the spectral upsampling result.
2. The hyperspectral and multispectral fusion reconstruction method according to claim 1, characterized in that, The observation value of the a-th band in the resampled multispectral data is represented as: ； Where a is a positive integer, This represents the spectral response function of the a-th band in the resampled multispectral data. Represents continuous hyperspectral reflectance under discrete wavelength sampling conditions. Represented as: ； Where b is a positive integer, The number of bands in the resampled hyperspectral data. This represents the b-th discrete band value of the resampled hyperspectral data. Let the response function of the b-th band of the resampled hyperspectral data be: ； in The first resampled hyperspectral data represents the... The first band of the resampled multispectral data The contribution weight of each band.
3. The hyperspectral and multispectral fusion reconstruction method according to claim 2, characterized in that, The mixed spectral response matrix is represented as follows: ； in, This represents the weighted overlap integral of the spectral response function of hyperspectral and multispectral sensors in the wavelength dimension. The spectral response function of the multispectral sensor is represented by the function at the 1st... The summation result over each wavelength dimension; This indicates row division. It is a positive integer.
4. The hyperspectral and multispectral fusion reconstruction method according to claim 1, characterized in that, The construction of the spatial encoder to extract the spatial latent information of the resampled hyperspectral data and the resampled multispectral data includes: Based on the adaptive point spread function, a trainable PSF convolution operator is constructed to model the spatial degradation process of the imaging system in an end-to-end learning framework, and to perform spatial downsampling on the resampled hyperspectral data. Based on the adaptive point spread function, a corresponding inverse point spread function module is constructed using the transposed convolution operator, and the inverse point spread function module is used to perform spatial upsampling on the resampled multispectral data.
5. The hyperspectral and multispectral fusion reconstruction method according to claim 1, characterized in that, The step of constructing a shared latent space alignment mechanism based on the spectral latent space information and the spatial latent information includes: The spectral encoder and spatial encoder are used to map the resampled hyperspectral data and the resampled multispectral data to different spatial modes, and the mapping results are aligned with the original hyperspectral data and the original multispectral data respectively to construct spatial reconstruction loss and spectral reconstruction loss. In the process of spatial mapping and spectral mapping, intermediate latent feature representations are extracted respectively, and the similarity between the two in the common feature space is constrained by the consistency loss function to obtain the latent space consistency loss; The spatial reconstruction loss, the spectral reconstruction loss, and the latent space consistency loss are combined to construct a joint training objective function with mutual constraints. The spatial network and the spectral network are jointly optimized end-to-end based on the joint training objective function.
6. A controller, characterized in that, include: It includes at least one processor and a memory for communicatively connecting with said at least one processor; The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the hyperspectral and multispectral fusion reconstruction method as described in any one of claims 1 to 5.
7. An electronic device, characterized in that, Includes the controller as described in claim 6.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the hyperspectral and multispectral fusion reconstruction method as described in any one of claims 1-5.