Hyperspectral image reconstruction method and system based on flow matching prior assistance

Abstract

The invention provides a hyperspectral image reconstruction method and system based on flow matching prior assistance, and the method comprises the steps: generating a low-dimensional condition vector based on compressed two-dimensional test data, and enabling the low-dimensional condition vector to be used for constraining a flow matching model process, and obtaining a hidden space vector; setting an initial iteration frequency and an initial signal vector, and determining a sensing matrix and measurement data; and based on the initial signal vector, the sensing matrix and the measurement data, performing loop iteration of N stages until a final reconstructed hyperspectral image is obtained, wherein the k-stage iteration specifically comprises two processes of generalized alternating projection and a de-noising device. And after N stages of iteration are completed, taking the hyperspectral image reconstruction estimation value of the Nth stage as a final reconstructed hyperspectral image. According to the method, the continuous probability modeling capability of stream matching is utilized, the spatial-spectral information recovery capability of the hyperspectral image is enhanced, and the problems of spectral distortion and spatial blur in a complex degraded scene in a traditional method are avoided.

Description

Technical Field

[0001] This application relates to the field of hyperspectral image reconstruction, and more specifically, to a hyperspectral image reconstruction method and system based on flow matching prior assistance. Background Technology

[0002] In the field of hyperspectral imaging, hyperspectral image reconstruction is a key technology aimed at recovering high-resolution, high-signal-to-noise-ratio hyperspectral images from low-dimensional observational data (such as compressed sensing measurements or low-resolution inputs). Traditional methods are mainly based on solving linear inverse problems or hand-designed priors (such as sparse representations, low-rank constraints, etc.), but these methods often rely on strong assumptions and have limited reconstruction performance in complex degradation scenarios. In recent years, deep learning methods (such as convolutional neural networks and Transformers) have shown great potential in hyperspectral reconstruction, but the generalization ability of the models in real-world scenarios remains limited due to the mismatch between training data and actual degradation patterns.

[0003] With the development of generative models, diffusion models have been introduced into hyperspectral reconstruction tasks due to their powerful data distribution modeling capabilities, enabling the generation of more refined spectral and spatial details. However, existing diffusion model-based methods typically rely on multi-step iterative denoising, resulting in high computational costs and difficulty in meeting real-time requirements. Furthermore, hyperspectral data exhibits extremely high inter-spectral correlations, and traditional methods are inefficient in modeling long-range dependencies, potentially leading to spectral distortion or spatial blurring in the reconstruction results.

[0004] A search revealed that Chinese patent application number 202510439917.X discloses a Transformer-based hyperspectral image reconstruction method. This method effectively preserves spectral consistency and spatial details during hyperspectral image reconstruction, improving reconstruction accuracy and exhibiting high computational efficiency. However, this method lacks a joint attention mechanism for spatial spectral data, failing to effectively aggregate global information; furthermore, it lacks the clean hyperspectral priors provided by flow matching, making the reconstruction results prone to blurring and distortion under complex degradation conditions. Summary of the Invention

[0005] In view of the deficiencies in the prior art, the purpose of this application is to provide a hyperspectral image reconstruction method and system based on flow matching prior assistance.

[0006] A first aspect of this application provides a hyperspectral image reconstruction method based on flow matching prior assistance, comprising:

[0007] Based on compressed two-dimensional test data, a low-dimensional condition vector is generated to constrain the flow matching model process and obtain the latent space vector.

[0008] Let the initial iteration number k = 1, and the initial signal vector be v.(0) And determine the perception matrix and measurement data;

[0009] Based on the initial signal vector v (0) The system uses the sensing matrix and measurement data to perform N stages of iterative iteration until the final reconstructed hyperspectral image is obtained.

[0010] The k-th iteration, 1≤k≤N, specifically includes:

[0011] Based on the signal vector v of the (k-1)th stage (k-1) The measurement data y and the perception matrix A are projected from the measurement space to the three-dimensional signal space through depthwise separable convolution, and the hyperspectral image reconstruction estimate x at the k-th stage is calculated. (k) ;

[0012] The hyperspectral image reconstruction estimate x (k) The latent space vector is input to the denoiser for progressive refinement and reconstruction, outputting the signal vector v of the k-th stage. (k) ;

[0013] After N iterations are completed, the hyperspectral image reconstruction estimate x(N) of the Nth stage is used as the final reconstructed hyperspectral image.

[0014] Optionally, the low-dimensional condition vector generated from the compressed two-dimensional test data is used to constrain the flow matching model process to obtain the latent space vector, including:

[0015] Use a residual fully connected layer (Resmlp) as the encoder LE;

[0016] Based on compressed two-dimensional test data, the encoder LE is used to generate a low-dimensional conditional vector c = LE(y);

[0017] Using the low-dimensional condition vector c as a constraint, a continuous normalized flow model is employed, and the transformation process is calibrated using an objective function. Learning the invertible mapping φ:R from the standard normal distribution N(0,I) to the latent space vector. d →R d , thus obtaining the latent space vector z.

[0018] Optionally, the encoder and the flow model are jointly optimized, and the loss function includes reconstruction error and flow matching error:

[0019] L total =λ1L F +λ2|x1-φ -1 (z)| 2 .

[0020] Optionally, the initial iteration number k = 1, and the initial signal vector is v.(0) And determine the perception matrix and measurement data;

[0021] Optionally, the signal vector v based on the (k-1)th stage (k-1) The measurement data y and the perception matrix A are projected from the measurement space to the three-dimensional signal space through depthwise separable convolution, and the hyperspectral image reconstruction estimate x at the k-th stage is calculated. (k)， ,include:

[0022] x (k) =v (k-1) +A T (AA T ) -1 (y-Av (k-1) )

[0023] Optionally, the denoiser is a U-shaped backbone network, comprising:

[0024] Encoder: Contains two-stage downsampling, each stage consisting of a Spectral-Spatial-PriorTransformer (SSP-Transformer) block, used to focus on spectral, spatial features and prior knowledge;

[0025] Decoder: Employs a symmetrical upsampling structure to progressively restore feature resolution;

[0026] Skip connections: connect the corresponding levels of the encoder and decoder to preserve low-frequency detail information and improve reconstruction fidelity.

[0027] Optionally, the Spectral-Spatial-PriorTransformer (SSP-Transformer) block:

[0028] Self-attention of the spatial dimension of the input features is calculated using spatial attention;

[0029] And cross-channel attention of the input features is calculated using spectral-prior attention;

[0030] The self-attention of the spatial dimension and the cross-channel attention are fused to obtain the output features.

[0031] Optionally, the Spectral-Spatial-PriorTransformer block includes decomposition units, spatial branches, spectral branches, prior branches, and fusion units;

[0032] The decomposition unit decomposes the input features into a first feature and a second feature;

[0033] The spatial branch calculates the self-attention of the spatial dimension of the first feature through spatial attention;

[0034] The spectral branch and the prior branch together compute the cross-channel attention of the second feature through spectral-prior attention;

[0035] The fusion unit fuses the self-attention of the spatial dimension and the cross-channel attention to obtain the output features.

[0036] Optionally, in the spatial branch, the first feature is preprocessed by the Norm layer; the preprocessed feature enters a structure composed of a series of Mblocks without batch normalization, and after processing by the Mblocks, the intermediate output of the spatial branch is obtained; subsequently, the intermediate output is fused with the information flowing from the spectral branch and the prior branch, and the fusion process is achieved by a 1x1 convolution operation.

[0037] The spectral branch employs an asymmetric, multi-scale, multi-head self-attention structure to process the second feature, resulting in the spectral branch feature, specifically:

[0038] The second feature is embedded using Q, K, and V to obtain the basic feature vector required for attention calculation.

[0039] The basic feature vector is compensated to generate intermediate variables for compensation;

[0040] The intermediate variables are downsampled and normalized to obtain the normalized result;

[0041] The normalization result is fused with the remaining intermediate variables to obtain the output spectral branching features;

[0042] The prior branch calculates the K and V values ​​based on the latent space vectors;

[0043] The attention weight is obtained by fusing the K value with the Q value in the spectral branch and performing a normalization operation.

[0044] The attention weights are fused with the V value of the prior branch and the mapping is adjusted to obtain the prior branch features.

[0045] A second aspect of this application provides a hyperspectral image reconstruction system based on flow matching prior assistance, comprising:

[0046] Stream matching module: Based on compressed two-dimensional test data, it generates low-dimensional condition vectors to constrain the stream matching model process and obtain latent space vectors;

[0047] Initial setup module: Let the initial iteration number k = 1, and the initial signal vector be v.(0) And determine the perception matrix and measurement data;

[0048] Loop module: based on the initial signal vector v (0) The system uses the sensing matrix and measurement data to perform N stages of iterative module iterations until the final reconstructed hyperspectral image is obtained.

[0049] The loop module in stage k, 1≤k≤N, specifically includes:

[0050] Generalized Alternating Projection (GC-GAP) submodule: Based on the signal vector v of the (k-1)th stage (k-1) The measurement data y and the perception matrix A are projected from the measurement space to the three-dimensional signal space through depthwise separable convolution, and the hyperspectral image reconstruction estimate x at the k-th stage is calculated. (k) ;

[0051] Denoiser module: Reconstructs the estimated value x from the hyperspectral image. (k) The latent space vector is input to the denoiser for progressive refinement and reconstruction, outputting the signal vector v of the k-th stage. (k) ;

[0052] Output module: After completing N stages of iteration, the hyperspectral image reconstruction estimate x(N) of the Nth stage is used as the final reconstructed hyperspectral image.

[0053] A third aspect of this application provides a terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to execute the hyperspectral image reconstruction method based on flow matching prior assistance, or to run the hyperspectral image reconstruction system based on flow matching prior assistance.

[0054] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, characterized in that, when executed by a processor, the program can be used to execute the aforementioned hyperspectral image reconstruction method based on flow matching prior assistance, or to run the aforementioned hyperspectral image reconstruction system based on flow matching prior assistance.

[0055] The hyperspectral image reconstruction method based on flow matching prior assistance provided in this application adopts a flow matching-assisted generation strategy. Leveraging the stable modeling capability of the flow matching method for high-dimensional spectral data distribution, it can accurately recover the spatial details and spectral features of hyperspectral images, effectively overcoming the blurring and distortion problems that traditional methods easily encounter under complex degradation conditions. Simultaneously, the flow matching method abandons the multi-step iterative calculation process of traditional diffusion models, significantly reducing inference time and enabling hyperspectral reconstruction to meet real-time processing requirements. It is particularly suitable for scenarios with stringent computational efficiency requirements, such as airborne and spaceborne remote sensing.

[0056] This application achieves high-precision and efficient hyperspectral image reconstruction, effectively solving the problems of low computational efficiency, spectral distortion, and insufficient generalization ability in existing technologies, and providing a better technical solution for multiple fields such as remote sensing monitoring, medical imaging, and industrial inspection.

[0057] Other technical effects resulting from the additional features will be further illustrated in the corresponding embodiments. Attached Figure Description

[0058] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0059] Figure 1 This is a flowchart illustrating a hyperspectral image reconstruction method based on flow matching prior assistance according to an exemplary embodiment;

[0060] Figure 2 This is a flowchart illustrating a one-stage training process for a hyperspectral image reconstruction method based on flow matching prior assistance, according to an exemplary embodiment.

[0061] Figure 3 This is a diagram illustrating the two-stage training and inference process of a hyperspectral image reconstruction method based on flow matching prior assistance, according to an exemplary embodiment.

[0062] Figure 4 This is a generalized alternating projection diagram illustrated according to an exemplary embodiment;

[0063] Figure 5 This is a structural diagram of a denoiser according to an exemplary embodiment;

[0064] Figure 6 This is a diagram illustrating the structure of an LE according to an exemplary embodiment;

[0065] Figure 7 This is a structural diagram of an SSP-Transformer according to an exemplary embodiment;

[0066] Figure 8 The following are structural branch diagrams of the SSP-Transformer according to an exemplary embodiment: (a) shows the spatial branch, and (b) shows the spectral-prior branch.

[0067] Figure 9 This is a structural diagram of a hyperspectral image reconstruction system based on flow matching prior assistance, according to an exemplary embodiment. Detailed Implementation

[0068] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all fall within the protection scope of the present application. Parts not described in detail in the following embodiments can be implemented using existing technology.

[0069] Traditional methods suffer from spectral distortion and spatial blurring in complex and degraded scenes. To address these issues, this application provides a hyperspectral image reconstruction method based on flow matching prior assistance.

[0070] Reference Figure 1 As shown in one embodiment of this application, a hyperspectral image reconstruction method based on flow matching prior assistance includes:

[0071] S100 generates low-dimensional condition vectors based on measurement data, which are used to constrain the flow matching model process and obtain latent space vectors.

[0072] S200, let the initial iteration number k = 1, and the initial signal vector be v. (0) And determine the perception matrix and measurement data;

[0073] S300, based on the initial signal vector v (0) The system uses the sensing matrix and measurement data to perform N stages of iterative iteration until the final reconstructed hyperspectral image is obtained.

[0074] The k-th iteration, 1≤k≤N, specifically includes:

[0075] S301, based on the signal vector v of the (k-1)th stage (k-1) The measurement data y and the perception matrix A are projected from the measurement space to the three-dimensional signal space through depthwise separable convolution, and the hyperspectral image reconstruction estimate x at the k-th stage is calculated. (k) ;

[0076] S302, reconstruct the estimated value x from the hyperspectral image. (k) The latent space vector is input to the denoiser for progressive refinement and reconstruction, outputting the signal vector v of the k-th stage. (k) ;

[0077] S400: After completing N stages of iteration, the hyperspectral image reconstruction estimate x(N) of the Nth stage is used as the final reconstructed hyperspectral image.

[0078] The embodiments described above in this application can significantly improve computational efficiency while ensuring reconstruction accuracy, providing a better solution for real-time applications of hyperspectral imaging in fields such as remote sensing, medicine, and industrial inspection.

[0079] To ensure both efficiency and quality, in some specific embodiments of this application, step S100, generating a low-dimensional condition vector based on compressed two-dimensional test data to constrain the flow matching model process and obtain the latent space vector, can be achieved through the following steps:

[0080] S101 inputs the two-dimensional test data into the encoder LE. After processing by the encoder, the output is a low-dimensional condition vector c = LE(y) mapped to the data, which is used to constrain the flow matching model process.

[0081] Specifically, the structure of the hidden space encoder (LE) is as follows: Figure 6 As shown, it includes, in sequence, a pixel recombination layer, an Mblock module, an average pooling layer, a blending channel, an activation function layer, and a linear layer.

[0082] S102 employs a Continuous Normalizing Flow (CNF) model to learn the invertible mapping φ:R from the standard normal distribution N(0,I) to the latent space vector. d →R d .

[0083] Stream matching objective function:

[0084] Where v t For the velocity field, x t For the interpolation path of time t∈[0,1].

[0085] Using the continuous normalized flow model described above, the latent space vector z is obtained based on the low-dimensional conditional vector obtained from S101. This latent space vector z is then passed to the denoising module as prior knowledge to guide the denoising process.

[0086] Specifically, the flow matching in this application is based on a generative model of Continuous Normalizing Flow (CNF), which has stable training characteristics. This is mainly reflected in:

[0087] Deterministic path learning:

[0088] Unlike diffusion models, which gradually denoise through random noise, flow matching directly learns the deterministic differential equation path ODE from the noise distribution to the target distribution, thus avoiding the gradient instability problem introduced by multi-step random noise in the training of diffusion models.

[0089] Gradient smoothness: Flow matching defines a continuous transformation of the data distribution by fitting a vector field. Its loss function (i.e., flow matching loss) usually has a smoother gradient, reducing the risk of gradient explosion / vanishing during training.

[0090] Implicit probability path: Flow matching can flexibly choose the probability path from noise to data (such as linear interpolation), while the diffusion model is fixed to a Gaussian noise path. This flexibility makes training easier to converge.

[0091] In this application, hyperspectral images (HSI) typically have a low signal-to-noise ratio, and traditional diffusion models are prone to spectral distortion during denoising due to the accumulation of randomness. Flow matching's deterministic path reduces random interference and improves training stability.

[0092] Specifically, the flow matching in this application is based on a generative model of Continuous Normalizing Flow (CNF), which has the advantage of continuous probabilistic modeling. Continuous probabilistic modeling refers to modeling the data distribution continuously through differential equations (such as ODE / SDE), and its advantages include:

[0093] High-resolution modeling capability: Continuous streams can accurately describe the distribution of data at any time step (such as intermediate states in super-resolution reconstruction), avoiding the quantization errors of discrete models (such as VAEs).

[0094] Reversibility: Flow-based models are inherently reversible, supporting bidirectional mapping from the latent space to the data space, making them suitable for tasks requiring accurate reconstruction, such as medical images.

[0095] Implicit likelihood calculation: It does not require explicit definition of the likelihood function and directly models the data distribution through path integral. It is suitable for complex high-dimensional data, such as hyperspectral cubes.

[0096] In this application, the spectral channels of hyperspectral images exhibit strong continuity, such as the gradation from near-infrared to infrared. Continuous probabilistic modeling of flow matching treats the spectral dimension as a continuous variable, capturing the gradation patterns between bands through ODE, which is superior to the discrete processing of CNN / Transformer.

[0097] If the latent space generation module is removed (i.e., the latent space vector z is not generated from the measurement data y for auxiliary reconstruction via flow matching), the entire reconstruction process will face the following problems:

[0098] Blind reconstruction lacking physical perception: Subsequent reconstruction modules can only rely on the original measurement data y and the perception matrix A for inversion, and cannot utilize the spectral structure priors encoded by the latent space vector z (such as the distribution of ground reflectance and the correlation between bands), which may lead to spectral distortion, spatial blurring, and reconstruction is prone to getting trapped in local optima (such as excessive smoothing of edges or texture details).

[0099] Decreased computational efficiency: The latent space vector z provides strong constraints for subsequent iterations; if it is missing, the number of iterations (such as more stages of S300 loops) or a more complex network structure is required to compensate, which slows down real-time performance.

[0100] The diffusion model requires stepwise denoising from pure noise, typically taking 100 to 1000 steps, each step being a stochastic process dependent on the complete model computation; while flow matching directly generates z from y without iteration. The deterministic ODE path of flow matching allows for solution in a single step or a very small number of steps (usually 1 to 20 steps), with a significantly lower computational cost than the diffusion model.

[0101] Therefore, the above embodiments of this application utilize the continuous probability modeling capability of flow matching to enhance the spatial-spectral information recovery capability of hyperspectral images, avoiding the spectral distortion and spatial blurring problems of traditional methods in complex degradation scenarios.

[0102] The embodiments described above in this application reduce the multi-step iterative computation of traditional diffusion models by using a flow matching prior-assisted generation strategy. Flow matching learns the direct mapping from noise to data by directly modeling deterministic differential equations (ODEs). Generation can be completed in one or a few steps by solving the ODE, without the need for iterative denoising. Furthermore, when using a numerical ODE solver (such as Runge-Kutta), the number of steps can be flexibly controlled by adjusting the tolerance, which significantly reduces the inference time and makes hyperspectral reconstruction more suitable for real-time or resource-constrained application scenarios, such as airborne / spaceborne remote sensing and portable spectroscopic equipment.

[0103] Compared with traditional diffusion models, the flow matching method described in the above embodiments of this application has more stable training characteristics, can reduce the dependence on large-scale labeled data, and avoids the pattern collapse problem in GAN (Generative Adversarial Network) training.

[0104] To obtain an encoder with more accurate parameters, in some specific embodiments of this application, the encoder and the flow model are jointly optimized, and the loss function includes reconstruction error and flow matching error:

[0105] L total =λ1L F +λ2|x1-φ -1 (z)| 2 .

[0106] L FLet x1 be the target hyperspectral image, z represent the prior generated by the flow matching, and φ be the flow matching loss. -1 This represents the subsequent reconstruction model, where λ1 and λ2 are weights.

[0107] The embodiments described above in this application, through joint optimization of reconstruction error and flow matching error, form a closed loop of bidirectional promotion between the encoder and the flow model: the encoder provides the flow model with a potential distribution reference of real data, while the flow model provides the encoder with regular distribution constraints to improve representation quality. Ultimately, this achieves a comprehensive improvement in the model's reconstruction accuracy, distribution modeling rationality, generalization ability, and functional scalability, making it particularly suitable for complex data scenarios such as hyperspectral images that have high requirements for both detail fidelity and distribution modeling.

[0108] Before entering the loop iteration, an initial loop input vector needs to be set. In some specific embodiments of this application, in S200, the initial iteration number k = 1, and the initial signal vector is v. (0) To determine the sensing matrix and measurement data, the following steps can be taken:

[0109] Set the number of iterations k = 1, and the total number of iterations to N;

[0110] Let the signal vector be v (0) The initial signal is normalized two-dimensional measurement data y, which is processed into a three-dimensional signal v by channel-by-channel replication. (0) ;

[0111] Set the perception matrix, perception matrix A = [D1,...,Dλ], That is, D is a combination of A diagonal matrix composed of the elements in the matrix. M is a two-dimensional mask;

[0112] Set the measurement data, that is, the two-dimensional measurement data y.

[0113] In each loop phase, in order to project the measurement onto the three-dimensional signal space for each input signal vector, some specific embodiments of this application, such as... Figure 4 As shown, S301, based on the signal vector v of the (k-1)th stage (k-1) The measurement data y and the perception matrix A are projected from the measurement space to the three-dimensional signal space through depthwise separable convolution, and the hyperspectral image reconstruction estimate x at the k-th stage is calculated. (k) Its implementation formula is expressed as:

[0114] x (k) =v (k-1)+A T (AA T ) -1 (y-Av (k-1) )

[0115] Where A represents the perception matrix, y represents the measurement data, and v (k-1) This represents the signal vector in the (k-1)th stage.

[0116] The embodiments described above in this application iteratively solve the optimization problem using generalized alternating projection. By introducing an auxiliary variable v and a deep network (such as depthwise separable convolution), the traditional iterative optimization problem is transformed into a learnable, lightweight module, significantly reducing the number of iterations. This solves the following technical problems: the underdeterminacy of ill-conditioned inverse problems (the perception matrix A is typically a wide matrix with an observation dimension much smaller than the signal dimension), leading to infinitely many solutions when directly solving for x; and the inefficiency of traditional optimization methods (traditional model-driven methods, such as ADMM, require hundreds of iterations to converge).

[0117] It is worth noting that without the alternating projection mechanism, and without gradually approximating the feasible solution through alternating projection, the traditional method requires a large number of iterations and is inefficient. Without the projection error correction in the alternating projection method, and without the use of a multi-stage denoising network to ensure spectral continuity, the reconstruction accuracy will also decrease.

[0118] To achieve better spatial-spectral information recovery capabilities for hyperspectral images, in some specific embodiments of this application, the denoiser in S302 employs a U-shaped backbone network, such as... Figure 5 As shown, it includes:

[0119] Encoder: Contains two-stage downsampling, each stage consisting of a Spectral-Spatial-PriorTransformer (SSP-Transformer) block, used to focus on spectral, spatial features and prior knowledge;

[0120] Decoder: Employs a symmetrical upsampling structure to progressively restore feature resolution;

[0121] Skip connections: connect the corresponding levels of the encoder and decoder to preserve low-frequency detail information and improve reconstruction fidelity.

[0122] The input to the denoiser in the k-th stage is the hyperspectral image reconstruction estimate x from the k-th stage. (k) The output is a signal vector v k A noise denoiser is used to denoise x. (k) A process of reprojecting back into the target signal space.

[0123] Among them, the Spectral-Spatial-PriorTransformer (SSP-Transformer) block, such as Figure 7 As shown, the architecture includes a pruning unit, a spatial branch, a spectral branch, a prior branch, and a fusion unit. The pruning unit divides the input features into a first feature and a second feature. The spatial branch, spectral branch, and prior branch utilize a collaborative attention mechanism. The spatial branch calculates the self-attention of the first feature in the spatial dimension using spatial attention, focusing on local structure. The spectral branch and prior branch use spectral-prior attention based on the second feature and the latent space vector to calculate cross-channel attention. The structures of the spatial branch and the spectral-prior branch are as follows: Figure 8 As shown.

[0124] Specifically, such as Figure 7 As shown, U i This refers to the input three-dimensional features. i represents the i-th scale, because the denoiser uses a U-shaped structure, and different Transformers have different scale inputs.

[0125] This involves dividing the UI into two parts based on channels, and then passing each part into the spatial branch. And the spectral-prior branch These are respectively called the first feature Second feature

[0126] In the spatial branch, firstly, the first feature The features are first preprocessed through a Norm layer. The preprocessed features are then fed into a structure consisting of a series of Mblocks without batch normalization. After processing by the Mblocks, the intermediate output of the spatial branch is obtained. Subsequently, the intermediate output is fused with the information flowing from the spectral branch and the prior branch. The fusion process is achieved through a 1x1 convolution operation.

[0127] The spectral branch employs asymmetric multi-scale self-attention (ACS-MHSA), and its execution process is as follows:

[0128] First, the spectral branch performs embedding calculations on the input information using Q (Query), K (Key), and V (Value), all of which are based on the second feature of the model input. (The subscript i represents the corresponding parameter in the i-th scale Transformer) is generated by the corresponding convolution weight W*, thereby obtaining the basic feature vector required for attention calculation.

[0129] Next, this branch will perform compensation-related processing, generating intermediate variables for compensation, including P obtained from the spatial branch.S (such as P that compensates for V) SV ) and P that compensates for Q and K SQK In addition, intermediate variables M will be defined, which will be used for subsequent adjustment of attention weights and feature optimization.

[0130] Finally, we proceed to the core computational stage of asymmetric cross-scale multi-head self-attention: first, relevant features are processed through downsampling (symbol ↓), then the attention score is normalized using the SoftMax function (divided by parameter α during calculation to adjust the scale), and finally the normalized result is compared with the compensation variable (P). SV ), weight matrix (W) CS The superscript CS indicates CrossSpectral (i.e., spectral branch) and related characteristics of V (V CS The fusion operation is performed to obtain the output of the spectral branch after ACS-MHSA processing, thus completing the feature extraction and optimization process of the entire spectral branch.

[0131] The above process can be represented by the following formula:

[0132] The embedding formulas for Q, K, and V are as follows:

[0133]

[0134] Where W* represents the weights of the corresponding convolution, and Q, K, and V correspond to the Query, Key, and Value in the attention mechanisms of each branch, respectively. The superscript CS represents the Cross Spectral branch, and CP represents the Cross Prior branch. The subscript i represents the parameters in the i-th scale of the Transformer. M represents intermediate variables. U represents the model input. P S P represents the spatial compensation obtained from the spatial branch of the spectral branch. SQK P represents the compensation for Q and K. SV This indicates compensation for V, and ↓ indicates a downsampling operation.

[0135] The entire asymmetric cross-scale self-attention mechanism is as follows:

[0136]

[0137] Where α is the adjustment scale. This represents the learnable weights in the spectral branch.

[0138] The prior branch is a variable multi-head cross-attention module. The query Q in the branch comes from the V information extracted from the spectral branch, which is extracted from a larger receptive field and has more spatial information.

[0139] In the prior branch, firstly, the core elements Q (Query), K (Key), and V (Value) required for the prior branch attention calculation are determined. The query Q originates from the V information extracted from the spectral branch and is related to the Q of the spectral branch. i cs (The superscript CS stands for Cross Spectral) is completely consistent. This Q information is obtained from a larger receptive field and contains richer spatial information. The key values ​​K and V are generated by mapping the prior basic input z (z represents the prior) through the corresponding weight matrix. The subscript i represents the corresponding parameter in the Transformer at the i-th scale.

[0140] Next, we move on to the core computational stage of multi-head cross-attention. First, the generated K and Q are fused and divided by the parameter α to adjust the scale of the attention score. Then, the SoftMax function is used to normalize the fused result to obtain the normalized attention weights.

[0141] Finally, the normalized attention weights are fused with V again and adjusted by mapping through the weight matrix to obtain the output features of the prior branch. These features can be used for subsequent information interaction with other branches (such as the spatial branch) to provide prior dimension feature support for hyperspectral image reconstruction.

[0142] The above process for handling prior branches can be expressed by the following formula:

[0143]

[0144] In the subscripts, i represents the parameter corresponding to the i-th scale of the Transformer; z represents the prior; QKV correspond to the Query, Key, and Value in each branch of the attention mechanism, respectively; α represents the adjustment parameter; U represents the input feature; W zK W zV W c1 This represents the weight value.

[0145] After being processed by the spatial branch, spectral branch, and prior branch respectively, the outputs of each branch are aggregated, then normalized using Norm, and finally passed through a fully-connected neural network (FNN) to obtain the final output.

[0146] The embodiments described above in this application effectively combine the de-degradation prior knowledge provided by flow matching to compensate for the degradation information provided by the measurement through the Transformer, which combines spatial-spectral-prior information attention mechanism. This solves the problem that previous hyperspectral reconstruction work usually only considered the information and relationship between space and spectrum, but it is difficult to extract spatial-spectral relationships through compressed measurements alone.

[0147] To better train the model corresponding to the hyperspectral image reconstruction method based on flow matching prior assistance, some specific embodiments of this application adopt a two-stage training method:

[0148] Phase 1: As attached Figure 2 A clean hyperspectral image x and a compressed measurement y are input into LE to directly generate prior-aided denoising.

[0149] Specifically, by leveraging the stable and accurate characteristics of clean image data, the problem of large gradient variance caused by measurement data noise or sparsity during direct end-to-end training is effectively avoided. This helps the model to initially learn reliable feature representations and reconstruct patterns, while avoiding the generation of prior z that deviates from the true spectral distribution, reducing pattern collapse, and laying a solid foundation for model training.

[0150] Phase Two: As attached Figure 3 The prior is generated using a flow model, and only the input measurement y is used as the control condition to generate the prior to assist in denoising.

[0151] Specifically, this stage builds upon the learning outcomes of the first stage, further optimizing and adjusting the prior information based on the characteristics of the measurement data. On one hand, learning the prior information in stages allows the model to generate more comprehensive and accurate prior information that conforms to the true distribution of hyperspectral images, improving the quality and applicability of the prior information. On the other hand, adjusting the prior information generation based on actual measurement data makes the model more closely aligned with the characteristics of data in real-world application scenarios.

[0152] The embodiments described above employ a two-stage training process. First, prior knowledge is learned from a clean HSI (High-Speed ​​Interchange Indicator), and then prior knowledge is generated from the flow matching model. The prior knowledge provided by flow matching assists the multi-stage reconstruction process. This addresses the problems of flow matching models trained directly end-to-end potentially having large gradient variances due to noise or sparsity in the measurement data, making model convergence difficult; and the problem of the generated prior z deviating from the true spectral distribution (mode collapse).

[0153] Based on the same technical concept, other embodiments of this application provide a hyperspectral image reconstruction system 100 based on flow matching prior assistance, such as... Figure 9 As shown, it includes:

[0154] Flow matching module 110: Based on compressed two-dimensional test data, it generates a low-dimensional condition vector to constrain the flow matching model process and obtain the latent space vector;

[0155] Initial setup module 120: Set the initial iteration number k = 1, and the initial signal vector v. (0) And determine the perception matrix and measurement data;

[0156] Loop Module 130: Based on the initial signal vector v (0) The system uses the sensing matrix and measurement data to perform N stages of iterative module iterations until the final reconstructed hyperspectral image is obtained.

[0157] The loop module in stage k, 1≤k≤N, specifically includes:

[0158] Generalized Alternating Projection (GC-GAP) submodule: Based on the signal vector v of the (k-1)th stage (k-1) The measurement data y and the perception matrix A are projected from the measurement space to the three-dimensional signal space through depthwise separable convolution, and the hyperspectral image reconstruction estimate x at the k-th stage is calculated. (k) ;

[0159] The denoiser module reconstructs the estimated value x from the hyperspectral image. (k) The latent space vector is input to the denoiser for progressive refinement and reconstruction, outputting the signal vector v of the k-th stage. (k) ;

[0160] Output module 140: After completing N stage iterations, the hyperspectral image reconstruction estimate x(N) of the Nth stage is used as the final reconstructed hyperspectral image.

[0161] The specific implementation techniques of each module / unit in the above examples of this application can be referred to the steps of the hyperspectral image reconstruction method based on flow matching prior assistance in the above embodiments, and will not be repeated here.

[0162] Based on the same technical concept, in other embodiments of this application, a terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to execute the above-described hyperspectral image reconstruction method based on flow matching prior assistance, or to run the above-described hyperspectral image reconstruction system based on flow matching prior assistance.

[0163] Based on the same technical concept, in other embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored. When executed by a processor, the program can be used to perform the above-described hyperspectral image reconstruction method based on flow matching prior assistance, or to run the above-described hyperspectral image reconstruction system based on flow matching prior assistance.

[0164] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDRSDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs, functional modules, etc. that implement the above methods), computer instructions, etc., and the aforementioned computer programs, computer instructions, etc., can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0165] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0166] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0167] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0168] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0169] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0170] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0171] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0172] The foregoing has described some specific embodiments of this application. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.

Claims

1. A hyperspectral image reconstruction method based on flow matching prior assistance, characterized in that, include: Based on the measurement data, a condition vector is generated to constrain the flow matching model process and obtain the latent space vector. Let the initial iteration number k = 1, and the initial signal vector be v. (0) And determine the perception matrix; Based on the initial signal vector v (0) The sensing matrix and the measurement data are iterated in N stages until the final reconstructed hyperspectral image is obtained; The k-th iteration, 1≤k≤N, specifically includes: Based on the signal vector v of the (k-1)th stage (k-1) The measurement data and the perception matrix are projected from the measurement space to the three-dimensional signal space through depthwise separable convolution to calculate the hyperspectral image reconstruction estimate x for the k-th stage. (k) ; The hyperspectral image reconstruction estimate x in the k-th stage (k) The latent space vector is input to a denoiser for progressive refinement and reconstruction, outputting the signal vector v of the k-th stage. (k) ; After completing N iterations, the hyperspectral image reconstruction estimate x of the Nth stage is... (N) This is the final reconstructed hyperspectral image.

2. The hyperspectral image reconstruction method based on flow matching prior assistance according to claim 1, characterized in that, The process of generating condition vectors based on measurement data to constrain the flow matching model process and obtain latent space vectors includes: Use a residual fully connected layer as the encoder; Based on the measurement data, the encoder is used to generate a condition vector; Using the conditional vector as a constraint, a continuous normalized flow model is adopted, and the transformation is calibrated with an objective function to learn the invertible mapping from the standard normal distribution to the latent space vector, thereby obtaining the latent space vector.

3. The hyperspectral image reconstruction method based on flow matching prior assistance according to claim 2, characterized in that, The encoder is jointly optimized with the continuous normalized flow model, and the loss function includes reconstruction error and flow matching error: L total =λ1L F +λ2|x1-φ -1 (z)| 2 . Among them, L total Let L represent the total loss function. F Let x1 be the target hyperspectral image, z represent the prior generated by the flow matching, and φ be the flow matching loss. -1 This represents the subsequent reconstruction model, where λ1 and λ2 are weights.

4. The hyperspectral image reconstruction method based on flow matching prior assistance according to claim 1, characterized in that, The denoiser is a U-shaped backbone network, comprising: The encoder contains two levels of downsampling, each consisting of a Spectral-Spatial-PriorTransformer block, which is used to focus on spectral, spatial features and prior knowledge. The decoder employs an upsampling structure symmetrical to the encoder for progressively restoring feature resolution; A jump connection layer connects the encoder and the corresponding layer of the decoder, and is used to retain low-frequency detail information and improve reconstruction fidelity.

5. The hyperspectral image reconstruction method based on flow matching prior assistance according to claim 4, characterized in that, The Spectral-Spatial-PriorTransformer block includes decomposition units, spatial branches, spectral branches, prior branches, and fusion units; The decomposition unit decomposes the input features into a first feature and a second feature; The spatial branch calculates the self-attention of the spatial dimension of the first feature through spatial attention; The spectral branch and the prior branch together compute the cross-channel attention of the second feature through spectral-prior attention; The fusion unit fuses the self-attention of the spatial dimension and the cross-channel attention to obtain the output features.

6. The hyperspectral image reconstruction method based on flow matching prior assistance according to claim 5, characterized in that, In the spatial branch, the first feature is preprocessed by the Norm layer; the preprocessed feature enters a structure composed of a series of Mblocks without batch normalization, and after processing by the Mblocks, the intermediate output of the spatial branch is obtained; subsequently, the intermediate output is fused with the information flowing from the spectral branch and the prior branch, and the fusion process is achieved by a 1x1 convolution operation. The spectral branch employs an asymmetric, multi-scale, multi-head self-attention structure to process the second feature, resulting in the spectral branch feature, specifically: The second feature is embedded using Q, K, and V to obtain the basic feature vector required for attention calculation. The basic feature vector is compensated to generate intermediate variables for compensation; The intermediate variables are downsampled and normalized to obtain the normalized result; The normalization result is fused with the remaining intermediate variables to obtain the output spectral branching features; The prior branch calculates the K and V values ​​based on the latent space vectors; The attention weight is obtained by fusing the K value with the Q value in the spectral branch and performing a normalization operation. The attention weights are fused with the V value of the prior branch and the mapping is adjusted to obtain the prior branch features.

7. The hyperspectral image reconstruction method based on flow matching prior assistance according to claim 1, characterized in that, The model corresponding to the hyperspectral image reconstruction method based on flow matching prior assistance adopts a two-stage training method: Phase 1: The clean hyperspectral image x and the compressed measurement y are both input into LE to directly generate prior-aided denoising; Second stage: Use the flow matching model to generate priors, and use only the input measurement y as the control condition to generate priors to assist in denoising.

8. A hyperspectral image reconstruction system based on flow matching prior assistance, characterized in that, include: Flow matching module: Based on measurement data, it generates condition vectors to constrain the flow matching model process and obtain latent space vectors; Initial setup module: Let the initial iteration number k = 1, and the initial signal vector be v. (0) And determine the perception matrix and measurement data; Loop module: based on the initial signal vector v (0) The system uses the sensing matrix and measurement data to perform N stages of iterative module iterations until the final reconstructed hyperspectral image is obtained. The loop module in stage k, 1≤k≤N, specifically includes: Generalized alternating projection submodule: based on the signal vector v of the (k-1)th stage (k-1) The measurement data y and the perception matrix A are projected from the measurement space to the three-dimensional signal space through depthwise separable convolution, and the hyperspectral image reconstruction estimate x at the k-th stage is calculated. (k) ; Denoising module: Reconstructs the estimated value x from the hyperspectral image. (k) The latent space vector is input to the denoiser for progressive refinement and reconstruction, outputting the signal vector v of the k-th stage. (k) ; Output module: After completing N stages of iteration, the hyperspectral image reconstruction estimate x(N) of the Nth stage is used as the final reconstructed hyperspectral image.

9. A terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it can be used to perform the method of any one of claims 1-7, or to run the system of claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program can be used to perform the method of any one of claims 1-7, or to run the system of claim 8.

Images