An enhanced multi-linear mixing hyperspectral unmixing method, system and device
By constructing a multi-branch variational autoencoder demixing network and combining spectral variability and nonlinear mixing effects, the problem of limited accuracy in hyperspectral demixing under complex scenarios was solved, and high-precision endmember extraction and abundance inversion were achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHAOXING UNIVERSITY
- Filing Date
- 2026-05-19
- Publication Date
- 2026-07-24
AI Technical Summary
Existing hyperspectral unmixing methods struggle to effectively handle spectral variability and nonlinear mixing phenomena in complex scenarios, resulting in limited unmixing accuracy.
A multi-branch variational autoencoder demixing network is constructed, which combines spectral variability and nonlinear mixing effects. Through joint loss function optimization training, the decoupled learning of spectral variability and nonlinear scattering parameters is achieved.
The accuracy and robustness of hyperspectral unmixing in complex environments were improved under unsupervised conditions, and the accuracy of endmember extraction and abundance inversion were enhanced.
Smart Images
Figure CN122223458B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of hyperspectral image processing technology, and more specifically, to an enhanced multilinear hybrid hyperspectral demixing method, system, and apparatus. Background Technology
[0002] Due to limitations in sensor spatial resolution and the complexity of surface features, mixed pixels are prevalent in hyperspectral images, making it impossible to directly and accurately identify feature categories. Hyperspectral unmixing aims to decompose mixed pixels into endmember spectra and their corresponding proportions, and is a crucial step in the refined interpretation of hyperspectral images.
[0003] Traditional linear mixing models are classic models commonly used in hyperspectral unmixing. These models assume that light interacts with only one type of substance in the scene before reaching the sensor. However, in real-world complex scenes, nonlinear interactions between endmembers are prevalent, limiting the effectiveness of unmixing algorithms based on purely linear assumptions. Furthermore, due to environmental factors such as terrain, illumination, and atmospheric conditions, the endmember spectral curves corresponding to the same ground feature will exhibit significant changes in actual imaging, a phenomenon known as spectral variability. In complex real-world environments, spectral variability and nonlinear mixing of ground feature spectra may coexist, and their coupling effect further increases the difficulty of solving hyperspectral unmixing problems. However, most existing hyperspectral unmixing methods typically focus only on addressing a single phenomenon of spectral variability or nonlinear mixing, limiting their unmixing performance and robustness in complex real-world scenarios.
[0004] Therefore, how to construct a demixing model that can synergistically characterize spectral variability and nonlinear multiple scattering effects, and achieve effective decoupling learning of various physical parameters under unsupervised conditions, is a technical problem that urgently needs to be solved in the field of hyperspectral demixing. Summary of the Invention
[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide an enhanced multilinear mixing hyperspectral unmixing method, system and device, which solves the problem of limited accuracy of hyperspectral unmixing in complex scenarios by jointly modeling spectral variability and nonlinear mixing effects.
[0006] To achieve the above objectives, the present invention adopts the following technical solution:
[0007] An enhanced multilinear mixing hyperspectral demixing method includes the following steps:
[0008] Step S1: Obtain hyperspectral image data of the area to be processed and preprocess it;
[0009] Step S2: Pre-extract endmembers and abundance information from the preprocessed hyperspectral image data;
[0010] Step S3: Construct a multi-branch variational autoencoder demixing network;
[0011] Step S4: Based on the enhanced multilinear mixture model that takes into account spectral variability, the pixel is reconstructed at the end of the multi-branch variational autoencoder demixing network.
[0012] Step S5: Construct a joint loss function and iteratively optimize and train the multi-branch variational autoencoder demixing network.
[0013] Step S6: Output the unmixing results to obtain the variant endmember spectra and the corresponding land cover abundance distributions.
[0014] Furthermore, in step S2, the pre-extraction of endmember and abundance information specifically involves: extracting the basic endmember spectra using vertex component analysis and generating the corresponding initial abundance using fully constrained least squares method.
[0015] Furthermore, in step S3, the multi-branch variational autoencoder demixing network contains four parallel feature extraction branches, which are used to extract scaling factors, perturbation terms, nonlinear scattering parameters, and abundance vectors from the input pixels, respectively.
[0016] Furthermore, the scaling factor branch, perturbation term branch, and nonlinear scattering parameter branch each use their respective parameterized encoders to map the input pixels to a low-dimensional manifold to obtain the corresponding latent representations.
[0017] Furthermore, the abundance branch employs a multilayer perceptron, with the end outputting an abundance vector using a Softmax activation function.
[0018] Furthermore, in step S4, the construction of the enhanced multilinear mixture model taking into account spectral variability includes the following steps:
[0019] Step S41: Establish a basic linear mixture model, including the endmember matrix, abundance vector, and additive noise term;
[0020] Step S42 introduces a scaling factor and a non-scale small perturbation to extend the basic linear mixture model into a model that senses spectral variability, where spectral variability is modeled as a combination of the scaling factor and the perturbation term.
[0021] Step S43: Introduce pixel-level nonlinear scattering parameters to control the probability of further interaction between photons;
[0022] Step S44: Combine the endmember physics terms containing spectral variability with the multiple scattering probability model to obtain the final enhanced multilinear mixture model.
[0023] Furthermore, in step S5, the construction of the joint loss function and the network training process include:
[0024] Step S51: Using the pre-extracted initial abundance as a prior, the multi-branch variational autoencoder demixing network is pre-trained using a pre-trained loss function that includes reconstruction loss and KL divergence loss.
[0025] Step S52: After pre-training is completed, the multi-branch variational autoencoder demixing network is formally trained using a joint loss function with added regularization term. The regularization term is used to constrain the desired properties of the variant endmember spectra.
[0026] Furthermore, the regularization term includes a term that minimizes the difference between the spectrum of the variant endmember corresponding to each pixel and the average spectrum of that endmember across all pixels.
[0027] The present invention also provides an enhanced multilinear mixing hyperspectral demixing system, the system comprising:
[0028] Hyperspectral image acquisition unit, used to capture and acquire hyperspectral images;
[0029] The preprocessing unit is used to perform reflectance correction and outlier removal on the hyperspectral image.
[0030] The endmember prior information extraction unit is used to extract endmember and abundance prior information from the preprocessed hyperspectral image data.
[0031] The hyperspectral unmixing unit is used to input the preprocessed hyperspectral image and the basic endmembers into the constructed multi-branch variational autoencoder unmixing network to obtain the variation endmember spectra and the corresponding land cover abundance matrix.
[0032] The present invention also provides an enhanced multilinear mixing hyperspectral demixing device, comprising:
[0033] Memory, used to store computer programs and data;
[0034] The processor is used to implement the steps of the enhanced multilinear mixing hyperspectral demixing method described above when executing a computer program.
[0035] The beneficial effects of this invention are:
[0036] 1. This invention successfully achieves decoupled learning of scaling factors, perturbation terms, and nonlinear scattering parameters under unsupervised conditions by constructing a demixing network containing a multi-branch variational autoencoder.
[0037] 2. This invention integrates the feature representation capabilities of deep generative models with the mechanism-driven approach of real physical models. Even under the complex interference of coupled spectral variability and nonlinear spectral mixing, it can still maintain high accuracy in endmember extraction and abundance inversion, thus improving the performance of hyperspectral unmixing in complex environments. Attached Figure Description
[0038] Figure 1 This is a flowchart of an enhanced multilinear mixing hyperspectral demixing method in this embodiment;
[0039] Figure 2 This is a structural framework diagram of a multi-branch variational autoencoder demixing network in this embodiment;
[0040] Figure 3 This is a structural framework diagram of the enhanced multilinear hybrid hyperspectral demixing system in this embodiment;
[0041] Figure 4 This is a structural framework diagram of the enhanced multilinear mixing hyperspectral demixing device in this embodiment.
[0042] Figure reference numerals: 1. Hyperspectral image acquisition unit; 2. Preprocessing unit; 3. Endmember prior information extraction unit; 4. Hyperspectral demixing unit; 5. Memory; 6. Processor. Detailed Implementation
[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0044] Example: An enhanced multilinear mixture hyperspectral unmixing method is proposed. This method first acquires hyperspectral images of ground features, then preprocesses the hyperspectral images to obtain basic endmember spectra and initial abundances through endmember and abundance information pre-extraction. Next, a multi-branch variational autoencoder unmixing network is constructed, inputting the preprocessed hyperspectral image into the network. The network extracts scaling factors, perturbation terms, nonlinear scattering parameters, and abundance vectors through four parallel branches. Subsequently, reconstruction calculations are performed based on the proposed enhanced multilinear mixture model that considers spectral variability. Finally, the network is optimized through a joint loss function, outputting the variable endmember spectra and abundance distribution matrix. This method effectively decouples the coupling interference of spectral variability and nonlinear multiple scattering under unsupervised conditions, achieving high-precision unmixing in complex scenarios.
[0045] Specifically, such as Figure 1 As shown, it includes the following steps:
[0046] Step S1: Obtain hyperspectral image data of the area to be processed and preprocess it.
[0047] Since the raw data recorded by the sensor is radiance value, it needs to be converted to reflectance to eliminate variations in light intensity and atmospheric effects. The formula for reflectance correction is:
[0048]
[0049] In the formula, To be at wavelength Reflectance at that location; For the sensor to record at wavelength Radiance at that location; The dark current response was obtained by measuring the black calibration plate; The radiance obtained by measuring a white reference plate; The known reflectance value of the white reference plate, typically 1 or a known high reflectance value.
[0050] After reflectivity correction, outlier removal is performed. Due to sensor noise, atmospheric correction errors, or cloud cover, the spectrum of some pixels may exhibit anomalies. The outlier removal method is as follows:
[0051] For reflectivity Perform threshold truncation if If < 0, then set to 0; if If the expression is greater than 1, then set it to 1, and leave the rest unchanged. The expression is:
[0052]
[0053] Step S2: In order to provide prior knowledge for subsequent deep networks, endmember and abundance information are pre-extracted from the preprocessed hyperspectral image.
[0054] This embodiment uses Vertex Component Analysis (VCA) algorithm to extract the basic endmember spectrum. ; In obtaining the basic endmember spectrum Then, the corresponding initial abundance was generated using the fully constrained least squares (FCLS) method. .
[0055] Step S3: Construct a multi-branch variational autoencoder demixing network.
[0056] like Figure 2 As shown, the multi-branch variational autoencoder demixing network contains four parallel feature extraction branches, which are used to extract scaling factors, perturbation terms, nonlinear scattering parameters, and abundance vectors from the input pixels, respectively.
[0057] Among them, the scaling factor, perturbation term, and nonlinear scattering parameter branches adopt a variational encoder structure, specifically:
[0058] Assuming three parameters are used respectively , , Parameterized encoder , , Input pixels Mapped to a low-dimensional manifold. Then, three vectors are introduced. , , Let represent the latent representations of the scaling factor, perturbation term, and scattering parameters learned from the input pixel, respectively. For each pixel, the following expression holds:
[0059]
[0060]
[0061]
[0062] In the formula, It is a diagonal scaling matrix; It is an additive perturbation matrix; These are pixel-level nonlinear scattering parameters; Indicates the column stacking operator; , and They represent respectively by , , Parameterized nonlinear mapping function; parameters , , Learning is achieved by maximizing the lower bound of the log-likelihood function of the data (ELBO).
[0063] The abundance branch uses a multilayer perceptron (MLP), with the softmax activation function hard-coded at the ends, outputting an abundance vector. .
[0064] Step S4: Based on the enhanced multilinear mixing model that takes into account spectral variability, the pixel is reconstructed at the end of the multi-branch variational autoencoder demixing network.
[0065] Specifically, it includes the following steps:
[0066] Step S41, establish the basic linear mixed model, whose expression is:
[0067]
[0068] In the formula, Let be the spectral vector of the nth pixel; It is an endmember matrix; For abundance vectors, The abundance sum is constrained to 1, that is ; The additive noise term introduced by the system and the environment.
[0069] Step S42, to characterize spectral variability, the basic endmember matrix obtained in step S2 is... By scaling factor Non-scale small perturbations The transformation is performed; the basic linear mixture model is extended to a model that senses spectral variation, and its expression is:
[0070]
[0071] In the formula, This refers to the fundamental end-member matrix under ideal or baseline conditions. Represents the basic endmember matrix Applying a spectral variability transformation operation, that is, modeling spectral variability as a scaling factor. and disturbance terms Combination, disturbance term This stems from the inherent structural differences in the imaging environment and materials.
[0072] Step S43: Introduce pixel-level nonlinear scattering parameters This is used to control the probability of further interactions. Within the framework of a multilinear mixture model, the simplified model expression is:
[0073]
[0074] In the formula, For a general endmember matrix, it will be... Substitute; It is an abundance vector; The additive noise term introduced by the system and the environment; It is a column vector whose elements are all 1s.
[0075] Step S44: The endmember physics term containing spectral variability is combined with the multiple scattering probability model to finally obtain the enhanced multilinear mixture model that takes into account spectral variability, and its expression is:
[0076]
[0077] In the formula, It is a column vector whose elements are all 1s.
[0078] Step S5: Construct a joint loss function and iteratively optimize and train the multi-branch variational autoencoder demixing network.
[0079] Specifically, it includes the following steps:
[0080] Step S51: To accelerate convergence and improve unmixing accuracy, the initial abundance extracted in step S2 is used as a priori to pre-train the multi-branch variational autoencoder unmixing network. The expression for the pre-training loss function is:
[0081]
[0082] In the formula, KL divergence term The weights; For the prior abundance loss term The weight.
[0083] To calculate the prior abundance loss, the abundance predicted by the network is calculated. Compared with the initial abundance of pre-extracted The mean square error between them is expressed as:
[0084]
[0085] In the formula, n is the cell index; N is the total number of cells; It is an L2 norm.
[0086] The reconstruction loss is expressed as follows:
[0087]
[0088] In the formula, Let be the spectral vector of the nth pixel; It is a diagonal scaling matrix; It is an additive perturbation matrix; These are pixel-level nonlinear scattering parameters; Basic endmember spectrum; It is a column vector whose elements are all 1s.
[0089] The KL divergence loss is expressed as follows:
[0090]
[0091] In the formula, The dimension of the low-dimensional manifold; For dimension indexing; , and These represent the three latent representation vectors of the j-th dimension, respectively. , , The variance; , , These represent the three latent representation vectors of the j-th dimension, respectively. , , The mean.
[0092] Step S52: After pre-training, the multi-branch variational autoencoder demixing network is formally trained using a joint loss function. The formal training loss function adds two regularization terms to the pre-training loss to improve the physical plausibility of the demixing results and the consistency of the endmember spectra. The expression for the joint loss function is:
[0093]
[0094] in, KL divergence term The weights; For endmember spectral uniformity regularization term The weights; For endmember spectral dispersion regularization term The weight.
[0095] This is the endmember spectral consistency regularization term, which constrains the difference between the variant endmember spectrum of each pixel and the average spectrum of that endmember across all pixels, preventing excessive variation in the endmember spectrum that would lose its physical meaning. Its expression is:
[0096]
[0097] In the formula, Number of endmember categories; For endmember category index; This represents the p-th mutated endmember extracted from the n-th pixel; The average spectrum of the p-th variant endmember among all pixels; This is the transpose operator; N is the total number of pixels.
[0098] This is the endmember spectral dispersion regularization term, used to constrain the differences in endmember spectra within the same pixel, thereby improving the discriminative power of endmember spectra. Its expression is:
[0099]
[0100] In the formula, This represents the number of bands in the hyperspectral data. This represents the q-th mutated endmember extracted from the n-th pixel; This is an end-member category index, with the same meaning as p.
[0101] Step S6: After the multi-branch variational autoencoder demixing network is trained, the demixing results are output to obtain the variational endmember spectra and the corresponding land cover abundance distributions.
[0102] This embodiment also provides an enhanced multilinear mixing hyperspectral demixing system, such as Figure 3 As shown, the system includes a hyperspectral image acquisition unit 1, a preprocessing unit 2, an endmember prior information extraction unit 3, and a hyperspectral demixing unit 4.
[0103] Among them, the hyperspectral image acquisition unit 1 is responsible for capturing and acquiring hyperspectral images.
[0104] Preprocessing unit 2 is responsible for preprocessing the hyperspectral image by performing reflectance correction and outlier removal.
[0105] The endmember prior information extraction unit 3 is responsible for extracting endmember and abundance prior information from the preprocessed hyperspectral image data.
[0106] The hyperspectral unmixing unit 4 is responsible for inputting the preprocessed hyperspectral image and the basic endmembers into the constructed multi-branch variational autoencoder unmixing network to obtain the variation endmember spectra and the corresponding land cover abundance matrix.
[0107] This embodiment also provides an enhanced multilinear mixing hyperspectral demixing device, such as... Figure 4 As shown, it includes a memory 5 and a processor 6; wherein, the memory 5 is used to store computer programs and data; the processor 6 is used to implement the steps of the above-described enhanced multilinear mixing hyperspectral demixing method when executing the computer program.
[0108] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.
Claims
1. An enhanced multilinear mixing hyperspectral demixing method, characterized in that, Includes the following steps: Step S1: Obtain hyperspectral image data of the area to be processed and preprocess it; Step S2: Pre-extract endmembers and abundance information from the preprocessed hyperspectral image data; Step S3: Construct a multi-branch variational autoencoder demixing network; Step S4: Based on the enhanced multilinear mixture model that takes into account spectral variability, the pixel is reconstructed at the end of the multi-branch variational autoencoder demixing network. Step S5: Construct a joint loss function and iteratively optimize and train the multi-branch variational autoencoder demixing network. Step S6: Output the unmixing results to obtain the variant endmember spectra and the corresponding land cover abundance distributions; In step S3, the multi-branch variational autoencoder unmixing network contains four parallel feature extraction branches, which are used to extract scaling factors, perturbation terms, nonlinear scattering parameters and abundance vectors from the input pixels, respectively. The scaling factor branch, perturbation term branch, and nonlinear scattering parameter branch each use their respective parameterized encoders to map the input pixels to a low-dimensional manifold to obtain the corresponding latent representations.
2. The enhanced multilinear mixing hyperspectral demixing method according to claim 1, characterized in that, In step S2, the pre-extraction of endmember and abundance information specifically involves: extracting the basic endmember spectra using vertex component analysis and generating the corresponding initial abundance using fully constrained least squares method.
3. The enhanced multilinear mixing hyperspectral demixing method according to claim 1, characterized in that, The abundance branch uses a multilayer perceptron, and the end outputs an abundance vector using a Softmax activation function.
4. The enhanced multilinear mixing hyperspectral demixing method according to claim 1, characterized in that, In step S4, the construction of the enhanced multilinear mixture model taking into account spectral variability includes the following steps: Step S41: Establish a basic linear mixture model, including the endmember matrix, abundance vector, and additive noise term; Step S42 introduces a scaling factor and a non-scale small perturbation to extend the basic linear mixture model into a model that senses spectral variability, where spectral variability is modeled as a combination of the scaling factor and the perturbation term. Step S43: Introduce pixel-level nonlinear scattering parameters to control the probability of further interaction between photons; Step S44: Combine the endmember physics terms containing spectral variability with the multiple scattering probability model to obtain the final enhanced multilinear mixture model.
5. The enhanced multilinear mixing hyperspectral demixing method according to claim 1, characterized in that, In step S5, the construction of the joint loss function and the network training process include: Step S51: Using the pre-extracted initial abundance as a prior, the multi-branch variational autoencoder demixing network is pre-trained using a pre-trained loss function that includes reconstruction loss and KL divergence loss. Step S52: After pre-training is completed, the multi-branch variational autoencoder demixing network is formally trained using a joint loss function with added regularization term. The regularization term is used to constrain the desired properties of the variant endmember spectra.
6. The enhanced multilinear mixing hyperspectral demixing method according to claim 5, characterized in that, The regularization term includes a term that minimizes the difference between the spectrum of the variant endmember corresponding to each pixel and the average spectrum of that endmember across all pixels.
7. An enhanced multilinear mixing hyperspectral demixing system for implementing the method of claim 1, characterized in that, The system includes: Hyperspectral image acquisition unit (1) is used to capture and acquire hyperspectral images; The preprocessing unit (2) is used to perform reflectance correction and outlier removal preprocessing on the hyperspectral image; The endmember prior information extraction unit (3) is used to extract endmember and abundance prior information from the preprocessed hyperspectral image data. The hyperspectral unmixing unit (4) is used to input the preprocessed hyperspectral image and the basic endmember into the constructed multi-branch variational autoencoder unmixing network to obtain the variation endmember spectrum and the corresponding land cover abundance matrix.
8. An enhanced multilinear mixing hyperspectral demixing device, characterized in that, include: Memory (5) is used to store computer programs and data; The processor (6) is configured to implement the steps of the enhanced multilinear mixing hyperspectral demixing method as described in any one of claims 1-6 when executing a computer program.