Methane column concentration inversion method and system based on physical driving variational autoencoder

Abstract

The application discloses a methane column concentration inversion method and system based on a physical driving variational autoencoder, relates to the technical field of greenhouse gas monitoring, and aims to solve the technical problem of low reliability of the existing data-driven methane column concentration inversion method. The methane column concentration inversion method based on the physical driving variational autoencoder comprises the following steps: a loss function containing Jacobian matrix consistency constraint is used to pre-train a decoder module, and the network weights of all layers of the decoder module are frozen; an observation spectrum is used as the input of an encoder module; the encoder module outputs a posterior probability distribution, and any concentration value is sampled and input into the decoder module to generate a simulated spectrum corresponding to the concentration value through forward simulation; when the difference between the simulated spectrum and the observation spectrum is the smallest, the optimal methane column concentration inversion encoder is determined, the observation spectrum is input into the encoder module, and the mean and standard deviation of the posterior probability distribution are output.

Description

Technical Field

[0001] This invention relates to the field of greenhouse gas monitoring technology, and more specifically, to a method and system for methane column concentration inversion based on a physical-driven variational autoencoder. Background Technology

[0002] Methane (CH4) is the world's second largest greenhouse gas. Its global warming potential over a 100-year timescale is about 28 times that of carbon dioxide, making it central to climate change research. By using spectrometers on orbiting satellites (such as Japan's GOSAT and Europe's Tropomi) to analyze the solar spectrum reflected from the Earth in the shortwave infrared band, the column concentration of methane in the atmosphere can be quantitatively determined. This has irreplaceable value for global carbon cycle research, identification and monitoring of high emission sources, and the formulation of climate policies.

[0003] In methane column concentration retrieval methods, traditional remote sensing retrieval relies on iterative radiative transfer models (RTMs), which can guarantee physical plausibility but are computationally intensive and time-consuming, making them unsuitable for real-time, large-scale processing or near-real-time operational applications. While purely data-driven machine learning methods are computationally efficient, they lack high-precision matching data, have limited generalization ability, low applicability in sparse data regions, and, as typical black-box models, lack physical interpretability. In recent years, some studies have attempted to use RTM simulations to screen sensitive bands for methane retrieval and further construct machine learning models. However, these studies only use physical models for data generation or sensitive channel screening preprocessing stages, without deeply integrating physical laws as strong constraints into the neural network training process. The differences between simulated spectra and actual noisy observational data have also been overlooked. Because various atmospheric states can produce similar spectral signals, the retrieval becomes an ill-posed problem with multiple solutions. If RTM is used to generate samples only during the training or preprocessing stage without using physical laws as training constraints, the data-driven model usually only gives a single definite value without quantifying the systematic uncertainty. At the same time, under noisy real observation conditions, this one-valued output cannot characterize the potential solution set, which can easily lead to misleading confidence judgments and make the inversion unreliable. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for methane column concentration inversion based on a physical-driven variational autoencoder, which addresses the technical problem of low reliability in existing data-driven methane column concentration inversion methods due to a lack of quantitative assessment capability for uncertainty in inversion results when dealing with ill-conditioned remote sensing inversion problems. Therefore, this invention achieves this through the following scheme.

[0005] In a first aspect, the present invention provides a method for methane column concentration inversion based on a physical-driven variational autoencoder, comprising:

[0006] Identify the decoder module and encoder module;

[0007] The decoder module is pre-trained using a loss function that includes Jacobian matrix consistency constraints, and the network weights of all its layers are frozen after training converges.

[0008] The encoder module is used to extract the absorption features of the spectrum and capture the dependencies between different bands;

[0009] The encoder module takes the observed spectrum as input and outputs the posterior probability distribution representing the methane concentration corresponding to the observed spectrum.

[0010] Any concentration value is sampled from the posterior probability distribution, and the concentration value is input into the decoder module to perform forward simulation to generate a simulated spectrum corresponding to the concentration value; by continuously minimizing the difference between the simulated spectrum and the observed spectrum, the optimal methane column concentration inversion encoder is determined when the difference between the two is minimized, and the observed spectrum is input into the encoder module after training.

[0011] The posterior probability distribution representing the methane concentration corresponding to the observed spectrum is output through a forward propagation, and the mean of the posterior probability distribution is determined as the final methane column concentration inversion value, while the standard deviation is output as an uncertainty measure.

[0012] Compared with existing technologies, the methane column concentration inversion method based on a physics-driven variational autoencoder of the present invention, after determining the decoder module and encoder module, pre-trains the decoder module using a loss function containing Jacobian matrix consistency constraints, and freezes the network weights of all its layers after training convergence; then, the observed spectrum is used as the input to the encoder module, and the encoder module outputs the posterior probability distribution representing the methane concentration corresponding to the observed spectrum; further, based on the above decoder module and encoder module, any concentration value can be sampled from the above posterior probability distribution, and the concentration value is input to the decoder module for forward simulation to generate a simulated spectrum corresponding to the concentration value; by continuously minimizing the difference between the simulated spectrum and the observed spectrum, the optimal methane column concentration inversion encoder is determined when the difference between the two is minimized; the observed spectrum is input to the trained encoder module, and the posterior probability distribution representing the methane concentration corresponding to the observed spectrum is directly output through one forward propagation, and the mean of the posterior probability distribution is determined as the final methane column concentration inversion value, and the standard deviation is used as an uncertainty measure and output, thus completing the inversion of the methane column concentration. In the above technical solution of the present invention, remote sensing quantitative inversion can be achieved through a physics-driven variational autoencoder architecture. The decoder module can be a physical process simulator pre-trained based on RTM simulation data, while the encoder module predicts the methane column concentration during the inversion process and further inputs the methane column concentration into the decoder module of the neural network to output spectral data, which is then compared with the spectrum actually observed by the satellite. This architecture achieves deep coupling between the physical model and the neural network, rather than the loose embedding in the prior art, ensuring that the entire inversion process is dominated by physical laws. Furthermore, the present invention adopts a joint training mechanism with dual physical constraints, especially the Jacobian matrix consistency constraint. The first layer of spectral reconstruction constraint ensures the physical closed loop of the inversion results of the neural network encoder module, while the second layer of Jacobian matrix consistency constraint ensures the physical closed loop of the inversion results of the neural network encoder module. The comparability matrix consistency constraint forces the neural network to learn the correct physical gradient, significantly narrowing down the potential solution set that matches the spectrum. Furthermore, to achieve the Jacobian matrix consistency constraint, this invention can output the Jacobian matrix for each atmospheric layer from the standard RTM, and convert it into the total column concentration Jacobian matrix corresponding to the single variable (total methane column concentration) in this invention through a pressure-weighted summation physical method. This Jacobian matrix is ​​used as the "true value" label for neural network training, which is the key to achieving gradient-level physical constraints. This invention improves the ability to quantitatively assess the uncertainty of the inversion results and has high reliability. It solves the technical problem that existing data-driven methane column concentration inversion methods lack the ability to quantitatively assess the uncertainty of the inversion results when dealing with ill-conditioned remote sensing inversion problems, resulting in low reliability of the inversion.

[0013] Furthermore, in the methane column concentration inversion method based on a physical-driven variational autoencoder of the present invention, the decoder determination module includes:

[0014] A neural network is identified and trained using data from a physical simulation of an atmospheric radiative transfer model, thereby forming the decoder module of the inversion architecture.

[0015] The neural network corresponds to the forward simulation process of the atmospheric radiative transfer model, which learns the mapping from total methane column concentration, observed geometric attitude, atmospheric and surface parameters to the corresponding simulated spectrum.

[0016] Furthermore, in the methane column concentration inversion method based on a physical-driven variational autoencoder of the present invention, the observed geometric attitude includes the solar zenith angle, satellite zenith angle, and relative azimuth angle; the atmospheric and surface parameters include temperature profiles, pressure profiles, water vapor profiles, and surface albedo.

[0017] Furthermore, in the methane column concentration inversion method based on a physics-driven variational autoencoder of the present invention, the step of pre-training the decoder module using a loss function containing Jacobian matrix consistency constraints, and freezing the network weights of all its layers after training convergence, includes:

[0018] The Jacobian matrix is ​​obtained by using the automatic differentiation function of the deep learning framework to obtain the derivative of the simulated spectrum output by the decoder module with respect to the concentration value.

[0019] The loss term is obtained by using the mean squared error to obtain the difference between two Jacobian matrix vectors.

[0020] The loss term forces the neural network to simulate the correct spectral output and learns the reasons for gradient changes to conform to the physical mechanism; when the decoder module converges under the drive of the decoding loss function, the network parameters corresponding to the decoder module are frozen.

[0021] Furthermore, in the methane column concentration inversion method based on a physical-driven variational autoencoder of the present invention, the decoding loss function is obtained by standard spectral fitting loss and Jacobian matrix consistency loss.

[0022] Furthermore, in the methane column concentration inversion method based on a physical-driven variational autoencoder of the present invention, determining the encoder module includes:

[0023] The encoder module uses a Transformer architecture to extract local absorption features of the spectrum and combines a self-attention mechanism to capture the dependencies between different bands.

[0024] The encoder module takes the observed spectrum as input, passes it through the network's forward propagation, and outputs the mean and standard deviation of the total methane column concentration.

[0025] Furthermore, in the methane column concentration inversion method based on a physical-driven variational autoencoder of the present invention, after training the encoder module, the encoder module infers a posterior probability distribution of the methane concentration that most likely produces the observed spectrum from an observed spectrum.

[0026] Furthermore, in the methane column concentration inversion method based on physical-driven variational autoencoder of the present invention, the weights of the encoder network corresponding to the encoder module are driven by a joint loss function, which is composed of a weighted sum of spectral reconstruction loss and KL divergence loss.

[0027] Furthermore, in the methane column concentration inversion method based on a physics-driven variational autoencoder of the present invention, the joint loss function is obtained at least through spectral reconstruction loss, KL divergence loss, and data constraint loss; the joint loss function is expressed as:

[0028] ;

[0029] in, Denotes the joint loss function. Indicates spectral reconstruction loss, This represents the KL divergence loss. This represents the first weight parameter. This represents the second weighting parameter. Indicates the data constraint loss, and , (•) indicates the mean square error. This represents the predicted mean of the total methane column concentration. This represents the actual observed total methane column concentration.

[0030] Secondly, the present invention provides a methane column concentration inversion system based on a physical-driven variational autoencoder, comprising:

[0031] Decoder determination module, used to determine the decoder module;

[0032] Encoder determination module, used to determine the encoder module;

[0033] A decoder training module is used to pre-train the decoder module using a loss function that includes a Jacobian matrix consistency constraint, and to freeze the network weights of all its layers after training convergence.

[0034] A posterior probability distribution determination module is used to take the observed spectrum as input to the encoder module, and the encoder module outputs the posterior probability distribution representing the methane concentration corresponding to the observed spectrum.

[0035] The inversion value determination module is used to sample any concentration value in the posterior probability distribution, input the concentration value into the decoder module for forward simulation to generate a simulated spectrum corresponding to the concentration value; by continuously minimizing the difference between the simulated spectrum and the observed spectrum, the optimal methane column concentration inversion encoder is determined when the difference is minimal, and the observed spectrum is input into the trained encoder module; the posterior probability distribution representing the methane concentration corresponding to the observed spectrum is output through one forward propagation, and the mean of the posterior probability distribution is determined as the final methane column concentration inversion value, and the standard deviation is output as an uncertainty measure.

[0036] Compared with the prior art, the beneficial effects of the methane column concentration inversion system based on physical-driven variational autoencoder of the present invention are the same as those of the methane column concentration inversion method based on physical-driven variational autoencoder described in the above technical solution, and will not be repeated here. Attached Figure Description

[0037] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0038] Figure 1 This is a schematic diagram of the architecture consisting of the decoder module and the encoder module in this invention. Detailed Implementation

[0039] To make the technical problems to be solved, the technical solutions, and the beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0040] It should be noted that when a component is referred to as being "fixed to" or "set on" another component, it can be directly on or indirectly on that other component. When a component is referred to as being "connected to" another component, it can be directly connected to or indirectly connected to that other component.

[0041] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified. "Several" means one or more, unless otherwise explicitly specified.

[0042] In methane column concentration retrieval methods, traditional remote sensing retrieval relies on iterative radiative transfer models (RTMs), which can guarantee physical plausibility but are computationally intensive and time-consuming, making them unsuitable for real-time, large-scale processing or near-real-time operational applications. While purely data-driven machine learning methods are computationally efficient, they lack high-precision matching data, have limited generalization ability, low applicability in sparse data regions, and, as typical black-box models, lack physical interpretability. In recent years, some studies have attempted to use RTM simulations to screen sensitive bands for methane retrieval and further construct machine learning models. However, these studies only use physical models for data generation or sensitive channel screening preprocessing stages, without deeply integrating physical laws as strong constraints into the neural network training process. The differences between simulated spectra and actual noisy observational data have also been overlooked. Because various atmospheric states can produce similar spectral signals, the retrieval becomes an ill-posed problem with multiple solutions. If RTM is used to generate samples only during the training or preprocessing stage without using physical laws as training constraints, the data-driven model usually only gives a single definite value without quantifying the systematic uncertainty. At the same time, under noisy real observation conditions, this one-valued output cannot characterize the potential solution set, which can easily lead to misleading confidence judgments and make the inversion unreliable.

[0043] To address the above technical problems, this invention provides a method for methane column concentration inversion based on a physical-driven variational autoencoder, comprising:

[0044] Identify the decoder module and encoder module;

[0045] The decoder module is pre-trained using a loss function that includes Jacobian matrix consistency constraints, and the network weights of all its layers are frozen after training converges.

[0046] The encoder module is used to extract the absorption features of the spectrum and capture the dependencies between different bands;

[0047] The encoder module takes the observed spectrum as input and outputs the posterior probability distribution representing the methane concentration corresponding to the observed spectrum.

[0048] Any concentration value is sampled from the posterior probability distribution, and the concentration value is input into the decoder module to perform forward simulation to generate a simulated spectrum corresponding to the concentration value; by continuously minimizing the difference between the simulated spectrum and the observed spectrum, the optimal methane column concentration inversion encoder is determined when the difference between the two is minimized, and the observed spectrum is input into the encoder module after training.

[0049] The posterior probability distribution representing the methane concentration corresponding to the observed spectrum is output through a forward propagation, and the mean of the posterior probability distribution is determined as the final methane column concentration inversion value, while the standard deviation is output as an uncertainty measure.

[0050] In the methane column concentration inversion method based on a physical-driven variational autoencoder of the present invention, after determining the decoder module and the encoder module, the decoder module is pre-trained using a loss function that includes a Jacobian matrix consistency constraint, and the network weights of all its layers are frozen after training convergence. Then, the observed spectrum is used as the input to the encoder module, which outputs the posterior probability distribution representing the methane concentration corresponding to the observed spectrum. Further, based on the decoder module and the encoder module, any concentration value can be sampled from the posterior probability distribution, and the concentration value is input to the decoder module for forward simulation to generate a simulated spectrum corresponding to that concentration value. By continuously minimizing the difference between the simulated spectrum and the observed spectrum, the optimal methane column concentration inversion encoder is determined when the difference is minimized. The observed spectrum is input to the trained encoder module, and the posterior probability distribution representing the methane concentration corresponding to the observed spectrum is directly output through one forward propagation. The mean of the posterior probability distribution is determined as the final methane column concentration inversion value, and the standard deviation is used as an uncertainty measure and output, thus completing the methane column concentration inversion. In the above technical solution of the present invention, remote sensing quantitative inversion can be achieved through a physics-driven variational autoencoder architecture. The decoder module can be a physical process simulator pre-trained based on RTM simulation data, while the encoder module predicts the methane column concentration during the inversion process and further inputs the methane column concentration into the decoder module of the neural network to output spectral data, which is then compared with the spectrum actually observed by the satellite. This architecture achieves deep coupling between the physical model and the neural network, rather than the loose embedding in the prior art, ensuring that the entire inversion process is dominated by physical laws. Furthermore, the present invention adopts a joint training mechanism with dual physical constraints, especially the Jacobian matrix consistency constraint. The first layer of spectral reconstruction constraint ensures the physical closed loop of the inversion results of the neural network encoder module, while the second layer of Jacobian matrix consistency constraint ensures the physical closed loop of the inversion results of the neural network encoder module. The comparability matrix consistency constraint forces the neural network to learn the correct physical gradient, significantly narrowing down the potential solution set that matches the spectrum. Furthermore, to achieve the Jacobian matrix consistency constraint, this invention can output the Jacobian matrix for each atmospheric layer from the standard RTM, and convert it into the total column concentration Jacobian matrix corresponding to the single variable (total methane column concentration) in this invention through a pressure-weighted summation physical method. This Jacobian matrix is ​​used as the "true value" label for neural network training, which is the key to achieving gradient-level physical constraints. This invention improves the ability to quantitatively assess the uncertainty of the inversion results and has high reliability. It solves the technical problem that existing data-driven methane column concentration inversion methods lack the ability to quantitatively assess the uncertainty of the inversion results when dealing with ill-conditioned remote sensing inversion problems, resulting in low reliability of the inversion.

[0051] To better understand the present invention, the following specific embodiments further illustrate the content of the present invention, but the content of the present invention is not limited to the following embodiments.

[0052] Example 1

[0053] This embodiment provides a method for methane column concentration inversion based on a physical-driven variational autoencoder, including:

[0054] Step 1: Identify the decoder module and encoder module;

[0055] Step 2: The decoder module is pre-trained using a loss function that includes Jacobian matrix consistency constraints, and the network weights of all its layers are frozen after training converges; the encoder module is used to extract the absorption features of the spectrum and capture the dependencies between different bands.

[0056] Step 3: Using the observed spectrum as input to the encoder module, the encoder module outputs the posterior probability distribution representing the methane concentration corresponding to the observed spectrum.

[0057] Step 4: Sample any concentration value in the posterior probability distribution and input the concentration value into the decoder module to perform forward simulation to generate a simulated spectrum corresponding to the concentration value; by continuously minimizing the difference between the simulated spectrum and the observed spectrum, the optimal methane column concentration inversion encoder is determined when the difference between the two is minimized.

[0058] Step 5: Input the observed spectrum into the trained encoder module, and output the posterior probability distribution representing the methane concentration corresponding to the observed spectrum through one forward propagation. The mean of the posterior probability distribution is determined as the final methane column concentration inversion value, and the standard deviation is used as an uncertainty measure and output.

[0059] Example 2

[0060] This embodiment provides a method for methane column concentration inversion based on a physical-driven variational autoencoder, including:

[0061] S100, determine the decoder module and encoder module.

[0062] Further, the decoder module is determined, including:

[0063] S111, a neural network is determined and trained using data from the physical simulation of the atmospheric radiative transfer model, thereby forming the decoder module of the inversion architecture;

[0064] S112, the neural network corresponds to the forward simulation process of the atmospheric radiative transfer model, which learns the mapping from total methane column concentration, observed geometric attitude, atmospheric and surface parameters to the corresponding simulated spectrum; among which, observed geometric attitude includes solar zenith angle, satellite zenith angle and relative azimuth angle; atmospheric and surface parameters include temperature profile, pressure profile, water vapor profile and surface albedo.

[0065] Further, the encoder module is determined, including:

[0066] S121, the encoder module uses the Transformer architecture to extract local absorption features of the spectrum and combines a self-attention mechanism to capture the dependencies between different bands;

[0067] S122, the encoder module takes the observed spectrum as input, passes it through the network's forward propagation, and outputs the mean and standard deviation of the total methane column concentration.

[0068] Furthermore, after training the encoder module, the encoder module infers the posterior probability distribution of the methane concentration that most likely produced the observed spectrum from an observed spectrum.

[0069] Furthermore, the weights of the encoder network corresponding to the encoder module are driven by a joint loss function, which is composed of a weighted average of the spectral reconstruction loss and the KL divergence loss; it can be expressed as: ;in, Denotes the joint loss function. Indicates spectral reconstruction loss, This represents the KL divergence loss. This represents the first weight parameter. This represents the second weighting parameter.

[0070] Furthermore, the joint loss function is obtained through spectral reconstruction loss, KL divergence loss, and data constraint loss; the joint loss function is expressed as:

[0071] ;in, Denotes the joint loss function. Indicates spectral reconstruction loss, This represents the KL divergence loss. This represents the first weight parameter. This represents the second weighting parameter. Indicates the data constraint loss, and , (•) indicates the mean square error. This represents the predicted mean of the total methane column concentration. This represents the actual observed total methane column concentration.

[0072] S200 pre-trains the decoder module using a loss function that includes Jacobian matrix consistency constraints, and freezes the network weights of all its layers after training convergence; the encoder module is used to extract the absorption features of the spectrum and capture the dependencies between different bands.

[0073] Furthermore, the decoder module is pre-trained using a loss function that includes Jacobian matrix consistency constraints, and the network weights of all its layers are frozen after training convergence, including:

[0074] S201, using the automatic differentiation function of the deep learning framework, the derivative of the analog spectrum output by the decoder module with respect to the concentration value is obtained, and the Jacobian matrix is ​​obtained;

[0075] S202, the loss term is obtained by using the mean square error to obtain the difference between two Jacobian matrix vectors;

[0076] S203, the loss term forces the neural network not only to simulate the correct spectral output, but also to learn the reasons for the gradient changes, so that it conforms to the physical mechanism; when the decoder module converges under the drive of the decoding loss function, the network parameters corresponding to the decoder module are frozen; the decoding loss function is obtained by standard spectral fitting loss and Jacobian matrix consistency loss.

[0077] S300 takes the observed spectrum as input to the encoder module, and the encoder module outputs the posterior probability distribution representing the methane concentration corresponding to the observed spectrum.

[0078] S400 samples any concentration value from the posterior probability distribution and inputs the concentration value into the decoder module to perform forward simulation to generate a simulated spectrum corresponding to the concentration value; by continuously minimizing the difference between the simulated spectrum and the observed spectrum, the optimal methane column concentration inversion encoder is determined when the difference between the two is minimized.

[0079] The S500 inputs the observed spectrum into the trained encoder module, and outputs the posterior probability distribution representing the methane concentration corresponding to the observed spectrum through one forward propagation. The mean of the posterior probability distribution is determined as the final methane column concentration inversion value, and the standard deviation is used as an uncertainty measure and output.

[0080] Example 3

[0081] Firstly, please refer to Figure 1 This embodiment provides a method for methane column concentration inversion based on a physical-driven variational autoencoder, including:

[0082] Step 1: Design and firmware of the neural network decoder module;

[0083] In this embodiment, the decoder module serves as the carrier of the physical knowledge. Data from the physical simulation of the vectorized Smart Matrix Operator Method (vSmartMOM) based on the matrix operator method is used to pre-train a neural network, thus forming the decoder module of the entire inversion architecture. This neural network corresponds to the forward simulation process of vSmartMOM, first learning from the total methane column concentration, the observed geometric attitude (such as solar zenith angle, satellite zenith angle, and relative azimuth angle), and other auxiliary atmospheric and surface parameters (such as temperature profiles, pressure profiles, water vapor profiles, and surface albedo) to the corresponding simulated spectrum. The mapping; the above atmospheric radiative transfer model vSmartMOM is an existing RTM tool, which, as a mature tool, is directly referenced to support the training and validation of neural networks.

[0084] Furthermore, after the decoder module is pre-trained, all layers of the decoder network corresponding to the decoder module are marked as untrainable in the PyTorch deep learning framework, meaning that the weight parameters will not be updated in subsequent steps. Thus, the decoder module becomes a differentiable, efficient, and fixed physical simulator, used only for forward inference, and will not change due to gradient updates.

[0085] Furthermore, in this embodiment, the loss function of the decoder module is:

[0086] ;in, This represents the loss function of the decoder module. This represents the spectral fitting loss, which is typically measured using mean squared error (MSE) to measure the difference between the analog spectrum output by the decoder network and the spectrum generated by the RTM. The Jacobian matrix consistency loss requires that the physical sensitivity (gradient) reflected by the decoder module be consistent with the real physical model (RTM). and The parameters for adjusting the loss weights of the two parameters are the first parameter and the second parameter, respectively.

[0087] Furthermore, the target Jacobian matrix is ​​derived. Standard RTM tools typically output the partial derivatives of the spectrum with respect to the gas concentrations in each atmospheric layer when calculating the Jacobian matrix, i.e., J... layer However, the latent variable in this invention is a single total methane column concentration. To align the Jacobian matrix calculated by the RTM with that derived from the neural network, this embodiment includes a transformation process that uses a chain rule and a weighted summation of pressure layers to obtain its Jacobian matrix with respect to the total column concentration, which is an N... bandsA vector of size ×1, N bands This represents the number of spectral bands, and in this embodiment, it is the "truth value" label of the decoder network gradient corresponding to the training decoder module.

[0088] Furthermore, the calculation process for the model Jacobian matrix in this step is as follows: using the automatic differentiation function of a deep learning framework (such as PyTorch's autograd), the simulated spectrum output by the decoder module is directly calculated. The derivative with respect to the concentration sample c, i.e.: ;

[0089] Furthermore, the loss is calculated: the difference between the two Jacobian matrix vectors is calculated using the mean squared error (MSE), i.e.: ;in, This represents the consistency loss function of the Jacobian matrix. This represents the Jacobian matrix obtained by automatic differentiation from the decoder network. This represents the reference Jacobian matrix derived from the radiative transfer model (RTM). This loss term forces the neural network not only to simulate the correct spectral output, but also to learn the reasons for the gradient changes, i.e. how the spectrum should respond for every unit change in concentration, thus fundamentally conforming to the physical mechanism.

[0090] Furthermore, once the decoder network converges under the training driven by the loss function of the decoder module, all its network parameters (weights and biases) will be frozen. This results in an efficient, differentiable physics simulator that is consistent with the high-fidelity physics model in both output values ​​and gradients.

[0091] Step 2: Design the neural network encoder module;

[0092] The encoder module in this embodiment is a deep neural network. Its structure can effectively extract local absorption features of the spectrum using the Transformer architecture, and combine a self-attention mechanism to capture the dependencies between different bands. The input of this encoder module is the observed spectrum. After forward propagation through the network, the output is the posterior probability distribution representing the methane concentration, which can also be considered as the mean of the total methane column concentration. and standard deviation To ensure the positive definiteness of subsequent calculations, the standard deviation can be expressed as the logarithm of the variance. In the form of.

[0093] Furthermore, in this embodiment, to establish a connection between the encoder and decoder and introduce probabilistic modeling to achieve uncertainty quantification, a standard technique from variational autoencoders (the reparameterization trick) is employed. This reparameterization trick can transform the sampling process from a complex posterior probability distribution into a process of sampling from a simple standard distribution followed by a deterministic transformation through a simple mathematical transformation. Specifically, it first starts from the standard normal distribution... Sampling a random noise Then the encoder module outputs and If a specific methane concentration sample c is generated and used as input to the decoder, then the methane concentration sample c can be expressed as: ;in, and These represent the mean and standard deviation of the total methane concentration in the column, respectively. This represents random noise.

[0094] Furthermore, a dual physical loss function is constructed and the model is trained; specifically, the goal of this process is to train the encoder module to learn how to obtain a single observed spectrum. In the process, a probability distribution of the methane concentration most likely to produce the spectrum is inferred; the weights of the encoder network in the encoder module are determined by minimizing a joint loss function. Driven by the following factors, the joint loss function is composed of weighted components: ;in, Denotes the joint loss function. Indicates spectral reconstruction loss, The loss is measured by the KL divergence measure. This represents the first weight parameter. This represents the second weighting parameter.

[0095] Furthermore, regarding spectral reconstruction loss , concentration sample The input is fed into the decoder module with fixed weights to obtain the simulated spectrum. Calculate the simulated spectrum and compare it with the original input true spectrum (i.e., the observed spectrum). The MSE (mean squared error) between the two is as follows: .

[0096] In this embodiment, the methane concentration predicted by the encoder module must be able to reproduce the observed facts through a gradient-constrained physical model, thereby ensuring the physical consistency of the inversion results. The loss is the KL divergence metric, which is the standard loss term of the variational autoencoder, used to constrain the posterior probability distribution p(c|R) of the encoder module output. obsThat is, given the true observed spectrum R obs Under the given conditions, the probability distribution p(c|R) of the total methane column concentration c is... obs The distribution approximates a pre-defined prior distribution, typically a standard normal distribution N(c;μ,σ). 2 The Kullback-Leibler Divergence (KL) loss function acts as a regularization mechanism, preventing overfitting and structuring the latent variable space. Furthermore, and The parameters are used to balance the spectral loss and the weighting of the KL divergence.

[0097] Furthermore, through backpropagation and optimization algorithms (such as Adam), the weights of the encoder network in the encoder module are continuously adjusted until the joint loss function converges. After training, for actual satellite spectra, the encoder module can achieve a single forward propagation and quickly output the mean (as the best estimate) and variance (as the uncertainty) of methane concentration. In the process of continuously adjusting the weights of the encoder network, the gradient of the loss function with respect to each weight parameter of the network can be calculated through the backpropagation algorithm, and the Adam optimizer is used to combine the first and second moments of the gradient to generate an adaptive learning rate for each parameter, thereby iteratively adjusting the weights to minimize the target loss function.

[0098] Furthermore, in some application scenarios, there exists a small amount of high-quality co-location observation data (i.e., the true methane concentration is known). This embodiment is highly flexible and can seamlessly integrate this valuable supervisory information, directly constraining the mean of the encoder output to be close to the true value. Based on this, the above joint loss function can be expressed as:

[0099] ;in, Denotes the joint loss function. Indicates spectral reconstruction loss, This represents the KL divergence loss. This represents the first weight parameter. This represents the second weighting parameter. Indicates the data constraint loss, and , (•) indicates the mean square error. This represents the predicted mean of the total methane column concentration. This represents the actual observed total methane column concentration.

[0100] Step 3: Sample any concentration value from the posterior probability distribution and input the concentration value into the decoder module for forward simulation to generate a simulated spectrum corresponding to that concentration value. By continuously minimizing the difference between the simulated spectrum and the observed spectrum, the optimal methane column concentration inversion encoder is determined when the difference is minimized. Input the observed spectrum into the trained encoder module, and output the posterior probability distribution representing the methane concentration corresponding to the observed spectrum directly through one forward propagation. The mean of the posterior probability distribution is determined as the final methane column concentration inversion value, and the standard deviation is used as an uncertainty measure and output. Specifically, this step is the inversion process of the actual satellite observation spectrum. Based on the decoder module and encoder module described in this embodiment, once the encoder module is trained, it can be used for actual inversion tasks. For any new satellite observation spectrum, it only needs to be input into the trained encoder module and a fast forward propagation calculation can be performed to output the mean of the total methane column concentration as the best point estimate of the total methane column concentration, thus obtaining the methane column concentration inversion value. This process is non-iterative, so the calculation speed is extremely fast, with several orders of magnitude improvement compared to traditional optimal estimation methods.

[0101] Furthermore, please refer again. Figure 1 , combined Figure 1 Further explanation of the invention: The actual satellite observation spectrum X is input to the encoder module, which employs a Transformer architecture to extract local absorption features of the spectrum and capture the dependencies between different bands through a self-attention mechanism. After network computation, the encoder outputs the mean μ and standard deviation σ of the posterior probability distribution of the total methane column concentration, forming a normal distribution N(μ,σ). In this process, optional data constraints can be introduced, i.e., using the known actual methane concentration to supervise and constrain the mean of the encoder output, thereby improving the model's accuracy and stability. Subsequently, a reparameterization method is used to sample the concentration variable z from N(μ,σ), which is the methane column concentration, mathematically expressed as: , , This represents random noise. The concentration variable z, along with an auxiliary variable, is input into the decoder module. The decoder consists of a neural network pre-trained and with frozen parameters based on simulated data from the Atmospheric Radiative Transfer Model (RTM). It is used to achieve a positive mapping from methane concentration to spectrum, outputting the simulated spectrum corresponding to that concentration. =d(z). Two types of physical constraints are set during the decoding process: (1) the sensitivity of the spectrum to methane concentration, that is, the Jacobian matrix of the decoder must be consistent with the reference Jacobian matrix of the RTM output; (2) the consistency between the actual spectrum and the simulated spectrum, that is, by minimizing the difference between the satellite observation spectrum and the spectrum generated by the decoder, the physical consistency of the inversion result is guaranteed. Through the above process, the present invention can quickly obtain the total methane column concentration from the satellite observation spectrum while maintaining physical consistency. The mean μ of the output is used as the best estimate of the methane column concentration, while the standard deviation σ reflects the uncertainty of the inversion result.

[0102] Secondly, the present invention provides a methane column concentration inversion system based on a physical-driven variational autoencoder, comprising:

[0103] Decoder determination module, used to determine the decoder module;

[0104] Encoder determination module, used to determine the encoder module;

[0105] The decoder training module is used to pre-train the decoder module using a loss function that includes Jacobian matrix consistency constraints, and freezes the network weights of all its layers after training convergence.

[0106] The posterior probability distribution determination module is used to take the observed spectrum as input to the encoder module, and the encoder module outputs the posterior probability distribution representing the methane concentration corresponding to the observed spectrum.

[0107] The inversion value determination module samples any concentration value from the posterior probability distribution and inputs the concentration value into the decoder module for forward simulation to generate a simulated spectrum corresponding to that concentration value. By continuously minimizing the difference between the simulated spectrum and the observed spectrum, the optimal methane column concentration inversion encoder is determined when the difference is minimized. The observed spectrum is then input into the trained encoder module. Through one forward propagation, the posterior probability distribution representing the methane concentration corresponding to the observed spectrum is output, and the mean of the posterior probability distribution is determined as the final methane column concentration inversion value. The standard deviation is used as an uncertainty measure and is also output.

[0108] In the description of the above embodiments, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0109] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for methane column concentration inversion based on a physical-driven variational autoencoder, characterized in that, include: Identify the decoder module and encoder module; The decoder module is pre-trained using a loss function that includes Jacobian matrix consistency constraints, and the network weights of all its layers are frozen after training converges. The encoder module is used to extract the absorption features of the spectrum and capture the dependencies between different bands; The encoder module takes the observed spectrum as input and outputs the posterior probability distribution representing the methane concentration corresponding to the observed spectrum. Any concentration value is sampled from the posterior probability distribution, and the concentration value is input into the decoder module to perform forward simulation to generate a simulated spectrum corresponding to the concentration value; by continuously minimizing the difference between the simulated spectrum and the observed spectrum, the optimal methane column concentration inversion encoder is determined when the difference between the two is minimized, and the observed spectrum is input into the encoder module after training. The posterior probability distribution representing the methane concentration corresponding to the observed spectrum is output through a forward propagation, and the mean of the posterior probability distribution is determined as the final methane column concentration inversion value, while the standard deviation is used as an uncertainty measure and output. The decoder module includes: A neural network is identified and trained using data from a physical simulation of an atmospheric radiative transfer model, thereby forming the decoder module of the inversion architecture. The neural network corresponds to the forward simulation process of the atmospheric radiative transfer model, which learns the mapping from total methane column concentration, observed geometric attitude, atmospheric and surface parameters to the corresponding simulated spectrum. Determining the encoder module includes: The encoder module uses a Transformer architecture to extract local absorption features of the spectrum and combines a self-attention mechanism to capture the dependencies between different bands. The encoder module takes the observed spectrum as input, passes it through the network's forward propagation, and outputs the mean and standard deviation of the total methane column concentration. The Jacobian matrix is ​​obtained by using the automatic differentiation function of the deep learning framework to obtain the derivative of the simulated spectrum output by the decoder module with respect to the concentration value. The loss term is obtained by using the mean squared error to obtain the difference between two Jacobian matrix vectors. The loss term forces the neural network to simulate the correct spectral output and learns the reasons for gradient changes to conform to the physical mechanism; when the decoder module converges under the drive of the decoding loss function, the network parameters corresponding to the decoder module are frozen.

2. The methane column concentration inversion method based on a physical-driven variational autoencoder according to claim 1, characterized in that, The observed geometric attitude includes solar zenith angle, satellite zenith angle, and relative azimuth angle; the atmospheric and surface parameters include temperature profile, pressure profile, water vapor profile, and surface albedo.

3. The methane column concentration inversion method based on a physical-driven variational autoencoder according to claim 2, characterized in that, The decoding loss function is obtained by using standard spectral fitting loss and Jacobian matrix consistency loss.

4. The methane column concentration inversion method based on a physical-driven variational autoencoder according to claim 3, characterized in that, After training the encoder module, the encoder module infers a posterior probability distribution of the methane concentration that most likely produced the observed spectrum from an observed spectrum.

5. The methane column concentration inversion method based on a physical-driven variational autoencoder according to claim 4, characterized in that, The weights of the encoder network corresponding to the encoder module are driven by a joint loss function, which is composed of at least a weighted sum of spectral reconstruction loss and KL divergence loss.

6. The methane column concentration inversion method based on a physical-driven variational autoencoder according to claim 5, characterized in that, The joint loss function is obtained through spectral reconstruction loss, KL divergence loss, and data constraint loss; the joint loss function is expressed as follows: ； in, Denotes the joint loss function. Indicates spectral reconstruction loss, This represents the KL divergence loss. This represents the first weight parameter. This represents the second weighting parameter. Indicates the data constraint loss, and , This represents the mean square error. This represents the predicted mean of the total methane column concentration. This represents the actual observed total methane column concentration.

7. A methane column concentration inversion system based on a physical-driven variational autoencoder, characterized in that, The methane column concentration inversion method based on a physical-driven variational autoencoder according to any one of claims 1 to 6 includes: Decoder determination module, used to determine the decoder module; Encoder determination module, used to determine the encoder module; A decoder training module is used to pre-train the decoder module using a loss function that includes a Jacobian matrix consistency constraint, and to freeze the network weights of all its layers after training convergence. A posterior probability distribution determination module is used to take the observed spectrum as input to the encoder module, and the encoder module outputs the posterior probability distribution representing the methane concentration corresponding to the observed spectrum. The inversion value determination module is used to sample any concentration value in the posterior probability distribution, input the concentration value into the decoder module for forward simulation to generate a simulated spectrum corresponding to the concentration value; by continuously minimizing the difference between the simulated spectrum and the observed spectrum, the optimal methane column concentration inversion encoder is determined when the difference is minimal, and the observed spectrum is input into the trained encoder module; the posterior probability distribution representing the methane concentration corresponding to the observed spectrum is output through one forward propagation, and the mean of the posterior probability distribution is determined as the final methane column concentration inversion value, and the standard deviation is output as an uncertainty measure.

Images