High-dimensional nonstationary variational inference method

Abstract

The application discloses a variational inference system combining fractional gradient flow and mixed entropy regularization. By introducing Caputo fractional time derivative to describe the non-stationary evolution of variational distribution, mixed entropy regularization term is constructed by using Rényi entropy and Fisher information functional, and fractional variational lower bound is derived. The application solves the problems of multi-modal inference and long-time memory modeling in high-dimensional non-stationary posterior distribution, and can be applied to fields such as extreme event prediction, gene regulation network and radar countermeasure.

Description

Technical Field

[0001] This invention relates to the fields of machine learning and Bayesian inference, and more specifically to a high-dimensional non-stationary variational inference method. Background Technology

[0002] Traditional Bayesian inference often fails when dealing with high-dimensional, non-stationary, and strongly coupled random fields due to the multimodal, heavy-tailed, and geometrically singular posterior distributions. Existing variational inference methods, mostly based on integer-order gradient descent and Kullback-Leibler divergence regularization, cannot effectively capture long-term memory effects and fractional-order diffusion behavior in non-Markov dynamics. Furthermore, the standard variational lower bound (ELBO) converges to local spurious extrema when the data likelihood is highly oscillating, causing inference biases that cannot be controlled by existing error bounds. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention aims to provide a high-dimensional non-stationary variational inference method based on a variational inference framework of fractional gradient flow and mixed entropy regularization. By embedding the variational distribution into a dynamical system in a Wasserstein space with fractional time derivatives and introducing a novel mixed entropy functional, adaptive approximation of non-stationary posterior geometry is achieved. The core of this invention lies in: (1) constructing the evolution equation of fractional variational gradient flow; (2) defining and calculating the mixed entropy regularization term; and (3) realizing non-stationary variational parameter updates in a high-dimensional parameter space. This method is mainly applied to extreme event probability prediction in climate-economic coupled systems; non-stationary dynamic inference in high-dimensional genome regulatory networks; blind estimation of clutter statistical distributions in cognitive radar countermeasure environments; and implicit state filtering with random volatility and fractional noise in financial engineering.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a high-dimensional non-stationary variational inference method, comprising the following steps: Step 1, modeling the time evolution of the variational distribution as a continuous equation with a Caputo fractional time derivative; Step 2, adding a hybrid entropy regularization term consisting of a linear combination of Rényi entropy and Fisher information functional to the free energy functional; Step 3, using Grünwald-Letnikov discretization to update the parameters, wherein each update step depends on the weighted average of all historical moments.

[0005] As a further improvement to the present invention, in step one, the time evolution of the variational distribution is modeled as a continuity equation with a Caputo fractional time derivative, specifically as follows: Let... As a latent variable, Define the variational distribution for the observed data sequence. The evolution follows a gradient flow guided by the following fractional Langevin equation: in: For free energy functionals about The first variational derivative of ; For Caputo-type fractional-order time derivative operators, the definition is: This operator introduces a power-law memory kernel containing historical gradient information, making the variational update have long-term dependencies. This is the Gamma function.

[0006] As a further improvement to the present invention, step two involves adding a hybrid entropy regularization term, consisting of a linear combination of Rényi entropy and Fisher information functional, to the free energy functional, as follows: Constructing the hybrid entropy functional : Where: the first term is the Rényi entropy, order. The second term is a variant of the Fisher information functional (multiplied by the parameter) used to enhance sensitivity to low-probability regions. ), used to control the smoothness and geometric curvature of the probability density; the final optimization objective is the fractional variational free energy: in This is a temperature-type regularization coefficient.

[0007] As a further improvement to the present invention, the parameter update in step three using Grünwald-Letnikov discretization is as follows: assuming the variational distribution is a Gaussian mixture. The mixing coefficient satisfy The fractional variational parameter update equation is: in: Similarly, this is the Caputo fractional derivative, applied to the time trajectory of each parameter. This refers to the step size hyperparameter. This represents the matrix second derivative with respect to the covariance matrix. Representing the Hadamard product; using the Grünwald-Letnikov approximation to discretize the fractional derivative: in, For time step, The coefficients are generalized binomial coefficients, making the current parameter update dependent on the parameter values ​​at all historical moments, with the weights exhibiting power-law decay. .

[0008] As a further improvement of the present invention, step two also includes the step of constructing the fractional variational free energy and deriving its corresponding fractional variational lower bound as the theoretical quantitative form of the optimization objective, specifically as follows: the fractional variational lower bound is defined as: in: For fractional integral operators, it is defined as: ; The Kullback-Leibler divergence; It is the Hessian determinant logarithm of the mixed entropy, used to automatically adjust the geometric complexity of the variational distribution.

[0009] The beneficial effects of this invention are as follows: By modeling the temporal evolution of the variational distribution as a continuous equation with a Caputo fractional time derivative, a power-law memory kernel containing historical gradient information is introduced, overcoming the Markov shortsightedness of traditional integer-order gradient descent and effectively capturing long-term memory effects and fractional-order diffusion behavior in non-stationary dynamics. By adding a mixed entropy regularization term composed of Rényi entropy and Fisher information functional to the free energy functional, the sensitivity to low-probability regions of the posterior distribution and the control over distribution smoothness are enhanced, enabling accurate approximation of multimodal, heavy-tailed, and geometrically singular high-dimensional posterior distributions. By employing Grünwald-Letnikov discretization for parameter updates, each update incorporates weighted information from all historical moments, ensuring the continuity and stability of parameter evolution in non-stationary scenarios and avoiding the problem of convergence to local pseudo-extrema in traditional methods when data likelihood oscillates significantly. Furthermore, the derived fractional-order variational lower bound provides strict theoretical error control, further improving the reliability of the inference results. Subsequent improvements clarified the specific form of the fractional Langevin equation, the mathematical definition of the mixed entropy functional, the computable parameter update rules under Gaussian mixture distribution, and the theoretically quantified optimization objective, which together constitute a complete and efficient high-dimensional nonstationary variational reasoning framework. Detailed Implementation

[0010] The present invention will be further described in detail below with reference to the given embodiments.

[0011] This embodiment of a high-dimensional nonstationary variational inference method includes the following steps: Step 1, modeling the temporal evolution of the variational distribution as a continuous equation with a Caputo fractional time derivative; Step 2, adding a hybrid entropy regularization term, consisting of a linear combination of Rényi entropy and Fisher information functional, to the free energy functional; Step 3, using Grünwald-Letnikov discretization to update parameters, where each update step depends on a weighted average of all historical moments. First, the evolution of the variational distribution is extended from the traditional integer-order gradient flow to a dynamical system in a fractional-order Wasserstein space, describing its temporal evolution through a continuous equation; then, a hybrid entropy regularization term is introduced based on the traditional free energy functional to balance the accuracy of posterior approximation with the geometric properties of the distribution; finally, the continuous fractional-order evolution equation is discretized into numerically computable parameter update rules, achieving adaptive adjustment of nonstationary parameters through a historical weighting mechanism. This process captures long-range dependencies in non-stationary data through fractional-order time derivatives, solves the modal collapse problem of multimodal posterior through hybrid entropy regularization, and ensures the stability of parameter evolution through historical weighted updates. It fundamentally solves the three core defects of traditional variational inference methods in high-dimensional non-stationary scenarios proposed in the background technology, and achieves high-precision approximation of complex non-stationary posterior distributions.

[0012] Furthermore, in step one, the time evolution of the variational distribution is modeled as a continuity equation with a Caputo fractional time derivative, specifically as follows: Let... As a latent variable, Define the variational distribution for the observed data sequence. The evolution follows a gradient flow guided by the following fractional Langevin equation: in: For free energy functionals about The first variational derivative of ; For Caputo-type fractional-order time derivative operators, the definition is: This operator introduces a power-law memory kernel containing historical gradient information, making the variational update have long-term dependencies. This is the Gamma function. The evolution of the variational distribution is combined with the gradient and fractional diffusion term of the free energy functional using the fractional Langevin equation. The Caputo fractional derivative, by integrating the derivative information from historical moments, endows the variational update with a long-term memory of past gradients. This step helps solve the problem that traditional methods cannot model non-Markovian dynamics, further improving the inference accuracy for long-term time series data and effectively capturing the long-range correlations prevalent in fields such as climate and finance.

[0013] Furthermore, in step two, a mixed entropy regularization term, consisting of a linear combination of Rényi entropy and Fisher information functional, is added to the free energy functional, specifically as follows: Constructing the mixed entropy functional : Where: the first term is the Rényi entropy, order. The second term is a variant of the Fisher information functional (multiplied by the parameter) used to enhance sensitivity to low-probability regions. ), used to control the smoothness and geometric curvature of the probability density; the final optimization objective is the fractional variational free energy: in This is a temperature-type regularization coefficient. The Rényi entropy is adjusted by changing the order. This approach is more sensitive than traditional KL divergence in detecting low-probability regions of the posterior distribution, avoiding the omission of secondary modes in multimodal distributions. Fisher's information functional, by penalizing drastic changes in probability density, ensures the smoothness of the variational distribution, preventing overfitting to observation noise. This step helps solve the mode collapse problem caused by traditional KL divergence regularization, further improving the geometric fidelity of the posterior distribution and enabling a more accurate characterization of the multimodal structure and heavy-tailed properties of the posterior distribution.

[0014] Furthermore, in step three, the Grünwald-Letnikov discretization is used to update the parameters, as follows: Assume the variational distribution is a Gaussian mixture. The mixing coefficient satisfy The fractional variational parameter update equation is: in: Similarly, this is the Caputo fractional derivative, applied to the time trajectory of each parameter. This refers to the step size hyperparameter. This represents the matrix second derivative with respect to the covariance matrix. Representing the Hadamard product; using the Grünwald-Letnikov approximation to discretize the fractional derivative: in, For time step, The coefficients are generalized binomial coefficients, making the current parameter update dependent on the parameter values ​​at all historical moments, with the weights exhibiting power-law decay. For the Gaussian mixture variational distribution commonly used in engineering, fractional-order update equations for the mean and covariance matrices are derived. The continuous fractional derivatives are transformed into a discrete historical weighted sum form through the Grünwald-Letnikov approximation. This ensures that each parameter update incorporates information from all historical parameters from the initial time step to the current time step, with the weights decaying power-lawfully over time. This step helps solve the computability problem of fractional-order equations in high-dimensional parameter spaces, additionally ensuring the numerical stability of parameter updates and avoiding inference failures caused by drastic parameter oscillations in non-stationary scenarios, as seen with integer-order methods.

[0015] Furthermore, step two also includes constructing the fractional variational free energy and deriving its corresponding fractional variational lower bound as a theoretical quantitative form of the optimization objective, specifically as follows: The fractional variational lower bound is defined as: in: For fractional integral operators, it is defined as: ; The Kullback-Leibler divergence; The Hessian determinant logarithm of the mixed entropy is used to automatically adjust the geometric complexity of the variational distribution. By introducing a fractional integral operator to handle the KL divergence term between the prior and the variational distribution, the traditional ELBO is extended to a fractional variational lower bound. Simultaneously, the Hessian determinant logarithm of the mixed entropy is used to automatically adjust the geometric complexity of the variational distribution. This step helps solve the problem of standard ELBO converging to local spurious extrema when the data likelihood is highly oscillating, and additionally provides strict theoretical error control, enabling the bias of the inference results to be effectively quantified and constrained.

[0016] In summary, this invention employs a technical solution that models the temporal evolution of the variational distribution as a Caputo fractional continuity equation, incorporates a hybrid entropy regularization term consisting of a linear combination of Rényi entropy and Fisher information functionals, uses Grünwald-Letnikov discretization to achieve parameter updates dependent on the weighted average of all historical moments, and derives a fractional variational lower bound as a theoretical quantification target. This solution addresses the problems of traditional variational inference methods in high-dimensional non-stationary scenarios, such as the inability to capture long-term memory effects, easy convergence to local pseudo-extremes, and difficulty in approximating multimodal heavy-tailed posterior distributions. It achieves adaptive high-precision approximation of non-stationary posterior geometric structures and can be widely applied in various fields, including extreme event prediction in climate-economic coupled systems, non-stationary dynamic inference in high-dimensional genome regulatory networks, blind estimation of clutter statistical distributions in cognitive radar countermeasure environments, and implicit state filtering with random volatility and fractional noise in financial engineering.

[0017] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A high-dimensional nonstationary variational inference method, characterized in that: Includes the following steps: Step 1: Model the time evolution of the variational distribution as a continuity equation with Caputo fractional time derivatives; Step 2: Add a mixed entropy regularization term, which is a linear combination of Rényi entropy and Fisher information functional, to the free energy functional; Step 3: The Grünwald-Letnikov discretization is used to update the parameters, where each update depends on the weighted average of all historical time points.

2. The high-dimensional non-stationary variational inference method according to claim 1, characterized in that: In step one, the time evolution of the variational distribution is modeled as a continuity equation with a Caputo fractional time derivative, as follows: Let... As a latent variable, Define a variational distribution for the observed data sequence. The evolution follows a gradient flow guided by the following fractional Langevin equation: in: For free energy functionals about The first variational derivative of ; For Caputo-type fractional-order time derivative operators, the definition is: This operator introduces a power-law memory kernel containing historical gradient information, making the variational update have long-term dependencies. This is the Gamma function.

3. The high-dimensional non-stationary variational inference method according to claim 2, characterized in that: In step two, the addition of a hybrid entropy regularization term, consisting of a linear combination of Rényi entropy and Fisher information functional, to the free energy functional is as follows: Constructing the hybrid entropy functional : Where: the first term is the Rényi entropy, order. The first term is used to enhance sensitivity to low-probability regions, and the second term is a variant of the Fisher information functional (multiplied by the parameter). ), used to control the smoothness and geometric curvature of the probability density; The ultimate optimization objective is the fractional variational free energy: in This is a temperature-type regularization coefficient.

4. The high-dimensional non-stationary variational inference method according to claim 3, characterized in that: In step three, the parameter update is achieved using Grünwald-Letnikov discretization as follows: Assume the variational distribution is a Gaussian mixture form. The mixing coefficient satisfy The fractional variational parameter update equation is: in: Similarly, this is the Caputo fractional derivative, applied to the time trajectory of each parameter. This refers to the step size hyperparameter. This represents the matrix second derivative with respect to the covariance matrix. Represents the Hadamard product; The Grünwald-Letnikov approximation is used to discretize the fractional derivative: in, For time step, The coefficients are generalized binomial coefficients, making the current parameter update dependent on the parameter values ​​at all historical moments, with the weights exhibiting power-law decay. .

5. The high-dimensional non-stationary variational inference method according to claim 4, characterized in that: Step two also includes constructing the fractional-order variational free energy and deriving its corresponding fractional-order variational lower bound as the theoretical quantitative form of the optimization objective, as detailed below: Define the lower bound of fractional variational evidence as: in: For fractional integral operators, it is defined as: ; The Kullback-Leibler divergence; It is the Hessian determinant logarithm of the mixed entropy, used to automatically adjust the geometric complexity of the variational distribution.

Images