A neural network optimization method and system based on thermodynamic dissipation constraint

Abstract

The present application relates to a kind of neural network optimization method and system based on thermodynamic dissipation constraint, method steps include: obtaining the current network parameter of the neural network to be optimized, training data is input into neural network, and loss function gradient is calculated;Approximation method is used to estimate the approximate inverse of Fisher information matrix, and global geometric gradient norm is calculated in combination with loss function gradient;Thermodynamic regulator is according to global geometric gradient norm, set dissipation budget is dynamically current thermodynamic adjustment step size;Network parameter is updated based on current thermodynamic adjustment step size;If Fisher information matrix is full rank, then use natural gradient direction to update network parameter;If Fisher information matrix is not full rank, then network parameter uses hybrid update strategy.Compared with prior art, the present application realizes the fundamental trade-off between learning speed and computational cost, while ensuring convergence performance, irreversible energy dissipation is minimized.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and machine learning technology, specifically to optimization algorithms in neural network training, and particularly to a natural gradient descent optimization method and system that combines information geometry and thermodynamic constraints to achieve high efficiency and energy saving. Background Technology

[0002] Limitations of Traditional Optimizers: Current mainstream neural network optimizers, such as stochastic gradient descent (SGD) and its variants (e.g., Adam), update their parameters based on Euclidean geometry. This means they assume the parameter space is flat. However, for parameterized probabilistic models (such as neural networks), the geometric structure implicit in their loss function is defined by the Fisher information matrix, which is a Riemannian manifold. Ignoring this geometric structure can lead to inefficient update directions, slow convergence, and, physically, unnecessary "shortcuts" and higher energy dissipation.

[0003] Challenges of Natural Gradient Descent (NGD): Natural Gradient Descent (NGD) pre-adjusts the gradient by using the inverse of the Fisher information matrix, ensuring the update direction follows the fastest descent direction on the manifold of the probability distribution, which is theoretically superior. However, its practical application faces three core challenges: (a) High computational complexity: Accurate computation and inversion of the Fisher matrix in large-scale neural networks are computationally infeasible; (b) Numerical instability: Fisher matrices are often ill-conditioned or lack rank, and direct inversion can lead to an explosion in update step size or incorrect update direction; (c) Lack of physical constraints: Existing methods do not link the optimization process to the energy consumption (or thermodynamic dissipation) of the physical system, failing to provide a controllable tradeoff between learning speed and computational cost / energy consumption.

[0004] Based on the summary of the shortcomings of existing technologies, there is an urgent need in this field for an optimization method that can accurately follow the geometric structure of the model information, is numerically stable and computationally efficient, and can explicitly control the energy consumption of the optimization process. Summary of the Invention

[0005] The purpose of this invention is to solve the problems existing in current neural network optimization and to provide a neural network optimization method and system based on thermodynamic dissipation constraints.

[0006] The objective of this invention can be achieved through the following technical solutions: As a first aspect of the present invention, a neural network optimization method based on thermodynamic dissipation constraints is provided, the method comprising the following steps: Obtain the current network parameters of the neural network to be optimized, input the training data into the neural network, and calculate the gradient of the loss function; An approximate method is used to estimate the approximate inverse of the Fisher information matrix, and the global geometric gradient norm is calculated by combining the gradient of the loss function. The thermodynamic regulator dynamically adjusts the set dissipation budget to the current thermodynamic adjustment step size based on the global geometric gradient norm. The network parameters are updated based on the current thermodynamic adjustment step size. If the Fisher information matrix is ​​full rank, the network parameters are updated using the natural gradient direction. If the Fisher information matrix is ​​not full rank, the network parameters are updated using a hybrid update strategy, which uses the pseudo-inverse of the Fisher information matrix in the value domain and performs Euclidean updates in the null space direction.

[0007] As a preferred technical solution, the estimation methods for the inverse approximation of the Fisher information matrix include: eigenvalue truncation method, Tikhonov damping method, and Kronecker factor approximation.

[0008] As a preferred technical solution, the Kronecker factor approximation is specifically implemented as follows: Update the input covariance matrix of each linear and convolutional layer in the current neural network using exponential moving average. With the output gradient covariance matrix ; Check if the set number of steps has been reached; if so, calculate and cache the eigenvalues ​​and pseudo-inverses of the input covariance matrix A and the output gradient covariance matrix G; otherwise, repeat the eigenvalues ​​and pseudo-inverses. Using the pseudo-inverse of the cache, the pre-adjusted gradient direction of each layer is calculated; and the global geometric gradient norm is calculated and substituted into the thermodynamic regulator to calculate the current thermodynamic adjustment step size.

[0009] As a preferred technical solution, the thermodynamic regulator calculates the thermodynamic adjustment step size as follows:

[0010]

[0011] in, Thermodynamic adjustment step size; For dissipation budget; For the global geometric gradient norm, This is the inverse approximation of the Fisher information matrix. The gradient of the loss function; To prevent division by zero of small constants; For the clipping function, Trim the boundary for the step size.

[0012] As a preferred technical solution, when the Fisher information matrix of a certain layer in the network is not full rank, the network parameters adopt a hybrid update strategy, adding a spatial Euclidean update term to the network parameters of that layer:

[0013] in, Thermodynamic adjustment step size; It is the Moore-Penrose pseudoinverse, used to perform minimal-dissipation natural gradient updates within the range of the Fisher information matrix; The gradient of the loss function; It is the projection matrix onto the null space; The learning rate is used to control the proportion of the step size for null space updates. This is the zero-space rollback ratio.

[0014] As a preferred technical solution, each time the training iteration of network parameters is completed, the dissipation ratio is calculated, which is the ratio of the predicted dissipation to the minimum dissipation estimated by the current approximation method. When the dissipation ratio exceeds a set threshold, switch to a higher-precision approximation method to estimate the approximate inverse of the Fisher information matrix.

[0015] As a second aspect of the present invention, a neural network optimization system based on thermodynamic dissipation constraints is provided, the system performing the neural network optimization method based on thermodynamic dissipation constraints as described above, including: Gradient calculation module: Calculates the gradient of the loss function through forward and backward propagation; Geometric Information Estimation Module: Estimates and updates the approximate inverse of the Fisher information matrix using an approximate method; Pre-adjuster solution module: Uses the approximate inverse pre-adjustment gradient of the Fisher information matrix and calculates the pre-adjusted gradient direction; Thermodynamic regulator module: Calculates the global geometric gradient norm and, in conjunction with the set dissipation budget, calculates the current thermodynamic regulation step size; Parameter update module: Updates network parameters based on the current thermodynamic adjustment step size; Monitoring and Decision Module: Records training metrics and calculates the dissipation ratio. When the dissipation ratio exceeds a set threshold, it switches to a higher-precision approximation method to update the approximate inverse of the Fisher information matrix.

[0016] As a preferred technical solution, the geometric information estimation module uses the Kronecker factor to approximately update the inverse of the Fisher information matrix, specifically as follows: Update the input covariance matrix of each linear and convolutional layer in the current neural network using exponential moving average. With the output gradient covariance matrix ; Check if the set number of steps has been reached; if so, calculate and cache the eigenvalues ​​and pseudo-inverses of the input covariance matrix A and the output gradient covariance matrix G; otherwise, repeat the eigenvalues ​​and pseudo-inverses.

[0017] As a preferred technical solution, the thermodynamic regulator module calculates the thermodynamic adjustment step size as follows:

[0018]

[0019] in, Thermodynamic adjustment step size; For dissipation budget; For the global geometric gradient norm, This is the inverse approximation of the Fisher information matrix. The gradient of the loss function; To prevent division by zero of small constants; For the clipping function, Trim the boundary for the step size.

[0020] As a preferred technical solution, the parameter update module updates the network parameters as follows: If the Fisher information matrix is ​​full rank, the network parameters are updated using the natural gradient direction. If the Fisher information matrix is ​​not of full rank, the network parameters adopt a hybrid update strategy, using the pseudo-inverse of the Fisher information matrix in the value domain and Euclidean updates in the null space direction.

[0021] Compared with the prior art, the present invention has the following beneficial effects: 1) The neural network optimization method based on thermodynamic dissipation constraints proposed in this invention internalizes thermodynamic constraints: by directly mapping the preset energy budget as a hyperparameter through the regulator, a fundamental trade-off between learning speed and computational cost is achieved.

[0022] 2) The optimization method proposed in this invention has geometrically aware adaptability. The thermodynamic adjustment step size is dynamically determined by the geometric norm, rather than preset or simply decayed. When the geometric gradient is steep, the step size is automatically reduced to control dissipation; otherwise, the step size is increased to accelerate learning, so that the update is more in line with the local geometry of the loss surface.

[0023] 3) This invention also proposes a robust and scalable architecture: singularity is handled by hybrid update. When the rank of the Fisher information matrix is ​​insufficient, it is updated in the null space using small-step Euclidean algorithm to ensure that the optimization does not stagnate. Combined with K-FAC cached feature decomposition, intermediate steps are reused to reduce computational overhead. And through a monitoring-switching mechanism, the approximate quality is guaranteed, forming an adaptive closed-loop system. Attached Figure Description

[0024] Figure 1 This is a flowchart of a thermodynamic natural gradient descent method according to the present invention.

[0025] Figure 2 The figure shows a comparison of the training results of the present invention and existing methods; a) is the training loss, b) is the validation accuracy (rounds), c) is the validation loss (rounds), d) is the prediction error, e) is the learning rate, and f) is the gradient norm.

[0026] Figure 3 This is a comparison diagram between the present invention and existing methods; a) is the final verification accuracy, b) is the optimal verification accuracy, c) is the convergence speed (number of rounds to reach the optimal accuracy), and d) is the average prediction error. Detailed Implementation

[0027] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0028] Example 1 To address the shortcomings of traditional neural network optimizer techniques, this invention proposes a method and system based on Thermodynamic Natural Gradient Descent (NGD-T). The core idea is to model the neural network training process as a physical process constrained by the second law of thermodynamics. A "thermodynamic regulator" transforms the user-specified energy (dissipation) budget into the optimal update step size for each step, thereby minimizing irreversible energy dissipation while ensuring convergence performance. The principle of the proposed NGD-T algorithm framework is as follows: Figure 1 As shown.

[0029] The theoretical basis of the thermodynamic natural gradient descent NGD-T proposed in this invention originates from the entropy action. Variational principle: The entropy action derives the Fisher information measure of the parameter space. This metric defines the Riemannian geometry of the model's probability distribution.

[0030] Parameter update speed Instantaneous dissipation is defined as .

[0031] Solving for the fixed loss rate of decrease constraint ( The variational problem of minimizing instantaneous dissipation.

[0032] The analytical solution is obtained: the optimal update direction is the natural gradient flow. And the theoretical minimum dissipation lower bound is derived. .

[0033] Based on this, the main loop of the thermodynamic natural gradient descent NGD-T algorithm is shown below, which discretizes the continuous theory into executable optimization steps: Input and gradient calculation: Obtain the current network parameters to be optimized. and loss gradient .

[0034] Geometric information processing: This involves estimating or calling an approximate inverse of the Fisher information matrix using approximate methods. And calculate the global geometric gradient norm of the key scalar. .

[0035] Update direction decision: If the Fisher information matrix is ​​full rank, the pure natural gradient direction is used. .

[0036] If the Fisher information matrix is ​​not of rank, enable hybrid update: use the pseudo-inverse in the range. And use small-step Euclidean updates in the zero space to ensure that optimization does not stagnate.

[0037] Thermodynamic regulation: via regulator formula The user-defined dissipation budget The dynamic mapping is set to the optimal step size. When the step size is large (geometrically steep), the step size is automatically reduced to control costs; When the time is short, increase the step size to accelerate learning.

[0038] Parameter update and output: Execute the update of network parameters. This completes one iteration.

[0039] I. Core theoretical foundation (variational principle based on entropy action).

[0040] Define instantaneous dissipation as: in parameter space, given the parameter update rate (direction vector) Its instantaneous irreversible dissipation under information geometry is defined as a quadratic form: ,in It is the Fisher information matrix. This definition is derived from the analogy of entropy production rate in statistical physics.

[0041] Construct a constrained optimization problem: At each step, we require the loss function to operate at a fixed rate. The descent, i.e., the constraint condition is: .

[0042] Derivation of the optimal direction: Solving the variational problem of minimizing instantaneous dissipation under a fixed rate of decrease in loss:

[0043] Using the Lagrange multiplier method, the unique optimal solution (the direction with minimum dissipation) is: This is precisely the direction of the natural gradient flow (NGF) in continuous time. Simultaneously, the lower bound for minimum dissipation is obtained as: .

[0044] II. Thermodynamic regulator.

[0045] To achieve discretization and control dissipation, this invention designs a thermodynamic regulator that transforms the theoretical lower bound into an operable step size selection strategy.

[0046] Calculate the global geometric gradient norm: This scalar characterizes the steepness of the gradient direction under the current geometry.

[0047] Step size mapping formula: The user sets a dissipation budget. (For example, the maximum energy consumption allowed per update step). To minimize the predicted dissipation from this update. Exactly equal to The theoretical step size can be solved. ( (Base step size).

[0048] Stabilizer with clipping: To ensure numerical stability, the final step size formula is as follows:

[0049] in, Thermodynamic adjustment step size; To prevent division by zero of small constants; For the clipping function, Trim the boundary for the step size. This adjuster is key to achieving the speed-cost tradeoff in this invention: when the geometric gradient is steep ( When the step size is large, the step size is automatically reduced to control dissipation; conversely, the step size is increased to accelerate learning.

[0050] III. Numerically stable Fisher inverse approximation method.

[0051] Due to the inverse of the accurate Fisher information matrix This invention employs and improves upon various approximate inversion techniques, ensuring compatibility with regulators.

[0052] Eigenvalue truncation method: for approximate Fisher information matrix Perform eigenvalue decomposition Set a relative threshold. Only retain eigenvalues The portion that constitutes the truncated reverse: .

[0053] Tikhonov damping method: using As an inverse approximation. Where the damping coefficient... It not only improves numerical stability, but also has a physical meaning within the framework of this invention: it is equivalent to introducing a viscosity into the system to prevent unstable and high-dissipation updates in regions with extremely high geometric curvature (corresponding to ill-conditioned Fisher information matrices).

[0054] Kronecker Factor Approximation (K-FAC) Ensemble: For large-scale neural networks, an efficient Kronecker factor approximation K-FAC method is employed. For each linear or convolutional layer in the network, its Fisher matrix is ​​approximated as the Kronecker product of two smaller matrices: ,in It is the covariance of the input (or input block). It is the covariance of the output gradient. Its inverse can be efficiently calculated as:

[0055] This invention is further optimized: each Recalculate and cache only after a step (e.g., K=50). and The feature decomposition can be reused in intermediate steps, which greatly reduces computational overhead.

[0056] IV. Hybrid update strategy for Fisher information matrix with insufficient rank.

[0057] When the Fisher matrix has insufficient rank (singular or nearly singular), its inverse or pseudo-inverse cannot provide an update direction in the null space. Therefore, this invention proposes a hybrid update strategy:

[0058] in, It is the Moore-Penrose pseudoinverse, the generalized inverse of a singular or rank-deficient matrix, which provides a least-squares solution for performing a minimal-dissipation natural gradient update within the range of the Fisher information matrix. It is the projection matrix onto the null space; , This is the zero-space rollback ratio, used to control the step size ratio of zero-space updates. A small learning rate, such as 0.001, allows for slow, controlled Euclidean descent in the null space direction, ensuring the optimization process is not completely blocked. This strategy balances geometric optimality and algorithmic completeness.

[0059] V. Dynamic Approximate Switching and Monitoring System.

[0060] To ensure long-term optimized geometric fidelity, this invention introduces a monitoring and switching mechanism: Dissipation ratio monitoring: Real-time calculation and monitoring of dissipation ratio.

[0061] in, To predict dissipation, the denominator uses the minimum dissipation estimated by the current approximation method.

[0062] Dynamic switching: Set a threshold (e.g., 1.5). When After several steps, it indicates that the currently used low-precision Fisher approximation (such as the diagonal empirical Fisher) introduces excessive geometric bias, causing the actual update direction to deviate from the optimal path and generating additional dissipation. The system will automatically switch to a higher-precision approximation method (such as K-FAC) to restore optimized thermodynamic efficiency.

[0063] This invention can be directly applied to various scenarios requiring neural network training, including but not limited to: computer vision (image classification, object detection), natural language processing (machine translation, text generation), scientific computing (physical simulation, drug discovery), embodied intelligence, autonomous driving, and algorithmic trading. It is particularly suitable for scenarios with high computational energy efficiency requirements, such as data center energy saving and online learning on mobile or embedded devices.

[0064] As one specific embodiment of the present invention, this embodiment takes training a small convolutional neural network (CNN) and ResNet18 on the CIFAR-10 image classification task as an example. The ResNet18 is trained on CIFAR-10 images. The experimental results comparing the method of the present invention with SGD and Adam are as follows: Figure 2 , Figure 3 As shown.

[0065] Initialization: Setting , , , , K-FAC was chosen as the Fisher approximation method, and damping was set. EMA attenuation factor Feature decomposition cache frequency Zero space backoff ratio .

[0066] Training cycle: a. Forward and Backward Propagation: Obtaining Current Parameters Obtain a batch of data from the data loader and calculate the loss gradient. .

[0067] b. Update geometric information: Update the K-FAC factors of the current layers using an exponential moving average. and .

[0068] c. Solve for the pre-adjustment direction: Check if the first... If so, calculate and cache. and The eigenvalue decomposition and pseudo-inverse of the model are used. The pre-adjusted gradient direction of each layer is calculated using the cached pseudo-inverse. (Uses the efficient Kronecker product-vector product algorithm, eliminating the need to construct large matrices).

[0069] d. Thermodynamic Regulation: Calculating the Global Geometric Gradient Norm Substitute the values ​​into the regulator formula to calculate the thermodynamic adjustment step size. .

[0070] e. Parameter Update: Perform an update on all network parameters: If a layer is found to have a severely insufficient Fisher factor rank (retaining very few feature values), a small spatial Euclidean update term is added to the network parameters of that layer.

[0071] f. Monitoring and Recording: Record the training loss and thermodynamic adjustment step size for this step. Global geometric gradient norm and predicted dissipation And evaluate it periodically on the validation set.

[0072] Dynamic decision-making: Calculate the average dissipation ratio every 100 steps. If you currently have experience using Fisher and If so, then switch to the K-FAC approximation in the next stage.

[0073] Results: After training, the CNN achieved approximately 92% accuracy on the CIFAR-10 test set, comparable to the Adam optimizer, but according to... Cumulative value estimation shows that the theoretical irreversible dissipation of its optimization process is significantly lower than that of Adam.

[0074] Example 2 As another embodiment of the present invention, this embodiment also provides an optimized system for implementing the above method embodiments, including the following modules: Gradient calculation module: Forward and backward propagation, calculating the gradient of the loss function. .

[0075] Geometric Information Estimation Module: Estimates or updates the Fisher information matrix or its factors based on the selected method (K-FAC / Empirical Fisher, etc.).

[0076] Pre-adjuster solution module: Calculates the pre-adjusted gradient direction .

[0077] Thermodynamic Regulator Module: Calculation and according to Calculate the current step size using the formula above. .

[0078] Parameter update module: Performs the final network parameter update. Or a hybrid update.

[0079] Monitoring and Decision Module: Records training metrics, calculates dissipation ratios, and triggers dynamic approximation switching.

[0080] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, mobile embedded device, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0081] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A neural network optimization method based on thermodynamic dissipation constraints, characterized in that, The method steps include: Obtain the current network parameters of the neural network to be optimized, input the training data into the neural network, and calculate the gradient of the loss function; An approximate method is used to estimate the approximate inverse of the Fisher information matrix, and the global geometric gradient norm is calculated by combining the gradient of the loss function. The thermodynamic regulator dynamically adjusts the set dissipation budget to the current thermodynamic adjustment step size based on the global geometric gradient norm. The network parameters are updated based on the current thermodynamic adjustment step size. If the Fisher information matrix is ​​full rank, the network parameters are updated using the natural gradient direction. If the Fisher information matrix is ​​not full rank, the network parameters are updated using a hybrid update strategy, which uses the pseudo-inverse of the Fisher information matrix in the value domain and performs Euclidean updates in the null space direction.

2. The neural network optimization method based on thermodynamic dissipation constraints according to claim 1, characterized in that, The estimation methods for the inverse approximation of the Fisher information matrix include: eigenvalue truncation method, Tikhonov damping method, and Kronecker factor approximation.

3. The neural network optimization method based on thermodynamic dissipation constraints according to claim 2, characterized in that, The Kronecker factor approximation is specifically implemented as follows: Update the input covariance matrix of each linear and convolutional layer in the current neural network using exponential moving average. With the output gradient covariance matrix ; Check if the set number of steps has been reached; if so, calculate and cache the eigenvalues ​​and pseudo-inverses of the input covariance matrix A and the output gradient covariance matrix G; otherwise, repeat the eigenvalues ​​and pseudo-inverses. Using the pseudo-inverse of the cache, the pre-adjusted gradient direction of each layer is calculated; and the global geometric gradient norm is calculated and substituted into the thermodynamic regulator to calculate the current thermodynamic adjustment step size.

4. The neural network optimization method based on thermodynamic dissipation constraints according to claim 1, characterized in that, The thermodynamic regulator calculates the thermodynamic adjustment step size as follows: in, Thermodynamic adjustment step size; For dissipation budget; For the global geometric gradient norm, This is the inverse approximation of the Fisher information matrix. The gradient of the loss function; To prevent division by zero of small constants; For the clipping function, Trim the boundary for the step size.

5. The neural network optimization method based on thermodynamic dissipation constraints according to claim 1, characterized in that, When the Fisher information matrix of a certain layer in the network is not of full rank, the network parameters adopt a hybrid update strategy, adding a spatial Euclidean update term to the network parameters of that layer: in, Thermodynamic adjustment step size; It is the Moore-Penrose pseudoinverse, used to perform minimal-dissipation natural gradient updates within the range of the Fisher information matrix; The gradient of the loss function; It is the projection matrix onto the null space; The learning rate is used to control the proportion of the step size for null space updates. This is the zero-space rollback ratio.

6. The neural network optimization method based on thermodynamic dissipation constraints according to claim 1, characterized in that, Each time a training iteration of the network parameters is completed, the dissipation ratio is calculated, which is the ratio of the predicted dissipation to the minimum dissipation estimated by the current approximation method. When the dissipation ratio exceeds a set threshold, switch to a higher-precision approximation method to estimate the approximate inverse of the Fisher information matrix.

7. A neural network optimization system based on thermodynamic dissipation constraints, characterized in that, The system executes the neural network optimization method based on thermodynamic dissipation constraints as described in any one of claims 1-6, including: Gradient calculation module: Calculates the gradient of the loss function through forward and backward propagation; Geometric Information Estimation Module: Estimates and updates the approximate inverse of the Fisher information matrix using an approximate method; Pre-adjuster solution module: Uses the approximate inverse pre-adjustment gradient of the Fisher information matrix and calculates the pre-adjusted gradient direction; Thermodynamic regulator module: Calculates the global geometric gradient norm and, in conjunction with the set dissipation budget, calculates the current thermodynamic regulation step size; Parameter update module: Updates network parameters based on the current thermodynamic adjustment step size; Monitoring and Decision Module: Records training metrics and calculates the dissipation ratio. When the dissipation ratio exceeds a set threshold, it switches to a higher-precision approximation method to update the approximate inverse of the Fisher information matrix.

8. A neural network optimization system based on thermodynamic dissipation constraints according to claim 7, characterized in that, The geometric information estimation module uses the Kronecker factor to approximately update the inverse of the Fisher information matrix, specifically as follows: Update the input covariance matrix of each linear and convolutional layer in the current neural network using exponential moving average. With the output gradient covariance matrix ; Check if the set number of steps has been reached; if so, calculate and cache the eigenvalues ​​and pseudo-inverses of the input covariance matrix A and the output gradient covariance matrix G; otherwise, repeat the eigenvalues ​​and pseudo-inverses.

9. A neural network optimization system based on thermodynamic dissipation constraints according to claim 7, characterized in that, The thermodynamic regulator module calculates the thermodynamic regulation step size as follows: in, Thermodynamic adjustment step size; For dissipation budget; For the global geometric gradient norm, This is the inverse approximation of the Fisher information matrix. The gradient of the loss function; To prevent division by zero of small constants; For the clipping function, Trim the boundary for the step size.

10. A neural network optimization system based on thermodynamic dissipation constraints according to claim 7, characterized in that, The parameter update module updates the network parameters as follows: If the Fisher information matrix is ​​full rank, the network parameters are updated using the natural gradient direction. If the Fisher information matrix is ​​not of full rank, the network parameters adopt a hybrid update strategy, using the pseudo-inverse of the Fisher information matrix in the value domain and Euclidean updates in the null space direction.

Images