Model optimization method, device, equipment and medium based on inverse normalization

Abstract

The present invention provides a model optimization method, device, equipment and medium based on inverse normalization, the method comprising: obtaining a deep neural network model and calculating the inverse matrix of each network layer in the deep neural network model; obtaining the inverse matrix of the Fisher information matrix and multiple intermediate gradient values ​​according to the inverse matrix of each network layer; calculating the average gradient term and the average square term according to the multiple intermediate gradient values; dividing the average square term according to the inverse matrix of the Fisher information matrix to obtain the gradient vector corresponding to each network layer; normalizing the gradient vector of each network layer to obtain the unitized average square term; adjusting the variables to be optimized in the deep neural network model according to the unitized average square term and the average gradient term to obtain the optimized deep neural network model, wherein the present application provides a model optimization scheme that can optimize the training accuracy of the neural network while only increasing a small amount of power consumption, so as to improve model performance.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a model optimization method, device, equipment and medium based on inverse normalization. Background Art

[0002] In recent years, with the rapid development of the field of artificial intelligence, the requirements for computing power have also increased rapidly, resulting in the training of a viable neural network possibly taking weeks or even months. In order to reduce the consumption of computing resources, a variety of model optimization methods have emerged. For example, in neural network training, currently commonly used optimization methods include the natural gradient algorithm, K-FAC algorithm, and ADAM algorithm for optimization.

[0003] However, these algorithms still have some problems when dealing with the training of large-scale neural networks. For example, a major problem with the natural gradient algorithm is that its parameter number is usually very large, resulting in excessive computational resource overhead for the inversion of the Fisher information matrix, which is unacceptable. On the other hand, the use of the K-FAC algorithm or the ADAM algorithm alone also has the above problems to a certain extent, and the calculation accuracy of the K-FAC algorithm or the ADAM algorithm is poor.

[0004] The above-mentioned characteristics of high computing resource consumption and low training accuracy make it impossible to effectively apply the commonly used model optimization methods to the model training process with a more complex network structure and high training accuracy requirements. For example, it is impossible to effectively train and optimize the SchNet network model built by network layers containing multiple physical meanings in the field of materials, resulting in poor prediction accuracy of the SchNet network model generated by the existing technology for molecular structure, properties, etc. Summary of the Invention

[0005] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0006] The embodiments of the present invention provide a model optimization method, electronic device and medium based on inverse normalization, which aim to solve the technical problems in the prior art that deep learning model training consumes a lot of computing resources, has low training accuracy, and cannot be effectively applied to the model training process with a relatively complex network structure and high training accuracy requirements.

[0007] To achieve the above objectives, a first aspect of an embodiment of the present invention provides a model optimization method based on inverse normalization, comprising:

[0008] Obtain a deep neural network model that has been preliminarily trained based on a preset sample data set, and calculate the inverse matrix of each network layer in the deep neural network model;

[0009] Obtaining the inverse matrix of the Fisher information matrix according to the inverse matrix of each network layer;

[0010] Obtaining a plurality of intermediate gradient values ​​according to the inverse matrix of the Fisher information matrix;

[0011] Calculating an average gradient term and an average square term based on the plurality of intermediate gradient values;

[0012] Dividing the mean square term according to the inverse matrix of the Fisher information matrix to obtain a gradient vector corresponding to each of the network layers;

[0013] Normalizing the gradient vector of each network layer to obtain a normalized mean square term;

[0014] The variables to be optimized in the deep neural network model are adjusted according to the normalized mean square term and the mean gradient term to obtain an optimized deep neural network model.

[0015] In some embodiments, calculating the inverse matrix of each network layer in the deep neural network model includes:

[0016] Calculate the forward propagation feature matrix and the backward propagation feature matrix of each network layer in the deep neural network model according to the sample data set;

[0017] The forward propagation feature matrix and the backward propagation feature matrix corresponding to each network layer are respectively subjected to Kronecker product calculation processing, and the obtained product result is determined as the inverse matrix of each network layer.

[0018] In some embodiments, obtaining the inverse matrix of the Fisher information matrix according to the inverse matrix of each network layer includes:

[0019] Obtaining a plurality of diagonal block units according to the inverse matrix of each of the network layers;

[0020] Splicing each of the diagonal block units according to a preset rule to obtain a Fisher information matrix, so that the main diagonal line of each of the diagonal block units corresponds to a part of the main diagonal line of the Fisher information matrix in sequence;

[0021] An inverse operation is performed on the Fisher information matrix to obtain an inverse matrix of the Fisher information matrix.

[0022] In some embodiments, obtaining a plurality of intermediate gradient values ​​according to the inverse matrix of the Fisher information matrix includes:

[0023] Obtaining an iterative gradient value for each iteration of the deep neural network model;

[0024] A plurality of intermediate gradient values ​​are obtained according to the iterative gradient value of each iteration and the inverse matrix of the Fisher information matrix.

[0025] In some embodiments, calculating the average gradient term and the average square term based on the plurality of intermediate gradient values ​​includes:

[0026] Initialize the average gradient term and the average square term;

[0027] performing a first exponentially weighted moving average update on the average gradient term according to a preset first attenuation coefficient and the plurality of intermediate gradient values ​​to obtain the average gradient term;

[0028] The average gradient term is updated by performing a second exponentially weighted moving average according to a preset second attenuation coefficient and the squares of the plurality of intermediate gradient values ​​to obtain the average square term.

[0029] In some embodiments, the partitioning of the mean square term according to the inverse matrix of the Fisher information matrix to obtain the gradient vector corresponding to each of the network layers includes:

[0030] Flattening the mean square term to obtain a gradient column vector;

[0031] The gradient column vector is multiplied by each of the diagonal block units in the inverse matrix of the Fisher information matrix to obtain a gradient vector of the network layer corresponding to each diagonal block unit, wherein the gradient vector is a column vector or a row vector.

[0032] In some embodiments, normalizing the gradient vector of each network layer to obtain a unitized mean square term includes:

[0033] Performing arithmetic mean calculation on the gradient vectors of each network layer respectively to obtain the average gradient vector of each network layer;

[0034] Multiplying the average gradient vector of each network layer by a unit vector of the same length to obtain a normalized gradient vector corresponding to each network layer;

[0035] A normalized mean square term is obtained according to the normalized gradient vector corresponding to each of the network layers.

[0036] In some embodiments, adjusting the variables to be optimized in the deep neural network model according to the normalized mean square term and the mean gradient term includes:

[0037] Get the current value of the variable to be optimized;

[0038] Calculating an adjustment value according to a preset learning rate, a preset smoothing term, the normalized mean square term, and the mean gradient term;

[0039] The value of the variable to be optimized is determined as the difference between the current value and the adjusted value.

[0040] In some embodiments, the sample dataset includes molecular structural data and property data, and the deep neural network model is a SchNet network model for predicting the structure and properties of molecules. After obtaining the optimized deep neural network model, the method further includes:

[0041] The target molecule-related data is input into the optimized SchNet network model to predict the molecular structure and molecular properties corresponding to the target molecule according to the output results of the optimized SchNet network model.

[0042] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present invention provides a model optimization device based on inverse normalization, comprising:

[0043] A first inverse matrix calculation unit is used to obtain a deep neural network model that has been preliminarily trained based on a preset sample data set, and calculate the inverse matrix of each network layer in the deep neural network model;

[0044] A second inverse matrix calculation unit, configured to obtain an inverse matrix of a Fisher information matrix according to the inverse matrix of each network layer;

[0045] an intermediate gradient value calculation unit, configured to obtain a plurality of intermediate gradient values ​​according to the inverse matrix of the Fisher information matrix;

[0046] an average gradient value calculation unit, configured to calculate an average gradient term and an average square term based on the plurality of intermediate gradient values;

[0047] A gradient vector calculation unit, configured to divide the mean square term according to the inverse matrix of the Fisher information matrix to obtain a gradient vector corresponding to each of the network layers;

[0048] A normalization calculation unit, configured to perform normalization processing on the gradient vector of each network layer to obtain a unitized mean square term;

[0049] A parameter adjustment unit is used to adjust the variables to be optimized in the deep neural network model according to the normalized mean square term and the mean gradient term to obtain an optimized deep neural network model.

[0050] To achieve the above-mentioned purpose, a third aspect of an embodiment of the present invention proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0051] To achieve the above-mentioned purpose, a fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0052] The model optimization method, device, equipment and medium based on inverse normalization proposed in the present invention obtain a deep neural network model that has been preliminarily trained according to a preset sample data set, and calculate the inverse matrix of each network layer in the deep neural network model, and obtain the inverse matrix of the Fisher information matrix based on the inverse matrix of each network layer, so as to preliminarily reduce the computing resources for obtaining the inverse matrix of the Fisher information matrix of the deep learning model; further, multiple intermediate gradient values ​​are obtained according to the inverse matrix of the Fisher information matrix, and the average gradient term and the average square term are calculated based on the multiple intermediate gradient values, so as to realize the process of averaging the newly obtained gradient term and square term according to the value obtained in each historical iteration, thereby facilitating subsequent normalization processing; further, the average square term is divided according to the inverse matrix of the Fisher information matrix to obtain the gradient vector corresponding to each network layer, so as to realize the corresponding block division of the output gradient according to the diagonal block characteristics caused by the neural network layering in the calculation process, and each block is regarded as a whole to reduce the amount of calculation. ; Further, the gradient vector of each of the network layers is normalized to obtain a unitized average square term, normalization is performed between blocks, and a corrected gradient is obtained, so as to realize the process of effectively improving the calculation accuracy through normalization; finally, the variables to be optimized in the deep neural network model are adjusted according to the unitized average square term and the average gradient term to obtain an optimized deep neural network model, wherein, through the above steps, the present application can provide a training scheme that can optimize the training accuracy of the neural network with only a small increase in power consumption, and can be effectively applied to the model training process with a relatively complex network structure and high requirements for training accuracy, so that the obtained network model has higher prediction accuracy and the prediction results are more accurate; for example, the method of the present application can be used to train and optimize the SchNet network model constructed by network layers containing multiple physical meanings in the field of materials, so that the optimized SchNet network model can have better performance advantages in atomic energy prediction in materials science fields such as molecular structure and properties, which is convenient for technical personnel in related fields to carry out research. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is a flow chart of a model optimization method based on inverse normalization provided by an embodiment of the present invention;

[0054] Figure 2 yes Figure 1 Flowchart of step S110 in FIG.

[0055] Figure 3 yes Figure 1 Flowchart of step S120 in FIG.

[0056] Figure 4 yes Figure 1 Flowchart of step S130 in FIG.

[0057] Figure 5 yes Figure 1 Flowchart of step S140 in FIG.

[0058] Figure 6 yes Figure 1 Flowchart of step S150 in FIG.

[0059] Figure 7 yes Figure 1 Flowchart of step S160 in FIG.

[0060] Figure 8 yes Figure 1 Flowchart of step S170 in FIG.

[0061] Figure 9 1 is a schematic structural diagram of a model optimization device based on inverse normalization provided by an embodiment of the present invention;

[0062] Figure 10 It is a schematic diagram of the hardware structure of the electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0064] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention pertains. The terms used herein are for the purpose of describing embodiments of the present invention only and are not intended to limit the present invention.

[0066] Artificial intelligence (AI), a key strategic technology, simulates and implements human intelligence through the fusion of data, computing, and algorithms. However, as AI models grow in complexity and scale, they require greater computing resources for training. In recent years, large-scale pre-trained models and optimization algorithms, such as GPT-3, GPT-4, and BERT, have garnered widespread attention. These models, with billions to tens of billions of parameters, require enormous computing power to train. AI technology has already begun to transform traditional areas of computational science. For example, AlphaFold2 has solved the challenging problem of protein structure prediction. Optimization algorithms are key technologies in AI research, particularly in training deep neural networks and reinforcement learning, which involve solving large-scale, non-convex, and non-smooth optimization problems. For various deep learning tasks, researchers are working to overcome challenges such as non-convexity, non-smoothness, and high dimensionality by constructing advanced and reliable optimization models and developing high-precision, real-time optimization algorithms, thereby addressing bottlenecks in the design of distributed and online optimization algorithms. Since the model structure and training process are also closely related to the model's performance, when the model structure is determined, finding the optimal solution for this model becomes a challenging problem. When designing the model structure and training process, researchers need to consider the complexity of the model and the limitations of computing resources to improve model performance and training effects.

[0067] Specifically, a variety of model optimization methods have emerged. For example, in neural network training, commonly used optimization methods include the natural gradient algorithm, K-FAC algorithm, and ADAM algorithm. However, these algorithms still have some problems when dealing with large-scale neural network training. For example, a major problem with the natural gradient algorithm is that its parameter count is usually very large, resulting in excessive computational resource overhead for the inversion of the Fisher information matrix, which is unacceptable. On the other hand, the use of the K-FAC algorithm or the ADAM algorithm alone also has the above problems to a certain extent, and the calculation accuracy of the K-FAC algorithm or the ADAM algorithm is poor. The above characteristics of high computational resource consumption and low training accuracy make it impossible to effectively apply commonly used model optimization methods to the training process of models with complex network structures and high training accuracy requirements. For example, it is impossible to effectively train and optimize the SchNet network model built by network layers containing multiple physical meanings in the field of materials, resulting in poor prediction accuracy of the SchNet network model generated by existing technologies for molecular structure, properties, etc.

[0068] Based on this, an embodiment of the present invention provides a model optimization method, electronic device and medium based on inverse normalization, which aims to solve the technical problems in the existing technology that deep learning model training consumes a lot of computing resources, has low training accuracy, and cannot be effectively applied to the model training process with a more complex network structure and high training accuracy requirements.

[0069] In this application, the present invention can be used in a variety of general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices, without specific limitation in the embodiments of the present invention.

[0070] Specifically, in an embodiment of the present invention, the model optimization method based on inverse normalization can be applied to NVIDIA GPU processors, and the deep learning model can be built by network layers containing multiple physical meanings. The network structure is relatively complex and has high requirements on training accuracy. For the SchNet network model in the field of materials, the corresponding sample data set includes molecular structural data and property data. After the SchNet network model is trained and the optimized SchNet network model is obtained, the target molecule-related data is input into the optimized SchNet network model to predict the molecular structure and molecular properties corresponding to the target molecule according to the output results of the optimized SchNet network model, so as to realize the training optimization of the SchNet network model built by network layers containing multiple physical meanings in the field of materials, so that the optimized SchNet network model can have better performance advantages in the prediction of atomic energy in the field of materials science such as molecular structure and properties, which is convenient for technical personnel in related fields to carry out research.

[0071] The embodiments of the present invention are further described below with reference to the accompanying drawings.

[0072] Figure 1 is an optional flow chart of a model optimization method based on inverse normalization provided by an embodiment of the present invention. Figure 1 The method may include but is not limited to steps S110 to S170.

[0073] Step S110, obtaining a deep neural network model that has been preliminarily trained based on a preset sample data set, and calculating the inverse matrix of each network layer in the deep neural network model;

[0074] Step S120, obtaining the inverse matrix of the Fisher information matrix according to the inverse matrix of each network layer;

[0075] Step S130, obtaining a plurality of intermediate gradient values ​​according to the inverse matrix of the Fisher information matrix;

[0076] Step S140, calculating an average gradient term and an average square term based on the multiple intermediate gradient values;

[0077] Step S150, dividing the mean square term according to the inverse matrix of the Fisher information matrix to obtain the gradient vector corresponding to each network layer;

[0078] Step S160, normalizing the gradient vector of each network layer to obtain a normalized mean square term;

[0079] Step S170: Adjust the variables to be optimized in the deep neural network model according to the normalized mean square term and the mean gradient term to obtain an optimized deep neural network model.

[0080] In some embodiments, for deep learning, model training can be expressed as solving the following large-scale non-convex and non-smooth problem. Assuming that the variable to be optimized in the deep neural network model is θ, the model optimization process can be understood as the process of solving the minimum value of the following network model expression J(θ):

[0081]

[0082] in, is a given sample data set, θ is the parameter of the neural network and the target variable to be optimized, f(x,θ) represents the output of the neural network when the input is x, ψ is the loss function, and the commonly used ones are Loss function and cross entropy loss function, etc. This application optimizes the model training accuracy by iteratively updating θ.

[0083] In steps S110 to S170 shown in the embodiment of the present invention, a deep neural network model that has been preliminarily trained according to a preset sample data set is obtained, and the inverse matrix of each network layer in the deep neural network model is calculated. The inverse matrix of the Fisher information matrix is ​​obtained according to the inverse matrix of each network layer, so as to preliminarily reduce the computing resources for obtaining the inverse matrix of the Fisher information matrix of the deep learning model; further, a plurality of intermediate gradient values ​​are obtained according to the inverse matrix of the Fisher information matrix, and the average gradient term and the average square term are calculated according to the plurality of intermediate gradient values, so as to realize the process of averaging the newly obtained gradient term and square term according to the value obtained in each historical iteration, so as to facilitate subsequent normalization processing; further, the average square term is divided according to the inverse matrix of the Fisher information matrix to obtain the gradient vector corresponding to each network layer. The method divides the output gradient into corresponding blocks according to the diagonal block characteristics caused by the neural network layering in the calculation process, and regards each block as a whole to reduce the amount of calculation; further, the gradient vector of each network layer is normalized to obtain the unitized average square term, and the normalization between blocks is performed to obtain the corrected gradient, so as to realize the process of effectively improving the calculation accuracy through normalization; finally, the variables to be optimized in the deep neural network model are adjusted according to the unitized average square term and the average gradient term to obtain the optimized deep neural network model. The above steps enable the present application to provide a training scheme that can optimize the training accuracy of the neural network with only a small increase in power consumption, and can be effectively applied to the model training process with a relatively complex network structure and high requirements on training accuracy, so that the obtained network model has higher prediction accuracy and the prediction results are more accurate.

[0084] In some embodiments, the present application can be based on the characteristics of existing deep neural network technology, and by adopting the calculation process of the K-FAC algorithm, realize the process of preliminarily calculating the Fisher information matrix, and obtaining multiple intermediate gradient values ​​according to the inverse matrix of the Fisher information matrix; then the process of the ADAM algorithm can be adopted to average the newly obtained gradient terms and square terms according to the values ​​obtained in each historical iteration, and calculate the average gradient term and the average square term; next, combining the K-FAC algorithm and the ADAM algorithm, considering the diagonal block characteristics caused by the neural network layer in the calculation process, the output gradient is divided into blocks corresponding to the network layer to obtain The gradient vector corresponding to each network layer, and each block is regarded as a whole, and then the blocks are normalized to obtain the corrected gradient (normalized average square term), and then the variables to be optimized are adjusted. Among them, due to the computational complexity of the newly added block division and normalization operations, compared with the computational complexity of the original K-FAC algorithm, the present application combines the averaging and normalization processing in the ADAM algorithm to effectively improve the calculation accuracy, while its computational complexity does not increase significantly, so that the model optimization scheme of the present application can improve the training accuracy of the neural network with only a small increase in power consumption, which is far less than the power consumption consumed when using only the ADAM algorithm.

[0085] The above is a general description of step S110 to step S170 . Step S110 to step S170 will be described in detail below.

[0086] Step S110: Obtain a deep neural network model that has been preliminarily trained based on a preset sample data set, and calculate the inverse matrix of each network layer in the deep neural network model.

[0087] See also Figure 2 , Figure 2 yes Figure 1 In the flowchart of step S110 in FIG. 1 , in some embodiments, calculating the inverse matrix of each network layer in the deep neural network model may include, but is not limited to, steps S210 to S220:

[0088] Step S210, calculating the forward propagation feature matrix and the backward propagation feature matrix of each network layer in the deep neural network model according to the sample data set;

[0089] In step S220, Kronecker product calculation is performed on the forward propagation feature matrix and the backward propagation feature matrix corresponding to each network layer, and the obtained product result is determined as the inverse matrix of each network layer.

[0090] In some embodiments, the deep learning model consists of several network layers, and the forward propagation feature matrix is ​​composed of the expected value of the gradient of the output before the nonlinear mapping of the network layer to the loss function value transmitted back; the back propagation feature matrix is ​​composed of the expected value of the output of the previous network layer after the nonlinear mapping. The matrix obtained by the Kronecker product of the forward propagation feature matrix and the back propagation feature matrix is ​​used as the inverse matrix of the network layer, which can greatly reduce the computing resources and storage resources for deep learning model training.

[0091] In some embodiments, the process of calculating the inverse matrix of the network layer in steps S210 to S220 is specifically as follows:

[0092]

[0093] in, is the matrix F l The inverse matrix, F l is the network layer matrix, G l is the forward propagation feature matrix, A l-1 is the back propagation feature matrix, is the Kronecker product, l represents the lth layer network formula of the deep neural network model, in summary, Represents the inverse matrix of each network layer. Since the Kronecker product greatly reduces the computational complexity and storage capacity of the network layer model, that is, it greatly simplifies the calculation process, the calculation of each network layer is performed using the Kronecker product approximation method, which can effectively reduce the computational resource consumption when calculating the inverse matrix of the network layer.

[0094] Step S120: Obtain the inverse matrix of the Fisher information matrix according to the inverse matrix of each network layer.

[0095] See also Figure 3 , Figure 3 yes Figure 1 In the flowchart of step S120 in FIG. 1 , in some embodiments, obtaining the inverse matrix of the Fisher information matrix according to the inverse matrix of each network layer may include but is not limited to steps S310 to S330:

[0096] Step S310, obtaining a plurality of diagonal block units according to the inverse matrix of each network layer;

[0097] Step S320: splicing each diagonal block unit according to a preset rule to obtain a Fisher information matrix, so that the main diagonal line of each diagonal block unit corresponds to a part of the main diagonal line of the Fisher information matrix in sequence;

[0098] Step S330: performing an inverse operation on the Fisher information matrix to obtain an inverse matrix of the Fisher information matrix.

[0099] In some embodiments, the process of calculating the inverse of the Fisher information matrix in the present application is similar to the K-FAC algorithm, which is an optimization algorithm used to train neural networks and aims to estimate the inverse of the Fisher information matrix of the model parameters. The Fisher information matrix is ​​a way to measure the sensitivity of the parameters to the gradient of the objective function, and its inverse matrix can be used to effectively update the model parameters during training.

[0100] Specifically, the Fisher information matrix F is assembled by diagonal blocks θ The process is as follows:

[0101] F θ ≈diag(F1,...,F l ,...,F L ) (3)

[0102] Among them, F θ is the Fisher information matrix, is the inverse matrix of the Fisher information matrix, F1,...,F l ,...,F L is the matrix of each network layer mentioned above, and L is the total number of network layers. Since the conventional technology calculates the inverse matrix of the Fisher information matrix, the computational overhead is relatively large, especially for large-scale neural network problems. Therefore, in order to reduce the amount of calculation, this application approximates the calculation, that is, only takes the diagonal blocks for approximate calculation, and obtains according to the above formula (3) The expression is:

[0103]

[0104] in, It is the inverse matrix of the above network layers.

[0105] In some embodiments, the present application generates multiple diagonal block units for each network layer according to the inverse matrix of the layer. The inverse matrix is ​​the inverse operation of the original matrix, which describes the sensitivity and correlation of the parameters in the network layer, and the generated diagonal block units represent the diagonal part of the inverse matrix of the network layer; further, each diagonal block unit is spliced ​​according to a preset rule to obtain a complete Fisher information matrix. The splicing rule is usually to place each diagonal block unit in the corresponding position in the Fisher information matrix to ensure that the main diagonal of the diagonal block unit corresponds to a part of the main diagonal of the Fisher information matrix. The purpose of this is to maintain the correlation between the parameters and to have a lower computational complexity when calculating the inverse matrix; further, the spliced ​​Fisher information matrix is ​​inversely operated to obtain the inverse matrix of the Fisher information matrix, which can be used to update the model parameters so as to more effectively optimize the model during the training process.

[0106] In some embodiments, through steps S310 to S330, the inverse matrix of each network layer is used as the diagonal block unit of the inverse matrix of the Fisher information matrix, and the inverse matrix obtained by splicing the diagonal block units according to preset rules is used as the inverse matrix of the Fisher information matrix. The diagonal block approximation method is adopted. Since only the interaction within each layer of the neural network is considered and the interaction between layers is ignored, although the diagonal block approximation method only considers the interaction within the layer to capture the main parameter relationship and ignores the interaction between layers, it is usually feasible in practice because the inter-layer correlation of the neural network is often relatively weak, while the correlation within the layer is strong. Therefore, in most cases, this simplification does not have a significant impact on the accuracy of the result, so that the present application can greatly simplify the calculation process without causing a significant impact on the accuracy, further reducing the consumption of computing resources. Among them, the preset rules mentioned can be in the order of the network layers from top to bottom in the deep learning model, and then multiple intermediate gradient values ​​for training the deep neural network model can be obtained according to the inverse matrix of the Fisher information matrix.

[0107] Step S130: obtaining a plurality of intermediate gradient values ​​according to the inverse matrix of the Fisher information matrix.

[0108] See also Figure 4 , Figure 4 yes Figure 1 In the flowchart of step S130 in FIG. 1 , in some embodiments, obtaining multiple intermediate gradient values ​​according to the inverse matrix of the Fisher information matrix may include but is not limited to steps S410 to S420:

[0109] Step S410, obtaining the iterative gradient value of each iteration of the deep neural network model;

[0110] Step S420 , obtaining a plurality of intermediate gradient values ​​according to the iterative gradient value of each iteration and the inverse matrix of the Fisher information matrix.

[0111] In some embodiments, the model can be monitored during the model training process, and the gradient value calculated after each iteration can be recorded to obtain the iterative gradient value of each iteration of the deep neural network model, that is, after each iteration, the function for calculating the gradient provided by the deep learning framework used is called to obtain the iterative gradient value of each iteration.

[0112] In some embodiments, the iterative gradient value of each iteration and the inverse matrix of the Fisher information matrix are used to obtain multiple intermediate gradient values ​​as follows:

[0113]

[0114] in, is the iterative gradient value of each iteration, and y is the intermediate gradient value. The result obtained by the calculation process similar to the above K-FAC algorithm is used as an intermediate variable to facilitate subsequent normalization processing and improve the training accuracy of this application.

[0115] Step S140 : Calculate an average gradient term and an average square term based on the multiple intermediate gradient values.

[0116] See also Figure 5 , Figure 5 yes Figure 1 In the flowchart of step S140 in FIG. 1 , in some embodiments, calculating the average gradient term and the average square term based on multiple intermediate gradient values ​​may include, but is not limited to, steps S510 to S530:

[0117] Step S510, initializing the average gradient term and the average square term;

[0118] Step S520, performing a first exponentially weighted moving average update on the average gradient term according to a preset first attenuation coefficient and a plurality of intermediate gradient values ​​to obtain an average gradient term;

[0119] Step S530 : performing a second exponentially weighted moving average update on the average gradient term according to a preset second attenuation coefficient and the squares of a plurality of intermediate gradient values ​​to obtain an average square term.

[0120] In some embodiments, initializing the average gradient term and the average square term includes setting the average gradient term and the average square term to zero.

[0121] In some embodiments, the average gradient term is updated by performing a first exponentially weighted moving average based on a preset first attenuation coefficient and a plurality of intermediate gradient values, and a specific expression of the average gradient term is obtained as follows:

[0122] m←β1m+(1-β1)y (6)

[0123] Wherein, m is the average gradient term, β1 is the preset first attenuation coefficient, and y is the above-mentioned intermediate gradient value.

[0124] In some embodiments, the average gradient term is updated by performing a second exponentially weighted moving average based on a preset second attenuation coefficient and the squares of multiple intermediate gradient values, and a specific expression for the average square term is obtained as follows:

[0125] v←β2v+(1-β2)y 2 (7)

[0126] Among them, v is the mean square term, β2 is the preset second attenuation coefficient, and y 2 is the square of the above intermediate gradient value.

[0127] In some embodiments, steps S510 to S530 are similar to the calculation process in the ADAM algorithm. The ADAM (Adaptive Moment Estimation) algorithm is a commonly used optimization algorithm for training deep neural networks, and a normalization processing step is adopted in its training process. That is, the above formula (6) is essentially an average processing of the intermediate gradient values ​​generated by each historical iteration in the calculation process; the above formula (7) is essentially an average processing of the square of the intermediate gradient values ​​obtained by each historical iteration in the calculation process, which facilitates the averaging of the historical gradients and then the normalization processing in the subsequent process.

[0128] Step S150: The mean square term is divided according to the inverse matrix of the Fisher information matrix to obtain the gradient vector corresponding to each network layer.

[0129] See also Figure 6 , Figure 6 yes Figure 1 In the flowchart of step S150 in FIG. 1 , in some embodiments, the mean square term is divided according to the inverse matrix of the Fisher information matrix to obtain the gradient vector corresponding to each network layer, which may include but is not limited to steps S610 to S620:

[0130] Step S610, flattening the mean square term to obtain a gradient column vector;

[0131] Step S620: Multiply the gradient column vector by each diagonal block unit in the inverse matrix of the Fisher information matrix to obtain the gradient vector of the network layer corresponding to each diagonal block unit, where the gradient vector is a column vector or a row vector.

[0132] In some embodiments, through the above steps S610 to S620, the output gradient is divided into corresponding blocks according to the diagonal block characteristics caused by the neural network layering in the calculation process. Through the above steps, the unitized average square term is obtained in the subsequent process. The momentum term can be introduced in the process of model training and normalized, thereby improving the speed and accuracy of convergence, and scaling the gradient, effectively improving the convergence effect, and improving the accuracy of model training.

[0133] Specifically, according to The diagonal block structure of v is transformed into a unit for the subsequent normalization, that is, for the l-th layer neural network, assuming v l The corresponding dimension in v is n l ×1 gradient vector.

[0134] In some embodiments, in K-FAC, the Fisher information matrix is ​​used to estimate the second-order information of the gradient. In order to combine K-FAC in the ADAM optimization step, the mean square term needs to be flattened and converted into a gradient column vector, that is, the mean square term matrix is ​​expanded into a vector. Step S610 is intended to convert the information of the mean square term from matrix form to vector form; while in K-FAC, the inverse matrix of the Fisher information matrix can be estimated by the inverse matrix approximation method. In step S620, the gradient column vector is multiplied with each diagonal block unit in the inverse matrix of the Fisher information matrix to obtain the gradient vector of the network layer corresponding to each diagonal block unit. This means that the gradient will be processed in blocks according to the network structure in order to better estimate the gradient information between different layers, and the gradient vector of each network layer will be normalized in the subsequent process so that the gradients of different network layers have similar scales, which helps to effectively optimize and better adjust the model parameters.

[0135] Step S160: Normalize the gradient vector of each network layer to obtain a normalized mean square term.

[0136] See also Figure 7 , Figure 7 yes Figure 1 In the flowchart of step S160 in FIG. 1 , in some embodiments, normalizing the gradient vector of each network layer to obtain a unitized mean square term may include, but is not limited to, steps S710 to S730:

[0137] Step S710, performing arithmetic mean calculation on the gradient vectors of each network layer to obtain the average gradient vector of each network layer;

[0138] Step S720: Multiply the average gradient vector of each network layer by a unit vector of the same length to obtain a normalized gradient vector corresponding to each network layer;

[0139] Step S730: Obtain a normalized mean square term based on the normalized gradient vector corresponding to each network layer.

[0140] In some embodiments, the specific process of obtaining the normalized mean square term is as follows: Calculate v l The average value p l =mean(v l ); Then, generate a new vector v l ′=p l 1 l , where 1 l is the length n l ×1 is a unit vector with all elements of 1; finally, the newly generated vector v l ′ is assembled into v′; where p l is the above average gradient vector, v l ′ is the normalized gradient vector, and v′ is the normalized mean square term. The normalized mean square term is used to adjust the parameters to be adjusted in the subsequent process to improve the training accuracy.

[0141] In summary, the above steps S150 to S160 combine the optimization process of K-FAC and ADAM, flatten the mean square term into a gradient column vector, use the inverse matrix of the Fisher information matrix to block the gradient vector, and normalize the gradient vector of each network layer to obtain better gradient estimation and model optimization effect.

[0142] Step S170: Adjust the variables to be optimized in the deep neural network model according to the normalized mean square term and the mean gradient term to obtain an optimized deep neural network model.

[0143] See also Figure 8 , Figure 8 yes Figure 1 In the flowchart of step S170 in FIG. 1 , in some embodiments, adjusting the variables to be optimized in the deep neural network model according to the normalized mean square term and the mean gradient term may include, but is not limited to, steps S810 to S830:

[0144] Step S810, obtaining the current value of the variable to be optimized;

[0145] Step S820, calculating an adjustment value based on a preset learning rate, a preset smoothing term, a normalized mean square term, and an average gradient term;

[0146] Step S830: Determine the value of the variable to be optimized as the difference between the current value and the adjusted value.

[0147] Corresponding to steps S810 to S830, the variable θ to be optimized in the deep neural network model is adjusted, and the expression is as follows:

[0148]

[0149] Among them, η is the preset learning rate, ∈ is the preset smoothing term, v′ is the above-mentioned normalized average square term, and m is the average gradient term. Replaces the gradient in the existing technology algorithm To adjust the optimized variable θ, the normalization process is introduced into the K-FAC algorithm, thereby greatly improving the training accuracy of the model and expectedly improving the calculation accuracy.

[0150] In some embodiments, the above steps are designed and developed on this basis by combining the characteristics of the original K-FAC algorithm and the ADAM algorithm. Compared with traditional optimization methods such as stochastic gradient descent, it has better accuracy and computing resource consumption advantages. By first adopting the calculation process of the K-FAC algorithm, a preliminary calculation output gradient is obtained, and the inverse matrix of the Fisher information matrix can be obtained without performing a real inverse calculation; then, the process of the ADAM algorithm is adopted, and the newly obtained gradient terms and square terms are averaged according to the values ​​obtained in each historical iteration; next, considering the diagonal block characteristics caused by the neural network layering in the calculation process, the output gradient is divided into corresponding blocks, and each block is regarded as a whole; finally, the blocks are normalized to obtain the corrected gradient. Since the computational amount of the newly added block division and normalization operations is very small compared to the computational amount of the original K-FAC algorithm, the computational amount of this application has not increased significantly, but the prediction accuracy has been significantly improved.

[0151] In some embodiments, the sample dataset includes molecular structural data and property data, and the deep neural network model is a SchNet network model for predicting the structure and properties of molecules. The method of the present application includes:

[0152] A SchNet network model that has been preliminarily trained based on preset molecular structural data and property data is obtained, and the inverse matrix of each network layer in the SchNet network model is calculated; the inverse matrix of the Fisher information matrix is ​​obtained based on the inverse matrix of each network layer; multiple intermediate gradient values ​​are obtained based on the inverse matrix of the Fisher information matrix; the average gradient term and the average square term are calculated based on the multiple intermediate gradient values; the average square term is divided and processed based on the inverse matrix of the Fisher information matrix to obtain the gradient vector corresponding to each network layer; the gradient vector of each network layer is normalized to obtain the unitized average square term; the variables to be optimized in the SchNet network model are adjusted based on the unitized average square term and the average gradient term to obtain the optimized SchNet network model; the target molecule related data is input into the optimized SchNet network model to predict the molecular structure and molecular properties corresponding to the target molecule based on the output results of the optimized SchNet network model, so that the optimized SchNet network model can have better performance advantages in the prediction of atomic energy in the field of materials science such as molecular structure and properties, which is convenient for technical personnel in related fields to carry out research.

[0153] In some embodiments, the present invention has been implemented and tested, and has good performance advantages in the prediction of atomic energy in the field of materials science. For example, in the task of predicting atomic energy (QM9) using the SchNet network in the field of materials science, the specific data are as follows: when comparing the atomic energy prediction task using SchNet based on the QM9 data set on the GPU, an accuracy test is performed. When the number of iterations (Epochs) exceeds 400, the loss (VALIDATION LOSS) of the present application solution on the validation set is 0.0003, the loss of K-FAC on the validation set is 0.0008, and the loss of ADAM on the validation set is 0.0003; at the same time, an energy error test is performed. When the number of iterations (Epochs) exceeds 400, the energy index of the mean absolute error (MAEof The energy index of the average absolute error of the solution in this application is smaller than that of ADAM and K-FAC at the same time, which effectively shows that under the NVIDIA GPU hardware environment, the convergence accuracy of the solution in this application is better than that of ADAM and K-FAC, that is, the solution in this application has indeed improved the two algorithms ADAM and K-FAC.

[0154] Second, reference Figure 9, an embodiment of the present invention further provides a model optimization device based on inverse normalization, comprising: a first inverse matrix calculation unit 901, used to obtain a deep neural network model that has been preliminarily trained according to a preset sample data set, and calculate the inverse matrix of each network layer in the deep neural network model; a second inverse matrix calculation unit 902, used to obtain the inverse matrix of the Fisher information matrix according to the inverse matrix of each network layer; an intermediate gradient value calculation unit 903, used to obtain multiple intermediate gradient values ​​according to the inverse matrix of the Fisher information matrix; an average gradient value calculation unit 904, used to calculate the average gradient term and the average square term according to the multiple intermediate gradient values; a gradient vector calculation unit 905, used to divide the average square term according to the inverse matrix of the Fisher information matrix to obtain the gradient vector corresponding to each network layer; a normalization calculation unit 906, used to normalize the gradient vector of each network layer to obtain a normalized average square term; a parameter adjustment unit 907, used to adjust the variables to be optimized in the deep neural network model according to the normalized average square term and the average gradient term to obtain an optimized deep neural network model.

[0155] In a third aspect, embodiments of the present invention further provide an electronic device comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for enabling communication between the processor and the memory. When the program is executed by the processor, the aforementioned model optimization method based on inverse normalization is implemented. The electronic device may be any intelligent terminal, including a tablet computer and an in-vehicle computer.

[0156] See also Figure 10 , Figure 10 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0157] The processor 1001 may be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided by the embodiments of the present invention.

[0158] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called by the processor 1001 to execute the model optimization method based on inverse normalization according to the embodiment of the present invention.

[0159] Input / output interface 1003, used to implement information input and output;

[0160] Communication interface 1004, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0161] Bus 1005 , which transmits information between various components of the device (e.g., processor 1001 , memory 1002 , input / output interface 1003 , and communication interface 1004 );

[0162] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0163] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned model optimization method based on inverse normalization.

[0164] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0165] In summary, compared with the existing technology, the present invention combines the calculation process of the K-FAC algorithm and the process of the ADAM algorithm. First, a preliminary calculation output gradient is obtained, and the inverse matrix of the Fisher information matrix can be obtained without performing a real inverse calculation; then the gradient term and the square term are averaged, and block division and normalization operations are performed. The above characteristics enable the present invention to provide a model optimization solution that can optimize the training accuracy of the neural network with only a small increase in power consumption, and solve the technical problems in the existing technology that deep learning model training consumes a lot of computing resources and has low training accuracy, and cannot be effectively applied to the model training process with a relatively complex network structure and high training accuracy requirements, so as to improve the model performance and enable the model to be effectively applied to the model training process with a relatively complex network structure and high training accuracy requirements.

[0166] The embodiments described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0167] It will be understood by those skilled in the art that Figures 1 to 8 The technical solutions shown in the figures do not constitute a limitation on the embodiments of the present invention, and may include more or fewer steps than those shown in the figures, or a combination of certain steps, or different steps.

[0168] The terms "first," "second," "third," "fourth," and the like (if any) in the description of the present invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments of the present invention described herein can be implemented in orders other than those illustrated or described herein. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products, or apparatus.

[0169] Those skilled in the art will appreciate that all or some of the steps in the method disclosed above, the corresponding system can be implemented as software, firmware, hardware and its appropriate combination. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include a computer-readable storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer-readable storage medium is included in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data) and is volatile and non-volatile, removable and non-removable. The medium includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, disk storage or other magnetic storage device, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0170] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the invention is not limited thereby. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the invention should be within the scope of the invention.

Claims

1. A model optimization method based on inverse normalization, characterized in that: include: Obtaining a deep neural network model that has been preliminarily trained based on a preset sample data set, and calculating the inverse matrix of each network layer in the deep neural network model; wherein the sample data set includes molecular structure data and property data; and the deep neural network model is a SchNet network model for predicting the structure and properties of molecules; Obtaining the inverse matrix of the Fisher information matrix according to the inverse matrix of each network layer; Obtaining a plurality of intermediate gradient values ​​according to the inverse matrix of the Fisher information matrix; Calculating an average gradient term and an average square term based on the plurality of intermediate gradient values; Dividing the mean square term according to the inverse matrix of the Fisher information matrix to obtain a gradient vector corresponding to each of the network layers; Normalizing the gradient vector of each network layer to obtain a normalized mean square term; Adjust the variables to be optimized in the deep neural network model according to the normalized mean square term and the mean gradient term to obtain an optimized deep neural network model; The target molecule-related data is input into the optimized SchNet network model to predict the molecular structure and molecular properties corresponding to the target molecule according to the output results of the optimized SchNet network model.

2. The model optimization method based on inverse normalization according to claim 1, characterized in that: Calculating the inverse matrix of each network layer in the deep neural network model includes: Calculate the forward propagation feature matrix and the backward propagation feature matrix of each network layer in the deep neural network model according to the sample data set; The forward propagation feature matrix and the backward propagation feature matrix corresponding to each network layer are respectively subjected to Kronecker product calculation processing, and the obtained product result is determined as the inverse matrix of each network layer.

3. The model optimization method based on inverse normalization according to claim 1, characterized in that: The obtaining of the inverse matrix of the Fisher information matrix according to the inverse matrix of each network layer includes: Obtain a plurality of diagonal block units according to the inverse matrix of each of the network layers; Splicing each of the diagonal block units according to a preset rule to obtain a Fisher information matrix, so that the main diagonal line of each of the diagonal block units corresponds to a part of the main diagonal line of the Fisher information matrix in sequence; An inverse operation is performed on the Fisher information matrix to obtain an inverse matrix of the Fisher information matrix.

4. The model optimization method based on inverse normalization according to claim 1, characterized in that: The method of obtaining a plurality of intermediate gradient values ​​according to the inverse matrix of the Fisher information matrix includes: Obtaining an iterative gradient value for each iteration of the deep neural network model; A plurality of intermediate gradient values ​​are obtained according to the iterative gradient value of each iteration and the inverse matrix of the Fisher information matrix.

5. The model optimization method based on inverse normalization according to claim 1, characterized in that: The calculating the average gradient term and the average square term according to the plurality of intermediate gradient values ​​comprises: Initialize the average gradient term and the average square term; performing a first exponentially weighted moving average update on the average gradient term according to a preset first attenuation coefficient and the plurality of intermediate gradient values ​​to obtain the average gradient term; The average gradient term is updated by performing a second exponentially weighted moving average according to a preset second attenuation coefficient and the squares of the plurality of intermediate gradient values ​​to obtain the average square term.

6. The model optimization method based on inverse normalization according to claim 3, characterized in that: The dividing process of the mean square term according to the inverse matrix of the Fisher information matrix to obtain the gradient vector corresponding to each of the network layers includes: Flattening the mean square term to obtain a gradient column vector; The gradient column vector is multiplied by each of the diagonal block units in the inverse matrix of the Fisher information matrix to obtain a gradient vector of the network layer corresponding to each diagonal block unit, wherein the gradient vector is a column vector or a row vector.

7. The model optimization method based on inverse normalization according to claim 1, characterized in that: Normalizing the gradient vector of each network layer to obtain a unitized mean square term includes: Performing arithmetic mean calculation on the gradient vectors of each network layer respectively to obtain the average gradient vector of each network layer; Multiplying the average gradient vector of each network layer by a unit vector of the same length to obtain a normalized gradient vector corresponding to each network layer; A normalized mean square term is obtained according to the normalized gradient vector corresponding to each of the network layers.

8. The model optimization method based on inverse normalization according to claim 1, characterized in that: The adjusting the variables to be optimized in the deep neural network model according to the normalized mean square term and the mean gradient term includes: Get the current value of the variable to be optimized; Calculating an adjustment value according to a preset learning rate, a preset smoothing term, the normalized mean square term, and the mean gradient term; The value of the variable to be optimized is determined as the difference between the current value and the adjusted value.

9. A model optimization device based on inverse normalization, characterized in that: include: a first inverse matrix calculation unit, configured to obtain a deep neural network model that has been preliminarily trained based on a preset sample data set, and calculate an inverse matrix of each network layer in the deep neural network model; wherein the sample data set includes molecular structural data and property data; and the deep neural network model is a SchNet network model for predicting molecular structure and properties; A second inverse matrix calculation unit, configured to obtain an inverse matrix of a Fisher information matrix according to the inverse matrix of each network layer; an intermediate gradient value calculation unit, configured to obtain a plurality of intermediate gradient values ​​according to the inverse matrix of the Fisher information matrix; an average gradient value calculation unit, configured to calculate an average gradient term and an average square term based on the plurality of intermediate gradient values; A gradient vector calculation unit, configured to divide the mean square term according to the inverse matrix of the Fisher information matrix to obtain a gradient vector corresponding to each of the network layers; A normalization calculation unit, configured to perform normalization processing on the gradient vector of each network layer to obtain a unitized mean square term; A parameter adjustment unit is used to adjust the variables to be optimized in the deep neural network model according to the normalized mean square term and the mean gradient term to obtain an optimized deep neural network model; input target molecule related data into the optimized SchNet network model to predict the molecular structure and molecular properties corresponding to the target molecule according to the output results of the optimized SchNet network model.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the model optimization method based on inverse normalization according to any one of claims 1 to 8 is implemented.

11. A computer-readable storage medium, characterized in that A computer program is stored, and when the computer program is executed by a processor, the model optimization method based on inverse normalization according to any one of claims 1 to 8 is implemented.

Images