Neural network training method based on back propagation algorithm

Abstract

The invention discloses a neural network training method based on a back propagation algorithm, and relates to the technical field of network training, and the method comprises the steps: obtaining the neuron output data of each neural network layer through a forward propagation process, and constructing the output distribution data reflecting the activation features of each layer of neuron; obtaining error data based on a back propagation process, establishing error term distribution of each neural network layer, and constructing an entropy change estimation factor for measuring network information state change; performing disturbance control on the error data according to the value trend of the entropy change estimation factor, and dynamically adjusting the parameter updating amplitude and direction of the neural network by the processed error data and the output distribution data; and weight iteration of the neural network is guided through the parameter updating behavior. According to the method, error disturbance control and dynamic parameter adjustment are combined, a whole set of mechanism of information perception-disturbance control-dynamic optimization is formed, and dynamic perception and accurate guidance of information state evolution in the neural network training process are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of network training, and in particular to a neural network training method based on a back propagation algorithm. Background Art

[0002] In recent years, with the rapid development of artificial intelligence technology, neural networks have become an important supporting technology in image recognition, natural language processing, intelligent control and other fields. In order to improve the learning effect and generalization ability of neural networks, the backpropagation algorithm, as a mainstream training method, has been widely used in the parameter optimization process of various neural network models. The algorithm calculates the output error and backpropagates the error information layer by layer to guide the adjustment of weights and bias parameters, thereby continuously reducing the deviation between the network output and the target value.

[0003] In the existing technology, the traditional backpropagation algorithm focuses on error backpropagation and gradient descent optimization during training, and lacks a detailed description and dynamic response mechanism for the state changes of each layer within the neural network. Especially in deep network training, as the network depth increases, the neuron activation characteristics and error information are prone to gradient vanishing, gradient explosion, unstable convergence and other problems during the multi-layer transmission process, resulting in a decrease in network training speed, uncontrolled parameter adjustment amplitude, and even training failure. In addition, existing backpropagation-based training methods usually adopt a fixed or simple adaptive learning rate adjustment strategy, lacking a dynamic intervention mechanism for the changing state of information within the network. They cannot effectively adapt to the complex nonlinear evolution characteristics of neural networks at different training stages, and are prone to problems such as overfitting, underfitting, and training oscillation.

[0004] To address the above problems, some existing technologies attempt to introduce regularization technology, gradient clipping methods or learning rate decay mechanisms to improve training stability. However, most of these improvement schemes are based on static control or adjustments based on empirical rules. They lack a mechanism for adaptive intervention based on the changing trend of information entropy within the neural network. It is difficult to achieve dynamic perception and precise guidance of the evolution of information states during neural network training. Therefore, it is necessary to design a neural network training method based on the backpropagation algorithm to solve the above problems. Summary of the Invention

[0005] The purpose of this section is to summarize some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract and title of this application to avoid obscuring the purpose of this section, the abstract and the title of the invention, and such simplifications or omissions should not be used to limit the scope of the present invention.

[0006] In view of the above problems existing in the existing neural network training method based on the back propagation algorithm, the present invention is proposed.

[0007] Therefore, the purpose of the present invention is to provide a neural network training method based on the backpropagation algorithm, which is suitable for solving the problem of lack of a mechanism for adaptive intervention based on the changing trend of information entropy within the neural network, and difficulty in achieving dynamic perception and precise guidance of the evolution of information status during neural network training.

[0008] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0009] In a first aspect, an embodiment of the present invention provides a neural network training method based on a back-propagation algorithm, comprising:

[0010] Obtain the neuron output data of each neural network layer through the forward propagation process, and construct the output distribution data reflecting the activation characteristics of neurons in each layer;

[0011] Based on the back-propagation process, error data is obtained, the error term distribution of each neural network layer is established, and an entropy change estimation factor is constructed to measure the change of network information state;

[0012] The error data is disturbed and controlled according to the value trend of the entropy change estimation factor. The processed error data and output distribution data are used to dynamically adjust the parameter update amplitude and direction of the neural network.

[0013] The parameter update behavior is used to guide the iteration of the neural network weights, so that the entropy change estimation factors of each layer of the neural network tend to converge to a stable interval, and the error term distribution converges to a target error threshold range.

[0014] As a preferred solution of the neural network training method based on the back propagation algorithm described in the present invention, the output distribution data includes the mean, variance and sparsity index of the output values ​​of each layer of neurons, and the relative entropy difference between layers is calculated based on the mean, variance and sparsity index of the output values ​​of each layer of neurons.

[0015] As a preferred solution of the neural network training method based on the back propagation algorithm of the present invention, the disturbance processing of the error data is based on the disturbance factor calculation model constructed by the entropy change estimation factor, and the construction process of the disturbance factor calculation model is as follows:

[0016] Based on the error term obtained during the back-propagation process, the error term is multiplied by the entropy change estimation factor to construct the initial structure of the disturbance factor calculation model;

[0017] The initial perturbation term is defined as:

[0018]

[0019] The hyperbolic tangent function is selected as the nonlinear mapping form of the disturbance adjustment function, the entropy change estimation factor change trend term and the adjustment coefficient are introduced, and the error disturbance expression is transformed into a disturbance adjustment formula.

[0020] In order to realize nonlinear regulation, tanh(.), adjustment coefficient α and ln(1+β|ΔS|) are introduced, and the expression is:

[0021]

[0022] Among them, Ψ1 is the final perturbation factor, dimensionless, used to adjust the error term of the output layer, α is the adjustment coefficient, β is the offset coefficient, ΔH L The magnitude of the change in the output layer entropy change estimation factor, The fractional normalization structure represents the normalized amount of change in the entropy change estimation factor;

[0023] The absolute value of the entropy change estimation factor and itself are combined to form a fractional normalization structure, and the minimum disturbance control term is introduced to obtain the disturbance factor calculation model, which is expressed as follows:

[0024] Ψ=Ψ1+γ;

[0025] Among them, Ψ is the final perturbation factor, which is input into the back propagation algorithm as a dynamic factor for error adjustment ratio, and γ represents the minimum perturbation value.

[0026] As a preferred solution of the neural network training method based on the back propagation algorithm described in the present invention, wherein:

[0027] If the entropy change estimation factor changes less than the lower threshold -ξ, it indicates that the response of the parameters of each layer in the neural network structure to the input sample gradually stabilizes, triggering the disturbance suppression mechanism;

[0028] If the absolute value of the change in the entropy change estimation factor is greater than the upper threshold ξ, it indicates that the neural network parameters are in the active adjustment stage, triggering the disturbance enhancement mechanism;

[0029] If the absolute value of the change in the entropy change estimation factor is not less than the lower threshold -ξ and not greater than the upper threshold ξ, it indicates that the neural network parameters have entered a stable adjustment stage, triggering the disturbance holding mechanism.

[0030] As a preferred solution of the neural network training method based on the back propagation algorithm described in the present invention, wherein:

[0031] The specific implementation steps of the disturbance suppression mechanism include:

[0032] Execute exponential decay function calculation, input the change amplitude of the entropy change estimation factor into the disturbance factor calculation model, and output the disturbance suppression factor for suppressing the error signal amplitude;

[0033] Multiply the error data obtained by the back propagation process by the disturbance suppression factor element by element, and input the result into the neural network back propagation process as a parameter update;

[0034] The specific implementation steps of the disturbance enhancement mechanism include:

[0035] Execute logarithmic growth function calculation, input the change amplitude of the entropy change estimation factor into the disturbance factor calculation model, and output the disturbance enhancement factor for enhancing the response strength of the error signal;

[0036] Multiply the error data obtained by the back propagation process by the disturbance enhancement factor element by element, and input the result into the neural network back propagation process as a parameter update;

[0037] The specific implementation steps of the disturbance holding mechanism include:

[0038] Keep the disturbance control factor constant and do not introduce disturbance changes;

[0039] The error data obtained by the back propagation process is multiplied element-by-element by the disturbance control factor of the previous cycle, and the result is input into the neural network back propagation process as a parameter update.

[0040] As a preferred solution of the neural network training method based on the back propagation algorithm described in the present invention, wherein:

[0041] The generation process of the disturbance suppression factor includes:

[0042] Extract the magnitude of the entropy change estimation factor;

[0043] The change amplitude of the entropy change estimation factor is input as an input variable into the exponential decay function to generate a disturbance suppression factor;

[0044] Normalized suppression disturbance suppression factor range;

[0045] The generation process of the disturbance enhancement factor includes:

[0046] Extract the magnitude of the entropy change estimation factor;

[0047] The change amplitude of the entropy change estimation factor is input as an input variable into the logarithmic growth function to generate the disturbance enhancement factor;

[0048] Limit the upper range of the disturbance enhancement factor.

[0049] As a preferred embodiment of the neural network training method based on the back propagation algorithm described in the present invention, wherein: in the disturbance suppression mechanism, the change amplitude of the entropy change estimation factor is refined by setting a first disturbance suppression threshold and a second disturbance suppression threshold, the neural network disturbance state interval is divided, and hierarchical disturbance processing is performed based on the judgment result, the change amplitude of the entropy change estimation factor in the current training cycle compared with the previous cycle is calculated, and the change amplitude of the entropy change estimation factor is generated, and the change amplitude of the entropy change estimation factor is compared with the preset first disturbance suppression threshold and the second disturbance suppression threshold for judgment;

[0050] If the change amplitude of the entropy change estimation factor is less than the first disturbance suppression threshold, the neural network training state is a low disturbance state, the exponential decay function is called to generate the first disturbance suppression factor, and the first parameter configuration scheme is adopted;

[0051] If the change amplitude of the entropy change estimation factor is not less than the first disturbance suppression threshold and less than the second disturbance suppression threshold, the neural network training process is determined to be in a medium disturbance state, and the exponential decay function is called to generate the second disturbance suppression factor, and the next step is executed;

[0052] When the neural network training state is a medium disturbance state, a first critical ratio threshold is further added, and the excess ratio of the change amplitude of the current entropy change estimation factor relative to the first disturbance suppression threshold is calculated;

[0053] If the excess ratio is less than the first critical ratio threshold, it is determined that the neural network training process is in a mild disturbance state, and the second parameter is used to configure the first sub-scheme;

[0054] If the excess ratio is greater than or equal to the first critical ratio threshold, it is determined that the neural network training process is in a state of intensified disturbance, and the second parameter configuration second sub-scheme is executed;

[0055] If the change amplitude of the entropy change estimation factor is greater than or equal to the second disturbance suppression threshold, it is determined that the neural network training process is in a high disturbance state, and an exponential decay function is called to generate a third disturbance suppression factor, and a third parameter configuration scheme is adopted;

[0056] In the disturbance enhancement mechanism, the change amplitude of the entropy change estimation factor is refined by setting a first disturbance enhancement threshold and a second disturbance enhancement threshold, the neural network disturbance state interval is divided, and hierarchical disturbance processing is performed based on the judgment result, and the change amplitude of the entropy change estimation factor is compared with the preset first disturbance enhancement threshold and the second disturbance enhancement threshold for judgment;

[0057] If the change amplitude of the entropy change estimation factor is less than the first disturbance enhancement threshold, it is determined that the neural network training process is in the insufficient disturbance stage, the logarithmic growth function is called to generate the first disturbance enhancement factor, and the fourth parameter configuration scheme is adopted;

[0058] If the change amplitude of the entropy change estimation factor is not less than the first disturbance enhancement threshold and is less than the second disturbance enhancement threshold, it is determined that the neural network training process is in the moderate disturbance stage, the logarithmic growth function is called to generate the second disturbance suppression factor, and the next step is executed;

[0059] When the current neural network training process is in the moderate disturbance stage, a second critical ratio threshold is further added, and the excess ratio of the change amplitude of the current entropy change estimation factor relative to the first disturbance enhancement threshold is calculated;

[0060] If the excess ratio is less than the second critical ratio threshold, it is determined that the neural network training process is in the disturbance mild enhancement stage, and the fifth parameter is used to configure the first sub-scheme;

[0061] If the excess ratio is greater than or equal to the second critical ratio threshold, it is determined that the neural network training process is in the disturbance significantly enhanced stage, and the fifth parameter is used to configure the second sub-scheme;

[0062] If the change amplitude of the entropy change estimation factor is greater than or equal to the second disturbance enhancement threshold, it is determined that the neural network training process is in the disturbance over-enhancement stage, the logarithmic growth function is called to generate the third disturbance suppression factor, and the sixth parameter configuration scheme is executed.

[0063] As a preferred solution of the neural network training method based on the back propagation algorithm of the present invention, the construction process of the disturbance control factor is as follows:

[0064] Extract the magnitude of the entropy change estimation factor;

[0065] The change amplitude of the entropy change estimation factor of the current cycle and its current value are combined to form a fractional normalization structure to generate a base value for calculating the disturbance control factor. The disturbance adjustment coefficient and the minimum disturbance threshold are introduced to limit the range of the disturbance control factor. The disturbance control factor of the previous cycle is kept constant to build a continuous adjustment structure for the disturbance control factor.

[0066] On the premise that the neural network is in a stable adjustment stage, the disturbance control factor calculated in the previous step is used as the control parameter of this cycle.

[0067] In a second aspect, an embodiment of the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, any step of the neural network training method based on the backpropagation algorithm as described in the first aspect of the present invention is implemented.

[0068] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, any step of the neural network training method based on the backpropagation algorithm as described in the first aspect of the present invention is implemented.

[0069] The present invention has the following beneficial effects: The present invention introduces an entropy change estimation factor to measure the changing trend of the output distribution of neurons in each layer in real time, giving the training process the ability of "dynamic information perception", so that the neural network can adaptively adjust the training pace, better adapt to complex data distribution, and improve training quality. Based on the changing trend of the entropy change estimation factor, the present invention introduces nonlinear perturbations to the error term, which can suppress the gradient when it is too large and moderately enhance it when it is too small, thereby dynamically stabilizing the entire backpropagation chain;

[0070] The present invention dynamically adjusts the amplitude and direction of each parameter update by combining the processed error data and output distribution data, and guides the entropy change estimation factors of each layer to gradually converge to a stable interval, so that the training process not only focuses on minimizing the error, but also takes into account the orderliness and structure within the network, thereby improving the robustness and interpretability of the model. The present invention is not a simple improvement on the traditional backpropagation training process, but systematically introduces the idea of ​​dynamic regulation of entropy change, combines error disturbance control with dynamic parameter adjustment, and forms a complete mechanism of information perception-disturbance control-dynamic optimization. This method is not limited to a specific neural network architecture and is applicable to a variety of application scenarios such as image recognition, natural language processing, speech recognition, and reinforcement learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort. Among them:

[0072] Figure 1 This is a schematic diagram of the overall process of the neural network training method based on the back propagation algorithm proposed in the present invention;

[0073] Figure 2 This is a schematic diagram of the judgment logic of the disturbance suppression mechanism of the neural network training method based on the back propagation algorithm proposed in the present invention;

[0074] Figure 3 This is a schematic diagram of the judgment logic of the disturbance enhancement mechanism of the neural network training method based on the back propagation algorithm proposed in the present invention. DETAILED DESCRIPTION

[0075] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0076] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0077] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0078] Furthermore, the present invention is described in detail with reference to schematic diagrams. For ease of illustration, when describing the embodiments of the present invention, cross-sectional views illustrating device structures may be partially enlarged and not to scale. Furthermore, the schematic diagrams are merely illustrative and should not limit the scope of protection of the present invention. Furthermore, in actual production, the three-dimensional dimensions of length, width, and depth should be included.

[0079] Example 1

[0080] Reference Figure 1-Figure 3 , which is an embodiment of the present invention, provides a neural network training method based on the backpropagation algorithm, which is particularly suitable for convolutional neural network (CNN) training in the field of image recognition, and is used to optimize model performance in scenarios such as industrial defect detection and medical image analysis. It can also be extended to other fields such as natural language processing and speech recognition.

[0081] Existing neural network training methods based on backpropagation algorithms have the following problems: some existing technologies attempt to introduce regularization technology, gradient clipping methods or learning rate decay mechanisms to improve training stability, but most of these improvement schemes are based on static control or adjustments based on empirical rules. They lack a mechanism for adaptive intervention based on the changing trend of information entropy within the neural network, making it difficult to achieve dynamic perception and precise guidance of the evolution of information states during neural network training.

[0082] This application provides a method that can effectively solve the above-mentioned problems. Next, we will combine multiple embodiments to explain in detail how to implement the neural network training method based on the backpropagation algorithm.

[0083] Figure 1 The overall flow chart of the neural network training method based on the back propagation algorithm is shown, including:

[0084] The neuron output data of each neural network layer is obtained through the forward propagation process, and the output distribution data reflecting the activation characteristics of each layer of neurons is constructed. The output distribution data includes the mean, variance and sparsity index of the output value of each layer of neurons.

[0085] In this embodiment, a convolutional neural network (CNN) is used as an example. Its typical structure includes convolutional layers, pooling layers, and fully connected layers. The convolutional layer is used to extract the spatial features of the input image and output a multi-channel feature map; the pooling layer downsamples the feature map and outputs the reduced-dimensional features; the fully connected layer maps the features into classification probabilities and outputs a vectorized representation. The specific implementation steps are as follows:

[0086] During the forward propagation process, an image is input and calculated through each layer of CNN to obtain the neuron activation value of each layer; for the convolution layer, the activation value is the multi-channel feature map; for the pooling layer, the activation value is the downsampled feature map; for the fully connected layer, the activation value is a one-dimensional vector; the activation value of each layer is processed by the activation function to ensure non-negativity.

[0087] For each layer activation value, the following statistical indicators are calculated to construct the output distribution data; for the mean, it reflects the average level of activation intensity; for the convolution layer, the mean of all pixel values ​​of each channel feature map is calculated to obtain the mean vector of each channel. Similarly, the mean of the downsampled feature map of each channel in the pooling layer can be obtained; for the fully connected layer, the mean of all neuron output values ​​is calculated; for the variance, it reflects the dispersion of the activation value. For the convolution layer, the variance of each channel feature map is calculated; for the fully connected layer, the variance of all neuron output values ​​is calculated;

[0088] Of particular importance is the sparsity index, which reflects the proportion of zero values ​​in the activation value and measures the sparsity of the activation. For the convolutional layer, the proportion of zero-valued pixels in the feature map of each channel is calculated. The specific formula is as follows:

[0089]

[0090] Among them, s i is the sparsity index of the i-th channel feature map, indicating the proportion of zero-value pixels in the channel feature map, and is used to quantify the sparsity of neuron activation. In the present invention, sparsity reflects the distribution characteristics of the activation values ​​of the convolutional layer and is an important component of the output distribution data, which is used for the subsequent calculation of the inter-layer relative entropy difference and the entropy change estimation factor; i is the channel index of the feature map, indicating the i-th channel of the convolutional layer output, H is the height of the feature map, W is the width of the feature map, h is the pixel index in the height direction of the feature map, indicating a specific pixel position of the feature map in the vertical direction, and w is the pixel index in the width direction of the feature map, indicating a specific pixel position of the feature map in the horizontal direction; x i,h,w is the pixel value of the i-th channel feature map at position (h,w), which represents the neuron activation value at that position. In the present invention, x i ,h,w is the activation value calculated by the convolution layer through the convolution operation and activation function during the forward propagation process, which is usually a non-negative value. i ,h,w=0) is an indicator function, which is used to count the number of zero-value pixels in the feature map, reflecting the degree to which the neuron is not activated after the activation function is activated. The double summation of all pixel positions of the feature map means traversing all pixels of the feature map of the i-th channel and accumulating the number of zero-valued pixels. H·W is the total number of pixels in the feature map.

[0091] For the fully connected layer, the proportion of zero values ​​in the output vector is calculated. The specific formula is as follows:

[0092]

[0093] Where s is the sparsity index of the fully connected layer, which represents the proportion of zero-valued elements in the output vector and is used to quantify the sparsity of neuron activation in the fully connected layer. In the present invention, sparsity is part of the output distribution data of the fully connected layer and is used to analyze activation features and support the calculation of entropy change estimation factors. j is the element index in the output vector of the fully connected layer, representing the output of the jth neuron. N is the total number of neurons in the fully connected layer, that is, the number of dimensions of the output vector. j is the output value of the jth neuron in the fully connected layer, that is, the jth element of the output vector, ∏(y j =0) is an indicator function, which is used to count the number of zero-valued elements in the output vector, reflecting the proportion of neurons that are not activated after activation by the activation function.

[0094] Different types of neural network layers (convolutional layer, pooling layer, fully connected layer) have different activation characteristics, so their mean, variance and sparsity indicators are calculated separately, and the output distribution data is stored in a structured form.

[0095] The relative entropy difference between layers is calculated based on the mean, variance and sparsity indicators of the neuron output values ​​of each layer.

[0096] It should be noted that the specific steps for calculating the relative entropy difference between layers are as follows:

[0097] Assuming that the activation values ​​of layer l follow an approximately normal distribution, a probability density function is constructed based on the mean and variance;

[0098]

[0099] Among them, P l(x) is the probability density function of the activation value of the neurons in the lth layer, which represents the distribution characteristics of the activation value. μl is the mean of the activation value of the neurons in the lth layer, which reflects the average level of activation intensity. is the variance of the activation value of the neurons in the lth layer, reflecting the dispersion of the activation value. The continuous variable of the x activation value is used to define the probability density;

[0100] Calculate the relative entropy difference between the probability distribution of the lth layer and the l+1th layer. The specific formula is as follows:

[0101]

[0102] Among them, D KL (l,l+1) is the relative entropy difference between the probability distribution of layer l and layer l+1, which quantifies the difference in activation distribution between layers. l (x) and P l+1 (x) are the probability density functions of the activation values ​​of neurons in layer l and layer l+1 respectively;

[0103] Based on the back-propagation process, we obtain error data, establish the error term distribution of each neural network layer, and construct an entropy change estimation factor to measure the change of network information state. The specific steps are as follows:

[0104] During the backpropagation process, the error term of the output layer is calculated based on the difference between the target output and the actual output. The specific formula is as follows:

[0105] δ L =(y p -y t )·f'(z L );

[0106] Among them, δ L is the error term of the output layer, which represents the deviation between the predicted output and the true output, y p Represents the predicted output value of the neural network, y t is the true label value, f'(z L ) is the derivative of the output layer activation function, z L is the input of the output layer;

[0107] For each layer error term δ L Calculate its statistical characteristics, construct the error term distribution, and calculate the mean of the error term. The formula is as follows:

[0108]

[0109] Among them, μδ L is the mean of the error term in the output layer, reflecting the average strength of the error, N L is the number of neurons in the output layer, δ L,iis the error term value of the i-th neuron in the output layer;

[0110] Calculate the square norm of the error term. The specific formula is as follows:

[0111]

[0112] in, is the square norm of the error term in the output layer, which represents the total energy of the error;

[0113] Construct the entropy change estimation factor for the output layer only, combining the relative entropy difference between the output layer and the previous layer and the square norm of the output layer error term. The specific formula is as follows:

[0114]

[0115] in, is the normalized output layer entropy change estimation factor, ∈ is the minimum positive value.

[0116] The error data is disturbed and controlled according to the value trend of the entropy change estimation factor. The processed error data and output distribution data are used to dynamically adjust the parameter update amplitude and direction of the neural network.

[0117] A disturbance factor calculation model is constructed based on the entropy change estimation factor and the disturbance of the error data is processed. The construction process of the disturbance factor calculation model is as follows:

[0118] Based on the error term obtained during the back-propagation process, the error term is multiplied by the entropy change estimation factor to construct the initial structure of the disturbance factor calculation model;

[0119] The initial perturbation term is defined as:

[0120]

[0121] The hyperbolic tangent function is selected as the nonlinear mapping form of the disturbance adjustment function, the entropy change estimation factor change trend term and the adjustment coefficient are introduced, and the error disturbance expression is transformed into a disturbance adjustment formula.

[0122] In order to realize nonlinear regulation, tanh(.), adjustment coefficient α and ln(1+β|ΔS|) are introduced, and the expression is:

[0123]

[0124] Among them, Ψ1 is the final perturbation factor, dimensionless, used to adjust the error term of the output layer, α is the adjustment coefficient, β is the offset coefficient, ΔH L The magnitude of the change in the output layer entropy change estimation factor, The fractional normalization structure represents the normalized amount of change in the entropy change estimation factor;

[0125] The absolute value of the entropy change estimation factor and itself are combined to form a fractional normalization structure, and the minimum disturbance control term is introduced to obtain the disturbance factor calculation model, which is expressed as follows:

[0126] Ψ=Ψ1+γ;

[0127] Among them, Ψ is the final disturbance factor, which is input into the back propagation algorithm as a dynamic factor of the error adjustment ratio, and γ is the minimum disturbance control term.

[0128] If the entropy change estimation factor changes less than the lower limit threshold -ξ, it indicates that the response of the parameters of each layer in the neural network structure to the input sample gradually stabilizes, triggering the disturbance suppression mechanism;

[0129] The specific implementation steps of the disturbance suppression mechanism include:

[0130] Execute exponential decay function calculation, input the change amplitude of the entropy change estimation factor into the disturbance factor calculation model, and output the disturbance suppression factor for suppressing the error signal amplitude;

[0131] The generation process of the disturbance suppression factor includes:

[0132] Extract the magnitude of the entropy change estimation factor;

[0133] The change amplitude of the entropy change estimation factor is input as an input variable into the exponential decay function to generate a disturbance suppression factor;

[0134] Normalized suppression disturbance suppression factor range;

[0135] The error data obtained by the back propagation process is multiplied element by element by the disturbance enhancement factor, and the result is input into the neural network back propagation process as a parameter update.

[0136] The error data obtained by the back-propagation process is multiplied element-by-element by the disturbance suppression factor, and the result is input into the neural network back-propagation process as a parameter update.

[0137] In the embodiment of the present application, an exponential decay function calculation is performed in the following form:

[0138]

[0139] in, is the disturbance suppression factor, which is used to reduce the amplitude of the output layer error term. κ is the attenuation coefficient. The amplitude of the entropy change estimation factor is integrated into the disturbance factor calculation model through the fractional normalization structure, as follows:

[0140]

[0141] in, is the final disturbance suppression factor.

[0142] If the absolute value of the change in the entropy change estimation factor is greater than the upper threshold ξ, it indicates that the neural network parameters are in the active adjustment stage, triggering the disturbance enhancement mechanism;

[0143] The specific implementation steps of the disturbance enhancement mechanism include:

[0144] Execute logarithmic growth function calculation, input the change amplitude of the entropy change estimation factor into the disturbance factor calculation model, and output the disturbance enhancement factor for enhancing the response strength of the error signal;

[0145] The generation process of the disturbance enhancement factor includes:

[0146] Extract the magnitude of the entropy change estimation factor;

[0147] The change amplitude of the entropy change estimation factor is input as an input variable into the logarithmic growth function to generate the disturbance enhancement factor;

[0148] Limit the upper range of the disturbance enhancement factor.

[0149] In the embodiment of the present application, the logarithmic growth function calculation is performed in the following form:

[0150]

[0151] in, is the disturbance enhancement factor, which is used to enhance the amplitude of the output layer error term. κ' is the growth coefficient, which controls the rate of logarithmic growth. The amplitude of the entropy change estimation factor is integrated into the disturbance factor calculation model through the normalized structure and the disturbance enhancement factor, in the following form:

[0152]

[0153] in, is the final disturbance enhancement factor.

[0154] If the absolute value of the change in the entropy change estimation factor is not less than the lower threshold -ξ and not greater than the upper threshold ξ, it indicates that the neural network parameters have entered a stable adjustment stage, triggering the disturbance holding mechanism.

[0155] The specific implementation steps of the disturbance holding mechanism include:

[0156] Keep the disturbance control factor constant and do not introduce disturbance changes;

[0157] The error data obtained by the back propagation process is multiplied element-by-element by the disturbance control factor of the previous cycle, and the result is input into the neural network back propagation process as a parameter update.

[0158] The construction process of the disturbance control factor is as follows:

[0159] Extract the magnitude of the entropy change estimation factor;

[0160] The change amplitude of the entropy change estimation factor of the current cycle and its current value are combined to form a fractional normalization structure to generate a base value for calculating the disturbance control factor. The disturbance adjustment coefficient and the minimum disturbance threshold are introduced to limit the range of the disturbance control factor. The disturbance control factor of the previous cycle is kept constant to build a continuous adjustment structure for the disturbance control factor.

[0161] On the premise that the neural network is in a stable adjustment stage, the disturbance control factor calculated in the previous step is used as the control parameter of this cycle.

[0162] Preferably, Figure 2 This is a schematic diagram of the disturbance suppression mechanism judgment logic of the neural network training method based on the backpropagation algorithm. In the disturbance suppression mechanism, the change amplitude of the entropy change estimation factor is refined by setting the first disturbance suppression threshold and the second disturbance suppression threshold. The neural network disturbance state interval is divided, and hierarchical disturbance processing is performed based on the judgment result. The change amplitude of the entropy change estimation factor in the current training cycle compared with the previous cycle, that is, the change amplitude of the entropy change estimation factor, is calculated. The change amplitude of the entropy change estimation factor is compared with the preset first disturbance suppression threshold and the second disturbance suppression threshold for judgment.

[0163] The first disturbance suppression threshold is used to identify low-disturbance states and trigger the disturbance suppression mechanism to stabilize parameter updates. This threshold is determined by analyzing the magnitude of the entropy change estimation factor of neural network training on multiple data sets and selecting the 25th percentile of the probability density distribution. This threshold is based on statistical data from at least 1,000 training iterations to ensure coverage of typical low-disturbance scenarios.

[0164] The second disturbance suppression threshold is used to distinguish between medium and high disturbance states, triggering a strong disturbance suppression mechanism to prevent training from getting out of control. This threshold is determined based on the 75th percentile of the distribution of the entropy change estimation factor change amplitude based on the same dataset. Combined with the analysis of 500 high-disturbance scenarios, its sensitivity to gradient explosion is verified.

[0165] Specifically, if the change amplitude of the entropy change estimation factor is less than the first disturbance suppression threshold, the neural network training state is a low disturbance state, an exponential decay function is called to generate a first disturbance suppression factor, and a first parameter configuration scheme is adopted;

[0166] The specific operation of the first parameter configuration scheme is as follows:

[0167] Multiply the current learning rate by the preset decay coefficient to generate a new learning rate;

[0168] It should be noted that multiplying the current learning rate by the preset decay coefficient to generate a new learning rate can reduce the step size of weight update. The reduced learning rate ensures more precise parameter adjustment.

[0169] Clipping is performed on the gradients calculated during backpropagation to limit the gradient norm to a preset range;

[0170] It should be noted that the clipping operation is performed on the gradient calculated during the back propagation process. The clipped gradient effectively prevents excessive parameter updates and maintains the smooth convergence of the neural network weights.

[0171] Updates to the neural network weights are performed based on the reduced learning rate and clipped gradients.

[0172] It should be noted that performing the update operation of the neural network weights can enable the neural network to gradually approach the target error threshold under a low disturbance state.

[0173] If the change amplitude of the entropy change estimation factor is not less than the first disturbance suppression threshold and less than the second disturbance suppression threshold, the neural network training process is determined to be in a medium disturbance state, and the exponential decay function is called to generate the second disturbance suppression factor, and the next step is executed;

[0174] When the neural network training state is a medium disturbance state, a first critical ratio threshold is further added, and the excess ratio of the change amplitude of the current entropy change estimation factor relative to the first disturbance suppression threshold is calculated;

[0175] It should be noted that in the backpropagation algorithm, the learning rate determines the magnitude of weight adjustment during the gradient descent process. It is a hyperparameter used to adjust the magnitude of weight updates during neural network training. Its value controls the step size of gradient descent. The preset attenuation coefficient is dynamically calculated through the change magnitude of the entropy change estimation factor. The specific calculation method is as follows:

[0176] From the above, we can see that the change range of the entropy change estimation factor is ΔH L ,define the preset attenuation coefficient as λ;

[0177] but

[0178] This expression ensures that when the entropy change estimate factor changes more, the decay coefficient is smaller (the learning rate decreases more), and vice versa, it is close to 1 (the learning rate changes less).

[0179] Among them, the first critical ratio threshold distinguishes between mild disturbance and aggravated disturbance in the medium disturbance state, and selects an appropriate parameter configuration scheme. The threshold is determined by cluster analysis based on the distribution of the excess ratio of the entropy change estimation factor relative to the first disturbance suppression threshold. The threshold is based on 800 medium disturbance training data to ensure the distinction between different disturbance degrees.

[0180] Specifically, if the excess ratio is less than a first critical ratio threshold, it is determined that the neural network training process is in a mild disturbance state, and the second parameter is used to configure the first sub-scheme;

[0181] The specific operations of the second parameter configuration first sub-scheme are as follows:

[0182] Multiply the current learning rate by the preset fine-tuning coefficient to generate a moderately reduced learning rate;

[0183] It should be noted that multiplying the current learning rate by the preset fine-tuning coefficient to generate a moderately reduced learning rate can balance training speed and stability. The fine-tuned learning rate ensures that the parameter update amplitude is moderate and supports the smooth optimization of the neural network.

[0184] Normalize the gradients calculated during backpropagation to scale the gradient vector to unit norm.

[0185] It should be noted that the normalized gradient can ensure the consistency of parameter update direction and improve the convergence efficiency of training;

[0186] Based on the fine-tuned learning rate and the normalized gradient, the neural network weights are updated; it should be noted that the updated weights can enable the neural network to maintain a stable optimization trend under mild perturbations.

[0187] Furthermore, in the first sub-scheme of the second parameter configuration, the fine-tuning coefficient is obtained by the following steps:

[0188] The relative deviation of the change amplitude of the entropy change estimation factor relative to the first disturbance suppression threshold is calculated to generate a deviation ratio; the deviation ratio is input into a preset linear mapping function to generate a fine-tuning coefficient within a preset range.

[0189] Specifically, if the excess ratio is greater than or equal to a first critical ratio threshold, it is determined that the neural network training process is in a state of aggravated disturbance, and the second parameter configuration second sub-scheme is executed;

[0190] The second parameter configuration second sub-scheme specific operations are as follows:

[0191] According to the excess ratio of the change in the entropy change estimation factor, the learning rate adjustment coefficient is dynamically calculated to generate a reduced learning rate;

[0192] It should be noted that dynamically calculating the learning rate adjustment coefficient and generating a reduced learning rate can effectively control the parameter update amplitude and adapt to the state of intensified disturbance;

[0193] Perform a sliding average operation on the gradient calculated during backpropagation to generate a smoothed gradient vector;

[0194] It should be noted that the smoothed gradient can reduce the mutation of parameter updates and improve the stability of the training process;

[0195] Update the neural network weights based on the dynamically adjusted learning rate and smoothed gradients;

[0196] It should be noted that the updated weights enable the neural network to gradually and stably converge under the condition of intensified disturbance.

[0197] Furthermore, in the second parameter configuration second sub-scheme, the fine-tuning coefficient is obtained by the following steps:

[0198] Calculating the excess ratio of the entropy change estimation factor relative to the first critical proportional threshold value to generate a proportional factor;

[0199] The scaling factor is input into a preset nonlinear mapping function to generate a fine-tuning coefficient within a preset range.

[0200] Specifically, if the change amplitude of the entropy change estimation factor is greater than or equal to the second disturbance suppression threshold, it is determined that the neural network training process is in a high disturbance state, an exponential decay function is called to generate a third disturbance suppression factor, and a third parameter configuration scheme is adopted;

[0201] The specific operations of the third parameter configuration scheme are as follows:

[0202] Multiply the current learning rate by a preset strong decay factor to generate a reduced learning rate;

[0203] It should be noted that multiplying the current learning rate by a preset strong decay coefficient to generate a significantly reduced learning rate can reduce the weight update step size. The reduced learning rate ensures that the parameter adjustment is highly controlled and prevents training oscillations.

[0204] Perform a strong clipping operation on the gradient calculated during backpropagation to limit the gradient norm to a lower preset range;

[0205] It should be noted that a strong clipping operation is performed on the gradients calculated during the back-propagation process. The clipped gradients effectively suppress excessive fluctuations in parameter updates and maintain training stability.

[0206] Perform updates to the neural network weights based on a significantly reduced learning rate and strongly clipped gradients;

[0207] It should be noted that the updated weights enable the neural network to gradually recover stable optimization under high disturbance conditions.

[0208] Furthermore, the strong attenuation coefficient is obtained by the following steps:

[0209] calculating the ratio of the entropy change estimation factor's change amplitude to the second disturbance suppression threshold value, and generating a proportional factor;

[0210] Input the scaling factor into a preset exponential decay function to generate a strong decay coefficient within a preset range.

[0211] Preferably, Figure 3 This is a schematic diagram of the disturbance enhancement mechanism judgment logic of the neural network training method based on the backpropagation algorithm. In the disturbance enhancement mechanism, the change amplitude of the entropy change estimation factor is refined by setting the first disturbance enhancement threshold and the second disturbance enhancement threshold. The neural network disturbance state interval is divided, and hierarchical disturbance processing is performed based on the judgment result. The change amplitude of the entropy change estimation factor is compared with the preset first disturbance enhancement threshold and the second disturbance enhancement threshold.

[0212] Among them, the first disturbance enhancement threshold is used to identify the insufficient disturbance state and trigger the disturbance enhancement mechanism to accelerate feature learning. The threshold is determined by analyzing the low-value distribution of the change amplitude of the entropy change estimation factor and selecting the 20th percentile. Based on 1000 low-disturbance training data, its effect on improving training speed is verified.

[0213] The second perturbation enhancement threshold distinguishes between moderate and excessive perturbation states, preventing training divergence. This threshold is determined based on the 80th percentile of the distribution of the magnitude of the entropy change estimate factor. This threshold, combined with an analysis of 600 high-perturbation enhancement scenarios, effectively limits excessive enhancement.

[0214] Specifically, if the change amplitude of the entropy change estimation factor is less than the first disturbance enhancement threshold, it is determined that the neural network training process is in the insufficient disturbance stage, the logarithmic growth function is called to generate the first disturbance enhancement factor, and the fourth parameter configuration scheme is adopted;

[0215] The specific operations of the fourth parameter configuration scheme are as follows:

[0216] Multiply the current learning rate by the preset enhancement factor to generate the improved learning rate;

[0217] It should be noted that multiplying the current learning rate by the preset enhancement coefficient to generate an increased learning rate can increase the weight update step size. The increased learning rate accelerates parameter adjustment and promotes rapid optimization of the neural network.

[0218] Perform an amplification operation on the gradient calculated during the backpropagation process to moderately increase the gradient norm to a preset range;

[0219] It should be noted that the gradient calculated during the backpropagation process is amplified to increase the gradient norm appropriately to a preset range. The amplified gradient enhances the strength of parameter updates and improves training efficiency.

[0220] Update the neural network weights based on the increased learning rate and amplified gradient;

[0221] It should be noted that the update operation of the neural network weights is performed, and the updated weights enable the neural network to quickly approach the target error threshold in the insufficient disturbance stage.

[0222] Furthermore, the enhancement coefficient is obtained by the following steps:

[0223] calculating a shortfall ratio of a change in the entropy change estimation factor relative to a first disturbance enhancement threshold value to generate a proportional factor;

[0224] The scaling factor is input into a preset linear growth function to generate an enhancement factor within a preset range.

[0225] If the change amplitude of the entropy change estimation factor is not less than the first disturbance enhancement threshold and is less than the second disturbance enhancement threshold, it is determined that the neural network training process is in the moderate disturbance stage, the logarithmic growth function is called to generate the second disturbance suppression factor, and the next step is executed;

[0226] When the current neural network training process is in the moderate disturbance stage, a second critical ratio threshold is further added, and the excess ratio of the change amplitude of the current entropy change estimation factor relative to the first disturbance enhancement threshold is calculated;

[0227] The second critical ratio threshold distinguishes between slightly enhanced and significantly enhanced states in the moderate disturbance stage, and selects an appropriate parameter configuration scheme. This threshold is determined by analyzing the excess ratio distribution of the entropy change estimation factor relative to the first disturbance enhancement threshold, and using decision tree analysis to determine the dividing point. It is based on 700 moderate disturbance training data (such as enterprise time series data) to ensure a balance between training efficiency and stability.

[0228] Specifically, if the excess ratio is less than a second critical ratio threshold, it is determined that the neural network training process is in a disturbance mild enhancement stage, and the first sub-scheme is configured using the fifth parameter;

[0229] The specific operations of the first sub-scheme of the fifth parameter configuration are as follows:

[0230] Multiply the current learning rate by the preset slight enhancement factor to generate a slightly increased learning rate;

[0231] It should be noted that multiplying the current learning rate by the preset slight enhancement coefficient to generate a slightly increased learning rate can moderately increase the weight update step size. The slightly increased learning rate ensures mild parameter adjustment and maintains training stability.

[0232] Perform a fine-tuning operation on the gradient calculated during the back-propagation process, slightly amplifying the gradient norm to a preset range;

[0233] It should be noted that fine-tuning is performed on the gradients calculated during backpropagation. The fine-tuned gradients enhance the response strength of parameter updates and improve training efficiency.

[0234] Update the neural network weights based on a slightly increased learning rate and fine-tuned gradients;

[0235] It should be noted that the update operation of the neural network weights is performed, and the updated weights enable the neural network to maintain efficient optimization in the stage of mild disturbance enhancement.

[0236] Furthermore, the mild enhancement coefficient is obtained by the following steps:

[0237] Calculating the ratio of the entropy change estimation factor's change amplitude to the first disturbance enhancement threshold to generate a proportional factor;

[0238] Input the scaling factor into a preset linear mapping function, generating a slight enhancement factor within a preset range.

[0239] Specifically, if the excess ratio is greater than or equal to a second critical ratio threshold, it is determined that the neural network training process is in a stage of significantly enhanced disturbance, and the fifth parameter is used to configure the second sub-scheme;

[0240] The specific operations of the second sub-scheme of the fifth parameter configuration are as follows:

[0241] Multiply the current learning rate by the preset significant enhancement factor to generate a significantly improved learning rate;

[0242] It should be noted that multiplying the current learning rate by the preset significant enhancement coefficient to generate a significantly improved learning rate can significantly increase the weight update step size. The significantly improved learning rate accelerates parameter adjustment and adapts to the significantly enhanced disturbance state.

[0243] Perform an enhancement operation on the gradient calculated during backpropagation, significantly amplifying the gradient norm to a preset range;

[0244] It should be noted that the gradient calculated during the back-propagation process is enhanced, and the enhanced gradient strengthens the parameter update force and improves the convergence speed of training;

[0245] Perform updates to neural network weights based on significantly increased learning rates and enhanced gradients;

[0246] It should be noted that the updated weights enable the neural network to efficiently approach the target error threshold when the disturbance is significantly enhanced.

[0247] Furthermore, the slightly significant enhancement coefficient is obtained by the following steps:

[0248] calculating the excess ratio of the entropy change estimation factor relative to the second critical proportional threshold value to generate a proportional factor;

[0249] The scaling factor is input into a preset nonlinear growth function to generate a significant enhancement coefficient within a preset range.

[0250] Specifically, if the change amplitude of the entropy change estimation factor is greater than or equal to the second disturbance enhancement threshold, it is determined that the neural network training process is in the disturbance over-enhancement stage, the logarithmic growth function is called to generate the third disturbance suppression factor, and the sixth parameter configuration scheme is executed;

[0251] The specific operations of the sixth parameter configuration scheme are as follows:

[0252] Multiply the current learning rate by the preset limit coefficient to generate a limited learning rate;

[0253] It should be noted that multiplying the current learning rate by the preset limit coefficient to generate a limited learning rate can reduce the weight update step size. The limited learning rate effectively controls the parameter adjustment amplitude and prevents excessive divergence of training;

[0254] Constraints are performed on the gradients calculated during backpropagation to limit the gradient norm to a lower preset range.

[0255] It should be noted that the gradient calculated during the back-propagation process is constrained. The constrained gradient suppresses excessive fluctuations in parameter updates and maintains training stability.

[0256] Update the neural network weights based on a limited learning rate and constrained gradients;

[0257] It should be noted that the update operation of the neural network weights is performed, and the updated weights enable the neural network to gradually restore stable optimization during the stage of excessive disturbance enhancement.

[0258] Furthermore, the restriction coefficient is obtained by the following steps:

[0259] calculating the ratio of the entropy change estimation factor's change amplitude to the second disturbance enhancement threshold value, and generating a proportional factor;

[0260] The scaling factor is input into a preset exponential decay function to generate a limiting coefficient within a preset range.

[0261] Step 4: Guide the iteration of neural network weights through parameter update behavior, so that the entropy change estimation factors of each layer of the neural network tend to converge to a stable interval, and the error term distribution converges to the target error threshold range.

[0262] For example, assume that in a certain intelligent manufacturing enterprise, an industrial defect detection system based on a deep neural network is used to identify surface defects on metal parts on the production line. The system uses a deep convolutional neural network (CNN) to process 256×256 resolution grayscale images captured by a high-speed camera to detect small defects such as scratches and cracks. Due to changes in the production line environment (such as lighting and temperature), the neural network experiences convergence instability during training, manifesting as fluctuations in defect recognition rate (e.g., from 85% to 70%) and occasional uncontrolled parameter updates, resulting in reduced detection accuracy. To address this issue, the enterprise uses the neural network training method based on the backpropagation algorithm proposed in this invention to optimize model training. The system obtains the output data of each neuron layer through forward propagation, constructs the output distribution data, and calculates the error data based on backpropagation to generate an entropy change estimation factor to measure the change in the network information state. When the entropy change estimation factor changes slightly, the disturbance suppression mechanism is triggered, generating a disturbance suppression factor through an exponential decay function, reducing the learning rate to stabilize parameter updates. When the entropy change estimation factor changes significantly, the disturbance enhancement mechanism is triggered, generating a disturbance enhancement factor through a logarithmic growth function, and the learning rate is moderately increased to accelerate feature learning. When the entropy change estimation factor changes moderately, the disturbance retention mechanism is triggered, maintaining the disturbance control factor of the previous cycle to maintain stable updates. After multiple iterations, the entropy change estimation factor of the neural network converges to a stable range, and the error distribution approaches the target threshold. In practical applications, this optimized neural network model significantly improves the stability of defect detection and reduces false alarms and missed detections. Maintenance personnel confirmed through training logs that the dynamic regulation of the entropy change estimation factor effectively avoids parameter oscillation, thereby ensuring the continuous operation of the production line and product quality.

[0263] In summary, the present invention introduces an entropy change estimation factor to measure the changing trend of the output distribution of neurons in each layer in real time, giving the training process the ability to "dynamically perceive information," enabling the neural network to adaptively adjust the training pace, better adapt to complex data distributions, and improve training quality. Based on the changing trend of the entropy change estimation factor, a nonlinear perturbation is applied to the error term, suppressing it when the parameter update amplitude is too large and moderately enhancing it when the parameter update amplitude is too small, thereby dynamically stabilizing the backpropagation process;

[0264] The present invention dynamically adjusts the amplitude and direction of each parameter update by combining the processed error data and output distribution data. By guiding the entropy change estimation factors of each layer to gradually converge to a stable range, the training process not only focuses on minimizing errors, but also takes into account the orderliness and structure within the network, thereby improving the robustness and interpretability of the model. The present invention is not a simple improvement on the traditional backpropagation training process, but systematically introduces the idea of ​​dynamic regulation of entropy change, combines error disturbance control with dynamic parameter adjustment, and forms a complete mechanism of information perception-disturbance control-dynamic optimization. This method is not limited to a specific neural network architecture and is applicable to a variety of application scenarios such as image recognition, natural language processing, speech recognition, and reinforcement learning.

[0265] Example 2 is an embodiment of the present invention, which is different from the previous embodiment in that:

[0266] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0267] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0268] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0269] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0270] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A neural network training method based on a back-propagation algorithm, characterized in that: include: Obtain the neuron output data of each neural network layer through the forward propagation process, and construct the output distribution data reflecting the activation characteristics of neurons in each layer; Based on the back-propagation process, error data is obtained, the error term distribution of each neural network layer is established, and an entropy change estimation factor is constructed to measure the change of network information state; The error data is disturbed and controlled according to the value trend of the entropy change estimation factor. The processed error data and output distribution data are used to dynamically adjust the parameter update amplitude and direction of the neural network. The parameter update behavior is used to guide the iteration of the neural network weights, so that the entropy change estimation factors of each layer of the neural network tend to converge to a stable interval, and the error term distribution converges to a target error threshold range.

2. The neural network training method based on the back propagation algorithm according to claim 1, characterized in that: The output distribution data includes the mean, variance and sparsity index of the output values ​​of neurons in each layer, and the relative entropy difference between layers is calculated based on the mean, variance and sparsity index of the output values ​​of neurons in each layer.

3. The neural network training method based on the back propagation algorithm according to claim 1, characterized in that: The disturbance processing of the error data is based on the disturbance factor calculation model constructed by the entropy change estimation factor, and the construction process of the disturbance factor calculation model is as follows: Based on the error term obtained during the back-propagation process, the error term is multiplied by the entropy change estimation factor to construct the initial structure of the disturbance factor calculation model; The hyperbolic tangent function is selected as the nonlinear mapping form of the disturbance adjustment function, the entropy change estimation factor change trend term and the adjustment coefficient are introduced, and the error disturbance expression is transformed into a disturbance adjustment formula. The absolute value of the entropy change estimation factor and itself are combined to construct a fractional normalization structure, and the minimum disturbance control term is introduced to obtain the disturbance factor calculation model.

4. The neural network training method based on the back propagation algorithm according to claim 3, characterized in that: If the entropy change estimation factor changes less than the lower threshold -ξ, it indicates that the response of the parameters of each layer in the neural network structure to the input sample gradually stabilizes, triggering the disturbance suppression mechanism; If the absolute value of the change in the entropy change estimation factor is greater than the upper threshold ξ, it indicates that the neural network parameters are in the active adjustment stage, triggering the disturbance enhancement mechanism; If the absolute value of the change in the entropy change estimation factor is not less than the lower threshold -ξ and not greater than the upper threshold ξ, it indicates that the neural network parameters have entered a stable adjustment stage, triggering the disturbance holding mechanism.

5. The neural network training method based on the back propagation algorithm according to claim 4, characterized in that: The specific implementation steps of the disturbance suppression mechanism include: Execute exponential decay function calculation, input the change amplitude of the entropy change estimation factor into the disturbance factor calculation model, and output the disturbance suppression factor for suppressing the error signal amplitude; Multiply the error data obtained by the back propagation process by the disturbance suppression factor element by element, and input the result into the neural network back propagation process as a parameter update; The specific implementation steps of the disturbance enhancement mechanism include: Execute logarithmic growth function calculation, input the change amplitude of the entropy change estimation factor into the disturbance factor calculation model, and output the disturbance enhancement factor for enhancing the response strength of the error signal; Multiply the error data obtained by the back propagation process by the disturbance enhancement factor element by element, and input the result into the neural network back propagation process as a parameter update; The specific implementation steps of the disturbance holding mechanism include: Keep the disturbance control factor constant and do not introduce disturbance changes; The error data obtained by the back propagation process is multiplied element-by-element by the disturbance control factor of the previous cycle, and the result is input into the neural network back propagation process as a parameter update.

6. The neural network training method based on the back propagation algorithm according to claim 5, characterized in that: The generation process of the disturbance suppression factor includes: Extract the magnitude of the entropy change estimation factor; The change amplitude of the entropy change estimation factor is input as an input variable into the exponential decay function to generate a disturbance suppression factor; Normalized suppression disturbance suppression factor range; The generation process of the disturbance enhancement factor includes: Extract the magnitude of the entropy change estimation factor; The change amplitude of the entropy change estimation factor is input as an input variable into the logarithmic growth function to generate the disturbance enhancement factor; Limit the upper range of the disturbance enhancement factor.

7. The neural network training method based on the back propagation algorithm according to claim 6, characterized in that: In the disturbance suppression mechanism, a first disturbance suppression threshold and a second disturbance suppression threshold are set to perform a detailed judgment on the change amplitude of the entropy change estimation factor, divide the neural network disturbance state interval, and perform hierarchical disturbance processing based on the judgment result. The change amplitude of the entropy change estimation factor in the current training cycle compared with the previous cycle is calculated to generate the change amplitude of the entropy change estimation factor, and the change amplitude of the entropy change estimation factor is compared with the preset first disturbance suppression threshold and the second disturbance suppression threshold for judgment; If the change amplitude of the entropy change estimation factor is less than the first disturbance suppression threshold, the neural network training state is a low disturbance state, the exponential decay function is called to generate the first disturbance suppression factor, and the first parameter configuration scheme is adopted; If the change amplitude of the entropy change estimation factor is not less than the first disturbance suppression threshold and less than the second disturbance suppression threshold, the neural network training process is determined to be in a medium disturbance state, and the exponential decay function is called to generate the second disturbance suppression factor, and the next step is executed; When the neural network training state is a medium disturbance state, a first critical ratio threshold is further added, and the excess ratio of the change amplitude of the current entropy change estimation factor relative to the first disturbance suppression threshold is calculated; If the excess ratio is less than the first critical ratio threshold, it is determined that the neural network training process is in a mild disturbance state, and the second parameter is used to configure the first sub-scheme; If the excess ratio is greater than or equal to the first critical ratio threshold, it is determined that the neural network training process is in a state of aggravated disturbance, and the second sub-scheme is configured using the second parameter; If the change amplitude of the entropy change estimation factor is greater than or equal to the second disturbance suppression threshold, it is determined that the neural network training process is in a high disturbance state, and an exponential decay function is called to generate a third disturbance suppression factor, and a third parameter configuration scheme is adopted; In the disturbance enhancement mechanism, the change amplitude of the entropy change estimation factor is refined by setting a first disturbance enhancement threshold and a second disturbance enhancement threshold, the neural network disturbance state interval is divided, and hierarchical disturbance processing is performed based on the judgment result, and the change amplitude of the entropy change estimation factor is compared with the preset first disturbance enhancement threshold and the second disturbance enhancement threshold for judgment; If the change amplitude of the entropy change estimation factor is less than the first disturbance enhancement threshold, it is determined that the neural network training process is in the insufficient disturbance stage, the logarithmic growth function is called to generate the first disturbance enhancement factor, and the fourth parameter configuration scheme is adopted; If the change amplitude of the entropy change estimation factor is not less than the first disturbance enhancement threshold and less than the second disturbance enhancement threshold, it is determined that the neural network training process is in the moderate disturbance stage, the logarithmic growth function is called to generate the second disturbance suppression factor, and the next step is executed; When the current neural network training process is in the moderate disturbance stage, a second critical ratio threshold is further added, and the excess ratio of the change amplitude of the current entropy change estimation factor relative to the first disturbance enhancement threshold is calculated; If the excess ratio is less than the second critical ratio threshold, it is determined that the neural network training process is in the disturbance mild enhancement stage, and the fifth parameter is used to configure the first sub-scheme; If the excess ratio is greater than or equal to the second critical ratio threshold, it is determined that the neural network training process is in the disturbance significantly enhanced stage, and the fifth parameter is used to configure the second sub-scheme; If the change amplitude of the entropy change estimation factor is greater than or equal to the second disturbance enhancement threshold, it is determined that the neural network training process is in the disturbance over-enhancement stage, the logarithmic growth function is called to generate the third disturbance suppression factor, and the sixth parameter configuration scheme is executed.

8. The neural network training method based on the back propagation algorithm according to claim 5, characterized in that: The construction process of the disturbance control factor is as follows: Extract the magnitude of the entropy change estimation factor; The change amplitude of the entropy change estimation factor of the current cycle and its current value are combined to form a fractional normalization structure to generate a base value for calculating the disturbance control factor. The disturbance adjustment coefficient and the minimum disturbance threshold are introduced to limit the range of the disturbance control factor. The disturbance control factor of the previous cycle is kept constant to build a continuous adjustment structure for the disturbance control factor. On the premise that the neural network is in a stable adjustment stage, the disturbance control factor calculated in the previous step is used as the control parameter of this cycle.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the neural network training method based on the back propagation algorithm described in any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the neural network training method based on the back propagation algorithm described in any one of claims 1 to 8 are implemented.

Images