Neural network optimization method for data classification

Abstract

This invention relates to the field of deep learning technology and discloses a neural network optimization method for data classification, comprising: inputting a high-dimensional feature data tensor into a neural network model to calculate a classification boundary margin vector; calculating a sample-level adaptive gradient adjustment factor by combining the historical gradient variance within a defined interval; extracting the alignment entropy of the original gradient directions of each hidden layer and generating a weighted smooth alignment entropy; generating a parameter update step size tensor to update the model weight parameters accordingly; detecting the increase in classification loss, and when a mutation threshold is triggered, rolling back the weight parameters and scaling the step size to update again. This invention reduces the impact of easily classified samples on the accumulated gradient by dynamically scaling, alleviates the excessive smoothing of the decision boundary, and improves classification accuracy; it uses alignment entropy to decouple the layer update step size to eliminate gradient propagation mismatch, and ensures the continuity and stability of parameter optimization by mutation detection.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and more specifically to a neural network optimization method for data classification. Background Technology

[0002] Neural networks typically update model weights using backpropagation and gradient descent. During training, the classification loss is calculated at the output based on the input data, and then the loss gradient is propagated forward layer by layer using the chain rule. This adjusts the parameters of each layer of the network, allowing the network to gradually extract high-dimensional features and fit the classification decision boundary. This is a common training method for neural networks when classifying data. However, existing neural network training methods still have certain limitations for datasets with high-dimensional features, unstable data distribution, and ambiguous class boundaries. When some classes contain a large number of samples, or when the proportion of easily classifiable samples is high, these samples will generate a large number of gradients in similar directions and accumulate continuously. This weakens the influence of a few classes and samples near the class boundary on parameter updates. As training progresses, the classification decision boundary is prone to over-smoothing, which reduces the model's ability to distinguish data in the class boundary region. At the same time, traditional optimization methods usually use a uniform global update step size for each layer of the network, without fully considering the differences in feature extraction levels between deep and shallow layers. This can easily lead to inconsistent gradient updates between different layers and increase the risk of the model converging to a local minimum.

[0003] To mitigate the negative impact of uneven sample difficulty and hyperparameter fluctuations on model optimization, existing technologies attempt to improve training performance by reconstructing the loss function structure or adaptively adjusting scaling factors. For example, Chinese invention patent application CN113361346A discloses a face recognition method that replaces the adjustment parameter with an adaptive scale parameter. By combining boundary-based and mining-based loss functions, it uses batch global statistics to adaptively calculate the scale parameter and adjustment parameter to adjust the relative importance of easy and difficult samples. However, this type of method mainly relies on macroscopic statistical indicators such as the mean or median of the batch to complete parameter mapping, and cannot perform sample-level fine-grained gradient adjustment for the deviation of the decision boundary margin and historical gradient variance of individual samples. At the same time, it fails to consider the difference in gradient direction dispersion between different hidden layers of the neural network, and also lacks a parameter rollback and step size decay mechanism to deal with the drastic fluctuations in loss caused by sudden high noise. When dealing with high-dimensional non-stationary data classification tasks, it is still difficult to balance the discrimination accuracy of the decision boundary and the stability of parameter optimization.

[0004] Therefore, the technical problem to be solved by this invention is how to accurately reflect the certainty of sample classification results throughout the training process of a neural network, dynamically adjust the influence of different samples on gradient updates based on historical fluctuations, set different update step sizes based on the consistency of gradient directions of each hidden layer, and roll back parameter updates and readjust the training state when the loss changes abruptly. Summary of the Invention

[0005] To address the problems in the background technology, the specific technical solution adopted is as follows: A neural network optimization method for data classification, comprising the following steps:

[0006] Step S1: Obtain the high-dimensional feature data tensor containing multiple samples, the historical variance moving average, the preceding smooth alignment entropy, and the preceding loss value. Input the high-dimensional feature data tensor into the neural network model and calculate the classification probability vector and classification boundary margin vector of multiple samples.

[0007] Step S2: Calculate the boundary scaling factor based on the classification boundary margin vector and the historical variance moving average, limit the historical variance moving average within the preset upper and lower limit range to obtain the corrected variance scalar, and use the corrected variance scalar to normalize the boundary scaling factor to generate the sample-level adaptive gradient adjustment factor.

[0008] Step S3: Extract the original alignment entropy of each hidden layer of the neural network model, and sum the weighted smooth alignment entropy with the original alignment entropy to generate the smooth alignment entropy.

[0009] Step S4: Generate the update step size tensor for each hidden layer based on the smooth alignment entropy and the sample-level adaptive gradient adjustment factor, and update the weight parameters of the neural network model based on the update step size tensor.

[0010] Step S5: Calculate the increase ratio of the current loss value after updating the weight parameters relative to the previous loss value. When the increase ratio is greater than the preset mutation threshold, restore the weight parameters to the state before the update, multiply the update step size tensor by the scaling factor, and update the weight parameters again.

[0011] Preferably, step S1 calculates the classification probability vector and classification margin vector of multiple samples, including the following steps: Step S11, calculate the log odds response value of multiple samples in the preset target category, and use the Softmax function to process the log odds response value to generate the classification probability vector; Step S12, determine the first response value corresponding to the true category of each sample in the multiple samples, and the maximum response value in the non-true category; Step S13, calculate the difference between the first response value and the maximum response value to obtain the margin value of each sample, and collect the margin values ​​of each sample to generate the classification margin vector.

[0012] Preferably, step S2 generates a sample-level adaptive gradient adjustment factor, including the following steps: Step S21, obtaining the original gradient tensor of the classification loss of each sample on the output layer of the neural network model; Step S22, calculating the deviation of the boundary margin value from a preset margin threshold; Step S23, using an exponential decay function to process the deviation and generate a boundary scaling factor. The calculation formula is: ,in, This represents the boundary margin value. As a preset margin threshold, The preset attenuation parameter is set; in step S24, the historical variance moving average is limited to the preset upper and lower limit range to obtain the corrected variance scalar, and the square root of the corrected variance scalar is calculated; in step S25, the boundary scaling coefficient is normalized using the square root to obtain the sample-level adaptive gradient adjustment factor for each sample.

[0013] Preferably, the preset margin threshold is 0.35, the preset attenuation parameter is 1.2, the lower limit of the preset upper and lower limit interval is 0.001, and the upper limit of the preset upper and lower limit interval is 10.0.

[0014] Preferably, step S3 extracts the original alignment entropy of each hidden layer of the neural network model, including the following steps: step S31, extracting the weight gradient tensors generated by multiple samples in the current batch at a specific hidden layer; step S32, calculating the cosine similarity between the weight gradient tensors of any two samples and constructing a similarity matrix; step S33, calculating the eigenvalue distribution of the similarity matrix and calculating the normalized information entropy based on the eigenvalue distribution; step S34, determining the normalized information entropy as the original alignment entropy of the specific hidden layer.

[0015] Preferably, step S3 involves weighted summation of the preceding smooth alignment entropy and the original alignment entropy to generate a smooth alignment entropy, including the following steps: Step S35, weighted summation of the preceding smooth alignment entropy and the original alignment entropy according to a preset smoothing weighting factor to obtain the smooth alignment entropy, wherein the preset smoothing weighting factor is 0.8.

[0016] Preferably, step S4 generates the update step size tensor for each hidden layer, including the following steps: Step S41, calculate the product of the preset baseline learning rate and the logarithmic gain mapping relationship of the smooth alignment entropy to obtain the basic update step size of the layer; Step S42, calculate the exponentially decaying layer weights according to the depth index of each hidden layer and the total number of layers in the network; Step S43, calculate the mean of the sample-level adaptive gradient adjustment factor for each sample; Step S44, multiply the mean, the basic update step size of the layer, and the exponentially decaying layer weights to generate the update step size tensor for each hidden layer.

[0017] Preferably, the preset baseline learning rate is 0.001 and the total number of network layers is 6.

[0018] Preferably, step S5 includes the following steps: Step S51, calculate the increase in the current loss value after updating the weight parameters relative to the previous loss value, and determine the ratio of the increase to the previous loss value as the increase ratio; Step S52, when the increase ratio is greater than the preset mutation threshold of 0.20, discard the modification to the weight parameters and restore the state before the update; Step S53, multiply the update step size tensor by a scaling factor of 0.5, and update the weight parameters again based on the multiplied update step size tensor.

[0019] Preferably, step S5 further includes the following step: step S54, when the action of restoring to the state before the update is triggered 3 times consecutively, the historical variance moving average is reset to the initial reset baseline 1.0.

[0020] This invention proposes a neural network optimization method for data classification. The beneficial effects of this invention are:

[0021] 1. In the optimization of neural networks for data classification, adjusting the impact of different samples on model weight updates based on the classification boundary margin vector and historical gradient fluctuations can reduce the proportion of gradients of high-margin, easily classified samples during training, allowing difficult samples near the classification boundary to receive more thorough correction. At the same time, limiting the range of the historical variance moving average can prevent abnormal update amplitude and numerical fluctuations caused by excessively small variance. This can reduce the dominant role of the same-direction gradients of a large number of easily classified samples on the training results, prevent the correction effect of a few categories and ambiguous boundary samples from being masked, thereby alleviating the problem of excessive smoothing of the classification decision boundary and improving the neural network's ability to distinguish complex class boundary regions.

[0022] 2. The original alignment entropy is obtained by utilizing the similarity between the weight gradients of different samples in each hidden layer. Combined with the prior smoothing alignment entropy, it is smoothed to reflect the dispersion of the gradient direction of samples in different hidden layers more stably, reducing the index fluctuations caused by random batch changes. On this basis, different update step size tensors are set for each hidden layer according to the network level, so that network layers of different depths can update parameters according to their respective gradient states, reducing the inconsistency between deep and shallow layer updates caused by uniform update step size, making the model training process smoother and improving the convergence efficiency of the model.

[0023] 3. After each weight parameter update, the loss change is detected. When the loss increases abnormally, the weight parameters before the update are restored, and the update step size is reduced before updating again. This can limit the impact of abnormal gradients on model parameters in a timely manner. When abnormal updates occur continuously, the relevant historical states are reset. This can also prevent the abnormal fluctuations in the early stage from affecting subsequent training. This can reduce the risk of drastic gradient fluctuations and gradient explosion caused by high-noise data and sudden outliers, reduce abnormal shifts in model parameters, and maintain good continuity and stability in the parameter optimization process. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart of the adaptive update process for neural network parameters in this invention;

[0026] Figure 2 This is a diagram illustrating loss mutation detection and parameter recovery in this invention. Detailed Implementation

[0027] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0028] A neural network optimization method for data classification includes the following steps:

[0029] Step S1: Obtain the high-dimensional feature data tensor containing multiple samples, the historical variance moving average, the preceding smooth alignment entropy, and the preceding loss value. Input the high-dimensional feature data tensor into the neural network model and calculate the classification probability vector and classification boundary margin vector of multiple samples.

[0030] Step S2: Calculate the boundary scaling factor based on the classification boundary margin vector and the historical variance moving average, limit the historical variance moving average within the preset upper and lower limit range to obtain the corrected variance scalar, and use the corrected variance scalar to normalize the boundary scaling factor to generate the sample-level adaptive gradient adjustment factor.

[0031] Step S3: Extract the original alignment entropy of each hidden layer of the neural network model, and sum the weighted smooth alignment entropy with the original alignment entropy to generate the smooth alignment entropy.

[0032] Step S4: Generate the update step size tensor for each hidden layer based on the smooth alignment entropy and the sample-level adaptive gradient adjustment factor, and update the weight parameters of the neural network model based on the update step size tensor.

[0033] Step S5: Calculate the increase ratio of the current loss value after updating the weight parameters relative to the previous loss value. When the increase ratio is greater than the preset mutation threshold, restore the weight parameters to the state before the update, multiply the update step size tensor by the scaling factor, and update the weight parameters again.

[0034] Preferably, step S1 calculates the classification probability vector and classification margin vector of multiple samples, including the following steps: Step S11, calculate the log odds response value of multiple samples in the preset target category, and use the Softmax function to process the log odds response value to generate the classification probability vector; Step S12, determine the first response value corresponding to the true category of each sample in the multiple samples, and the maximum response value in the non-true category; Step S13, calculate the difference between the first response value and the maximum response value to obtain the margin value of each sample, and collect the margin values ​​of each sample to generate the classification margin vector.

[0035] Preferably, step S2 generates a sample-level adaptive gradient adjustment factor, including the following steps: Step S21, obtaining the original gradient tensor of the classification loss of each sample on the output layer of the neural network model; Step S22, calculating the deviation of the boundary margin value from a preset margin threshold; Step S23, using an exponential decay function to process the deviation and generate a boundary scaling factor. The calculation formula is: ,in, This represents the boundary margin value. As a preset margin threshold, The preset attenuation parameter is set; in step S24, the historical variance moving average is limited to the preset upper and lower limit range to obtain the corrected variance scalar, and the square root of the corrected variance scalar is calculated; in step S25, the boundary scaling coefficient is normalized using the square root to obtain the sample-level adaptive gradient adjustment factor for each sample.

[0036] Preferably, the preset margin threshold is 0.35, the preset attenuation parameter is 1.2, the lower limit of the preset upper and lower limit interval is 0.001, and the upper limit of the preset upper and lower limit interval is 10.0.

[0037] Preferably, step S3 extracts the original alignment entropy of each hidden layer of the neural network model, including the following steps: step S31, extracting the weight gradient tensors generated by multiple samples in the current batch at a specific hidden layer; step S32, calculating the cosine similarity between the weight gradient tensors of any two samples and constructing a similarity matrix; step S33, calculating the eigenvalue distribution of the similarity matrix and calculating the normalized information entropy based on the eigenvalue distribution; step S34, determining the normalized information entropy as the original alignment entropy of the specific hidden layer.

[0038] Preferably, step S3 involves weighted summation of the preceding smooth alignment entropy and the original alignment entropy to generate a smooth alignment entropy, including the following steps: Step S35, weighted summation of the preceding smooth alignment entropy and the original alignment entropy according to a preset smoothing weighting factor to obtain the smooth alignment entropy, wherein the preset smoothing weighting factor is 0.8.

[0039] Preferably, step S4 generates the update step size tensor for each hidden layer, including the following steps: Step S41, calculate the product of the preset baseline learning rate and the logarithmic gain mapping relationship of the smooth alignment entropy to obtain the basic update step size of the layer; Step S42, calculate the exponentially decaying layer weights according to the depth index of each hidden layer and the total number of layers in the network; Step S43, calculate the mean of the sample-level adaptive gradient adjustment factor for each sample; Step S44, multiply the mean, the basic update step size of the layer, and the exponentially decaying layer weights to generate the update step size tensor for each hidden layer.

[0040] Preferably, the preset baseline learning rate is 0.001 and the total number of network layers is 6.

[0041] Preferably, step S5 includes the following steps: Step S51, calculate the increase in the current loss value after updating the weight parameters relative to the previous loss value, and determine the ratio of the increase to the previous loss value as the increase ratio; Step S52, when the increase ratio is greater than the preset mutation threshold of 0.20, discard the modification to the weight parameters and restore the state before the update; Step S53, multiply the update step size tensor by a scaling factor of 0.5, and update the weight parameters again based on the multiplied update step size tensor.

[0042] Preferably, step S5 further includes the following step: step S54, when the action of restoring to the state before the update is triggered 3 times consecutively, the historical variance moving average is reset to the initial reset baseline 1.0.

[0043] Example 1: The current input batch size is And the feature dimension is High-dimensional feature data tensors The total number of network layers is And the number of target categories is The neural network model is used for big data classification. The network output layer is designed for the first... Each sample generates a log-odds response vector. Using the Softmax activation function based on the relation Calculate the classification probability vector ,in, Represents an exponential function. Indicates sample Belongs to the Predicted probabilities for each category; simultaneously determine the true category. Corresponding first response value Maximum response value in non-real categories Calculate the difference between the two to obtain the boundary margin value. The classification boundary margin vector is composed of the boundary margin values ​​of each sample in the current batch.

[0044] In step S2, the scaling of the sample gradient is calculated based on the classification boundary margin vector and the historical variance moving average. With preset margin threshold deviation Using preset attenuation parameters The boundary scaling factor is calculated using the exponential decay function: Simultaneously, update the historical variance moving average based on the moving average relationship. : ,in, Indicates the first In the next iteration, the original gradient magnitude variance of the current batch, and the initial reset benchmark. Using the lower limit and upper limit By limiting the historical variance moving average to a preset upper and lower limit range, the corrected variance scalar is obtained. Calculate its square root, and use this square root with the zero-prevention term. Normalize the boundary scaling factors to generate sample-level adaptive gradient adjustment factors: When calculating the variance of the original gradient magnitude of the current batch, first calculate the original gradient tensor of the classification loss for each sample in the current batch relative to the log-odds response value of the output layer of the neural network model, and then obtain the two-dimensional norm magnitude of this original gradient tensor; then calculate the variance of the original gradient magnitude of the current batch. Calculate the arithmetic mean of the gradient modulus values ​​corresponding to each sample, calculate the discrete variance of each gradient modulus value relative to the arithmetic mean, and determine the variance obtained as the original gradient modulus variance of the current batch.

[0045] For depth indexing in neural network models The Hidden layer: Extracts the weight gradient tensor generated by each sample in the current batch. ,in Indicates the first The total number of parameters in the hidden layer, and the calculation of samples within the batch. With sample In the Cosine similarity of gradient directions in hidden layers: ,in, Represents the vector dot product. express Norm; constructing a similarity matrix from the cosine similarity between samples. For the similarity matrix Perform eigenvalue decomposition to obtain nonnegative eigenvalues Calculate the eigenvalue probability distribution using the following formula. : Then calculate the normalized information entropy and determine it as the original alignment entropy: Then, a preset smoothing weighting factor was used. The smooth alignment entropy is generated by weighted summing the previous smooth alignment entropy and the current original alignment entropy according to the following formula. : , where the initial smooth alignment entropy .

[0046] Combined with a preset baseline learning rate With smooth alignment entropy Logarithmic gain mapping relationship, calculate hierarchical basic update step size: Indexed by hidden layer depth With the total number of network layers Calculate the weights of the exponentially decaying hierarchy: Calculate the mean of the sample-level adaptive gradient adjustment factor for each sample in the current batch: Update the step size of the mean and the hierarchy. With exponential decay level weights Multiply to generate the first product. Hidden layer update step size tensor: In this embodiment, Using a single-element tensor form, the element values ​​are determined by the above product; in updating the... Hidden layer weight parameters At that time, the element value is broadcast according to the corresponding element of the weight parameter tensor.

[0047] When generating the weighted sample gradient, the sample-level adaptive gradient adjustment factor corresponding to each sample is multiplied element-wise with the original weight gradient tensor generated by that sample in the corresponding hidden layer to obtain the weighted sample gradient tensor corresponding to that sample; then the current batch is... The weighted sample gradient tensors corresponding to each sample are summed and their arithmetic averages are taken to form the batch-weighted gradient sum. When the sample-level adaptive gradient adjustment factor corresponding to difficult boundary samples is large, the proportion of its gradient in the batch-weighted gradient increases accordingly; the gradient contribution of easily classified samples decreases as the sample-level adaptive gradient adjustment factor decreases, based on the updated step size tensor. And the weighted gradient update of this batch Hidden layer weight parameters .

[0048] After updating the weight parameters, calculate the current batch classification cross-entropy loss value. Compared to the previous batch loss value The increase rate: When the proportion increases Greater than the preset mutation threshold If necessary, discard the current weight parameter modification, restore the weight parameters to their state before the update, and then update the step size tensor. Multiply by scaling factor And update the weight parameters again based on the scaled update step tensor; when continuous When the action to restore to the state before the update is triggered, the historical variance moving average will be used. Reset to initial reset reference And reset the historical state of the time-domain smoothing operator.

[0049] Example 2: In the case of containing In a dataset of high-dimensional samples, the batch size will be... Feature Dimension High-dimensional feature data tensors Input neural network model, the total number of layers in the model Number of target categories The network output layer is for the first Each sample generates a log-odds response vector. The classification probability vector is calculated using the Softmax activation function according to the following formula. : ,in, Represents an exponential function. Indicates sample Belongs to the Predicted probabilities for each category; simultaneously determine the true category. Corresponding first response value and the maximum response value in the non-real category. The difference between the two is used to obtain the boundary margin value: The classification boundary margin vector is formed by the boundary margin values ​​of each sample in the current batch. To simulate noise interference in the working condition, the signal-to-noise ratio is superimposed on the high-dimensional feature data tensor. Gaussian white noise and frequency Harmonic disturbances, preset margin threshold The corrected weights are used to balance the difficulty of boundary samples with the gradient amplification of noisy outliers. This is taken when the boundary samples are densely distributed. This causes the margin to deviate from the amount Maintain positive exponential response; preset decay parameters Used to control boundary scaling factor The decay rate is taken for the current feature dimension. Preset smoothing weighting factor To balance the high-frequency entropy jitter introduced by small-batch sampling with the dynamic tracking capability of gradient alignment distribution, when the sampling batch size is... Time to take ,according to Generate a smoothed aligned entropy that preserves distribution drift information between adjacent iterations while maintaining filter stability. The historical variance moving average is calculated according to... Update, in which, Indicates the first In the next iteration, the original gradient magnitude variance of the current batch, and the initial reset benchmark. ; Using the lower limit and upper limit The corrected variance scalar is obtained by limiting the historical variance moving average to a preset upper and lower limit range. Then, generate the sample-level adaptive gradient adjustment factor according to the following formula: Among them, the prevention of zero minor items Preset baseline learning rate Used to determine the hierarchical base update step size according to the following formula: The embodiment used In a layered neural network model, the input layer dimension is , No. Hidden layer to the first Hidden layers all use a number of channels. A fully connected layer, followed by a modified linear unit activation function and a dropout rate. The regularization layer; the first Hidden layers use dimensions The fully connected bottleneck layer, the first Layer is The output classification head contains dimensionality, and the test dataset includes... A high-dimensional feature sample, among which One sample was used as a training sample. Each sample is used as an independent test sample, with a fixed random seed. Data partitioning and parameter initialization were performed, and the classification accuracy and convergence loss were statistically analyzed according to the same measurement procedures during the testing phase.

[0050] The experiment consisted of one sample group based on this invention and seven control groups. The sample group of this invention employed sample-level boundary margin scaling, hidden layer alignment entropy step size adjustment, and mutation detection reset. The parameters were set as follows: , , , The initial reset benchmark for the historical variance moving average Control group 1 used the conventional Adam optimizer, a globally fixed learning rate of 0.001, and standard cross-entropy loss; control group 2 removed the sample-level adaptive gradient adjustment factor. Control group 3 sets the sample gradient weights to a constant 1; control group 4 removes the step size adjustment corresponding to the hidden layer gradient direction alignment entropy, so that each hidden layer updates the step size using a uniform learning rate; control group 5 removes the loss mutation detection and hedging rollback mechanism, and does not adjust the step size when gradient shocks occur; control group 5 sets the preset margin threshold to... Control group 6 was set as Control group 7 set the preset smoothing weighting factor to Each group uses the same initial weights and performs 100 rounds of training iterations, recording the first response value. Maximum response value for non-realistic categories Boundary margin values Sample-level adaptive gradient adjustment factor 5th hidden layer smooth alignment entropy The fifth hidden layer updates the step size tensor. Increase the proportion The training loss fluctuation range, the classification accuracy of the boundary region, and the convergence loss value after 100 rounds.

[0051] Take input samples Its original response vector is ,according to The calculated predicted probability is 0.743; the first response value... Maximum response value for non-real categories ,Depend on The boundary margin value is 2.1, and the deviation is... With preset attenuation parameters Substitution Obtain the boundary scaling factor At this point, the historical variance moving average Within the preset upper and lower limit range of 0.001 to 10.0, the input is substituted after the range is limited. The sample-level adaptive gradient adjustment factor is obtained. .

[0052] Continue extracting the eigenvalue distribution of the 5th hidden layer similarity matrix to calculate the original alignment entropy. Preorder smooth alignment entropy Preset smoothing weighting factor ,according to The smooth alignment entropy is 0.61, obtained from... Obtain the basic update step size of the 5th hidden layer, and then... Obtain the exponential decay level weights; combine with the batch average adjustment factor. Calculate the step size tensor of the 5th hidden layer. The weight parameters of the 5th hidden layer are updated based on the update step tensor.

[0053] In the 45th iteration, the high-noise impulses increased the current loss value from 0.32 to 0.42. The resulting increase ratio is 0.3125, which is greater than the preset mutation threshold of 0.20. At this point, the weight parameters are restored to their original state, and the update step size tensor is multiplied by a scaling factor of 0.5 to obtain... Then, the weight parameters are updated again using the scaled update step tensor. After 100 rounds of training, the sample group of this invention... , At that time, the batch average adjustment factor was 0.6125, the element value of the step size tensor of the 5th hidden layer was 0.000765, the loss fluctuation amplitude in the later stage of training was 0.012, the classification accuracy of the boundary region was 94.6%, and the convergence loss value after 100 rounds was 0.182.

[0054] In the sample-level gradient adjustment section, control group 2 did not use the preset margin threshold, retained the preset smoothing weighting factor of 0.8, and fixed the sample gradient weights to 1. Its batch average adjustment factor was 1.0000, the element value of the step size tensor in the 5th hidden layer was 0.001250, the loss fluctuation in the later training period was 0.062, the classification accuracy in the boundary region was 85.3%, and the convergence loss after 100 rounds was 0.348. Control group 5 reduced the preset margin threshold to 0.05 while maintaining the preset smoothing weighting factor of 0.8. The resulting batch average adjustment factor was 0.9150, the element value of the step size tensor in the 5th hidden layer was 0.001142, the loss fluctuation in the later training period was 0.054, and the boundary region classification accuracy was 85.3%. The boundary region classification accuracy was 86.2%, and the convergence loss after 100 rounds was 0.320. In control group 6, the preset margin threshold was increased to 0.85 while maintaining the preset smoothing weighting factor of 0.8. The batch average adjustment factor was reduced to 0.2210, the element value of the tensor of the 5th hidden layer update step size was 0.000276, the loss fluctuation amplitude in the later stage of training was 0.021, the boundary region classification accuracy was 84.5%, and the convergence loss after 100 rounds was 0.382. As can be seen from the above data, after removing the sample-level adaptive gradient adjustment factor, the cumulative gradient of easily classified samples once again occupies a large proportion. When the preset margin threshold deviates from 0.35, the batch average adjustment factor and the 5th hidden layer update step size change accordingly.

[0055] In the hidden layer update step size adjustment part, control group 3 retains the preset margin threshold of 0.35 but does not set the preset smoothing weighting factor. Its batch average adjustment factor is 0.6125, the update step size of the 5th hidden layer is 0.000612, the loss fluctuation amplitude in the later stage of training is 0.048, the classification accuracy of the boundary region is 87.1%, and the convergence loss value after 100 rounds is 0.295. Control group 7 retains the preset margin threshold of 0.35 and reduces the preset smoothing weighting factor to 0.1. The batch average adjustment factor is still 0.6125, the update step size of the 5th hidden layer is 0.000910, the loss fluctuation amplitude in the later stage of training increases to 0.078, the classification accuracy of the boundary region is 89.1%, and the convergence loss value after 100 rounds is 0.245. After control group 3 adopts a uniform step size, the update amplitude between the shallow feature extractor and the deep classification head is no longer adjusted with the smoothing alignment entropy. After control group 7 reduces the preset smoothing weighting factor, the entropy value change caused by small batch sampling is more included in the calculation of subsequent update step sizes.

[0056] In the mutation detection and recovery section, control group 4 maintained a preset margin threshold of 0.35, a preset smoothing weighting factor of 0.8, and a batch average adjustment factor of 0.6125. The update step size of the 5th hidden layer was 0.000765, but rollback and step size scaling after loss mutation detection were not performed. Its loss fluctuation amplitude in the later stage of training was 0.115, the classification accuracy of the boundary region was 88.4%, and the convergence loss value after 100 rounds was 0.276. As a reference for routine optimization, control group 1 did not use the preset margin threshold and preset smoothing weighting factor, the batch average adjustment factor was 1.0000, the update step size of the 5th hidden layer was 0.001000, the loss fluctuation amplitude in the later stage of training was 0.085, the classification accuracy of the boundary region was 81.2%, and the convergence loss value after 100 rounds was 0.415.

[0057] In the above experiment, the sample-level adaptive gradient adjustment factor changed the proportion of samples in the batch gradient corresponding to different boundary margin values. The smooth alignment entropy and the exponentially decaying layer weights worked together on the update step size tensor of each hidden layer. When the increase ratio exceeded the preset mutation threshold, the weight parameters before the update were restored and the update step size tensor was scaled before being updated again. After each control group changed the above processing steps, the batch average adjustment factor, hidden layer update step size, loss fluctuation amplitude in the later stage of training, classification accuracy in the boundary region, and convergence loss all showed corresponding changes.

[0058] Example 3: After the high-dimensional non-stationary feature data stream is input, the processor will set the sample batch size to be... Feature dimension is High-dimensional feature data tensors The total number of layers transmitted to the network is The number of target categories is The neural network model addresses the decreased certainty in classification decisions caused by imbalanced class distributions and ambiguous boundary samples. It extracts the boundary margin distribution histogram from an offline calibrated test set and statistically analyzes the boundary margin values. The lower quantile in the difficult boundary interval is used to determine the preset margin threshold. ,in, Indicates sample Real category The corresponding first response value, This represents the maximum response value in the non-true category; simultaneously, it is based on the margin deviation. The preset attenuation parameter is obtained by calculating the slope of the suppression response to the gradient of easily classifiable samples with high margin. The boundary scaling factor is generated using the exponential decay relationship. Establish the correspondence between sample classification determinism and gradient scaling magnitude.

[0059] To address the historical gradient fluctuations caused by sampling pulses and sudden noise, the processor in the first... In the next training iteration, according to Update the historical variance moving average ,in The original gradient magnitude variance of the current batch, the initial reset benchmark. Using the lower limit and upper limit By limiting the historical variance moving average to a preset upper and lower limit range, the corrected variance scalar is obtained. Take the square root of the corrected variance scalar, and combine it with the zero-preservation term. ,according to Calculate the sample-level adaptive gradient adjustment factor This adjusts the gradient contribution of samples corresponding to different boundary margin values ​​by scaling the adjustment factor.

[0060] For the hierarchical structure of neural network models, according to The network layers are numbered, where the first... Layer to the first The first layer is the hidden layer that participates in the calculation of the original alignment entropy and the update step size. The layer is the output classification header, targeting the first... Hidden layer ( Extract the weight gradient tensor generated for each sample. ,in, For the first Total number of parameters in the hidden layer; Calculate the cosine similarity between samples: And construct a similarity matrix ,in Represents the vector dot product. express Norm, when used to decompose the similarity matrix into eigenvalues, normalizes the resulting eigenvalues ​​into a probability distribution: Then calculate the normalized information entropy to obtain the original alignment entropy. : To reduce the high-frequency changes in entropy caused by random small-batch sampling, a preset smoothing weighting factor is used. The smooth alignment entropy is generated by weighted summation of the preceding smooth alignment entropy and the original alignment entropy according to the following formula. : , where the initial smooth alignment entropy Based on the preset baseline learning rate and smooth alignment entropy Calculate the basic update step size of the hierarchy: Combined with hidden layer depth index With the total number of network layers Calculate the weights of the exponentially decaying hierarchy: For the current batch Calculate the mean of the sample-level adaptive gradient adjustment factor for each sample: The mean Hierarchical base update step size With exponential decay level weights Multiply to determine the first... Hidden layer update step size tensor: ,in, The step size value corresponding to the hidden layer update step size tensor is used to update the hidden layer weight parameter tensor.

[0061] When generating the weighted sample gradient, the original sample weight gradient tensor generated by a specific sample in a designated hidden layer is scaled with the corresponding sample-level adaptive gradient adjustment factor by multiplying the tensor element-wise as a scalar. The resulting tensor is used as the weighted sample gradient of that sample in the corresponding hidden layer. The weighted sample gradient retains the original gradient direction and adjusts the gradient magnitude according to the sample-level adaptive gradient adjustment factor. Then, it is updated according to the following formula: Hidden layer weight parameters : After updating the weight parameters, the processor calculates the total classification cross-entropy loss for the current batch. Compared to the total loss of the previous batch The increase rate: when At that time, discard the weight parameters of the current batch. Modify and restore to the state before the update, and adjust the update step tensor to Then update the weight parameters again; when in continuous In each training batch When the action to restore to the state before the update is triggered, the historical variance moving average will be used. Force reset to initial reset reference And reset the historical state of the time-domain smoothing operator to Then, gradient adjustment and update step size calculation are performed again from the reset historical variance moving average and smooth alignment entropy.

[0062] Example 4: This example combines Figures 1 to 2 The paper explains the neural network optimization methods used for data classification, such as... Figure 1As shown, step S1 obtains a high-dimensional feature data tensor containing multiple samples and prior parameters, inputs it into the neural network model, and calculates the classification probability vector and classification boundary margin vector of multiple samples. Step S2 calculates the boundary scaling factor based on the classification boundary margin vector and the historical variance moving average, limits the correction variance scalar and normalizes the scaling factor to generate a sample-level adaptive gradient adjustment factor. Step S3 extracts the original alignment entropy of each hidden layer of the neural network model, and weights and sums the prior smooth alignment entropy and the original alignment entropy to generate the smooth alignment entropy. Step S4 generates the update step size tensor of each hidden layer based on the smooth alignment entropy and the sample-level adaptive gradient adjustment factor, and updates the weight parameters of the neural network model based on the update step size tensor. Step S5 calculates the increase ratio of the current loss value relative to the previous loss value. When it is greater than the preset mutation threshold, the weight parameters are restored to the state before the update, the update step size is scaled and the weight parameters are updated again.

[0063] like Figure 2 As shown, when the system is in the preceding state, a high-dimensional feature data tensor is input to update the weight parameters of the neural network model. After each weight parameter update, the increase in classification loss is detected, and gradient adjustment and step size calculation are performed to return to the preceding state. Alternatively, if the weight loss exceeds a preset mutation threshold, the system can trigger the restoration of the weight parameters to the state before the update. Then, the step size tensor is multiplied by a scaling factor and the weight parameters are updated again. When the action of continuously triggering the restoration to the state before the update occurs, the system is reset to the initial reset baseline. After resetting the historical state of the temporal smoothing operator, the system returns to the preceding state.

[0064] Example 5: Before deploying the neural network model on a specific high-dimensional data classification computing node, the processor first extracts the batch size from the data stream to be classified. Feature Dimension Pre-calibration sample tensor And transmit it to the total number of network layers. The neural network model, under the condition of idle iteration without updating the model weight parameters, extracts the target class number of each hidden layer. Log-odds response vector Calculate the variance of the original gradient magnitude of the preceding batch. This variance characterizes the original gradient fluctuation level corresponding to the previous calibration sample; the initial reset benchmark is the historical variance moving average. Fixed as Simultaneously, the initial information entropy corresponding to the normalized probability distribution of the eigenvalues ​​of the similarity matrix of each hidden layer is calculated, and this is used as the initial state of the smooth alignment entropy of each hidden layer. In the cold start phase of the first training iteration, after receiving the first batch of high-dimensional feature data tensors, the system first performs a forward propagation to obtain the classification cross-entropy loss value of the first batch of samples, and uses this value as the initial benchmark of the preceding loss value. In the first batch of backpropagation, after extracting the original alignment entropy of each hidden layer, the original alignment entropy calculated in the first batch is used as the initial value of the preceding smooth alignment entropy and input into the temporal smoothing operator, thereby completing the initialization of each preceding state variable in the cold start phase.

[0065] Regarding the preset mutation threshold of 0.20 and the reset trigger condition in step S5, a signal-to-noise ratio of 0.20 is applied to the test sample set during the offline calibration phase. to The noise impulse sequence, recording continuous The percentage increase per training batch: When the proportion increases In to Within the interval, the model maintains a normal stochastic gradient descent trajectory; when noise impulses increase the proportion Exceed At that time, the system will update the step tensor. Multiply by scaling factor Then restore the weight parameters to their original state, and then update the weight parameters again based on the scaled update step tensor. This process continues until the weight parameters are fully updated. When the action to restore to the state before the update is triggered, the historical variance moving average will be used. Reset to initial reset reference At the same time, smooth alignment of entropy history states will be achieved. Reset to Then, gradient adjustment and weight parameter updates continue from the reset state.

[0066] Example 6: During hyperparameter calibration and adjustment, the processor calls the initialization test module to extract the offline gradient distribution features of the samples. The processor calls the initialization test module to extract the offline gradient distribution features of the samples, setting the sample batch size to... Feature dimension is calibration data tensor The total number of layers in the input network is The number of target categories is The neural network model calculates the output log-odds response vector. With classification probability vector Calculate the true category response value Maximum response value of non-real categories The difference is used to determine the sample classification boundary margin: Margin margin values ​​for all samples within the batch Sort in ascending order, and take the [number]th [item]. quantiles Determine the preset margin threshold According to the deviation The logarithmic slope of the rate of change of the original gradient magnitude of the output layer with respect to the classification cross-entropy loss is used to determine the preset decay parameter. The boundary scaling factor can be obtained using the following formula. : Meanwhile, during the warm-up training phase, continuous calculations were performed. The original gradient magnitude variance of each batch Determine the variance distribution in Boundary extreme values ​​under confidence intervals and Based on this, a lower limit is set. and upper limit Update the historical variance moving average according to the following formula. : The corrected variance scalar is obtained by limiting it to a preset upper and lower limit interval. Combined with zero-minimum items Generate sample-level adaptive gradient adjustment factors according to the following formula. : When the number of target categories or the scale of data distribution changes, if the number of target categories changes from Expand to If the statistical distribution of the sample classification margins shrinks overall, then the first sample will remain in ascending order. For quantile values, resampling is performed to calculate the corresponding preset margin threshold. If the magnitude of the output layer response value changes, the slope is refitted by calculating the logarithmic rate of change of the classification cross-entropy loss relative to the output layer gradient to obtain a preset decay parameter suitable for that magnitude. For preset upper and lower limit intervals, values ​​below the lower limit are considered acceptable. At this time, the system gradient is in a state of extremely low fluctuation. If the value is too small, it may cause overflow during division normalization, exceeding the upper limit value. At that time, the sample-level adaptive gradient adjustment factor is significantly compressed, therefore in Cut off the upper and lower limits within the confidence interval.

[0067] When determining the hidden layer update step size, the processor extracts the depth index. The The weight gradient tensor generated between samples by the hidden layer Calculate the cosine similarity: And construct a similarity matrix Find the probability distribution of the eigenvalues ​​of the matrix: Then calculate the normalized information entropy to obtain the original alignment entropy: Through continuous In each iteration batch, for Power spectral density analysis was performed on the autocorrelation coefficient to determine the weights corresponding to the autocorrelation decay time constant. Based on this, a preset smoothing weighting factor is set. According to the relational formula: Generate smooth alignment entropy , where the initial smooth alignment entropy Combined with a preset benchmark learning rate Calculate the basic update step size of the hierarchy: Multiply by the exponentially decaying hierarchy weight: Batch average adjustment factor: Generate and update the step size tensor: And update the weight parameters according to the following formula: At the same time, the proportion of data collected during the preceding training phase without abnormalities is increased: The mean and standard deviation, calculated by adding the mean to the standard deviation. The statistical criteria for multiple standard deviations are derived as follows Based on this, the preset mutation threshold is determined to be... .

[0068] When an increase ratio is detected When the value is greater than 0.20, the processor discards the modifications made to the weight parameter θl in the current batch and restores it to the state before the update, adjusting the update step size tensor to... The weight parameters are then updated again. If the action to restore the state before the update is triggered three times in three consecutive training batches, the processor determines that the system has encountered high-intensity noise or data distribution shift, triggering the state reset process and resetting the historical variance moving average. Reset to initial reset reference Simultaneously, the historical state of the smooth alignment entropy is reset to the initial smooth alignment entropy. In the actual reset process, to avoid a sudden drop in the denominator scaling term when the historical variance moving average is reset from a high value to 1.0, which would cause the update step size of subsequent iterations to increase abnormally, the system starts a step size smoothing transition mechanism while performing a forced variance reset. Specifically, the baseline learning rate of the first iteration after the reset is set to 0.2 times the original baseline learning rate of 0.001, i.e., 0.0002, and then gradually restored to the original baseline learning rate of 0.001 in the subsequent 5 training batches in a linear increment, so as to transition the parameter update magnitude after the state reset.

[0069] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A neural network optimization method for data classification, characterized in that, Includes the following steps: Step S1: Obtain the high-dimensional feature data tensor containing multiple samples, the historical variance moving average, the preceding smooth alignment entropy, and the preceding loss value. Input the high-dimensional feature data tensor into the neural network model and calculate the classification probability vector and classification boundary margin vector of multiple samples. Step S2: Calculate the boundary scaling factor based on the classification boundary margin vector and the historical variance moving average, limit the historical variance moving average within the preset upper and lower limit range to obtain the corrected variance scalar, and use the corrected variance scalar to normalize the boundary scaling factor to generate the sample-level adaptive gradient adjustment factor. Step S3: Extract the original alignment entropy of each hidden layer of the neural network model, and sum the weighted smooth alignment entropy with the original alignment entropy to generate the smooth alignment entropy. Step S4: Generate the update step size tensor for each hidden layer based on the smooth alignment entropy and the sample-level adaptive gradient adjustment factor, and update the weight parameters of the neural network model based on the update step size tensor. Step S5: Calculate the increase ratio of the current loss value after updating the weight parameters relative to the previous loss value. When the increase ratio is greater than the preset mutation threshold, restore the weight parameters to the state before the update, multiply the update step size tensor by the scaling factor, and update the weight parameters again.

2. The neural network optimization method for data classification according to claim 1, characterized in that, Step S1 calculates the classification probability vector and classification margin vector of multiple samples, including the following steps: Step S11, calculate the log odds response value of multiple samples in the preset target category, and use the Softmax function to process the log odds response value to generate the classification probability vector; Step S12, determine the first response value corresponding to the true category of each sample in the multiple samples, and the maximum response value in the non-true category; Step S13, calculate the difference between the first response value and the maximum response value to obtain the margin value of each sample, and collect the margin values ​​of each sample to generate the classification margin vector.

3. The neural network optimization method for data classification according to claim 2, characterized in that, Step S2 generates a sample-level adaptive gradient adjustment factor, including the following steps: Step S21, obtaining the original gradient tensor of the classification loss of each sample on the output layer of the neural network model; Step S22, calculating the deviation of the boundary margin value from the preset margin threshold; Step S23, using an exponential decay function to process the deviation and generate a boundary scaling factor. The calculation formula is: ,in, This represents the boundary margin value. As a preset margin threshold, The preset attenuation parameter is set; in step S24, the historical variance moving average is limited to the preset upper and lower limit range to obtain the corrected variance scalar, and the square root of the corrected variance scalar is calculated; in step S25, the boundary scaling coefficient is normalized using the square root to obtain the sample-level adaptive gradient adjustment factor for each sample.

4. The neural network optimization method for data classification according to claim 3, characterized in that, The preset margin threshold is 0.35, the preset attenuation parameter is 1.2, the lower limit of the preset upper and lower limit interval is 0.001, and the upper limit of the preset upper and lower limit interval is 10.

0.

5. The neural network optimization method for data classification according to claim 1, characterized in that, Step S3 extracts the original alignment entropy of each hidden layer of the neural network model, including the following steps: Step S31, extract the weight gradient tensors generated by multiple samples in the current batch in a specific hidden layer; Step S32, calculate the cosine similarity between the weight gradient tensors of any two samples and construct a similarity matrix; Step S33, calculate the eigenvalue distribution of the similarity matrix and calculate the normalized information entropy based on the eigenvalue distribution; Step S34, determine the normalized information entropy as the original alignment entropy of the specific hidden layer.

6. The neural network optimization method for data classification according to claim 5, characterized in that, Step S3 involves weighted summation of the preceding smooth alignment entropy and the original alignment entropy to generate the smooth alignment entropy, including the following steps: Step S35, weighted summation of the preceding smooth alignment entropy and the original alignment entropy according to a preset smoothing weighting factor to obtain the smooth alignment entropy, where the preset smoothing weighting factor is 0.

8.

7. The neural network optimization method for data classification according to claim 3, characterized in that, Step S4 generates the update step size tensor for each hidden layer, including the following steps: Step S41, calculate the product of the preset baseline learning rate and the logarithmic gain mapping relationship of the smooth alignment entropy to obtain the basic update step size of the layer; Step S42, calculate the exponentially decaying layer weights according to the depth index of each hidden layer and the total number of layers in the network; Step S43, calculate the mean of the sample-level adaptive gradient adjustment factor for each sample; Step S44, multiply the mean, the basic update step size of the layer, and the exponentially decaying layer weights to generate the update step size tensor for each hidden layer.

8. The neural network optimization method for data classification according to claim 7, characterized in that, The preset baseline learning rate is 0.001, and the total number of network layers is 6.

9. A neural network optimization method for data classification according to claim 1, characterized in that, Step S5 includes the following steps: Step S51, calculate the increase in the current loss value after updating the weight parameters relative to the previous loss value, and determine the ratio of the increase to the previous loss value as the increase ratio; Step S52, when the increase ratio is greater than the preset mutation threshold of 0.20, discard the modification to the weight parameters and restore the state before the update; Step S53, multiply the update step size tensor by a scaling factor of 0.5, and update the weight parameters again based on the multiplied update step size tensor.

10. A neural network optimization method for data classification according to claim 9, characterized in that, Step S5 also includes the following steps: Step S54, when the action of restoring to the state before the update is triggered 3 times in a row, the historical variance moving average is reset to the initial reset baseline 1.0.

Images