A callback-based deep neural network error localization method

Abstract

The present invention discloses a callback-based deep neural network error location method, which is divided into three steps: model preparation, error detection, and statistical analysis. In the model preparation stage, a reasonable DNN model is selected to meet the problem requirements. In the error detection stage, key parameters are continuously tracked, and parameters such as training loss, verification loss, training accuracy, and verification accuracy are monitored through a callback mechanism. In the statistical analysis stage, methods such as sliding average, rate of change calculation, threshold determination, and visual analysis are used to accurately locate errors in the model and improve the diagnostic accuracy and comprehensiveness of the deep learning model. Error location is performed for underfitting and overfitting problems in the DNN model, which realizes real-time diagnosis and location of deep learning model training problems, improves the comprehensiveness and accuracy of location, and provides more comprehensive support for error diagnosis of deep learning models.

Description

Technical Field

[0001] The present invention belongs to the technical field of DNN error location, and in particular relates to a deep neural network error location method based on callback. Background Art

[0002] In recent years, deep neural networks (DNNs) have been widely used in various fields, and with them comes the need to effectively monitor, diagnose, and solve problems that may arise during the training process. Traditional methods often rely on manual inspection and interpretation of training indicators, which is not only time-consuming, but may also lack precision and real-time performance.

[0003] To address these issues, a popular approach based on a callback mechanism has emerged recently, allowing real-time monitoring of the status of model training. Figure 1 The existing technology Deeplocalize[1] framework prepares DNN models to collect dynamic trajectories, using a callback mechanism to pass a special callback method Callback as a parameter during model training. This custom callback function allows developers to capture and save model variables. It also records key values ​​such as the training data of the forward phase, the results after applying the forward function, the results of applying the activation function, the loss value and accuracy, and the update of the weights and biases in the back-propagation phase. During the training process, online statistical analysis is performed on its learning rate, input data, activation function, and loss function, and the status of the program is compared with pre-defined error conditions. Finally, it reports whether the program contains an error and in which layers and stages the error exists.

[0004] Based on Deeplocalize[1], the researchers achieved this goal by designing a special callback function. This function can capture and record key model variables during the forward and backward propagation phases of model training. In addition, they introduced online statistical analysis, which compares the current state of the model with a predefined series of error conditions (such as learning rate problems, input data anomalies, inappropriate activation / loss function selection, etc.). Although this approach has been proven to be effective in some cases, it also has limitations. For example, it may not be able to fully capture all potential problems, especially those that are not easy to detect in the early stages of training. Existing deep neural network error localization methods based on callback mechanisms can capture some problems with learning rate, input data, and activation / loss functions, but their localization capabilities are insufficient for problems such as underfitting and overfitting.

[0005] References[1]Wardat M,Le W,Rajan H.Deeplocalize:Fault localization for deep neural networks[C].2021IEEE / ACM 43rd International Conference onSoftware Engineering(ICSE).IEEE,2021:251-262. Summary of the invention

[0006] Purpose of the invention: In view of the above problems, the present invention proposes a method for locating errors in deep neural network training problems. In particular, during the training process of deep neural networks, a callback mechanism is used to locate underfitting and overfitting, thereby providing developers with a more comprehensive and accurate real-time diagnostic tool for deep learning model training problems.

[0007] Technical solution: To achieve the purpose of the present invention, the technical solution adopted by the present invention is: a callback-based deep neural network error location method, comprising the following steps:

[0008] Step 1: DNN model preparation: Select the DNN error model and build a training data set. When training the model, add a callback function to save the model variables and collect training information.

[0009] Step 2: DNN error detection: During the forward propagation process, the following five indicators are tracked: loss value, accuracy, verification loss value, verification accuracy, number of neurons and number of layers, and the values ​​of each indicator are recorded after each iteration;

[0010] Step 3: DNN statistical analysis: Based on the indicator values ​​obtained in step 2, underfitting and overfitting are located to complete real-time diagnosis of deep learning model training.

[0011] Furthermore, the step 3 detects two types of errors, underfitting and overfitting, of the DNN during training through statistical analysis, and the steps are as follows:

[0012] Step 3-1: Detect DNN underfitting; use statistical tools to obtain and analyze the loss and accuracy of training and validation to identify underfitting phenomena;

[0013] Step 3-2: Detect DNN overfitting; use statistical tools to obtain and analyze the loss and accuracy of training and validation, and use the complexity of the model as a reference factor to evaluate the status and performance of the model through visual indicators;

[0014] Step 3-3: Match the state of the DNN model with the predefined error condition, report the error in the DNN model, and indicate where it occurs.

[0015] Furthermore, the step 3-1 detects DNN underfitting, specifically including:

[0016] Sliding average analysis: Calculate the sliding average of training loss and validation loss within a certain period range to smooth short-term fluctuations and obtain a stable loss trend;

[0017] Change rate calculation: Compare the loss of the current cycle with the loss of the previous cycle and calculate the change rate of the loss. If the change rate is lower than the set value in several cycles, it means that the model has not learned enough information from the data;

[0018] Difference between training and validation accuracy: monitor the difference between training accuracy and validation accuracy. If the difference between training accuracy and 100% is less than the set value but the difference between validation accuracy and training accuracy is greater than the set value, it indicates an underfitting problem.

[0019] Threshold determination: Set thresholds T1 and T2 for the loss change rate and the difference between training and validation accuracy, respectively. If the loss change rate is lower than T1 for several consecutive cycles, or the difference between training and validation accuracy is higher than T2, an underfitting warning is issued.

[0020] Visual analysis: Draw charts of training loss, validation loss, training accuracy, and validation accuracy to observe the changing trends of each indicator over time and assist in the judgment of underfitting.

[0021] Furthermore, the step 3-2 detects DNN overfitting, specifically including:

[0022] Loss and accuracy analysis: Monitor the training loss and validation loss. If the training loss continues to decrease, but the validation loss starts to increase or changes more, it means that the model is overfitting. Compare the training accuracy and validation accuracy. If the training accuracy increases at a rate exceeding the set threshold in p% of the cycle and the difference between the training accuracy and 100% is within the set range, but the validation accuracy does not increase at a rate exceeding the set threshold in p% of the cycle and the difference between the validation accuracy and 100% is not within the set range, and starts to decrease, it means that the model is overfitting.

[0023] Sliding average: Calculate the sliding average of training loss and validation loss within a certain period range, add a new period each time and delete the oldest period, monitor the loss trend of both training loss and validation loss, and determine whether the difference between training loss and validation loss increases. If the increase exceeds the set threshold, it means that the model is overfitting;

[0024] Visual analysis: Draw charts of training loss, validation loss, training accuracy, and validation accuracy to view the changing trends of each indicator over time and determine whether overfitting occurs.

[0025] Beneficial effects: Compared with the prior art, the technical solution of the present invention has the following beneficial technical effects:

[0026] The callback-based deep neural network error localization method proposed in the present invention further improves Deeplocalize, and can additionally accurately locate underfitting and overfitting problems, providing more comprehensive support for error diagnosis of deep learning models. Compared with traditional methods, this method is more detailed and comprehensive, especially in error detection and statistical analysis. Tools including sliding average, rate of change calculation, threshold determination and visual analysis are introduced to improve the stability and performance of the deep neural network model. The method of the present invention not only expands the existing error localization technology, but also provides more comprehensive, accurate and innovative support for research and application in the field of deep learning, which can further improve the reliability and efficiency of the deep neural network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is the existing Deeplocalize framework diagram.

[0028] Figure 2 It is a schematic diagram of the callback-based deep neural network error localization method of the present invention.

[0029] Figure 3 It is an example diagram of an underfitting DNN model.

[0030] Figure 4 It is a visual analysis of the loss and accuracy of the underfitting DNN model detection results.

[0031] Figure 5 It is the correct DNN model 1.

[0032] Figure 6 This is an example diagram of an overfitting DNN model.

[0033] Figure 7 It is a visual analysis of the loss and accuracy of the overfitting DNN model detection results.

[0034] Figure 8 It is the correct DNN model 2. DETAILED DESCRIPTION

[0035] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0036] Figure 1is an existing Deeplocalize framework diagram. The present invention discloses a callback-based deep neural network error localization method, the framework is as follows Figure 2 As shown in the figure, the second part of Deeplocalize is improved to locate underfitting and overfitting. Deeplocalize is divided into two modules: model preparation and dynamic analysis. The error location of Deeplocalize includes:

[0037] (1) DNN model preparation. Filter posts with a score of no less than 5 points on the Stack Overflow website to ensure that the posts obtained involve DNN-related errors. Use "error" and its synonyms as keywords for searching; review the search results and exclude posts that contain only partial or incomplete code; check the content of the post to determine whether complete training data is provided, that is, the training data is not missing, or whether a recognized data set, that is, an official data set, is used; study the answers related to StackOverflow to determine the repair solution for the DNN error model, and organize the DNN model data accordingly; when training the model, add a callback function (Callback) to collect the necessary training information. Deeplocalize's dynamic analysis is divided into two parts: error detection and statistical analysis.

[0038] (2) DNN error detection. DNN training is divided into two stages: forward propagation and back propagation. In the forward propagation stage, the following five key parameters are continuously tracked: input and labeled training data, output after applying the forward function, output after applying the activation function, loss value, and accuracy, and their values ​​are recorded after each iteration; in the back propagation stage, the following three key parameters are continuously monitored: weight, bias, and weight increment obtained by gradient descent for each layer.

[0039] (3) DNN statistical analysis. Deeplocalize's statistical analysis is used to detect three types of errors in DNNs during training. (3-1) Detecting incorrect learning rates. During the backpropagation process, the learning rate has an impact on the update of weights. During the backpropagation process, the learning library iteratively calculates gradient descent. During the training process, the mean and standard deviation of the weights in the correct model are constantly changing. In contrast, the mean and standard deviation of the weights in the incorrect model are constant. If there is a problem with the learning rate, it can be detected from the output of the gradient of the output layer. (3-2) Detecting incorrect input data. During the forward propagation phase, the output before / after the activation function is applied is detected. The mean and standard deviation of the output of each layer are then calculated. The output of the first layer will be checked for numerical errors, such as NaN (Not a Number, NaN) after / before the activation function is applied. (3-3) Detecting incorrect activation / loss functions. After completing the forward propagation phase, the loss and accuracy are calculated. There are two indicators that indicate that there is a problem with the model. First, if one of the two indicators has a numerical error like NaN. Second, the loss starts to increase, or the accuracy starts to decrease after some iterations. (3-4) Compare the state of the DNN model with the defined error conditions. Report whether the DNN model contains an error and at which layers and stages the error exists.

[0040] The callback-based deep neural network error location method described in the present invention specifically includes the following steps:

[0041] Step 1: DNN model preparation.

[0042] Step 1-1: Filter posts with a score of 5 or above in Stack Overflow to ensure that they involve DNN-related errors. Use "error" and its synonyms as keywords for the search.

[0043] Step 1-2: Review the search results and exclude posts that contain only partial or incomplete code.

[0044] Step 1-3: Review the post content to determine whether complete training data is provided or whether a recognized dataset is used.

[0045] Step 1-4: Study the answers related to Stack Overflow to determine the fixes for the DNN model errors and compile the DNN model data accordingly.

[0046] Step 1-5: When the model is training, add a designed callback function to the fit() method to collect necessary training information.

[0047] Step 2: DNN error detection.

[0048] During the forward propagation process, keep track of the following five indicators: loss value, accuracy, validation loss value, validation accuracy, number of neurons and number of layers, and record their values ​​after each iteration.

[0049] Step 3: DNN statistical analysis.

[0050] To detect two types of errors in DNN training, the specific steps are as follows:

[0051] Step 3-1: Detect DNN underfitting. This step focuses on the loss and accuracy of training and validation. Applying statistical tools such as sliding average, rate of change, and threshold determination can more accurately identify underfitting.

[0052] Step 3-1-1: Sliding average analysis. Calculate the sliding average of training loss and validation loss within a certain period range to smooth short-term fluctuations and obtain a stable loss trend.

[0053] Step 3-1-2: Calculate the rate of change. Compare the loss of the current cycle with the loss of the previous cycle and calculate the rate of change of the loss. If the rate of change is continuously low (lower than the set value for several cycles), it means that the model has not learned enough information from the data.

[0054] Step 3-1-3: Difference between training and validation accuracy. Monitor the difference between training accuracy and validation accuracy. If the training accuracy gradually approaches 100% but the difference between validation accuracy and training accuracy is greater than 0.15, it indicates an underfitting problem.

[0055] Step 3-1-4: Threshold determination. Set thresholds for the rate of change of loss and the difference between training and validation accuracy. If the rate of change of loss for several consecutive cycles is lower than the set threshold, or the difference between training and validation accuracy is higher than the set threshold, an underfitting warning is issued.

[0056] Step 3-1-5: Visual analysis. Draw a chart of training loss, validation loss, training accuracy, and validation accuracy. This helps to visually observe their changing trends over time, and thus assists in the judgment of underfitting.

[0057] Step 3-2: Detect DNN overfitting. Analyze the loss and accuracy of training and validation, and consider the complexity of the model as a reference factor. By visualizing these indicators, you can intuitively evaluate the state and performance of the model.

[0058] Step 3-2-1: Loss and accuracy analysis.

[0059] Monitor training loss and validation loss: If training loss continues to decrease, but validation loss starts to increase or varies more, it indicates that the model is overfitting. Compare training accuracy and validation accuracy: If training accuracy increases rapidly in 30% of the cycles and approaches 100%, but validation accuracy does not increase rapidly in 30% of the cycles and approaches 100% and starts to decrease, it also indicates that the model is overfitting.

[0060] Step 3-2-2: Sliding average.

[0061] Calculate the sliding average of the training loss and validation loss within a certain period range (10 periods), add a new period each time and delete the oldest period, monitor the loss trend of both training loss and validation loss, and determine whether the difference between training loss and validation loss increases. If the increase exceeds the set threshold, it means that the model is overfitting.

[0062] Step 3-2-3: Model complexity.

[0063] Evaluate the complexity of the model, such as the number of neuron layers, etc. Overly complex models, i.e. models with more than 8 neuron layers, are more likely to overfit.

[0064] Step 3-2-4: Visual analysis.

[0065] Plot the training loss, validation loss, training accuracy, and validation accuracy to see how they change over time to visually determine if overfitting is occurring.

[0066] Step 3-3: Ensure that the state of the DNN model matches the predefined error conditions, report any errors in the DNN model promptly, and accurately indicate where they occur.

[0067] The present invention is Figure 3 Error localization of the DNN model instance in:

[0068] (1) Moving average analysis: At the end of each cycle, the training loss and validation loss of the current cycle are recorded. Then, based on the defined window size (e.g., the most recent 10 cycles), the average training loss and average validation loss within this window are calculated. By comparing the average training loss with the training loss of the current cycle, the rate of decline of the average loss is calculated.

[0069] (2) Rate of change calculation: Compare the loss of the current cycle with the loss of the previous cycle. Calculate the rate of change of the loss by using the following formula: (previous cycle loss - current cycle loss) / previous cycle loss. If the rate of change is continuously low (e.g., less than or equal to 10%) in the 20% to 80% cycle, an underfitting warning is issued.

[0070] (3) Model complexity analysis (difference between training and validation accuracy): In this step, observe the difference between the training accuracy and the validation accuracy. If the training accuracy gradually approaches 100% but the difference between the validation accuracy and the training accuracy is greater than 0.15, there may be an underfitting problem.

[0071] (4) Threshold determination: In order to more specifically determine underfitting, the given code detects the rate of change and the difference between training and validation accuracy at the end of each cycle. If the rate of decrease of the average loss is less than or equal to 10% in 40% of the cycles from 20% to 80%, an underfitting warning will be issued. If both the training and validation accuracy are less than 50%, an underfitting warning will also be issued.

[0072] (5) Visual analysis: Plot the curves of training loss, validation loss, training accuracy, and validation accuracy to visually observe their changing trends over time.

[0073] Figure 3 The goal of the DNN model is to classify handwritten digits and output the probability distribution of each digit. Because the complexity of the DNN model is not enough, underfitting occurs. The training results of the model are: loss: 0.2624-accuracy: 0.9270-val_loss: 0.2676-val_accuracy: 0.9256.

[0074] Finally, through Figure 4 Visual analysis and DNN model detection results show that the loss rate of the DNN model in cycles 3-10 is less than or equal to 10% for more than 40% of the cycles, indicating that the model begins to have underfitting problems in cycle 3 (because Figure 3 The model complexity is insufficient). Figure 5 The repaired correct model 1 is shown. The training results of the model are: loss: 0.0213-accuracy: 0.9931-val_loss: 0.0855-val_accuracy: 0.9773. Its training loss, validation loss, training accuracy and validation accuracy are all high.

[0075] The present invention is Figure 6 Error localization of the DNN model instance in:

[0076] (1) Loss and accuracy analysis: If the training loss continues to decrease, but the validation loss starts to increase or changes more, it means that the model is overfitting. Compare the training accuracy and validation accuracy: If the training accuracy increases rapidly in 30% of the cycles and approaches 100%, but the validation accuracy does not increase rapidly in 30% of the cycles and approaches 100% and starts to decrease, it also means that the model is overfitting.

[0077] (2) Sliding average analysis: Calculate the sliding average of the training loss and validation loss within a certain period range (10 periods), add a new period and delete the oldest period each time to observe the loss trend of both training loss and validation loss, and determine whether the difference between training loss and validation loss increases.

[0078] (3) Model complexity analysis (difference between training and validation accuracy): If the training accuracy is close to 100%, but the validation accuracy does not keep up and begins to decline, this may indicate that the model has an overfitting problem.

[0079] (4) Visual analysis: Plot the curves of training loss, validation loss, training accuracy, and validation accuracy to visually observe their changing trends over time.

[0080] Figure 6 The main role of the model shown is to be used for multi-classification problems, such as handwritten digit classification (MNIST dataset), which is to classify the input handwritten digit image into 10 possible categories, namely the numbers 0 to 9. Since the model has a large capacity and the training data is small, the model will overfit the training data. The training results of the model are: Epoch 6 / 200, loss: 0.0335-accuracy: 1.0000-val_loss: 0.5090-val_accuracy: 0.8514; Epoch 200 / 200, loss: 1.0596e-05-accuracy: 1.0000-val_loss: 0.7935-val_accuracy: 0.8633.

[0081] Finally, through Figure 7 Visual analysis and DNN model test results show that the DNN model training loss continues to decrease from epoch 5, but the validation loss increases, indicating that the model begins to have overfitting problems at epoch 5 (because Figure 6 The model complexity is too high, there are too many training cycles, and there is not enough training data). Then, Figure 8 The correct model 2 after repair is shown. A Dropout layer is added between the two hidden layers to help reduce overfitting. The training results of the model are: Epoch 20 / 200, loss: 0.0323-accuracy: 0.9960-val_loss: 0.5190-val_accuracy: 0.8693, and its training loss, validation loss, training accuracy, and validation accuracy remain stable.

Claims

1. A callback-based deep neural network error location method, characterized in that: The following steps are involved: Step 1: DNN model preparation; Select the DNN error model and build a training data set. When training the model, add a callback function to save the model variables and collect training information. Step 2: DNN error detection; During the forward propagation process, the following five indicators are tracked: loss value, accuracy, validation loss value, validation accuracy, number of neurons and number of layers, and the values ​​of each indicator are recorded after each iteration; Step 3: DNN statistical analysis: Based on the index values ​​obtained in step 2, underfitting and overfitting are located to complete real-time diagnosis of deep learning model training; Step 3-1: Detect DNN underfitting; use statistical tools to obtain and analyze the loss and accuracy of training and validation, and identify underfitting phenomena, including: Moving average analysis: Calculate the sliding average of training loss and validation loss within a certain period range to smooth short-term fluctuations and obtain a stable loss trend; Rate of change calculation: Compare the loss of the current cycle with the loss of the previous cycle and calculate the rate of change of the loss. If the rate of change is lower than the set value for several cycles, it means that the model has not learned enough information from the data. Difference between training and validation accuracy: Monitor the difference between the training accuracy and the validation accuracy. If the difference between the training accuracy and 100% is less than the set value but the difference between the validation accuracy and the training accuracy is greater than the set value, it indicates an underfitting problem. Threshold determination: Set thresholds T1 and T2 for the rate of change of loss and the difference between training and validation accuracy, respectively. If the rate of change of loss is lower than T1 for several consecutive cycles, or the difference between training and validation accuracy is higher than T2, an underfitting warning is issued. Visual analysis: Draw charts of training loss, validation loss, training accuracy, and validation accuracy to observe the changing trends of each indicator over time and assist in the judgment of underfitting; Step 3-2: Detect DNN overfitting; use statistical tools to obtain and analyze the loss and accuracy of training and validation, and use the complexity of the model as a reference factor to evaluate the state and performance of the model through visual indicators, including: Loss and accuracy analysis: Monitor the training loss and validation loss. If the training loss continues to decrease, but the validation loss starts to increase or changes more, it means that the model is overfitting. Compare the training accuracy and the verification accuracy. If the training accuracy rate exceeds the set threshold in the p% period and the difference between the training accuracy and 100% is within the set range, but the verification accuracy rate does not exceed the set threshold in the p% period and the difference between the verification accuracy and 100% is not within the set range, and begins to decline, it means that the model is overfitting. Sliding Average: Calculate the sliding average of the training loss and validation loss within a certain period range, add a new period each time and delete the oldest period, monitor the loss trend of both training loss and validation loss, and determine whether the difference between the training loss and validation loss increases. If the increase exceeds the set threshold, it means that the model is overfitting; Visual analysis: Draw charts of training loss, validation loss, training accuracy, and validation accuracy to view the changing trends of each metric over time and determine whether overfitting occurs. Step 3-3: Match the state of the DNN model with the predefined error condition, report the error in the DNN model, and indicate where it occurs.

Images