Maximum contribution neuron propagation path determination method based on reverse tracking

Abstract

The invention relates to a maximum contribution neuron propagation path determination method based on reverse tracking, and belongs to the field of computer vision of machine learning. The method specifically comprises the following steps: step 1, calculating the activity degree of each layer of neurons in an electroencephalogram mapping model based on a response value statistics method; 2, finding a neuron and a path which contribute to the greatest degree in a decision-making process from classification output reverse tracking; drawing a key path of the electroencephalogram mapping model; according to the method, neurons and paths which contribute to the maximum in the decision making process are found, and the working principle of the electroencephalogram mapping model is explained by tracking the paths of the neurons which contribute to the maximum.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision of machine learning and relates to a method for determining a maximum contributing neuron propagation path based on reverse tracing. Background Art

[0002] EEG mapping model refers to an artificial intelligence model that can promote the understanding of the image to EEG signal conversion process through the interpretation of the EEG mapping model. At present, the interpretability methods of artificial intelligence models are roughly divided into two technical directions: data-based interpretability methods and model-based interpretability methods.

[0003] The data-based interpretability method is to explain the model by preprocessing the data and visually displaying the data. Machine learning can extract useful knowledge and rules from a large amount of data information. The premise of obtaining a good model is to have a certain understanding of the data to be processed, such as the characteristics of the data, the distribution of the data, the internal connection of the data, etc. Therefore, the interpretability analysis of the model can start from the pre-understanding of the data, fully understand the problem to be dealt with, and reasonably establish the corresponding model to solve it, so as to obtain the optimal solution to the problem. Data visualization is a type of interpretability method that researchers have focused on in recent years. Specifically, it includes many methods such as deconvolution, depooling, and heat maps. These visualization methods give people a direct impression of the internal mechanism of the neural network, but these methods cannot deeply understand the internal structure of the model and it is difficult to directly explain the decision logic of the model.

[0004] Model-based interpretability analysis methods are mainly divided into two categories: proxy models and automatic feature extraction. The proxy model method is to build a new model to simulate the input and output of the black box model, and then use the proxy model to understand the original black box model. A typical example is the linear proxy model based on model-independent local interpretable descriptions (Local Interpretable Model-Agnostic Explanations, LIME) proposed by Ribeiro et al. Automatic feature extraction is to analyze and explain its decision logic by studying the relationship between input, output and internal elements of the model. Specifically, there are the KT method that uses if-then rules for automatic feature extraction for each layer and each neuron, the method of automatic feature extraction using sampling, and the sensitivity analysis method that judges the importance of input variables by changing the connection weights, partial derivatives, and input variables. Model-based interpretability methods generally have problems such as simple structure and limited functions, and building complex interpretable neural networks is a very arduous task. Summary of the invention

[0005] The technical problem solved by the present invention is: to overcome the shortcomings of the prior art, propose a method for determining the propagation path of the most contributing neuron based on reverse tracing, find the neurons and paths that contribute the most in the decision-making process, and explain the working principle of the EEG mapping model by tracing the path of the most contributing neuron.

[0006] The solution of the present invention is:

[0007] A method for determining the maximum contribution neuron propagation path based on reverse tracing, comprising:

[0008] Step 1: Calculate the activity level of each layer of neurons in the EEG mapping model based on the response value statistics method;

[0009] Step 2: Trace back from the classification output to find the neurons and paths that contribute most to the decision-making process; draw the key path of the EEG mapping model.

[0010] In the above-mentioned method for determining the maximum contribution neuron propagation path based on reverse tracing, it is characterized in that: in the step 1, the method for calculating the activity level of each layer of neurons in the EEG mapping model is:

[0011] S11. Perform forward propagation calculation on each image sample separately, and record the response value of each neuron on different samples;

[0012] S12, obtain the output features of the model at each layer, and use the mean response value on each channel of the neural network to represent the strength of the neuron's response;

[0013] S13, summarize the results of individual samples to obtain the average response strength of each neuron in each layer of the batch;

[0014] S14, arranging in descending order according to the average response intensity to obtain a sequence of neurons with activity levels from high to low;

[0015] S15, setting a threshold, taking the neurons ranked in the top few and with average response strength higher than the threshold as active neurons, and outputting a single inference result;

[0016] S16. Repeat S11-S15 with different image samples. After multiple rounds of batch calculations, the average response strength of each neuron on all training samples is accumulated.

[0017] S17. Compare the distribution of active neurons at different levels and observe the evolution trajectory of model feature extraction;

[0018] S18. By analyzing the average response strength of each neuron in different levels of the statistical model to the sample set, key neurons with higher response values ​​are identified, and it is determined that these neurons have higher activity.

[0019] In the above-mentioned method for determining the maximum contributing neuron propagation path based on reverse tracing, in the step 2, the layer-by-layer interpretation method LRP and the gradient class activation map Grad-CAM interpretation method are combined, and the classification output is used to weight the gradient of the intermediate layer feature map to quantitatively calculate the influence of each neuron on the final classification result, that is, the contribution of the neuron.

[0020] In the above-mentioned method for determining the propagation path of the maximum contributing neuron based on back-tracing, the specific method of back-tracing from the classification output to find the neuron and path that contributes the most in the decision-making process is:

[0021] S21, forward propagation of the neural network is performed to obtain the output value of each neuron;

[0022] S22, back propagate from the classification results of the output layer to calculate the contribution of each neuron to the final output;

[0023] S23, trace back to the input layer along the neuron path with the largest contribution, and obtain the interpretation path based on the neuron with the largest contribution;

[0024] S24. Use the reverse maximum contribution neuron path to evaluate the stability of model predictions.

[0025] In the above-mentioned method for determining the maximum contributing neuron propagation path based on reverse tracing, in S21, the neural network consists of an intermediate layer and an output layer, the intermediate layer is connected to the input image sample, and the output layer is connected to the intermediate layer.

[0026] In the above-mentioned method for determining the maximum contribution neuron propagation path based on reverse tracing, in S22, the contribution of each neuron to the final output is calculated as follows:

[0027] Assume that the neural network has L layers, and the output of the i-th neuron in the first layer is

[0028] The contribution of neuron j in the output layer is C j for:

[0029]

[0030] Neurons in the middle layer C i The contribution of the neural network level is defined as

[0031]

[0032] Among them, w ik is the set of child nodes of neuron i;

[0033] w ik is the weight from neuron i to k.

[0034] In the above-mentioned method for determining the maximum contributing neuron propagation path based on reverse tracing, in S23, the depth-first search algorithm DFS is used to always select the maximum N contributing input neurons of the current neuron as the next neuron, and recursively calculate the contribution of each neuron; select the neuron path that contributes Top K, and use this method layer by layer based on the classification result to find the explanation path for the classification decision.

[0035] In the above-mentioned method for determining the maximum contribution neuron propagation path based on reverse tracing, in S24, the specific method of evaluating the stability of the model prediction by using the reverse maximum contribution neuron path is:

[0036] S241, perform multiple forward and reverse reasoning on each sample to obtain a reverse key path;

[0037] S242. Count the reverse critical paths of all samples and find the standard reverse critical paths that appear frequently. The standard reverse critical paths reflect the model's discrimination patterns for different categories.

[0038] S243. For the misclassified samples, analyze the difference between the reverse critical path of the sample and the corresponding standard reverse critical path to find out the blind spots of the model judgment;

[0039] S244. Count the overlap between the reverse critical path of all test samples and the standard reverse critical path as an evaluation indicator of the prediction stability of the model.

[0040] In the above-mentioned method for determining the maximum contribution neuron propagation path based on reverse tracing, in S243, the calculation method of the overlap between the reverse critical path of the test sample and the standard reverse critical path is:

[0041] Set the i-th test sample to x i ; Set x i The reverse critical path is P i , the standard reverse critical path of the same sample is P s ;

[0042] Calculate x i The path difference Diff(x i )for:

[0043]

[0044] In the above-mentioned method for determining the maximum contribution neuron propagation path based on reverse tracing, in S244, the calculation method of the Stability of the overlap between the reverse critical path of the test sample and the standard reverse critical path is:

[0045]

[0046] The overlap degree Stability is the reverse path stability. By analyzing the path difference and stability, the reliability of the diagnosis model judgment can be achieved and its blind spots can be discovered.

[0047] The beneficial effects of the present invention compared with the prior art are:

[0048] (1) The present invention constructs the propagation path of the most contributing neuron based on the contribution of active neurons in the EEG mapping model and the correlation between neurons at different levels, and interprets the EEG mapping model with the help of the logical rules of the propagation path and a clear hierarchical visualization display method;

[0049] (2) The present invention adopts a method of tracing the path of the neuron with the largest contribution from the classification output in reverse. This method combines the layer-by-layer interpretation method (LRP) with the gradient-weighted class activation mapping (Grad-CAM) interpretation method, and uses the classification output to weight the gradient of the intermediate layer feature map to quantitatively calculate the degree of influence of each neuron on the final classification result. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a flow chart for determining the transmission path of the neuron with the greatest contribution according to the present invention.

[0051] Figure 2 This is a graph showing the activity of key neurons in the present invention;

[0052] Figure 3 Schematic diagram of the key path of the EEG mapping model drawn for the present invention. DETAILED DESCRIPTION

[0053] The present invention will be further described below in conjunction with the embodiments.

[0054] The present invention provides a method for determining the propagation path of the most contributing neuron based on reverse tracing, discovers the neurons and paths that contribute the most in the decision-making process, and explains the working principle of the EEG mapping model by tracing the path of the most contributing neuron.

[0055] The method for determining the propagation path of the maximum contributing neuron based on reverse tracing, such as Figure 1 As shown, the specific steps include:

[0056] Step 1: Calculate the activity level of each layer of neurons in the EEG mapping model based on the response value statistics method. In step 1, the method for calculating the activity level of each layer of neurons in the EEG mapping model is:

[0057] S11. Perform forward propagation calculation on each image sample separately, and record the response value of each neuron on different samples.

[0058] S12. Obtain the output features of the model at each layer, and use the mean response value on each channel of the neural network to indicate the strength of the neuron's response.

[0059] S13. Summarize the results of individual samples to obtain the average response strength of each neuron in each layer of the batch.

[0060] S14. Arrange in descending order according to the average response intensity to obtain a sequence of neurons with activity levels from high to low.

[0061] S15. Set a threshold, and take the neurons that rank in the top few and whose average response strength is higher than the threshold as active neurons, and output a single inference result.

[0062] S16. Repeat S11-S15 using different image samples. After multiple rounds of batch calculations, the average response strength of each neuron on all training samples is accumulated.

[0063] S17. Compare the distribution of active neurons at different levels and observe the evolution trajectory of model feature extraction.

[0064] S18. By using the average response strength of each neuron in different levels of the statistical model to the sample set, key neurons with higher response values ​​are identified, and it is determined that these neurons have higher activity, such as Figure 2 shown.

[0065] Step 2: trace back from the classification output to find the neurons and paths that contribute most to the decision-making process; draw the key paths of the EEG mapping model, such as Figure 3 shown.

[0066] In step 2, the layer-by-layer interpretation method LRP and the gradient class activation map Grad-CAM interpretation method are combined, and the classification output is used to weight the gradient of the intermediate layer feature map to quantitatively calculate the influence of each neuron on the final classification result, that is, the contribution of the neuron.

[0067] The specific method of tracing back from the classification output to find the neurons and paths that contribute most to the decision-making process is:

[0068] S21. Perform forward propagation on the neural network to obtain the output value of each neuron.

[0069] In S21, the neural network consists of an intermediate layer and an output layer, the intermediate layer is connected to the input image sample, and the output layer is connected to the intermediate layer.

[0070] S22, back propagating from the classification result of the output layer, calculating the contribution of each neuron to the final output. In S22, the calculation method of the contribution of each neuron to the final output is:

[0071] Assume that the neural network has L layers, and the output of the i-th neuron in the first layer is

[0072] The contribution of neuron j in the output layer is C j for:

[0073]

[0074] Neurons in the middle layer C i The contribution of the neural network level is defined as

[0075]

[0076] Among them, w ik is the set of child nodes of neuron i;

[0077] w ik is the weight from neuron i to k.

[0078] S23. Trace back to the input layer along the neuron path with the largest contribution to obtain the interpretation path based on the neuron with the largest contribution.

[0079] In S23, the depth-first search algorithm DFS is used to always select the maximum N contributing input neurons of the current neuron as the next neuron, and recursively calculate the contribution of each neuron; select the neuron path that contributes Top K, and use this method layer by layer based on the classification result to find the explanation path for the classification decision.

[0080] S24. Use the reverse maximum contribution neuron path to evaluate the stability of model predictions.

[0081] In S24, the specific method of evaluating the stability of the model prediction using the reverse maximum contribution neuron path is:

[0082] S241. Perform multiple forward and reverse reasoning on each sample to obtain the reverse critical path.

[0083] S242. Count the reverse critical paths of all samples and find the standard reverse critical paths that appear frequently. The standard reverse critical paths reflect the model's discrimination patterns for different categories.

[0084] S243. For the misclassified samples, analyze the difference between the reverse critical path of the sample and the corresponding standard reverse critical path to find out the blind spots of the model judgment.

[0085] In S243, the calculation method of the overlap between the reverse critical path of the test sample and the reverse critical path of the standard is:

[0086] Set the i-th test sample to x i ; Set x i The reverse critical path is P i , the standard reverse critical path of the same sample is P s ;

[0087] Calculate x i The path difference Diff(x i )for:

[0088]

[0089] S244. Count the overlap between the reverse critical path of all test samples and the standard reverse critical path as an evaluation indicator of the prediction stability of the model.

[0090] In S244, the calculation method of the Stability of the overlap between the reverse critical path of the test sample and the standard reverse critical path is:

[0091]

[0092] The classification accuracy is defined as:

[0093]

[0094] The overlap degree Stability is the reverse path stability. By analyzing the path difference and stability, the reliability of the diagnosis model judgment can be achieved and its blind spots can be discovered.

[0095] The present invention constructs the propagation path of the most contributing neuron according to the contribution of active neurons in the EEG mapping model and the correlation between neurons at different levels, and interprets the EEG mapping model with the help of the logical rules of the propagation path and a clear hierarchical visualization display method.

[0096] The present invention adopts a method of tracing the path of the maximum contributing neuron backwards from the classification output. This method combines the layer-by-layer interpretation method (LRP) with the interpretation method based on the gradient class activation map (Gradient-weighted Class Activation Mapping, Grad-CAM), and uses the classification output to weight the gradient of the intermediate layer feature map to quantitatively calculate the influence of each neuron on the final classification result.

[0097] Although the present invention has been disclosed as above in the form of a preferred embodiment, it is not intended to limit the present invention. Any person skilled in the art may make possible changes and modifications to the technical solution of the present invention by using the methods and technical contents disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope of the technical solution of the present invention.

Claims

1. A method for determining the maximum contribution neuron propagation path based on reverse tracing, characterized in that: include: Step 1: Calculate the activity level of each layer of neurons in the EEG mapping model based on the response value statistics method; Step 2: trace back from the classification output to find the neurons and paths that contribute most to the decision-making process; Draw out the critical path of the EEG mapping model.

2. The method for determining the maximum contributing neuron propagation path based on reverse tracing according to claim 1, characterized in that: In step 1, the method for calculating the activity level of neurons in each layer in the EEG mapping model is: S11. Perform forward propagation calculation on each image sample separately, and record the response value of each neuron on different samples; S12, obtain the output features of the model at each layer, and use the mean response value on each channel of the neural network to represent the strength of the neuron's response; S13, summarize the results of individual samples to obtain the average response strength of each neuron in each layer of the batch; S14, arranging in descending order according to the average response intensity to obtain a sequence of neurons with activity levels from high to low; S15, setting a threshold, taking the neurons ranked in the top few and with average response strength higher than the threshold as active neurons, and outputting a single inference result; S16. Repeat S11-S15 with different image samples. After multiple rounds of batch calculations, the average response strength of each neuron on all training samples is accumulated. S17. Compare the distribution of active neurons at different levels and observe the evolution trajectory of model feature extraction; S18. By analyzing the average response strength of each neuron in different levels of the statistical model to the sample set, key neurons with higher response values ​​are identified, and it is determined that these neurons have higher activity.

3. The method for determining the maximum contribution neuron propagation path based on reverse tracing according to claim 1 is characterized in that: In the step 2, the layer-by-layer interpretation method LRP and the gradient class activation map Grad-CAM interpretation method are combined, and the classification output is used to weight the gradient of the intermediate layer feature map to quantitatively calculate the influence of each neuron on the final classification result, that is, the contribution of the neuron.

4. The method for determining the maximum contribution neuron propagation path based on reverse tracing according to claim 3 is characterized in that: The specific method of tracing back from the classification output to find the neurons and paths that contribute most to the decision-making process is: S21, forward propagation of the neural network is performed to obtain the output value of each neuron; S22, back propagate from the classification results of the output layer to calculate the contribution of each neuron to the final output; S23, trace back to the input layer along the neuron path with the largest contribution, and obtain the interpretation path based on the neuron with the largest contribution; S24. Use the reverse maximum contribution neuron path to evaluate the stability of model predictions.

5. The method for determining the maximum contribution neuron propagation path based on reverse tracing according to claim 4 is characterized in that: In S21, the neural network consists of an intermediate layer and an output layer, the intermediate layer is connected to the input image sample, and the output layer is connected to the intermediate layer.

6. The method for determining the maximum contribution neuron propagation path based on reverse tracing according to claim 5, characterized in that: In S22, the contribution of each neuron to the final output is calculated as follows: Assume that the neural network has L layers, and the output of the i-th neuron in the first layer is The contribution of neuron j in the output layer is C j for: Neurons in the middle layer C i The contribution of the neural network level is defined as Among them, w ik is the set of child nodes of neuron i; w ik is the weight from neuron i to k.

7. The method for determining the maximum contribution neuron propagation path based on reverse tracing according to claim 6 is characterized in that: In S23, a depth-first search algorithm DFS is used to always select the maximum N contributing input neurons of the current neuron as the next neuron, and recursively calculate the contribution of each neuron; Select the neuron paths that contribute to the Top K, and use this method layer by layer based on the classification results to find the explanation path for the classification decision.

8. The method for determining the maximum contribution neuron propagation path based on reverse tracing according to claim 4 is characterized in that: In S24, the specific method of evaluating the stability of the model prediction using the reverse maximum contribution neuron path is: S241, perform multiple forward and reverse reasoning on each sample to obtain a reverse key path; S242. Count the reverse critical paths of all samples and find the standard reverse critical paths that appear frequently. The standard reverse critical paths reflect the model's discrimination patterns for different categories. S243. For the misclassified samples, analyze the difference between the reverse critical path of the sample and the corresponding standard reverse critical path to find out the blind spots of the model judgment; S244. Count the overlap between the reverse critical paths of all test samples and the standard reverse critical paths as an evaluation indicator of the prediction stability of the model.

9. The method for determining the maximum contribution neuron propagation path based on reverse tracing according to claim 8, characterized in that: In S243, the calculation method of the overlap between the reverse critical path of the test sample and the reverse critical path of the standard is: Set the i-th test sample to x i ; Set x i The reverse critical path is P i , the standard reverse critical path of the same sample is P s ; Calculate x i The path difference Diff(x i )for:

10. The method for determining the maximum contribution neuron propagation path based on reverse tracing according to claim 9, characterized in that: In S244, the calculation method of the Stability of the overlap between the reverse critical path of the test sample and the standard reverse critical path is: The overlap degree Stability is the reverse path stability. By analyzing the path difference and stability, the reliability of the diagnosis model judgment can be achieved and its blind spots can be discovered.

Images