A method and system for interpreting the prediction results of a deep neural network model

Abstract

The application provides a method and system for explaining the prediction result of a deep neural network model. In the method, some operations are added in the process of normal model training and prediction, that is, intermediate data in the model training is recorded, neuron activation data in the final model prediction is used, and some calculations are performed on the intermediate data in the model training and the neuron activation data in the final model prediction, so that the prediction result of the deep neural network model is explained. The method does not change the original operation of the model training and prediction, and does not affect the structure and performance of the model.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and more specifically, to a method, system, electronic device, and computer storage medium for interpreting the prediction results of a deep neural network model. Background Technology

[0002] In recent years, the development of deep neural network-related technologies has propelled the application of artificial intelligence to a new peak, producing many high-performance models and giving rise to many new application scenarios.

[0003] However, deep neural networks also have a fatal flaw—the black-box nature, meaning that the model's predictions are difficult to interpret. This makes the models lack in terms of security, transferability, fairness, and privacy. This problem significantly hinders their application in many fields. For example, in the field of medical image recognition, identifying a patient's X-ray to determine if they have a certain disease is a high-risk process, as the results directly relate to the patient's life and health. Therefore, the model needs not only to provide a prediction but also to explain the reasons behind its judgment; an unexplainable prediction is difficult for users to accept. Furthermore, in financial risk control and investment research, user decisions based on model predictions can affect hundreds of millions of dollars. To control risk, financial decision-makers generally require models to provide the prediction process and detailed explanations. For example, users need to know not only the probability of the risk but also which training samples were used to obtain the prediction and which specific indicators in those training samples led the model to make this judgment.

[0004] Therefore, how to explain the technology of deep neural networks has always been a technical challenge and has been continuously studied along with the development of deep neural networks.

[0005] Currently, deep neural network interpretation techniques are mainly divided into two categories:

[0006] 1) Post-training interpretation refers to the methods of analyzing and interpreting the model after training is complete. This includes many methods, such as analyzing the model's representation of data features, interpreting the original model using another model, and analyzing the model using advanced mathematical and physical methods, etc.

[0007] 2) Ad hoc models (transparent models) refer to modifications made during model design to inherently possess interpretable characteristics. This method can guide the model to train interpretable modules during training, achieving a certain level of interpretability. However, it generally impacts model performance and increases the difficulty of model training. Furthermore, this method is less effective for more complex models.

[0008] To date, the relevant technologies are still in the exploratory stage, and no unified and effective method has been found to interpret the prediction results of deep neural networks well. Moreover, the deeper and more complex the neural network model, the more difficult it is to analyze and understand. Summary of the Invention

[0009] In order to at least solve the technical problems existing in the background art, the present invention provides a method, system, electronic device and computer storage medium for interpreting the prediction results of a deep neural network model.

[0010] The first aspect of the present invention provides a method for interpreting the prediction results of a deep neural network model, comprising the following steps:

[0011] S1, During the training phase, record the gradient changes at each neural node during backpropagation of each training sample, and calculate the cumulative gradient data for each training sample.

[0012] S2, In the prediction phase, the gradient accumulation data of each training sample and the neuron activation data of the prediction sample are used to calculate the influence value of each training sample on the prediction result, and then the target training sample with the greatest influence is found by sorting.

[0013] S3, using the neuron activation data of the prediction sample to perform backpropagation calculation, and combining it with the gradient accumulation data of the training sample, to obtain the influence of each sub-element in the target training sample on the prediction result.

[0014] S4. Based on the influence of each target training sample on the prediction result and the influence of each sub-element of the target training sample on the prediction result, an interpretation of the prediction result of the neural network model is generated.

[0015] Furthermore, step S1 records not the gradients of parameters ω and θ, but the gradient of the output value y of each neuron, which is the intermediate calculation result of the backpropagation algorithm.

[0016] Furthermore, the formula for the neuron's output value y is as follows:

[0017]

[0018] Where x is the input value, y is the output value, ω is the weight corresponding to each input value, θ is the offset value, and Act() is called the activation function.

[0019] Further, in step S1, the calculation of the cumulative gradient data for each training sample includes:

[0020] Calculate the neuron output adjustment vector g obtained for each training sample each time it participates in model training;

[0021] The adjustment vector g of each training sample is summed to obtain a contribution matrix with the dimension of "number of training samples" × "number of neurons n", which is the gradient accumulation data.

[0022] Furthermore, the formula for calculating the adjustment vector g is as follows:

[0023]

[0024] In the above formula, C is the loss function, and y (n) This is the output of the nth neuron, where n is the total number of neurons in the model.

[0025] Furthermore, the formula for calculating the influence value of each training sample on the prediction result in step S2 is as follows:

[0026]

[0027] Among them, g (i) Let a be the contribution vector of the i-th training sample to the output value of each neuron; i To predict the output value of the i-th neuron in the model when the input data is processed by the model; n is the number of neurons in the model.

[0028] Further, in step S3, backpropagation calculation is performed using the neuron activation data of the predicted samples, and combined with the gradient accumulation data of the training samples, to obtain the influence of each sub-element in the target training samples on the prediction result, including:

[0029] Backpropagation is performed using the neuron activation data of the prediction sample to calculate the influence factor of each layer of neurons on the neurons of the previous layer. This factor is then calculated with the gradient accumulation data of the training sample and backpropagated to the input layer. This yields the magnitude of the influence of each sub-element of the target training sample on the prediction result during training.

[0030] Furthermore, in step S3, the formula for calculating the influence of each character or pixel on the prediction result is as follows:

[0031]

[0032] in, This represents the influence value transmitted from layer l to layer l-1. For the prediction of the i-th layer of layer l-1 (l-1) The output value of each neuron; For training samples, the i-th layer of the (l-1)-th layer (l-1) The contribution value of each neuron; For the i-th layer of layer l-1 (l-1)The i-th neuron in layer l is connected to the i-th neuron. (l) The ω parameter of each neuron; n is the number of neurons in the l-th layer;

[0033] Where, when layer l is the output layer, d represents the parameters of the predicted output layer, i.e. When layer l-1 is the input layer, d is the input data of the training samples, and g is 1.

[0034] A second aspect of the present invention provides a system for interpreting the prediction results of a deep neural network model, comprising an acquisition module, a processing module, and a storage module; the processing module is electrically connected to the acquisition module and the storage module.

[0035] The storage module is used to store executable computer program code;

[0036] The acquisition module is used to acquire data from the training and prediction phases of the deep neural network model and transmit it to the processing module.

[0037] The processing module is configured to execute the method described in the preceding one by invoking the executable computer program code in the storage module.

[0038] A third aspect of the present invention provides an electronic device comprising: a memory storing executable program code; a processor coupled to the memory; the processor invoking the executable program code stored in the memory to perform the method as described in any of the preceding claims.

[0039] A fourth aspect of the present invention provides a computer storage medium storing a computer program that, when executed by a processor, performs the method described in any of the preceding claims.

[0040] The beneficial effects of this invention are as follows:

[0041] 1) A novel method is proposed to effectively interpret the prediction results of black-box deep neural networks;

[0042] 2) The method of the present invention does not change the model structure, does not affect the training and inference process of the model, and does not reduce the model performance;

[0043] 3) The method of the present invention can not only give the magnitude of the influence of each training sample on the prediction result, but also explain how these training samples affect the prediction result.

[0044] 4) The method of the present invention is universal. Any interpretation of a deep neural network can refer to this method. Even new model structures designed later can be interpreted and analyzed based on the basic principles of the present invention.

[0045] 5) The method of this invention provides a new approach to analyzing and optimizing the inference process of deep learning models. Attached Figure Description

[0046] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a schematic diagram of the structure of an artificial neuron in a deep neural network disclosed in an embodiment of the present invention;

[0048] Figure 2 This is a schematic diagram of the structure of the fully connected neural network disclosed in the embodiments of the present invention;

[0049] Figure 3 This is a flowchart illustrating a method for interpreting the prediction results of a deep neural network model disclosed in an embodiment of the present invention.

[0050] Figure 4 This is a schematic diagram illustrating a specific way in which the prediction results are affected, as disclosed in an embodiment of the present invention.

[0051] Figure 5 This is a schematic diagram of the structure of an interpretation system for the prediction results of a deep neural network model disclosed in an embodiment of the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0053] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The singular forms “a,” “said,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms, and “multiple” generally includes at least two unless the context clearly indicates otherwise.

[0054] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0055] It should be understood that although the terms first, second, third, etc., may be used to describe ... in the embodiments of this application, these ... should not be limited to these terms. These terms are only used to distinguish .... For example, without departing from the scope of the embodiments of this application, first ... can also be referred to as second ..., and similarly, second ... can also be referred to as first ....

[0056] Depending on the context, the words “if” or “suppose” as used here can be interpreted as “when” or “in response to determination” or “in response to detection.” Similarly, depending on the context, the phrases “if determination” or “if detection (of the stated condition or event)” can be interpreted as “when determination” or “in response to determination” or “when detection (of the stated condition or event)” or “in response to detection (of the stated condition or event).”

[0057] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a product or system comprising a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a product or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the product or system that includes said element.

[0058] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0059] To make the solution of this invention clearer, the basic principles of deep neural networks are first introduced as follows:

[0060] Deep neural networks are models designed based on the working principles of neurons in the brain. Their basic unit is the artificial neuron, and multiple layers of artificial neurons are interconnected to form a deep learning model. A diagram of the artificial neuron structure can be found here. Figure 1 As shown.

[0061] Multilayer artificial neurons connect their output segments to their input segments, forming a deep neural network. Below, we will use the simplest fully connected neural network as an example for a detailed explanation; its structure can be found in [reference needed]. Figure 2As shown in the diagram, each circle represents an artificial neuron; the leftmost layer is the input layer, which receives the data used for prediction; the rightmost layer is the output layer, which outputs the prediction result; and the middle layer is the hidden layer. The number of layers and the number of neurons in each layer can be set according to actual needs. The output value of each neuron in one layer serves as the input value for the next layer's neurons, and each neuron in the current layer is connected to every neuron in the next layer—this is the structural characteristic of a fully connected neural network.

[0062] The parameters ω and θ of each neuron in a neural network, and the accurate prediction results, are obtained through model training. During training, training samples are input into the model, and forward propagation (starting from the input layer and proceeding layer by layer) yields an output. This output, along with the true label, is then used to calculate the loss function, resulting in a loss value—the difference between the actual and expected output values. The goal of model optimization is to minimize this loss value, leading to more accurate predictions.

[0063] Because there are too many parameters, the minimum value cannot be directly obtained through calculation. However, we can gradually approach the minimum of the loss function using certain methods. For example, if we compare the loss function to a valley, the minimum value is at the bottom of the valley. By continuously moving downhill until we can't go any further, we find the minimum. This method is called gradient descent. Finding the downhill direction is equivalent to finding the opposite direction of the gradient. By gradually adjusting the parameters along the opposite direction of the gradient, we eventually obtain the neuron parameters that minimize the loss function. This method of finding the gradient is called backpropagation. The calculation starts from the output layer and propagates to the previous layer, calculating the gradient of the neuron parameters in each layer, ultimately obtaining the gradient value of each parameter in the entire neural network.

[0064] See Figure 3 The flowchart shown illustrates a method for interpreting the prediction results of a deep neural network model, comprising the following steps:

[0065] S1, During the training phase, record the gradient changes at each neural node during backpropagation of each training sample, and calculate the cumulative gradient data for each training sample.

[0066] S2, In the prediction phase, the gradient accumulation data of each training sample and the neuron activation data of the prediction sample are used to calculate the influence value of each training sample on the prediction result, and then the target training sample with the greatest influence is found by sorting.

[0067] S3, for the target training sample, backpropagation calculation is performed using the neuron activation data of the prediction sample, and combined with the gradient accumulation data of the target training sample, to obtain the influence of each sub-element (i.e., character or pixel) in the target training sample on the prediction result.

[0068] S4. Based on the influence of each target training sample on the prediction result and the influence of each sub-element of the target training sample on the prediction result, an interpretation of the prediction result of the neural network model is generated.

[0069] In this embodiment, the fundamental reason why a trained model makes a certain prediction lies in the training data. During the model's training process, relevant training samples are used. Through the backpropagation algorithm, the parameters of each neuron are continuously adjusted, allowing the model to activate the corresponding neurons based on these parameters and make the final judgment. These training samples and the input data for prediction are not exactly the same, but they must have similar correlations, and this correlation is the source of the model's interpretability.

[0070] The above summarizes the core logic upon which the technical solution of this invention is based. Specifically, during the normal model training and prediction process, some operations are added: by recording intermediate training data and then using the neuron activation data predicted by the model to perform calculations with the intermediate training data, the prediction results of the deep neural network model are explained. This method does not change the original model training and prediction operations; therefore, it does not affect the model's structure or performance.

[0071] Furthermore, step S1 records not the gradients of parameters ω and θ, but the gradient of the output value y of each neuron, which is the intermediate calculation result of the backpropagation algorithm.

[0072] In this embodiment, the parameter values ​​of the deep neural network are obtained through gradual training and adjustment based on training samples and their labels. The reason why this invention needs to record the gradient of y is that when a neuron is activated, its output is positive, and it will participate in subsequent calculations. When y is not activated, that is, when the output value is zero, this neuron does not contribute to the subsequent calculation results.

[0073] Therefore, it is necessary to know whether the output value y of each neuron increases or decreases when using the corresponding training samples for parameter tuning. According to the principle of the backpropagation algorithm, the amount by which each training sample increases the output value y of each neuron is the negative of the gradient of y.

[0074] Furthermore, the formula for the neuron's output value y is as follows:

[0075]

[0076] Where x is the input value, y is the output value, ω is the weight corresponding to each input value, θ is the offset value, and Act() is called the activation function.

[0077] In this embodiment, the activation function can be a non-linear function such as Sigmoid or ReLU.

[0078] Further, in step S1, the calculation of the cumulative gradient data for each training sample includes:

[0079] Calculate the neuron output adjustment vector g obtained for each training sample each time it participates in model training;

[0080] The adjustment vector g of each training sample is summed to obtain a contribution matrix with the dimension of "number of training samples" × "number of neurons n", which is the gradient accumulation data.

[0081] In this embodiment, the contribution matrix represents the contribution of each training sample to the output value of each neuron.

[0082] Furthermore, the formula for calculating the adjustment vector g is as follows:

[0083]

[0084] In the above formula, C is the loss function, and y (n) This is the output of the nth neuron, where n is the total number of neurons in the model.

[0085] Furthermore, the formula for calculating the influence value of each training sample on the prediction result in step S2 is as follows:

[0086]

[0087] Among them, g (i) Let a be the contribution vector of the i-th training sample to the output value of each neuron; i To predict the output value of the i-th neuron in the model when the input data is processed by the model; n is the number of neurons in the model.

[0088] In this embodiment, the activation function of the neuron is similar to a step function; the output is 0 when not activated and positive when activated. Therefore, in subsequent calculations, whether multiplication or addition is performed, the output of an unactivated neuron is 0. In other words, unactivated neurons have no impact on the prediction result in this prediction. Each value in vector g is the contribution of the training sample to the output of each neuron during training. Therefore, the value f calculated by formula (3) is the contribution of a certain training sample to the current prediction result. By sorting the f values ​​of all training samples, we can find out which training samples have the greatest impact on the prediction result, that is, the model has the ability to obtain the current prediction result through training on these training samples.

[0089] Further, in step S3, backpropagation calculation is performed using the neuron activation data of the predicted samples, and combined with the gradient accumulation data of the training samples, to obtain the influence of each sub-element in the target training samples on the prediction result, including:

[0090] Backpropagation is performed using the neuron activation data of the prediction sample to calculate the influence factor of each layer of neurons on the neurons of the previous layer. This factor is then calculated with the gradient accumulation data of the training sample and backpropagated to the input layer. This yields the magnitude of the influence of each sub-element of the target training sample on the prediction result during training.

[0091] In this embodiment, the present invention employs the backpropagation method of influence to analyze how each training sample affects the prediction result. Specifically: since a model calculates prediction data by propagating from the input layer to the output layer layer by layer to obtain the calculation result, and during training, the adjustment of model parameters is achieved by gradually obtaining the gradient of each parameter through backpropagation before adjustment. Therefore, the propagation of influence should also be similar to the backpropagation of gradients, starting from the output layer and moving forward, first obtaining the influence of the previous layer's neurons on the output layer, and then propagating backward layer by layer, finally obtaining the influence of each pixel in the input layer (i.e., the training sample image) on the prediction result.

[0092] Furthermore, in step S3, the formula for calculating the influence of each character or pixel on the prediction result is as follows:

[0093]

[0094] in, This represents the influence value transmitted from layer l to layer l-1. For the prediction of the i-th layer of layer l-1 (l-1) The output value of each neuron; For training samples, the i-th layer of the (l-1)-th layer (l-1) The contribution value of each neuron; For the i-th layer of layer l-1 (l-1) The i-th neuron in layer l is connected to the i-th neuron. (l) The ω parameter of each neuron; n is the number of neurons in the l-th layer;

[0095] Where, when layer l is the output layer, d represents the parameters of the predicted output layer, i.e. When layer l-1 is the input layer, d is the input data of the training samples, and g is 1.

[0096] In this embodiment, the present invention uses the neuron activation status 'a' of the prediction model, combined with the neuron output contribution 'g' of the training samples, and then sums the influence values ​​transmitted from layer l according to the weight 'ω' of the neurons in each layer. Finally, the training sample that has the greatest impact on the prediction result and its specific influence on the prediction result are obtained.

[0097] For verification examples, please refer to Figure 4 As shown, Figure 4 The first image in the diagram is the prediction sample; the subsequent black images represent the top 12 training samples ranked from highest to lowest influence; the light gray images illustrate the specific influence of each pixel in the preceding black image, with darker pixels having greater influence. It can be seen that the results offer some interpretability for the model's predictions.

[0098] It should be noted that the influence of each training sample on the prediction result can be expressed in the following manner: Figure 4 The accompanying drawings can be described using textual descriptions, such as... Figure 4 The illustrated content is interpreted according to preset rules, but this invention does not limit this interpretation.

[0099] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of an interpretation system for the prediction results of a deep neural network model disclosed in an embodiment of the present invention. For example... Figure 5 As shown, an interpretation system for the prediction results of a deep neural network model according to an embodiment of the present invention includes an acquisition module (101), a processing module (102), and a storage module (103); the processing module (102) is connected to the acquisition module (101) and the storage module (103);

[0100] The storage module (103) is used to store executable computer program code;

[0101] The acquisition module (101) is used to acquire data from the training and prediction phases of the deep neural network model and transmit it to the processing module (102).

[0102] The processing module (102) is configured to execute the method described in the preceding one by invoking the executable computer program code in the storage module (103).

[0103] The specific functions of the interpretation system for the prediction results of a deep neural network model in this embodiment are the same as those in the above embodiments. Since the system in this embodiment adopts all the technical solutions of the above embodiments, it has at least all the beneficial effects brought about by the technical solutions of the above embodiments, and will not be described in detail here.

[0104] This invention also discloses an electronic device, comprising: a memory storing executable program code; a processor coupled to the memory; the processor calling the executable program code stored in the memory to execute the method described in the foregoing embodiments.

[0105] This invention also discloses a computer storage medium storing a computer program, which is executed by a processor to perform the methods described in the foregoing embodiments.

[0106] The apparatus / system according to embodiments of this disclosure may include a processor, memory for storing and executing program data, permanent memory such as a disk drive, a communication port for processing communication with external devices, and a user interface device, etc. The method is implemented as a software module or may be stored on a computer-readable recording medium as computer-readable code or program instructions executable by a processor. Examples of computer-readable recording media may include magnetic storage media (e.g., read-only memory (ROM), random access memory (RAM), floppy disk, hard disk, etc.), optical reading media (e.g., CD-ROM, DVD, etc.). The computer-readable recording medium may be distributed across computer systems connected to a network, and the computer-readable code may be stored and executed in a distributed manner. The medium may be computer-readable, stored in memory, and executed by a processor.

[0107] Embodiments of this disclosure can be designated as functional block components and various processing operations. Functional blocks can be implemented as various numbers of hardware and / or software components that perform specific functions. For example, embodiments of this disclosure can implement direct circuit components, such as memories, processing circuits, logic circuits, lookup tables, etc., that can perform various functions under the control of one or more microprocessors or other control devices. Components of this disclosure can be implemented by software programming or software components. Similarly, embodiments of this disclosure can include various algorithms implemented by combinations of data structures, procedures, routines, or other programming components, and can be implemented by programming or scripting languages ​​(such as C, C++, Java, assembler, etc.). Functional aspects can be implemented by algorithms executed by one or more processors. Furthermore, embodiments of this disclosure can implement related techniques for electronic environment setup, signal processing, and / or data processing. Terms such as “mechanism,” “element,” “unit,” etc., can be used broadly and are not limited to mechanical and physical components. These terms can refer to a series of software routines associated with processors, etc.

[0108] Specific embodiments are described in this disclosure as examples, and the scope of the embodiments is not limited thereto.

[0109] While embodiments of this disclosure have been described, those skilled in the art will understand that various changes in form and detail may be made therein without departing from the spirit and scope of this disclosure as defined by the appended claims. Therefore, the above embodiments of this disclosure should be interpreted as exemplary and are not limiting in any way. For example, each component described as a single unit may be executed in a distributed manner, and similarly, components described as distributed may be executed in a combined manner.

[0110] All examples or example terms (e.g., etc.) used in the embodiments of this disclosure are for the purpose of describing embodiments of this disclosure and are not intended to limit the scope of embodiments of this disclosure.

[0111] Furthermore, unless otherwise explicitly stated, expressions such as “necessary” or “important” associated with certain components do not necessarily indicate that the components are absolutely necessary.

[0112] Those skilled in the art will understand that embodiments of this disclosure may be implemented in modified forms without departing from the spirit and scope of this disclosure.

[0113] Because this disclosure allows for various changes to the embodiments thereof, it is not limited to the specific embodiments described herein, and it will be understood that all changes, equivalents, and alternatives that do not depart from the spirit and scope of this disclosure are included herein. Therefore, the embodiments of this disclosure described herein should be understood as illustrative in all respects and should not be construed as limiting.

[0114] Furthermore, terms such as "unit" and "module" refer to a unit that can be implemented as hardware or software or a combination of hardware and software to process at least one function or operation. "Unit" and "module" can be stored in a storage medium to be addressed and can be implemented as a program that can be executed by a processor. For example, "unit" and "module" can refer to components such as software components, object-oriented software components, class components, and task components, and can include processes, functions, attributes, procedures, subroutines, program code segments, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, or variables.

[0115] In this disclosure, the statement "A may include one of a1, a2, and a3" can broadly indicate that examples that can be included in element A include a1, a2, or a3. This statement should not be construed as limiting the meaning of examples included in element A to a1, a2, and a3. Therefore, as examples included in element A, elements other than a1, a2, and a3 should not be interpreted as excluding elements. Furthermore, this statement indicates that element A may include a1, a2, or a3. This statement does not imply that the elements included in element A must be selected from a specific set of features. That is, this statement should not be construed restrictively as indicating that a1, a2, or a3 must be selected from a set that includes a1, a2, and a3 to be included in element A.

[0116] Furthermore, in this disclosure, the expression "at least one of a1, a2 and / or a3" means one of "a1", "a2", "a3", "a1 and a2", "a1 and a3", "a2 and a3", and "a1, a2 and a3". Therefore, it should be noted that unless explicitly described as "at least one of a1, at least one of a2, and at least one of a3", the expression "at least one of a1, a2 and / or a3" should not be interpreted as "at least one of a1", "at least one of a2", and "at least one of a3".

Claims

1. A method for interpreting the prediction results of a deep neural network model based on the gradient of training samples, characterized in that, Includes the following steps: S1, during the training phase, records the gradient changes at each neural node during backpropagation of each training sample, and calculates the cumulative gradient data for each training sample, including: Calculate the neuron output adjustment vector obtained for each training sample during each model training iteration. The adjustment vector for each training sample The contributions are summed separately to obtain a contribution matrix with the dimension of "number of training samples" × "number of neurons n", which is the gradient accumulation data. The adjustment vector The calculation formula is as follows: ； In the above formula, C is a loss function, is the output of the nth neuron, and n is the total number of neurons in the model. S2, In the prediction phase, the gradient accumulation data of each training sample and the neuron activation data of the prediction sample are used to calculate the influence value of each training sample on the prediction result, and then the target training sample with the greatest influence is found by sorting; wherein, the prediction sample is an X-ray image. The formula for calculating the influence value of each training sample on the prediction result is as follows: ； wherein, is the contribution value of the i-th neuron for the training sample; is the output value of the i-th neuron in the model when the prediction input data is calculated by the model; n is the number of neurons of the model; S3, using the neuron activation data of the predicted samples for backpropagation calculation, and combining it with the gradient accumulation data of the training samples, the magnitude of the influence of each sub-element in the target training sample on the prediction result is obtained, and the calculation formula is as follows: ； in, This represents the influence value transmitted from layer l to layer l-1. For the prediction of the (l-1)th layer The output value of each neuron; For training samples, the first layer of the (l-1)th layer... The contribution value of each neuron; For the l-1 layer The first neuron to the lth layer The ω parameter of each neuron; n is the number of neurons in the l-th layer; wherein, when the l-th layer is an output layer, d is a prediction result output layer parameter, i.e. when the (l-1)-th layer is an input layer, d is an input data of a training sample, and g is 1. S4. Based on the magnitude of the influence value of each target training sample on the prediction result and the magnitude of the influence value of each sub-element of the target training sample on the prediction result, an explanation of the prediction result of the neural network model is generated, that is, an explanation of how the training samples affect the prediction result. 2.The method of claim 1, wherein: Step S1 records not the gradients of parameters ω and θ, but the gradient of the output value y of each neuron, which is the intermediate calculation result of the backpropagation algorithm; where ω is the weight corresponding to each input value and θ is the offset value. 3.The method of claim 2, wherein: The formula for the neuron's output value y is as follows: ； Where x is the input value, y is the output value, ω is the weight corresponding to each input value, θ is the offset value, and Act() is the activation function. 4.The method of claim 3, wherein: In step S3, backpropagation calculation is performed using the neuron activation data of the predicted samples, and combined with the gradient accumulation data of the training samples, to obtain the influence of each sub-element in the target training samples on the prediction result, including: Backpropagation is performed using the neuron activation data of the prediction sample to calculate the influence factor of each layer of neurons on the neurons of the previous layer. This factor is then calculated with the gradient accumulation data of the training sample and backpropagated to the input layer. This yields the magnitude of the influence of each sub-element of the target training sample on the prediction result during training.

5. A system for interpreting the prediction results of a deep neural network model based on the gradient of training samples, comprising an acquisition module, a processing module, and a storage module; wherein the processing module is electrically connected to the acquisition module and the storage module; The storage module is used to store executable computer program code; The acquisition module is used to acquire data from the training and prediction phases of the deep neural network model and transmit it to the processing module. characterized in that The processing module is configured to execute the method as described in any one of claims 1-4 by calling the executable computer program code in the storage module.

6. An electronic device, comprising: Memory containing executable program code; A processor coupled to the memory; characterized in that: the processor calls the executable program code stored in the memory to perform the method as described in any one of claims 1-4.

Images