Model Behavior Interpretability Method, System, Medium and Device

Abstract

The present application discloses a method, system, medium, and device for interpreting model behavior, which belongs to the field of neural network model technology. The method includes: determining the contribution of each test unit in the test sample to the prediction result based on the model prediction result; sorting the contribution, and selecting a first preset number of test units as interpretation units based on the contribution sorting result; respectively calculating the correlation between each training sample in the training set and the interpretation unit; sorting the correlation, and selecting a second preset number of training samples as interpretation samples based on the correlation sorting result. The method interprets the output results of the neural network model by obtaining interpretation units and interpretation samples related to the training results, and combines the neural network interpretability scheme of explaining the model behavior in the training and testing stages, forming an inference and interpretation link from training data to input test samples to model prediction results, and better explaining the model behavior.

Description

Technical Field

[0001] The present application relates to the field of neural network model technology, and in particular to a method, system, medium and device for interpreting model behavior. Background Art

[0002] Deep learning neural network models are "black box" models, making it difficult for users to understand how they process data and arrive at predictions. This results in users being unable to control the model's operation, inevitably leading to unexpected results. This can affect further optimization of the model and negatively impact the user experience. Therefore, a suitable interpretation method is crucial for deep learning network models to be used or further optimized. Summary of the Invention

[0003] In view of the problem in the existing technology that deep learning neural network models have a "black box" nature, which makes it impossible for users to control the operation process of deep learning neural network models, and inevitably leads to unexpected results, affecting the user's experience of using the neural network model, this application proposes a model behavior explainability method, system, medium and device.

[0004] In one technical solution of the present application, a model behavior interpretability method is provided, including: determining the contribution of each test unit in the test sample to the prediction result based on the model prediction result; sorting the contribution, and selecting a first preset number of test units as explanation units based on the contribution sorting result; respectively calculating the correlation between each training sample in the training set and the explanation unit; sorting the correlation, and selecting a second preset number of training samples as explanation samples based on the correlation sorting result.

[0005] In another technical solution of the present application, a model behavior explainability system is provided, including: a contribution calculation module, which determines the contribution of each test unit in the test sample to the prediction result based on the model prediction result; an explanation unit determination module, which sorts the contribution and selects a first preset number of test units as explanation units based on the contribution sorting result; a correlation calculation module, which calculates the correlation between each training sample in the training set and the explanation unit respectively; an explanation sample determination module, which sorts the correlation and selects a second preset number of training samples as explanation samples based on the correlation sorting result.

[0006] In another technical solution of the present application, a computer-readable storage medium is provided, wherein the storage medium stores computer instructions, and the computer instructions are operated to execute the model behavior interpretability method in Solution 1.

[0007] In another technical solution of the present application, a computer device is provided, wherein the computer device includes a processor and a memory, the memory stores computer instructions, and the processor operates the computer instructions to execute the model behavior interpretability method in solution one.

[0008] The beneficial effects of the present application are as follows: the present application explains the output results of the neural network model by obtaining explanation units and explanation samples related to the training results during the training and testing stages of the neural network model, and combines the neural network interpretability scheme that explains the model behavior during the training and testing stages to form an inference and explanation link from training data to input test samples to model prediction results, thereby better explaining the model behavior. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0010] Figure 1 An embodiment of the model behavior interpretability method of the present application is shown;

[0011] Figure 2 An embodiment of determining the contribution of a test unit to a prediction result in the model behavior interpretability method of the present application is shown;

[0012] Figure 3 An embodiment of the model behavior explainability system of the present application is shown.

[0013] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0014] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0015] The terms "first," "second," "third," "fourth," and the like (if any) in the specification and claims of the present application and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequential sequence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in orders other than those illustrated or described herein. In addition, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusions, e.g., a product or apparatus comprising a series of steps or units is not necessarily limited to those units explicitly listed, but may include other units not explicitly listed or inherent to those products or apparatuses.

[0016] Deep learning neural network models have a "black box" nature. It is difficult for users to know how deep learning neural network models process data and how they obtain prediction results. As a result, users cannot control the operation process of the deep learning neural network models, and will inevitably obtain unexpected results, which will affect the further optimization of the deep learning neural network models and affect the user experience of the neural network models.

[0017] The use of neural network models is divided into two stages: training and testing. Taking a text training model as an example, training refers to using labeled data to guide the model's learning, while testing involves inputting raw text into the trained model and the model providing its predictions.

[0018] Based on the two phases of training and testing, this solution proposes to combine these two phases to explain model performance. Specifically, the solution aims to address the following questions: which parts of the input text the model focuses on when making predictions, and which training examples in the training set are most "correlated" with these parts. This approach allows the solution to comprehensively explain model behavior from both the training and testing perspectives, rather than focusing on either one in isolation.

[0019] To this end, this application proposes a model behavior interpretability method. First, based on the model prediction results, the contribution of each test unit in the test sample to the prediction result is determined; then, the contribution is sorted, and a first preset number of test units are selected as explanation units based on the contribution ranking results; then, the correlation between each training sample in the training set and the explanation unit is calculated; finally, the correlation is sorted, and a second preset number of training samples are selected as explanation samples based on the correlation ranking results. The model prediction results are explained by the obtained explanation units and explanation samples. That is, the reason why the model obtains the prediction result is due to the existence of the explanation units in the test sample and the explanation samples in the training set. This explains the model behavior and facilitates the control of the model behavior.

[0020] By obtaining explanation units and explanation samples related to the training results during the training and testing phases of the neural network model, the output results of the neural network model are explained, and a neural network interpretability scheme that explains the model behavior during the combined training and testing phases is formed to form an inference explanation link from training data to input test samples to model prediction results, so as to better explain the model behavior. This application can be applied to the interpretation of model results and can be directly used for downstream tasks of multiple neural network model predictions, including model behavior analysis, error analysis, adversarial sample generation, adversarial sample defense, and corrected model predictions, with a wide range of applications. In addition, the interpretability method of this application can be applied to any neural network model structure and downstream tasks, and has extremely strong scalability.

[0021] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0022] Figure 1 An embodiment of the model behavior explainability method of the present application is shown.

[0023] exist Figure 1 In the illustrated embodiment, the model behavior explainability method of the present application includes process S101, which determines the contribution of each test unit in the test sample to the prediction result based on the model prediction result.

[0024] In this embodiment, the model's usage process is divided into two phases: training and testing. During the testing phase, the model generates predictions based on the input test samples. The predictions generated are related to the input test samples. Therefore, the model interprets the predictions generated by the model based on the components of the test samples that contribute significantly to the predictions. Therefore, each test unit in the test samples input to the model is analyzed, and the contribution of each test unit to the prediction is calculated.

[0025] By calculating the contribution of each test unit in the test sample to the prediction result output by the model, we can obtain the different influences of different test units on the prediction result output by the model according to the level of contribution. That is, the higher the contribution, the higher the influence of the corresponding test sample on the prediction result obtained by the model, thereby explaining the prediction result output by the model.

[0026] Optional, Figure 2 An embodiment of determining the contribution of a test unit to a prediction result in the model behavior explainability method of the present application is shown.

[0027] like Figure 2 In the embodiment shown, determining the contribution of each test unit in the test sample to the prediction result based on the model prediction result includes process S201, determining the score corresponding to the prediction result; and process S202, determining the gradient corresponding to each test unit based on the score, and then determining the contribution corresponding to each test unit, wherein the gradient is positively correlated with the contribution.

[0028] Specifically, the contribution of each test unit in the test sample is calculated using the following formula (1), where formula (1) is expressed as follows:

[0029]

[0030] As shown in formula (1), e represents a test unit in the test sample of the input model, where e can be a word or a sentence. y Represents the score of the prediction result y output by the model, It's S y The gradient of a test unit e, and w y (e) represents the score of test unit e. This formula can be used to calculate the contribution score of each test unit in the test sample to the model prediction result y. The higher the score, the more important the test unit is and the greater its contribution to the model prediction result y.

[0031] Specifically, the above process is described below with reference to a specific example.

[0032] For example, the content of a test sample is "The movie is very attractive, and I particularly like its soundtrack". If this test sample is input into the neural network model, the prediction result is "positive". Among them, for the neural network model, due to its "black box" characteristics, it is difficult for people to explain the behavior of the model outputting the result "positive". Therefore, this application starts with the test sample input into the neural network model, determines the contribution of each test unit in the test sample to the prediction result of the neural network model, and then explains the behavior of the model. According to the model prediction result "positive", the contribution of each test unit in the test sample is calculated. For example, the contribution of the test units such as "very attractive", "like", "movie" and "soundtrack" in the test sample is calculated according to a predetermined calculation method, and finally the contribution of each test unit in the test sample to the model prediction result is obtained. The model behavior is explained by calculating the contribution of each test unit to the model prediction result.

[0033] exist Figure 1In the illustrated embodiment, the model behavior interpretability method of the present application includes process S102 of sorting contribution degrees and selecting a first preset number of test units as explanation units based on the contribution degree sorting results.

[0034] In this embodiment, after calculating the contribution of each test unit in the test sample to the model prediction result, the corresponding test units are sorted according to the size of the contribution. According to the sorting results, a preset number of test units are selected from large to small as explanation units for the model prediction result. Among them, the higher the contribution corresponding to the test unit, the greater the influence of the test unit on the prediction result obtained by the model, and the more reasonable explanation can be provided for the behavior of the model in obtaining the prediction result.

[0035] Specifically, in combination with the above example, according to the model's prediction result of "positive", the test sample "the movie is very attractive, and I particularly like its soundtrack" is analyzed, wherein the contribution of each test sample to the prediction result is ranked as: "very attractive", "like", "movie" and "soundtrack". According to a pre-set first preset number, the corresponding test unit is selected as the explanation unit. For example, when the first preset number is set to 1, the test sample "very attractive" is used as the explanation unit for explaining the model's prediction result of "positive"; when the first preset number is set to 2, the two test units "very attractive" and "like" in the test sample are used as the explanation units for explaining the model's prediction result of "positive", and the model behavior is explained.

[0036] By analyzing the contribution of each test unit in the test sample to the model prediction results, and selecting one or more test units with the highest contribution as the explanation units of the model prediction results, the behavior of the model is explained, the accuracy of the explanation of the model behavior is ensured, and the correct guidance for the subsequent use of the model is provided.

[0037] exist Figure 1 In the illustrated embodiment, the model behavior interpretability method of the present application includes process S103 of respectively calculating the correlation between each training sample in the training set and the explanation unit.

[0038] In this embodiment, the use of the neural network model includes two stages: training and testing. The neural network model is explained in terms of testing through process S101 and process S102. In order to ensure the accuracy of the interpretation of the neural network model and provide more reasonable guidance for the subsequent use of the neural network model, it is necessary to explain the behavior of the neural network model in terms of the training of the neural network model. Combined with the interpretation results of the neural network model in the above process S101 and process S102, the behavior of the neural network model is explained at the training level after the neural network model, as described below:

[0039] After step S102, the explanatory units in the test sample that contribute most to the prediction results of the neural network model are obtained. The correlation between each training sample and the explanatory unit is calculated in the training set. The higher the correlation between the training text and the explanatory unit in the training set, the greater the impact of the training sample on the prediction results of the neural network model.

[0040] The correlation between each training sample and the explanation unit in the training set is calculated by the following formula (2), which is as follows:

[0041]

[0042] In this formula, z is a training sample in the training set. is the gradient of the above score with respect to the model parameters θ, H θ is the Hessian matrix of the training set, and That is, the loss calculated by the model on the training sample z The gradient of the model parameter θ. This formula shows the correlation score between each training sample z and the explanation unit e found above, where the larger the score, the more relevant the training sample is to the corresponding explanation unit.

[0043] exist Figure 1 In the illustrated embodiment, the model behavior interpretability method of the present application includes process S104 of sorting the relevance and selecting a second preset number of training samples as explanation samples based on the sorted relevance results.

[0044] In this embodiment, after calculating the correlation between each training sample in the training set and the interpretation unit, the correlations are sorted by magnitude, and based on the sorting results, a second predetermined number of training samples are selected as interpretation texts to explain the prediction results of the neural network model. Because the interpretation samples have a high correlation with the interpretation unit determined above, the obtained interpretation samples can better explain the prediction results of the neural network model, thereby improving the accuracy of the interpretation of the neural network model's behavior.

[0045] Specifically, the above process is described below with an example.

[0046] For the test sample "The movie is very attractive, and I particularly like its soundtrack," analysis reveals that the explanatory unit that contributes most to the model's prediction of "positive" is "very attractive." The correlation between this explanatory unit and each training sample in the training set is then calculated to obtain an explanatory sample. For example, when training the model, the training set includes samples 1, 2, and 3. Sample 1 contains the word "attractive," and its training result is "positive"; sample 2 contains the word "like," and its training result is "positive"; sample 3 contains the word "hate," and its training result is "negative." After obtaining the explanatory unit "very attractive," the correlation between this explanatory unit and each sample is calculated. Clearly, the correlation with sample 1 is the highest, followed by sample 2, and the lowest with sample 3. One or more training samples in the training set are selected as explanatory samples as needed. If one is selected as an explanatory sample, sample 1, which has the highest correlation with the explanatory unit, is selected as the explanatory sample.

[0047] Through the above method, the present application starts from the prediction results of the model, finds the most contributing explanation unit in the test sample according to the prediction results, and then finds the training samples related to the explanation unit in the training set as explanation samples based on the explanation unit. The prediction results of the model are explained by the explanation unit and the explanation sample. This helps the staff to better explain the behavior of the neural network model and facilitates the control of the behavior of the model. In addition, the model behavior interpretability method of the present application can be directly used for multiple downstream tasks, including: model behavior analysis, error analysis, adversarial sample generation, adversarial sample defense and correction model prediction. It has a wide range of applications. The interpretability solution can be used for any model structure and downstream tasks and has extremely strong scalability.

[0048] Figure 3 An embodiment of the model behavior explainability system of the present application is shown.

[0049] like Figure 3 As shown, the model behavior explainability system of the present application includes: a contribution calculation module 301, which determines the contribution of each test unit in the test sample to the prediction result based on the model prediction result; an explanation unit determination module 302, which sorts the contribution and selects a first preset number of test units as explanation units based on the contribution sorting result; a correlation calculation module 303, which calculates the correlation between each training sample in the training set and the explanation unit respectively; an explanation sample determination module 304, which sorts the correlation and selects a second preset number of training samples as explanation samples based on the correlation sorting result.

[0050] Optionally, in the interpretation unit determination module, the interpretation unit determination module is used to determine the score corresponding to the prediction result, and determine the gradient corresponding to each test unit based on the score, and then determine the contribution corresponding to each test unit, wherein the gradient is positively correlated with the contribution.

[0051] Optionally, in the explanation sample determination module, the explanation sample determination module is used to calculate the relevance score of each training sample to the explanation unit, and determine the relevance according to the relevance score, wherein the relevance score is positively correlated with the relevance.

[0052] In one embodiment of the present application, a computer-readable storage medium stores computer instructions, wherein the computer instructions are operated to execute the model behavior interpretability method described in any embodiment. The storage medium can be directly in hardware, in a software module executed by a processor, or in a combination of the two.

[0053] The software modules may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from and write information to the storage medium.

[0054] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination thereof. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. In the alternative, the storage medium may be integral to the processor. The processor and storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In the alternative, the processor and storage medium may reside as discrete components in the user terminal.

[0055] In a specific embodiment of the present application, a computer device includes a processor and a memory, the memory storing computer instructions, wherein: the processor operates the computer instructions to execute the model behavior interpretability method described in any embodiment.

[0056] In the embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0057] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0058] The above are merely embodiments of the present application and are not intended to limit the patent scope of the present application. Any equivalent structural transformations made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A model behavior interpretability method, characterized in that: include: Determine the contribution of each test unit in the test sample to the prediction result based on the model prediction result. For the test sample whose prediction result is "positive" and the content is "the movie is very attractive, and I particularly like its soundtrack", calculate the contribution of the test units "very attractive", "like", "movie", and "soundtrack" in the test sample to the prediction result respectively; The contribution degrees are sorted by size, and a first preset number of the test units are selected from large to small as explanation units according to the contribution degree sorting results, and the test units are sorted by contribution degree as follows: "very attractive", "like", "movie", and "soundtrack". When the first preset number is 1, the test unit with "very attractive" in the test sample is selected as the explanation unit with the prediction result of "positive"; Calculating the relevance of each training sample in the training set with the explanation unit respectively. For the training set including sample 1, sample 2, and sample 3, calculating the relevance of sample 1, sample 2, and sample 3 with the explanation unit of "extremely attractive" respectively. The relevance is sorted, and a second preset number of the training samples are selected as explanation samples based on the relevance sorting result, where the first sample contains the word "attractive", the second sample contains the word "like", and the third sample contains the word "hate", and the relevance of the first, second, and third samples to the explanation unit "very attractive" decreases in sequence. When the second preset number is 1, the first sample is selected as the explanation sample; Explain the prediction results of the model by combining the explanation unit in the testing phase and the explanation samples in the training phase; The application scenarios of the model behavior explainability method include downstream tasks of neural network model prediction, and the downstream tasks include model behavior analysis, error analysis, adversarial sample generation, adversarial sample defense, and correcting model predictions.

2. The model behavior interpretability method according to claim 1, characterized in that Determining the contribution of each test unit in the test sample to the prediction result based on the model prediction result includes: Determining a score corresponding to the prediction result; The gradient corresponding to each of the test units is determined according to the score, and then the contribution corresponding to each of the test units is determined, wherein the gradient is positively correlated with the contribution.

3. The model behavior interpretability method according to claim 1, characterized in that The step of calculating the correlation between each training sample in the training set and the explanation unit includes: Calculating the relevance score of each of the training samples to the explanation unit; The relevance is determined based on the relevance score, wherein the relevance is positively correlated with the relevance score.

4. A model behavior explainability system, characterized by: include: a contribution calculation module, which determines the contribution of each test unit in the test sample to the prediction result based on the model prediction result, and for the test sample whose prediction result is "positive" and the content is "the movie is very attractive, and I particularly like its soundtrack", calculates the contribution of the test units "very attractive", "like", "movie", and "soundtrack" in the test sample to the prediction result respectively; an interpretation unit determination module, which sorts the contribution degrees by size, and selects a first preset number of the test units as interpretation units based on the contribution degree sorting results from large to small, wherein the test units are sorted by contribution degree as follows: "very attractive", "like", "movie", and "soundtrack". When the first preset number is 1, the test units in the test sample that are "very attractive" are selected as the interpretation units for the "positive" prediction result; a correlation calculation module, which calculates the correlation between each training sample in the training set and the explanation unit, and for the training set including sample 1, sample 2, and sample 3, calculates the correlation between sample 1, sample 2, and sample 3 and the explanation unit of "highly attractive"; an explanation sample determination module, which sorts the relevances and selects a second preset number of the training samples as explanation samples based on the sorted relevances, wherein the first sample contains the word "attractive", the second sample contains the word "like", and the third sample contains the word "hate", and the relevances of the first, second, and third samples to the explanation unit "very attractive" decrease in descending order. When the second preset number is 1, the first sample is selected as the explanation sample; An explanation module, which explains the prediction results of the model by combining the explanation unit in the test phase and the explanation samples in the training phase; The application scenarios of the model behavior explainability method include downstream tasks of neural network model prediction, and the downstream tasks include model behavior analysis, error analysis, adversarial sample generation, adversarial sample defense, and correcting model predictions.

5. The model behavior explainability system according to claim 4, characterized in that In the interpretation unit determination module, the interpretation unit determination module is used to determine the score corresponding to the prediction result, and determine the gradient corresponding to each of the test units based on the score, and then determine the contribution corresponding to each of the test units, wherein the gradient is positively correlated with the contribution.

6. The model behavior explainability system according to claim 4, characterized in that In the explanation sample determination module, the explanation sample determination module is used to calculate the relevance score of each training sample to the explanation unit, and determine the relevance according to the relevance score, wherein the relevance is positively correlated with the relevance score.

7. A computer-readable storage medium, characterized in that The storage medium stores computer instructions, which are operated to execute the model behavior interpretability method described in any one of claims 1-3.

8. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores computer instructions, wherein the processor operates the computer instructions to execute the model behavior interpretability method described in any one of claims 1-3.

Images