Method, device, storage medium and electronic device for explaining model

Abstract

The embodiments of the present specification disclose a method, apparatus, storage medium and electronic device for explaining a model. Text sequence information is input into a recurrent neural network model to be explained. For each time step in the input process of the text sequence information, a linear proxy model corresponding to the time step is constructed according to the behavior of the recurrent neural network model in the prediction process corresponding to the time step. The linear proxy model is trained to obtain a trained linear proxy model. An integrated proxy model is constructed according to the trained linear proxy model corresponding to each time step. According to the output result of the recurrent neural network model on the text sequence information, a feature attribution score corresponding to the text unit at each time step is obtained based on the integrated proxy model. According to the feature attribution score, explanation information for explaining the recurrent neural network model is obtained.

Description

Technical Field

[0001] The present invention relates to computer technology, and in particular to a method, device, storage medium and electronic device for interpreting a model. Background Art

[0002] In the prior art, due to the black box characteristics of the prediction results of deep learning models, it is difficult for users to understand the decision-making mechanism and reasons from the model predictions, and it is difficult to give an intuitive and easy-to-understand explanation for them. Therefore, it is necessary to introduce an interpretable algorithm to explicitly represent the internal logical relationship between the model's prediction results and the corresponding samples.

[0003] Although task data in sequence scenarios can provide richer contextual semantic information to help the model make decisions, it also has the characteristics of high feature dimension, complex data form and content, and because of the time dimension, its actual application business scenarios are complex and changeable, making it difficult to directly apply existing machine learning interpretable methods. Recurrent neural network model is a type of model commonly used in real-world scenarios to process sequence data. It can achieve better prediction results with fewer parameters and relatively simple structure. Therefore, it is necessary to propose an effective interpretable method for the prediction of recurrent neural network model in sequence scenarios to provide explanation for the model's prediction and help understand the decision logic of the model in this scenario. Summary of the invention

[0004] The purpose of the embodiments of this specification is to provide a method, device, storage medium and electronic device for interpreting a model.

[0005] The embodiments of this specification provide a method for explaining a model. A special interpretable method is obtained by characterizing the unique structure and prediction mechanism of a deep learning model such as a recurrent neural network model, which is widely used in various business scenarios. The method integrates the unique structural characteristics of the recurrent neural network model into the design of the interpretable method by constructing an integrated proxy model, and explicitly uses the prediction mechanism inside the recurrent neural network in the modeling of the interpretable method. Compared with the existing general interpretable methods, it can better help users understand the prediction mechanism of such models, and while calculating the attribution score of the input feature, it can also consider the contribution and influence of the hidden layer state, the core element in the recurrent neural network model, in the model prediction, thereby providing a more intuitive and reasonable feature attribution explanation, and can produce an attribution explanation that is more in line with the user's way of thinking, helping users better understand the prediction mechanism of the recurrent neural network model. The method includes:

[0006] Input text sequence information into the recurrent neural network model to be interpreted, for each time step in the input process of the text sequence information, construct a linear proxy model corresponding to the time step according to the behavior of the recurrent neural network model in the prediction process corresponding to the time step, train the linear proxy model, and obtain a trained linear proxy model, wherein the input of the linear proxy model is the text unit corresponding to the text sequence information at the time step and the current hidden layer state vector of the recurrent neural network model after the previous time step of the time step, and the output of the linear proxy model is the predicted hidden layer state vector of the recurrent neural network model after the time step;

[0007] Constructing an integrated proxy model according to the trained linear proxy model corresponding to each time step;

[0008] According to the output result of the recurrent neural network model on the text sequence information, based on the integrated agent model, the feature attribution score corresponding to the text unit of each time step is obtained, and according to the feature attribution score, the explanation information for explaining the recurrent neural network model is obtained.

[0009] Furthermore, the training of the linear proxy model to obtain a trained linear proxy model includes:

[0010] Constructing training samples according to the input features of the linear proxy model;

[0011] The linear proxy model is trained according to the training samples to obtain a trained linear proxy model.

[0012] Furthermore, constructing training samples according to the input features of the linear proxy model includes:

[0013] A training sample is obtained by applying random perturbations to the input features of the linear proxy model.

[0014] Furthermore, the training samples are distributed in a space adjacent to the input features of the linear proxy model.

[0015] Furthermore, the training of the linear proxy model according to the training samples to obtain the trained linear proxy model includes:

[0016] According to the training samples and the current hidden layer state vector of the recurrent neural network model after the time step, the linear proxy model is fitted based on a linear regression algorithm to obtain a trained linear proxy model.

[0017] Furthermore, constructing an integrated proxy model according to the trained linear proxy model corresponding to each time step includes:

[0018] The trained linear proxy models corresponding to each time step are sequentially concatenated head to tail in chronological order to obtain an integrated proxy model.

[0019] Furthermore, the step of obtaining the feature attribution score corresponding to the text unit at each time step based on the output result of the recurrent neural network model on the text sequence information and based on the integrated proxy model includes:

[0020] The output result of the recurrent neural network model on the text sequence information is used as the initial feature attribution score, and the initial feature attribution score is back-propagated to the integrated proxy model to obtain the feature attribution score corresponding to the text unit at each time step.

[0021] Furthermore, back-propagating the initial feature attribution score to the integrated proxy model to obtain the feature attribution score corresponding to the text unit at each time step includes:

[0022] The initial feature attribution score is back-propagated to the integrated proxy model, so that the initial feature attribution score is back-propagated to the last linear proxy model in the integrated proxy model, and the feature attribution score corresponding to the text unit at the time step corresponding to the last linear proxy model is obtained, so that the feature attribution score is further back-propagated to the previous linear proxy model corresponding to the last linear proxy model, and so on, until the feature attribution score corresponding to the text unit at the time step corresponding to the first linear proxy model in the integrated proxy model is obtained.

[0023] The embodiment of this specification also provides a device for interpreting a model, comprising:

[0024] A proxy model training module, used for inputting text sequence information into a recurrent neural network model to be interpreted, for each time step in the input process of the text sequence information, constructing a linear proxy model corresponding to the time step according to the behavior of the recurrent neural network model in the prediction process corresponding to the time step, training the linear proxy model, and obtaining a trained linear proxy model, wherein the input of the linear proxy model is the text unit corresponding to the text sequence information at the time step and the current hidden layer state vector of the recurrent neural network model after the previous time step corresponding to the time step, and the output of the linear proxy model is the predicted hidden layer state vector of the recurrent neural network model after the time step;

[0025] An integrated proxy model construction module, used to construct an integrated proxy model according to the trained linear proxy model corresponding to each time step;

[0026] A model interpretation module is used to obtain feature attribution scores corresponding to text units at each time step based on the output results of the recurrent neural network model on the text sequence information and based on the integrated agent model, obtain interpretation information for interpreting the recurrent neural network model based on the feature attribution scores.

[0027] The embodiments of the present specification also provide a computer program product, wherein the computer program product stores at least one instruction, and the at least one instruction is suitable for being loaded by a processor and executing the above method steps.

[0028] The embodiments of the present specification also provide a storage medium, wherein the storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the steps of the above method.

[0029] An embodiment of the present specification also provides an electronic device, comprising: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the steps of the above method.

[0030] In the embodiments of the present specification, a special interpretable method is obtained by characterizing the unique structure and prediction mechanism of deep learning models such as recurrent neural network models that are widely used in various business scenarios. The method integrates the unique structural characteristics of the recurrent neural network model into the design of the interpretable method by constructing an integrated proxy model, and explicitly uses the prediction mechanism inside the recurrent neural network in the modeling of the interpretable method. Compared with the existing general interpretable methods, it can better help users understand the prediction mechanism of such models, and while calculating the attribution score for the input features, it can also consider the contribution and influence of the hidden layer state, the core element in the recurrent neural network model, in the model prediction, thereby providing a more intuitive and reasonable feature attribution explanation, and can produce an attribution explanation that is more in line with the user's way of thinking, helping users better understand the prediction mechanism of the recurrent neural network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A flowchart of a method for explaining a model provided in an embodiment of this specification;

[0032] Figure 2 A schematic diagram of the structure of an integrated proxy model provided as an example in the embodiments of this specification;

[0033] Figure 3 A flowchart of a method for back propagation of an integrated proxy model provided as an example in an embodiment of this specification;

[0034] Figure 4A schematic diagram of a structure of a device for explaining a model provided in an embodiment of this specification;

[0035] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0037] See also Figure 1 , is a flow chart of a method for interpreting a model provided in an embodiment of this specification. In an embodiment of this specification, the method for interpreting a model is applied to a device for interpreting a model (hereinafter referred to as a "model interpretation device") or an electronic device equipped with a model interpretation device. Figure 1 The process shown is described in detail, and the method for explaining the model may specifically include the following steps:

[0038] S102, inputting text sequence information into the recurrent neural network model to be interpreted, for each time step in the input process of the text sequence information, constructing a linear proxy model corresponding to the time step according to the behavior of the recurrent neural network model in the prediction process corresponding to the time step, training the linear proxy model to obtain a trained linear proxy model, wherein the input of the linear proxy model is the text unit corresponding to the text sequence information at the time step and the current hidden layer state vector of the recurrent neural network model after the previous time step of the time step, and the output of the linear proxy model is the predicted hidden layer state vector of the recurrent neural network model after the time step.

[0039] In some embodiments, the text sequence information includes a plurality of text units (tokens) arranged in a front-to-back manner. In some embodiments, the recurrent neural network model is a type of deep learning model commonly used in processing sequence data. By updating the vector called the hidden state inside the model, the memory and forgetting of the information in the sequence data are realized. The hidden state vector is a core element inside the recurrent neural network model. When the model processes the input data, it updates the state vector step by step in time to obtain the information in the input features and determine which part of the information to remember or forget. In some embodiments, the recurrent neural network model in this example embodiment can be any recurrent neural network model used to predict text tasks, including but not limited to sentiment analysis tasks, news classification tasks, etc., and this example embodiment does not specifically limit this.

[0040] In some embodiments, in the process of inputting text sequence information into the recurrent neural network model to be interpreted, the text units in the text sequence information will be sequentially input into the recurrent neural network model according to the time step (i.e., time interval) based on the arrangement order in the text sequence information, that is, in each time step, there will be at least one text unit in the text sequence information input into the recurrent neural network model, for example, in the first time step, the first N text units in the text sequence information are input into the recurrent neural network model, in the second time step, the M text units in the text sequence information that are arranged after the N text units are input into the recurrent neural network model, in the third time step, the P text units in the text sequence information that are arranged after the M text units are input into the recurrent neural network model, and so on, until all the text units in the text sequence information have been input into the recurrent neural network model. In some embodiments, each time step corresponds to the same time interval, and the number or length of the text units input into the recurrent neural network model in each time step may be random and not fixed, or may be preset and fixed, and this exemplary embodiment does not make any special limitation on this.

[0041] In some embodiments, for each time step in the input process of the text sequence information, a linear proxy model corresponding to the time step is constructed according to the behavior of the recurrent neural network model in the prediction process corresponding to the time step, and the behavior data includes but is not limited to any behavior performed by the recurrent neural network model in the time step (for example, internal calculation logic, etc.), which is not specifically limited in this exemplary embodiment, or, the linear proxy model corresponding to the time step is constructed according to the behavior of the recurrent neural network model in the prediction process corresponding to the time step and the text unit corresponding to the text sequence information in the time step (that is, the text unit in the text sequence information that is input into the recurrent neural network model in the time step). In some embodiments, each time step corresponds to a linear proxy model, and for each time step, the linear proxy model input is the text unit corresponding to the text sequence information in the time step and the current hidden state vector of the recurrent neural network model after the previous time step of the time step (the recurrent neural network will update the hidden state vector time by time step), and the output of the linear proxy model is the predicted hidden state vector of the recurrent neural network model after the time step, that is, the linear proxy model is used to predict the latest hidden state vector of the recurrent neural network model after the update of the time step. In some embodiments, the linear proxy model corresponding to each time step is trained to obtain a trained linear proxy model. It should be noted that those skilled in the art should understand that any training method can be included in the protection scope of this specification, and this specification does not limit this.

[0042] S104, constructing an integrated proxy model according to the trained linear proxy model corresponding to each time step. In some embodiments, the trained linear proxy models corresponding to each time step can be fused (integrated) into a proxy model by a preset model fusion method. It should be noted that those skilled in the art should understand that any model fusion method can be included in the protection scope of this specification, and this specification does not limit this. In some embodiments, the trained linear proxy models corresponding to each time step can also be combined according to certain rules to construct an integrated proxy model. It should be noted that those skilled in the art should understand that any combination rule can be included in the protection scope of this specification, and this specification does not limit this.

[0043] S106, according to the output result of the recurrent neural network model on the text sequence information, based on the integrated agent model, obtain the feature attribution score corresponding to the text unit of each time step, and according to the feature attribution score, obtain the explanation information for explaining the recurrent neural network model.

[0044] In some embodiments, the output result (i.e., prediction result) of the recurrent neural network model on the text sequence information is obtained, and based on the output result and the integrated proxy model obtained above, the feature attribution score corresponding to the text unit at each time step is obtained, wherein the feature attribution score corresponding to each time step is used to characterize the "importance" of the text unit corresponding to the time step (i.e., the text unit input into the recurrent neural network model in the time step) for the cycle, the prediction result output by the neural network model. The feature attribution score is used to represent the influence of different input features on the model prediction by explicitly calculating the causal relationship between the input features (i.e., the text units input into the recurrent neural network model in each time step) and the prediction result of the recurrent neural network model.

[0045] In some embodiments, the specific method of obtaining the feature attribution score corresponding to each time step based on the integrated proxy model may be to input the output result (i.e., prediction result) of the recurrent neural network model on the text sequence information into the integrated proxy model to obtain the feature attribution score corresponding to each time step output by the integrated proxy model, or the prediction result may be back-propagated in the integrated proxy model, thereby obtaining the feature attribution score corresponding to each time step in the back-propagation process. In some embodiments, the explanation information for explaining the recurrent neural network model may be determined based on the feature attribution scores corresponding to each time step, for example, the feature attribution scores corresponding to each time step may be concatenated as the explanation information for explaining the recurrent neural network model, or the explanation information for explaining the recurrent neural network model may be determined based on the feature attribution scores corresponding to each time step, as well as the text sequence information and / or the prediction result of the recurrent neural network model on the text sequence information. In some embodiments, the explanation information may be used to provide an explanation for the prediction result of the recurrent neural network model, to help users understand the principle and mechanism of the decision-making of the recurrent neural network model, and to make its prediction result transparent and explainable.

[0046] This example embodiment is a special interpretable method that is characterized by design for the unique structure and prediction mechanism of deep learning models such as recurrent neural network models that are widely used in various business scenarios. This method integrates the unique structural characteristics of the recurrent neural network model into the design of the interpretable method by constructing an integrated proxy model, and explicitly uses the prediction mechanism inside the recurrent neural network in the modeling of the interpretable method. Compared with the existing general interpretable methods, it can better help users understand the prediction mechanism of such models, and while calculating the attribution score for the input features, it can also consider the contribution and influence of the hidden layer state, the core element in the recurrent neural network model, in the model prediction, thereby providing a more intuitive and reasonable feature attribution explanation, and can produce an attribution explanation that is more in line with the user's way of thinking, helping users better understand the prediction mechanism of the recurrent neural network model.

[0047] In some embodiments, the training of the linear proxy model to obtain the trained linear proxy model includes: constructing training samples according to the input features of the linear proxy model; and training the linear proxy model according to the training samples to obtain the trained linear proxy model. In some embodiments, for the linear proxy model corresponding to each time step, multiple training samples corresponding to the linear proxy model are constructed according to the input features of the linear proxy model, i.e., the text unit corresponding to the time step (the text unit input to the recurrent neural network model in the time step), and the linear proxy model is trained by the multiple training samples to obtain the trained linear proxy model. For example, a preset data enhancement method can be used to perform data enhancement processing on the text unit, obtain multiple enhanced text units corresponding to the text unit and use them as training samples corresponding to the linear proxy model. It should be noted that those skilled in the art should understand that any data enhancement method can be included in the protection scope of this specification, and this specification does not limit this.

[0048] In some embodiments, constructing training samples according to the input features of the linear proxy model includes: obtaining training samples by applying random perturbations to the input features of the linear proxy model. In some embodiments, by referring to the LIME interpretable method, for the linear proxy model corresponding to each time step, random perturbations are applied to the input features of the linear proxy model, i.e., the text units corresponding to the time step (the text units input to the recurrent neural network model in the time step), to obtain multiple perturbation samples as training samples of the linear proxy model.

[0049] In some embodiments, the training samples are distributed in the adjacent space of the input features of the linear proxy model. In some embodiments, the perturbation samples obtained by applying random perturbations to the input features of the linear proxy model are distributed in the adjacent space of the input features of the linear proxy model, so that the linear proxy model can be helped to learn the prediction logic of the recurrent neural network model at the current time step corresponding to the linear proxy model.

[0050] In some embodiments, the training of the linear proxy model according to the training sample to obtain the trained linear proxy model includes: fitting the linear proxy model based on the linear regression algorithm according to the training sample and the current hidden state vector of the recurrent neural network model after the time step to obtain the trained linear proxy model. In some embodiments, the latest actual hidden state vector of the recurrent neural network model after the time step is obtained, the actual hidden state vector is used as the label information corresponding to the training sample, and the linear proxy model is fitted based on the linear regression algorithm according to the training sample and its corresponding label information (i.e., regression training is performed on the linear proxy model) to obtain the trained linear proxy model.

[0051] In some embodiments, the integrated proxy model is constructed based on the trained linear proxy models corresponding to each time step, including: splicing the trained linear proxy models corresponding to each time step in sequence from beginning to end in chronological order to obtain an integrated proxy model. In some embodiments, the trained linear proxy models corresponding to the respective time steps are spliced ​​from beginning to end in chronological order corresponding to the time steps to obtain a spliced ​​integrated proxy model, the output of the previous linear proxy model is the output of the next linear proxy model, the input of the first linear proxy model is the output of the integrated proxy model, and the output of the last linear proxy model is the output of the integrated proxy model, that is, the output of the trained linear proxy model corresponding to a certain time step in chronological order is the input of the trained linear proxy model corresponding to the next time step corresponding to the time step, and the input of the trained linear proxy model corresponding to a certain time step in chronological order is the output of the trained linear proxy model corresponding to the previous time step corresponding to the time step. As an example, Figure 2 As shown in the figure, the input of agent model 1 corresponding to the first time step is the input of the integrated agent model, the output of agent model 1 is the input of agent model 2 corresponding to the second time step, and so on. The output of the agent model corresponding to the previous time step is the input of the agent model corresponding to the next time step, and the output of the Nth time step (i.e., the last time step) is the output of the integrated agent model.

[0052] In some embodiments, the output result of the recurrent neural network model on the text sequence information is used as the initial feature attribution score, and the initial feature attribution score is back-propagated to the integrated proxy model to obtain the feature attribution score corresponding to the text unit at each time step based on the output result of the recurrent neural network model on the text sequence information. In some embodiments, the output result (i.e., the prediction result) of the recurrent neural network model on the text sequence information is used as the initial feature attribution score, and the initial feature attribution score is back-propagated in the integrated proxy model, that is, the initial feature attribution score is passed to each linear proxy model in the integrated proxy model in accordance with the back-propagation rule, and the feature attribution score corresponding to each time step is obtained in the back-propagation process.

[0053] In some embodiments, back-propagating the initial feature attribution score to the integrated proxy model to obtain the feature attribution score corresponding to the text unit at each time step includes: back-propagating the initial feature attribution score to the integrated proxy model so that the initial feature attribution score is back-propagated to the last linear proxy model in the integrated proxy model, obtaining the feature attribution score corresponding to the text unit at the time step corresponding to the last linear proxy model, and continuing to back-propagate the feature attribution score to the previous linear proxy model corresponding to the last linear proxy model, and so on, until the feature attribution score corresponding to the text unit at the time step corresponding to the first linear proxy model in the integrated proxy model is obtained. In some embodiments, the initial feature attribution score is back-propagated to the integrated proxy model, so that the initial feature attribution score is back-propagated to the last linear proxy model in the integrated proxy model, and during the back-propagation process of the initial feature attribution score in the last linear proxy model, the feature attribution score of the time step corresponding to the last linear proxy model is obtained, and the feature attribution score is continued to be back-propagated to the previous linear proxy model corresponding to the last linear proxy model in the integrated proxy model, and during the back-propagation process of the feature attribution score in the previous linear proxy model, a new feature attribution score of the time step corresponding to the previous linear proxy model is obtained, and the new feature attribution score is continued to be back-propagated, and so on, until the feature attribution score of the time step corresponding to the first linear proxy model in the integrated proxy model is obtained, so as to gradually transfer the feature attribution score from back to front in each linear proxy model within the integrated proxy model according to the back-propagation rule. As an example, Figure 3As shown, the initial feature attribution score is back-propagated to the integrated proxy model, so that the initial feature attribution score is back-propagated to the proxy model N corresponding to the Nth time step (i.e., the last time step) in the integrated proxy model, and the feature attribution score N corresponding to the Nth time step is obtained in the back-propagation process, and the feature attribution score N is further back-propagated to the proxy model N-1 corresponding to the N-1th time step (i.e., the second to last time step) in the integrated proxy model, and the feature attribution score N-1 corresponding to the N-1th time step is obtained in the back-propagation process, and so on, until the feature attribution score 1 corresponding to the first time step in the integrated proxy model is obtained in the back-propagation process of the proxy model 1 corresponding to the first time step.

[0054] Figure 4 A schematic diagram of a device for interpreting a model provided in an embodiment of the specification, the device for interpreting a model (hereinafter referred to as "model interpretation device 1") can be implemented as all or part of an electronic device through software, hardware or a combination of both. According to some embodiments, the model interpretation device 1 includes an agent model training module 11, an integrated agent model construction module 12 and a model interpretation module 13.

[0055] The proxy model training module 11 is used to input the text sequence information into the recurrent neural network model to be interpreted, and for each time step in the input process of the text sequence information, a linear proxy model corresponding to the time step is constructed according to the behavior of the recurrent neural network model in the prediction process corresponding to the time step, and the linear proxy model is trained to obtain a trained linear proxy model, wherein the input of the linear proxy model is the text unit corresponding to the text sequence information at the time step and the current hidden layer state vector of the recurrent neural network model after the previous time step corresponding to the time step, and the output of the linear proxy model is the predicted hidden layer state vector of the recurrent neural network model after the time step;

[0056] An integrated proxy model construction module 12 is used to construct an integrated proxy model according to the trained linear proxy model corresponding to each time step;

[0057] The model interpretation module 13 is used to obtain the feature attribution scores corresponding to the text units at each time step based on the output results of the recurrent neural network model on the text sequence information and based on the integrated agent model, obtain the interpretation information for explaining the recurrent neural network model according to the feature attribution scores.

[0058] In some embodiments, the step of training the linear proxy model to obtain a trained linear proxy model includes:

[0059] Constructing training samples according to the input features of the linear proxy model;

[0060] The linear proxy model is trained according to the training samples to obtain a trained linear proxy model.

[0061] In some embodiments, constructing a training sample according to the input features of the linear proxy model includes:

[0062] A training sample is obtained by applying random perturbations to the input features of the linear proxy model.

[0063] In some embodiments, the training samples are distributed in a neighboring space of input features of the linear proxy model.

[0064] In some embodiments, the step of training the linear proxy model according to the training samples to obtain the trained linear proxy model includes:

[0065] According to the training samples and the current hidden layer state vector of the recurrent neural network model after the time step, the linear proxy model is fitted based on a linear regression algorithm to obtain a trained linear proxy model.

[0066] In some embodiments, constructing the integrated proxy model according to the trained linear proxy model corresponding to each time step includes:

[0067] The trained linear proxy models corresponding to each time step are sequentially concatenated head to tail in chronological order to obtain an integrated proxy model.

[0068] In some embodiments, obtaining the feature attribution score corresponding to the text unit at each time step based on the output result of the recurrent neural network model on the text sequence information and based on the integrated proxy model includes:

[0069] The output result of the recurrent neural network model on the text sequence information is used as the initial feature attribution score, and the initial feature attribution score is back-propagated to the integrated proxy model to obtain the feature attribution score corresponding to the text unit at each time step.

[0070] In some embodiments, back-propagating the initial feature attribution score to the integrated proxy model to obtain the feature attribution score corresponding to the text unit at each time step includes:

[0071] The initial feature attribution score is back-propagated to the integrated proxy model, so that the initial feature attribution score is back-propagated to the last linear proxy model in the integrated proxy model, and the feature attribution score corresponding to the text unit at the time step corresponding to the last linear proxy model is obtained, so that the feature attribution score is further back-propagated to the previous linear proxy model corresponding to the last linear proxy model, and so on, until the feature attribution score corresponding to the text unit at the time step corresponding to the first linear proxy model in the integrated proxy model is obtained.

[0072] The above device embodiments correspond to the method embodiments. For specific descriptions, please refer to the description of the method embodiments, which will not be repeated here. The device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. For specific descriptions, please refer to the corresponding method embodiments.

[0073] The embodiment of the present specification also provides a computer storage medium, which can store multiple instructions, and the instructions are suitable for being loaded by a processor to execute the method of the embodiment of the present specification.

[0074] The embodiments of the present specification also provide a computer program product, which stores at least one instruction, and the at least one instruction is loaded by the processor to execute the method of the embodiments of the present specification.

[0075] The embodiments of this specification also provide Figure 5 The structural diagram of the electronic device shown in FIG. Figure 5 At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the above-mentioned voice activity detection method.

[0076] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the executor of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0077] In the 1990s, it was very clear whether the improvement of a technology was hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with hardware entity modules. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0078] The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, and the memory controller may also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that, in addition to implementing the controller in a purely computer-readable program code manner, the controller may be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller may be considered as a hardware component, and the devices for implementing various functions included therein may also be considered as structures within the hardware component. Or even, the devices for implementing various functions may be considered as both software modules for implementing the method and structures within the hardware component.

[0079] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0080] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0081] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0082] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0083] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0084] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0085] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0086] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0087] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0088] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0089] It should be understood by those skilled in the art that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0090] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0091] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0092] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A method for explaining a model, comprising: Input text sequence information into the recurrent neural network model to be interpreted, for each time step in the input process of the text sequence information, construct a linear proxy model corresponding to the time step according to the behavior of the recurrent neural network model in the prediction process corresponding to the time step, train the linear proxy model, and obtain a trained linear proxy model, wherein the input of the linear proxy model is the text unit corresponding to the text sequence information at the time step and the current hidden layer state vector of the recurrent neural network model after the previous time step of the time step, and the output of the linear proxy model is the predicted hidden layer state vector of the recurrent neural network model after the time step; Constructing an integrated proxy model according to the trained linear proxy model corresponding to each time step; According to the output result of the recurrent neural network model on the text sequence information, based on the integrated agent model, a feature attribution score corresponding to the text unit of each time step is obtained, and according to the feature attribution score, explanation information for explaining the recurrent neural network model is obtained; Wherein, according to the output result of the recurrent neural network model on the text sequence information, based on the integrated proxy model, the feature attribution score corresponding to the text unit at each time step is obtained, including: taking the output result of the recurrent neural network model on the text sequence information as the initial feature attribution score, back-propagating the initial feature attribution score to the integrated proxy model, and obtaining the feature attribution score corresponding to the text unit at each time step.

2. According to the method of claim 1, the training of the linear proxy model to obtain the trained linear proxy model comprises: Constructing training samples according to the input features of the linear proxy model; The linear proxy model is trained according to the training samples to obtain a trained linear proxy model.

3. The method according to claim 2, wherein constructing a training sample according to the input features of the linear proxy model comprises: A training sample is obtained by applying random perturbations to the input features of the linear proxy model.

4. According to the method of claim 3, the training samples are distributed in the adjacent space of the input features of the linear proxy model.

5. The method according to claim 4, wherein the step of training the linear proxy model according to the training samples to obtain the trained linear proxy model comprises: According to the training samples and the current hidden layer state vector of the recurrent neural network model after the time step, the linear proxy model is fitted based on a linear regression algorithm to obtain a trained linear proxy model.

6. The method according to claim 1, wherein constructing an integrated proxy model according to the trained linear proxy model corresponding to each time step comprises: The trained linear proxy models corresponding to each time step are sequentially concatenated head to tail in chronological order to obtain an integrated proxy model.

7. According to the method of claim 1, the back-propagating the initial feature attribution score to the integrated proxy model to obtain the feature attribution score corresponding to the text unit at each time step comprises: The initial feature attribution score is back-propagated to the integrated proxy model, so that the initial feature attribution score is back-propagated to the last linear proxy model in the integrated proxy model, and the feature attribution score corresponding to the text unit at the time step corresponding to the last linear proxy model is obtained, so that the feature attribution score is further back-propagated to the previous linear proxy model corresponding to the last linear proxy model, and so on, until the feature attribution score corresponding to the text unit at the time step corresponding to the first linear proxy model in the integrated proxy model is obtained.

8. An apparatus for interpreting a model, comprising: A proxy model training module, used for inputting text sequence information into a recurrent neural network model to be interpreted, for each time step in the input process of the text sequence information, constructing a linear proxy model corresponding to the time step according to the behavior of the recurrent neural network model in the prediction process corresponding to the time step, training the linear proxy model, and obtaining a trained linear proxy model, wherein the input of the linear proxy model is the text unit corresponding to the text sequence information at the time step and the current hidden layer state vector of the recurrent neural network model after the previous time step corresponding to the time step, and the output of the linear proxy model is the predicted hidden layer state vector of the recurrent neural network model after the time step; An integrated proxy model construction module, used to construct an integrated proxy model according to the trained linear proxy model corresponding to each time step; A model interpretation module is used to obtain, based on the integrated proxy model and according to the output results of the recurrent neural network model on the text sequence information, feature attribution scores corresponding to the text units at each time step, and obtain interpretation information for interpreting the recurrent neural network model based on the feature attribution scores; wherein the obtaining, based on the integrated proxy model and according to the output results of the recurrent neural network model on the text sequence information, feature attribution scores corresponding to the text units at each time step comprises: using the output results of the recurrent neural network model on the text sequence information as initial feature attribution scores, back-propagating the initial feature attribution scores to the integrated proxy model, and obtaining feature attribution scores corresponding to the text units at each time step.

9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: include: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the steps of the method as claimed in any one of claims 1 to 7.

11. A computer program product having at least one instruction stored thereon, characterized in that: When the at least one instruction is executed by the processor, the steps of the method described in any one of claims 1 to 7 are implemented.

Images