Model interpretation method, device, electronic device and storage medium

Abstract

The embodiments of the present application provide a model interpretation method, device, electronic device and storage medium. The method includes: obtaining a pre-trained black box model, a sample set of the target task scenario and a target sample to be interpreted; inputting the sample set into the black box model to obtain feature distribution information; inputting the target sample into the black box model for model prediction to obtain a model prediction score; traversing the features of each dimension in the target sample; calculating the weighted average score corresponding to the feature of each dimension according to the feature distribution information; determining the importance of the feature of each dimension in the target sample according to the difference between the weighted average score and the model prediction score; and interpreting the model of the black box model based on the target sample according to the importance. The present application can help engineers interpret the behavior of the model in a specific sample to perform feature significance analysis.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model interpretation method, device, electronic device and storage medium. Background Art

[0002] In risk control scenarios, especially when model predictions are used for customer behavior interception and warnings, erroneous interception and warning signals not only fail to control risk but can significantly impact the customer experience. Therefore, risk control scenarios place high demands on model interpretability. Therefore, engineers in risk control scenarios often need to explain their models. If they can't, the model's effectiveness is easily questioned by frontline business. This leads engineers to choose models with high interpretability.

[0003] In related technologies, there's often a conflict between model interpretability and performance. Models with high interpretability often perform poorly, such as logistic regression and single decision trees, which offer good interpretability but poor performance. However, some effective models, such as neural networks and GBDT (an ensemble of multiple decision trees), struggle to explain the behavior of individual samples. Faced with this conflict between interpretability and performance, risk control engineers often settle for the second best option, choosing models with high interpretability but poor performance. Therefore, helping engineers explain the model's behavior on specific samples for feature significance analysis has become a pressing technical challenge in this field. Summary of the Invention

[0004] The main purpose of the embodiments of the present application is to propose a model interpretation method, device, electronic device and storage medium that can help engineers interpret the behavior of the model in specific samples to perform feature significance analysis.

[0005] To achieve the above objectives, a first aspect of an embodiment of the present application provides a model interpretation method, the method comprising:

[0006] Obtain a pre-trained black box model, a sample set of the target task scenario, and the target sample to be explained;

[0007] Inputting the sample set into the black box model to obtain feature distribution information;

[0008] Input the target sample into the black box model to perform model prediction and obtain a model prediction score;

[0009] Traversing the features of each dimension in the target sample, and calculating the weighted average score corresponding to the features of each dimension according to the feature distribution information;

[0010] Determine the importance of the feature in each dimension to the target sample based on the difference between the weighted average score and the model prediction score;

[0011] The model of the black box model based on the target sample is explained according to the importance of the features.

[0012] In some embodiments, the feature distribution information includes the feature value corresponding to each dimension of the feature in the sample set and the proportion of the feature value in the sample set.

[0013] In some embodiments, calculating the weighted average score corresponding to the feature in each dimension based on the feature distribution information includes:

[0014] Replacing the feature of each dimension in the target sample with the feature value corresponding to the feature of the same dimension in the sample set to obtain a new sample;

[0015] Performing a model score on the new sample using the black box model to obtain a model score;

[0016] According to the model scoring score and the proportion of the feature value in the sample set, the weighted average score corresponding to the feature in each dimension of the target sample is obtained.

[0017] In some embodiments, determining the importance of the feature in each dimension in the target sample based on the difference between the weighted average score and the model prediction score includes:

[0018] Calculate the difference between the weighted average score corresponding to the feature in each dimension and the model prediction score to obtain a first difference value;

[0019] Calculate the difference between the weighted average score corresponding to the features of all dimensions and the model prediction score to obtain a second difference value;

[0020] A first feature importance value is obtained by dividing the first difference value by the second difference value, wherein the first feature importance value is used to characterize the importance of the feature of each dimension in the target sample, and the size of the first feature importance value is proportional to the importance.

[0021] In some embodiments, explaining the model of the black box model based on the target sample according to the importance level includes:

[0022] sorting the features according to the importance values ​​of the first features;

[0023] Determine the top N features with the highest importance values ​​of the first feature;

[0024] The first N features are used to locally interpret the model of the black box model based on the target sample, where N is a positive integer greater than zero.

[0025] In some embodiments, the step of explaining the black box model based on the target sample according to the importance level includes:

[0026] Calculating the first feature importance value corresponding to the feature of each dimension of each sample in the sample set;

[0027] averaging the sum of the first feature importance values ​​corresponding to the features of each dimension of each sample to obtain a second feature importance value, wherein the second feature importance value is used to represent the importance of the feature of each dimension in the sample set, and the magnitude of the second feature importance value is proportional to the importance;

[0028] sorting the features of each dimension in the sample set according to the importance value of the second feature;

[0029] Determine the features of the top N dimensions with the highest importance values ​​of the second feature;

[0030] The features of the first N dimensions are used to globally interpret the model of the black box model based on the target sample, where N is a positive integer greater than zero.

[0031] In some embodiments, the black box model is a classification model.

[0032] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a model interpretation device, comprising:

[0033] The acquisition module is used to obtain the pre-trained black box model, the sample set of the target task scenario, and the target sample to be explained;

[0034] A feature analysis module, configured to input the sample set into the black box model to obtain feature distribution information;

[0035] A prediction module is used to input the target sample into the black box model to perform model prediction and obtain a model prediction score;

[0036] A calculation module is used to traverse the features of each dimension in the target sample and calculate the weighted average score corresponding to the features of each dimension according to the feature distribution information;

[0037] An analysis module is configured to determine the importance of the feature in each dimension in the target sample based on the difference between the weighted average score and the model prediction score;

[0038] An explanation module is used to explain the model of the black box model based on the target sample according to the importance of the feature.

[0039] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory, a processor, a program stored on the memory and runnable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the method described in the first aspect above is implemented.

[0040] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method described in the first aspect above.

[0041] The model interpretation method, device, electronic device and storage medium proposed in the present application obtain a pre-trained black box model, a sample set of the target task scenario and a target sample to be interpreted; input the sample set into the black box model to obtain feature distribution information; input the target sample into the black box model for model prediction to obtain a model prediction score; traverse the features of each dimension in the target sample, and calculate the weighted average score corresponding to each dimension feature according to the feature distribution information; determine the importance of each dimension feature in the target sample according to the difference between the weighted average score and the model prediction score; interpret the black box model based on the target sample according to the importance of the feature. The model interpretation method of the embodiment of the present application does not need to care about the model implementation details, and regards the model as an invisible black box. By adjusting the model input and observing the changes in the model output to analyze the importance of a certain feature to the model, it helps engineers explain the behavior of the model in specific samples to perform feature significance analysis, and can explain the behavior of the model in a single case and the overall behavior of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a schematic diagram of a system architecture platform for executing a model interpretation method provided by one embodiment of the present application;

[0043] Figure 2 This is a flow chart of a model interpretation method provided by one embodiment of the present application;

[0044] Figure 3 yes Figure 2 A sub-flowchart of the specific method of step S400 in FIG.

[0045] Figure 4 yes Figure 2 A sub-flowchart of the specific method of step S500 in FIG.

[0046] Figure 5 yes Figure 2 A sub-flowchart of the specific method of step S600 in FIG.

[0047] Figure 6 yes Figure 2 Another sub-flowchart of the specific method of step S600;

[0048] Figure 7 is a schematic diagram of a model interpretation device provided by an embodiment of the present application; DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0050] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0052] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0053] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0054] Deep learning (DL) is a new research direction within the field of machine learning (ML). It was introduced to bring ML closer to its original goal: artificial intelligence (AI). Deep learning involves learning the inherent patterns and representational hierarchies of sample data. The information gained from this learning process significantly aids in interpreting data such as text, images, and sound. Its ultimate goal is to enable machines to acquire human-like analytical and learning capabilities, enabling them to recognize data such as text, images, and sound. Deep learning is a complex machine learning algorithm that has achieved results in speech and image recognition that far surpass previous technologies. Deep learning has also achieved significant success in search technology, data mining, machine learning, machine translation, natural language processing, multimedia learning, speech recognition, recommendation and personalization technologies, and other related fields. Deep learning enables machines to mimic human activities such as seeing, hearing, and thinking, solving many complex pattern recognition challenges and significantly advancing AI-related technologies.

[0055] In risk control scenarios, especially when model predictions are used for customer behavior interception and warnings, erroneous interception and warning signals not only fail to control risk but can significantly impact the customer experience. Therefore, risk control scenarios place high demands on model interpretability. Therefore, engineers in risk control scenarios often need to explain their models. If they can't, the model's effectiveness is easily questioned by frontline business. This leads engineers to choose models with high interpretability.

[0056] In related technologies, there's often a conflict between model interpretability and performance. Models with high interpretability often perform poorly, such as logistic regression and single decision trees, which offer good interpretability but poor performance. However, some effective models, such as neural networks and GBDT (an ensemble of multiple decision trees), struggle to explain the behavior of individual samples. Faced with this conflict between interpretability and performance, risk control engineers often settle for the second best option, choosing models with high interpretability but poor performance. Therefore, helping engineers explain the model's behavior on specific samples for feature significance analysis has become a pressing technical challenge in this field.

[0057] To solve the above technical problems, an embodiment of the present invention provides a model interpretation method, device, electronic device and storage medium, which obtains a pre-trained black box model, a sample set of the target task scenario and a target sample to be interpreted; inputs the sample set into the black box model to obtain feature distribution information; inputs the target sample into the black box model for model prediction to obtain a model prediction score; traverses the features of each dimension in the target sample, and calculates the weighted average score corresponding to each dimension feature according to the feature distribution information; determines the importance of each dimension feature in the target sample according to the difference between the weighted average score and the model prediction score; interprets the black box model based on the target sample according to the importance of the feature. The model interpretation method of the embodiment of the present application does not need to care about the model implementation details, and regards the model as an invisible black box. By adjusting the model input and observing the changes in the model output, the importance of a certain feature to the model is analyzed, thereby helping engineers to explain the behavior of the model in a specific sample to perform feature significance analysis, and can explain the behavior of the model in a single case and the overall behavior of the model.

[0058] First, let’s analyze some of the terms used in this application:

[0059] 1. Black box model: For a model, its internal structure is unclear, but the input can be given and the black box model returns the output result.

[0060] Input: Tabular data is a common data type in risk control scenarios, distinct from images, text, and other data. In tabular data, a row is a sample, and a column is a feature. Features are categorized into numerical features and discrete features.

[0061] Output: Taking the classification model as an example, the model will output the probability distribution of each classification label:

[0062] P=[P0,P1,……,P C-1 ]

[0063] There are B categories in total, P i is the probability distribution of the i-th category label predicted by the model, where P i ≤1 and

[0064] 2. Explainability:

[0065] Generally, model interpretability is divided into two types: local interpretability and global interpretability.

[0066] Local interpretability refers to the model explaining a specific case. For example, if a model intercepts a risk in a certain case, it is necessary to explain why the model intercepted the case and the features based on which the interception action was taken.

[0067] Globally interpretable: This usually refers to the model's behavior for all samples. For example, what are the main features learned by the model for this task scenario?

[0068] The technical solutions of the embodiments of the present application are further described below with reference to the accompanying drawings.

[0069] like Figure 1 As shown, Figure 1 1 is a schematic diagram of a system architecture platform 100 for executing a model interpretation method provided by an embodiment of the present invention.

[0070] exist Figure 1 In the example of FIG, the system architecture platform 100 is provided with a processor 110 and a memory 120, wherein the processor 110 and the memory 120 can be connected via a bus or other means. Figure 1 The bus connection is taken as an example.

[0071] The memory 120 is a non-transitory computer-readable storage medium that can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory 120 may include a high-speed random access memory and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 120 may optionally include a memory remotely located relative to the processor 110, and these remote memories may be connected to the system architecture platform via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0072] Those skilled in the art will appreciate that the system architecture platform can be applied to existing communication network systems and subsequently evolved mobile communication network systems, etc., and this embodiment does not specifically limit this.

[0073] It will be understood by those skilled in the art that Figure 1 The system architecture platform shown in the figure does not constitute a limitation on the embodiments of the present invention, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0074] The system architecture platform 100 can be an independent system architecture platform, or it can be a cloud system architecture platform 100 that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0075] Based on the above system architecture platform, various embodiments of the model interpretation method of the present invention are proposed below.

[0076] like Figure 2 As shown, Figure 2 It is a flowchart of a model interpretation method provided by an embodiment of the present invention. The model interpretation method can be applied to the above-mentioned architecture platform, and the model interpretation method includes but is not limited to step S100, step S200, step S300, step S400, step S500 and step S600.

[0077] Step S100: Obtain a pre-trained black box model, a sample set of the target task scenario, and a target sample to be explained.

[0078] It's understandable that a trained black-box model can be a classification model that, given a given input, outputs a probability distribution for each classification label. The sample set for the target task scenario refers to each sample that can serve as input for the model to predict the outcome. This data set represents the data and feature distribution for that scenario, helping us understand the distribution patterns of data and features within that scenario. The format of the specific sample to be interpreted aligns with the format of the bulk data for that task scenario.

[0079] Step S200: Input the sample set into the black box model to obtain feature distribution information.

[0080] It is understood that the feature distribution information includes the feature value corresponding to each dimension feature in the sample set and the proportion of the feature value in the sample set. According to the sample set of the target task scenario, the distribution law of each feature is statistically obtained, and the feature distribution obtained is:

[0081] D=[D0,D1,……,D K-1 ]

[0082] Among them, D i is the distribution of the i-th dimension feature.

[0083]

[0084] The total number of features in the i-th dimension is L i Different values, V i,j is the jth value of the i-th dimension feature, C i,j is the proportion of the j-th value of the i-th dimension feature in the sample set,

[0085] It should be noted that if there is no specific data, but the distribution law is known, it can be used directly, and its characteristic distribution format is the same as mentioned above.

[0086] Step S300: Input the target sample into the black box model for model prediction to obtain a model prediction score.

[0087] It is understandable that, assuming that the target sample has K-dimensional features, the target sample F=[F0,F1,……,F K-1 ], where F i is the value of the i-th dimension feature of sample F. Assume that the black box model now predicts F as category b, and obtains the model score S of the current sample now , S now =Model b (F), Model b (F) is the score predicted by the model for sample F in the bth category.

[0088] Step S400: traverse the features of each dimension in the target sample and calculate the weighted average score corresponding to each dimensional feature according to the feature distribution information.

[0089] It is understandable that traversing each dimension feature F i For each dimension, calculate the weighted average score of the feature value traversal under the dimension feature

[0090]

[0091] in, It is sample F General F i Replaced by V i,j The model score of the new sample after , that is, the i-dimensional feature of the target sample is replaced by the j-th value of the i-dimensional feature of the sample set, C i,j The weight is the proportion of the j-th value of the i-th dimension feature of the sample set in the sample set.

[0092]

[0093] Among them, [F0,……,F i-1 ,V i,j ,F i+1 ,……,F K-1 ] is Lieutenant General F i Replaced by V i,j The new sample after.

[0094] Step S500: Determine the importance of each dimension feature in the target sample based on the difference between the weighted average score and the model prediction score.

[0095] It is understandable that the importance of each dimension feature in the target sample can be represented by the size of the feature importance value. For example, for each dimension feature F i , calculate the importance value of the feature

[0096]

[0097] Current value F i The more important the target sample F is, the more S now and The greater the difference, The larger the , the more K-dimensional features there are.

[0098] Step S600 , explaining the black box model based on the target sample according to the importance of the features.

[0099] It is understood that model explanations include local and global explanations. Local explanations refer to the feature interpretation of the target sample, while global explanations refer to the feature analysis of the black-box model. For example, for local explanations, the features with the highest important eigenvalues ​​of the target sample can be used as the features that determine the target sample's classification by the black-box model. For global explanations, the features with the highest important eigenvalues ​​are used as the salient features of the black-box model.

[0100] It is understandable that the model interpretation method of the embodiment of the present application can be applied to mainstream model solutions such as deep neural network models and ensemble tree models. This solution has the following characteristics / advantages:

[0101] 1. Highly versatile. This method is applicable to mainstream deep neural network models, ensemble tree models, and other model solutions. This method does not require attention to model implementation details; it primarily analyzes the importance of a feature to the model by adjusting the model input and observing changes in the model output.

[0102] 2. This method has local interpretability. This method can explain the behavior of the model on a single case.

[0103] 3. This method is fully interpretable. This method can explain the overall behavior of the model, such as analyzing the model's salient features.

[0104] Based on this, the model interpretation method of the embodiment of the present application does not need to care about the details of the model implementation. The model can be regarded as an invisible black box. By adjusting the model input and observing the changes in the model output, the importance of a certain feature to the model can be analyzed, thereby helping engineers to explain the behavior of the model in specific samples and perform feature significance analysis. It can explain the behavior of the model in a single case and the overall behavior of the model.

[0105] Reference Figure 3 In one embodiment, step S400 includes but is not limited to step S410, step S420 and step S430.

[0106] Step S410: Replace each dimension feature in the target sample with the feature value corresponding to the feature of the same dimension in the sample set to obtain a new sample;

[0107] Step S420, performing model scoring on the new sample using the black box model to obtain a model scoring score;

[0108] Step S430: Obtain the weighted average score corresponding to each dimension feature in the target sample based on the model scoring score and the proportion of the feature value in the sample set.

[0109] It is understandable that traversing each dimension feature F i For each dimension, calculate the weighted average score of the feature value traversal under the dimension feature

[0110]

[0111] in, It is sample F General F i Replaced by V i,j The model score of the new sample after , that is, the i-dimensional feature of the target sample is replaced by the j-th value of the i-dimensional feature of the sample set, C i,j The weight is the proportion of the j-th value of the i-th dimension feature of the sample set in the sample set.

[0112]

[0113] Among them, [F0,……,F i-1 ,V i,j ,F i+1 ,……,F K-1 ] is Lieutenant General F i Replaced by V i,j The new sample after.

[0114] Reference Figure 4 In one embodiment, step S500 includes but is not limited to step S510, step S520 and step S530.

[0115] Step S510: Calculate the difference between the weighted average score corresponding to each dimensional feature and the model prediction score to obtain a first difference value;

[0116] Step S520: Calculate the difference between the weighted average score corresponding to all dimensional features and the model prediction score to obtain a second difference value;

[0117] Step S530: Obtain a first feature importance value by dividing the first difference value by the second difference value, wherein the first feature importance value is used to characterize the importance of each dimensional feature in the target sample, and the size of the first feature importance value is proportional to the importance.

[0118] It can be understood that the first difference value is obtained by calculating the difference between the weighted average score corresponding to each dimension feature and the model prediction score Calculate the difference between the weighted average score corresponding to all dimensional features and the model prediction score to obtain the second difference value

[0119] The importance of each dimension feature in the target sample can be represented by the size of the feature importance value. For example, for each dimension feature F i , calculate the importance value of the feature

[0120]

[0121] Current value F i The more important the target sample F is, the more S now and The greater the difference, The larger the , the more K-dimensional features there are.

[0122] Reference Figure 5 In one embodiment, step S600 includes but is not limited to step S610, step S620 and step S630.

[0123] Step S610, sorting the features according to the importance value of the first feature;

[0124] Step S620, determining the top N features with the highest first feature importance values;

[0125] Step S630: Use the first N features to make the black box model based on the target sample locally interpretable, where N is a positive integer greater than zero.

[0126] It is understandable that the explanation of the model includes local interpretability, which is the feature interpretation of the target sample. For example, for local interpretability, we can take the N features with the highest important feature values ​​of the target sample. The feature that determines whether the target sample is classified into this category by the black box model is And take The N features with the highest values. That is, the important features of the target sample are The features with the highest values.

[0127] Reference Figure 6 In one embodiment, step S600 further includes but is not limited to step S640, step S650, step S660, step S670 and step S680.

[0128] Step S640, calculating the first feature importance value corresponding to the feature of each dimension of each sample in the sample set;

[0129] Step S650: averaging the sum of the first feature importance values ​​corresponding to the features of each dimension of each sample to obtain a second feature importance value, wherein the second feature importance value is used to represent the importance of each dimensional feature in the sample set, and the magnitude of the second feature importance value is proportional to the importance;

[0130] Step S660: sorting each dimension feature in the sample set according to the importance value of the second feature;

[0131] Step S670, determining the features of the top N dimensions with the highest second feature importance values;

[0132] Step S680: Use the features of the first N dimensions to make the black box model based on the target sample globally interpretable, where N is a positive integer greater than zero.

[0133] It is understood that the explanation of the model includes global interpretability, where global interpretability is a feature analysis of the black box model. For example, for global interpretability, the features with the highest important eigenvalues ​​are used as the salient features of the black box model.

[0134] For example, for the sample set E, E={E0,E1,……,E M-1}, E i is a sample in the sample set.

[0135] The feature importance of the j-th dimension feature of the model for:

[0136]

[0137] in, is the feature importance of the j-th dimension of sample E_i.

[0138] The salient features of this model are The highest dimensional features.

[0139] It can be understood that, under the premise of having a sample set and a black box model, the feature distribution information of the sample set is first obtained through the sample set, where the feature distribution information includes the values ​​of each dimensional feature and the proportion of each value in the sample set. Afterwards, for the features of a certain dimension of the target sample, the features of that dimension are replaced with the values ​​of the features of the same dimension of the feature distribution information of the sample set to obtain each replaced target sample. The existing model is used to score each replaced target sample and combine the proportion of each value in the sample set to obtain the weighted average score of the features of one dimension of the original target sample (the calculation method can be to multiply the score and the value proportion and then add them together). The rest of the original target sample The weighted average score of features in other dimensions can also be obtained by the above method; then, the original target sample is scored using the existing model, and the important index of one dimensional feature of the original target sample can be obtained based on the previously obtained weighted average score and the model score of the original target sample (the calculation method can be the absolute value of the difference between the weighted average score of the dimensional feature and the model score divided by the sum of the absolute values ​​of the difference between the weighted average score of all dimensional features and the model score). Similarly, the important indexes of other dimensional features of the original target sample can also be completed using the above method to obtain the important indexes of all dimensional features of the original target sample. It can be seen that the important features of the target sample can be the dimensional features with the highest important indexes, thereby realizing local interpretability of the model.

[0140] In addition, by using the above method to obtain the important indicators of the characteristics of one dimension of the target sample, the important indicators of the characteristics of all samples in the sample set in one dimension can be obtained. By summarizing them, the important indicators of the sample set in one dimension can be obtained (the calculation method can be the sum of the important indicators of the characteristics of all samples in the same dimension divided by the number of all samples). Similarly, the important indicators of the sample set in other dimensions can be obtained by the above method, and the important indicators of all dimensional characteristics of the original sample set can be obtained. The significant features of the sample set can be the dimensional features with the highest important indicators, which realizes the global interpretability of the model.

[0141] To sum up, this application starts from the characteristics of one dimension, replaces the characteristics of the target sample according to the various values ​​of the characteristics of one dimension of the sample set, and then scores the target sample after feature replacement. The weighted average score is obtained according to the score and the proportion of various values ​​of the characteristics of one dimension of the sample set. Then, the important indicators of the dimensional characteristics of the target sample are obtained according to the weighted average score and the model score of the original target sample (without feature replacement), which realizes the local and global interpretability of the model and solves the interpretability problem of the model, especially the black box model.

[0142] Reference Figure 7 One embodiment of the present invention further provides a model interpretation device, comprising:

[0143] An acquisition module 710 is used to acquire a pre-trained black box model, a sample set of a target task scenario, and a target sample to be explained;

[0144] Feature analysis module 720, used to input the sample set into the black box model to obtain feature distribution information;

[0145] The prediction module 730 is used to input the target sample into the black box model for model prediction and obtain the model prediction score;

[0146] A calculation module 740 is used to traverse the features of each dimension in the target sample and calculate the weighted average score corresponding to each dimension feature based on the feature distribution information;

[0147] Analysis module 750, used to determine the importance of each dimension feature in the target sample based on the difference between the weighted average score and the model prediction score;

[0148] The explanation module 760 is used to explain the model of the black box model based on the target sample according to the importance of the features.

[0149] In one embodiment, the calculation module 740 is also used to replace each dimensional feature in the target sample with the feature value corresponding to the feature of the same dimension in the sample set to obtain a new sample; perform model scoring on the new sample through the black box model to obtain a model scoring score; and obtain a weighted average score corresponding to each dimensional feature in the target sample based on the model scoring score and the proportion of the feature value in the sample set.

[0150] In one embodiment, the analysis module 750 is also used to calculate the difference between the weighted average score corresponding to each dimensional feature and the model prediction score to obtain a first difference value; calculate the difference between the weighted average score corresponding to all dimensional features and the model prediction score to obtain a second difference value; divide the first difference value by the second difference value to obtain a first feature importance value, wherein the first feature importance value is used to characterize the importance of each dimensional feature in the target sample, and the size of the first feature importance value is proportional to the importance.

[0151] In one embodiment, the analysis module 760 is also used to sort the features according to the size of the first feature importance value; determine the top N features with the highest first feature importance values; and use the top N features to locally interpret the black box model based on the target sample, where N is a positive integer greater than zero.

[0152] In one embodiment, the analysis module 760 is also used to calculate the first feature importance value corresponding to the feature of each dimension of each sample in the sample set; average the sum of the first feature importance values ​​corresponding to the feature of each dimension of each sample to obtain a second feature importance value, wherein the second feature importance value is used to characterize the importance of each dimensional feature in the sample set, and the size of the second feature importance value is proportional to the importance; sort each dimensional feature in the sample set according to the size of the second feature importance value; determine the features of the top N dimensions with the highest second feature importance values; and use the features of the top N dimensions to globally interpret the black box model based on the target sample, wherein N is a positive integer greater than zero.

[0153] It should be noted that the technical means used, technical problems solved and technical effects achieved in the various embodiments of the above-mentioned model interpretation device are consistent with those in the embodiments of the model interpretation method. They will not be described in detail here. Please refer to the embodiments of the model interpretation method for details.

[0154] In addition, an embodiment of the present invention provides an electronic device, which includes: a memory, a processor, and an electronic device program stored in the memory and executable on the processor.

[0155] The processor and the memory may be connected via a bus or other means.

[0156] It should be noted that the electronic device in this embodiment may include: Figure 1 The memory and processor in the embodiment shown can constitute Figure 1 Part of the system architecture platform in the illustrated embodiment, both belong to the same inventive concept, so both have the same implementation principles and beneficial effects, and will not be described in detail here.

[0157] The non-transient software program and instructions required to implement the model interpretation method of the above embodiment are stored in the memory. When executed by the processor, the model interpretation method of the above embodiment is executed, for example, the above described Figure 2 Method steps S100 to S600, Figure 3 Steps S410 to S430 of the method, Figure 4 Method steps S510 to S530, Figure 5 Method steps S610 to S630 and Figure 6 Method steps S640 to S680 in .

[0158] The electronic device includes components such as a radio frequency (RF) circuit, a memory, an input unit, a display unit, a sensor, an audio circuit, a wireless fidelity (WiFi) module, a processor, and a power supply. Those skilled in the art will appreciate that this embodiment does not limit the structure of the electronic device to a single component, and the electronic device may include more or fewer components than those in this embodiment, or may combine certain components or arrange the components differently.

[0159] RF circuits can be used to receive and send signals during information transmission or calls. In particular, after receiving the downlink information from the base station, it is sent to the processor for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF circuit can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communication can use any communication standard or protocol, including but not limited to the Global System of Mobile communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0160] The memory can be used to store software programs and modules. The processor executes the various functional applications and data processing of the electronic device by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store an operating system, at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created based on the use of the electronic device (such as audio data, a phone book, etc.). In addition, the memory can include a high-speed random access memory and a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0161] The input unit can be used to receive input digital or character information, and to generate key signal input related to the settings and function control of the electronic device. Specifically, the input unit may include a touch panel and other input devices. The touch panel, also known as a touch screen, can collect touch operations on or near it (such as operations on or near the touch panel using any suitable object or accessory such as a finger, stylus, etc.) and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel may include two parts: a touch detection device and a touch controller. The touch detection device detects the touch direction and detects the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device and converts it into touch point coordinates, which are then sent to the processor, and can receive and execute commands sent by the processor. In addition, the touch panel can be implemented using various types such as resistive, capacitive, infrared and surface acoustic wave. In addition to the touch panel, the input unit may also include other input devices. Specifically, other input devices may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, etc.

[0162] The display unit can be used to display input information or provided information and various menus of the electronic device. The display unit may include a display panel, and optionally, the display panel may be configured in the form of a liquid crystal display (Liquid Crystal Display, LCD), an organic light-emitting diode (Organic Light-Emitting Diode, OLED), etc. Further, the touch panel may cover the display panel, and when the touch panel detects a touch operation on or near it, it is transmitted to the processor to determine the category of the touch event, and then the processor provides a corresponding visual output on the display panel according to the category of the touch event. Although the touch panel and the display panel are two independent components to realize the input and output functions of the electronic device, in some embodiments, the touch panel and the display panel can be integrated to realize the input and output functions of the electronic device.

[0163] The electronic device may also include at least one sensor, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel according to the brightness of the ambient light, and the proximity sensor may turn off the display panel and / or backlight when the electronic device is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that segment the posture of the electronic device (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration segmentation related functions (such as pedometer, tapping), etc.; as for other sensors that the electronic device can also be configured with, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be described here.

[0164] The audio circuit, speaker, and microphone provide an audio interface. The audio circuit converts received audio data into electrical signals and transmits them to the speaker, which then converts them into sound signals for output. The microphone, on the other hand, converts collected sound signals into electrical signals, which are then received by the audio circuit and converted into audio data. The audio data is then processed by the processor and then transmitted via the RF circuit to, for example, another electronic device, or stored in a memory for further processing.

[0165] WiFi is a short-range wireless transmission technology. Electronic devices can use WiFi modules to send and receive emails, browse the web, and access streaming media, providing wireless broadband internet access. WiFi modules are not essential components of electronic devices and can be omitted as needed without altering the essence of the invention.

[0166] The processor is the control center of an electronic device, connecting all parts of the electronic device using various interfaces and circuits. By running or executing software programs and / or modules stored in memory and accessing data stored in memory, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor may include one or more processing units; preferably, the processor may integrate an application processor and a modem processor, wherein the application processor primarily handles the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into the processor.

[0167] The electronic device also includes a power source (such as a battery) to power each component. Preferably, the power source can be logically connected to the processor through a power management system, so that the power management system can manage charging, discharging, and power consumption.

[0168] Although not shown, the electronic device may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0169] In this embodiment, the processor included in the terminal device is capable of executing the pedestrian re-segmentation method of the previous embodiment.

[0170] In addition, an embodiment of the present invention further provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are used to execute the model interpretation method on the terminal side, for example, to execute the above-described Figure 2 Method steps S100 to S600, Figure 3 Steps S410 to S430 of the method, Figure 4 Method steps S510 to S530, Figure 5 Method steps S610 to S630 and Figure 6 Method steps S640 to S680 in .

[0171] Those skilled in the art will appreciate that all or some of the steps and systems in the method disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include computer storage media (or non-transitory media) and communication media (or temporary media). As known to those skilled in the art, the term computer storage media is included in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data) and is volatile and non-volatile, removable, and non-removable. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory, or other memory technology, CD-ROM, digital versatile disks (DVD), or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage, or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication media typically includes computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0172] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above implementation. Those skilled in the art can also make various equivalent modifications or substitutions under the shared conditions that do not violate the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A model interpretation method, characterized in that: The method comprises: Obtain a pre-trained black box model, a sample set of the target task scenario, and the target sample to be explained; Inputting the sample set into the black box model to obtain feature distribution information; wherein the feature distribution information includes the feature value corresponding to each dimension of the feature in the sample set and the proportion of the feature value in the sample set; Input the target sample into the black box model to perform model prediction and obtain a model prediction score; Traversing the features of each dimension in the target sample, and calculating the weighted average score corresponding to the features of each dimension in the target sample according to the feature distribution information; Determine the importance of the feature in each dimension to the target sample based on the difference between the weighted average score and the model prediction score; Interpreting the model of the black box model based on the target sample according to the importance of the feature in the target sample to obtain local interpretability and global interpretability, wherein the local interpretability is the feature interpretation of the target sample, and the global interpretability is the feature analysis of the black box model; The calculating the weighted average score corresponding to the feature of each dimension in the target sample based on the feature distribution information includes: replacing the feature of each dimension in the target sample with the feature value corresponding to the feature of the same dimension in the sample set to obtain a new sample; performing model scoring on the new sample through the black box model to obtain a model scoring score; and obtaining the weighted average score corresponding to the feature of each dimension in the target sample based on the model scoring score and the proportion of the feature value in the sample set.

2. The method according to claim 1, characterized in that Determining the importance of the feature in each dimension in the target sample based on the difference between the weighted average score and the model prediction score includes: Calculate the difference between the weighted average score corresponding to the feature in each dimension and the model prediction score to obtain a first difference value; Calculate the difference between the weighted average score corresponding to the features of all dimensions and the model prediction score to obtain a second difference value; A first feature importance value is obtained by dividing the first difference value by the second difference value, wherein the first feature importance value is used to characterize the importance of the feature of each dimension in the target sample, and the size of the first feature importance value is proportional to the importance.

3. The method according to claim 2, characterized in that The step of explaining the model of the black box model based on the target sample according to the importance includes: sorting the features according to the importance values ​​of the first features; Determine the top N features with the highest importance values ​​of the first feature; The first N features are used to locally interpret the model of the black box model based on the target sample, where N is a positive integer greater than zero.

4. The method according to claim 2, characterized in that The step of explaining the model of the black box model based on the target sample according to the importance includes: Calculating the first feature importance value corresponding to the feature of each dimension of each sample in the sample set; averaging the sum of the first feature importance values ​​corresponding to the features of each dimension of each sample to obtain a second feature importance value, wherein the second feature importance value is used to represent the importance of the feature of each dimension in the sample set, and the magnitude of the second feature importance value is proportional to the importance; sorting the features of each dimension in the sample set according to the importance value of the second feature; Determine the features of the top N dimensions with the highest importance values ​​of the second feature; The features of the first N dimensions are used to globally interpret the model of the black box model based on the target sample, where N is a positive integer greater than zero.

5. The method according to any one of claims 1 to 4, characterized in that The black box model is a classification model.

6. A model interpretation device, characterized in that: The device comprises: The acquisition module is used to obtain the pre-trained black box model, the sample set of the target task scenario, and the target sample to be explained; A feature analysis module, configured to input the sample set into the black box model to obtain feature distribution information; wherein the feature distribution information includes a feature value corresponding to a feature of each dimension in the sample set and a proportion of the feature value in the sample set; A prediction module is used to input the target sample into the black box model to perform model prediction and obtain a model prediction score; A calculation module is configured to traverse the features of each dimension in the target sample and calculate a weighted average score corresponding to the features of each dimension in the target sample according to the feature distribution information; the calculation of the weighted average score corresponding to the features of each dimension in the target sample according to the feature distribution information includes: replacing the features of each dimension in the target sample with feature values ​​corresponding to the features of the same dimension in the sample set to obtain a new sample; performing a model scoring on the new sample using the black box model to obtain a model scoring score; and obtaining a weighted average score corresponding to the features of each dimension in the target sample according to the model scoring score and the proportion of the feature values ​​in the sample set; An analysis module is configured to determine the importance of the feature in each dimension in the target sample based on the difference between the weighted average score and the model prediction score; An interpretation module is used to interpret the model of the black box model based on the target sample according to the importance of the feature in the target sample, so as to obtain local interpretability and global interpretability, wherein the local interpretability is the feature interpretation of the target sample, and the global interpretability is the feature analysis of the black box model.

7. An electronic device, characterized in that: The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the steps of the model interpretation method according to any one of claims 1 to 5 are realized.

8. A storage medium, which is a computer-readable storage medium and is used for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the model interpretation method described in any one of claims 1 to 5.

Images