Model interpretation method and apparatus, storage medium, and computer program product

Abstract

Embodiments of the present application provide a model interpretation method and apparatus, a storage medium, and a computer program product. The method comprises: determining an attention weight value of first multimedia data in a forward propagation process in a first model, and determining a gradient value of the first multimedia data in a backward propagation process in the first model, wherein the first multimedia data is any input data input to the first model; using the attention weight value and the gradient value to determine an attention value of the first multimedia data; and using the attention value as interpretability information of the first multimedia data, so as to use the interpretability information to interpret a decision-making process of the first model.

Description

Model explanation method and device, storage medium, and computer program product

[0001] Cross-reference to Related Applications

[0002] This application is based on and claims priority to Chinese Patent Application No. 202410650299.9, filed on May 23, 2024, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0003] The present application relates to the technical field of computers, and in particular to a model explanation method and device, a storage medium, and a computer program product. BACKGROUND

[0004] With the development of deep learning, the performance of various models is getting better and better, and breakthroughs have been made in various fields. However, the decision-making process of most models is like a black box and is difficult to understand. When the model decision fails, it is impossible to understand the decision-making process of the model, and it is also impossible to effectively adjust and improve it. This opacity can cause critical problems in some fields such as security. However, with the development of models, the complexity of the structure of the model and the diversity of feature interaction patterns result in low explainability of the model and low trustworthiness of the model. SUMMARY

[0005] The embodiments of the present application provide a model explanation method and device, a storage medium, and a computer program product, which can improve the explainability and trustworthiness of the model.

[0006] The technical solution of the present application is implemented as follows:

[0007] In a first aspect, the embodiments of the present application provide a model explanation method, which comprises:

[0008] determining an attention weight value of first multimedia data in a forward propagation process of a first model; determining a gradient value of the first multimedia data in a backward propagation process of the first model; the first multimedia data being any input data input into the first model;

[0009] determining an attention value of the first multimedia data using the attention weight value and the gradient value;

[0010] using the attention value as explainability information of the first multimedia data to explain the decision-making process of the first model using the explainability information.

[0011] In a second aspect, the embodiments of the present application provide a model explanation device, which comprises:

[0012] determining a gradient value of the first multimedia data in a back propagation process of the first model; the first multimedia data being any input data inputting the first model; determining an attention value of the first multimedia data by using the attention weight value and the gradient value;

[0013] interpreting the attention value as explainability information of the first multimedia data, so as to interpret a decision-making process of the first model by using the explainability information.

[0014] In a third aspect, an embodiment of the present application provides a model interpretation device, which comprises a processor, a memory and a communication bus; the processor implements the model interpretation method described above when executing a running program stored in the memory.

[0015] In a fourth aspect, an embodiment of the present application provides a storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the model interpretation method described above.

[0016] In a fifth aspect, an embodiment of the present application provides a computer program product comprising a computer program, and the computer program is executed by a processor to implement the model interpretation method described above.

[0017] The embodiments of the present application provide a model interpretation method and device, a storage medium and a computer program product. The method comprises the following steps: determining an attention weight value of first multimedia data in a forward propagation process of a first model; determining a gradient value of the first multimedia data in a back propagation process of the first model; the first multimedia data being any input data inputting the first model; determining an attention value of the first multimedia data by using the attention weight value and the gradient value; and interpreting the attention value as explainability information of the first multimedia data, so as to interpret a decision-making process of the first model by using the explainability information. By using the above implementation scheme, the attention distribution information in the forward propagation process and the gradient information in the back propagation process are fused, the influence of deep feature information on the decision result of the model is learned, a deeper understanding is provided, the transparency of the model is greatly enhanced, and the explainability of the model is improved. Moreover, the attention value is based on the gradient information in the back propagation process of the model, so that the attention value dynamically changes with the model training, the dynamic information in the model training process can be captured based on the attention value, and then the model training process can be optimized, thereby improving the trustworthiness of the model. BRIEF DESCRIPTION OF DRAWINGS

[0018] FIG. 1 is a flowchart of a model interpretation method according to an embodiment of the present application;

[0019] FIG. 2 is a flowchart of a model explanation method according to an embodiment of the present application;

[0020] FIG. 3 is a structural component diagram of an exemplary model explanation device according to an embodiment of the present application;

[0021] FIG. 4 is a structural schematic diagram of a model explanation device according to an embodiment of the present application;

[0022] FIG. 5 is a structural schematic diagram of a model explanation device according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to enable a more detailed understanding of the features and technical content of the embodiments of the present application, the implementation of the embodiments of the present application will be described in detail below with reference to the accompanying drawings, which are only used for reference and are not intended to limit the embodiments of the present application.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0025] In the following description, "some embodiments" are described, which describe a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict. It should be noted that the terms "first, second, third" used in the embodiments of the present application are only used to distinguish similar objects, and do not represent a specific order of the objects. It can be understood that "first, second, third" can be interchanged in a specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0026] The embodiments of the present application provide a model explanation method, as shown in FIG. 1, which can include:

[0027] S101, determining the attention weight value of the first multimedia data in the forward propagation process of the first model; determining the gradient value of the first multimedia data in the back propagation process of the first model; the first multimedia data is any input data input into the first model.

[0028] The model explanation method proposed in the embodiments of the present application can be applied to the scenario where the Transformer model performs any software classification. Among them, the first model in the embodiments of the present application is not limited to the Transformer model, but can also be other deep learning models such as Convolutional Neural Networks (CNN), Deep Neural Network (DNN), Recurrent Neural Network (RNN), etc. The specific selection can be made according to the actual situation, and the embodiments of the present application do not make specific limitations; the scenario in the embodiments of the present application is not limited to malware classification, but can also be any field such as other images, voice, natural language processing, control, etc. The specific selection can be made according to the actual situation, and the embodiments of the present application do not make specific limitations.

[0029] In the embodiments of the present application, for the original multimedia data input into the first model, data preprocessing is first performed to obtain multimedia data. Through data preprocessing, the original multimedia data is converted into an input format that the first model can accept. In this way, preparation is provided for the subsequent processing process of the first model, helping to improve the performance of the first model.

[0030] It should be noted that the multimedia data can be image data, audio data, text data, video data, etc. The specific selection can be made according to the application scenario corresponding to the first model, and the embodiments of the present application do not make specific limitations.

[0031] In the embodiments of the present application, data preprocessing includes the following four key stages: data reading stage, data cleaning stage, feature extraction stage and feature representation stage.

[0032] In the data reading stage, the original multimedia data is read from the data source. This process involves different types of files, such as binary files, Portable Executable (PE) files, text files, etc. In order to process different types of files, different data reading methods need to be used. For example, for binary files, the byte sequence is directly read; for PE files, a special library (such as the Portable Executable format (PELile) library in Python) is used to parse the file and extract metadata and section information.

[0033] In the data cleaning stage, the read original multimedia data is cleaned. This process includes removing unnecessary special characters, deleting blank lines, handling missing values, etc. The purpose of this step is to eliminate noise that may affect the performance of the model and convert the data into a format suitable for subsequent processing.

[0034] In the feature extraction stage, meaningful features are extracted from the cleaned multimedia data. For binary files, their byte frequency distribution is calculated as features; for PE files, their metadata (e.g., file size, section number, etc.) and section information (e.g., size of each section, permission, etc.) are extracted as features. In addition, some more advanced techniques, such as static analysis or dynamic analysis, can be used to extract more features (e.g., API call sequences).

[0035] In the feature representation stage, the extracted features are converted into a vector form that can be processed by the model. This process usually involves some encoding methods, such as One-hot encoding or Term Frequency-Inverse Document Frequency (TF-IDF) encoding. For example, an Application Programming Interface (API) call sequence is One-hot encoded into a One-hot vector; or a byte frequency distribution is TF-IDF encoded into a TF-IDF vector.

[0036] In an optional embodiment, the first model is a Transformer model, and the attention weight value of the first multimedia data at each layer of the Transformer model is determined; the gradient value of the first multimedia data at each layer of the Transformer model is determined.

[0037] In the embodiments of the present application, for any input data input into the first model, first forward propagation of the model is performed, and in the hth layer of each Transformer model, an attention weight matrix A h ∈R n×n is calculated, where n is the length of the first multimedia data X. The element A h (i,j) of the attention weight matrix A h represents the attention weight of position i to position j in the hth layer.

[0038] In the embodiments of the present application, then, backward propagation of the model is performed, and the gradient value G h of the hth layer of each Transformer model is calculated. The size of G h reflects the degree of update of the hth layer parameters of the model in the training process.

[0039] It should be noted that after the gradient value of the first multimedia data at each layer of the Transformer model is determined, the gradient value is normalized. Specifically, first, the absolute value sum S of the gradient of all m layers of the Transformer model is calculated, see formula (1).

[0040] Then the gradient G of each layer is divided by S to obtain the normalized gradient value W of each layer, see formula (2). h h

[0041] S102, using the attention weight value and the gradient value, determining the attention value of the first multimedia data.

[0042] In an optional embodiment, the first model is a Transformer model, and the attention weight value of each layer and the gradient value of each layer are processed to obtain the attention value of each layer; the attention values of each layer are accumulated to obtain the attention value of the first multimedia data.

[0043] In the embodiment of the application, the gradient value of each layer is applied to the attention weight value of the corresponding layer to obtain the attention value of each layer, see formula (3). h h h (3)

[0044] Then A' is added to obtain the attention value A of the first multimedia data, see formula (4). h

[0045] S103, taking the attention value as the explainability information of the first multimedia data, and using the explainability information to explain the decision-making process of the first model.

[0046] It should be noted that the embodiments of the application introduce a new perspective to understand and explain the behavior of the Transformer model, which combines the attention degree of the model to the input multimedia data and the importance of each Transformer layer in the model training process.

[0047] It should be noted that after taking the attention value as the explainability information of the first multimedia data, the explainability information of the first multimedia data can also be displayed.

[0048] In an optional embodiment, based on the explainability information, display parameter information of the first multimedia data is determined; based on the display parameter information, the corresponding multimedia data is displayed.​​​​​

[0049] In the embodiment of the present application, the display parameter information can be display font size, display font color, etc., which can be selected according to actual conditions, and the embodiment of the present application does not make specific limitations.

[0050] Taking display font size as an example, the display font size is proportional to the explainability information, that is, the larger the explainability information is, the larger the font is displayed, so it can be directly seen which multimedia data has a large impact on the output of the first model.

[0051] Taking display font color as an example, the display font color is proportional to the explainability information, that is, the larger the explainability information is, the darker the font color is displayed, so it can be directly seen which multimedia data has a large impact on the output of the first model.

[0052] In the embodiment of the present application, the word cloud scheme or the text heat map scheme can be used to display the explainability information of the first multimedia data.

[0053] In the embodiment of the present application, the word cloud scheme is used for display, the first multimedia data can be a word, the word and the corresponding explainability information can be used as input, a word cloud is created by using a word cloud generation tool, and in the word cloud, the display font size is proportional to the explainability information.

[0054] It should be noted that the display font size can also be set to be inversely proportional to the explainability information, and the specific mapping relationship can be selected according to actual conditions, and the embodiment of the present application does not make specific limitations.

[0055] In another optional embodiment, based on the explainability information, the marking information of the first multimedia data is determined; the first multimedia data is marked by using the marking information to obtain the marked first multimedia data; and the marked first multimedia data is displayed.

[0056] In the embodiment of the present application, each first multimedia data is associated with its corresponding explainability information, and specifically, the underline intensity can be used as marking information to represent the explainability information of each first multimedia data. For example, the larger the explainability information is, the thicker the underline is used to mark. Finally, the marked first multimedia data is displayed, so it can be directly seen which multimedia data has a large impact on the output of the first model based on the marking information.

[0057] It should be noted that for all the multimedia data input into the first model, the positions of each multimedia data in all the multimedia data and the explainability information of each multimedia data can be displayed.

[0058] It can be understood that the fusion of the attention distribution information of the forward propagation and the gradient information of the backward propagation, the learning of the influence of the deep feature information on the decision result of the model, provides a deeper understanding, can greatly enhance the transparency of the model, and improves the explainability of the model. And the attention value is based on the gradient information in the backward propagation process of the model, so that the attention value will dynamically change with the progress of the model training, and the dynamic information in the model training process can be captured based on the attention value, and then the model training process can be optimized, thereby improving the trustworthiness of the model.

[0059] It should be noted that before S103, a global or feature level model explanation method is also included, and the embodiment of the present application uses SHAP (SHapley Additive exPlanations) to perform explainability analysis on the first model. The interaction effect of the features is introduced based on SHAP to reveal the causal relationship behind the model prediction result, enhance the transparency of the model, and provide a basis for model optimization. Referring to FIG. 2, it specifically includes:

[0060] S201, respectively determining a first contribution value of the first multimedia data to the decision result data and a second contribution value of the second multimedia data to the decision result; the second multimedia data is multimedia data other than the first multimedia data; and the decision result data is generated after the first model processes the first multimedia data and the second multimedia data.

[0061] In the embodiment of the present application, an interpreter is first constructed, which uses the SHAP method and references the Shapley value in game theory to explain the model prediction. The Shapley value is a fair distribution solution that assigns a contribution value to each feature (first multimedia data), which represents the contribution of the feature to the decision result data. The interpreter also calculates the interaction effect between features, that is, how the features affect each other to contribute to the prediction result.

[0062] In the embodiment of the present application, the initialized interpreter is used to explain the first multimedia data, and the contribution value φ i (v) of each multimedia data is calculated, as shown in formula (5).

[0063] Wherein, v is a function of a feature set, S is a different feature set composed of different features, N is a total feature set, S is a subset of N, N\{i} is a set after removing element i from set N, |S| is the size of the feature set, and |N| is the size of the total feature set.

[0064] It should be noted that the calculation of the first contribution value of the first multimedia data and the calculation of the second contribution value of the second multimedia data can be calculated according to formula (5), which will not be repeated here.

[0065] S202, based on the first contribution value and the second contribution value, determining a third contribution value of the first multimedia data under the interaction of the second multimedia data.

[0066] It should be noted that the interaction effect between multimedia data describes the mutual influence between features. It is obtained by calculating the difference between the SHAP value of one feature in the presence and absence of another feature. Specifically, if the interaction effect of the first multimedia data and the second multimedia data is positive, it means that when the first multimedia data and the second multimedia data exist at the same time, their joint contribution to the decision result data is greater than the sum of their individual contributions. On the contrary, if the interaction effect of the first multimedia data and the second multimedia data is negative, it means that when the first multimedia data and the second multimedia data exist at the same time, their joint contribution to the decision result data is less than the sum of their individual contributions.

[0067] In the embodiments of the present application, the first contribution value and the second contribution value can be added to obtain the third contribution value of the first multimedia data under the interaction of the second multimedia data. It should be noted that the addition is only one optional calculation method proposed in the embodiments of the present application, and the specific calculation method can be selected according to the actual situation, and the embodiments of the present application do not make specific limitations.

[0068] It can be understood that the model explanation method at the global or feature level can more comprehensively understand the dynamic interaction between features, thereby providing a deeper model explanation, providing more abundant information for model optimization, and helping to improve the performance and accuracy of the model.

[0069] Correspondingly, under the premise of performing S201-S202, the specific implementation process of S103 can refer to S203.

[0070] S203, fusing the third contribution value and the attention value to obtain the explainability information of the first multimedia data.

[0071] In an optional embodiment, for the Transformer model, the third contribution value and the attention value are both in the form of a matrix, and the matrix of the third contribution value and the matrix of the attention value can be added to realize the fusion of the third contribution value and the attention value. The specific fusion method is not limited in the present application, and can be fused according to the actual situation.

[0072] It should be noted that the first contribution value and the third shared sum attention value can be fused to obtain the explainability information of the first multimedia data. Specifically, it can be selected according to actual conditions, and the embodiments of the application are not limited specifically.

[0073] It should be noted that the explainable information can still be subjected to subsequent display process. Specifically, refer to the description of S103, and the embodiments of the application will not be repeated.

[0074] Based on the above embodiments, the embodiments of the application propose a processing device of a malware classification model-oriented explainability method. Referring to FIG. 3, the device includes a data preprocessing module, a global / feature level explainability module, a local / attention mechanism explainability module, and a fusion visualization module. The data preprocessing module sequentially performs data reading, data cleaning, feature extraction, and feature representation processes. The global / feature level explainability module sequentially performs interpreter initialization, SHAP value calculation of features, and feature interaction calculation processes. The local / attention mechanism explainability module sequentially performs gradient value calculation under model forward propagation, gradient value calculation under model backward propagation, normalization calculation, and multi-attention head fusion calculation processes. The fusion visualization module can use a word cloud scheme and / or a text heat map scheme to realize display of explainability information.

[0075] The embodiments of the application provide a model explanation device 1. As shown in FIG. 4, the model explanation device 1 includes:

[0076] A determination unit 10 is configured to determine an attention weight value of a first multimedia data in a forward propagation process of a first model; determine a gradient value of the first multimedia data in a backward propagation process of the first model; the first multimedia data is any input data input into the first model; and determine an attention value of the first multimedia data by using the attention weight value and the gradient value.

[0077] An explanation unit 11 is configured to take the attention value as explainability information of the first multimedia data, and use the explainability information to explain a decision process of the first model.

[0078] Optionally, the device further includes a fusion unit.

[0079] The determining unit 10 is further configured to determine a first contribution value of the first multimedia data to a decision result data and a second contribution value of second multimedia data to the decision result, the second multimedia data being multimedia data other than the first multimedia data, the decision result data being generated by processing the first multimedia data and the second multimedia data by a first model, and determine a third contribution value of the first multimedia data under interaction of the second multimedia data based on the first contribution value and the second contribution value.

[0080] The fusion unit is configured to fuse the third contribution value and the attention value to obtain the explainability information of the first multimedia data.

[0081] Optionally, the first model is a Transformer model.

[0082] The determining unit 10 is further configured to determine an attention weight value of the first multimedia data at each layer of the Transformer model, determine a gradient value of the first multimedia data at each layer of the Transformer model, process the attention weight value of each layer and the gradient value of each layer to obtain an attention value of each layer, and accumulate the attention value of each layer to obtain the attention value of the first multimedia data.

[0083] Optionally, the apparatus further includes a display unit.

[0084] The display unit is configured to display the explainability information of the first multimedia data.

[0085] Optionally, the determining unit 10 is further configured to determine display parameter information of the first multimedia data based on the explainability information.

[0086] The display unit is further configured to display corresponding multimedia data based on the display parameter information.

[0087] Optionally, the determining unit 10 is further configured to determine marking information of the first multimedia data based on the explainability information, and mark the first multimedia data by using the marking information to obtain marked first multimedia data.

[0088] The display unit is further configured to display the marked first multimedia data.

[0089] The model explanation device provided in the embodiment of the application determines an attention weight value of first multimedia data in a forward propagation process of a first model; determines a gradient value of the first multimedia data in a backward propagation process of the first model; the first multimedia data is any input data input into the first model; determines an attention value of the first multimedia data by using the attention weight value and the gradient value; and takes the attention value as explainability information of the first multimedia data, so as to use the explainability information to explain a decision process of the first model. It can be seen that the model explanation device provided in the embodiment of the application fuses attention distribution information of the forward propagation and gradient information of the backward propagation, learns an influence of deep feature information on a decision result of the model, provides a deeper understanding, greatly enhances transparency of the model, and improves explainability of the model. The attention value is based on the gradient information in the backward propagation process of the model, so that the attention value dynamically changes with the model training, the dynamic information in the model training process can be captured based on the attention value, and then the model training process can be optimized, thereby improving trustworthiness of the model.

[0090] FIG. 5 is a schematic diagram of a component structure of a model explanation device 1 provided in an embodiment of the application. In actual application, based on the same disclosure concept of the above embodiment, as shown in FIG. 5, the model explanation device 1 of the embodiment includes a processor 12, a memory 13, and a communication bus 14.

[0091] The processor 12 can be at least one of an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a CPU, a controller, a microcontroller, and a microprocessor. It can be understood that, for different devices, the electronic device used to implement the function of the processor can also be other devices, and the embodiment is not limited in this regard.

[0092] In the embodiment of the application, the communication bus 14 is used to realize connection and communication between the processor 12 and the memory 13; and the processor 12 realizes the following model explanation method when executing a running program stored in the memory 13.

[0093] determine an attention weight value of the first multimedia data in a forward propagation process of the first model; determine a gradient value of the first multimedia data in a backward propagation process of the first model; the first multimedia data is any input data input into the first model; determine an attention value of the first multimedia data by using the attention weight value and the gradient value; and use the attention value as explainability information of the first multimedia data to explain a decision-making process of the first model by using the explainability information.

[0094] Further, the processor 12 is further configured to determine a first contribution value of the first multimedia data to decision result data and a second contribution value of second multimedia data to the decision result data, respectively; the second multimedia data is multimedia data other than the first multimedia data; the decision result data is generated after the first model processes the first multimedia data and the second multimedia data; determine a third contribution value of the first multimedia data under the interaction of the second multimedia data based on the first contribution value and the second contribution value; and fuse the third contribution value and the attention value to obtain explainability information of the first multimedia data.

[0095] Further, the first model is a Transformer model,

[0096] The processor 12 is further configured to determine an attention weight value of the first multimedia data at each layer of the Transformer model; determine a gradient value of the first multimedia data at each layer of the Transformer model; process the attention weight value of each layer and the gradient value of each layer to obtain an attention value of each layer; and accumulate the attention value of each layer to obtain the attention value of the first multimedia data.

[0097] Further, the processor 12 is further configured to display the explainability information of the first multimedia data.

[0098] Further, the processor 12 is further configured to determine display parameter information of the first multimedia data based on the explainability information; and display corresponding multimedia data based on the display parameter information.

[0099] Further, the processor 12 is further configured to determine marking information of the first multimedia data based on the explainability information; mark the first multimedia data by using the marking information to obtain marked first multimedia data; and display the marked first multimedia data.

[0100] The embodiment of the present application provides a storage medium, which stores a computer program, the computer readable storage medium stores one or more programs, the one or more programs can be executed by one or more processors, and the one or more programs are applied to a model explanation device, and the computer program implements the model explanation method.

[0101] Based on the above embodiment, the embodiment of the present application provides a computer program product, which comprises a computer program, the computer program can be executed by one or more processors, and the computer program implements the model explanation method.

[0102] It should be noted that, in this paper, the term "comprise", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "comprises a" does not exclude the presence of another identical element in the process, method, article or device comprising the element.

[0103] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment method can be realized by means of software and necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present disclosure essentially or say the part that contributes to the related art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a plurality of instructions for making an image display device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) execute the method described in each embodiment of the present disclosure.

[0104] The above is only a preferred embodiment of the present application, and is not intended to limit the protection scope of the present application.

Claims

1. A model explanation method, the method comprising: determining attention weight values of first multimedia data in a forward propagation process of a first model; determining gradient values of the first multimedia data in a backward propagation process of the first model; the first multimedia data being any input data input into the first model; determining an attention value of the first multimedia data using the attention weight values and the gradient values; using the attention value as explainability information of the first multimedia data to explain a decision-making process of the first model using the explainability information.

2. The method of claim 1, wherein, Before the using the attention value as explainability information of the first multimedia data, the method further comprises: respectively determining a first contribution value of the first multimedia data to a decision result data and a second contribution value of a second multimedia data to the decision result data; the second multimedia data being multimedia data other than the first multimedia data; the decision result data being generated after the first model processes the first multimedia data and the second multimedia data; determining a third contribution value of the first multimedia data under interaction of the second multimedia data based on the first contribution value and the second contribution value; the using the attention value as explainability information of the first multimedia data comprising: fusing the third contribution value and the attention value to obtain the explainability information of the first multimedia data.

3. The method of claim 1, wherein, the first model being a Transformer model, the determining attention weight values of first multimedia data in a forward propagation process of a first model comprising: determining gradient values of the first multimedia data in a backward propagation process of the first model comprising: determining attention weight values of the first multimedia data at each layer of the Transformer model; determining gradient values of the first multimedia data at each layer of the Transformer model; the using the attention weight values and the gradient values to determine an attention value of the first multimedia data comprising: processing the attention weight values of each layer and the gradient values of each layer to obtain an attention value of each layer; accumulating the attention values of each layer to obtain the attention value of the first multimedia data.

4. The method of claim 1, wherein, After the using the attention value as explainability information of the first multimedia data, the method further comprises: displaying the explainability information of the first multimedia data.

5. The method of claim 4, wherein, the displaying the explainability information of the first multimedia data comprising: determining display parameter information of the first multimedia data based on the explainability information; displaying corresponding multimedia data based on the display parameter information.

6. The method of claim 4, wherein, the displaying the explainability information of the first multimedia data comprising: determining marking information of the first multimedia data based on the explainability information; marking the first multimedia data using the marking information to obtain marked first multimedia data; displaying the marked first multimedia data.

7. A model explanation device, the device comprising: determining a gradient value of the first multimedia data in a back propagation process of the first model; determining a gradient value of the first multimedia data in a back propagation process of the first model; the first multimedia data is any input data input into the first model; the attention weight value and the gradient value are used to determine an attention value of the first multimedia data; an explanation unit configured to use the attention value as explainability information of the first multimedia data to explain a decision-making process of the first model.

8. A model explanation device, the device comprising: a processor, a memory and a communication bus; the processor executes a running program stored in the memory to implement the method of any one of claims 1-6.

9. A storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the method of any one of claims 1-6.

10. A computer program product comprising a computer program, the computer program being executed by a processor to implement the method of any one of claims 1-6.

Images