Method for obtaining an interpretable result of a game reinforcement learning model and related device

Abstract

The application provides a method for obtaining an interpretable result of a game reinforcement learning model and related equipment. The method comprises: inputting original data with a multi-dimensional state into the game reinforcement learning model to obtain a first expected return of each of a plurality of actions; for each dimension of the multi-dimensional state, obtaining perturbation data from the original data by adding noise to the dimension of the state, obtaining a second expected return of each of the plurality of actions by inputting the perturbation data into the game reinforcement learning model, and calculating an importance of the dimension of the state for each of the plurality of actions based on the first expected return and the second expected return; and forming an interpretable result of a state-to-action mapping strategy of the game reinforcement learning model by statistically processing the importance. It can be seen that the method realizes visual interpretation of a mapping relationship between a state and an action.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the explainable technology of artificial intelligence models, and in particular to a method for obtaining an explainable result of a game reinforcement learning model and related equipment. BACKGROUND

[0002] In related explainable technology of artificial intelligence models, the explainable result often uses a visual explanation method of a saliency map.

[0003] In the visual explanation method of a saliency map, especially in the saliency map method based on perturbation, an image is often needed as input data of the artificial intelligence model, and the artificial intelligence model with a vector as input data cannot be explained.

[0004] Therefore, a solution is needed that can realize visual explanation of an artificial intelligence model with a vector input and obtain a saliency map that is easy to understand. SUMMARY

[0005] Therefore, the purpose of the present application is to provide a method for obtaining an explainable result of a game reinforcement learning model and related equipment.

[0006] To achieve the above purpose, the present application provides a method for obtaining an explainable result of a game reinforcement learning model, comprising:

[0007] inputting original data with a multi-dimensional state into the game reinforcement learning model to obtain a first expected return of each of a plurality of actions;

[0008] for each dimension of the multi-dimensional state, obtaining a plurality of perturbation data from the original data by adding noise to the dimension, obtaining a second expected return of each of the plurality of actions by inputting each of the perturbation data into the game reinforcement learning model, and calculating an importance of the dimension for each of the plurality of actions based on the first expected return and the second expected return;

[0009] statistically processing the importance to form an explainable result of a state-to-action mapping strategy of the game reinforcement learning model.

[0010] Further, the method for obtaining an explainable result of a game reinforcement learning model further comprises:

[0011] presenting the explainable result in the form of a saliency map.

[0012] Further, obtaining a first expected return of each of a plurality of actions by inputting original data with a multi-dimensional state into the game reinforcement learning model comprises:

[0013] determining a target action to be explained and other actions except the target action from all the actions;

[0014] outputting, according to the target action, a first target expected return between the original data and the target action by using the game reinforcement learning model;

[0015] outputting, according to the other actions, a first other expected return between the original data and each of the other actions by using the game reinforcement learning model;

[0016] taking the first target expected return and all the first other expected returns as first expected returns corresponding to the original data.

[0017] Further, the second expected return of each of the plurality of actions is obtained by inputting each of the perturbed data into the game reinforcement learning model, comprising:

[0018] For each of the perturbed data, performing operations:

[0019] outputting, according to the target action, a second target expected return between the perturbed data and the target action by using the game reinforcement learning model;

[0020] outputting, according to the other actions, a second other expected return between the perturbed data and each of the other actions by using the game reinforcement learning model;

[0021] taking the second target expected return and all the second other expected returns as second expected returns corresponding to the perturbed data.

[0022] Further, the importance of the dimension state to each of the plurality of actions is calculated, comprising:

[0023] calculating the influence of the perturbed dimension state on the target action to be explained, and determining the specificity of the dimension state to the target action;

[0024] calculating the influence of the perturbed dimension state on the other actions, and determining the relevance of the dimension state to the other actions;

[0025] fusing the specificity and the relevance to obtain the importance of the dimension state to the target action.

[0026] Further, the influence of the perturbed dimension state on the target action is calculated, comprising:

[0027] determining, according to the determined target action, a first target probability distribution of the game reinforcement learning model outputting the first target expected return;

[0028] And determine the second objective probability distribution of the expected reward of the second objective output by the game reinforcement learning model;

[0029] Calculate a first difference between the first target probability distribution and each of the second target probability distributions, and determine the influence of the disturbed state in that dimension of the original data on the target action based on the first difference.

[0030] Furthermore, the impact of the perturbed state in this dimension on the target action to be interpreted is calculated, including:

[0031] For the determined target action, determine the target value difference between the first target expected reward and the second target expected reward output by the game reinforcement learning model;

[0032] For each of the other actions, determine the other value difference between the first other expected reward and the second other expected reward output by the game reinforcement learning model;

[0033] Calculate a second difference between the target value difference and each of the other value differences, and determine the impact of the perturbed state of that dimension in the original data on the target action based on the second difference.

[0034] Further, determining the specificity of this dimension state for the target action includes:

[0035] When it is determined that the influence of the disturbed state of a certain dimension on the target action is positively correlated with the first difference and / or the second difference, then it is determined that the specificity of the state of that dimension to the target action is positively correlated with the first difference and / or the second difference.

[0036] Furthermore, the effect of the disturbed state in this dimension on the other actions is calculated, including:

[0037] For each of the other actions, determine a first other probability distribution for the game reinforcement learning model to output the first other expected reward;

[0038] And determine the second other probability distribution of the second other expected reward output by the game reinforcement learning model;

[0039] Based on the relative entropy between the first other probability distribution and the second other probability distribution, determine the impact of the disturbed state in that dimension of the original data on all the other actions.

[0040] Further, determining the relevance of this dimension state to the other actions includes:

[0041] determining that the relevance of the dimension state to the other action is negatively correlated with the relative entropy when it is determined that the influence of the dimension state that is disturbed on the other action is positively correlated with the relative entropy.

[0042] Further, fusing the specificity and the relevance to obtain the importance of the dimension state to the target action, including:

[0043] normalizing the relative entropy of the dimension state in the original data;

[0044] calculating the harmonic mean of the first difference value and the normalized relative entropy to obtain the importance of the dimension state to the target action;

[0045] and / or calculating the harmonic mean of the second difference value and the normalized relative entropy to obtain the importance of the dimension state to the target action.

[0046] Further, by statistically processing the importance, forming an interpretable result of the state-to-action mapping strategy of the game reinforcement learning model, including:

[0047] for each dimension state in the original data, determining the importance to each of the actions;

[0048] for each of the actions, ranking the dimension states in a monotonically decreasing order according to the calculated values of the importance, and scoring the dimension states in a monotonically decreasing order;

[0049] in response to the score being higher, determining that the influence of the dimension state on the action is greater.

[0050] Further, presenting the interpretable result in the form of a saliency map, including:

[0051] generating different colors for each dimension state in the saliency map according to the influence of each dimension state on each of the actions.

[0052] Further, after presenting the interpretable result in the form of a saliency map, further including:

[0053] for each dimension state, performing an operation:

[0054] generating a saliency subgraph about the specificity and the relevance for the dimension state according to the size of the specificity and the relevance between the dimension state and each of the actions.

[0055] Based on the same inventive concept, the application also provides a device for obtaining an interpretable result of a game reinforcement learning model, including a preprocessing module, a perturbation module, and an explanation module.

[0056] The preprocessing module is configured to obtain a first expected return of each action by inputting original data with a multi-dimensional state into the game reinforcement learning model.

[0057] The perturbation module is configured to obtain a plurality of pieces of perturbation data from the original data by adding noise to each dimension of the multi-dimensional state, obtain a second expected return of each action by inputting each piece of the perturbation data into the game reinforcement learning model, and calculate an importance of the dimension of the state to each action based on the first expected return and the second expected return.

[0058] The explanation module is configured to form an interpretable result of a state-to-action mapping strategy of the game reinforcement learning model by performing statistics on the importance.

[0059] Based on the same inventive concept, the present application further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor implements the method for obtaining an interpretable result of a game reinforcement learning model according to any one of the above embodiments when executing the program.

[0060] Based on the same inventive concept, the present application further provides a non-transitory computer readable storage medium, wherein the non-transitory computer readable storage medium stores computer instructions for causing the computer to execute the method for obtaining an interpretable result of a game reinforcement learning model.

[0061] As can be seen from the above, the method for obtaining an interpretable result of a game reinforcement learning model and related devices provided by the present application obtain the expected reward of each action after the perturbation of each dimension of the state, i.e., the second expected return, based on the perturbation of the input original data, and determine the importance of the perturbed dimension of the state to each action by using the first expected return, and on the basis of the original data and the obtained perturbation data being both vector data, the importance of each dimension of the state to each action in the original data is obtained, and the importance is visually interpreted in the form of a saliency map, thereby solving the problem of the game reinforcement learning model that cannot be interpreted with vector input. BRIEF DESCRIPTION OF DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the present application or related art, the following will briefly introduce the drawings needed to be used in the embodiments or related art descriptions. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0063] Figure 1 A flowchart of a method for obtaining an interpretable result of a game reinforcement learning model according to an embodiment of the present application;

[0064] Figure 2 A flowchart of a method for calculating a perturbation result according to an embodiment of the present application;

[0065] Figure 3 A flowchart of a method for calculating an influence on a target action based on a first difference value according to an embodiment of the present application;

[0066] Figure 4 A flowchart of a method for calculating an influence on a target action based on a second difference value according to an embodiment of the present application;

[0067] Figure 5 A flowchart of a method for calculating an influence on other actions according to an embodiment of the present application;

[0068] Figure 6 A flowchart of a method for generating a saliency map according to an embodiment of the present application;

[0069] Figure 7 A structural schematic diagram of an apparatus for obtaining an interpretable result of a game reinforcement learning model according to an embodiment of the present application;

[0070] Figure 8 A structural schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0071] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to specific embodiments and with reference to the accompanying drawings.

[0072] It should be noted that, unless otherwise defined, technical terms or scientific terms used in the embodiments of the present application should be understood as their common meanings to those having ordinary skills in the art to which the present application pertains. The terms “first”, “second”, and similar terms used in the embodiments of the present application do not represent any order, number, or importance, but are only used to distinguish different components. The terms “comprise”, “include”, and similar terms mean that the elements or objects before the terms encompass the elements or objects listed after the terms and their equivalents, and do not exclude other elements or objects. The terms “connect” and “connected” and similar terms do not mean only physical or mechanical connections, but can also include electrical connections, whether direct or indirect.

[0073] As described in the background section, the related explanation methods of artificial intelligence models are still difficult to meet the needs of visual explanation in actual production.

[0074] The applicant has found that the main problems of the related explanation methods of artificial intelligence models are as follows:

[0075] In interpretable techniques for related artificial intelligence models, saliency maps are often used to make the results of the visualization more intuitive and easier to understand.

[0076] However, in the visualization and interpretation methods of saliency maps, especially in perturbation-based saliency map methods, they can often only interpret artificial intelligence models that use images as input data. However, in a wider range of scenarios, artificial intelligence models that use vectors as input data, such as reinforcement learning models, cannot be visualized and interpreted using existing methods.

[0077] The embodiments of this application are described in detail below with reference to the accompanying drawings.

[0078] refer to Figure 1 An embodiment of this application describes a method for obtaining interpretable results from a game reinforcement learning model, comprising the following steps:

[0079] Step S101: By inputting the original data with multidimensional states into the game reinforcement learning model, the first expected reward for each of the multiple actions is obtained.

[0080] In the embodiments of this application, the model to be explained that needs to be visualized can be any type of trained artificial intelligence model (hereinafter referred to as "model"). In this embodiment, a game reinforcement learning model about deep reinforcement learning is used as a specific example of the model to be explained.

[0081] The game reinforcement learning model can handle both image and vector inputs. In this embodiment, vector input is used as an example, and a multiplayer game scenario is used as a specific application example of the game reinforcement learning model.

[0082] Specifically, the game scenario of multiplayer battles is used as the environment for the game reinforcement learning model. Multiple raw data points are input into the game reinforcement learning model. The game reinforcement learning model interacts with the environment and outputs actions corresponding to each raw data point. In other words, a large number of vector data pairs can be obtained: <1 raw data point, action>. Each raw data point input into the game reinforcement learning model can include multiple states, such as: the health value of a specified character in the game scenario, whether the character's skills are available, whether the character is invisible, and the distance between the character and the enemy. The actions output by the game reinforcement learning model can be the character's movement, skill activation, return to base, and attack.

[0083] In this context, for the original data pair as a vector, the change of each state can be used as the dimension of that state, and the dimensions of all states can be used as the state space of the original data pair; furthermore, all actions can be used as the action space of the vector data pair.

[0084] Further, taking one data pair as an example, based on the input of 1 original data, the game reinforcement learning model can also output the expected return between the original data and each action.

[0085] In this embodiment, based on the above-mentioned pre-set game reinforcement learning model, the prepared multiple original data are input into the model, wherein each original data is an original undisturbed data, and each original data is represented as s.

[0086] Further, taking one original data as an example, each state in s is regarded as an undisturbed state, and the total dimension of the state space is represented as n; among the output actions, the action that needs to be explained is determined, the action to be explained is referred to as a target action, and is represented as a; the remaining actions are referred to as other actions, and are represented as a; and the total dimension of the action space is represented as p; among the obtained data pairs, the data pair related to the target action is represented as , and the data pair related to the other action is represented as .

[0087] Further, in the expected return output by the game reinforcement learning model, the expected return of any action in the original data s is selected as the first expected return, wherein among all the first expected returns, the first expected return corresponding to the target action is referred to as the first target expected return, and is represented as , and the first expected return corresponding to the other action is referred to as the first other expected return, and is represented as Q(s, a).

[0088] In step S102, for each dimension of the multi-dimensional state, a plurality of perturbation data are obtained from the original data by adding noise to the dimension of the state, a second expected return of each of the plurality of actions is obtained by inputting each of the perturbation data into the game reinforcement learning model, and the importance of the dimension of the state to each of the plurality of actions is calculated based on the first expected return and the second expected return.

[0089] In the embodiments of the present application, based on the above-mentioned pre-set game reinforcement learning model and the input of multiple original data, each original data needs to be disturbed, and the perturbation data obtained after the disturbance is input into the game reinforcement learning model to obtain the expected return output after the disturbance of each original data.

[0090] In this embodiment, taking one original data as an example, the disturbance process needs to traverse each dimension of the original data, and only one dimension of the state is disturbed in each disturbance, that is, each dimension of the state needs to be disturbed once, and thus the influence of each dimension of the state on the action can be obtained.

[0091] Specifically, in the embodiment, Gaussian noise can be selected to disturb the current state to be explained, and the disturbed data obtained is taken as one piece of disturbance data. After each dimension of the state of the original data is disturbed, a plurality of pieces of disturbance data can be obtained, and each piece of disturbance data is denoted as s', wherein each piece of disturbance data corresponds to a disturbed state of one dimension.

[0092] Further, all the obtained disturbance data are input into the model. Taking one piece of disturbance data as an example, a data pair <one piece of disturbance data, action> output by the model can be obtained, wherein in the obtained data pair, the data pair related to the target action is denoted as and the data pairs related to other actions are denoted as <s', a>.

[0093] Further, based on the input piece of disturbance data, the model can also output the expected return of each action selected by the piece of disturbance data, and the expected return is taken as the second expected return.

[0094] wherein the second expected return corresponding to the target action is taken as the second target expected return, and is denoted as wherein the second expected return corresponding to other actions is taken as the second other expected return, and is denoted as Q(s', a).

[0095] Further, based on the model, a plurality of second target expected returns and a plurality of second other expected returns corresponding to each piece of disturbance data can be obtained.

[0096] In the embodiment, based on the obtained first expected return and second expected return, the comprehensive influence of each dimension of the state on the target action can be determined in each piece of original data, that is, the mapping relationship between the dimension of the state and the target action is explained. By taking each action as the target action in turn, the mapping relationship between the dimension of the state and each action can be obtained, and in the embodiment, the mapping relationship is regarded as the importance of the dimension of the state to the target action.

[0097] Specifically, in one embodiment of the application, for each dimension of the state, the comprehensive influence of the dimension of the state on the target action can be divided into specificity and correlation, wherein the specificity describes the influence of the dimension of the state on the target action only, and the greater the influence of the dimension of the state on the current target action, the higher the specificity; the correlation describes the influence of the dimension of the state on other actions, and the greater the influence of the dimension of the state on other actions, the smaller the influence of the dimension of the state on the current target action, and the lower the correlation.

[0098] Further, the specificity and the correlation can be fused to obtain the importance described above.

[0099] In some other embodiments, the importance of the state to each action can also be regarded as the degree of correlation between the two, and the degree of correlation can be quantified by a set evaluation index, for example, the degree of correlation between the state and the action can be described by calculating the Pearson correlation coefficient.

[0100] Specifically, for the target action, the second target expected return between each state and the action can be determined in the manner of the foregoing embodiments.

[0101] Further, for each state and the target action, the Pearson correlation coefficient can be calculated by using the covariance between the second target expected return and the target action on the basis of the quantification of the target action.

[0102] Further, by calculating the Pearson correlation coefficient, a linear relationship between the two can be constructed,

[0103] Specifically, the value of the Pearson correlation coefficient can be between 0 and 1, and the greater the value, the stronger the degree of correlation between the two, that is, the higher the importance of the state to the target action; if the Pearson correlation coefficient is equal to 0, it means that there is no degree of correlation between the two, that is, the lowest degree of correlation.

[0104] In some other embodiments, other evaluation indexes known to those skilled in the art can also be used to determine the degree of correlation, such as the Spearman correlation coefficient, the Kendall correlation coefficient, or other rank correlation coefficients, and the evaluation index and its calculation method are not specifically limited in the present application.

[0105] In one embodiment of the present application, as shown in Figure 2 the calculation of the importance of the state to each of the actions includes the following steps:

[0106] Step S201, calculate the influence of the disturbed state on the target action to be explained, and determine the specificity of the state to the target action.

[0107] In this step, for each piece of original data, the specificity of each state is determined by disturbing the state of each dimension respectively.

[0108] In the present embodiment, taking the state of one dimension as an example, after the state of the dimension is disturbed by the above steps, one piece of disturbance data corresponding to the state of the dimension can be obtained, that is, for each state in the disturbance data, only the state of one dimension is different from the original data, and it can be considered that the disturbance data s' obtained after the disturbance is similar to the original data s before the disturbance.

[0109] Further, based on the similar original data s and the perturbed data s', the influence of the perturbed state on the target action, i.e., the specificity, is determined.

[0110] In one embodiment of the present application, as shown in Figure 3 the specificity of the perturbed state of the dimension and the target action specifically includes the following steps:

[0111] Step S301, according to the determined target action, determining a first target probability distribution of the first target expected return output by the game reinforcement learning model.

[0112] In this step, first, any one action is selected from all actions as a target action, and the remaining actions except the target action are selected as other actions.

[0113] Further, the first target expected return and the second target expected return are determined from the first expected return and the second expected return output by the model in the above step.

[0114] Further, based on the determined target action, the probability distribution of the game reinforcement learning model outputting the first target expected return can be determined as the first target probability distribution, specifically, it can be calculated according to the following formula:

[0115]

[0116] wherein, as the first target probability distribution.

[0117] Step S302, determining a second target probability distribution of the second target expected return output by the game reinforcement learning model.

[0118] In this step, based on the determined target action, the probability distribution of the game reinforcement learning model outputting the second target expected return can be determined as the second target probability distribution, specifically, it can be calculated according to the following formula:

[0119]

[0120] wherein, the second target probability distribution is represented as

[0121] Step S303, calculating a first difference value between the first target probability distribution and each of the second target probability distributions, and according to the first difference value, determining the influence of the perturbed state of the dimension in the original data on the target action.

[0122] In this step, according to the first target probability distribution and the second target probability distribution determined in the above steps, a first difference value between the two is calculated, and the first difference value is taken as the influence of the disturbed dimension state on the target action.

[0123] Specifically, the calculation can be performed according to the following formula:

[0124]

[0125] Wherein, the first difference value is denoted as Δp.

[0126] It can be seen that due to the intervention of the disturbance, the expected return of the dimension state on the target action (i.e. the first target expected return) changes with the expected return of the disturbed state on the same target action (i.e. the second target expected return), and the current change degree is also the influence of the disturbed dimension state on the target action.

[0127] Therefore, the first difference value can be taken as a quantitative representation of the change degree of the expected return.

[0128] It can be known that when the above change degree, i.e. the first difference value, is greater, the influence of the disturbed dimension state on the target action is greater; and when the first difference value is smaller, the influence of the disturbed dimension state on the target action is smaller.

[0129] In some embodiments, since the calculation of the first difference value is for a certain target action, the first difference value can be taken as an auxiliary quantitative reference of the specificity of the dimension state on the target action.

[0130] Specifically, when the first difference value is greater, the specificity of the dimension state on the target action is higher; and when the first difference value is smaller, the specificity of the dimension state on the target action is smaller.

[0131] In other embodiments, the first difference value can also be equivalent to the specificity of the dimension state on the target action.

[0132] Reference Figure 4 In another embodiment of the present application, the specificity of the disturbed dimension state on the target action can also be directly determined by using the first expected return and the second expected return, specifically including the following steps:

[0133] Step S401, for a certain target action, determining a target value difference between the first target expected return and the second target expected return output by the game reinforcement learning model.

[0134] In this step, first, any one action is selected from all actions as a target action, and the remaining actions other than the target action are taken as other actions.

[0135] Further, the first target expected return and the second target expected return are determined from the first expected return and the second expected return output by the model in the above step.

[0136] Further, a target value difference between the first target expected return and the second target expected return is calculated according to the following formula:

[0137]

[0138] wherein ΔQ1 is the target value difference.

[0139] In step S402, for each of the other actions, an other value difference between the first other expected return and the second other expected return output by the game reinforcement learning model is determined.

[0140] In this step, the target action and the other actions are determined based on the above.

[0141] Further, the first other expected return and the second other expected return are determined from the first expected return and the second expected return output by the model in the above step.

[0142] Further, an other value difference between the first other expected return and the second other expected return is calculated according to the following formula:

[0143] ΔQ2 = Q(s, a) - Q(s', a)

[0144] wherein ΔQ2 is the target value difference.

[0145] In step S403, a second difference value between the target value difference and each of the other value differences is calculated, and according to the second difference value, an influence of the dimension of the state that is disturbed in the original data on the target action is determined.

[0146] In this step, the difference value ΔQ = ΔQ1 - ΔQ2 between the target value difference and the other value difference determined in the above step is taken as the second difference value, and the second difference value is taken as the influence of the dimension of the state that is disturbed on the target action.

[0147] Specifically, due to the intervention of the disturbance, the expected return of the dimension of the state on the target action (i.e., the first target expected return) changes from the expected return of the state after the disturbance on the same target action (i.e., the second target expected return), and the current change amount is the target value difference.

[0148] Further, the change between the expected return of the state in the dimension for the target action (i.e., the first other expected return) and the expected return of the state after the disturbance for the same other action (i.e., the second other expected return) is the other value difference.

[0149] It can be seen that, if the change in the expected return of the target action caused by the disturbance on the state in the dimension is greater than the change in the expected return of the other action, it can be determined that the state in the dimension after the disturbance has higher specificity for the target action; and if the change in the expected return of the target action caused by the disturbance on the state in the dimension is less than the change in the expected return of the other action, it can be determined that the state in the dimension after the disturbance has lower specificity for the target action.

[0150] In some embodiments, the second difference value can also be equivalent to the specificity of the state in the dimension for the target action.

[0151] In other embodiments, the second difference value can be used as an auxiliary quantitative reference for the specificity of the state in the dimension for the target action.

[0152] Specifically, a specificity coefficient can be set for the specificity, and based on the specificity coefficient, a linear relationship between the second difference value and the specificity can be described by the following formula:

[0153] Φ = ΔQ x γ

[0154] Wherein, Φ represents the value of the calculated specificity, and γ represents the set specificity coefficient.

[0155] In the embodiments of the present application, the specificity coefficient can be obtained according to expert experience or provided by a calibration engineer.

[0156] It can be seen that, in the present embodiment, the second difference value is not directly equivalent to the value of the specificity, but since there is a certain linear relationship between the two, the second difference value can still be regarded as a reference quantity for measuring the specificity, and a specific operation relationship between the two can be constructed in the case of setting the specificity coefficient as described above.

[0157] Step S202, calculate the influence of the disturbed state in the dimension on the other action, and determine the correlation of the state in the dimension for the other action.

[0158] In the present step, for each piece of original data, the correlation of each dimension state is determined by respectively disturbing the state in each dimension.

[0159] In the present embodiment, taking the state in one dimension as an example, after the state in the dimension is disturbed by the above steps, one piece of disturbance data corresponding to the state in the dimension can be obtained.

[0160] Further, based on the similar original data s and the perturbed data s', the influence of the perturbed dimension state on other actions, i.e., the relevance to the target action, is determined.

[0161] In one embodiment of the present application, as shown in Figure 5 The determination of the relevance of the dimension state to the target action specifically includes the following steps:

[0162] Step S501, for each of the other actions, determine a first other probability distribution of the first other expected return output by the game reinforcement learning model.

[0163] In this step, based on the target action and the other actions determined in the above step S301 or step S401, and the first other expected return and the second other expected return determined, the probability distribution of the first other expected return output by the game reinforcement learning model can be determined as the first other probability distribution, specifically, it can be calculated according to the following formula:

[0164]

[0165] wherein P rem (s, a) is the first other probability distribution.

[0166] Step S502, determine a second other probability distribution of the second other expected return output by the game reinforcement learning model.

[0167] In this step, based on the determined target action, the probability distribution of the second other expected return output by the game reinforcement learning model can be determined as the second other probability distribution, specifically, it can be calculated according to the following formula:

[0168]

[0169] wherein the second other probability distribution is represented as P rem (s', a).

[0170] Step S503, according to the relative entropy between the first other probability distribution and the second other probability distribution, determine the influence of the perturbed dimension state in the original data on all the other actions.

[0171] In this step, due to the intervention of the above perturbation, the expected return of the dimension state on other actions (i.e., the first expected return) and the expected return of the perturbed state on other actions (i.e., the second expected return) change, and the current degree of change is the influence of the perturbed dimension state on other actions.

[0172] In the embodiment, the variation degree of the expected return is described by the discrete degree between the first other probability distribution and the second other probability distribution.

[0173] Specifically, the variation degree of the expected return can be described by the relative entropy, i.e., the KL divergence, and the relative entropy is denoted as D KL .

[0174] It can be seen that when the relative entropy is smaller, the first other probability and the second other probability are more consistent, i.e., the difference is smaller, and the influence of the disturbed state on the other actions is smaller, and when the relative entropy is larger, the influence of the disturbed state on the other actions is larger.

[0175] In the embodiment, the relativity is designed to describe the influence on the other actions, and specifically, the relativity describes the relative influence of the disturbed state on the target action and the other actions.

[0176] Specifically, when the influence of the disturbed state on the other actions is smaller, the influence on the target action is larger, and the correlation with the other actions is smaller, and the correlation with the target action is larger, and when the influence of the disturbed state on the other actions is larger, the influence on the target action is smaller, and the correlation with the other actions is larger, and the correlation with the target action is smaller.

[0177] In the embodiment, the relative entropy is taken as the quantification of the correlation of the disturbed state with the other actions and the target action.

[0178] In step S203, the relativity and the specificity are fused to obtain the importance degree of the state to the target action.

[0179] In the step, the specificity obtained in step S201 and the relativity obtained in step S202 are taken as the disturbance result of the disturbed first state to the target action.

[0180] That is, the disturbance result is the common evaluation of the first state from the specificity and the relativity.

[0181] Further, the specificity and the relativity determined in step S201 and step S202 can be further fused to obtain the comprehensive influence, i.e., the importance degree, of the disturbed state to the target action, wherein the first difference or the second difference can be taken as the quantification of the specificity, and the relative entropy can be taken as the quantification of the relativity.

[0182] Specifically, based on the target action and other actions determined in step S301, the relative entropy of the disturbed state is normalized to ensure that the values can be kept in the same scale when the fusion calculation of the specificity and the correlation is performed. Specifically, the normalization method is shown in the following formula:

[0183]

[0184] wherein K represents the normalization result of the relative entropy.

[0185] Further, the harmonic mean of the specificity and the correlation is calculated, and the harmonic mean is taken as the importance of the disturbed state of the dimension to the target action, and is denoted as S(f).

[0186] In some embodiments, the first difference and the relative entropy can also be used to perform the calculation of the following formula to obtain the harmonic mean:

[0187]

[0188] In another embodiment of the present application, the second difference and the relative entropy can also be used to perform the calculation of the following formula to obtain the harmonic mean:

[0189]

[0190] Further, according to the above method of calculating the importance, a plurality of importance of each dimension of the state can be obtained in each original data, and the mapping strategy of the state to the action of the game reinforcement learning model can be explained according to the importance, and a saliency map about each dimension of the state can be further generated.

[0191] In some other embodiments, considering that the specificity and the correlation have different degrees of influence on the correctness of the evaluation result when evaluating the influence of the state on the target action, for example, in some cases, the specificity accounts for a more important proportion in the evaluation of the comprehensive influence of the state on the target action, therefore, different weights can be set for the specificity and the correlation according to the expert experience, and the two weights satisfy the following constraint condition:

[0192] W α +W β =1

[0193] wherein W α represents the weight of the specificity, and W β represents the weight of the correlation.

[0194] Step S103, by statistically processing the importance, an interpretable result of the state to action mapping strategy of the game reinforcement learning model is formed.

[0195] In this step, based on the importance degrees obtained in the above step S102, the importance degrees of each action for each dimension can be counted by ranking.

[0196] In an embodiment of the present application, as shown in Figure 6 The forming of the interpretable result of the state-to-action mapping strategy of the game reinforcement learning model by counting the importance degrees can include the following steps:

[0197] Step S601, for each state in the original data, determine the importance degree of each action.

[0198] In this step, based on the above calculation of the importance degree, it can be seen that the calculation result of the importance degree represents the importance of a state of a dimension to a target action. Further, the remaining actions need to be taken in turn as the target action, and the above steps S201, S202 and S203 are repeated to calculate the importance degree of the state of the dimension to the current target action when each action is taken as the target action. That is, the number of times of repeating the importance degree calculation should be the same as the dimension p of the action space, so as to obtain the importance degree of the first state of the dimension to each action, that is, p importance degrees are obtained.

[0199] Further, for each piece of original data, each state in it needs to be perturbed. Since the dimension of the state space in each piece of original data is n, that is, n times of perturbation are needed for each piece of original data, and the above p times of importance degree calculation is performed after each state is perturbed.

[0200] Further, for each piece of original data, based on the above n times of perturbation and n x p times of importance degree calculation, n x p importance degrees can be obtained for each piece of original data corresponding to n-dimensional states and p actions.

[0201] Step S602, for each action, rank the states in a monotonically decreasing order of the calculated importance degrees, and score the states in a monotonically decreasing order.

[0202] In this step, each of the p actions needs to be traversed, and based on each action, the calculation results of the importance degrees of the states for the action are counted and ranked.

[0203] Specifically, for any one of the p actions, the calculation results of the importance degrees of the states for the action are compared, and the states are ranked in a monotonically decreasing order.

[0204] In some embodiments, a predetermined number of states can be selected from all the states, for example, the top n states in the ranking, to reduce the redundancy in the statistical process.

[0205] Further, the states that complete the ranking are scored, specifically, the score of the state ranked first is evaluated as 1, and the scores of the other states ranked after the first are evaluated as: to form a result of scores decreasing in turn; wherein the bit number represents the position of the state in the ranking, for example, the state ranked second has a bit number of 2 and a score of

[0206] Step S603, in response to the higher the score, determining that the state has a greater impact on the action.

[0207] In this step, based on the ranking in the above step S602, the order of the impact of each state on the target action in terms of degree can be determined, and the saliency map of each state of the action is generated according to the impact on the target action.

[0208] Specifically, according to the above scoring and ranking of each state, it can be seen that based on the selected target action, when the score of the state is higher, that is, the importance of the state to the target action is higher, then the impact of the state on the target action is greater; when the score of the state is lower, that is, the importance of the state to the target action is lower, then the impact of the state on the target action is smaller.

[0209] Further, according to the degree of impact of each state on the target action, the saliency map of each state of the target action in the original data is generated.

[0210] Specifically, in the saliency map, each state is assigned a different color according to the degree of impact, for example, the state that has a great impact on the target action can be red, the state that has a small impact can be blue, etc., in some other embodiments, different shades in the same color system can also be used to represent the different degrees of impact of each state, for example, the state that has a great impact can be a deep color, and the state that has a small impact can be a light color, etc.

[0211] Further, all p-dimensional actions are traversed, and each action is taken in turn as a target action to obtain the saliency map of each state in the original data based on each action.

[0212] Further, for all the original data, the above steps S602 and S603 can be repeated to obtain the saliency map of each action for each piece of original data.

[0213] In another embodiment of the present application, based on the quantification results of the specificity and the correlation obtained in steps S201 and S202, a saliency map subgraph about the specificity and the correlation can be generated according to the quantification results.

[0214] Specifically, for each dimension state in the original data, based on the determined target action, a saliency map of different colors or different color depths can be generated for each dimension state according to the size of the specificity of each dimension state, i.e., the size of the first difference or the second difference, to visually explain the specificity of each dimension state for the target action.

[0215] Further, a saliency map of different colors or different color depths can be generated for each dimension state according to the size of the correlation of each dimension state, i.e., the size of the relative entropy or the size of the normalized relative entropy, to visually explain the correlation of each dimension state for the target action.

[0216] It can be seen that the method for obtaining an interpretable result of a game reinforcement learning model in the embodiments of the present application is based on perturbing the input original data to obtain the expected reward of each action after perturbation of each dimension state, i.e., the second expected return, and determining the importance of the perturbed dimension state for each action by the first expected return determined in advance. On the basis that the input original data of the game reinforcement learning model and the obtained perturbed data are both vector data, the importance of each dimension state in the original data for each action is obtained, and the importance is visually explained in the form of a saliency map, solving the problem of the inability to interpret the vector input of the game reinforcement learning model.

[0217] It should be noted that the method of the embodiments of the present application can be executed by a single device, such as a computer or a server, etc. The method of the present embodiment can also be applied in a distributed scenario, and completed by multiple devices cooperating with each other. In this distributed scenario, one of the multiple devices can only execute one or more steps in the method of the embodiments of the present application, and the multiple devices can interact with each other to complete the method.

[0218] It should be noted that some embodiments of the present application have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order than the order described above and still achieve the desired results. Also, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0219] Based on the same inventive concept, embodiments of the present application also provide a device for obtaining an interpretable result of a game reinforcement learning model, corresponding to any of the above-mentioned embodiment methods.

[0220] Reference Figure 7 The device for obtaining an interpretable result of a game reinforcement learning model comprises a preprocessing module 701, a perturbation module 702, and an explanation module 703.

[0221] The preprocessing module 701 is configured to obtain a first expected return of each of a plurality of actions by inputting original data having a multi-dimensional state into the game reinforcement learning model.

[0222] The perturbation module 702 is configured to, for each dimension of the multi-dimensional state, obtain a plurality of perturbation data from the original data by adding noise to the dimension, obtain a second expected return of each of the plurality of actions by inputting each of the perturbation data into the game reinforcement learning model, and calculate an importance of the dimension for each of the plurality of actions based on the first expected return and the second expected return.

[0223] The explanation module 703 is configured to form an interpretable result of a state-to-action mapping strategy of the game reinforcement learning model by performing statistics on the importance, wherein the interpretable result can be presented in the form of a saliency map.

[0224] As an optional embodiment, the preprocessing module 701 is specifically configured to:

[0225] determine a target action to be explained and other actions except the target action from all the actions;

[0226] output a first target expected return between the original data and the target action by using the game reinforcement learning model according to the target action;

[0227] output a first other expected return between the original data and each of the other actions by using the game reinforcement learning model according to the other actions;

[0228] take the first target expected return and all the first other expected returns as the first expected return corresponding to the original data.

[0229] As an optional embodiment, the perturbation module 702 is specifically configured to:

[0230] for each of the perturbation data, perform an operation:

[0231] According to the target action, a second target expected return between the perturbation data and the target action is output by the game reinforcement learning model;

[0232] According to the other actions, a second other expected return between the perturbation data and each of the other actions is output by the game reinforcement learning model;

[0233] The second target expected return and all the second other expected returns are taken as a second expected return corresponding to the perturbation data of the corresponding strip.

[0234] Further, the influence of the perturbed dimension state on the target action to be explained is calculated, and the specificity of the dimension state to the target action is determined;

[0235] The influence of the perturbed dimension state on the other actions is calculated, and the relevance of the dimension state to the other actions is determined;

[0236] The specificity and the relevance are fused to obtain the importance of the dimension state to the target action.

[0237] Wherein, the influence of the perturbed dimension state on the target action to be explained includes:

[0238] According to the determined target action, a first target probability distribution of the game reinforcement learning model outputting the first target expected return is determined;

[0239] And a second target probability distribution of the game reinforcement learning model outputting the second target expected return is determined;

[0240] A first difference value between the first target probability distribution and each of the second target probability distributions is calculated, and according to the first difference value, the influence of the perturbed dimension state in the original data on the target action is determined.

[0241] In some other embodiments, the influence of the perturbed dimension state on the target action to be explained can also include:

[0242] For the determined target action, a target value difference between the first target expected return and the second target expected return output by the game reinforcement learning model is determined;

[0243] For each of the other actions, an other value difference between the first other expected return and the second other expected return output by the game reinforcement learning model is determined;

[0244] A second difference value between the target value difference and each of the other value differences is calculated, and according to the second difference value, the influence of the perturbed dimension state in the original data on the target action is determined.

[0245] It can be seen that determining the specificity of the dimension state to the target action comprises:

[0246] When the first difference value and / or the second difference value is greater, it can be determined that the influence of the disturbed dimension state on the target action is greater, and it can be determined that the specificity of the dimension state to the target action is higher at this time.

[0247] Further, calculating the influence of the disturbed dimension state on the other actions comprises:

[0248] For each of the other actions, determining a first other probability distribution of the game reinforcement learning model outputting the first other expected return;

[0249] And determining a second other probability distribution of the second other expected return output by the game reinforcement learning model;

[0250] According to the relative entropy between the first other probability distribution and the second other probability distribution, determining the influence of the disturbed dimension state in the original data on all the other actions.

[0251] It can be seen that when the relative entropy is greater, it can be determined that the influence of the disturbed dimension state on the other actions is greater, and it can be determined that the relevance of the dimension state to the target action is lower at this time.

[0252] Further, fusing the specificity and the relevance to obtain the importance of the dimension state to the target action comprises:

[0253] Normalizing the relative entropy of the dimension state in the original data;

[0254] Calculating the harmonic mean of the first difference value and the normalized relative entropy to obtain the importance of the dimension state to the target action;

[0255] And / or calculating the harmonic mean of the second difference value and the normalized relative entropy to obtain the importance of the dimension state to the target action.

[0256] As an optional embodiment, the interpretation module 703 is specifically configured to:

[0257] For each dimension state in the original data, determining the importance to each of the actions;

[0258] For each of the actions, ranking the dimension states in a monotonically decreasing order according to the calculated values of the importance, and scoring the dimension states in the monotonically decreasing order;

[0259] The higher the score, the greater the influence of the state on the action.

[0260] Further, different colors are generated for each state in the saliency map according to the influence of each state on each action.

[0261] In some other embodiments, for each state, the following operations can also be performed to generate a saliency map subgraph:

[0262] A saliency map subgraph about the uniqueness and relevance between the state and each action is generated for the state according to the size of the uniqueness and relevance.

[0263] For the convenience of description, the above apparatus is described in various modules according to functions. Of course, the functions of the modules can be implemented in one or more software and / or hardware when implementing the embodiments of the present application.

[0264] The apparatus of the above embodiments is used to implement the method of obtaining an interpretable result of a game reinforcement learning model in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which are not described here.

[0265] Based on the same inventive concept, corresponding to the method of any of the above embodiments, the embodiments of the present application also provide an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method of obtaining an interpretable result of a game reinforcement learning model according to any of the above embodiments when executing the program.

[0266] Figure 8 A more specific hardware structure of an electronic device provided by the present embodiment is shown, which can include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 for communication within the device.

[0267] The processor 1010 can be implemented in the form of a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., for executing related programs to implement the technical solutions provided by the embodiments of the present application.

[0268] The memory 1020 can be implemented in the form of a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, etc. The memory 1020 can store an operating system and other application programs, and when the technical solutions provided in the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0269] The input / output interface 1030 is configured to connect an input / output module to realize information input and output. The input / output module can be configured as a component in the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. The input device can include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device can include a display, a speaker, a vibrator, an indicator light, etc.

[0270] The communication interface 1040 is configured to connect a communication module (not shown in the figure) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired manner (such as a USB, a network cable, etc.) or through a wireless manner (such as a mobile network, WIFI, Bluetooth, etc.).

[0271] The bus 1050 includes a channel to transmit information between various components (such as the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040) of the device.

[0272] It should be noted that although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040, and the bus 1050, in the specific implementation process, the device can also include other components necessary for normal operation. In addition, those skilled in the art can understand that the above device can also only include the components necessary to implement the embodiments of the present application, and does not have to include all the components shown in the figure.

[0273] The device of the above embodiment is used to implement the method of obtaining an interpretable result of a game reinforcement learning model in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which are not described here.

[0274] Based on the same inventive concept, corresponding to the method of any of the above embodiments, the present application also provides a non-transitory computer readable storage medium storing computer instructions for causing the computer to execute the method of obtaining an interpretable result of a game reinforcement learning model according to any of the above embodiments.

[0275] The computer readable media of the present embodiments includes permanent and non-permanent, removable and non-removable media can be implemented by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape disk storage or other magnetic storage device, or any other non-transmission medium that can be used to store information accessible to a computing device.

[0276] The storage medium of the above embodiments stores computer instructions for causing the computer to execute the method of obtaining an interpretable result of a game reinforcement learning model as described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which are not described here.

[0277] Those skilled in the art should understand that the discussion of any of the above embodiments is only exemplary and is not intended to imply that the scope of the present application (including claims) is limited to these examples; the above embodiments or technical features between different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes of the different aspects of the embodiments of the present application as described above. In order to be brief, they are not provided in detail.

[0278] In addition, in order to simplify the description and discussion, and so as not to make the embodiments of the present application difficult to understand, the well-known power / ground connections of integrated circuit (IC) chips and other components can or can not be shown in the provided drawings. In addition, the devices can be shown in the form of block diagrams in order to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented the embodiments of the present application (i.e. these details should be fully within the understanding of those skilled in the art). Where specific details (e.g. circuits) are set forth in order to describe the exemplary embodiments of the present application, it will be apparent to those skilled in the art that the embodiments of the present application can be implemented without these specific details or with variations of these specific details. Therefore, these descriptions should be considered illustrative rather than limiting.

[0279] While the present application has been described in connection with certain embodiments thereof, many modifications, substitutions, changes, and of forms will be apparent to those of ordinary skill in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) can use the embodiments discussed.

[0280] Embodiments of the present application are intended to embrace all such alterations, modifications, and variations that fall within the scope of the appended claims. Accordingly, the application is intended to be governed by the scope of the claims and their equivalents.

Claims

1. A method for obtaining an interpretable result of a game reinforcement learning model, characterized in that, The method comprises the following steps: inputting original data with multi-dimensional states into the game reinforcement learning model to obtain first expected returns of a plurality of actions respectively; for each dimension of the multi-dimensional states, obtaining a plurality of perturbation data from the original data by adding noise to the dimension, obtaining second expected returns of the plurality of actions respectively by inputting each piece of the perturbation data into the game reinforcement learning model, and calculating the importance of the dimension to the plurality of actions respectively based on the first expected returns and the second expected returns; wherein, the calculation of the importance of the dimension to the plurality of actions respectively comprises: calculating the influence of the perturbed dimension on a target action to be explained, and determining the specificity of the dimension to the target action; calculating the influence of the perturbed dimension on other actions, and determining the correlation of the dimension to the other actions; fusing the specificity and the correlation to obtain the importance of the dimension to the target action; statistically processing the importance to form an interpretable result of a state-to-action mapping strategy of the game reinforcement learning model.

2. The method of claim 1, wherein, Further comprising: presenting the interpretable result in the form of a saliency map.

3. The method of claim 1, wherein, The step of inputting original data with multi-dimensional states into the game reinforcement learning model to obtain first expected returns of a plurality of actions respectively comprises: determining a target action to be explained and other actions except the target action from all the actions; outputting a first target expected return between the original data and the target action by the game reinforcement learning model according to the target action; outputting a first other expected return between the original data and each of the other actions by the game reinforcement learning model according to the other actions; taking the first target expected return and all the first other expected returns as first expected returns corresponding to the original data.

4. The method of claim 3, wherein, The step of obtaining second expected returns of a plurality of actions respectively by inputting each piece of the perturbation data into the game reinforcement learning model comprises: for each piece of the perturbation data, performing the following operations: outputting a second target expected return between the perturbation data and the target action by the game reinforcement learning model according to the target action; outputting a second other expected return between the perturbation data and each of the other actions by the game reinforcement learning model according to the other actions; taking the second target expected return and all the second other expected returns as second expected returns corresponding to the perturbation data.

5. The method of claim 4, wherein, The step of calculating the influence of the perturbed dimension on a target action to be explained comprises: determining a first target probability distribution of the game reinforcement learning model outputting the first target expected return according to the determined target action; and determining a second target probability distribution of the game reinforcement learning model outputting the second target expected return; calculating a first difference value between the first target probability distribution and each of the second target probability distributions, and determining the influence of the perturbed dimension of the original data on the target action according to the first difference value.

6. The method of claim 4, wherein, The computing the influence of the dimension state being perturbed on the target action to be explained comprises: For the determined target action, determining a target value difference between a first target expected return and a second target expected return output by the game reinforcement learning model; For each of the other actions, determining an other value difference between a first other expected return and a second other expected return output by the game reinforcement learning model; Computing a second difference value between the target value difference and each of the other value differences, and determining the influence of the dimension state being perturbed in the original data on the target action according to the second difference value.

7. The method of claim 5, wherein, The determining the specificity of the dimension state on the target action comprises: When it is determined that the influence of the dimension state being perturbed on the target action is positively correlated with the first difference value, then it is determined that the specificity of the dimension state on the target action is positively correlated with the first difference value.

8. The method of claim 6, wherein, The determining the specificity of the dimension state on the target action comprises: when it is determined that the influence of the dimension state being perturbed on the target action is positively correlated with the second difference value, then it is determined that the specificity of the dimension state on the target action is positively correlated with the second difference value.

9. The method of claim 4, wherein, The computing the influence of the dimension state being perturbed on the other actions comprises: For each of the other actions, determining a first other probability distribution of the first other expected return output by the game reinforcement learning model; And determining a second other probability distribution of the second other expected return output by the game reinforcement learning model; According to the relative entropy between the first other probability distribution and the second other probability distribution, determining the influence of the dimension state being perturbed in the original data on all the other actions.

10. The method of claim 9, wherein, The determining the relevance of the dimension state on the other actions comprises: In response to the greater the relative entropy, the greater the influence of the dimension state being perturbed on the other actions, then the lower the relevance of the dimension state on the target action; When it is determined that the influence of the dimension state being perturbed on the other actions is positively correlated with the relative entropy, then it is determined that the relevance of the dimension state on the other action is negatively correlated with the relative entropy.

11. The method of claim 5, wherein, The fusing the specificity and the relevance to obtain the importance degree of the dimension state on the target action comprises: Normalizing the relative entropy between the first other probability distribution of the first other expected return and the second other probability distribution of the second other expected return of the dimension state in the original data; Computing a harmonic mean of the first difference value and the normalized relative entropy to obtain the importance degree of the dimension state on the target action.

12. The method of claim 6, wherein, The fusing the specificity and the relevance to obtain the importance degree of the dimension state on the target action comprises: Normalizing the relative entropy between the first other probability distribution of the first other expected return and the second other probability distribution of the second other expected return of the dimension state in the original data; Computing a harmonic mean of the second difference value and the normalized relative entropy to obtain the importance degree of the dimension state on the target action.

13. The method according to claim 11 or 12, characterized in that, The forming of the interpretable result of the state-to-action mapping strategy of the game reinforcement learning model by counting the importance degrees comprises: For each dimension of state in the original data, determining the importance degree of each action; For each action, ranking the dimensions of state in a monotonically decreasing order according to the calculated values of the importance degrees, and scoring the dimensions of state in the monotonically decreasing order to obtain scores of the dimensions of state; In response to the higher the score, the greater the influence of the dimension of state on the action.

14. The method of claim 2, wherein, The presenting of the interpretable result in the form of a saliency map comprises: According to the influence size of each dimension of state on each action, generating different colors for each dimension of state in the saliency map.

15. The method of claim 5 or 6, wherein, After presenting the interpretable result in the form of a saliency map, further comprising: For each dimension of state, performing operations: According to the size of the uniqueness and correlation between the dimension of state and each action, generating a saliency map subgraph about the uniqueness and correlation for the dimension of state.

16. An apparatus for obtaining an interpretable result of a game reinforcement learning model, comprising: A preprocessing module, a perturbation module, and an explanation module; The preprocessing module is configured to obtain first expected returns of a plurality of actions respectively by inputting original data with a multi-dimensional state into the game reinforcement learning model; The perturbation module is configured to, for each dimension of state in the multi-dimensional state, obtain a plurality of perturbation data from the original data by adding noise to the dimension of state, obtain second expected returns of the plurality of actions respectively by inputting each piece of the perturbation data into the game reinforcement learning model, and calculate an importance degree of the dimension of state for the plurality of actions respectively based on the first expected returns and the second expected returns; The explanation module is configured to form an interpretable result of a state-to-action mapping strategy of the game reinforcement learning model by counting the importance degrees.

17. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable by the processor, wherein, The processor implements the method of any one of claims 1 to 15 when executing the computer program.

18. A non-transitory computer-readable storage medium, comprising: The non-transitory computer-readable storage medium stores computer instructions for causing a computer to execute the method of any one of claims 1 to 15. The non-transitory computer-readable storage medium stores computer instructions for causing a computer to execute the method of any one of claims 1 to 15.

Images