Multi-granularity strategy interpretation method for deep reinforcement learning and related equipment

Abstract

The invention provides a multi-granularity strategy interpretation method for deep reinforcement learning and related equipment. The method comprises the following steps: acquiring an original image set and a discrete action set corresponding to each original image in the original image set; and based on the original image set and the discrete action set, adopting a saliency map interpretation method to determine a saliency image corresponding to each original image. And classifying all saliency images corresponding to the original image set according to each discrete action in the discrete action set to obtain a saliency image set corresponding to each discrete action. And based on the saliency map set, determining a decision rule set corresponding to each discrete action through a clustering-based global strategy interpretation method. The local decision explanation based on saliency graph explanation guides generation of global strategy explanation, and logic consistency between the local decision explanation and the global strategy explanation is ensured.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular, to a multi-granularity policy interpretation method for deep reinforcement learning and related devices. Background Art

[0002] Deep Reinforcement Learning (DRL) combines deep learning and reinforcement learning, and has shown great potential in solving complex tasks in recent years. However, due to the non-linearity of the deep neural network itself and the particularity of reinforcement learning, the decision-making process of the DRL model becomes difficult to understand and explain. If the decision-making process of the DRL model is not transparent, the user's trust in the system will decrease. This lack of interpretability may also lead to unexpected behaviors during the execution of the system, thus bringing potential safety hazards or economic losses. Therefore, improving the interpretability of the deep reinforcement learning model and enhancing its safety and credibility in practical applications have become the main research motivation and goal of current interpretable deep reinforcement learning.

[0003] At present, the mainstream method of interpretable deep reinforcement learning is the post-hoc interpretation method. This type of method regards the model as a black box and shows the decision-making basis of the model by performing feature attribution analysis on the decision results output by the model. According to the different ranges and granularities of the interpretation, the post-hoc interpretation method is usually divided into local decision interpretation and global policy interpretation. However, both local decision interpretation and global policy interpretation have certain limitations, which affect the accuracy and clarity of the interpretation. Summary of the Invention

[0004] In view of this, the purpose of this application is to propose a multi-granularity policy interpretation method for deep reinforcement learning and related devices to solve the problem that the post-hoc interpretation method has poor interpretation accuracy and clarity.

[0005] Based on the above purpose, this application provides a multi-granularity policy interpretation method for deep reinforcement learning, including:

[0006] Obtain a set of original images and a set of discrete actions corresponding to each original image in the set of original images:

[0007] Based on the set of original images and the set of discrete actions, use the saliency map interpretation method to determine the saliency image corresponding to each original image;

[0008] Classify all the saliency images corresponding to the set of original images according to each discrete action in the set of discrete actions to obtain a set of saliency maps corresponding to each discrete action;

[0009] Based on the set of saliency maps, a set of decision rules corresponding to each discrete action is determined through a clustering-based global strategy interpretation method.

[0010] In some embodiments, based on the set of original images and the set of discrete actions, a saliency map interpretation method is used to determine a saliency image corresponding to each original image, including:

[0011] For each original image, the original image is superpixel segmented according to a preset number to form multiple groups of superpixel fragments;

[0012] For each group of superpixel fragments, multiple hard masks are randomly generated; the number of the multiple hard masks is the same as the number of each group of superpixel fragments;

[0013] Each hard mask is multiplied by the original image respectively to generate multiple perturbed images; based on the original image, the multiple perturbed images, the set of discrete actions, and the deep learning model, the weight coefficient of each hard mask is determined;

[0014] According to the weight coefficients, all hard masks are weighted averaged, and through Monte Carlo sampling, a saliency image corresponding to the original image is determined.

[0015] In some embodiments, the superpixel segmenting the original image according to a preset number to form multiple groups of superpixel fragments includes:

[0016] According to the original image, a preset number of images to be processed are generated;

[0017] For each image to be processed, the image to be processed is superpixel segmented according to preset superpixel segmentation parameters to generate a group of superpixel fragments; wherein, the superpixel segmentation parameters corresponding to each image to be processed are different.

[0018] In some embodiments, the determining the weight coefficient of each hard mask based on the original image, the multiple perturbed images, the set of discrete actions, and the deep learning model includes:

[0019] For each hard mask, based on the original image, the perturbed image corresponding to the hard mask, the set of discrete actions, and the deep learning model, the difference in the expected Q-value distribution of the action to be explained executed by the agent under the original image before and after the perturbation is calculated and determined;

[0020] Based on the original image, the perturbed image corresponding to the hard mask, the set of discrete actions, and the deep learning model, the expected Q-value distribution before the perturbation and the expected Q-value distribution after the perturbation of other actions except the action to be explained are respectively calculated and determined;

[0021] Calculate the KL divergence based on the expected Q-value distribution before the perturbation and the expected Q-value distribution after the perturbation, and normalize the KL divergence;

[0022] Based on the difference in the expected Q-value distribution and the normalized KL divergence, calculate and determine the weight coefficient of the hard mask.

[0023] In some embodiments, the determining, based on the set of saliency maps, a set of decision rules corresponding to each discrete action through a clustering-based global strategy interpretation method includes:

[0024] For each set of saliency maps, use a clustering algorithm to cluster all the saliency maps included in the set of saliency maps to obtain a plurality of clusters;

[0025] For each cluster, use an importance algorithm to determine the most important saliency map corresponding to the cluster;

[0026] Take all the most important saliency maps corresponding to each set of saliency maps as the set of decision rules for the discrete action corresponding to the set of saliency maps.

[0027] In some embodiments, the using an importance algorithm to determine the most important saliency map corresponding to the cluster includes:

[0028] For each saliency map in the cluster, calculate the importance value of the saliency map;

[0029] Take the saliency map corresponding to the maximum importance value as the most important saliency map.

[0030] In some embodiments, the calculating the importance value of the saliency map includes:

[0031] Determine the set of discrete actions of the original image corresponding to the saliency map;

[0032] Determine the optimal action and the sub-optimal action of the original image in the set of discrete actions;

[0033] Based on a preset action state function, calculate respectively to obtain a first action state value corresponding to the optimal action and a second action state value corresponding to the sub-optimal action;

[0034] Take the difference between the first action state value and the second action state value as the importance value.

[0035] Based on the same inventive concept, the present application also provides a multi-granularity policy interpretation device for deep reinforcement learning, including:

[0036] An acquisition module, configured to acquire a set of original images, and a set of discrete actions corresponding to each original image in the set of original images;

[0037] A local interpretation module, configured to determine a saliency image corresponding to each original image by using a saliency map interpretation method based on the set of original images and the set of discrete actions;

[0038] A classification module, configured to classify all the saliency images corresponding to the set of original images according to each discrete action in the set of discrete actions, to obtain a set of saliency maps corresponding to each discrete action;

[0039] A global interpretation module, configured to determine a set of decision rules corresponding to each discrete action by using a global policy interpretation method based on clustering, based on the set of saliency maps.

[0040] Based on the same inventive concept, the present application further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable by the processor, where the processor implements the method as described above when executing the computer program.

[0041] Based on the same inventive concept, the present application further provides a non-transitory computer-readable storage medium, where the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause a computer to execute the method as described above.

[0042] As can be seen from the above, a multi-granularity policy interpretation method and related devices for deep reinforcement learning provided by the present application, the method includes: acquiring a set of original images, and a set of discrete actions corresponding to each original image in the set of original images. Based on the set of original images and the set of discrete actions, using a saliency map interpretation method to determine a saliency image corresponding to each original image, which can obtain a more accurate and precise local saliency map while ensuring a lower complexity. Classifying all the saliency images corresponding to the set of original images according to each discrete action in the set of discrete actions, to obtain a set of saliency maps corresponding to each discrete action. Based on the set of saliency maps, using a global policy interpretation method based on clustering to determine a set of decision rules corresponding to each discrete action. The present application guides the generation of global policy interpretation based on local decision interpretation of saliency maps, ensuring the logical consistency between the two. Local interpretation not only reflects the logic of specific decisions, but also can incorporate global decision rules, thereby ensuring the unity and mutual support between different granularity interpretations, making the multi-granularity interpretation framework more in line with the logical requirements in practical applications, and enhancing the reliability and effectiveness of the interpretation results. Description of the Drawings

[0043] To more clearly illustrate the technical solutions in the present application or related technologies, the following will briefly introduce the drawings required for use in the embodiments or related technology descriptions. Obviously, the drawings in the following descriptions are only embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0044] Figure 1 It is a schematic flowchart of a multi-granularity policy interpretation method for deep reinforcement learning according to an embodiment of the present application;

[0045] Figure 2 It is a schematic diagram of saliency map interpretation based on superpixel perturbation according to an embodiment of the present application;

[0046] Figure 3 It is a schematic diagram of global policy interpretation based on saliency map clustering according to an embodiment of the present application;

[0047] Figure 4 It is a schematic structural diagram of a multi-granularity policy interpretation device for deep reinforcement learning according to an embodiment of the present application;

[0048] Figure 5 It is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. Specific Embodiments

[0049] To make the objectives, technical solutions, and advantages of the present application clearer and more understandable, the following further elaborates on the present application in detail with reference to specific embodiments and the accompanying drawings.

[0050] It should be noted that unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should have the ordinary meaning understood by those of ordinary skill in the art to which the present application belongs. The "first", "second", and similar terms used in the embodiments of the present application do not indicate any order, quantity, or importance, but are only used to distinguish different components. The terms such as "including" or "comprising" mean that the elements or objects appearing before this word cover the elements or objects listed after this word and their equivalents, without excluding other elements or objects. The terms such as "connected" or "coupled" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The terms such as "upper", "lower", "left", and "right" are only used to represent relative positional relationships, and when the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0051] As described in the background art, deep reinforcement learning (DRL) combines deep learning and reinforcement learning, and has shown great potential in solving complex tasks in recent years. Deep learning can automatically extract features from a large amount of data through a deep neural network, while reinforcement learning enables an agent to interact with the environment and optimize its decision-making strategy based on reward signals. By combining the two, DRL can achieve remarkable results in tasks such as autonomous driving and game battles. However, due to the non-linearity of the deep neural network itself and the particularity of reinforcement learning, the decision-making process of the DRL model becomes difficult to understand and explain. This "black box" nature makes it difficult to apply in task scenarios that require high reliability and user trust, such as military and intelligent healthcare. If the decision-making process of the model is opaque, the user's trust in the system will decrease. This lack of interpretability may also lead to unexpected behaviors during the execution of the system, resulting in potential safety hazards or economic losses. Therefore, improving the interpretability of deep reinforcement learning models and enhancing their security and credibility in practical applications have become the main research motivation and goal of current explainable deep reinforcement learning (XDRL).

[0052] At present, the mainstream method of XDRL is the post-hoc explanation method. This type of method treats the model as a black box and demonstrates the decision-making basis of the model by performing feature attribution analysis on the decision results output by the model. According to the different scope granularities of the explanations, the post-hoc explanation method is usually divided into local decision explanation and global policy explanation. Local decision explanation mainly focuses on explaining the decisions of the model in specific states. It helps users understand how the model makes judgments in specific situations and can reveal which features have important impacts on the decisions in such situations. However, the limitation of local decision explanation is the lack of a comprehensive understanding of the overall behavior of the model. Although users can understand the decision-making basis of the model in a specific instance, it is difficult to infer the overall strategy and behavior patterns of the model in other different situations. Different from local explanations, global policy explanation focuses on reasoning about the overall behavior of the model. This method helps users form an understanding of the overall logic of the model by analyzing the overall structure of the model, the learned strategy, and the importance of features.

[0053] To make up for the limitations of local and global explanations respectively, multi-granularity explanation methods have been proposed in related technologies. This method combines the advantages of local and global explanations, aiming to enable users to understand the behavior of the model from multiple perspectives by providing explanations at different levels. Specifically, the multi-granularity explanation method allows users to understand the decision-making logic of the model from global to local and from rough to fine. Users can not only understand the overall behavior pattern of the model but also dig deep into the details of specific decisions. This hierarchical explanation method can not only help users comprehensively grasp the decision-making basis of the model but also enhance users' trust and understanding of the model's behavior. In related technologies, multi-granularity explanations are mainly provided based on the combination of policy summaries and saliency maps. Although this method can achieve a comprehensive explanation of the agent's decision-making process to a certain extent, there are still some significant drawbacks.

[0054] 1) The problem of logical unity between local and global explanations: In the related technical solutions, the calculations of local and global explanations are two independent processes that have nothing to do with each other. This method forms multi-granularity explanations through simple combination, ignoring the internal connection between local decisions and global policies. Local explanations focus on individual decisions or single-step behaviors, while global explanations describe the rules of the agent's overall decision-making. For the same agent, the two explanations should conform to the same logic. If the two are separated, it may lead to inconsistencies in the displayed decision-making logic of the agent. Therefore, the logical unity between local and global explanations cannot be ensured, and the full picture of the agent's behavior cannot be accurately captured.

[0055] 2) The problem of model correlation of local explanations: The related technology generates a saliency map explanation of the policy summary based on the LRP method (Layer-wise Relevant Propagation, a method for explaining the decisions of neural networks). This method relies on obtaining the activation values of internal neurons of the model and their corresponding weight information to calculate the relevance of features. This process means that the generation of explanations is closely related to the structure of a specific model and it is difficult to be completely model-independent. This dependence on the model limits the generality of the explanation method and it cannot be widely applied to different types of model algorithms. If the model changes, the effectiveness and accuracy of the explanations may be affected, so it lacks good model adaptation ability.

[0056] 3) Comprehensibility and integrity issues of global interpretation: The policy summary explanation form generated by the related technology based on the HIGHLIGHTS-DIV method (HIGHLIGHTS-DIV is a global interpretation method based on policy summary, which measures the importance of a state by calculating the expected return difference between the optimal decision and the worst decision of the intelligent agent in a certain state) is relatively complex, and it is difficult to directly extract the decision-making logic of the intelligent agent from it. This may make users feel confused or unclear when understanding the decision-making process of the intelligent agent. In addition, the related technology generates explanations by selecting summary trajectories based on state importance. Although this method can reduce the calculation amount to a certain extent, it cannot guarantee that the explanations completely cover the decision-making space of the intelligent agent. Under the same decision, similar states may be selected multiple times, resulting in redundant information in the explanations, thus affecting the clarity of the explanations.

[0057] In view of this, the present application proposes a multi-granularity policy interpretation method for deep reinforcement learning, which is applied to the multi-granularity interpretation task of a deep reinforcement learning intelligent agent with visual input and discrete action space, so as to solve the deficiencies existing in the above-mentioned related technologies.

[0058] To facilitate the understanding of the technical solution of the present application, the reinforcement learning involved in the present application is briefly described below:

[0059] The main components of reinforcement learning include: Agent, Environment, State, Action, and Reward. The intelligent agent selects actions based on the current state, the environment changes to a new state according to the actions of the intelligent agent and gives rewards, and the intelligent agent updates its policy after receiving the rewards. Introducing a deep neural network into the reinforcement learning algorithm can enable the intelligent agent to process complex and high-dimensional state spaces. Taking the classic deep reinforcement learning algorithm DQN (Deep Q-Network) as an example, the state space of the intelligent agent is defined as The action space is At time t, the state of the intelligent agent is s, which is input into the model f, and the output is the Q value Q(s,a) of each action. The Q value Q(s,a) represents the future cumulative reward that the intelligent agent may obtain by executing action a in state s. a is a vector containing all actions in the action space Q(s,a). Therefore, the action taken by the intelligent agent in this state s is the action with the largest Q value Then there is:

[0060]

[0061] The following will detail the embodiments of the present application with reference to the accompanying drawings.

[0062] The present application proposes a multi-granularity policy interpretation method for deep reinforcement learning, referring toFigure 1 , including the following steps:

[0063] Step 102, obtain the original image set and the discrete action set corresponding to each original image in the original image set.

[0064] Specifically, the original image set is all the states corresponding to the agent in a single task. Each original image serves as a state, and at each moment t in a single task, the agent corresponds to an original image. Each original image corresponds to a discrete action set, and each discrete action in the discrete action set is the set of actions that the agent can take under that original image. Exemplarily, taking a game scenario as an example, the original image set can be a game round, the original image is a screenshot of a certain frame of the game interface, and the discrete action set is the set of possible operations that the agent can perform under the screenshot of a certain frame of the game interface. For each screenshot of the game interface, the action selected by the agent from the discrete action set is also the action to be explained in this embodiment, and is also denoted as the action to be explained.

[0065] Step 104, based on the original image set and the discrete action set, use the saliency map interpretation method to determine the saliency image corresponding to each original image.

[0066] Further, step 104 includes:

[0067] Step 1041, for each original image, perform superpixel segmentation on the original image according to a preset number to form multiple groups of superpixel fragments.

[0068] Specifically, the preset number is determined according to multiple experiments. Exemplarily, the preset number can be 5.

[0069] Further, step 1041 includes:

[0070] Generate a preset number of images to be processed according to the original image.

[0071] First, copy the original image to obtain a preset number of original images as the images to be processed.

[0072] For each image to be processed, perform superpixel segmentation on the image to be processed according to the preset superpixel segmentation parameters to generate a group of superpixel fragments; wherein, the superpixel segmentation parameters corresponding to each image to be processed are different.

[0073] For each image to be processed, perform superpixel segmentation to divide each image to be processed into a set of superpixel fragments. During segmentation, segment the image to be processed according to the superpixel segmentation parameters. After segmentation, the image to be processed contains multiple superpixel blocks, and the number of multiple superpixel blocks is equal to the superpixel segmentation parameters. The superpixel segmentation parameters corresponding to each image to be processed are different and show a gradually increasing trend, that is, the number of formed superpixel blocks gradually increases. Figure 2 shows a schematic diagram of the interpretation of the saliency map based on superpixel perturbation. As Figure 2 shown, the input state s represents an original image, and perform superpixel segmentation at L levels on the input state s, where L is a preset number. Figure 2 In it, the value of L is 4. After segmentation, obtain a list of superpixel fragments Seg of length L, and g(Seg l ) is the number of the l-th group of superpixel fragments. The value range of l is 0 to L. As the value of l increases, the number of superpixel fragments g(Seg l ) contained in the image to be processed after segmentation is increasing. After segmentation, L groups of superpixel fragments are formed.

[0074] Step 1042: For each group of superpixel fragments, randomly generate multiple hard masks; the number of the multiple hard masks is the same as the number of each group of superpixel fragments; multiply each hard mask with the original image respectively to generate multiple perturbed images.

[0075] Specifically, as Figure 2 shown, for each group of superpixel fragments, randomly generate g(Seg l ) hard masks m, and the values within the superpixel blocks in each mask m are 0 or 1. Multiply each hard mask m with the original image s respectively to obtain multiple perturbed images s′ = m ⊙ s. The total number of hard masks m is G, and the number of multiple perturbed images is also G.

[0076] Step 1043: Based on the original image, the multiple perturbed images, the discrete action set, and the deep learning model, determine the weight coefficient of each hard mask.

[0077] As Figure 2 shown, input the multiple perturbed images into the deep learning DRL model, and calculate and output the score (i.e., the weight coefficient) of each hard mask according to specificity and relevance in the DRL model. After weighted averaging, obtain the saliency image Sal(s) corresponding to the original image (input state s).

[0078] Further, step 1043 includes:

[0079] For each hard mask, based on the original image, the perturbed image corresponding to the hard mask, the discrete action set, and a deep learning model, calculate and determine the difference in the expected Q-value distribution of the action to be explained executed by the agent under the original image before and after perturbation;

[0080] Based on the original image, the perturbed image corresponding to the hard mask, the discrete action set, and a deep learning model, calculate and determine the expected Q-value distribution before perturbation and the expected Q-value distribution after perturbation of actions other than the action to be explained, respectively;

[0081] Calculate the KL divergence based on the expected Q-value distribution before perturbation and the expected Q-value distribution after perturbation, and normalize the KL divergence;

[0082] Based on the difference in the expected Q-value distribution and the normalized KL divergence, calculate and determine the weight coefficient of the hard mask.

[0083] Specifically, determining the weight coefficient of each hard mask is equivalent to scoring each hard mask. In the prior art, the difference in Q-values before and after perturbation, Q(s,a)-Q(s′,a), is usually used to represent the score of the mask. Q(s,a) represents the Q-value corresponding to each action a before perturbation, and Q(s′,a) represents the Q-value corresponding to each action a after perturbation. Input the corresponding state s and each action a into the deep learning model, and the corresponding Q(s,a) can be output through the deep learning model. Similarly, Q(s′,a) can be obtained. However, for DRL tasks, the perturbation also needs to consider the specificity and relevance to the target action. Specificity means that when only removing a certain feature, the impact on the Q-value of the action to be explained is much greater than that of other actions when the significance of this feature is high, which is represented by a relatively large expected Q-value before and after perturbation.

[0084] Determine the expected Q-value distribution of the action to be explained before perturbation according to the following formula (2)

[0085]

[0086] where is a probability distribution, representing the expected Q-value distribution of action under state s.

[0087] Determine the expected Q-value distribution P(s′,a) of the action to be explained after perturbation according to the following formula (3):

[0088]

[0089] where s′ in formula (3) represents the perturbed image corresponding to the hard mask, Denotes the action under perturbation s'. The expected distribution of Q values. Then, the difference Δp in the expected distribution of Q values before and after perturbation is shown in the following formula (4):

[0090]

[0091] Where Δp represents the change in the expected distribution of Q values of the action before and after perturbation, and is used to measure the specificity of the impact of this perturbation s' on the action . Relevance means that it is necessary to ensure that the impact of the perturbation on the relative expected Q value of a' is small, and only then is the significance of this feature high. It is expressed that the similarity between the distributions of a's relative Q expectations before and after perturbation should be high. The expected distribution of Q values P

[0092] (s,a) of other actions except the action to be explained is calculated by the following formula (5): rem

[0093]

[0094] Where P rem (s,a) is a probability distribution, representing the expected distribution of Q values of other actions a' except the action in state s.

[0095] The expected distribution of Q values P rem (s',a) of other actions except the action to be explained is calculated by the following formula (6):

[0096]

[0097] Where P rem (s',a) is a probability distribution, representing the expected distribution of Q values of other actions a' except the action in state s'. Then, the KL divergence D KL is calculated by the following formula (7):

[0098] D KL = P rem (s',a) ‖ P rem (s,a) (7)

[0099] Through D KL , the impact of the perturbation s' on other actions a' can be measured, and thus the relevance of the impact on the action a can be represented. D KL is an index for measuring the difference between two probability distributions. The larger the value, the less similar the two probability distributions are. The normalized KL divergence K is calculated by the following formula (8):

[0100]

[0101] Through normalization, D is mapped within the interval [0, 1], such that the closer the value of K is to 1, the more similar the two probability distributions are. The weight coefficient w of the hard mask is calculated by the following formula (9): KL The weight coefficient w of the hard mask is calculated by the following formula (9):

[0102]

[0103] Step 1044: Perform weighted averaging on all hard masks according to the weight coefficient, and determine the saliency image corresponding to the original image through Monte Carlo sampling.

[0104] Specifically, taking w as the weight of the hard mask m, perform weighted averaging on all hard masks. According to Monte Carlo sampling, the saliency image Sal(s) corresponding to the original image s can be obtained, which is specifically calculated by the following formula (10):

[0105]

[0106] where represents the expected value of the random variable, which is the average of the masks obtained from multiple samples here, and m l,i represents the i-th hard mask in the l-th group, and w l,i represents the weight corresponding to m l,i

[0107] The foregoing steps present a model-agnostic local decision explanation method, and design a local explanation method based on multi-level superpixel perturbations for the deep reinforcement learning scenario. Compared with the widely used pixel perturbation-based explanation method, it can capture the geometric features of objects in the environment more effectively, making the generated saliency map more accurate. And compared with the object-level perturbation based on the object detection algorithm, it is more efficient and simple. On this basis, the consideration of the specificity and relevance of perturbations to the target action is introduced, and the saliency calculation formula of the superpixel perturbation mask is redesigned to make it more suitable for the saliency analysis of the state features in the deep reinforcement learning scenario. This application can generate high-quality local explanations without relying on a specific neural network architecture and has stronger versatility.

[0108] Step 106: Classify all the saliency images corresponding to the original image set according to each discrete action in the discrete action set, and obtain a set of saliency maps corresponding to each discrete action.

[0109] Specifically, classify all the saliency images in terms of the action dimension to obtain a set of saliency maps D(a) corresponding to each discrete action. Figure 3 shows a schematic diagram of the global policy explanation based on saliency map clustering. As Figure 3 ​As shown, all saliency images are classified according to actions 1, 2, etc. in the discrete action set.

[0110] Step 108: Based on the set of saliency maps, determine a set of decision rules corresponding to each discrete action through a clustering-based global policy interpretation method.

[0111] Specifically, to ensure the logical unity of multi-granularity interpretation, the global policy interpretation needs to contain the information of local decision interpretation. This step proposes the idea of saliency map clustering, introduces feature importance information for clustering, and realizes the generation of global policy interpretation guided by local decision interpretation. By combining the example decisions of each type of scenario for the agent under each action, a comprehensive and understandable decision rule table can be formed. Through this table, users can view in which scenarios the agent may make a specified action, thereby understanding the global policy of the agent and predicting actions in similar scenarios.

[0112] Furthermore, step 108 includes:

[0113] For each set of saliency maps, use a clustering algorithm to cluster all the saliency maps included in the set of saliency maps to obtain multiple clusters;

[0114] For each cluster, use an importance algorithm to determine the most important saliency map corresponding to the cluster;

[0115] Take all the most important saliency maps corresponding to each set of saliency maps as the set of decision rules for the discrete action corresponding to the set of saliency maps.

[0116] For each set of saliency maps, apply a clustering algorithm to obtain multiple clusters. The number of clusters is k, as Figure 3 shown, and the clusters are denoted as C1, C2,... Ck kSince the number of clustering clusters cannot be determined in advance, a method that does not require specifying the number of clusters needs to be used, such as the density-based clustering algorithm DBSCAN (Density-Based Spatial Clustering of Applications with Noise), which identifies clusters by examining the neighborhood of data points and can automatically identify the number of clusters. When executing the clustering algorithm, the minimum number of data points minPts to be included in the clustering and the maximum neighborhood ε of data points in each clustering cluster need to be given. In this way, each obtained clustering cluster represents a type of scenario when the decision result (the action selected by the agent under s) is a. In this type of scenario, the agent has a similar decision logic and can be regarded as satisfying the same decision rule. Then, according to the importance algorithm, the importance I(s) of the original image s corresponding to each saliency map in each clustering cluster is calculated, and the saliency map of the most important state in the cluster represents the abstraction of this decision rule.

[0117] Further, the method of using the importance algorithm to determine the most important saliency map corresponding to the clustering cluster includes:

[0118] For each saliency map in the clustering cluster, calculate the importance value of the saliency map.

[0119] Specifically, determine the discrete action set of the original image corresponding to the saliency map; determine the optimal action and the sub-optimal action of the original image in the discrete action set; based on the preset action-state function Q(s,a), calculate the first action-state value max a Q(s,a) corresponding to the optimal action and the second action-state value secondhighest a Q(s,a) corresponding to the sub-optimal action; take the difference between the first action-state value max a Q(s,a) and the second action-state value secondhighest a Q(s,a) as the importance value I(s). Then, take the saliency map corresponding to the maximum importance value as the most important saliency map

[0120] Further, the calculation of the importance value I(s) of the saliency map is shown in formula (11),

[0121] I(s) = max a Q(s,a) - secondhighest a Q(s,a) (11)

[0122] max aQ(s,a) represents the Q-value of performing the optimal action under the original image s, secondhighest a Q(s,a) represents the Q-value of performing the sub-optimal action under the original image s. By calculating the Q-value difference of the action performed under the original image s relative to the sub-optimal action, the importance of the original image s to the global decision-making process is measured, avoiding the problem that the action with the lowest Q-value has no practical significance when the training is insufficient. After calculating the importance values corresponding to each saliency map, the saliency map corresponding to the maximum importance value is used as the most important saliency map i max represents the serial number of the saliency map corresponding to the maximum importance value in the clustering cluster.

[0123] Based on the above steps 102 to 108, the multi-granularity policy interpretation method for deep reinforcement learning proposed in this application can obtain complete local and global interpretations in one end-to-end process. At the local interpretation level, the superpixel perturbation method is adopted, which can obtain more accurate and precise local saliency maps while ensuring low complexity. At the global interpretation level, the state feature saliency maps obtained from the local interpretation are clustered, which improves the integrity of the global interpretation rules and reduces their redundancy while ensuring the logical unity of the overall multi-granularity policy interpretation. Different from the existing technology where local interpretation and global interpretation are independent and simply combined, this proposal introduces the idea of saliency map clustering, and guides the generation of global policy interpretation through local decision interpretation, ensuring the logical consistency between the two. Local interpretation not only reflects the logic of specific decisions, but also can be integrated into global decision rules, thus ensuring the unity and mutual support between different granularity interpretations, making the multi-granularity interpretation framework more in line with the logical requirements in practical applications, and enhancing the reliability and effectiveness of the interpretation results.

[0124] It should be noted that the method of the embodiment of this application can be executed by a single device, such as a computer or a server, etc. The method of this embodiment can also be applied to a distributed scenario, and completed by multiple devices cooperating with each other. In this case of a distributed scenario, one of the multiple devices can only execute one or more steps of the method of the embodiment of this application, and these multiple devices will interact with each other to complete the described method.

[0125] It should be noted that some embodiments of this application have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be executed in a different order from that in the above embodiments and still achieve the desired results. Additionally, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In certain embodiments, multi-tasking and parallel processing are also possible or may be advantageous.

[0126] Based on the same inventive concept, corresponding to the method of any of the above embodiments, the present application further provides a multi-granularity policy interpretation device for deep reinforcement learning.

[0127] Reference Figure 4 , the multi-granularity policy interpretation device for deep reinforcement learning includes:

[0128] An acquisition module 402, configured to acquire a set of original images and a set of discrete actions corresponding to each original image in the set of original images;

[0129] A local interpretation module 404, configured to determine a saliency image corresponding to each original image by using a saliency map interpretation method based on the set of original images and the set of discrete actions;

[0130] A classification module 406, configured to classify all the saliency images corresponding to the set of original images according to each discrete action in the set of discrete actions, and obtain a set of saliency maps corresponding to each discrete action;

[0131] A global interpretation module 404, configured to determine a set of decision rules corresponding to each discrete action based on the set of saliency maps by using a global policy interpretation method based on clustering.

[0132] In some embodiments, the local interpretation module 404 is configured to, for each original image, perform superpixel segmentation on the original image according to a preset number to form multiple groups of superpixel fragments;

[0133] Based on the multiple groups of superpixel fragments and the original image, determine multiple perturbed images;

[0134] Based on the original image, the multiple perturbed images, the set of discrete actions, and a deep learning model, determine the weight coefficients of each hard mask;

[0135] Perform weighted averaging on all the hard masks according to the weight coefficients, and determine the saliency image corresponding to the original image through Monte Carlo sampling.

[0136] In some embodiments, the local interpretation module 404 is configured to generate a preset number of images to be processed according to the original image;

[0137] For each image to be processed, perform superpixel segmentation on the image to be processed according to preset superpixel segmentation parameters to generate a group of superpixel fragments; wherein, the superpixel segmentation parameters corresponding to each image to be processed are different.

[0138] In some embodiments, the local interpretation module 404 is configured to randomly generate a plurality of hard masks for each set of superpixel fragments; the number of the plurality of hard masks is the same as the number of each set of superpixel fragments;

[0139] Multiply each hard mask with the original image respectively to generate a plurality of perturbed images.

[0140] In some embodiments, the local interpretation module 404 is configured to, for each hard mask, calculate and determine the difference in the expected Q-value distribution of the action to be interpreted executed by the agent under the original image before and after perturbation based on the original image, the perturbed image corresponding to the hard mask, the discrete action set, and the deep learning model;

[0141] Based on the original image, the perturbed image corresponding to the hard mask, the discrete action set, and the deep learning model, calculate and determine the expected Q-value distribution before perturbation and the expected Q-value distribution after perturbation of other actions except the action to be interpreted respectively;

[0142] Calculate the KL divergence based on the expected Q-value distribution before perturbation and the expected Q-value distribution after perturbation, and normalize the KL divergence;

[0143] Calculate and determine the weight coefficient of the hard mask based on the difference in the expected Q-value distribution and the normalized KL divergence.

[0144] In some embodiments, the global interpretation module 404 is configured to, for each set of saliency maps, cluster all the saliency maps included in the set of saliency maps by using a clustering algorithm to obtain a plurality of clusters;

[0145] For each cluster, use an importance algorithm to determine the most important saliency map corresponding to the cluster;

[0146] Use all the most important saliency maps corresponding to each set of saliency maps as the decision rule set of the discrete action corresponding to the set of saliency maps.

[0147] In some embodiments, the global interpretation module 404 is configured to calculate the importance value of each saliency map in the cluster;

[0148] Use the saliency map corresponding to the maximum importance value as the most important saliency map.

[0149] In some embodiments, the global interpretation module 404 is configured to determine the discrete action set of the original image corresponding to the saliency map;

[0150] Determine the optimal action and the sub-optimal action of the original image in the discrete action set;

[0151] Based on a preset action state function, calculate a first action state value corresponding to the optimal action and a second action state value corresponding to the sub - optimal action respectively;

[0152] Take the difference between the first action state value and the second action state value as the importance value.

[0153] For convenience of description, when describing the above device, it is divided into various modules according to functions and described separately. Of course, when implementing the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0154] The device in the above - mentioned embodiment is used to implement the corresponding multi - granularity policy interpretation method for deep reinforcement learning in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be elaborated here.

[0155] Based on the same inventive concept, corresponding to the method in any of the above embodiments, the present application further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the multi - granularity policy interpretation method for deep reinforcement learning in any of the above embodiments.

[0156] Figure 5 FIG. shows a more specific schematic diagram of the hardware structure of the electronic device provided in this embodiment. The device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. Among them, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are communicatively connected to each other inside the device through the bus 1050.

[0157] The processor 1010 may be implemented in a general - purpose CPU (Central Processing Unit), a microprocessor, an application - specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0158] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, etc. The memory 1020 can store the operating system and other application programs. When implementing the technical solutions provided in the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 1020 and called and executed by the processor 1010.

[0159] The input / output interface 1030 is used to connect to the input / output module to implement information input and output. The input / output module can be configured as a component in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Among them, the input device can include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device can include a display, a speaker, a vibrator, an indicator light, etc.

[0160] The communication interface 1040 is used to connect to the communication module (not shown in the figure) to implement communication interaction between this device and other devices. Among them, the communication module can implement communication in a wired manner (such as USB, network cable, etc.) or in a wireless manner (such as mobile network, WIFI, Bluetooth, etc.).

[0161] The bus 1050 includes a path for transmitting information between various components of the device (such as the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0162] It should be noted that although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040, and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, those skilled in the art can understand that the above device may also only include the components necessary to implement the solutions of the embodiments of this specification, and do not have to include all the components shown in the figure.

[0163] The electronic device in the above embodiment is used to implement the corresponding multi-granularity policy interpretation method for deep reinforcement learning in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be elaborated here.

[0164] Based on the same inventive concept, corresponding to the method in any of the above embodiments, the present application also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the multi-granularity policy interpretation method for deep reinforcement learning as described in any of the foregoing embodiments.

[0165] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD), or other optical storage, magnetic cassette tapes, magnetic tape magnetic disk storage, or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device.

[0166] The computer instructions stored in the storage medium of the above embodiment are used to cause the computer to execute the multi-granularity policy interpretation method for deep reinforcement learning described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be elaborated here.

[0167] Based on the same concept, corresponding to any of the above method embodiments, the present application also provides a computer program product, including computer program instructions, which, when running on a computer, cause the computer to execute the method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be elaborated here.

[0168] Those of ordinary skill in the art should understand that the discussion of any of the above embodiments is only exemplary and is not intended to imply that the scope of the present application is limited to these examples; under the concept of the present application, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations in different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of brevity.

[0169] In addition, for simplicity of explanation and discussion, and so as not to make the embodiments of the present application difficult to understand, well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Further, the devices may be shown in block diagram form in order to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present application are to be implemented (i.e., these details should be fully within the understanding of those skilled in the art). In cases where specific details (such as circuits) are set forth to describe exemplary embodiments of the present application, it will be apparent to those skilled in the art that the embodiments of the present application may be practiced without these specific details or with variations of these specific details. Accordingly, these descriptions should be regarded as illustrative rather than restrictive.

[0170] Although the present application has been described in connection with specific embodiments thereof, many alternatives, modifications, and variations thereof will be apparent to those of ordinary skill in the art based on the foregoing description. For example, other memory architectures (such as dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0171] Embodiments of the present application are intended to cover all such alternatives, modifications, and variations that fall within the broad scope of the present application. Accordingly, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of the present application shall be included within the protection scope of the present application.

Claims

1. A multi-granularity policy interpretation method for deep reinforcement learning, characterized in that, Including: Obtain a set of original images and a set of discrete actions corresponding to each original image in the set of original images; Based on the set of original images and the set of discrete actions, use the saliency map interpretation method to determine a saliency image corresponding to each original image; Classify all the saliency images corresponding to the set of original images according to each discrete action in the set of discrete actions to obtain a set of saliency maps corresponding to each discrete action; Based on the set of saliency maps, determine a set of decision rules corresponding to each discrete action through a clustering-based global policy interpretation method.

2. The method according to claim 1, wherein The step of using the saliency map interpretation method based on the set of original images and the set of discrete actions to determine a saliency image corresponding to each original image includes: For each original image, perform superpixel segmentation on the original image according to a preset number to form multiple groups of superpixel fragments; For each group of superpixel fragments, randomly generate multiple hard masks; the number of the multiple hard masks is the same as the number of each group of superpixel fragments; Multiply each hard mask with the original image respectively to generate multiple perturbed images; based on the original image, the multiple perturbed images, the set of discrete actions, and a deep learning model, determine the weight coefficient of each hard mask; Perform weighted average on all the hard masks according to the weight coefficients, and determine the saliency image corresponding to the original image through Monte Carlo sampling.

3. The method according to claim 2, wherein The step of performing superpixel segmentation on the original image according to a preset number to form multiple groups of superpixel fragments includes: Generate a preset number of images to be processed according to the original image; For each image to be processed, perform superpixel segmentation on the image to be processed according to preset superpixel segmentation parameters to generate a group of superpixel fragments; Wherein, the superpixel segmentation parameters corresponding to each image to be processed are different.

4. The method according to claim 2, wherein The step of determining the weight coefficient of each hard mask based on the original image, the multiple perturbed images, the set of discrete actions, and the deep learning model includes: For each hard mask, based on the original image, the perturbed image corresponding to the hard mask, the set of discrete actions, and the deep learning model, calculate and determine the difference in the expected distribution of Q-values of the action to be explained executed by the agent under the original image before and after perturbation; Based on the original image, the perturbed image corresponding to the hard mask, the set of discrete actions, and the deep learning model, calculate and determine the expected distribution of Q-values before perturbation and the expected distribution of Q-values after perturbation of other actions except the action to be explained respectively; Calculate the KL divergence based on the expected distribution of Q-values before perturbation and the expected distribution of Q-values after perturbation, and normalize the KL divergence; Based on the difference in the expected distribution of Q-values and the normalized KL divergence, calculate and determine the weight coefficient of the hard mask.

5. The method according to claim 1, wherein The step of determining a set of decision rules corresponding to each discrete action through a clustering-based global policy interpretation method based on the set of saliency maps includes: For each set of saliency maps, use a clustering algorithm to cluster all the saliency maps included in the set of saliency maps to obtain multiple clustering clusters; For each cluster, an importance algorithm is used to determine the most important saliency map corresponding to the cluster; All the most important saliency maps corresponding to each saliency map set are used as the decision rule set of the discrete action corresponding to the saliency map set.

6. The method according to claim 1, wherein The using of the importance algorithm to determine the most important saliency map corresponding to the cluster includes: For each saliency map in the cluster, calculate the importance value of the saliency map; The saliency map corresponding to the maximum importance value is used as the most important saliency map.

7. The method according to claim 6, wherein The calculating of the importance value of the saliency map includes: Determine the discrete action set of the original image corresponding to the saliency map; Determine the optimal action and the sub-optimal action of the original image in the discrete action set; Based on a preset action state function, calculate the first action state value corresponding to the optimal action and the second action state value corresponding to the sub-optimal action respectively; The difference between the first action state value and the second action state value is used as the importance value.

8. A multi-granularity policy interpretation device for deep reinforcement learning, characterized in that, Includes: An acquisition module, configured to acquire a set of original images and a set of discrete actions corresponding to each original image in the set of original images; A local interpretation module, configured to determine the saliency image corresponding to each original image by using a saliency map interpretation method based on the set of original images and the set of discrete actions; A classification module, configured to classify all the saliency images corresponding to the set of original images according to each discrete action in the set of discrete actions, and obtain a set of saliency maps corresponding to each discrete action; A global interpretation module, configured to determine the decision rule set corresponding to each discrete action by using a global strategy interpretation method based on clustering based on the set of saliency maps.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that When the processor executes the program, it implements the method according to any one of claims 1 to 8.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 8.

Images