Awards for data analysis methods, devices, equipment, storage media, and software products

Abstract

This application provides a reward data analysis method, apparatus, device, storage medium, and program product. The method includes: collecting target data generated by a model executing a game; wherein the target data includes: reward data obtained from the robot's action data under the corresponding state data at each moment in the game, and the mapping relationship between the state data, action data, and reward data; using a neural network model to infer reward data corresponding to the state data and action data; determining a first SHAP value for the state data relative to the expected value and a second SHAP value for the action data relative to the expected value; the first SHAP value characterizes the degree of influence of the state data on the expected value, and the second SHAP value characterizes the degree of influence of the action data on the expected value; adjusting the reward data corresponding to the robot's state data according to the first SHAP value, and adjusting the reward data corresponding to the robot's action data according to the second SHAP value.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, device, storage medium, and program product for analyzing reward data. Background Technology

[0002] In related technologies, it is common practice to set up AI robots in games to complete corresponding tasks. When training the reinforcement learning model for the AI ​​robot, it usually relies on the human experience of engineers. By tuning parameters one by one and observing the robot's performance after tuning, the reward data corresponding to the AI ​​robot's performance can be further adjusted to debug the robot. Because the debugging process often needs to be repeated many times, and reinforcement learning models are slow to train, the related technologies suffer from low training efficiency and poor training results. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a reward data analysis method, apparatus, device, storage medium and program product.

[0004] In view of the above objectives, in a first aspect, this application provides a reward data analysis method, the method comprising:

[0005] Collect target data generated by the model to be interpreted during the game; wherein, the target data includes: reward data obtained by the robot performing action data under the state data corresponding to each moment in the game, and the mapping relationship between the state data, the action data and the reward data;

[0006] Using a pre-trained neural network model, the expected value of the reward data corresponding to the state data and the action data is inferred from the mapping relationship;

[0007] Determine a first SHAP value for the state data relative to the expected value and a second SHAP value for the action data relative to the expected value; wherein, the first SHAP value is used to characterize the degree of influence of the state data on the expected value, and the second SHAP value is used to characterize the degree of influence of the action data on the expected value.

[0008] The reward data corresponding to the robot's state data is adjusted according to the first SHAP value, and the reward data corresponding to the robot's action data is adjusted according to the second SHAP value.

[0009] In a second aspect, this application provides a reward data analysis apparatus, the apparatus comprising:

[0010] The acquisition module is configured to acquire target data generated by the model to be interpreted during a game; wherein, the target data includes: reward data obtained by the robot performing action data under the state data corresponding to each moment in the game, and the mapping relationship between the state data, the action data and the reward data;

[0011] The first determining module is configured to use a pre-trained neural network model to infer the expected value of the reward data corresponding to the state data and the action data based on the mapping relationship;

[0012] The second determining module is configured to determine a first SHAP value of the state data with respect to the expected value and a second SHAP value of the action data with respect to the expected value; wherein the first SHAP value is used to characterize the degree of influence of the state data on the expected value, and the second SHAP value is used to characterize the degree of influence of the action data on the expected value.

[0013] The adjustment module is configured to adjust the reward data corresponding to the robot's state data according to the first SHAP value, and to adjust the reward data corresponding to the robot's action data according to the second SHAP value.

[0014] In a third aspect, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the reward data analysis method as described in the first aspect.

[0015] In a fourth aspect, this application provides a computer-readable storage medium storing computer instructions for causing a computer to perform the reward data analysis method as described in the first aspect.

[0016] In a fifth aspect, this application provides a computer program product including computer program instructions that, when executed on a computer, cause the computer to perform the reward data analysis method as described in the first aspect.

[0017] As can be seen from the above, the reward data analysis method, apparatus, device, storage medium, and program product provided in this application collect target data generated after the model to be interpreted executes a game. The target data may include reward data obtained from the robot's action data under the state data corresponding to each moment in the game, as well as the mapping relationship between the state data, the action data, and the reward data. Further, a pre-trained neural network model can be used to perform forward reasoning on the state data and action data based on the mapping relationship, thereby obtaining the expected value of the reward data corresponding to both the state data and the action data. Further still, a first SHAP value of the state data relative to the expected value and a second SHAP value of the action data relative to the expected value can be determined, where the first SHAP value characterizes the degree of influence of the state data on the expected value, and the second SHAP value characterizes the degree of influence of the action data on the expected value. Subsequently, the reward data corresponding to the robot's state data can be adjusted according to the first SHAP value, and the reward data corresponding to the robot's action data can be adjusted according to the second SHAP value. It can determine reward data based on the robot's state and action data during a game, and can adjust the reward data corresponding to these state and action data in a targeted manner, thereby more accurately determining the impact of each reward data on the robot's performance. This eliminates the need to debug each piece of data generated by the robot individually, improving the training efficiency and effectiveness of the AI ​​robot. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 An exemplary flowchart of a reward data analysis method provided in an embodiment of this application is shown.

[0020] Figure 2 An exemplary schematic diagram is shown showing the distribution of the mean expected value corresponding to 20 types of reward data in an embodiment of this application.

[0021] Figure 3 An exemplary schematic diagram is shown in an embodiment of this application, illustrating the expectation value analysis of reward data corresponding to attack action data.

[0022] Figure 4 An exemplary schematic diagram is shown illustrating the expectation value analysis of reward data corresponding to multiple action data in an embodiment of this application.

[0023] Figure 5 An exemplary structural diagram of a reward data analysis device provided in an embodiment of this application is shown.

[0024] Figure 6 This illustration shows an exemplary structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0026] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0027] As described in the background section, in related technologies, it is common practice to set up AI robots in games to complete corresponding tasks. When training the reinforcement learning model corresponding to the AI ​​robot, it usually relies on the human experience of engineers. By adjusting parameters one by one and observing the robot's performance after parameter adjustment, the reward data corresponding to the performance of the AI ​​robot can be further adjusted to debug the robot.

[0028] The inventors' research revealed that, in most cases, engineers need to carefully and continuously adjust parameters to improve algorithm performance, while also preventing agents from exploiting loopholes to manipulate scores. Engineers spend a significant amount of time in a loop of "training -> fine-tuning the reward function -> training -> fine-tuning the reward function." However, RL models are inherently slow to train, making this process time-consuming, laborious, and inefficient, and heavily reliant on the RL engineer's experience. Currently, there is a lack of reliable and practical methods for RL robot reward analysis. When RL engineers encounter robot behavior that doesn't meet expectations (e.g., due to overly conservative approaches), they often rely on their past debugging experience, using a process of "model training -> observing robot performance -> adjusting and improving the reward function" to debug and optimize the RL robot. Because this debugging process often needs to be repeated multiple times, and reinforcement learning models are inherently slow to train, related technologies suffer from low training efficiency and poor training results, and heavily depend on the RL engineer's experience.

[0029] Therefore, this application provides a reward data analysis method, apparatus, device, storage medium, and program product, which collects target data generated after the model to be interpreted executes a game. The target data may include reward data obtained from the robot's action data performed under the corresponding state data at each moment in the game, as well as the mapping relationship between the state data, the action data, and the reward data. Further, a pre-trained neural network model can be used to perform forward reasoning on the state data and action data based on the mapping relationship, thereby obtaining the expected value of the reward data corresponding to both the state data and the action data. Further still, a first SHAP value of the state data relative to the expected value and a second SHAP value of the action data relative to the expected value can be determined, where the first SHAP value characterizes the degree of influence of the state data on the expected value, and the second SHAP value characterizes the degree of influence of the action data on the expected value. Subsequently, the reward data corresponding to the robot's state data can be adjusted according to the first SHAP value, and the reward data corresponding to the robot's action data can be adjusted according to the second SHAP value. It can determine reward data based on the robot's state and action data during a game, and can adjust the reward data corresponding to these state and action data in a targeted manner, thereby more accurately determining the impact of each reward data on the robot's performance. This eliminates the need to debug each piece of data generated by the robot individually, improving the training efficiency and effectiveness of the AI ​​robot.

[0030] The reward data analysis method provided in this application will be specifically described below through specific embodiments.

[0031] Figure 1An exemplary flowchart of a reward data analysis method provided in an embodiment of this application is shown.

[0032] refer to Figure 1 The reward data analysis method provided in this application embodiment specifically includes the following steps:

[0033] S102: Collect target data generated by the model to be interpreted during the game; wherein, the target data includes: reward data obtained by the robot performing action data under the state data corresponding to each moment in the game, and the mapping relationship between the state data, the action data and the reward data.

[0034] S104: Using a pre-trained neural network model, the expected value of the reward data corresponding to the state data and the action data is obtained by reasoning based on the mapping relationship.

[0035] S106: Determine a first SHAP value of the state data for the expected value and a second SHAP value of the action data for the expected value; wherein, the first SHAP value is used to characterize the degree of influence of the state data on the expected value, and the second SHAP value is used to characterize the degree of influence of the action data on the expected value.

[0036] S108: Adjust the reward data corresponding to the robot's state data according to the first SHAP value, and adjust the reward data corresponding to the robot's action data according to the second SHAP value.

[0037] In some embodiments, an interpretable model, such as an RL model (i.e., a reinforcement learning model), can be selected. The interpretable model is used to execute a game to obtain the state and action data generated by the robot at each time step, as well as the reward data corresponding to the data pairs formed by the state and action data. Specifically, the interpretable model is used to repeatedly execute combat games. After each game, data generated by all robots in that game is collected, including state data, action data, and reward data. Common state data for robots may include: their own class, health, enemy health, enemy distance, all skill cooldowns, equipped weapons, etc. State data can also include their own state and the states of other robots. Common action data for robots may include: standing still, moving in various directions, and all available skills. Action data generally represents their own actions. Reward data represents the rewards given to the robot. Common reward data can include: win / loss reward (e.g., +50 points for a win, -50 points for a loss), damage reward (e.g., damage multiplied by a coefficient, such as 0.5), healing reward (e.g., healing amount multiplied by a coefficient), etc. Engineers mainly adjust the reward data by designing these coefficients, thereby adjusting the robot's performance. It should be noted that reward data can include multiple sub-items, and each sub-item corresponds to all state and action data in the entire game. For example, the win / loss reward corresponds to all state and action data in the entire game, and the damage reward also corresponds to all state and action data in the entire game. Each reward data is independent of the others.

[0038] It should be noted that the state data, action data, and reward data generated by the robot at each time step in the game can be represented as state-action-reward pairs. Action data can include multiple different preset action data, with each robot corresponding one-to-one with a preset action. This can be that each robot executes a different preset action, or multiple robots execute one preset action while others execute another. For example, the data generated by the i-th robot at the k-th time step in a game can be denoted as state-action-reward(i,k), where reward(i,k) represents the reward data generated by the i-th robot executing a unique and fixed preset action data in the state corresponding to the k-th time step in the game. For example, the unique preset action data executed by a robot in the game, such as the forward movement action, the reward data obtained by the robot executing the preset action data under the state data corresponding to each time step in the game, and the mapping relationship between state data, preset action data, and reward data. When the action data is uniquely determined, that is, under the preset action data in the game, once the robot's state data changes at each time step, the corresponding reward data changes. It can be understood that the state data is multi-dimensional data, the preset action data is one-dimensional data, and the reward data corresponds to both the state data and the preset action data. It can be represented as state data - action data - reward data.

[0039] In some embodiments, a neural network model can be pre-trained, including acquiring historical target data generated by the robot in historical games, such as historical state data, historical action data, and historical reward data from historical games. Specifically, the historical target data can include historical reward data obtained by the robot executing historical action data under the historical state data at each moment in the historical game, as well as the mapping relationships between historical state data, historical action data, and historical reward data. Further, an offline reinforcement learning algorithm can be used to learn the original policy based on the historical target data, thereby obtaining a neural network model. When the similarity between the training result obtained by the neural network model based on historical data and the historical result corresponding to the historical target data reaches a preset similarity threshold, a decomposed reward expectation value learning structure (DRQ, Decomposed Reward Q-Learning) can be introduced into the neural network model to obtain a pre-trained neural network model. Specifically, an offline reinforcement learning algorithm, such as the Batch Constrained Deep Q-Learning algorithm, can be used to learn the original robot policy from the historical target data. The BCQ algorithm can learn a policy from a fixed dataset without needing to interact with the environment, enabling offline learning.

[0040] Specifically, BCQ puts the collected historical target data into an experience pool for learning, ensuring that the data in the experience pool is similar in distribution to the current policy data, reducing inference errors, and making the learned policy close to the original robot's policy. The historical target data can be used for further verification. When the accuracy of the learned policy on historical data actions reaches a certain threshold, such as 90%, that is, when the similarity between the training results obtained from the historical target data and the corresponding historical results reaches a preset similarity threshold, it is determined that the original robot policy has been learned well by BCQ, and a well-trained neural network model can be obtained.

[0041] Figure 2 An exemplary schematic diagram is shown showing the distribution of the mean expected value corresponding to 20 types of reward data in an embodiment of this application.

[0042] Furthermore, state data and action data can be input as data pairs into a neural network model. The model then uses this pair to perform forward reasoning, determining the expected value of the reward corresponding to each data pair based on the mapping relationships between state, action, and reward data. Specifically, after introducing the DRQ structure into the neural network model, DRQ, based on the deep Q-Learning algorithm, maps each type of reward data to a sub-network, with each sub-network outputting the expected value of that reward data. DRQ finally selects the action based on the aggregated total expected value. Therefore, refer to... Figure 2 This allows the network to output the expected value of the corresponding reward data for each state-action data pair during forward reasoning (each type of reward data in the reward design, such as win / loss reward, damage reward, healing reward, etc., has a corresponding expected value). By analyzing the mean and overall proportion of the expected values ​​corresponding to the reward data, we can understand the approximate impact of the reward design on the current robot behavior and see which reward data mainly influences the robot's behavior. In other words, the expected value can be used to characterize which action data will yield the maximum reward data when the robot is holding state data A. The reward data can include sub-reward data of different dimensions, such as win / loss reward data, damage reward data, and healing reward data. The output of sub-network A can be the expected value A corresponding to the win / loss reward data, which can be used to characterize the win / loss reward data; the output of sub-network B can be the expected value B corresponding to the damage reward data, which can be used to characterize the damage reward data; and the output of sub-network C can be the expected value C corresponding to the healing reward data, which can be used to characterize the healing reward data. The total expected value obtained by summing up all the expected values ​​can be used to characterize reward data that contains multiple sub-reward data.

[0043] It should be noted that the forward reasoning process can include: inputting state data and action data into each node of the input layer in a neural network; using a characteristic function to determine the output of each layer in the neural network based on the mapping relationship between state data, action data, and reward data; using a threshold function to judge the output of the output layer; if the output reward data corresponds to the input state data and action data, a correct output result is obtained, and the expected value of the reward data is further determined. The neurons in the same layer are completely parallel; that is, neuron A, which receives the first set of state data and action data, processes the data in complete parallel with neuron B, which receives the second set of state data and action data. The forward reasoning of the neural network depends only on the input (e.g., state data and action data) and the parameters of the neural network itself, which are obtained by training the network using a learning algorithm; therefore, it is an adaptive reasoning process.

[0044] In some embodiments, based on the collected state-action-partial Q-value pair dataset, the TarbularSHAP method can be used to calculate the SHAP (SHapley Additive exPlanation) value of the expected value of state data and action data for each reward data, thereby obtaining the main state data and action data affecting the expected value of each reward data and the magnitude of their influence. Specifically, state data and action data can be used as features, and the reward data corresponding to the state data and action data determined according to the mapping relationship can be used as labels to train a regression-type ensemble tree supervised model. For example, the LightGBM model, an efficient and accurate distributed gradient boosting tree, can be specifically chosen to better learn the (state, action) -> partial Q-value mapping. That is, the LightGBM ensemble tree model is chosen to learn the accurate mapping between state data and action data.

[0045] Furthermore, the TreeSHAP interpretive AI method can be used to interpret each state-action-item Q-value pair, calculating the SHAP value of the expected value of each dimension of state data and action data with respect to each reward data, as its influence magnitude. Specifically, a first SHAP value corresponding to the expected value of state data and a second SHAP value corresponding to the expected value of action data can be determined. The first SHAP value can be used to characterize the degree of influence of state data on the expected value, and the second SHAP value can be used to characterize the degree of influence of action data on the expected value.

[0046] For example, reward data A, used to represent the reward for winning or losing, has multiple different state data and action data. This is because for each robot, executing preset action data X under different state data in game 1 can yield one reward data A1, and executing preset action data Y under different state data in game 2 can still yield another reward data A2. Therefore, reward data A is affected by multiple different state data and multiple different preset action data.

[0047] It should be noted that the TreeSHAP interpretive method requires an accurate ensemble tree supervision model for interpretation, explaining why the supervision model outputs corresponding action data for a specific state data. Therefore, an ensemble tree supervision model is trained.

[0048] Figure 3 An exemplary schematic diagram is shown in an embodiment of this application, illustrating the expectation value analysis of reward data corresponding to attack action data.

[0049] In some embodiments, visualization tools can be invoked to generate visualized data analysis results; wherein, the visualized data analysis results are used to indicate the distribution of the first SHAP value relative to the state data and the distribution of the second SHAP value relative to the action data. Specifically, refer to Figure 3 The summary analysis chart shows the top 20 factors influencing the expected value of the robot's attack reward and their value distribution. The horizontal axis represents the distribution of SHAP (Shape Up and Power) values, with values ​​greater than 0 indicating a positive impact and values ​​less than 0 indicating a negative impact. Colors represent the magnitude of state or action data values. The vertical axis sorts the influencing factors from top to bottom according to their importance, based on the mean of the absolute values ​​of the SHAP values ​​of the influencing factors in the dataset. The chart shows that enemy armor, whether the "Shake Blade" skill has been used, and enemy health are the top three factors affecting attack reward. Higher enemy armor, not having used the "Shake Blade" skill, and higher enemy health have a significant positive impact on the expected value of this reward data.

[0050] Figure 4 An exemplary schematic diagram is shown illustrating the expectation value analysis of reward data corresponding to multiple action data in an embodiment of this application.

[0051] Furthermore, for details that are difficult to observe in the Summary analysis chart, you can refer to... Figure 4 The detailed analysis of the Dependence chart allows for a full understanding of the different impacts of various important factors at different values ​​on the reward data. Figure 4The distribution of the impact of different action data on the expected value of the attack reward data is shown. It can be seen that releasing the AutoCombo, LeftAttack, RightAttack, and ShengLong actions has a large positive impact on the expected value of this reward data, while the BlueLeftAttack, BlueRightAttack, and Shock actions have a large negative impact. Engineers can increase the frequency of actions with large positive impacts and decrease the frequency of actions with large negative impacts by increasing the attack reward, thereby specifically improving the robot's offensive capabilities.

[0052] In some embodiments, specifically, the state data may include: first state data, which can determine whether the first SHAP value corresponding to the first state data reaches a first preset SHAP value threshold. For example, if the first SHAP value corresponding to the first state data reaches the first preset SHAP value threshold, the reward data corresponding to the first state data can be increased to the first preset reward data, thereby increasing the reward obtained by the robot for the positive influence by increasing the duration of the state with a large positive influence. For example, if the first SHAP value corresponding to different action data performed by the robot in the forward walking state reaches the first preset SHAP value threshold, the reward data obtained by the robot in the forward walking state can be increased, for example, the reward data can be increased from 1 to 5, so that the robot increases the duration of the forward walking state in order to obtain a higher reward.

[0053] Alternatively, it can be determined whether the first SHAP value corresponding to the first state data is lower than a second preset SHAP value threshold. If the first SHAP value corresponding to the first state data is lower than the second preset SHAP value threshold, then the reward data corresponding to the first state data is reduced to the second preset reward data. This reduces the duration of states with significant negative impact, thereby reducing the reward the robot receives for that negative impact. For example, if the first SHAP value corresponding to different action data performed by the robot in a backward walking state is lower than the second preset SHAP value threshold, then the reward data obtained by the robot in the backward walking state can be reduced, for example, from 1 to 0.1, so that the robot reduces the duration of the backward walking state in order to obtain a higher reward.

[0054] The same settings apply to motion data. Motion data can include: first preset motion data; determining whether the second SHAP value corresponding to the first preset motion data reaches the first preset SHAP value threshold; if the second SHAP value corresponding to the first motion data reaches the first preset SHAP value threshold, then the reward data corresponding to the first motion data is increased to the first preset reward data. This increases the frequency of actions with a large positive impact, thereby increasing the reward the robot receives for that positive impact. For example, if the first SHAP value corresponding to the robot executing AutoCombo motion data in different states reaches the first preset SHAP value threshold, then the reward data obtained by the robot executing AutoCombo motion data can be increased, for example, from 1 to 5, causing the robot to increase the frequency of releasing AutoCombo motion data in order to obtain a higher reward.

[0055] Alternatively, it can be determined whether the second SHAP value corresponding to the first preset action data is lower than the second preset SHAP value threshold. If the second SHAP value corresponding to the first preset action data is lower than the second preset SHAP value threshold, then the reward data corresponding to the first preset action data is reduced to the second preset reward data. This reduces the frequency of actions with significant negative impact, thereby reducing the reward the robot receives for that negative impact. For example, if the first SHAP value corresponding to the robot executing Shock action data in different states is lower than the second preset SHAP value threshold, the reward data obtained by the robot executing Shock action data can be reduced, for example, from 1 to 0.1, causing the robot to reduce the frequency of releasing Shock action data in order to obtain a higher reward.

[0056] As can be seen from the above, the reward data analysis method, apparatus, device, storage medium, and program product provided in this application collect target data generated after the model to be interpreted executes a game. The target data may include reward data obtained from the robot's action data under the state data corresponding to each moment in the game, as well as the mapping relationship between the state data, the action data, and the reward data. Further, a pre-trained neural network model can be used to perform forward reasoning on the state data and action data based on the mapping relationship, thereby obtaining the expected value of the reward data corresponding to both the state data and the action data. Further still, a first SHAP value of the state data relative to the expected value and a second SHAP value of the action data relative to the expected value can be determined, where the first SHAP value characterizes the degree of influence of the state data on the expected value, and the second SHAP value characterizes the degree of influence of the action data on the expected value. Subsequently, the reward data corresponding to the robot's state data can be adjusted according to the first SHAP value, and the reward data corresponding to the robot's action data can be adjusted according to the second SHAP value. It can determine reward data based on the robot's state and action data during a game, and can adjust the reward data corresponding to these state and action data in a targeted manner, thereby more accurately determining the impact of each reward data on the robot's performance. This eliminates the need to debug each piece of data generated by the robot individually, improving the training efficiency and effectiveness of the AI ​​robot.

[0057] It should be noted that the method in this embodiment can be executed by a single device, such as a computer or server. The method can also be applied in a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method in this embodiment, and the multiple devices will interact with each other to complete the method described.

[0058] It should be noted that the above description describes some embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0059] Figure 5 An exemplary structural diagram of a reward data analysis device provided in an embodiment of this application is shown.

[0060] Based on the same inventive concept, corresponding to any of the above embodiments, this application also provides a reward data analysis device.

[0061] refer to Figure 5 The reward data analysis device includes: a data acquisition module, a first determination module, a second determination module, and an adjustment module; wherein,

[0062] The acquisition module is configured to acquire target data generated by the model to be interpreted during a game; wherein, the target data includes: reward data obtained by the robot performing action data under the state data corresponding to each moment in the game, and the mapping relationship between the state data, the action data and the reward data;

[0063] The first determining module is configured to use a pre-trained neural network model to infer the expected value of the reward data corresponding to the state data and the action data based on the mapping relationship;

[0064] The second determining module is configured to determine a first SHAP value of the state data with respect to the expected value and a second SHAP value of the action data with respect to the expected value; wherein the first SHAP value is used to characterize the degree of influence of the state data on the expected value, and the second SHAP value is used to characterize the degree of influence of the action data on the expected value.

[0065] The adjustment module is configured to adjust the reward data corresponding to the robot's state data according to the first SHAP value, and to adjust the reward data corresponding to the robot's action data according to the second SHAP value.

[0066] In one possible implementation, the action data includes: multiple different preset action data;

[0067] The acquisition module is further configured as follows:

[0068] The game is executed using the model to be explained to obtain reward data obtained by each robot performing preset action data under the state data corresponding to each time step in the game, as well as the mapping relationship between the state data, the preset action data and the reward data; wherein, the reward data is used to indicate the reward corresponding to the robot's behavior in the game, and the robot corresponds one-to-one with the preset action data.

[0069] In one possible implementation, the device further includes: a training module;

[0070] The training module is configured as follows:

[0071] Acquire the historical target data generated by the robot in historical games; wherein, the historical target data includes: historical reward data obtained by the robot performing target historical action data under the historical state data corresponding to each moment in the historical game, and the mapping relationship between the historical state data, the historical action data and the historical reward data;

[0072] The original policy is learned based on the historical target data using an offline reinforcement learning algorithm to obtain the neural network model;

[0073] If it is determined that the similarity between the training result obtained by the neural network model based on the historical target data and the historical result corresponding to the historical target data reaches a preset similarity threshold, then the decomposable reward expectation value learning structure is introduced into the neural network model to obtain a pre-trained neural network model.

[0074] In one possible implementation, the first determining module is further configured as follows:

[0075] The state data and the action data are input into the neural network model as a data pair, and the neural network model is used to perform forward reasoning on the data pair according to the mapping relationship to obtain the expected value of the reward data corresponding to the state data and the action data.

[0076] In one possible implementation, the second determining module is further configured as follows:

[0077] The state data and the action data are used as features, and the reward data corresponding to the state data and the action data determined according to the mapping relationship is used as labels. An ensemble tree supervised model is trained based on the features and labels.

[0078] The TreeSHAP interpretability method is used to interpret the ensemble tree supervision model to obtain the first SHAP value of the state data for the expected value and the second SHAP value of the action data for the expected value.

[0079] In one possible implementation, the second determining module is further configured as follows:

[0080] The ensemble tree supervised model is used to learn the mapping relationship between the state data, the action data, and the reward data.

[0081] In one possible implementation, the device further includes: a visualization module;

[0082] The visualization module is configured as follows:

[0083] A visualization tool is invoked to generate visualization data analysis results; wherein, the visualization data analysis results are used to indicate the distribution of the first SHAP value relative to the state data and the distribution of the second SHAP value relative to the action data.

[0084] In one possible implementation, the state data includes: first state data;

[0085] The adjustment module is further configured to:

[0086] Determine whether the first SHAP value corresponding to the first state data reaches a first preset SHAP value threshold.

[0087] If the first SHAP value corresponding to the first state data reaches the first preset SHAP value threshold, then the reward data corresponding to the first state data is added to the first preset reward data.

[0088] In one possible implementation, the state data includes: first state data;

[0089] The adjustment module is further configured to:

[0090] Determine whether the first SHAP value corresponding to the first state data is lower than a second preset SHAP value threshold.

[0091] If the first SHAP value corresponding to the first state data is lower than the second preset SHAP value threshold, then the reward data corresponding to the first state data is reduced to the second preset reward data.

[0092] In one possible implementation, the action data includes: first preset action data;

[0093] The adjustment module is further configured to:

[0094] Determine whether the second SHAP value corresponding to the first preset action data reaches the first preset SHAP value threshold.

[0095] If the second SHAP value corresponding to the first preset action data reaches the first preset SHAP value threshold, then the reward data corresponding to the first preset action data is added to the first preset reward data.

[0096] In one possible implementation, the action data includes: first preset action data;

[0097] The adjustment module is further configured to:

[0098] Determine whether the second SHAP value corresponding to the first preset action data is lower than the second preset SHAP value threshold;

[0099] If the second SHAP value corresponding to the first preset action data is lower than the second preset SHAP value threshold, then the reward data corresponding to the first preset action data is reduced to the second preset reward data.

[0100] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware.

[0101] The apparatus described above is used to implement the corresponding reward data analysis method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0102] Figure 6 This illustration shows an exemplary structural diagram of an electronic device provided in an embodiment of this application.

[0103] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the reward data analysis method described in any of the above embodiments. Figure 6 This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 610, a memory 620, an input / output interface 630, a communication interface 640, and a bus 650. The processor 610, memory 620, input / output interface 630, and communication interface 640 are interconnected internally via the bus 650.

[0104] The processor 610 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0105] The memory 620 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 620 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 620 and is called and executed by the processor 610.

[0106] The input / output interface 630 is used to connect input / output modules to enable information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0107] The communication interface 640 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (e.g., USB, Ethernet cable) or wireless means (e.g., mobile network, Wi-Fi, Bluetooth).

[0108] Bus 650 includes a pathway for transmitting information between various components of the device, such as processor 610, memory 620, input / output interface 630, and communication interface 640.

[0109] It should be noted that although the above-described device only shows the processor 610, memory 620, input / output interface 630, communication interface 640, and bus 650, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0110] The electronic devices described above are used to implement the corresponding reward data analysis methods in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0111] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the reward data analysis method as described in any of the above embodiments.

[0112] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0113] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the reward data analysis method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0114] Based on the same inventive concept, corresponding to the reward data analysis method described in any of the above embodiments, this disclosure also provides a computer program product, which includes computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer to cause the computer and / or the processor to perform the reward data analysis method. Corresponding to the execution entity for each step in each embodiment of the reward data analysis method, the processor executing the corresponding step can belong to the corresponding execution entity.

[0115] The computer program product of the above embodiments is used to cause the computer and / or the processor to execute the reward data analysis method as described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0116] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application (including the claims) is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in the details for the sake of brevity.

[0117] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0118] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0119] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. A method of rewarding data analysis, characterized by, The method comprises: Collecting target data generated by a game executed by a to-be-interpreted model; wherein the target data comprises reward data obtained by an action data corresponding to state data of each robot at each time in the game and a mapping relationship between the state data, the action data and the reward data; Using a pre-trained neural network model to infer the expected value of the reward data corresponding to the state data and the action data according to the mapping relationship; Determining a first SHAP value of the state data with respect to the expected value and a second SHAP value of the action data with respect to the expected value, comprising: using a TreeSHAP interpretive method to interpret a trained ensemble tree supervised model to obtain the first SHAP value of the state data with respect to the expected value and the second SHAP value of the action data with respect to the expected value; wherein the first SHAP value is used to represent the influence degree of the state data on the expected value, and the second SHAP value is used to represent the influence degree of the action data on the expected value; Adjusting the reward data corresponding to the state data of the robot according to the first SHAP value, and adjusting the reward data corresponding to the action data of the robot according to the second SHAP value.

2. The method of claim 1, wherein, The action data comprises a plurality of different preset action data; The collecting of the target data generated by the game executed by the to-be-interpreted model comprises: Using the to-be-interpreted model to execute the game to obtain reward data obtained by preset action data corresponding to state data of each robot at each time step in the game, and a mapping relationship between the state data, the preset action data and the reward data; wherein the reward data is used to indicate the reward corresponding to the behavior of the robot in the game, and the robot is one-to-one corresponding to the preset action data.

3. The method of claim 1, wherein, The pre-training process of the neural network model comprises: Obtaining historical target data generated by the robot in historical games; wherein the historical target data comprises historical reward data obtained by historical action data corresponding to historical state data of the robot at each time in the historical game and a mapping relationship between the historical state data, the historical action data and the historical reward data; Using an offline reinforcement learning algorithm to learn an original strategy according to the historical target data to obtain the neural network model; If it is determined that the similarity between the training result of the neural network model according to the historical target data and the historical result corresponding to the historical target data reaches a preset similarity threshold, a decomposable reward expectation learning structure is introduced into the neural network model to obtain a pre-trained neural network model.

4. The method of claim 2, wherein, The using of the pre-trained neural network model to infer the expected value of the reward data corresponding to the state data and the action data according to the mapping relationship comprises: input the state data and the action data as a data pair into the neural network model, and perform forward reasoning on the data pair according to the mapping relationship by using the neural network model to obtain an expected value of reward data corresponding to the state data and the action data.

5. The method of claim 1, wherein, The integrated tree supervised model comprises: input the state data and the action data as features, input the reward data corresponding to the state data and the action data determined according to the mapping relationship as labels, and train an integrated tree supervised model according to the features and the labels.

6. The method of claim 5, wherein, The process of training the integrated tree supervised model according to the features and the labels comprises: learning, by using the integrated tree supervised model, a mapping relationship among the state data, the action data and the reward data.

7. The method of claim 1, wherein, After determining the first SHAP value of the state data with respect to the expected value and the second SHAP value of the action data with respect to the expected value, the method further comprises: calling a visualization tool to generate a visualized data analysis result, wherein the visualized data analysis result is used to indicate a distribution of the first SHAP value with respect to the state data and a distribution of the second SHAP value with respect to the action data.

8. The method of claim 1, wherein, The state data comprises first state data. The method of adjusting the reward data corresponding to the state data of the robot according to the first SHAP value comprises: determining whether the first SHAP value corresponding to the first state data reaches a first preset SHAP value threshold value; if the first SHAP value corresponding to the first state data reaches the first preset SHAP value threshold value, increasing the reward data corresponding to the first state data to a first preset reward data.

9. The method of claim 1, wherein, The state data comprises first state data. The method of adjusting the reward data corresponding to the state data of the robot according to the first SHAP value comprises: determining whether the first SHAP value corresponding to the first state data is lower than a second preset SHAP value threshold value; if the first SHAP value corresponding to the first state data is lower than the second preset SHAP value threshold value, decreasing the reward data corresponding to the first state data to a second preset reward data.

10. The method of claim 1, wherein, The action data comprises first preset action data. The method of adjusting the reward data corresponding to the action data of the robot according to the second SHAP value comprises: determining whether the second SHAP value corresponding to the first preset action data reaches a first preset SHAP value threshold value; if the second SHAP value corresponding to the first preset action data reaches the first preset SHAP value threshold value, increasing the reward data corresponding to the first preset action data to a first preset reward data.

11. The method of claim 1, wherein, The action data comprises first preset action data. The method of adjusting the reward data corresponding to the action data of the robot according to the second SHAP value comprises: determining whether the second SHAP value corresponding to the first preset action data is lower than a second preset SHAP value threshold value; If the second SHAP value corresponding to the first preset action data is lower than a second preset SHAP value threshold, the reward data corresponding to the first preset action data is reduced to second preset reward data.

12. A rewarding data analysis apparatus, characterized by comprising: The device comprises: The acquisition module is configured to acquire target data generated by a game executed by a to-be-interpreted model; wherein the target data comprises reward data obtained by executing action data under state data corresponding to a robot in the game at each time point, and a mapping relationship between the state data, the action data and the reward data; The first determination module is configured to infer an expected value of the reward data corresponding to the state data and the action data according to the mapping relationship by using a neural network model trained in advance; The second determination module is configured to determine a first SHAP value of the state data with respect to the expected value and a second SHAP value of the action data with respect to the expected value, comprising: using a TreeSHAP interpretive method to interpret an integrated tree supervised model trained to obtain the first SHAP value of the state data with respect to the expected value and the second SHAP value of the action data with respect to the expected value; wherein the first SHAP value is used to represent the influence degree of the state data on the expected value, and the second SHAP value is used to represent the influence degree of the action data on the expected value; The adjustment module is configured to adjust the reward data corresponding to the state data of the robot according to the first SHAP value, and adjust the reward data corresponding to the action data of the robot according to the second SHAP value.

13. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to realize the method of any one of claims 1-11.

14. A computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to make the computer realize the method of any one of claims 1-11.

15. A computer program product comprising computer program instructions which, when run on a computer, cause the computer to perform the method of any one of claims 1-11.

Images