Explainable reinforcement learning method for uav game playing
Patent Information
- Application Number
- CN202510554246.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-29
- Publication Date
- 2026-10-30
AI Technical Summary
[0004]本发明提供了一种用于无人机博弈的可解释强化学习方法，以解决深度强化学习模型可解性较差，无法大面积推广到无人机模拟对抗领域的问题
[0037]针对以往基于状态显著性解释模型的方法存在的对非图像状态环境适配性较差的问题，本发明方法利用状态均值作为状态特征掩码，通过状态特征掩码与真实状态特征相互组合形成新状态，利用新状态与真实状态的任务价值输出差值衡量特征对决策的贡献度；
Smart Images
Figure CN123014061A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to drone-based game competition, and more particularly to an interpretable reinforcement learning method for drone-based game competition. Background Technology
[0002] Unmanned aerial vehicle (UAV) game-playing adversarial planning is now widely used in the design and testing phases of various fields such as aerial reconnaissance and target surveillance. Deep reinforcement learning, as a major branch of machine learning, has become one of the mainstream methods in UAV game-playing planning due to the powerful expressive capabilities of its deep neural networks and their strong autonomous decision-making abilities in complex environments. However, the complex backbone structure and large number of parameters of deep neural networks also make the model's output decisions lack interpretability. This lack of interpretability directly leads to a lack of trust in the model's decisions by relevant users, and also directly results in the model being applicable only to simulations and unable to be widely promoted in real-world applications.
[0003] To address the poor interpretability of decision outputs in previous deep reinforcement learning models, researchers have conducted extensive research on interpretable reinforcement learning models. Currently, there are two main approaches. One focuses on improving the intrinsic interpretability of the model, such as using differential decision trees to approximate the decision-making mechanism of deep reinforcement learning models. Compared to deep neural networks, decision tree models have a more intuitive structure and a decision-making mechanism more similar to humans, thus possessing stronger interpretability. However, their simpler and more intuitive structure also leads to insufficient expressive power. In complex task environments such as drone games, it is impractical to model decisions for all states using simple decision trees. Forcedly constructed decision trees will also face decision errors due to decision approximation, which is unacceptable in the highly precise drone game environment. Other studies use decision path graphs to approximate the decision-making mechanism of deep reinforcement learning models, thus facing the same problems as the differential decision tree approach. The other approach focuses on generating post-hoc explanations of the model. For example, in the context of autonomous driving missions, saliency map algorithms, which interpret through perturbation inputs, are used. Saliency map algorithms can intuitively analyze the influence of different state features in an image state on the decision output. However, in the context of drone game planning missions, states are often non-image information. When perturbing the state by setting a mask, the mask cannot be set to zero as simply as in image states, making the selection of the state mask a research challenge. Other studies have adopted the LIME (Local Interpretable Model-Agnostic Explanations) method, which essentially still perturbs the state. In addition to facing similar problems as saliency map algorithms that interpret through perturbation inputs, these methods only output the state basis of the model's decision, without readily available quantitative indicators to measure the fidelity of the output's explanatory information. Summary of the Invention
[0004] This invention provides an interpretable reinforcement learning method for drone-based game simulation, addressing the problem that deep reinforcement learning models suffer from poor solvability and cannot be widely applied to drone-based adversarial simulation. This method combines state feature importance calculation with interpretation fidelity quantification to achieve model interpretability. On one hand, it calculates the importance of state features based on historical drone data; on the other hand, it quantifies the fidelity of the interpretation results, thereby objectively evaluating the quality of the interpretation.
[0005] The technical solution adopted in this invention is: an interpretable reinforcement learning method for drone gaming, which includes the following steps:
[0006] Step 1: Based on several configured reward indicators, set a corresponding sub-reward function for each reward indicator, and obtain the game adversarial reward function based on the weighted sum of all sub-reward functions;
[0007] A deep reinforcement learning model for a value network is constructed, which consists of a task value network group consisting of a game decision network and a task value network. The input of the game decision network is the drone state at the current decision step, and the output is the drone action at the current decision step. The input of the task value network is the drone state and drone action at the current decision step, and the output is the value at the current decision step.
[0008] Among them, the drone status includes several status features used to characterize the red and blue drones, such as the position coordinates of the red drone, the position coordinates of the blue drone, the altitude of the red drone, the speed of the red drone, the deviation angle between the orientation of the red drone and the line connecting the centroids of the red and blue drones, the deviation angle between the orientation of the blue drone and the line connecting the centroids of the red and blue drones, the angle coefficient of the red drone, and the angle coefficient of the blue drone.
[0009] Step 2: Construct an experience replay pool, and train and update the network parameters of the deep reinforcement learning model of the value network based on the game experience in the experience replay pool until the network converges.
[0010] Each game experience represents a state change brought about by a decision step, including the original state of the drone at the decision step, the drone's actions, each sub-reward, and the state of the drone in the next decision step (new state).
[0011] Step 3: Use the value network deep reinforcement learning model trained in Step 2 to interact with the environment, obtain multiple game trajectories, and calculate the mean of each state feature of the drone state at each decision step in each game trajectory.
[0012] Based on the established state feature importance measurement rules, the importance value f of each state feature to the decision of the deep reinforcement learning model of the value network is calculated. i Where, the subscript i is the state feature index of the UAV state;
[0013] Step 4, based on the maximum importance value f i The corresponding state features yield the key state features of a single decision step, i.e., the state features that have the greatest impact on the decision.
[0014] Step 5: Based on the fidelity of key state features in a single decision step, form and output explanatory information for the deep reinforcement learning model of the value network; where fidelity is used to characterize the influence of key state features on decision-making.
[0015] Furthermore, step 4 also includes:
[0016] Based on the set observation-decision step interval, the importance value f of each state feature within the observation-decision step interval is accumulated. i The importance value of the state characteristics of the observation decision step interval is obtained. Where n represents the starting point of the observation-decision step interval;
[0017] Based on all The key state features of the decision interval are obtained from the state features corresponding to the maximum value in the range.
[0018] In step 5, the fidelity of key state features regarding the decision interval is added to the explanatory information.
[0019] Furthermore, in step 3, the specific rule for measuring the importance value of state features is as follows:
[0020] f i =∑ S∈N (|Φ|-1)! (n-|Φ|)! [Q(Φ,a)-Q(Φ\{i},a)]
[0021] Where S represents the state space, N is the set of state features, and Φ is the average state vector s of all UAV states that contain state feature i and belong to any non-empty subset of N and include the game trajectory. mean The set formed by the missing state features in the set, where Φ\{i} is any non-empty subset of N that does not contain state feature i and is bounded by s. mean The set formed by the missing state features in the middle, where n represents the total number of state features and a represents the action.
[0022] Furthermore, the fidelity of the key state features for a single decision step is set as follows:
[0023]
[0024] Where φ(i) represents the fidelity of the key state feature i in a single decision step, V() represents the state value function of reinforcement learning, and s(i) represents the state value of the key state feature i.
[0025] Furthermore, the fidelity of the key state features in the decision interval is set as follows:
[0026]
[0027] Where t represents the decision step number, T represents the number of decision steps included in the decision interval, i.e., [n, n+T] is the corresponding decision interval, and V (t) (s(i)) represents the state value function value of the key state feature i, s(i), at decision step t.
[0028] Furthermore, the reward metrics include: angle, altitude, distance, energy, and events.
[0029] Furthermore, the drone's actions include: aileron output, elevator output, rudder output, and throttle output.
[0030] Furthermore, the game decision network and task value network of the deep reinforcement learning model of the value network are respectively set as follows:
[0031] The game-theoretic decision network consists of a many-to-one recurrent neural network, fully connected layers, and activation mapping layers. The drone state at the current decision step and the drone states at the previous k-1 time steps are input into the many-to-one recurrent neural network, where k represents the length of the network. The many-to-one recurrent neural network aggregates the input drone states and inputs the aggregated result into a network of size n. s ×n a In the fully connected layer, the fully connected layer maps its input to the range (-1, 1) based on the set activation function, and then maps the final output to the value range of the action space by multiplying by a scaling factor, outputting the drone action for the current decision step; where n s n a These represent the vector dimensions of the drone's state and its actions, respectively.
[0032] The task value network consists of a stitching layer and a fully connected layer. The stitching layer concatenates the input UAV state and UAV actions into state-action pairs, which are then input into a network of size (n). s +n a A fully connected layer of 1×1 outputs the value of the current decision step.
[0033] Furthermore, in step 2, network convergence specifically refers to: the loss L of the game strategy network. ω And the loss of the task value network Lθ convergence;
[0034] Among them, L θ =δ t Q θ (s,a); L ω =Q θ (s,a)logπ ω (s|a);
[0035] Where θ and ω represent the network parameters of the task value network and the game strategy network, respectively, and δ t Let δ be the TD-target for the t-th decision step. t =r t +γQ θ (s t+1 ,a t+1 )-Q θ (s t ,a t ), where γ is the hyperparameter discount factor, r t Let π be the adversarial reward function for the t-th decision step, where s represents the drone state and a represents the drone state; ω (s|a) represents the action a that the game strategy network needs to take in the current decision step, based on the drone's state s at the current decision step; Q θ (s,a) represents the output of the task value network.
[0036] The technical solution provided by this invention brings at least the following beneficial effects:
[0037] To address the issue of poor adaptability to non-image state environments in previous methods based on state saliency interpretation models, this invention uses the state mean as a state feature mask. A new state is formed by combining the state feature mask with the real state features. The difference between the task value output of the new state and the real state is used to measure the contribution of the features to the decision.
[0038] To address the limitations of previous interpretable deep reinforcement learning methods that focused on a single decision step, this invention extends the state importance analysis of a single decision step to the state feature importance analysis over a period of time, outputting the state that has the greatest impact on the decision over that period, thus explaining the state basis of the model's decision from a macro perspective.
[0039] To address the lack of quantitative metrics for interpretability performance in previous interpretable deep reinforcement learning methods, the method of this invention has a fixed quantitative standard to measure the fidelity of its interpretation. By defining a state value function and perturbing the input by changing important feature values, the difference between the upper and lower limits of the perturbed output is defined as the fidelity of the interpretation, thereby verifying the rationality of the interpretation. Attached Figure Description
[0040] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:
[0041] Figure 1 A flowchart illustrating an interpretable reinforcement learning method for unmanned aerial vehicle (UAV) gaming, provided in an embodiment of the present invention.
[0042] Figure 2 This is a schematic diagram of the structure of the deep reinforcement learning model for the value network in an embodiment of the present invention;
[0043] Figure 3 This is a diagram of the game decision network structure in an embodiment of the present invention;
[0044] Figure 4 This is a task value network structure diagram in the application embodiment. Detailed Implementation
[0045] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions of the embodiments of the present invention will be described in detail and completely below with reference to the accompanying drawings. Obviously, the embodiments described with reference to the accompanying drawings are exemplary and intended to explain the present invention, but should not be construed as limiting the present invention.
[0046] This invention proposes an interpretable reinforcement learning method for drone game playing. It is an interpretation method that combines the calculation of state feature importance and the quantification of interpretation fidelity. On the one hand, this invention can calculate the importance of state features based on historical drone data, and on the other hand, it can quantify the fidelity of the interpretation results, thereby objectively evaluating the quality of the interpretation results.
[0047] In one embodiment, see Figure 1 The aerial adversarial game method based on interpretable reinforcement learning provided in this embodiment of the invention includes the following steps:
[0048] Step 1: Construct a deep reinforcement learning model for the value network and initialize the network parameters;
[0049] In this embodiment of the invention, based on a number of reward indicators, a corresponding sub-reward function is set for each reward indicator, and the game-playing reward function r is obtained based on the weighted sum of all sub-reward functions;
[0050] The constructed value network deep reinforcement learning model includes a task value network group consisting of a game decision network and a task value network. The input of the game decision network is the drone state at the current decision step, and the output is the drone action at the current decision step. The input of the task value network is the drone state and drone action at the current decision step, and the output is the value at the current decision step. The drone state includes several state features used to characterize the red and blue drones.
[0051] Step 2, initialize the game environment;
[0052] Step 3: Collect game experience and update the network parameters of the deep reinforcement learning model of the value network, including the network parameters of the task value network and the network parameters of the game decision network.
[0053] Step 4: Check if the network has converged (whether the loss function has converged). If yes, proceed to step 5; otherwise, return to step 2.
[0054] Step 5: Calculate the state mean and, based on the set state feature importance value measurement rule, calculate the importance value of each state feature to the decision of the deep reinforcement learning model of the value network, and then calculate the state feature importance value of a single decision step; calculate the state feature importance value of the decision interval.
[0055] Step 6: Output explanatory information, that is, output decision explanation guarantee information for key feature states, which is the fidelity used to characterize the ability of key state features to influence decision-making.
[0056] In one embodiment, the sub-reward function and the game-playing reward function r of the method proposed in this invention are specifically set as follows:
[0057] (1) Design r (1) As an angular reward, its expression is:
[0058] r (1) =c1cosθ1-c2cosθ2+b#(1)
[0059] Where θ1 is the deviation angle between the orientation of the red drone and the line connecting the centroids of both the red and blue drones, and θ2 is the deviation angle between the orientation of the blue drone and the line connecting the centroids of both the red and blue drones. c1 is the angle coefficient of the red drone, c2 is the angle coefficient of the blue drone, c1, c2∈(0,1), where b is the global angle offset, b∈(-5,5);
[0060] (2) Design r (2) As a high reward, its expression is:
[0061] r (2) =(h-h2)-τ(h-h1) 2 #(2)
[0062] Where h is the altitude of our (Red) UAV relative to the ground. h1 is the optimal cruising altitude of the UAV. h2 is the warning altitude of the UAV; if the UAV flies below the warning altitude, there is a greater risk of crashing. τ∈(0,1) is the warning attenuation coefficient.
[0063] (3) Design r (3) As a distance reward, its expression is:
[0064]
[0065] Where (x1,y1,z1) and (x2,y2,z2) are the coordinates of the red UAV and the blue UAV in the NED coordinate system, respectively.
[0066] (4) Design r (4) As an energy reward, its expression is:
[0067]
[0068] Where g is the acceleration due to gravity and v is the speed of our drone.
[0069] (5) Design r (5) As an event reward, its expression is:
[0070] r (5) =εc b -c r +(1-ε)#(5)
[0071] Among them, c b The identifier for the blue team's plane crash is c. r ε is the red team's crash identifier. The crash identifier is 1 only if the plane crashes, and 0 otherwise. ε∈(0,1) is the blue team's influence factor.
[0072] (6) Calculate the game-playing reward function r:
[0073] r = a1r (1) +a2r (2) +a3r (3) +α4r (4) +a5r (5) +…#(6)
[0074] Where r represents each sub-reward function designed as in steps (1) to (5), and n represents the number of sub-reward functions. r The values can vary depending on the design concept and are not limited to the five values given in this embodiment. a1, a2, a3, a4, a5… are the weight coefficients of the sub-reward functions.
[0075] In one embodiment, the drone state constructed in this embodiment is n. s The (custom value) dimension state space S is:
[0076] S={x1,y1,z1,x2,y2,z2,h,v,θ1,θ2,c r ,c b ,…}#(7)
[0077] Where x1, y1, z1 represent the position coordinates of the red drone, x2, y2, z2 represent the position coordinates of the blue drone, h, v are the altitude and velocity of our (red) drone, θ1 is the deviation angle between the orientation of the red drone and the line connecting the centers of mass of both red and blue drones, θ2 is the deviation angle between the orientation of the blue drone and the line connecting the centers of mass of both red and blue drones, and c b The identifier for the blue team's plane crash is c. r This is the identifier for the Red Team's plane crash.
[0078] And constructing n a (Custom value) dimension action space:
[0079] A=(a1,a2,a3,a4)#(8)
[0080] Where a1, a2, a3, and a4 represent the aileron output (range of (-5, 5)), elevator output (range of (-3, 3)), rudder output (range of (-1, 1)), and throttle output (range of (-10, 10)) of the UAV, respectively.
[0081] like Figure 2 As shown, the deep reinforcement learning model for the value network in this embodiment of the invention is specifically configured as follows:
[0082] The deep reinforcement learning model for value networks consists of a game decision network (denoted as π) and a task value network C with parameter θ. The game decision network π, with parameter ω, takes the state of the current decision step as its input and outputs the action of the current decision step. Its process can be represented as follows:
[0083] a~π ω (a|s)#(9)
[0084] Where, π ω () represents a game-theoretic decision network based on parameter ω, a represents the UAV action vector, and s represents the UAV state vector.
[0085] The task value network C takes the state and action of the current decision step as input and outputs the value q of the corresponding state-action pair, expressed as:
[0086] q = Q θ (s,a)#(10)
[0087] Among them, Q θ () represents the output of the task value network C.
[0088] In one embodiment, the game decision network of the deep reinforcement learning model of the value network in this embodiment consists of a many-to-one recurrent neural network, a fully connected layer, and an activation mapping layer, as follows: Figure 3 As shown. For the state s of the i-th decision step. i The game decision network will s i The states of the first k-1 time steps are all input into a many-to-one recurrent neural network of length k. The many-to-one recurrent neural network converges s i The results include the state information of the first k-1 time steps, and are input into a system of size n. s ×n a In the fully connected layer, the activation mapping layer following the fully connected layer first uses the tanh activation function to map the input to the range (-1, 1), and then maps the final output to the range of the action space by multiplying by a scaling factor. The output is the action 'a' of the i-th decision step. i The task value network consists of a splicing layer and a fully connected layer, such as... Figure 4 As shown. The input to the task value network is the state s at a certain moment. i and action a i The splicing layer will display state s i With action a i Concatenate into state-action pairs (s i ||a i Then input the state-action pair into a system of size (n). s +n a A fully connected layer of size 1×1 outputs the value q of the i-th decision step. i .
[0089] In one embodiment, the experience replay pool constructed in this embodiment is:
[0090] A deep reinforcement learning model is used to continuously interact with the environment and the reward for each decision step is calculated according to formula (6). The state of the drone at a certain moment, the decision output by the agent according to formula (9), and the state of the drone at the next moment are recorded as the game experience tuple T:
[0091] T=(s i ,a i ,r i ,s i+1 )#(11)
[0092] The generated M game experiences are stored in the experience replay pool.
[0093] In one embodiment, the constructed multi-value network deep reinforcement learning model interacts with the environment to achieve cumulative reward convergence. This involves collecting game experience based on the game environment to update the network parameters of the multi-value network deep reinforcement learning model. The specific process is as follows:
[0094] 1) The parameters of all networks in the value network swarm are updated using the temporal difference algorithm and gradient descent algorithm, respectively, based on the experience obtained from random sampling from the experience replay pool, with the learning rate set to α. θ The parameter update expression is then:
[0095]
[0096] in, Q represents the gradient of parameter θ. θ Represents the task value network, δ t The TD-target for the t-th decision step is expressed as:
[0097] δ t =r t +γQ θ (s t+1 ,a t+1 )-Q θ (s t ,a t )#(13)
[0098] Where γ is the hyperparameter discount factor, r t Let t be the game adversarial reward function calculated by formula (6) for the t-th decision step.
[0099] 2) Update the parameters ω of the game decision network using the Bellman equation and gradient descent algorithm:
[0100]
[0101] Where, α ω Q is the learning rate. θ (s,a) represents the output of the task value network.
[0102] Repeat steps 1) to 2) until the game strategy network and the task value network converge.
[0103] After updating the network parameters of the deep reinforcement learning model of the value network, multiple game process records are output simultaneously, recording the state of all decision steps in the trajectory. The mean of all states is calculated and denoted as:
[0104]
[0105] The design of state feature importance values measures the importance of each state feature to the reinforcement learning model's decision. For a given n...s A state vector s, and the importance value f of the i-th state feature. i for:
[0106] f i =∑ s∈N (|Φ|-1)! (n-|Φ|)! [Q(Φ,a)-Q(Φ\{i},a)]#(16)
[0107] Where N is the set containing all state features, and Φ is any non-empty subset of N containing feature i, bounded by s. mean The set formed by the missing features in the set, where Φ\{i} is any non-empty subset of N that does not contain feature i and is bounded by s. mean The set formed by corresponding missing features.
[0108] The trained reinforcement learning model interacts with the environment, and the importance value of each state feature is calculated using formula (16), denoted as... Where t represents the t-th decision step, and i represents the i-th state feature in the state vector, by comparing different values of i... The magnitude of i is used to output the state feature that has the greatest impact on the model's decision in the t-th decision step. Therefore, the state feature i that has the greatest impact on the model's decision can be expressed by the formula:
[0109]
[0110] Set the observation decision step interval T. In this embodiment, it is set to T∈(10,1000). Assuming the starting point of the decision interval is n, calculate the importance value of the state features within T decision steps:
[0111]
[0112] in, This represents the importance of the decision feature from the t-th decision step. It is determined by comparing different values of i. The magnitude of i is used to output the state feature that has the greatest impact on the model's decision output during the period from the nth decision step to the (n+T)th decision step. Therefore, the state feature i that has the greatest impact on the model's decision output during the decision interval can be expressed by the formula:
[0113]
[0114] Calculate the fidelity of explanation for a single decision step. In this embodiment, the fidelity of explanation is defined as the ability of a specified state feature to influence the decision. For a state s in a decision step, the state value function V is defined as:
[0115] V(s)=E a (Q(s,a))#(20)
[0116] Among them, E a () represents the mathematical expectation of the drone's actions.
[0117] For the state feature i with the greatest influence obtained, the value of state s changes with the value of i, denoted as s(i). Then, the interpretation fidelity φ(i) of the method in this embodiment can be expressed by the formula:
[0118]
[0119] The higher the fidelity, the greater the influence of the most important state feature output by this embodiment on the decision-making process, i.e., the more accurate the interpretation.
[0120] Calculate the fidelity of the multi-decision-step explanation. For the state feature i with the greatest influence obtained, let V be the state value function of the t-th decision step. (t) Then, taking the nth decision step as the starting point, the fidelity Φ of the decision steps explained within a decision interval of size T is... n (i) can be expressed by the formula:
[0121]
[0122] Thus, when outputting explanatory information, explanatory information representing the current key state characteristics is formed based on the fidelity calculated by formulas (21) and (22).
[0123] Table 1 Parameter Table
[0124]
[0125]
[0126] In one embodiment, the specific implementation steps of the interpretable reinforcement learning method for drone gaming based on the relevant parameters given in Table 1 include:
[0127] Step S1: Construct a game decision network, in which the length of the many-to-one recurrent neural network is 8, the size of the fully connected layer 1 is 12×4, the activation function of the activation mapping layer is tanh, and the mapping coefficients of the four-dimensional action space are (5, 3, 1, 10).
[0128] Step S2: Construct the task value network, with the size of the fully connected layer being 16×1.
[0129] Step S3: Initialize the parameters of the game decision network and the task value network;
[0130] Step S4: Initialize the game-playing environment;
[0131] Step S5: Initialize the experience replay pool;
[0132] Step S6, record the current state as s i , will s i The input is fed into a game decision network to obtain the action output a. i ;
[0133] Step S7, calculate r according to formula (6) i ;
[0134] Step S8, execute action a i Interact with the environment and output a new state s i+1 ;
[0135] Step S9, transfer the experience (s) i ,a i ,r i ,s i+1 Stored in the experience replay pool;
[0136] Step S10: Repeat steps S4 to S9 until the game ends.
[0137] Step S11: Randomly sample 10 experience tuples from the experience replay pool and update the parameters of the game decision network and task value network using formulas (12) to (15);
[0138] Step S12, repeat steps 4 to 11 until the network loss decreases by less than 3%;
[0139] Step S13: Use a game-theoretic decision network to interact with the environment, save the state of 2000 decision steps during the interaction process and calculate the average state.
[0140] Step S14: Interact the trained game decision network with the environment to obtain the state-action sequence to be interpreted. Calculate the importance value of the state features of each decision step in the sequence according to formula (16) and the task value network, and determine the state feature with the greatest impact on the decision in each decision step according to formula (17).
[0141] Step S15: Using T=300 as the decision interval size, divide the sequence to be explained obtained in step 14 into several decision intervals. For each decision interval, use the task value network and formulas (18) and (19) to calculate the state feature that has the greatest impact on all decision steps within the decision interval.
[0142] Step S16: Calculate the fidelity of the decision state characteristics of a single decision step according to formula (21), and verify the reliability of the interpretation by comparing the calculation results with those of other state characteristics.
[0143] Step S17: Calculate the fidelity of the decision state characteristics of all decision steps within the decision interval according to formula (22), and verify the reliability of the interpretation by comparing the calculation results with those of other state characteristics.
[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the application, and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. An interpretable reinforcement learning method for drone gaming, characterized in that, Includes the following steps: Step 1: Based on several configured reward indicators, set a corresponding sub-reward function for each reward indicator, and obtain the game adversarial reward function based on the weighted sum of all sub-reward functions; A deep reinforcement learning model for a value network is constructed, which consists of a task value network group consisting of a game decision network and a task value network. The input of the game decision network is the drone state at the current decision step, and the output is the drone action at the current decision step. The input of the task value network is the drone state and drone action at the current decision step, and the output is the value at the current decision step. Among them, the drone status includes several status features used to characterize the red team drones and the blue team drones; Step 2: Construct an experience replay pool, and train and update the network parameters of the deep reinforcement learning model of the value network based on the game experience in the experience replay pool until the network converges. Each game experience represents the state change brought about by a decision step, including the original state of the drone at the decision step, the drone's actions, each sub-reward, and the state of the drone in the next decision step. Step 3: Use the value network deep reinforcement learning model trained in Step 2 to interact with the environment, obtain multiple game trajectories, and calculate the mean of each state feature of the drone state at each decision step in each game trajectory. Based on the established state feature importance measurement rules, the importance value f of each state feature to the decision of the deep reinforcement learning model of the value network is calculated. i Where, the subscript i is the state feature index of the UAV state; Step 4, based on the maximum importance value f i The corresponding state features yield the key state features of a single decision step, i.e., the state features that have the greatest impact on the decision. Step 5: Based on the fidelity of key state features in a single decision step, form and output explanatory information for the deep reinforcement learning model of the value network; where fidelity is used to characterize the influence of key state features on decision-making.
2. The method as described in claim 1, characterized in that, Step 4 also includes: Based on the set observation-decision step interval, accumulate the importance value fi of each state feature within the observation-decision step interval. i The importance value F of the state features of the observation decision step interval is obtained. i (n) , where n represents the starting point of the observation-decision step interval; Based on all F i (n) The key state features of the decision interval are obtained from the state features corresponding to the maximum value in the range. In step 5, the fidelity of key state features regarding the decision interval is added to the explanatory information.
3. The method as described in claim 1 or 2, characterized in that, In step 3, the specific rule for measuring the importance value of state features is as follows: f i =∑ S∈N (|Φ|-1)!(n-|Φ|)![Q(Φ,a)-Q(Φ\{i},a)] Where S represents the state space, N is the set of state features, and Φ is the average state vector s of all UAV states that contain state feature i and belong to any non-empty subset of N and include the game trajectory. mean The set formed by the missing state features in the set, where Φ\{i} is any non-empty subset of N that does not contain state feature i and is bounded by s. mean The set formed by the missing state features in the middle, where n represents the total number of state features and a represents the action.
4. The method as described in claim 1, characterized in that, The fidelity of key state features in a single decision step is set as follows: Where φ(i) represents the fidelity of the key state feature i in a single decision step, V() represents the state value function of reinforcement learning, and s(i) represents the state value of the key state feature i.
5. The method as described in claim 2, characterized in that, The fidelity of the key state features in the decision interval is set as follows: Where t represents the decision step number, T represents the number of decision steps included in the decision interval, i.e., [n, n+T] is the corresponding decision interval, and V (t) (s(i)) represents the state value function value of the key state feature i, s(i), at decision step t.
6. The method as described in claim 1 or 2, characterized in that, Reward metrics include: angle, altitude, distance, energy, and events.
7. The method as described in claim 1 or 2, characterized in that, The actions of a drone include: aileron output, elevator output, rudder output, and throttle output.
8. The method as described in claim 1 or 2, characterized in that, The drone status includes: the position coordinates of the red drone, the position coordinates of the blue drone, the altitude of the red drone, the speed of the red drone, the deviation angle between the orientation of the red drone and the line connecting the centers of mass of both red and blue drones, the deviation angle between the orientation of the blue drone and the line connecting the centers of mass of both red and blue drones, the angle coefficient of the red drone, and the angle coefficient of the blue drone.
9. The method as described in claim 1 or 2, characterized in that, The game decision network and task value network of the deep reinforcement learning model of the value network are set as follows: The game-theoretic decision network consists of a many-to-one recurrent neural network, fully connected layers, and activation mapping layers. The drone state at the current decision step and the drone states at the previous k-1 time steps are input into the many-to-one recurrent neural network, where k represents the length of the network. The many-to-one recurrent neural network aggregates the input drone states and inputs the aggregated result into a network of size n. s ×n a In the fully connected layer, the fully connected layer maps its input to the range (-1, 1) based on the set activation function, and then maps the final output to the value range of the action space by multiplying by a scaling factor, outputting the drone action for the current decision step; where n s n a These represent the vector dimensions of the drone's state and its actions, respectively. The task value network consists of a stitching layer and a fully connected layer. The stitching layer concatenates the input UAV state and UAV actions into state-action pairs, which are then input into a network of size (n). s +n a A fully connected layer of 1×1 outputs the value of the current decision step.
10. The method as described in claim 9, characterized in that, In step 2, network convergence specifically refers to: the loss L of the game strategy network. ω And the loss of the task value network L θ convergence; Among them, L θ =d t Q θ (s,a);L ω =Q θ (s,a)logπ ω (s|a); Where θ and ω represent the network parameters of the task value network and the game strategy network, respectively, and δ t Let δ be the TD-target for the t-th decision step. t =r t +γQ θ (s t+} ,a t+1 )-Q θ (s t ,a t ), where γ is the hyperparameter discount factor, r t Let be the adversarial reward function for the t-th decision step, s represent the drone state, and a represent the drone state. t+1 ,a t+1 Let s represent the UAV state and UAV action at decision step t+1, respectively; t ,a t Let π represent the UAV state and UAV action at decision step t, respectively; ω (s|a) represents the action a that the game strategy network needs to take in the current decision step, based on the drone's state s at the current decision step; Q θ (s,a) represents the output of the task value network.