A reward generation and optimization method for force confrontation reinforcement learning

Abstract

The application provides a reward generation and optimization method for force confrontation reinforcement learning. Firstly, the basic structure of a reward function hidden in an expert strategy is deduced by using a reverse reinforcement learning method based on existing human expert strategy based on experience rule reasoning and actual interaction examples of a simulation environment, so that the basic form of the original reward function for a specific complex force decision task is determined at a relatively coarse granularity. Then, the reward function is optimized based on the idea of reward remodeling. On the one hand, an internal exploration encouragement mechanism is introduced to guide the force intelligent agent to obtain an additional reward in a novel confrontation state, so as to increase the density and effectiveness of the reward signal. On the other hand, the cumulative reward in a multi-step sequence is used as a reward calculation unit to increase the stability of the reward signal.

Description

Technical Field

[0001] This invention relates to the field of intelligent force confrontation behavior decision modeling technology based on reinforcement learning. Specifically, it relates to a reward generation and optimization method based on inverse reinforcement learning and an incentive-exploration mechanism in force confrontation reinforcement learning. Background Technology

[0002] Computer technology is currently an important tool and effective approach in combat simulation research. Among them, Computer Generated Forces (CGF) technology is a crucial supporting technology for military simulation, especially distributed interactive combat simulation. CGF is a virtual combat force entity created by computers that can autonomously control or guide all or part of its own actions and behaviors. These entities can represent individual equipment platforms, weapon systems, or even entire combat units, and can interact with other virtual or real forces in a virtual battlefield environment built based on distributed interactive simulation technology.

[0003] Human combat behavior (such as situational awareness assessment and commander's command and control decision-making) has always been a key research focus in Combat Force Decision Modeling (CGF). This involves enabling equipment entities to automatically react and make rational decisions regarding the state and events of the virtual battlefield environment without human interaction, and to take corresponding actions and behaviors to complete designated tasks. Because force simulation systems are dynamic, complex, and large-scale systems, generally possessing complex nonlinear and uncertain characteristics, traditional game-theoretic behavioral modeling methods are insufficient for modeling complex war decision-making activities. The development of CGF behavioral modeling technology in combat simulation complements research in artificial intelligence, with its theoretical foundation deeply rooted in the field of AI. In recent years, with the rapid development and deepening of AI technology, methods for CGF behavioral decision-making modeling based on deep reinforcement learning technology have become a hot research topic.

[0004] In military confrontations, force agents based on deep reinforcement learning algorithms continuously learn from experience and update their deep neural networks through ongoing interaction with the battlefield environment, thereby guiding their continuous behavioral decisions. This has become a key technology that needs to be focused on and broken through in military intelligent confrontation behavioral decision-making. In reinforcement learning-based force behavioral decision-making modeling, the agent's goal is formally represented as a special scalar value signal called reward, which is transmitted to the agent through the simulation environment. The agent learns how to maximize its cumulative reward for a specific military confrontation task. Reward is the only signal guiding the agent's continuous intelligent evolution through interaction with the simulation environment. Reward is a function that needs to be designed manually, and mathematically it should reflect the top-level designer's understanding or cognition of the task goal that the agent must ultimately achieve. For example, in force confrontations, if an aircraft agent is to complete an autopilot mission, the reward should reflect the gap between the current aircraft state and the ideal cruise state; if an aircraft formation is to complete a group confrontation mission, the reward should represent the probability or certainty that the current aircraft formation can defeat the enemy formation.

[0005] Human design of rewards requires extreme caution, as it is inherently subjective and experience-based, involving continuous trial and error and adjustments. It largely depends on the designer's prior domain knowledge, mathematical foundation, and depth of understanding of the task problem, and even carries an element of luck. Therefore, an unreasonable reward design can easily lead to "goal bias" in the evolution of reinforcement learning-based force behavior decisions, failing to guide the agent towards the ideal task objective. Furthermore, under a conservative reward design bias, in complex joint force confrontation simulation scenarios, force agents are very prone to the dilemmas of "reward sparsity" and "reward delay": the agent cannot receive enough effective reward signals in a timely manner, resulting in slow learning and requiring long periods and extensive trial and error to find a good strategy, or even failing to learn effectively at all.

[0006] In typical force-on-force simulation scenarios, the numerous participating force platforms and the highly coupled behavioral decisions of each platform, coupled with the dynamic nature of the environment, often result in a highly random and complex combat process, with the environmental state rapidly shifting amidst these dynamic changes. Furthermore, joint force-on-force simulation tasks are often complex or involve multiple sub-tasks, making it difficult to mathematically define a specific reward function. If the combat task has a long-term objective, even with a clearly defined objective state, the agent still needs to engage in prolonged simulation interactions with the environment to reach the termination state, making reward sparsity and delays even more difficult to avoid. Simultaneously, the policy space that the force agent needs to explore is vast, potentially containing hundreds or even thousands of different tactical combinations that can effectively achieve the final objective. Therefore, facing complex force behavior decision-making problems, it is extremely difficult to design a reasonably structured, highly representative, and dense reward function to drive the force agent to complete the task using reinforcement learning algorithms.

[0007] In summary, the generation and optimization of reinforcement learning rewards in intelligent combat force behavior decision-making is a key issue that needs to be addressed. For specific joint force confrontation simulation tasks, generating reward functions that can effectively drive reinforcement learning force agents towards the task objective, and optimizing these functions to more efficiently guide the force behavior decision-making model to converge quickly during training, has significant theoretical and military application value in force confrontation simulation scenarios.

[0008] To address the problem of reward generation and optimization in reinforcement learning for intelligent combat force behavior decision modeling, this paper studies a reward signal generation and optimization method based on inverse reinforcement learning and reward reshaping. The abstraction of reward signal is a priori design problem. The designed reward function must, in principle, enable the agent to maximize the reward while also achieving the specific task objective. Therefore, the key to reward design is to truly represent the final objective that the agent needs to achieve in mathematical form. However, the design of reward function for force combat has the following problems: (1) Formal reward function design is difficult. In slightly complex force combat scenarios, when designing reward functions for intermediate steps, a slight misstep in the design of the reward function will cause "target tilt" in the evolution of force behavior decision, which will fail to guide the agent to approach the ideal task objective. The formal reward design requires great care. It is highly subjective and experiential, and largely depends on the designer's prior knowledge of the domain, mathematical foundation, and depth of understanding of the task problem. (2) In the simulation scenario of force combat, the problem of "sparse reward" is more obvious. First, force confrontation relies on real-time interaction with the underlying simulation system, and the advancement rate setting of the simulation system usually affects the instantaneous calculation of reward signals. Second, specific force confrontation tasks generally require long-term complex interactions to reach the termination state, and the effects of force behavior at intermediate time steps often have delayed feedback. Summary of the Invention

[0009] To overcome the design problems of reward functions in current force-based reinforcement learning applications, this invention aims to introduce inverse reinforcement learning algorithms into intelligent force behavior modeling. By combining an incentive-based exploration mechanism with a reward reconstruction mechanism for multi-step sequence units, a technical solution for reward function generation and optimization in force-based reinforcement learning is formed. The technical solution adopted in this invention first starts with existing examples of interaction between human expert strategies based on empirical rule reasoning and simulation environments. It uses inverse reinforcement learning to deduce the basic structure of the implicit reward function in the expert strategy, thereby determining the basic form of the original reward function for specific complex force decision-making tasks at a relatively coarse-grained level. Then, based on the idea of ​​reward reshaping, the reward function is optimized: on the one hand, an inherent incentive-based exploration mechanism is introduced to guide the force agent to obtain additional rewards in sufficiently novel adversarial states, increasing the density and effectiveness of the reward signal; on the other hand, the cumulative reward on the multi-step sequence is used as the reward calculation unit, increasing the stability of the reward signal.

[0010] The technical solution of the present invention is as follows:

[0011] A reward generation and optimization method for force-based reinforcement learning, the specific steps of which are as follows:

[0012] S1. Generate a reward function based on inverse reinforcement learning. The specific process is as follows:

[0013] Step 101: Use human expert strategy π E The driven intelligent agent interacts with the military confrontation environment for several rounds, resulting in multiple interaction sequences as shown in the following formula, which serve as a human example;

[0014] τ:s1,a1,s2,a2,…,s t ,a t ,…,s T ~π E

[0015] Where a certain interaction sequence is denoted by τ, s t Let a be the state at time step t. t For the action at time step t, a dataset consisting of m human examples of unrewarded interactions is obtained. Where τ i Represents the i-th interaction sequence;

[0016] Step 102: Obtain the reward value from the interaction data at each time step of the human example, and then back-infer the human expert policy π. E The implicit reward function in (a|s) is represented by the feature function shown below, where s is the current state and a is the human expert policy π. E The output action first selects the reward function r. * =w·φ(s,a):

[0017] r * (s,a;w)=w·φ(s,a)∈[0,1]

[0018] Where φ(s,a) is the state-action pair.<s,a> The characteristic basis functions at a given location reflect the state-action pair.<s,a> A mapping φ to a scalar value: S×A→[0,1] k Where S is the state space, A is the action space, and k represents the number of basis functions; the characteristic basis functions can be selected from one of the following: polynomial function, Fourier function, or radial basis function; where, A vector of parameter values ​​corresponding to a set of feature basis functions;

[0019] Step 103: Randomly initialize the unit vector

[0020] Step 104: After obtaining the reward value w·φ(s,a), further utilize this reward value to drive the agent in a positive reinforcement learning process, and select a reinforcement learning policy optimization method. The general policy π(a|s;θ) is optimized and the policy parameters θ are updated in the direction of maximizing the expected cumulative reward. This process is iterated continuously. The goal of training is to make the policy learned through positive reinforcement learning similar to the expected cumulative reward of the human expert policy. At this point, the reward network r... * (s,a;w) is the reward function derived from inverse reinforcement learning;

[0021] Step 105: Randomly initialize an original general policy π 0 (a|s;θ);

[0022] Step 106: Select a hyperparameter value ∈ that measures the performance gap between human expert strategies and general strategies; initialize the iteration round i = 1;

[0023] Step 107: Combining the reward function formula in Step 102, the cumulative expected reward can be derived as follows;

[0024]

[0025] Among them, E π Let t be the mathematical expectation of the human expert strategy, t be the time step index, T be the termination time step, and γ be the reward function decay factor.

[0026] The expected function of the policy-related features is defined as follows;

[0027]

[0028] Step 108: The expected cumulative reward under different policy drivers can be expressed as w·μ(π); given a dataset of m human examples... Afterwards, the characteristic expectation μ of the human expert strategy can be obtained. E As shown in the following formula:

[0029]

[0030] in, Let represent the state and action at the k-th time step in the i-th sequence, respectively;

[0031] Step 109: Based on the reward function r *(0) =w 0 ·φ(s,a), using strategy π 0 The system drives the intelligent agents of the forces to interact with the adversarial simulation environment in several rounds, resulting in a dataset consisting of rewarded interaction data. Based on this dataset, π is calculated using the following strategy: 0 Feature expectation μ (0) ;

[0032]

[0033] Step 110: Represent the policy performance using feature expectation. The goal of inverse reinforcement learning can be expressed as continuously optimizing the general policy π to a certain policy. Its characteristic expectation is similar to human expert strategy π E The characteristic expectations are similar, as shown in the following formula:

[0034]

[0035] Therefore, for any feature parameter ‖ρ‖≤1, the expected cumulative reward satisfies the following equation:

[0036]

[0037] in, For the mathematical expectation of the general strategy π, To optimize the strategy The mathematical expectation;

[0038] Step 111: Measure the performance of the policy using feature expectation, and find a policy π that sufficiently approximates the human expert policy. E General strategy Through multiple rounds of iterative optimization of the general strategy π, a series of strategies {π} are formed. i For each iteration of the function :i=0,…,n}, a maximum boundary value t>∈ is found according to the following formula, corresponding to the reward function r. * =w (i) ·φ, enabling human expert strategies π E The difference between the feature expectation and all policies before this iteration is always greater than t until such a maximum boundary value cannot be found. This indicates that the general policy has been optimized to be close enough to the human expert policy. The reward function corresponding to the boundary value of the previous iteration is the reward function inferred from the human expert policy. Now we start the i-th iteration.

[0039]

[0040] Where, max t,w t is the maximum value of t when t and w satisfy the st condition described in the formula, μ j Let be the mathematical expectation of the j-th general strategy;

[0041] Calculate t (i) =max w:‖w‖≤1 [min j∈{0,1…,(i-1)} w·(μ E -μ (j) )], obtain the parameter vector w that satisfies this formula. (i) And update the reward function to r *(i) =w (i)·φ(s,a);

[0042] Step 112: If t (i) If ≤∈, then the process ends and returns the policy sequence {π}. i :i=0,…,n} and reward function r *(i) =w (i) ·φ(s,a), the reward function output in this step is the reward function implicit in the human expert strategy obtained by inverse reinforcement learning back reasoning. This reward function serves as the native reward function in complex force behavior decision-making tasks; otherwise, continue.

[0043] Step 113: Use strategy optimization methods Based on the reward function r *(i) =w (i) The optimal strategy π is obtained by following the direction of maximizing the expected cumulative reward φ(s,a). (i) ; A dataset consisting of rewarded interactive data was obtained.

[0044] Step 114: Calculate strategy π based on this dataset according to the following formula. i Feature expectation μ (i) ;

[0045]

[0046] Step 115: Set the iteration round i = i + 1, and return to step 111;

[0047] S2. Reshape and optimize the reward function based on the incentive mechanism for exploration;

[0048] S3. Redesign the interaction sequence.

[0049] Preferably, the specific process of step S2 is as follows:

[0050] Step 201: Data preprocessing, initializing the sample space D, preparing positive and negative samples for training the classifier, using D + Let D represent the space of positive examples. - Represents the negative sample space;

[0051] Step 202: Collect training samples, and in advance, make the military agent interact with the military adversarial environment in n rounds according to a random strategy; let the state of the military agent at the i-th time step be... The state at the j-th time step is If |ij|≤k, then the state pair Store in D + Otherwise, change the state pair Store in D - ;

[0052] Step 203: Train the empirical comparison network using existing samples in the sample space D to perform logistic regression training;

[0053] Step 204: The classification metric reaches the expected value, and training is complete;

[0054] Step 205: Construct an experience storage space to store the old states that the agent has already passed in each round of interaction. The old states are those that have obtained additional rewards. The elements in the space are fixed-dimensional state feature vectors extracted from the force confrontation situation, denoted as M = {s1, s2, ..., s...}. |M|} represents the number of states in the current experience storage space;

[0055] Step 206: Construct an empirical comparison network to compare two states and obtain the "reachability" between the two states: Input two state vectors and output the probability that one state can be transitioned to another state within k steps. Use this probability value as the predicted value of the "reachability" between the two states; Assuming the dimension of the state vector is n, the input layer width of the empirical comparison network is 2n and the output layer width is 1, that is, the empirical comparison network fits the mapping C: S×S→[0,1];

[0056] Step 207: The military agent interacts with the environment. At each interaction time step, it extracts the current state vector s and compares it with each state vector {s1, s2, ..., s} in the experience storage space M through an experience comparison network. |M|} compare the current state with all states that have been visited in this round of interaction to obtain the "reachability" between the current state and all states that have been visited in this round of interaction, using c i This is represented and stored in the reachability cache space, as shown in the following formula:

[0057] c i =C(s) i ,s)∈[0,1],i=1,2,…,|M|

[0058] Where C(s) i (s) represents the current state s and the state vector s in the experience storage space m. i Accessibility between them;

[0059] Step 208: Aggregate all |M| "reachability" values ​​to obtain the overall reachability between the current state and the experience storage space, and score it as shown in the following formula:

[0060] C(M,s)=F(c1,c2,…,c |M| )∈[0,1]

[0061] Where F is an aggregation function, a hyperparameter, used to select one of |M| "reachability" as the overall reachability score between the current state and the memory storage space, which can be selected from the minimum, maximum, decimal, and ninetieths functions;

[0062] Step 209: After obtaining the overall reachability score between the current state and the experience storage space, calculate the additional reward for the current state using the following formula:

[0063] b s =B(M,s)=α(β-C(M,s))

[0064] Where B represents the additional reward function for the current state, and its formula is α(β-C(M,s)). β∈[0,1] are hyperparameters, α is used to adjust the scale of the additional reward signal to the same level as the original reward signal in the task; the value of β is a bias parameter that is adjusted according to the length of the interaction sequence in each interaction cycle;

[0065] Step 210: After obtaining the additional reward for the current state, compare it with the pre-set additional reward threshold b. notelty Compare the results; if the additional reward is above the threshold, add the current state to the experience storage space.

[0066] Step 211: As shown in the following formula, utilize the additional reward value b obtained through the exploration incentive mechanism. S The denser reward signal is obtained by adding the original reward value learned in step S1 based on inverse reinforcement learning.

[0067]

[0068] in, b represents the native reward value of state s obtained through inverse reinforcement learning in step S1; s This indicates that the additional reward obtained in state s is based on the incentive mechanism for exploration, and the sum of the two is the overall reward value in state s.

[0069] Preferably, the specific process of step S3 is as follows:

[0070] Step 301: Divide and reconstruct the original interaction sequence between the agent and the simulation environment into k steps; based on the original sequence, sample every k steps as a sequence unit, and connect n+1 consecutive sequence units to form a reconstructed sequence; the first state and corresponding action in each sequence unit are used as the state and action of the reconstructed sequence, and the index of the sequence unit corresponds to the index of the corresponding state, action and cumulative reward of the reconstructed sequence.

[0071] Within each sequence unit, the cumulative reward is calculated according to the following formula, and this cumulative reward serves as the reward value for the corresponding index in the reconstructed sequence;

[0072]

[0073] Where γ is the decay factor, which is adjustable as a hyperparameter and represents the degree of decay of the reward signal at different time steps; m k is a positive integer, adjustable as a sliding reward peak parameter, ranging from [1, k]. It represents which time step in the k-step sequence has the largest weight for the reward signal. Within the sequence unit, the reward signals on both sides of this time step decay proportionally according to the decay factor. m Let m be the reward function in the k-step sequence;

[0074] Step 302: The sequence step size k is an adjustable hyperparameter, and should be set based on a comprehensive consideration of the specific simulation scale and the characteristics of troop interaction time.

[0075] For the last (n+1)th sequence unit with a length of less than k steps, the termination state is directly taken as the reconstruction of this unit; this hyperparameter, together with the sequence step size k, determines the granularity of the multi-step sequence unit cumulative reward reconstruction process.

[0076] The present invention provides a reward generation and optimization method for force confrontation reinforcement learning, which has the following advantages:

[0077] 1. This invention introduces inverse reinforcement learning technology, which can make full use of the domain knowledge and data accumulated by rule sets, finite state machines and other behavior decision-making methods based on expert experience, and transform them into native reward functions that can be used in force confrontation reinforcement learning. It has high engineering usability and portability in the field of force behavior decision modeling.

[0078] 2. This invention introduces an intrinsic incentive mechanism based on reachability to encourage the simulated agent of the force to interact more fully with the simulation environment, promote the agent to actively approach the "unreachable state" and obtain additional rewards, thereby improving the incentive effect of the original reward signal and further improving the problem of slow learning of force behavior decision under sparse reward conditions.

[0079] 3. The reward reconstruction mechanism of the multi-step sequence unit proposed in this invention considers the multi-step interaction sequence between the force agent and the adversarial environment as a decision unit, enabling the force agent and the adversarial simulation environment to complete more sufficient interaction at multiple time steps, enhancing the stability and triggering capability of the reward signal, and effectively alleviating the problems of reward sparsity and reward delay.

[0080] 4. This invention forms a set of technical solutions for the generation and optimization of reinforcement learning reward signals in force confrontation behavior decision-making modeling scenarios, which can promote the effective application of reinforcement learning methods in force confrontation simulation scenarios in terms of theoretical exploration and technical means. Attached Figure Description

[0081] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the embodiments will be briefly described below. Referring to the accompanying drawings will provide a clearer understanding of the features and advantages of the present invention. The drawings are illustrative and should not be construed as limiting the present invention in any way. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort. Wherein:

[0082] Figure 1 This is a flowchart of the overall process for reward generation and optimization based on inverse reinforcement learning and reward reshaping.

[0083] Figure 2 This is a flowchart of reverse reinforcement learning.

[0084] Figure 3 This is a diagram illustrating the working principle of the mechanism that encourages exploration.

[0085] Figure 4 This is a schematic diagram of interactive sequence units.

[0086] Figure 5 This is a flowchart of the reverse reinforcement learning implementation process. Detailed Implementation

[0087] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other.

[0088] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0089] The technical solution adopted in this invention first starts with existing examples of interaction between human expert strategies based on empirical rule reasoning and simulation environments. It uses inverse reinforcement learning to deduce the basic structure of the implicit reward function in the expert strategy, thereby determining the basic form of the native reward function for specific complex force decision-making tasks at a relatively coarse-grained level. Then, based on the idea of ​​reward reshaping, the reward function is optimized: on the one hand, an inherent incentive mechanism is introduced to guide the force agent to obtain additional rewards in sufficiently novel adversarial states, increasing the density and effectiveness of the reward signal; on the other hand, the cumulative reward over a multi-step sequence is used as the reward calculation unit to increase the stability of the reward signal. The overall process is as follows: Figure 1 As shown, the specific steps are as follows:

[0090] The first step is the generation of the reward function based on inverse reinforcement learning: The native reward function is generated using inverse reinforcement learning. First, the task objective is analyzed, and appropriate feature basis functions are selected as linear fits to the reward function, determining the basic structure of the implicit reward function of the human expert policy at a coarse-grained level. Then, starting from existing examples of interaction between human expert policies based on empirical rules and the actual simulation environment, a forward reinforcement learning process is driven by the reward value output by this linear reward function. While optimizing the forward reinforcement learning policy to a level comparable to the human policy, the parameter values ​​of the linear reward function are updated simultaneously, thus deriving the basic form of the native reward function for complex force decision-making tasks. The inverse reinforcement learning process is as follows: Figure 2 As shown;

[0091] The second step involves reshaping and optimizing the reward function based on an incentive-based exploration mechanism: Building upon the idea of ​​reward reshaping, an intrinsic incentive-based exploration mechanism is introduced to guide the agent to actively explore potential "scarce experiences" under reward sparse conditions. Additional rewards are given to the agent in sufficiently novel states, further increasing the density and effectiveness of the reward signal. The working principle of the incentive-based exploration mechanism is as follows: Figure 3 As shown;

[0092] The third step is to reconstruct the interaction sequence: the cumulative reward on the multi-step sequence is used as the reward calculation unit to increase the stability and density of the reward signal, and to further optimize the reward function. The interaction sequence unit is illustrated as follows: Figure 4 As shown.

[0093] The overall technical solution of the reward reshaping generation method based on inverse reinforcement learning proposed in this invention is as follows: Figure 1 As shown, the entire system mainly consists of three parts: reward function generation based on inverse reinforcement learning, reward function reshaping and optimization based on an incentive-based exploration mechanism, and design for reconstructing interaction sequences. The specific technical solutions are as follows:

[0094] The first step is the generation of a reward function based on inverse reinforcement learning: starting from existing rule-based and knowledge-based human expert policies, the basic form of the reward function implicit in the human expert policies is inferred in reverse, representing the cognitive mechanism of solving task objectives implicit in human prior knowledge. This basic reward function is then used as the original reward signal for further reinforcement learning training, driving the military agent to continuously optimize its strategy using reinforcement learning algorithms, starting from the existing human expert policies, ultimately achieving better performance than the human expert policies. The specific process is as follows: Figure 5 As shown.

[0095] Specific steps:

[0096] Step 101: Use human expert strategy π E The driven intelligent agent interacts with the military confrontation environment for several rounds, resulting in multiple interaction sequences as shown in the following formula, which are called human examples.

[0097] τ:s1,a1,s2,a2,…,s t ,a t ,…,s T ~π E

[0098] Where a certain interaction sequence is denoted by τ, s t Let a be the state at time step t. t For the action at time step t, a dataset consisting of m human examples of unrewarded interactions is obtained. Where τ i Represents the i-th interaction sequence;

[0099] Step 102: Obtain the reward value from the interaction data at each time step of the human example, and then back-infer the human expert policy π. E The implicit reward function in (a|s) is represented by the feature function shown below, where s is the current state and a is the human expert policy π. E The output action first selects the reward function r. * =w·φ(s,a):

[0100] r * (s,a;w)=w·φ(s,a)∈[0,1]

[0101] Where φ(s,a) is the state-action pair.<s,a> The characteristic basis functions at a given location reflect the state-action pair.<s,a> A mapping φ to a scalar value: S×A→[0,1] k Where S is the state space, A is the action space, and k represents the number of basis functions; the characteristic basis functions can be selected from one of the following: polynomial function, Fourier function, or radial basis function; where, A vector of parameter values ​​corresponding to a set of feature basis functions;

[0102] Step 103: Randomly initialize the unit vector in, A vector of parameter values ​​corresponding to a set of basis functions;

[0103] Step 104: After obtaining the reward value w·φ(s,a), further utilize this reward value to drive the agent in a positive reinforcement learning process, and select a reinforcement learning policy optimization method. (As described above using the policy gradient method), the general policy π(a|s; θ) is optimized and the policy parameters θ are updated in the direction of maximizing the expected cumulative reward. This process is iterated continuously. The goal of training is to make the policy learned through positive reinforcement learning similar to the expected cumulative reward of the human expert policy. At this point, the reward network r... * (s,a;w) is the reward function derived from inverse reinforcement learning.

[0104] Step 105: Randomly initialize an original general policy π 0 (a|s;θ);

[0105] Step 106: Select a hyperparameter value ∈ to measure the performance gap between human expert strategies and general strategies; initialize the iteration round i = 1.

[0106] Step 107: Combining the reward function formula in Step 102, the cumulative expected reward can be derived as follows.

[0107]

[0108] Among them, E π Let t be the mathematical expectation of the human expert strategy, t be the time step index, T be the termination time step, and γ be the reward function decay factor.

[0109] The expected function of policy-related features is defined as follows.

[0110]

[0111] Step 108: The expected cumulative reward under different policy drivers can be expressed as w·μ(π). Given a dataset consisting of m human examples... Afterwards, the characteristic expectation μ of the human expert strategy can be obtained. E As shown in the following formula:

[0112]

[0113] in, Let represent the state and action at the k-th time step in the i-th sequence, respectively;

[0114] Step 109: Based on the reward function r *(0) =w 0 ·φ(s,a), using strategy π 0 The system drives the intelligent agents of the forces to interact with the adversarial simulation environment in several rounds, resulting in a dataset consisting of rewarded interaction data. Based on this dataset, π is calculated using the following strategy: 0 Feature expectation μ (0) .

[0115]

[0116] Step 110: Represent the policy performance using feature expectation. The goal of inverse reinforcement learning can be expressed as continuously optimizing the general policy π to a certain policy. Its characteristic expectation is similar to human expert strategy π E The characteristic expectations are similar, as shown in the following formula:

[0117]

[0118] Therefore, for any feature parameter ‖ρ‖≤1, the expected cumulative reward satisfies the following equation:

[0119]

[0120] in, For the mathematical expectation of the general strategy π, To optimize the strategy The mathematical expectation;

[0121] Step 111: Based on the above derivation, the problem can be transformed into: using the expected value of features to measure the performance of the policy, in order to find a policy π that sufficiently approximates the human expert policy. E General strategy Through multiple rounds of iterative optimization of the general strategy π, a series of strategies {π} are formed. i For each iteration of the function :i=0,…,n}, a maximum boundary value t>∈ is found according to the following formula, corresponding to the reward function r. * =w (i) ·φ, enabling human expert strategies π E The difference between the expected features and all policies before this iteration is always greater than t, until no such maximum boundary value can be found. This indicates that the general policy has been optimized to be close enough to the human expert policy. The reward function corresponding to the boundary value of the previous iteration is the reward function inferred from the human expert policy. Now we start the i-th iteration.

[0122]

[0123] Where, max t,wt is the maximum value of t when t and w satisfy the st condition described in the formula, μ j Let be the mathematical expectation of the j-th general strategy;

[0124] Calculate t (i) =max w:‖w‖≤1 [min j∈{0,1…,(i-1)} w·(μ E -μ (j) )], obtain the parameter vector w that satisfies this formula. (i) And update the reward function to r *(i) =w (i) ·φ(s,a).

[0125] Step 112: If t (i) If ≤∈, then the process ends and returns the policy sequence {π}. i :i=0,…,n} and reward function r *(i) =w (i) ·φ(s,a), the reward function output in this step is the reward function implicit in the human expert strategy obtained by inverse reinforcement learning back reasoning. This reward function can be used as the native reward function in complex force behavior decision-making tasks; otherwise, continue.

[0126] Step 113: Use strategy optimization methods Based on the reward function r *(i) =w (i) The optimal strategy π is obtained by following the direction of maximizing the expected cumulative reward φ(s,a). (i) ; A dataset consisting of rewarded interactive data was obtained.

[0127] Step 114: Calculate strategy π based on this dataset according to the following formula. i Feature expectation μ (i) .

[0128]

[0129] Step 115: Set the iteration round i = i + 1, and return to step 111.

[0130] The second step involves reshaping and optimizing the reward function based on an incentive mechanism for exploration. This involves introducing a mechanism into the reward design that intrinsically encourages agents to interact more actively and effectively with the adversarial simulation environment, rather than waiting for delayed reward signals. This reward function incentivizes agents to approach states that are difficult to reach and more "novel" to their existing "experience," thereby generating additional effective reward signals. Specifically, a neural network is constructed as a step predictor. The network input consists of state vectors representing two different adversarial situations, and the predicted number of steps required to separate these two states. The output of the step predictor is compared with a predefined step threshold k to determine whether a state belongs to "scarce experience." After determining whether a state belongs to "scarce experience," the reward intensity is increased for states in "scarce experience" and decreased for states in "familiar experience." The working principle is as follows: Figure 3 As shown.

[0131] Specific steps:

[0132] Step 201: Data preprocessing, initializing the sample space D, preparing positive and negative samples for training the classifier, using D + Let D represent the space of positive examples. - This represents the negative sample space.

[0133] Step 202: Collect training samples. To ensure the sample space covers the entire state space as comprehensively as possible, the agent is pre-processed to interact with the adversarial environment through n rounds according to a random strategy. Let the state of the agent at the i-th time step be... The state at the j-th time step is If |ij|≤k, then the state pair Store in D + Otherwise, change the state pair Store in D - .

[0134] Step 203: Train the empirical comparison network using existing samples in the sample space D to perform logistic regression.

[0135] Step 204: The classification index reaches the expected value, and the training is complete.

[0136] Step 205: Construct an experience storage space to store the old states that the agent has already explored in each round of interaction (here, old states are not all explored states, but only states that have obtained additional rewards). The elements in the space are fixed-dimensional state feature vectors extracted from the force confrontation situation, denoted as M = {s1, s2, ..., s...}. |M|} represents the number of states in the current experience storage space.

[0137] Step 206: Construct an empirical comparison network to compare two states and obtain the "reachability" between them. The input is two state vectors, and the output is the probability that one state can transition to another within k steps. This probability value is used as the predicted value of the "reachability" between the two states. Let the dimension of the state vector be n, then the input layer width of the empirical comparison network is 2n, and the output layer width is 1. That is, the empirical comparison network fits the mapping C: S×S→[0,1].

[0138] Step 207: The military agent interacts with the environment. At each interaction time step, it extracts the current state vector s and compares it with each state vector {s1, s2, ..., s} in the experience storage space M through an experience comparison network. |M|} compare the current state with all states that have been visited in this round of interaction to obtain the "reachability" between the current state and all states that have been visited in this round of interaction, using c i This is represented and stored in the reachability cache space, as shown in the following formula:

[0139] c i =C(s) i ,s)∈[0,1],i=1,2,…,|M|

[0140] Where C(s) i (s) represents the current state s and the state vector s in the experience storage space M. i Accessibility between them;

[0141] Step 208: Aggregate all |M| "reachability" values ​​to obtain the overall reachability between the current state and the experience storage space, and score it as shown in the following formula:

[0142] C(M,s)=F(c1,c2,…,c |M| )∈[0,1]

[0143] Here, F is an aggregation function, considered a hyperparameter, used to select one of |M| "reachability" criteria as the overall reachability score between the current state and the memory storage space. For example, it can be selected from functions such as minimum, maximum, decimal, and ninetieths.

[0144] Step 209: After obtaining the overall reachability score between the current state and the experience storage space, calculate the additional reward for the current state using the following formula:

[0145] b s =B(M,s)=α(β-C(M,s))

[0146] Where B represents the additional reward function for the current state, and its formula is α(β-C(M,s)). β∈[0,1] are hyperparameters, α is used to adjust the scale of the additional reward signal to the same level as the original reward signal in the task; the value of β is a bias parameter that is adjusted according to the length of the interaction sequence in each interaction cycle.

[0147] Step 210: After obtaining the additional reward for the current state, compare it with the pre-set additional reward threshold b. notelty A comparison is made. If the additional reward exceeds the threshold, the current state is added to the experience storage space.

[0148] Step 211: As shown in the following formula, utilize the additional reward value b obtained through the exploration incentive mechanism. S The denser reward signal is obtained by adding the original reward value learned in step 1 based on inverse reinforcement learning.

[0149]

[0150] in, b represents the native reward value of state s obtained through inverse reinforcement learning in step 1; s This indicates that the additional reward obtained in state s is based on the incentive mechanism for exploration, and the sum of the two is the overall reward value in state s.

[0151] The third step involves considering the multi-step interaction sequence between the agent and the adversarial environment as a single decision unit. This involves aggregating reward signals from multiple consecutive time steps to obtain the actual reward signal. This ensures sufficient interaction between the agent and the adversarial simulation environment across multiple time steps, enhancing the stability and triggering capability of the reward signal, and effectively mitigating the impact of reward sparsity and delay. A schematic diagram of the interaction sequence unit is shown below. Figure 4 As shown.

[0152] Specific steps:

[0153] Step 301: Segment and reconstruct the original interaction sequence between the agent and the simulation environment in k steps. Based on the original sequence, sample every k steps as a sequence unit, and connect n+1 consecutive sequence units to form a reconstructed sequence. The first state and corresponding action within each sequence unit are used as the state and action of the reconstructed sequence, and the index of the sequence unit corresponds to the index of the corresponding state, action, and cumulative reward in the reconstructed sequence.

[0154] Within each sequence unit, the cumulative reward is calculated according to the following formula, and this cumulative reward serves as the reward value for the corresponding index in the reconstructed sequence.

[0155]

[0156] Where γ is the decay factor, which is adjustable as a hyperparameter and represents the degree of decay of the reward signal at different time steps; mk is a positive integer, adjustable as a sliding reward peak parameter, ranging from [1, k]. It represents which time step in the k-step sequence has the largest weight for the reward signal. Within the sequence unit, the reward signals on both sides of this time step decay proportionally according to the decay factor. m Let m be the reward function in the k-step sequence;

[0157] Step 302: The sequence step size k is an adjustable hyperparameter, and its setting should be based on a comprehensive consideration of factors such as the specific simulation scale and the characteristics of force interaction time. A larger k generally corresponds to a smaller simulation scale and a greater force interaction delay, meaning more simulation clock steps are needed for the actual effect to occur after the force agent performs an action. This hyperparameter is related to the sliding reward peak parameter m. k Together, they determine the granularity of the multi-step sequence unit cumulative reward reconstruction process.

[0158] For the last (n+1)th sequence unit with a length of less than k steps, the termination state is directly taken as the reconstruction of this unit. This hyperparameter, together with the sequence step size k, determines the granularity of the cumulative reward reconstruction process for multi-step sequence units.

[0159] The above description is merely an example of the implementation of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A reward generation and optimization method for force-based reinforcement learning, characterized in that, The specific steps are as follows: S1. Generate a reward function based on inverse reinforcement learning. The specific process is as follows: Step 101: Using human expert strategies The driven intelligent agent interacts with the military confrontation environment for several rounds, resulting in multiple interaction sequences as shown in the following formula, which serve as a human example; Among them, a certain interaction sequence is used express, For the first The state at time step, For the first The action during the time step is obtained A dataset consisting of unrewarded interactions from human examples ,in Indicates the first 10 interactive sequences; Step 102: Obtain the reward value from the interaction data at each time step of the human example, and then back-infer the human expert strategy that needs to be derived. The implicit reward function is represented by the feature function shown in the following equation, where s is the current state and a is the human expert strategy. The output action first requires selecting the reward function. : in, State-action pair The characteristic basis functions at a given location reflect the state-action pair. A mapping to scalar values Where S is the state space and A is the action space. Indicates the number of basis functions; the characteristic basis functions are polynomial functions, Fourier functions, and radial basis functions. Step 103: Randomly initialize the unit vector ; Step 104: Obtain reward value This reward value is then used to drive the agent in a positive reinforcement learning process, and a reinforcement learning policy optimization method is selected. Optimize the general strategy in the direction of maximizing the expected cumulative reward. And update the strategy parameters This process is iterated continuously, with the training goal of making the cumulative reward expectation of the policy learned through positive reinforcement learning similar to that of the human expert policy. At this point, the reward network... This is the reward function derived from inverse reinforcement learning; Step 105: Randomly initialize an original general policy ; Step 106: Select a hyperparameter value to measure the performance gap between human expert strategies and general strategies. Initialize the iteration rounds ; Step 107: Combining the reward function formula in Step 102, the cumulative expected reward is derived as follows; in, Let t be the mathematical expectation of the human expert strategy, t be the time step index, and T be the termination time step. The reward function decay factor; The expected function of the policy-related features is defined as follows; Step 108: The expected cumulative reward under different strategies is expressed as follows: When given Dataset formed from human examples Subsequently, the expected characteristics of human expert strategies were obtained. As shown in the following formula: in, They represent the first No. 1 in the sequence The state and actions at each time step; Step 109: Based on the reward function Use strategy The system drives the intelligent agents of the forces to interact with the adversarial simulation environment in several rounds, resulting in a dataset consisting of rewarded interaction data. Based on this dataset, the calculation strategy is as follows: Feature expectation ; Step 110: Represent the performance of the policy using feature expectation. The objective of inverse reinforcement learning is to make the general policy... Continuously optimize to a strategy This makes its characteristic expectations consistent with human expert strategies. The characteristic expectations are similar, as shown in the following formula: Therefore, for any characteristic parameter The expected cumulative reward satisfies the following formula: in, For general strategies The mathematical expectation, To optimize the strategy The mathematical expectation; Step 111: Measure the performance of the policy using feature expectation, and find a policy that sufficiently approximates the strategy of human experts. General strategy : Optimize the general strategy through multiple rounds of iteration A series of strategies were formed in the process. In each iteration, a maximum boundary value is found according to the following formula. Corresponding reward function This enables human expert strategies The difference in feature expectation between this round and all previous strategies is always greater than The process continues until no such maximum boundary value can be found, indicating that the general policy has been optimized to be sufficiently close to the human expert policy. The reward function corresponding to the boundary value of the previous iteration is the reward function inferred from the human expert policy. The following section continues... Round iteration; in, for The formula is satisfied. The maximum value of t under the given condition. For the first Mathematical expectation of a general strategy; calculate Find the parameter vector that satisfies this equation. And update the reward function to ; Step 112: If If the condition is met, the process ends and the strategy sequence is returned. and reward function The reward function output in this step is the reward function implicit in the human expert strategy obtained through inverse reinforcement learning reasoning. This reward function serves as the native reward function in complex force behavior decision-making tasks; otherwise, continue. Step 113: Use strategy optimization methods Based on the reward function The optimal strategy is obtained by maximizing the expected cumulative reward. ; A dataset consisting of rewarded interactive data was obtained. ; Step 114: Based on this dataset, calculate according to the following strategy. Feature expectation ; Step 115: Set the iteration rounds Return to step 111; S2. Reshape and optimize the reward function based on the incentive mechanism for exploration; S3. Redesign the interaction sequence.

2. The reward generation and optimization method for force confrontation reinforcement learning according to claim 1, characterized in that, The specific process for step S2 is as follows: Step 201: Data preprocessing, initializing the sample space Prepare positive and negative samples for training the classifier, using... Represents the positive sample space. Represents the negative sample space; Step 202: Collect training samples and prepare the military agents to engage in combat against a randomized environment. Round-robin interaction; assuming the military intelligent agent is the first The state at each time step is , No. The state at each time step is ,like Then the state pair deposit Otherwise, change the state pair deposit ; Step 203: Using the sample space The existing samples in the network are used to train the empirical comparison network using logistic regression. Step 204: The classification metric reaches the expected value, and training is complete; Step 205: Construct an experience storage space to store the old states that the agent has already passed in each round of interaction. These old states are those that have already obtained additional rewards. The elements in the space are fixed-dimensional state feature vectors extracted from the force confrontation situation. It means that, among them This indicates the number of states in the current experience storage space; Step 206: Construct an empirical comparison network to compare the current state with the old state and obtain the "reachability" between the two states: Input two state vectors, output the reachability from one state to the other. The probability of transitioning to another state within one step is used as a prediction of the "reachability" between the two states; let the dimension of the state vector be . The input layer width of the empirical comparison network is... The output layer width is 1, meaning the empirical comparison network fits the mapping. ; Step 207: The military agent interacts with the environment, and at each interaction time step, extracts the current state vector. Through experience comparison network and experience storage space Each state vector in By comparing the current state with all states that have been visited in this round of interaction, we can obtain the "reachability" between the current state and all states that have been visited in this round of interaction. This is indicated and stored in the reachability cache space; As shown in the following formula: in Indicates the current state With experience storage space The state vector in Accessibility between them; Step 208: For all The overall reachability between the current state and the experience storage space is scored by aggregating the "reachability" data, as shown in the following formula: in, It is an aggregate function, which takes one hyperparameter and is used to extract from... One of the "reachability" criteria is selected as the overall reachability score between the current state and the memory storage space, and the selection is made from the minimum value, maximum value, decimal place, and ninetieth place functions; Step 209: After obtaining the overall reachability score between the current state and the experience storage space, calculate the additional reward for the current state using the following formula: in, B The additional reward function for the current state is expressed by the following formula: , and [0,1] is a hyperparameter. This is used to adjust the scale of the additional reward signal to the same level as the scale of the native reward signal in the task; The value is a bias parameter that is adjusted based on the length of the interaction sequence in each interaction cycle. Step 210: After obtaining the additional reward for the current state, compare it with the pre-set additional reward threshold. Compare the results; if the additional reward is above the threshold, add the current state to the experience storage space. Step 211: As shown in the following formula, utilize the additional reward value obtained through the exploration incentive mechanism. The denser reward signal is obtained by adding the original reward value learned in step S1 based on inverse reinforcement learning. in, This represents the state learned in step S1 based on inverse reinforcement learning. The native reward value below; This indicates a state obtained based on a mechanism that encourages exploration. The additional reward below, the two are added together to get the status. The overall reward value below.

3. A reward generation and optimization method for force confrontation reinforcement learning according to any one of claims 1-2, characterized in that, The specific process for step S3 is as follows: Step 301: Organize the original interaction sequence between the agent and the simulation environment according to... Perform segmentation and reconstruction step by step; based on the original sequence, every... Each step of sampling is performed as a sequence unit, continuously. Each sequence unit is connected to form a reconstructed sequence; the first state and corresponding action within each sequence unit serve as the state and action of the reconstructed sequence, and the index of the sequence unit corresponds to the index of the corresponding state, action, and cumulative reward of the reconstructed sequence; Within each sequence unit, the cumulative reward is calculated according to the following formula, and this cumulative reward serves as the reward value for the corresponding index in the reconstructed sequence; in, It is the decay factor, which is adjustable as a hyperparameter and represents the degree of decay of the reward signal at different time steps; It is a positive integer, and its value is adjustable as a parameter for the peak sliding reward, with a range of [range missing]. Between, indicating The reward signal at the time step with the largest weight in the sequence is determined, and the reward signals on both sides of this time step within the sequence unit are attenuated proportionally by the attenuation factor. for The first step in the sequence One reward function; Step 302: Sequence Step Size As a hyperparameter, it can be adjusted and set according to the specific simulation scale and the characteristics of troop interaction time. For the last one The length is insufficient For each step of the sequence unit, the terminated state is directly taken as the reconstruction of this unit; this decay factor With sequence step size Together, they determine the granularity of the multi-step sequence unit cumulative reward reconstruction process.

Images