A trajectory decision control method and system based on inverse critic guide imitation
Patent Information
- Application Number
- CN202611027454.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-10
- Publication Date
- 2026-09-15
Smart Images
Figure CN122755486A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence, reinforcement learning, and decision control technology, specifically to a trajectory decision control method and system based on inverse commentator-guided imitation. Background Technology
[0002] In scenarios such as robot control, autonomous driving, and intelligent scheduling, reinforcement learning enables agents to continuously select actions based on environmental states and optimize decision-making strategies through task rewards. These tasks typically require agents not only to obtain high task rewards but also to meet safety constraints such as collision, energy consumption, risk values, or resource consumption. However, conducting online reinforcement learning training directly in real-world environments is often costly and may introduce safety risks such as collisions, equipment damage, or system instability. Therefore, offline safety reinforcement learning trains strategies using fixed offline datasets, preventing agents from interacting with the environment during the training phase, thus making it suitable for decision-making and control tasks with high safety requirements.
[0003] Existing offline security reinforcement learning methods typically use the cumulative reward and cumulative cost in offline trajectories to guide policy learning. However, these trajectory-level signals mainly reflect the performance of existing behaviors in the dataset. When the learned policy deviates from the original data distribution, it may not accurately represent the actual benefits and security costs after the new policy is actually implemented, thus affecting the policy's performance and constraint satisfaction capabilities. Existing offline security reinforcement learning methods have at least the following drawbacks: (1) Distribution shift problem in offline training: Offline security reinforcement learning can only be trained using a fixed dataset and cannot correct the policy through additional environmental interactions. When the learned policy deviates from the original behaviors in the dataset, it is easy to enter states and action areas with insufficient data coverage, leading to a decrease in policy performance or an increase in security costs; (2) The cumulative reward and cumulative cost signals are not stable enough: Existing constraint sequence modeling methods typically use the cumulative reward and cumulative cost in offline trajectories as condition signals, but these signals mainly reflect the performance of existing trajectories and are difficult to accurately represent the actual benefits and security risks after the new policy is implemented; (3) Separating reward evaluation and security evaluation can easily lead to error accumulation: Some methods learn reward commentators and cost commentators separately and then guide policy updates through weighted combination. When both reward and cost estimates are biased, combined evaluation may mislead strategy selection, leading to overly conservative strategies or violations of safety constraints.
[0004] In summary, existing offline security reinforcement learning methods rely solely on fixed offline datasets and do not interact with the environment. When the learning strategy deviates from the original data distribution, it may not accurately represent the benefits and security costs of the new strategy after its actual execution, thus affecting the performance and constraint satisfaction of the strategy and resulting in low security of trajectory decision control. Summary of the Invention
[0005] To address the shortcomings of existing technologies that rely solely on fixed offline datasets without additional interaction with the environment, leading to decreased policy performance or increased security costs, this invention proposes a trajectory decision control method and system based on inverse commentator-guided imitation. By selecting safe and high-reward reference trajectories from offline data, an inverse commentator capable of jointly evaluating returns and safety is trained. This inverse commentator is then used to guide the learning of constraint sequence policies, thereby improving the safety, stability, and task performance of the policy under offline training conditions, thus solving the problems existing in the prior art.
[0006] A trajectory decision control method based on inverse commentator-guided imitation, applied to continuous decision-making tasks with safety constraints, includes the following steps: Obtain a fixed offline trajectory dataset of the target application object; each trajectory data includes status, action, reward, and security cost; and filter the trajectory data whose cumulative security cost meets the preset security cost budget from the fixed offline trajectory dataset to form a feasible trajectory set; Based on the set of feasible trajectories, a constraint sequence dataset containing state sequences, action sequences, reward targets, and safety cost budgets is constructed; at the same time, based on the cumulative reward, trajectory data with rewards higher than a threshold are selected from the set of feasible trajectories to construct a reference trajectory set. A decision model is trained using a set of feasible trajectories and a set of reference trajectories. The decision model includes an inverse commentator and a constraint sequence policy. The training process includes the following steps: The inverse commentator is trained using inverse reinforcement learning with the set of reference trajectories to jointly evaluate the task reward and safety of state-action pairs; the constraint sequence policy is trained using supervised imitation learning with the constraint sequence dataset to learn the actions of feasible trajectories in the constraint sequence dataset; the actions generated by the current policy are input into the trained inverse commentator to construct the inverse commentator guidance loss; based on the inverse commentator guidance loss, the constraint sequence policy is guided and optimized with the objective of maximizing the joint evaluation value; the inverse commentator and the constraint sequence policy are alternately updated until the training termination condition is met, and a decision policy that satisfies the safety cost constraint is output. The current trajectory dataset of the target application object is input into the trained decision model, and the action command at the current moment is output based on the constraint sequence strategy to control the target application object to complete the task.
[0007] Furthermore, the training process of the inverse commenter specifically includes the following steps: Given the current policy Define soft value function ,in Represents the entropy coefficient; Inverse commenter; 'a' represents action; 's' represents state; 'E' represents expectation; Based on the soft-value function, the inverse commenter is optimized using a training objective; the training objective is expressed as: ,in, Indicates the discount factor. This represents the initial state of the reference trajectory. This represents the set of reference trajectories; the training objective is to enable the inverse commentator to jointly evaluate the task benefits and safety of state-action pairs in the reference trajectories.
[0008] Furthermore, a gradient penalty term is introduced during the training of the inverse commentator; the gradient penalty term is used to limit the local variation of the inverse commentator near the offline data; the gradient penalty term is expressed as: ,in, Indicates the interpolated sample. This represents the target gradient norm.
[0009] Furthermore, the total loss function of the inverse commenter is the negative value of the training target plus the gradient penalty weight multiplied by the gradient penalty term.
[0010] Furthermore, the total training loss function of the constraint sequence policy includes supervised imitation loss and inverse commentator guidance loss, expressed as follows: ,in, This represents the supervised imitation loss of the sequence policy. This indicates the inverse commenter-guided loss. This indicates the guiding weight.
[0011] Furthermore, the supervised imitation loss is = ,in, The input sequence of constraints at time t represents the input condition. This represents the actions within the set of feasible trajectories. This represents a dataset containing a sequence of constraints.
[0012] Furthermore, the inverse commentator guided loss is the inverse commentator. Regarding the current state and strategy to generate actions The negative expected value of the evaluation value is expressed as = .
[0013] Furthermore, the set of feasible trajectories is determined based on a preset safety cost budget. Calculate the cumulative safety cost for each trajectory data point to filter the set of feasible trajectories that meet the safety constraints: ,in, Representing the trajectory The cumulative security costs.
[0014] The present invention also includes a trajectory decision control system based on inverse commentator-guided imitation, applied to continuous decision-making tasks with safety constraints, comprising: The acquisition module is used to acquire a fixed offline trajectory dataset of the target application object; each trajectory data includes status, action, reward and security cost; and selects trajectory data whose cumulative security cost meets the preset security cost budget from the fixed offline trajectory dataset to form a feasible trajectory set; The reference trajectory selection module is used to construct a constraint sequence dataset containing state sequences, action sequences, reward targets, and safety cost budgets based on the feasible trajectory set; at the same time, based on the cumulative reward, it selects trajectory data with rewards higher than a threshold from the feasible trajectory set to construct a reference trajectory set. The model training module is used to train a decision model using a set of feasible trajectories and a set of reference trajectories. The decision model includes an inverse commentator and a constraint sequence policy. The training process includes the following steps: training the inverse commentator using the set of reference trajectories through inverse reinforcement learning, enabling it to jointly evaluate the task benefits and safety of state-action pairs; performing supervised imitation learning on the constraint sequence policy using the constraint sequence dataset, enabling it to learn the actions of feasible trajectories in the constraint sequence dataset; inputting the actions generated by the current policy into the trained inverse commentator to construct the inverse commentator guidance loss; based on the inverse commentator guidance loss, optimizing the constraint sequence policy with the objective of maximizing the joint evaluation value; and alternately updating the inverse commentator and the constraint sequence policy until the training termination condition is met, outputting a decision policy that satisfies the safety cost constraint. The control module is used to input the current trajectory dataset of the target application object into the trained decision model, and output the action command at the current moment based on the constraint sequence strategy to control the target application object to complete the task.
[0015] This invention provides a trajectory decision control method based on inverse commentator-guided imitation, which has the following beneficial effects: This invention constructs a reference trajectory set by selecting safe and high-reward trajectories from a fixed offline dataset and trains an inverse commentator using inverse reinforcement learning. This commentator jointly evaluates the reward and safety of state-action pairs, providing a stable and reliable evaluation signal for the policy during training and effectively reducing the adverse effects of distribution offset on policy learning. The invention uses a single inverse commentator to jointly evaluate reward and safety, and alternately updates the inverse commentator and the constraint sequence policy. This allows the policy to not only mimic safe trajectories but also tend to select safe and high-reward actions with high inverse commentator evaluation scores, thus achieving higher task rewards while satisfying safety cost constraints. Furthermore, the invention employs inverse commentator-guided loss instead of traditional trajectory-level conditional signals, guiding the policy optimization towards maximizing the inverse commentator evaluation value, avoiding inconsistencies between trajectory-level signals and the actual policy execution. This method, relying solely on a fixed offline dataset and without additional interaction with the environment, can learn decision-making policies that simultaneously satisfy safety constraints and task reward requirements, making it suitable for continuous decision-making tasks in safety-critical fields such as robot control, autonomous driving, and intelligent scheduling. Attached Figure Description
[0016] Figure 1 This is a schematic diagram of the trajectory decision control method based on inverse commentator-guided imitation in an embodiment of the present invention. Detailed Implementation
[0017] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.
[0018] This invention proposes a trajectory decision control method based on inverse commentator-guided imitation. Under a fixed offline dataset, this method first filters feasible trajectories that satisfy the safety cost budget, then selects high-reward trajectories as a reference trajectory set. Subsequently, it uses this reference trajectory set to train an inverse commentator, enabling it to jointly evaluate the reward and safety of states and actions. Finally, the evaluation results of the inverse commentator are incorporated into the constraint sequence policy learning process to generate a decision policy that satisfies the safety cost constraints.
[0019] like Figure 1 As shown, the method specifically includes the following steps: S1. Obtain the fixed offline trajectory dataset of the target application object; the target application object is robot control, autonomous driving, intelligent scheduling, and other continuous decision-making tasks with safety constraints.
[0020] Fixed offline trajectory dataset Each trajectory includes state, action, reward, and safety cost information, and can be represented as: ,in, Indicates state, Indicates an action, Indicates a reward. This represents the security cost. No further interaction with the environment is required during training. The fixed offline dataset refers to a dataset collected before training and not expanded during training; this dataset is typically generated from historical strategies, expert demonstrations, or existing system operation records, and contains information such as state, actions, rewards, and security costs. All training steps in this invention are completed based on the fixed offline dataset, requiring no additional environmental interaction.
[0021] S2. Select trajectory data from the fixed offline trajectory dataset whose cumulative safety cost meets the preset safety cost budget to form a feasible trajectory set, specifically through: Based on the preset security cost budget Calculate the cumulative safety cost for each trajectory and select the set of feasible trajectories that meet the safety constraints: ,in, Representing the trajectory The cumulative security costs.
[0022] S3. Based on the cumulative reward, select trajectory data with rewards higher than the threshold from the feasible trajectory set, and construct a high-reward reference trajectory set, specifically as follows: In the set of feasible trajectories In the middle, the tracks are sorted according to cumulative rewards, and the tracks with higher rewards are selected. Construct a reference trajectory set from the given trajectories: ,in, This represents a set of reference trajectories that simultaneously satisfy safety constraints and offer high mission rewards.
[0023] S4. Construct a constraint sequence dataset based on the feasible trajectory set: Based on the feasible trajectory set Construct a constraint sequence dataset for training the serialization model. This dataset is the same as the feasible trajectory data, except that the trajectories are modified into a serialized form. The dataset includes state sequences, historical action sequences, reward objectives, and safety cost budgets, which are used to train constraint sequence policies.
[0024] S5. The decision model is trained using the feasible trajectory set and the reference trajectory set. The decision model includes an inverse commenter and a constraint sequence policy to output a decision policy that satisfies the safety cost constraint.
[0025] S5.1 Inverse Commentator Training: Constructing the Inverse Commentator and using a set of reference trajectories Training is then performed. The inverse evaluator does not estimate rewards and costs separately, but instead uses inverse reinforcement learning to jointly evaluate state-action pairs, reflecting both task gains and safety. During training, given the current policy... Define a soft-value function: ,in Let represent the entropy coefficient. The training objective of the inverse commenter can be expressed as: ,in, Indicates the discount factor. Indicates the initial state of the reference trajectory; This represents the state of the trajectory at the next moment, starting from the current moment. The objective is to enable the inverse commentator to assign higher ratings to safe, high-reward behaviors within the reference trajectory.
[0026] In this invention, the inverse commentator is typically used to evaluate the value of a state or action. The inverse commentator is an evaluation model trained using a safe, high-reward reference trajectory, used to uniformly score state-action pairs. Unlike training reward and cost commentators separately, this invention uses a single inverse commentator to simultaneously reflect task rewards and safety, thereby reducing the error accumulation caused by the separation of reward and safety evaluations.
[0027] Inverse reinforcement learning is a learning method that infers evaluation signals from demonstrated behaviors. While ordinary reinforcement learning typically learns a policy given a known reward function, inverse reinforcement learning learns "what states and actions are better" from expert or reference behaviors. This invention utilizes the concept of inverse reinforcement learning to train an inverse evaluator from safe and high-reward reference trajectories, enabling it to jointly evaluate the reward and safety of actions.
[0028] S5.2 Gradient Penalty Stabilization: A gradient penalty term is added during the training of the inverse commentator to limit the local variation of the inverse commentator near offline data. ,in, Indicates the interpolated sample. Let represent the target gradient norm. The total loss of the inverse commentator is: ,in, This represents the gradient penalty weights. This step is used to improve the training stability of the inverse commentator and reduce the adverse effects of offline distribution shift. It is the L2 norm of the gradient of the inverse commentator.
[0029] Gradient penalty is a regularization method that limits the magnitude of changes in the output of a neural network. During offline training, the policy may encounter states and actions with insufficient data coverage, leading to unstable evaluations by the commentator in these areas. This invention incorporates gradient penalty during the training of the inverse commentator, making its changes near offline data smoother and thus reducing the adverse effects of distribution shift on policy learning.
[0030] S5.3, Policy Learning Guided by Inverse Commentator: Constructing Constraint Sequence Policies This allows the policy to generate actions based on state sequences, historical actions, reward objectives, and safety cost budgets. The policy first learns actions in feasible trajectories from a dataset of constrained sequences using supervised imitation loss. = ,in, This represents the sequence of constraints input at time t. Further, it represents the actions generated by the current policy. Input the inverse commentator, construct the inverse commentator guided loss: = The strategy training includes both imitation learning loss and inverse commentator guidance loss: ,in, This represents the supervised imitation loss of the sequence policy. This indicates the inverse commenter-guided loss. This indicates the guiding weight.
[0031] Among them, the constrained sequence policy model is a type of policy model that represents the decision-making process as a sequence generation problem. This model can take a state sequence, historical actions, reward objectives, and safety cost budget as inputs, and output the current or future action. This invention adds an inverse commentator to this type of policy model, making the policy not only mimic safe trajectories but also tend to select actions that are rated higher by the inverse commentator.
[0032] S5.4 Alternating Training: During training, the inverse commentator and the constraint sequence policy are updated alternately. The inverse commentator is used to learn the evaluation signals of safe, high-reward actions, while the policy is guided by the inverse commentator based on imitating safe trajectories.
[0033] S5.5 Strategy Output: After training, the output is a decision strategy that satisfies the safety cost constraint. This strategy can generate actions under a given safety budget and can be used for continuous decision-making tasks with safety constraints, such as robot control, autonomous driving, and intelligent scheduling.
[0034] S6. Input the current trajectory dataset of the target application object into the trained decision model, and output the action command at the current moment based on the constraint sequence strategy to control the target application object to complete the task.
[0035] This invention can be implemented in offline security reinforcement learning benchmark tasks. The following examples use three environments from the DSRL benchmark, including SafetyGym, BulletGym, and MetaDrive; each embodiment is trained using a fixed offline dataset, and no additional interaction with the environment is performed during training.
[0036] For overall performance, the following two types of quantitative indicators are designed: 1) Normalized Reward This value is used to measure the ability of a strategy to complete a given task. A higher value indicates better task performance. The calculation formula is as follows: ; in, The trajectory reward obtained from the execution of the strategy. and The maximum and minimum reward values for the corresponding task in the offline dataset are calculated.
[0037] 2) Normalized Cost The value (), used to measure whether a strategy meets security constraints, indicates that the lower the value, the more secure the strategy. Its calculation formula is:
[0038] ; in, The cumulative cost obtained from the execution of the strategy. To pre-set a security budget. When At that time, it is assumed that the strategy satisfies the security constraints.
[0039] This experiment sets up three baseline methods for comparison: 1) BC-Safe is a safe behavior cloning method. This method uses only trajectories whose cumulative cost satisfies the safety budget for supervised learning, making the policy imitate the actions in the safe trajectory as closely as possible. This method does not require the construction of additional reward or cost functions, the training process is relatively simple, and it usually achieves good safety, thus serving as a basic comparative method for "imitation based solely on safe data". However, BC-Safe is essentially still a supervised imitation learning method; it can only copy actions from existing safe trajectories and lacks the ability to further search for high-reward actions. Therefore, when the safe trajectory itself is conservative or the task scenario is complex, this method is prone to achieving lower task rewards.
[0040] 2) CAPS (IQL) is a constraint-adaptive policy switching method based on implicit Q-learning. This method does not train a single fixed policy, but rather learns a set of candidate policies with different reward-cost trade-offs on offline data. During testing, it selects the policy from the candidate policies that is expected to satisfy the cost constraints and has higher future rewards, based on the current security budget and state. This method can adapt to different security constraints and is suitable as a comparative method for "multi-policy switching security reinforcement learning". However, CAPS (IQL) still relies on the trade-off between reward and cost for policy selection. When reward estimation, cost estimation, or policy switching judgment is inaccurate, problems may arise such as overly conservative policies or unstable satisfaction of security constraints.
[0041] 3) B2R (Boundary-to-Region Supervision) is a constraint sequence modeling method based on boundary-to-region supervision. This method improves upon the symmetrical treatment of cumulative reward and cumulative cost in the safety decision transformer, arguing that cumulative reward can serve as a flexible performance objective, while cumulative cost should represent a strict safety boundary. Therefore, B2R realigns the cost condition signals, unifying trajectories that satisfy the safety budget under the same safety boundary, enabling the sequence model to learn action generation strategies within the safety region. This method enhances the safety and task performance of constraint sequence models and is suitable as a comparative method for "improved safety decision transformers." However, B2R still primarily relies on reward and cost arrival values from offline trajectories as condition signals. When the learned strategy deviates from the original data distribution, these trajectory-level condition signals may not be entirely consistent with the actual benefits and safety after the strategy is executed.
[0042] Example 1: SafetyGym Safety Navigation Control Task Composition: This embodiment uses the SafetyGym environment from the DSRL benchmark. This type of task mainly simulates the navigation and control of an intelligent agent in a scenario with obstacles, target areas, and safety constraints. The intelligent agent needs to select actions based on its own state and environmental observations, maximizing task rewards while controlling safety costs such as collisions, boundary crossings, or entering dangerous areas.
[0043] Training: A fixed offline dataset is used as the training data. First, feasible trajectories that meet the constraints are selected based on a preset safety cost budget. Second, trajectories with higher cumulative rewards are selected from the feasible trajectories as the reference trajectory set. Then, an inverse commentator is trained and gradient penalty is added to stabilize the training process. Finally, the inverse commentator is used to guide the learning of the constraint sequence policy.
[0044] Effects: In the SafetyGym environment, the present invention achieved an average normalized reward of 0.74 and an average normalized cost of 0.58. The normalized cost is lower than the safety threshold of 1, indicating that the present invention can maintain a high task reward while meeting safety constraints.
[0045] Example 2: BulletGym Robot Safety Control Task Composition: This embodiment uses the BulletGym environment from the DSRL benchmark. This type of task primarily simulates the motion control problem of robots or intelligent agents in a physically simulated environment, including running, circling, and obstacle avoidance. The intelligent agent needs to maintain high motion performance while ensuring that the cumulative safety cost does not exceed a preset budget.
[0046] Training: The same offline training method as in Example 1 is used. A set of feasible trajectories is obtained by filtering the fixed offline dataset through safe trajectories, and then high-reward trajectories are selected from this set to train the inverse reviewer. During the policy learning phase, the state sequence, historical actions, reward objective, and safety cost budget are used as inputs, and the evaluation results from the inverse reviewer guide action generation.
[0047] Effects: In the BulletGym environment, this invention achieves an average normalized reward of 0.66 and an average normalized cost of 0.65. Compared to safety behavior cloning and constraint sequence baselines, this invention achieves higher task rewards while maintaining safety constraints satisfied.
[0048] Example 3: MetaDrive Automated Driving Offline Safety Control Task Composition: This embodiment uses the MetaDrive autonomous driving task from the DSRL benchmark test. This type of task includes driving scenarios with different difficulties and traffic densities. The agent needs to select driving actions based on vehicle status, road information, and historical actions, while controlling safety costs such as collisions and boundary crossings to maximize the reward of the driving task.
[0049] Training: A fixed offline driving dataset is used for training. First, driving trajectories that meet the safety cost budget are selected; then, a reference trajectory set is constructed by selecting trajectories with higher cumulative rewards; next, an inverse commentator is trained using the reference trajector, and gradient penalty is used to improve the stability of the inverse commentator; finally, the evaluation results of the inverse commentator are incorporated into the constraint sequence policy training process to obtain a driving policy that meets the safety budget.
[0050] Results: In the MetaDrive environment, this invention achieved an average normalized reward of 0.76 and an average normalized cost of 0.61. The results demonstrate that this invention can balance task performance and safety constraints in complex driving scenarios, avoiding the safety cost exceeding limits problem that often occurs with ordinary offline strategies.
[0051] The innovation of this invention lies in: (1) Reference trajectory construction method based on safety constraints and task reward: First, select feasible trajectories that meet the safety cost budget from the fixed offline dataset, and then select high reward trajectories from the feasible trajectories to construct a set of safe and high reward reference trajectories.
[0052] (2) Training method of single inverse commentator based on inverse reinforcement learning: The inverse commentator is trained by using a safe and high-reward reference trajectory, so that it can jointly evaluate the reward and safety of state-action pairs, instead of training the reward commentator and the cost commentator separately.
[0053] (3) Introducing a stabilization method of gradient penalty in the training of the inverse reviewer: By constraining the local variation of the inverse reviewer near the offline data through gradient penalty, the impact of offline distribution offset on the reviewer evaluation results can be reduced.
[0054] (4) Inverse commenter-guided constraint sequence policy learning method: The evaluation results of the inverse commenter are introduced into the constraint sequence policy training process, so that the policy tends to select high-return, low-risk actions on the basis of imitating the safe trajectory.
[0055] The present invention has the following beneficial effects: (1) This invention demonstrates good security and task performance in various offline security reinforcement learning tasks. In the DSRL benchmark test, this invention was validated in three environments: SafetyGym, BulletGym, and MetaDrive. In the SafetyGym task, the average normalized reward of this invention reached 0.74, and the average normalized cost was 0.58; in the BulletGym task, the average normalized reward of this invention reached 0.66, and the average normalized cost was 0.65; in the MetaDrive task, the average normalized reward of this invention reached 0.76, and the average normalized cost was 0.61. The average normalized cost in the above three environments is lower than the security threshold of 1, indicating that this invention can maintain a high task reward while meeting the security cost constraint.
[0056] (2) Compared with ordinary behavior cloning, safe behavior cloning, traditional constraint value function methods, and constraint sequence models, this invention achieves a more stable balance between reward and safety. Although ordinary behavior cloning can obtain higher rewards in some tasks, it is prone to generating high safety costs; although safe behavior cloning can reduce costs, it usually sacrifices rewards significantly. This invention learns an inverse commentator from safe, high-reward trajectories and uses the inverse commentator to guide policy updates, so that the policy does not simply imitate safe trajectories, but further tends to select actions that have both higher rewards and lower risks. Therefore, this invention achieves higher or near-optimal normalized rewards in multiple tasks, while remaining within the overall safety constraints.
[0057] (3) This invention improves the stability of the inverse commentator under offline training conditions by introducing a gradient penalty term. Offline security reinforcement learning cannot correct erroneous policies through online interaction. Therefore, if the commentator has a large estimation error in a region with insufficient data coverage, it is easy to lead the policy to produce unsafe actions. This invention uses gradient penalty to limit the variation of the inverse commentator in local regions, making the inverse commentator less sensitive to distribution shifts. Experimental results show that after appropriately adding gradient penalty, the normalized cost and normalized reward of the policy are more stable, which can reduce the problem of security budget violation caused by unstable commentator estimation.
[0058] (4) The inverse commentator trained by this invention can provide more reliable safety guidance than the traditional combination of reward commentator and cost commentator. In boundary state analysis, traditional methods usually combine reward commentator and cost commentator by weight, but when some states have both high rewards and high risks, this combined evaluation is prone to being misleading. This invention uses a single inverse commentator to jointly evaluate benefits and safety, which can more clearly distinguish between safe high-reward states, misleading states, and dangerous states, so that the strategy can still obtain more accurate action guidance when approaching the safety boundary.
[0059] Based on the above inventive concept, this invention also proposes a trajectory decision control system based on inverse commentator-guided imitation, comprising: The acquisition module is used to acquire a fixed offline trajectory dataset of the target application object; each trajectory data includes status, action, reward and security cost; and selects trajectory data from the fixed offline trajectory dataset whose cumulative security cost meets the preset security cost budget to form a feasible trajectory set.
[0060] The reference trajectory selection module is used to construct a constraint sequence dataset containing state sequences, historical action sequences, reward targets, and safety cost budgets based on the feasible trajectory set; at the same time, based on the cumulative reward, it selects trajectory data with rewards higher than a threshold from the feasible trajectory set to construct a reference trajectory set.
[0061] The model training module is used to train the decision model using a set of feasible trajectories and a set of reference trajectories. The decision model includes an inverse commentator and a constraint sequence policy. The training process includes the following steps: using the set of reference trajectories, the inverse commentator is trained through inverse reinforcement learning to jointly evaluate the task reward and safety of state-action pairs; using the constraint sequence dataset, the constraint sequence policy is subjected to supervised imitation learning to learn the actions of feasible trajectories in the constraint sequence dataset; the actions generated by the current policy are input into the trained inverse commentator to construct the inverse commentator guidance loss; based on the inverse commentator guidance loss, the constraint sequence policy is guided to optimize with the objective of maximizing the joint evaluation value; the inverse commentator and constraint sequence policy are alternately updated until the training termination condition is met, and a decision policy that satisfies the safety cost constraint is output.
[0062] The control module is used to input the current trajectory dataset of the target application object into the trained decision model, and output the action command at the current moment based on the constraint sequence strategy to control the target application object to complete the task.
[0063] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A trajectory decision control method based on inverse critic guide imitation, characterized in that, Includes the following steps: Obtain a fixed offline trajectory dataset of the target application object; each trajectory data includes status, action, reward, and security cost; Then, trajectory data whose cumulative safety cost meets the preset safety cost budget are selected from the fixed offline trajectory dataset to form a feasible trajectory set; Based on the feasible trajectory set, a constraint sequence dataset containing state sequence, action sequence, reward target, and safety cost budget is constructed; at the same time, based on the cumulative reward, trajectory data with rewards higher than the threshold are selected from the feasible trajectory set to construct a reference trajectory set. A decision model is trained using a set of feasible trajectories and a set of reference trajectories. The decision model includes an inverse commentator and a constraint sequence policy. The training process includes the following steps: The inverse commentator is trained using inverse reinforcement learning with the set of reference trajectories to jointly evaluate the task reward and safety of state-action pairs; the constraint sequence policy is trained using supervised imitation learning with the constraint sequence dataset to learn the actions of feasible trajectories in the constraint sequence dataset; the actions generated by the current policy are input into the trained inverse commentator to construct the inverse commentator guidance loss; based on the inverse commentator guidance loss, the constraint sequence policy is guided and optimized with the objective of maximizing the joint evaluation value; the inverse commentator and the constraint sequence policy are alternately updated until the training termination condition is met, and a decision policy that satisfies the safety cost constraint is output. The current trajectory dataset of the target application object is input into the trained decision model, and the action command at the current moment is output based on the constraint sequence strategy to control the target application object to complete the task.
2. The trajectory decision control method based on inverse critic guide imitation according to claim 1, wherein, The training process of the inverse commenter specifically includes the following steps: Given the current policy Define soft value function ,in Represents the entropy coefficient; Inverse commenter; 'a' represents action; 's' represents state; 'E' represents expectation; Based on the soft-value function, the inverse commenter is optimized using a training objective; the training objective is expressed as: ,in, Indicates the discount factor. This represents the initial state of the reference trajectory. This represents the set of reference trajectories; the training objective is to enable the inverse commentator to jointly evaluate the task benefits and safety of state-action pairs in the reference trajectories.
3. The trajectory decision control method based on inverse commentator-guided imitation according to claim 2, characterized in that, The training process of the inverse commentator also incorporates a gradient penalty term; this gradient penalty term is used to limit the local variation of the inverse commentator near offline data; the gradient penalty term is expressed as: ,in, Indicates the interpolated sample. This represents the target gradient norm.
4. The trajectory decision control method based on inverse commentator-guided imitation according to claim 1, characterized in that, The total loss function of the inverse commenter is the negative value of the training target plus the gradient penalty weight multiplied by the gradient penalty term.
5. The trajectory decision control method based on inverse commentator-guided imitation according to claim 1, characterized in that, The total training loss function of the constraint sequence policy includes supervised imitation loss and inverse commentator guidance loss, expressed as follows: ,in, This represents the supervised imitation loss of the sequence policy. Indicates the reverse commenter-guided loss. This indicates the guiding weight.
6. The trajectory decision control method based on inverse commentator-guided imitation according to claim 5, characterized in that, The supervised imitation loss is = ,in, The input sequence of constraints at time t represents the input condition. This represents the actions within the set of feasible trajectories. This represents a dataset containing a sequence of constraints.
7. The trajectory decision control method based on inverse commentator-guided imitation according to claim 6, characterized in that, The inverse commentator guided loss is the inverse commentator. Regarding the current state and strategy to generate actions The negative expected value of the evaluation value is expressed as = .
8. The trajectory decision control method based on inverse commentator-guided imitation according to claim 1, characterized in that, The set of feasible trajectories is determined based on a preset safety cost budget. Calculate the cumulative safety cost for each trajectory data point to filter the set of feasible trajectories that meet the safety constraints: ,in, Representing the trajectory The cumulative security costs.
9. A trajectory decision control system based on inverse commentator-guided imitation, characterized in that, include: The acquisition module is used to acquire a fixed offline trajectory dataset of the target application object; Each trajectory data includes status, action, reward, and safety cost; Then, trajectory data whose cumulative safety cost meets the preset safety cost budget are selected from the fixed offline trajectory dataset to form a feasible trajectory set; The reference trajectory selection module is used to construct a constraint sequence dataset containing state sequences, action sequences, reward targets, and safety cost budgets based on the feasible trajectory set; at the same time, based on the cumulative reward, it selects trajectory data with rewards higher than a threshold from the feasible trajectory set to construct a reference trajectory set. The model training module is used to train a decision model using a set of feasible trajectories and a set of reference trajectories. The decision model includes an inverse commentator and a constraint sequence policy. The training process includes the following steps: training the inverse commentator using the set of reference trajectories through inverse reinforcement learning, enabling it to jointly evaluate the task benefits and safety of state-action pairs; performing supervised imitation learning on the constraint sequence policy using the constraint sequence dataset, enabling it to learn the actions of feasible trajectories in the constraint sequence dataset; inputting the actions generated by the current policy into the trained inverse commentator to construct the inverse commentator guidance loss; based on the inverse commentator guidance loss, optimizing the constraint sequence policy with the objective of maximizing the joint evaluation value; and alternately updating the inverse commentator and the constraint sequence policy until the training termination condition is met, outputting a decision policy that satisfies the safety cost constraint. The control module is used to input the current trajectory dataset of the target application object into the trained decision model, and output the action command at the current moment based on the constraint sequence strategy to control the target application object to complete the task.