Feasible constraint strategy optimization method and device for intelligent agent control
The FCPO algorithm addresses safety and efficiency issues in reinforcement learning by integrating penalty functions and trust regions, ensuring stable and efficient intelligent agent control.
Patent Information
- Application Number
- CN202411180546.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-27
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-08-27
AI Technical Summary
Traditional reinforcement learning methods for intelligent agent control often violate safety constraints, leading to damage to the environment or the agent itself, and lack stability and efficiency in practical applications.
A feasible constraint strategy optimization method using the FCPO algorithm, combining penalty function and trust region methods, with ADMM decomposition, to ensure safe and efficient intelligent agent control.
The FCPO algorithm ensures stable convergence and wider applicability, improving safety, stability, and efficiency in intelligent agent control by optimizing constraint strategies.
Smart Images

Figure CN118707854B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a feasible constraint strategy optimization method and device for intelligent body control. Background Art
[0002] Reinforcement learning, as an important machine learning paradigm, has achieved remarkable achievements, especially in the field of intelligent agent control (such as robot control and self-driving cars).
[0003] The traditional reinforcement learning methods currently used for intelligent agent control mainly include primal-dual methods, trust region-based methods, and penalty function-based methods. However, the application of these methods is not only limited in scope and has certain limitations, but also in the process of exploring the environment, the behavior of the intelligent agent may violate certain safety constraints, thereby causing damage to the environment or the intelligent agent itself, and cannot meet the requirements of safety, stability, and efficiency. Summary of the invention
[0004] In view of this, the present invention provides a feasible constraint strategy optimization method and device for intelligent agent control, so as to solve the above-mentioned problems existing in the prior art.
[0005] The present invention provides a feasible constraint strategy optimization method for intelligent agent control, comprising the following steps.
[0006] Obtain information about constraints for controlling the movement of the agent;
[0007] Inputting the constraint information into the FCPO model to obtain control information for controlling the movement of the intelligent agent output by the FCPO model; wherein the FCPO model is used to determine the optimal constraint strategy according to the constraint information, and output the control information for controlling the movement of the intelligent agent based on the optimal constraint strategy; the FCPO model is obtained based on fixed random seed training;
[0008] Based on the control information, the intelligent agent is controlled to move.
[0009] According to a feasible constraint strategy optimization method for intelligent agent control provided by the present invention, the constraint condition information for mobile control of the intelligent agent includes:
[0010] Speed constraint: When the control agent reaches the maximum moving speed, the control agent's moving speed is less than or equal to the first threshold;
[0011] Circular constraint: when the intelligent agent is controlled to move in a preset circular area, the distance between the intelligent agent and the center of the circular area during the movement is greater than or equal to a second threshold;
[0012] Goal constraint: When controlling the agent to reach one or more target locations, the agent must avoid obstacles while moving to the target locations.
[0013] According to a feasible constraint strategy optimization method for intelligent agent control provided by the present invention, the determining of the optimal constraint strategy according to the constraint condition information comprises the following steps:
[0014] Determine the reward function and the cost function based on the constraint information for controlling the movement of the intelligent agent;
[0015] Based on the reward function and the cost function, establishing a first constraint strategy through a constraint strategy iteration algorithm;
[0016] The first constraint strategy is optimized by the PPO algorithm to establish a second constraint strategy; wherein the KL divergence between the second constraint strategy and the first constraint strategy is defined as a trust interval constraint;
[0017] The second constraint strategy is optimized by the ADMM algorithm to determine the optimal constraint strategy for controlling the movement of the intelligent agent while satisfying the constraint conditions for controlling the movement of the intelligent agent.
[0018] According to a feasible constraint strategy optimization method for intelligent agent control provided by the present invention, the reward function and the cost function are determined according to the constraint condition information for mobile control of the intelligent agent, including:
[0019] When the constraint information for controlling the movement of the agent is a speed constraint, the reward function is expressed as:
[0020]
[0021] in, Indicates at time Total rewards, represents the reward value when the agent is in a normal state at each time step, represents the reward value when the agent moves forward, Indicates the penalty value when the action performed by the agent is too large;
[0022] The cost function is expressed as:
[0023]
[0024] in, Indicates at time The total cost, represents the indicator function, represents the current speed of the agent, Represents the speed threshold of the agent.
[0025] According to a feasible constraint strategy optimization method for intelligent agent control provided by the present invention, the reward function and the cost function are determined according to the constraint condition information for mobile control of the intelligent agent, including:
[0026] When the constraint information for controlling the movement of the agent is a circular constraint, the reward function is expressed as:
[0027]
[0028] in, Indicates at time Total rewards, represents the Euclidean distance from the agent to the origin, represents the radius of a circular geometric figure, and Represents the agent along Axis and The velocity components of the axis, and Indicates that the agent is Axis and Coordinates on the axis;
[0029] The cost function is expressed as:
[0030]
[0031] in, Indicates at time The total cost, represents the indicator function, Express Take the absolute value, It is a safety boundary.
[0032] According to a feasible constraint strategy optimization method for intelligent agent control provided by the present invention, the reward function and the cost function are determined according to the constraint condition information for mobile control of the intelligent agent, including:
[0033] When the constraint information for controlling the movement of the agent is the target constraint, the reward function is expressed as:
[0034]
[0035] in, Indicates at time Total rewards, represents the distance between the agent and the target point at the previous time step, represents the distance between the agent and the target point at the current time step, is the discount factor, Indicates the reward value when the goal is completed. represents the indicator function, Indicates the threshold of judgment;
[0036] The cost function is expressed as:
[0037]
[0038] in, Indicates at time The total cost, represents the penalty value per unit time, represents the size of the agent, Represents the distance between the agent and the obstacle.
[0039] The present invention also provides a feasible constraint strategy optimization device for intelligent agent control, comprising the following modules:
[0040] An acquisition module is used to obtain constraint information for controlling the movement of the intelligent agent;
[0041] A processing module, used for inputting the constraint information into the FCPO model, and obtaining control information for controlling the movement of the intelligent agent output by the FCPO model; wherein the FCPO model is used for determining an optimal constraint strategy according to the constraint information, and outputting control information for controlling the movement of the intelligent agent based on the optimal constraint strategy; the FCPO model is obtained by training based on a fixed random seed;
[0042] A control module is used to control the movement of the intelligent agent based on the control information.
[0043] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements any of the above-described feasible constraint strategy optimization methods for intelligent agent control.
[0044] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described feasible constraint strategy optimization methods for intelligent agent control.
[0045] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-mentioned feasible constraint strategy optimization methods for intelligent agent control.
[0046] The present invention provides a feasible constraint strategy optimization method and device for intelligent agent control. Based on the constraint condition information for mobile control of the intelligent agent, the optimal constraint strategy for mobile control of the intelligent agent is output through the FCPO model, and then the mobile control of the intelligent agent is performed according to the optimal constraint strategy. Not only better strategy benefits are achieved while satisfying the constraint conditions, and the degree of default is also reduced, but the FCPO model also improves the computational efficiency of outputting the optimal constraint strategy and has a wider scope of application. The mobile control of the intelligent agent is performed based on the optimal constraint strategy, which improves the safety, stability and efficiency of the mobile control of the intelligent agent. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0048] Figure 1 It is a flow chart of the feasible constraint strategy optimization method for intelligent agent control provided by the present invention.
[0049] Figure 2 It is a schematic diagram of the safe speed intelligent agent environment provided by the present invention.
[0050] Figure 3 It is a schematic diagram of the safe navigation intelligent agent environment provided by the present invention.
[0051] Figure 4 It is a schematic diagram of the task environment provided by the present invention.
[0052] Figure 5 It is a training curve diagram provided by the present invention in controlling an intelligent agent to perform a safe speed task.
[0053] Figure 6 It is a training curve diagram provided by the present invention in controlling an intelligent agent to perform a safe navigation task.
[0054] Figure 7 It is a structural schematic diagram of a feasible constraint strategy optimization device for intelligent agent control provided by the present invention.
[0055] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0056] The present invention belongs to the field of artificial intelligence and machine learning technology. Specifically, the present invention relates to a feasible constraint strategy optimization algorithm for safe reinforcement learning. As an important machine learning paradigm, reinforcement learning has achieved remarkable achievements, such as showing excellent performance in various fields such as autonomous vehicles and robot control. However, due to the exploratory nature of traditional reinforcement learning methods, it is full of challenges to directly apply them to the real world. For example, in the process of exploring the environment, the behavior of the agent may violate certain safety constraints, thereby causing damage to the environment or the agent itself. Therefore, how to consider safety factors in the decision-making process has become one of the main bottlenecks restricting the practical application of reinforcement learning. A popular method to solve such problems is to use a constrained Markov decision process (CMDP) to model the problem, thereby converting safety considerations into operational policy set constraints, but the ability of the agent to freely explore the surrounding environment is limited.
[0057] In fact, many algorithms for solving CMDP have emerged in recent years. These algorithms can be roughly divided into three categories: the first category is based on the primal-dual method. Representative algorithms include the Lagrange function method, etc. This type of algorithm transforms the constrained optimization problem into an unconstrained problem by constructing a Lagrange function, and uses the classic reinforcement learning method to mine the optimal strategy. It has high computational efficiency and is simple, but it may encounter convergence oscillation and instability problems; the second category is based on trust region optimization. Representative algorithms include constrained policy optimization (CPO) and first-order policy space constrained optimization (FOCOPS). This type of algorithm draws on the idea of trust region policy optimization (TRPO) algorithm, has high computational efficiency and good stability, but it should be noted that the optimization problem constructed by this method can only be solved when there is an intersection between the trust region and the feasible region; the third type of method is the penalty function method, which directly transforms the constrained optimization problem into an unconstrained optimization problem, but the use of penalty functions may affect the quality and stability of problem solving, because it is still difficult to determine the size of the penalty coefficient and narrow the gap with the optimal solution. In addition, there are some other methods such as projection-based algorithms, but they all have certain defects to a greater or lesser extent and fail to solve the CMDP problem well.
[0058] Since safe reinforcement learning is crucial in safety-critical applications, it requires that the policy must satisfy clear constraints. However, existing primal-dual methods are instable, and trust region-based methods usually produce infeasible policies during training due to initialization and approximation errors. Current methods have inherent defects.
[0059] In order to solve the above problems, the present invention combines the penalty function and the trust region method and proposes a feasible constrained strategy optimization algorithm. The algorithm combines the penalty function and the trust region method. On the basis of ensuring the availability, convergence, stability and efficiency of the algorithm, it not only solves the feasibility problem of the strategy, but also has a wider scope of application, higher computational efficiency, and more stable convergence effect. It can effectively solve the constrained optimization problem in secure reinforcement learning.
[0060] Specifically, the present invention proposes a feasible constraint policy optimization method for intelligent agent control, which seamlessly integrates the penalty function method and the trust region method, and introduces the feasible constraint policy optimization (FCPO) to solve the feasibility limitation of the strategy; by adopting an accurate penalty function to expand the feasible domain range of the optimization problem, the inherent constraints of the initial feasible domain are eliminated; by using the objective function as the measure of the strategy, a policy iteration algorithm similar to TRPO is introduced to ensure monotonic convergence; through reasonable approximation, the FCPO algorithm is proposed, and the alternating direction multiplier method (ADMM) is used to decompose the strategy optimization problem into several simple sub-problems, so that efficient optimization can be achieved. In summary, the present invention mainly includes the following three points:
[0061] (1) Theoretical research: A sequential policy optimization method is proposed, which is characterized by being able to monotonically improve constraint violations outside the feasible domain while improving algorithm performance while satisfying the constraints;
[0062] (2) FCPO algorithm: The proposed FCPO successfully solves the constraints related to the feasibility of the initial strategy while ensuring the stable convergence of the trust region method. In order to reduce the optimization complexity, the present invention introduces the use of ADMM decomposition, decomposing the original problem into three easily solvable sub-problems, and adopts first-order approximate update, which greatly improves the optimization efficiency;
[0063] (3) Robust performance: The unique capabilities of the FCPO algorithm have been demonstrated through a series of experiments. The FCPO algorithm exhibits the characteristics of stable convergence and can provide robust performance in various situations, demonstrating the reliability and effectiveness of the FCPO algorithm in optimization tasks.
[0064] These advantages make the FCPO algorithm a novel and efficient solution to the problem of constrained optimization in reinforcement learning. It has important theoretical significance and broad application prospects. It has promoted the development of the cutting-edge direction of safe reinforcement learning and is expected to promote the implementation of reinforcement learning technology in more practical scenarios. It will have a profound impact on the development and application of artificial intelligence technology.
[0065] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0066] Combine the following Figures 1 to 8 The feasible constraint strategy optimization method and device for intelligent agent control of the present invention are described.
[0067] Figure 1 is a flow chart of a feasible constraint strategy optimization method for intelligent agent control provided by the present invention, such as Figure 1 As shown, the method includes the following steps: step 101, step 102 and step 103.
[0068] Step 101: Obtain constraint information for controlling the movement of an intelligent agent.
[0069] The constraint information for controlling the movement of the agent includes:
[0070] Speed constraint: When the control agent reaches the maximum moving speed, the control agent's moving speed is less than or equal to the first threshold;
[0071] Circular constraint: when the intelligent agent is controlled to move in a preset circular area, the distance between the intelligent agent and the center of the circular area during the movement is greater than or equal to a second threshold;
[0072] Goal constraint: When controlling the agent to reach one or more target locations, the agent must avoid obstacles while moving to the target locations.
[0073] Safety-gymnasium has developed a comprehensive Gym environment that provides a series of reinforcement learning tasks that focus on safety and reliability. The embodiment of the present invention designs three different experimental task scenarios to verify the FCPO algorithm. Among them, the speed constraint is the safe speed task in the embodiment of the present invention, the circle constraint is the Circle task in the safe navigation task in the embodiment of the present invention, and the goal constraint is the Goal task in the safe navigation task in the embodiment of the present invention. The following is a detailed introduction to the contents of these three tasks with constraints.
[0074] 1. Safe speed mission
[0075] Speed tasks are an important class of real-world applications where an agent must maximize its speed while respecting a speed constraint (i.e., the first threshold). Figure 2 is a schematic diagram of the safe speed agent environment, such as Figure 2 As shown. In the speed task, a safety limit is imposed on the speed of the agent in Mujoco. The embodiment of the present invention uses four agents, each with its own configuration. Figure 2 (a) is SafetyAnt (i.e., a safe quadrupedal agent), (b) is SafetyHalfCheetah (i.e., a safe bipedal agent), (c) is SafetyHumanid (i.e., a safe three-dimensional bipedal agent), and (d) is SafetySwimmer (i.e., a safe three-link swimming agent).
[0076] 2. Safe Navigation Mission
[0077] The safe navigation task includes the Circle task and the Goal task. Figure 3 is a schematic diagram of the safe navigation agent environment, such as Figure 3 As shown in the figure, four types of agents are shown: (a) is the point front agent (Point front), (b) is the point side agent (Point front), (c) is the vehicle front agent (Point front) and (d) is the vehicle side agent (Point front).
[0078] Figure 4 is a schematic diagram of the task environment, where Figure 4 (a) in the figure is a schematic diagram of the circular navigation task environment (Circle task), as shown in Figure 4 As shown in (a) in the Circle task, the goal of the agent is to navigate in a circular area while optimizing the speed and maintaining the distance from the center (i.e., the second threshold) to obtain a reward, but venturing into an unsafe area will be punished. The embodiment of the present invention uses four types of agents, namely, point front agent, point side agent, vehicle front agent, and vehicle side agent. The observation value is a vector that contains the information of the current state of the agent and the pseudo LIDAR point.
[0079] Figure 4 (b) in the figure is a schematic diagram of the target navigation task environment (Goal task), as shown in Figure 4 As shown in (b) in the Goal task, the goal of the agent is to reach a series of target locations. After reaching a target, the target location is randomly reset to a new location, but the environment layout remains unchanged. When the agent successfully reaches the target location (enters the target circle), it will receive a sparse reward; and moving towards the target will receive a dense reward. The observation value is the same as the Circle task environment.
[0080] Step 102: input the constraint information into the FCPO model to obtain control information for mobile control of the intelligent agent output by the FCPO model; wherein the FCPO model is used to determine the optimal constraint strategy according to the constraint information, and output the control information for mobile control of the intelligent agent based on the optimal constraint strategy; the FCPO model is obtained based on fixed random seed training.
[0081] Wherein, determining the optimal constraint strategy according to the constraint condition information includes the following steps: step 1, step 2, step 3 and step 4.
[0082] Step 1: Determine the reward function and cost function based on the constraint information for controlling the movement of the agent.
[0083] When the constraint information for controlling the movement of the agent is a speed constraint, the reward function is expressed as:
[0084]
[0085] in, Indicates at time Total rewards, represents the reward value when the agent is in a normal state at each time step, represents the reward value when the agent moves forward, Indicates the penalty value when the action performed by the agent is too large;
[0086] The cost function is expressed as:
[0087]
[0088] in, Indicates at time The total cost, represents the indicator function, represents the current speed of the agent, Represents the speed threshold of the agent.
[0089] According to the embodiment provided by the present invention, when the constraint information for controlling the movement of the agent is a speed constraint, that is, when the agent performs a safe speed task, the reward function It consists of three parts:
[0090] (1) : At each time step, if the agent is in a normal state, it will receive a fixed reward .
[0091] (2) : The reward for forward movement, calculated as . is the time step, After executing the action coordinate, Before executing the action coordinates. If the agent moves forward (positive direction), the reward is positive.
[0092] (3) : If the agent performs an action that is too large, it will receive a negative reward The calculation formula is The sum of squares, where is a control parameter, the default value is 0.5.
[0093] The total reward is , the cost function It can be expressed as:
[0094] ,in represents the current speed of the agent, is the speed threshold, which is set to the upper limit of 50% of the agent speed after 1 million steps of optimization using the PPO algorithm.
[0095] When the constraint information for controlling the movement of the agent is a circular constraint, the reward function is expressed as:
[0096]
[0097] in, Indicates at time Total rewards, represents the Euclidean distance from the agent to the origin, represents the radius of a circular geometric figure, and Represents the agent along Axis and The velocity components of the axis, and Indicates that the agent is Axis and Coordinates on the axis;
[0098] The cost function is expressed as:
[0099]
[0100] in, Indicates at time The total cost, represents the indicator function, Express Take the absolute value, It is a safety boundary.
[0101] According to the embodiment provided by the present invention, when the constraint information for controlling the movement of the agent is a circular constraint, that is, when the agent performs the Circle task in the safe navigation task, the reward function and the cost function is defined as follows:
[0102]
[0103]
[0104] in, and Represents the agent along Axis and The velocity components of the axis, and Indicates that the agent is Axis and Coordinates on the axis. represents the Euclidean distance from the agent to the origin, Corresponds to the radius of the circular geometry. represents the indicator function, It is a safety boundary.
[0105] When the constraint information for controlling the movement of the agent is the target constraint, the reward function is expressed as:
[0106]
[0107] in, Indicates at time Total rewards, represents the distance between the agent and the target point at the previous time step, represents the distance between the agent and the target point at the current time step, is the discount factor, Indicates the reward value when the goal is completed. represents the indicator function, Indicates the threshold of judgment;
[0108] The cost function is expressed as:
[0109]
[0110] in, Indicates at time The total cost, represents the penalty value per unit time, represents the size of the agent, Represents the distance between the agent and the obstacle.
[0111] According to the embodiment provided by the present invention, when the constraint information for controlling the movement of the agent is a target constraint, that is, when the agent performs the Goal task in the safe navigation task, the reward function and the cost function is defined as follows:
[0112]
[0113]
[0114] represents the distance between the agent and the target point at the previous time step, Represents the distance between the agent and the target point at the current time step. is the discount factor, Is a positive value representing the reward obtained when the goal is completed. is the judgment threshold, is the size of the agent, is the distance between the agent and the obstacle, is the penalty per unit time.
[0115] Step 2: Based on the reward function and the cost function, a first constraint strategy is established through a constraint strategy iteration algorithm.
[0116] According to the embodiment provided by the present invention, before establishing the first constraint strategy through the constraint strategy iteration algorithm, it is necessary to pre-define it. Specifically, a CMDP model is pre-defined as a tuple ,in, is the state space; It is the action space; It means to apply action Afterwards from the status arrive The probability of state transition; is the reward function; is a set of constraints (where It is A cost function, It is limits); is the discount factor; The initial state distribution of .
[0117] In CMDP, the strategy is defined in The probability distribution on Indicates in status Next select action probability; Represents the set of all possible strategies.
[0118] In addition, stable parameterized strategies is defined in The probability distribution on Indicates in status Next action probability; Represents the set of all parameter strategies, express dimensional space vector.
[0119] By executing the policy From the initial state start time steps later, the state The probability of , From the initial state start state after time steps.
[0120] set up From the initial state The stationary state distribution of the Markov chain at the beginning, Indicated in The value of the discount factor at the time step is defined as The initial distribution The discounted access distribution on Indicates the initial state Follow the initial distribution The expected function when .
[0121] Will Defined as The generated trajectory distribution is and , Indicates the start The action after time steps is From the initial state start state after time steps.
[0122] The state value function of the reward is expressed as , the state-action value function of the reward is expressed as , the advantage function of the reward is .
[0123] According to the embodiment provided by the present invention, similarly, the first The state value function of the cost is defined as ; The cost The state-action value function is defined as ; The cost The advantage function is defined as ; Set the first The sub-cost payback is defined as ; The expected discounted return is defined as ; Constraint Set The set of feasible strategies Defined as .
[0124] The constrained policy iteration algorithm is generally used to solve the CMDP problem. This process is expressed as an unbiased constrained policy iteration problem:
[0125] (1a)
[0126] (1b)
[0127] in, Indicated in In the iteration, the current strategy The expected discounted return under .
[0128] In early studies, the strategy was to use a fixed strategy for iterative updates to collect samples to construct local optimization problems, highlighting the effectiveness of incorporating local trust region constraints when constructing these problems, thereby improving efficiency and performance.
[0129] Therefore, in the In the iteration, the present invention expects to use the current strategy To formulate a local optimization problem, we can get the updated strategy On this basis, CPO proposed the upper and lower bounds of the objective function and constraint function, which are explained as follows:
[0130] (2a)
[0131] in, is the updated expected discounted return, is the expected discounted return, For current strategy The advantage function of the reward under , is Action the total variation dispersion between distributions;
[0132] (2b)
[0133] in, is the updated expected discount Cost return, The expected discount Cost return, For current strategy The cost of The advantage function of , is Action The total variation among the distributions.
[0134] The goal of secure reinforcement learning is to search for the optimal constraint strategy by gradually updating the optimization. At this time, the updated strategy is obtained based on the constraint policy iteration algorithm , and combined with the upper and lower bound theorems of the objective function and constraint function, the first constraint strategy is established.
[0135] Step 3: Optimize the first constraint strategy through the PPO algorithm to establish a second constraint strategy; wherein the KL divergence between the second constraint strategy and the first constraint strategy is defined as a trust interval constraint.
[0136] Specifically, optimizing the first constraint strategy is to ensure monotonic improvement of the constraint strategy. Unlike traditional reinforcement learning, secure reinforcement learning needs to comply with strategy constraints while optimizing the objective function, while traditional methods such as Trust Region Policy Optimization (TRPO) often find it difficult to ensure continuous monotonic improvement of the strategy.
[0137] In order to provide a theoretical guarantee for the FCPO algorithm proposed in the present invention, previous works usually assume that the initial solution satisfies the constraints, thereby ensuring that the objective function is continuously improved while satisfying the constraints.
[0138] In order to extend the applicability of the algorithm to the entire strategy space, the embodiment of the present invention introduces a new metric to measure the relative advantages and disadvantages of two strategies that may not strictly comply with the constraints. To solve this problem, the embodiment of the present invention introduces the exact penalty function method in problem (1), which is restated as:
[0139] (3a)
[0140] (3b)
[0141] in is a positive penalty parameter, is the exact penalty function, is the slack variable vector.
[0142] The fundamental difference between problem (1) and problem (3) lies in their feasible domains. The feasible domain of problem (1) is defined by constraint (1b), while problem (3) expands the feasible solution space to the entire strategy space. Since any strategy can be a feasible solution, the embodiment of the present invention uses objective function (3a) to measure the pros and cons of a strategy. Since the penalty coefficient is a finite value, the objective function of the embodiment of the present invention becomes a measure of the safe reinforcement learning strategy. However, it is often difficult to calculate the state distribution of the strategy in problem (3). In actual situations, it has become a common practice to approximate the distribution by sampling fixed strategies. For convenience, the embodiment of the present invention introduces the following function: , , as well as ,and , , as well as It is a symbol with no actual meaning.
[0143]
[0144] Based on the above-established definitions, the embodiment of the present invention sets the following optimization problem (4) and introduces a constrained strategy iteration algorithm 1 with a non-increasing guarantee for function (3a).
[0145] (4a)
[0146] (4b)
[0147] Under the condition that the current strategy meets the constraints, the penalty function ensures the feasibility of the strategy. Under this condition, the algorithm can monotonically improve the reward function of the strategy while ensuring that the output strategy remains within the specified constraints.
[0148] The penalty function is used to measure the degree of constraint violation. In practical applications, directly solving problem (4) may result in a very small step size for each policy iteration update. To solve this problem, the embodiment of the present invention adopts a first-order method, using the neighboring policy optimization PPO algorithm to optimize the first constraint strategy, thereby obtaining the second constraint strategy, and drawing inspiration from the trust region method in TRPO, taking the KL divergence between the new and old strategies (i.e., the second constraint strategy and the first constraint strategy) as a trust interval constraint, so that the algorithm can obtain a larger step size update capability. The optimization problem is expressed as follows:
[0149] (5a)
[0150] (5b)
[0151] (5c)
[0152] in: , represents the trust region boundary.
[0153] Assumptions is the optimal solution to problem (3), then No. The upper bound of the return is:
[0154] (6)
[0155] in: .
[0156] Due to the use of penalty functions and trust region methods, the FCPO algorithm of the embodiment of the present invention converges the slack variables to 0 as the number of iterations increases during the optimization process. In contrast to CPO and FOCOPS, the optimization problems they use have feasible solutions only when the Slater conditions are met. Therefore, these algorithms have limitations when optimizing strategies that do not meet the constraints. The optimization method of the embodiment of the present invention can effectively solve this problem.
[0157] Step 4: Optimize the second constraint strategy through the ADMM algorithm to determine the optimal constraint strategy for mobile control of the intelligent agent while satisfying the constraint conditions for mobile control of the intelligent agent.
[0158] Specifically, the essence of optimizing the second constraint strategy is to find the optimal update strategy and then obtain the optimal constraint strategy.
[0159] The exact penalty function method can maintain the optimality of the solution, but it introduces non-smoothness due to the existence of the penalty term. Directly applying the Lagrange dual ascent algorithm to solve this problem may affect the stability of the algorithm, especially when the Lagrange multiplier fluctuates violently during training.
[0160] The optimization problem (5) involves many variables and complex constraints, which brings great challenges to optimization. The ADMM algorithm is derived from the primal-dual augmented Lagrangian problem and provides a decomposition method that can simplify complex optimization problems into multiple sub-problems that are easy to solve.
[0161] The above second constraint strategy is optimized through the decomposition method in the ADMM algorithm, and the original optimization problem is decomposed into three simpler sub-problems.
[0162] The formal description of the primal-dual augmented Lagrangian problem is as follows:
[0163] (7)
[0164] (8)
[0165] in is the proximity factor, and is the Lagrange multiplier. For simplicity, the symbols are introduced in the embodiment of the present invention. .
[0166] Next, we apply the ADMM algorithm to efficiently solve the three reconstructed sub-problems, and then In the iteration, the following three steps are performed to obtain the optimal constraint strategy for mobile control of the intelligent agent while satisfying the constraint conditions for mobile control of the intelligent agent.
[0167] (1) By solving the following optimization problem, based on the previous estimate and Get the optimal constraint strategy, where , Indicates The first round of training Iterations:
[0168] (9a)
[0169] (2) Update :
[0170] (9b)
[0171] (3) Update :
[0172] (9c)
[0173] Given a previous estimate and , for problem (9a), the optimal strategy is Take the following form:
[0174] (10)
[0175] in is a constant normalization factor, ensuring Belongs to policy set , dual variable is the solution to the following convex optimization problem:
[0176] (11)
[0177] From the above results, we can find that solving the optimal constrained strategy problem (9a) is equivalent to solving an unconstrained problem, that is, maximizing the weighted average of reward and cost.
[0178] The constraint coefficient in the embodiment of the present invention is composed of the error accumulation term , error term and the difference term This indicates that the FCPO algorithm proposed in the present invention implicitly includes a PID controller for learning Lagrange multipliers, thereby enhancing the stability of learning.
[0179] When solving problem (9a), we allow Existing in strategy space In this case, the obtained strategy may not belong to the parameterized strategy space. , so evaluating or sampling To solve this problem, in the embodiment of the present invention, a parameterized strategy is used. Replacement strategy On the other hand, in order to improve computational efficiency and simplify complexity, a first-order method is adopted in the algorithm proposed in the embodiment of the present invention, and the PPO algorithm is used to optimize the strategy, and the line search method and advantage function clipping are combined to ensure compliance with the KL divergence constraint.
[0180] Step 103: Based on the control information, control the movement of the agent.
[0181] In this embodiment of the present invention, in order to evaluate the effectiveness of the feasible region relaxation method we proposed in improving the performance of the local strategy search algorithm, not only the control information output by the FCPO model for controlling the movement of the agent is used to conduct experiments on the movement control of the agent, but also CPO and FOCOPS are used as baseline algorithms for experiments.
[0182] In addition, to evaluate the impact of the ADMM algorithm on optimization stability, in the embodiment of the present invention, CPPOPID is introduced into the baseline. In addition, to explore the difference of directly using the penalty function to process the constraints, the embodiment of the present invention also includes the P3O algorithm as another baseline in the comparative analysis.
[0183] Figure 5 This is a training curve diagram for controlling the agent to perform a safe speed task, where the reward and cost are obtained from running 10 million steps. The black dotted line in the figure indicates that the target cost value is set to 25; Figure 6 It is a training curve diagram in controlling the agent to perform a safe navigation task. The black dotted line in the figure represents the target cost value set to 25. Among them, the X-axis represents the number of samples used, and the Y-axis represents the average total reward or average total cost in the past 100 rounds.
[0184] like Figure 5 and Figure 6 As shown in Figure 3, in most tasks, the FCPO model outperforms other baseline algorithms in terms of rewards while also complying with the cost constraints; in addition, FCPO exhibits smoother convergence on the training curve and has lower variance.
[0185] In the SafetySwimmer task, although the average reward performance of FCPO is slightly lower than that of CPPOPID, the training curve of CPPOPID shows great variance and severe dependence on initialization. Compared with related local policy search methods such as CPO and FOCOPS, FOCOPS converges unstablely, the cost fluctuates greatly in the early stage, and the variance is also higher in tasks such as Goal. On the contrary, CPO converges stably and performs well in satisfying constraints, but the reward performance is sometimes poor in some tasks, and second-order information is required, which has high computational overhead. Although both FCPO and P3O use exact penalty functions to handle constraint violations, P3O has always lagged behind other algorithms in reward performance, especially in tasks such as Goal.
[0186] Comprehensive evaluation shows that FCPO stands out due to its simple implementation, stable convergence, and excellent performance, but FCPO performs slowly in cost convergence for complex problems, especially in the Goal environment.
[0187] Based on any of the above embodiments, in order to evaluate the performance of the algorithm and analyze the generalization of the algorithm, the embodiment of the present invention first uses a fixed random seed to train the model, and then uses 10 different random seeds to perform 100 experiments for testing.
[0188] Table 1 is a performance evaluation table of the algorithms provided by the embodiments of the present invention. As shown in Table 1, the performance of each algorithm in these experiments is summarized in Table 1. It is worth noting that FCPO always shows the ability to meet the constraints in most environments, and achieves the best performance except for the Goal environment. It should be pointed out that adding Gaussian noise to the actions during training will cause the evaluation performance to differ from the training performance.
[0189] Table 1 Algorithm performance evaluation table
[0190]
[0191] Based on any of the above embodiments, in order to further evaluate the performance of the algorithm and analyze the sensitivity of the algorithm, in an embodiment of the present invention, a series of experimental tasks are carried out, and the results are recorded in Table 2. Table 2 is an algorithm sensitivity analysis table provided in an embodiment of the present invention. As shown in Table 2, in an embodiment of the present invention, the sensitivity of the hyperparameter σ is deeply analyzed in combination with Table 2.
[0192] Table 2 Algorithm sensitivity analysis table
[0193]
[0194] The penalty factor is very important in the FCPO algorithm. However, as can be seen from Table 2, the performance of the FCPO algorithm is significantly insensitive to the choice of this hyperparameter.
[0195] In summary, the feasible constraint strategy optimization method for intelligent agent control proposed in the present invention is based on the FCPO algorithm, which better solves the problem of Markov decision process with constraints in reinforcement learning. The FCPO algorithm innovatively combines the ideas of the exact penalty function method and the trust region method, breaking through the limitations of traditional algorithms. On the one hand, by introducing the exact penalty function, the feasible domain of the FCPO algorithm is expanded from the local area to the entire strategy space, and is no longer limited by the feasibility of the initial strategy; on the other hand, the FCPO algorithm designs an objective function, which is used as a metric for evaluating the quality of the strategy, thereby ensuring that the FCPO algorithm has monotonic convergence during the optimization process, that is, both the strategy performance and the degree of constraint violation can be monotonically converged and optimized; the FCPO algorithm also greatly improves the computational efficiency of the algorithm with the help of technical means such as the ADMM algorithm and the first-order approximate strategy update, and the implicitly included PID controller structure also gives the FCPO algorithm good convergence stability.
[0196] Based on the embodiments provided by the present invention, compared with the existing baseline algorithm, the FCPO algorithm not only achieves better strategy benefits while satisfying the constraints, but also has a smaller degree of default, and the entire training process is more controllable, converges more smoothly, and the algorithm has a wider scope of application. On this basis, the present invention proposes a feasible constraint strategy optimization method for intelligent agent control, which obtains control information for intelligent agent movement control through the FCPO algorithm, realizes safe speed control and safe navigation control during intelligent agent movement, and improves the safety, stability and efficiency of intelligent agent movement control.
[0197] The following is a description of the feasible constraint strategy optimization device for intelligent agent control provided by the present invention. The feasible constraint strategy optimization device for intelligent agent control described below and the feasible constraint strategy optimization method for intelligent agent control described above can be referenced to each other.
[0198] Based on any of the above embodiments, Figure 7 Schematic diagram of a feasible constraint strategy optimization device for intelligent agent control provided by an embodiment of the present invention, such as Figure 7 As shown, the feasible constraint strategy optimization device for intelligent agent control provided by the embodiment of the present invention includes an acquisition module 701, a processing module 702 and a control module 703, wherein:
[0199] The acquisition module 701 is used to obtain the constraint information for controlling the movement of the intelligent agent; the processing module 702 is used to input the constraint information into the FCPO model to obtain the control information for controlling the movement of the intelligent agent output by the FCPO model; wherein the FCPO model is used to determine the optimal constraint strategy according to the constraint information, and output the control information for controlling the movement of the intelligent agent based on the optimal constraint strategy; the FCPO model is obtained based on fixed random seed training; the control module 703 is used to control the movement of the intelligent agent based on the control information.
[0200] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the feasible constraint strategy optimization method for intelligent agent control, the method comprising:
[0201] Obtain information about constraints for controlling the movement of the agent;
[0202] Inputting the constraint information into the FCPO model to obtain control information for controlling the movement of the intelligent agent output by the FCPO model; wherein the FCPO model is used to determine the optimal constraint strategy according to the constraint information, and output the control information for controlling the movement of the intelligent agent based on the optimal constraint strategy; the FCPO model is obtained based on fixed random seed training;
[0203] Based on the control information, the intelligent agent is controlled to move.
[0204] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0205] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the feasible constraint strategy optimization method for intelligent agent control provided by the above methods, the method comprising:
[0206] Obtain information about constraints for controlling the movement of the agent;
[0207] Inputting the constraint information into the FCPO model to obtain control information for controlling the movement of the intelligent agent output by the FCPO model; wherein the FCPO model is used to determine the optimal constraint strategy according to the constraint information, and output the control information for controlling the movement of the intelligent agent based on the optimal constraint strategy; the FCPO model is obtained based on fixed random seed training;
[0208] Based on the control information, the intelligent agent is controlled to move.
[0209] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the feasible constraint strategy optimization method for agent control provided by the above methods, the method comprising:
[0210] Obtain information about constraints for controlling the movement of the agent;
[0211] Inputting the constraint information into the FCPO model to obtain control information for controlling the movement of the intelligent agent output by the FCPO model; wherein the FCPO model is used to determine the optimal constraint strategy according to the constraint information, and output the control information for controlling the movement of the intelligent agent based on the optimal constraint strategy; the FCPO model is obtained based on fixed random seed training;
[0212] Based on the control information, the intelligent agent is controlled to move.
[0213] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0214] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0215] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A feasible constraint strategy optimization method for intelligent agent control, characterized in that: include: Obtain information about constraints for controlling the movement of the agent; Inputting the constraint information into the FCPO model to obtain control information for controlling the movement of the intelligent agent output by the FCPO model; wherein the FCPO model is used to determine the optimal constraint strategy according to the constraint information, and output the control information for controlling the movement of the intelligent agent based on the optimal constraint strategy; the FCPO model is obtained based on fixed random seed training; Determining the optimal constraint strategy according to the constraint condition information comprises the following steps: Determine the reward function and the cost function based on the constraint information for controlling the movement of the intelligent agent; Based on the reward function and the cost function, establishing a first constraint strategy through a constraint strategy iteration algorithm; The first constraint strategy is optimized by the PPO algorithm to establish a second constraint strategy; wherein the KL divergence between the second constraint strategy and the first constraint strategy is defined as a trust interval constraint; By using the ADMM algorithm, the second constraint strategy is optimized to determine the optimal constraint strategy for controlling the movement of the intelligent agent while satisfying the constraint conditions for controlling the movement of the intelligent agent; The step of determining the reward function and the cost function according to the constraint information for controlling the movement of the agent includes: When the constraint information for controlling the movement of the agent is a speed constraint, the reward function is expressed as: ; in, Indicates at time Total rewards, represents the reward value when the agent is in a normal state at each time step, represents the reward value when the agent moves forward, Indicates the penalty value when the action performed by the agent is too large; The cost function is expressed as: ; in, Indicates at time The total cost, represents the indicator function, represents the current speed of the agent, represents the speed threshold of the agent; Based on the control information, the intelligent agent is controlled to move.
2. The feasible constraint strategy optimization method for agent control according to claim 1 is characterized in that: The constraint information for controlling the movement of the agent includes: Speed constraint: When the control agent reaches the maximum moving speed, the control agent's moving speed is less than or equal to the first threshold; Circular constraint: when the intelligent agent is controlled to move in a preset circular area, the distance between the intelligent agent and the center of the circular area during the movement is greater than or equal to a second threshold; Goal constraint: When controlling the agent to reach one or more target locations, the agent must avoid obstacles while moving to the target locations.
3. The feasible constraint strategy optimization method for agent control according to claim 1 is characterized in that: The step of determining the reward function and the cost function according to the constraint information for controlling the movement of the agent includes: When the constraint information for controlling the movement of the agent is a circular constraint, the reward function is expressed as: ; in, Indicates at time Total rewards, represents the Euclidean distance from the agent to the origin, represents the radius of a circular geometric figure, and Represents the agent along Axis and The velocity components of the axis, and Indicates that the agent is Axis and Coordinates on the axis; The cost function is expressed as: ; in, Indicates at time The total cost, represents the indicator function, Express Take the absolute value, It is a safety boundary.
4. The feasible constraint strategy optimization method for agent control according to claim 1 is characterized in that: The step of determining the reward function and the cost function according to the constraint information for controlling the movement of the agent includes: When the constraint information for controlling the movement of the agent is the target constraint, the reward function is expressed as: ; in, Indicates at time Total rewards, represents the distance between the agent and the target point at the previous time step, represents the distance between the agent and the target point at the current time step, is the discount factor, Indicates the reward value when the goal is completed. represents the indicator function, Indicates the threshold of judgment; The cost function is expressed as: ; in, Indicates at time The total cost, represents the penalty value per unit time, represents the size of the agent, Represents the distance between the agent and the obstacle.
5. A feasible constraint strategy optimization device for intelligent agent control, characterized in that: include: An acquisition module is used to obtain constraint information for controlling the movement of the intelligent agent; A processing module, used for inputting the constraint information into the FCPO model, and obtaining control information for controlling the movement of the intelligent agent output by the FCPO model; wherein the FCPO model is used for determining an optimal constraint strategy according to the constraint information, and outputting control information for controlling the movement of the intelligent agent based on the optimal constraint strategy; the FCPO model is obtained by training based on a fixed random seed; Determining the optimal constraint strategy according to the constraint condition information comprises the following steps: Determine the reward function and the cost function based on the constraint information for controlling the movement of the intelligent agent; Based on the reward function and the cost function, establishing a first constraint strategy through a constraint strategy iteration algorithm; The first constraint strategy is optimized by the PPO algorithm to establish a second constraint strategy; wherein the KL divergence between the second constraint strategy and the first constraint strategy is defined as a trust interval constraint; By using the ADMM algorithm, the second constraint strategy is optimized to determine the optimal constraint strategy for controlling the movement of the intelligent agent while satisfying the constraint conditions for controlling the movement of the intelligent agent; The step of determining the reward function and the cost function according to the constraint information for controlling the movement of the agent includes: When the constraint information for controlling the movement of the agent is a speed constraint, the reward function is expressed as: ; in, Indicates at time Total rewards, represents the reward value when the agent is in a normal state at each time step, represents the reward value when the agent moves forward, Indicates the penalty value when the action performed by the agent is too large; The cost function is expressed as: ; in, Indicates at time The total cost, represents the indicator function, represents the current speed of the agent, represents the speed threshold of the agent; A control module is used to control the movement of the intelligent agent based on the control information.
6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the feasible constraint strategy optimization method for agent control as described in any one of claims 1 to 4 is implemented.
7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the feasible constraint strategy optimization method for agent control as described in any one of claims 1 to 4 is implemented.
8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the feasible constraint strategy optimization method for agent control as described in any one of claims 1 to 4 is implemented.
