A flexible constraint multi-agent system motion planning method based on reinforcement learning
By using a reinforcement learning algorithm and a reheat mechanism based on the Actor-Critic framework, the motion planning problem of flexible constrained multi-agent systems in complex environments is solved, achieving smooth trajectories and efficient training, and improving the feasibility and stability of the planning.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-03
- Publication Date
- 2026-03-20
AI Technical Summary
Existing technologies struggle to achieve smooth and feasible motion planning for flexible constrained multi-agent systems in complex, continuous environments, and the training process involves large amounts of data and long durations, making it prone to getting trapped in local optima.
We employ a reinforcement learning algorithm based on the Actor-Critic framework, combining A* search and backfire mechanisms. We discretize the state space at the cost of Euclidean distance, set an appropriate reward function and optimize the reward mechanism, and statistically analyze the success rate to save network parameters and avoid local optima.
It achieves smooth motion trajectory planning in complex environments, reduces the amount of training data and time, improves the success rate and algorithm stability, and ensures the continuity and feasibility of planning.
Smart Images
Figure CN116300953B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the field of artificial intelligence, and particularly relates to a flexible constraint multi-agent system motion planning method based on reinforcement learning. BACKGROUND
[0002] A system composed of multiple mobile agents carrying power mechanisms and directly or indirectly constrained by flexible objects such as tethered or flexible net connected is a flexible constraint multi-agent constraint. The cooperative transportation system in which multiple rotor unmanned aerial vehicles jointly lift a load through a tether and the space net system composed of multiple satellites connected by a fiber net are both flexible constraint multi-agent systems. In addition, flexible constraint multi-agent systems have many application examples in underwater, ground and space environments.
[0003] The prior art "Q-learning Based Path Planning Method for UAVs Using Priority Shifting[C].2022International Conference on Unmanned Aircraft Systems(ICUAS),IEEE,2022:421-426." introduces a method of motion planning in a tabular discrete environment through reinforcement learning. The action set adopts eight squares around the current position of the system. Through flexible setting of the reward function, the method in the document shows effectiveness under different performance optimization indicators. However, in actual application, the over-discretization of the environment is difficult to adapt to complex environments full of irregular obstacles, and the discretized actions are difficult to meet the continuity requirements of motion planning.
[0004] The prior art "Research for UAV Path Planning Method Based on Guided SARSA Algorithm[C].2022IEEE 2nd International Conference on Software Engineering and Artificial Intelligence(SEAI),IEEE,2022:220-224." introduces a method of motion planning in continuous space through a discrete action set. However, due to the overly simple selection of the action set, the motion trajectory generated by the planning method in the document has many right-angle turns, which is difficult to meet the continuity requirements of motion planning.
[0005] The planning method in the prior art "Path Planning for Mobile Robot's Continuous Action Space Based on Deep Reinforcement Learning [C]. 2018 International Conference on Big Data and Artificial Intelligence (BDAI), IEEE, 2018: 42-46." can generate a trajectory with high smoothness in a continuous space by using a continuous action set. However, the planning method in the document requires a large amount of data. As the complexity of the scene increases, the method in the document is prone to converge to a local optimum. SUMMARY
[0006] The purpose of the present application is to provide a flexible constraint multi-agent system motion planning method based on reinforcement learning, to solve the problem of motion planning based on a continuous action set while ensuring the smoothness and feasibility of the planned trajectory in a complex continuous environment.
[0007] The present application adopts the following technical solution: a flexible constraint multi-agent system motion planning method based on reinforcement learning, comprising the following steps:
[0008] Step 1: discretize the twelve-dimensional state space of six-degree-of-freedom rigid body motion at a fixed scale, and select the midpoint of each dimension as the characteristic state of the discretized interval in the discretized state space, and take the target state required by the cross-section condition of the flexible constraint multi-agent system as the starting point, and take the characteristic points of each state interval in the discretized state space as the end point, and use the Euclidean distance between the discrete state points as the cost of A* search to calculate the cost of the target state to each state in the configuration space,
[0009] Step 2: set a reward function based on the cost of the target state to the characteristic state of each discrete interval in the configuration space,
[0010] Step 3: train the system based on the reinforcement learning algorithm of the Actor-Critic framework, and calculate the success rate of the flexible constraint multi-agent system without collision and reaching the end point within N rounds,
[0011] Step 4: when the success rate is greater than or equal to the upper threshold, and the flexible constraint multi-agent system does not collide, overturn, and is within the limit of the motion ability, calculate the proportion of the optimization term in the reward function,
[0012] Step 5: when the proportion of the optimization term is greater than or equal to 85%, save the Actor network parameters and Critic network parameters, and output the state trajectory of the round with the maximum reward.
[0013] Further, the reward function expression in step 2 is:
[0014] r t = e -m (r guide + r collide ) + (1 - e -m ) r execute ,
[0015] wherein r t is the reward obtained by the reward function expression at time t, m is a backfire coefficient, the initial value is 0, r guide is a guide reward, r collide is a collision penalty reward, and r execute is an optimization reward.
[0016] wherein the guide reward is a negative number corresponding to the action in the state table in the configuration space when the flexible constraint multi-agent system selects the action, and the collision penalty reward is a penalty value when the flexible constraint multi-agent system collides.
[0017] wherein the algorithm of the optimization reward is:
[0018]
[0019] wherein x(t) is the system state at time t, Δt is the decision period of the system, is the integral performance index in the HJB equation.
[0020] Further, wherein the formula of the time difference error in the Actor-Critic in step 3 is:
[0021] δ t = r t + γV(x t+1 ) - V(x t )
[0022] wherein δ t is the time difference error, r t is the reward obtained by the reward function expression at time t, V(x t ) is the state value function of the state x t at time t, V(x t+1 ) is the state value function of the state x t+1 at time t+1, and γ is a reward discount, usually taking a value between 0.85 and 0.99.
[0023] Further, wherein the formula of the advantage function in the Actor-Critic algorithm in step 3 is:
[0024]
[0025] In the formula, A t is a generalized advantage function, l is an accumulated intermediate variable, t is time, t f is the end of the round time, gamma is the reward discount, lambda is the generalized advantage function discount rate, usually 0.96, delta t+1 is the time difference error.
[0026] The beneficial effects of the present application are:
[0027] 1. The present application realizes motion planning of a flexible constraint multi-agent system in a complex environment based on a continuous action set by introducing a deep reinforcement learning algorithm with a tempering mechanism, and obtains smooth, achievable and outstanding performance motion trajectories.
[0028] 2. The present application reduces the amount of data used for training in a complex environment, shortens the training time, ensures the optimization effect of the related indicators, improves the success rate of training, and improves the stability of the algorithm.
[0029] 3. The setting of the reward function determines the function and result orientation of the reinforcement learning training strategy. In a complex environment, if necessary guidance is not added, the exploration of the system can only rely on limited probability sampling; this leads to the need for a large amount of data sampling in the training of a complex environment, greatly prolonging the training time; the task configuration space is discretized at a suitable scale; starting from the target state, A* search is performed on other states at the cost of Euclidean distance, and the cost of the target state to each state in the configuration space is calculated respectively.
[0030] 4. The present application sets a reward function update mechanism that changes adaptively with the training situation, selects a negative reward of a suitable size as a penalty when a collision occurs, and counts the success rate of N times. If the success rate reaches the upper threshold, the value is continuously increased and the relevant parameters are saved, so that the components of the reward function are more biased towards performance optimization and the influence of the guide is weakened.
[0031] 5. The present application introduces a tempering mechanism, which can return to the previous neural network parameter version and reward function design when trapped in a local optimum, without having to start training again. When the success rate of N times of training reaches the upper threshold, the reward function update mechanism can be used to continue training after saving the parameters. If the success rate is lower than the lower threshold, the neural network parameters and the reward function design are rolled back to the previous version for periodic training, without having to start completely from scratch. Otherwise, the training is continued for N times to observe the training effect. BRIEF DESCRIPTION OF DRAWINGS
[0032] Figure 1 is a flowchart of the present application;
[0033] Figure 2Flowchart of the present application;
[0034] Figure 3 Loss of the Actor network in the training process of the present application;
[0035] Figure 4 Loss of the Critic network in the training process of the present application;
[0036] Figure 5 Round total reward curve in the training process of the present application. DETAILED DESCRIPTION
[0037] The present application will be described in detail below in conjunction with the accompanying drawings and specific embodiments.
[0038] It should be understood that the structures, proportions, sizes, etc. shown in the drawings of the present specification are only used to cooperate with the content disclosed in the present specification, to be understood and read by those skilled in the art, and do not have technical significance, and any modification of the structure, change of the proportional relationship, or adjustment of the size, without affecting the effect and purpose that can be achieved by the present application, should still fall within the scope of the technical content disclosed by the present application.
[0039] The present application discloses a flexible constraint multi-agent system motion planning method based on reinforcement learning, as shown in Figure 1 and Figure 2 , which consists of the following steps:
[0040] Step 1: Discretize the twelve-dimensional state space of six-degree-of-freedom rigid body motion at a fixed scale, and select the midpoint of each dimension as the characteristic state of this discretized interval in the discretized state space, and take the target state required by the cross-section condition of the flexible constraint multi-agent system as the starting point, and take the characteristic points of each state interval in the discretized state space as the end point, and use the Euclidean distance between the discrete state points as the cost of A* search to calculate the cost of the target state to each state in the configuration space.
[0041] Step 2: Set the reward function based on the cost of the target state to the characteristic state of each discrete interval in the configuration space, and calculate the immediate reward after the flexible constraint multi-agent system selects an action through the reward function. The immediate reward is calculated through the guide reward, collision penalty reward, and optimization reward.
[0042] The expression of the reward function is:
[0043] r t =e -m (r guide +r collide )+(1-e -m )r execute ,
[0044] wherein m is a tempering coefficient, the initial value is 0, r t is the reward obtained by the reward function expression at t, r guide is the guidance reward, r collide is the collision penalty reward, r execute is the optimization reward.
[0045] wherein the guidance reward is a negative number corresponding to the action in the state table of the configuration space when the flexible constraint multi-agent system selects the action, and the collision penalty reward is a penalty value when the flexible constraint multi-agent system collides.
[0046] wherein the algorithm of the optimization reward is:
[0047]
[0048] wherein x(t) is the system state at t, Δt is the decision-making period of the system, is the integral performance index in the HJB equation.
[0049] Step 3: Train the system based on the reinforcement learning algorithm of the Actor-Critic framework, and count the success rate of the flexible constraint multi-agent system not colliding and reaching the end point within N rounds.
[0050] wherein the formula for calculating the time difference error in the Actor-Critic in step 3 is:
[0051] δ t =r t +γV(x t+1 )-V(x t )
[0052] In the formula, δ t is the time difference error, r t is the reward obtained by the reward function expression at t, V(x t ) is the state value function of the state x t at t, V(xt+1) is the state value function of the state xt+1 at t+1, and γ is the reward discount, usually taking a value between 0.85 and 0.99,
[0053] wherein the formula for the advantage function in the Actor-Critic algorithm in step 3 is:
[0054]
[0055] In the formula, A t is the generalized advantage function, l is an accumulated intermediate variable, t is the time, and t fis the end of episode time, gamma is the reward discount, lambda is the generalized advantage function discount rate, usually 0.96, delta t+1 is the timing difference error.
[0056] Step 4: When the success rate is greater than the upper threshold, and the flexible constraint multi-agent system does not have collision, overturning, and is within the motion ability limit, the proportion of the optimization term in the reward function is calculated.
[0057] In this step, the following three items are checked:
[0058] 1. Whether the motion planning result at this time meets the check feasibility requirement, that is, no collision, overturning and other safety hazards caused by events in the motion process.
[0059] 2. Whether the motion planning result meets the speed obstacle and position obstacle limit of the mobile robot.
[0060] 3. According to the loss function value curve of the Actor network and the Critic network, whether the neural network is converged.
[0061] If at least one of the above three requirements is not met, the annealing coefficient m is increased by 0.1, and step 3 is re-entered. The annealing coefficient m is reduced by 0.2 in the next round, if the reduced annealing coefficient is less than 0, it is re-assigned to the initial value 0, and step 3 is re-entered until the three requirements are met.
[0062] Step 5: When the proportion of the optimization term is greater than 85%, the Actor network parameters and the Critic network parameters are saved, and the state trajectory of the round with the maximum reward is output. Check whether the motion planning result at this time meets the requirements. If it has been met, save the Actor network parameters theta a , the Critic network parameters theta c , the annealing coefficient m, and output the final motion trajectory.
[0063] The Actor network loss in the training process of the application is as shown in Figure 3 , the Critic network loss in the training process is as shown in Figure 4 , and Figure 3 and Figure 4 It can be seen that the Critic network loss and the Actor network loss of the application tend to 0, which indicates that the algorithm converges, and the total reward curve of the round in the training process is as shown in Figure 5 , and Figure 5 It can be seen that the reward curve tends to increase, which indicates that the strategy improvement effect is good.
[0064] The above merely preferred embodiments of the present application and are not intended to limit the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1. A motion planning method for a flexible constrained multi-agent system based on reinforcement learning, characterized in that, It consists of the following steps: Step 1: Discretize the twelve-dimensional state space of the six-DOF rigid body motion at a fixed scale. Within the discretized state space, select the midpoint of each dimension as the characteristic state of this discretized interval. Starting from the target state required by the cross-sectional conditions of the flexible constrained multi-agent system, and ending at the characteristic points of each state interval within the discretized state space, the Euclidean distance between discrete state points is used as the cost. The search calculates the cost from the target state to each state in the configuration space. Step 2: Set the reward function based on the cost from the target state to the characteristic states of each discrete interval within the configuration space. Step 3: Train the system using a reinforcement learning algorithm based on the Actor-Critic framework, and calculate the success rate of the flexible constrained multi-agent system reaching the endpoint without collision within N rounds. Step 4: When the success rate is greater than or equal to the upper threshold, and the flexible constrained multi-agent system does not experience collisions or overturning and remains within its motion capability limits, calculate the proportion of optimization terms in the reward function. Step 5: When the proportion of optimization items is ≥ At that time, save the Actor network parameters and Critic network parameters, and output the state trajectory of the round with the maximum reward; The reward function expression in step 2 is: ， in, yes The reward obtained from the time-reward function expression. This is the tempering coefficient, initially set to 0. To guide rewards, As a collision penalty reward, To optimize rewards, The guidance reward is the negative number in the configuration space state table corresponding to the action selected by the flexible constrained multi-agent system, and the collision penalty reward is the penalty value when a collision occurs in the flexible constrained multi-agent system. The algorithm for optimizing the reward is as follows: ， in, yes System status at all times It is the system's decision-making cycle. It is the integral performance index in the HJB equation.
2. The motion planning method for a flexible constrained multi-agent system based on reinforcement learning according to claim 1, characterized in that, in, The formula for the timing difference error in Actor-Critic in step 3 is: ； In the formula, For timing difference error, yes The reward obtained from the time-reward function expression. for Moment State The state value function, V(x) t+1 )for +1 time state x t+1 The state value function, This is a reward discount, typically ranging from 0.85 to 0.
99.
3. A motion planning method for a flexible constrained multi-agent system based on reinforcement learning according to any one of claims 1-2, characterized in that, in, The formula for the advantage function in the Actor-Critic algorithm in step 3 is: ， In the formula, For the generalized dominance function, To accumulate intermediate variables, For time, t f The end time of the round. It's a reward discount. The discount rate for the generalized advantage function is usually taken as... δ t+1 This refers to timing difference error.