AGV obstacle avoidance adaptive control system and method based on reinforcement learning

Abstract

The invention discloses an AGV obstacle avoidance adaptive control system and method based on reinforcement learning, and belongs to the technical field of AGV obstacle avoidance adaptive control, the system comprises an infrastructure module, a reinforcement learning module and a simulation experiment module, the infrastructure module is electrically connected with the reinforcement learning module, and the simulation experiment module is electrically connected with the reinforcement learning module. The reinforcement learning module is electrically connected with the simulation experiment module. By adopting the method and the system, the safety and the accuracy of a transportation task can be ensured, the adaptability of the AGV obstacle avoidance self-adaptive control system to the environment is improved, the adaptability to the environment and the integration of the system are improved through the arrangement of the simulation experiment module, algorithm optimization, simulation and experiment verification and field testing, and the system is suitable for popularization and application. Therefore, safer, more efficient and more intelligent operation is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of AGV obstacle avoidance adaptive control, and in particular to an AGV obstacle avoidance adaptive control system and method based on reinforcement learning. Background Art

[0002] The advantages of the AGV obstacle avoidance adaptive control system mainly include the following aspects: the AGV obstacle avoidance adaptive control system can perceive the surrounding environment in real time through built-in sensors and safety devices to avoid collisions with obstacles or personnel, significantly improving the safety of operations, especially in complex working environments, where AGVs can accurately avoid obstacles and reduce potential accident risks; the AGV obstacle avoidance adaptive control system can adjust the driving path in real time according to changes in the working environment and adapt to different working scenarios. This flexibility enables AGVs to operate efficiently in various complex environments without human intervention; because AGVs can accurately control the driving speed and direction, rough operations that may occur in manual operations are avoided, thereby reducing the risk of cargo damage; the AGV obstacle avoidance adaptive control system can operate continuously and stably, reducing The downtime caused by human factors has improved production efficiency; the AGV obstacle avoidance adaptive control system reduces vehicle collisions and maintenance needs and reduces maintenance costs through anti-collision equipment and intelligent control; the AGV obstacle avoidance adaptive control system can optimize logistics management, make the goods more orderly, and the workshop more tidy, thereby improving the efficiency and quality of logistics management; the AGV obstacle avoidance adaptive control system can flexibly plan paths and schedule tasks according to changes in production lines and changes in demand, and has strong adaptability; the AGV connected to the central control system can monitor the working status in real time and collect data, providing strong support for optimizing the production process and decision-making. In summary, the AGV obstacle avoidance adaptive control system has significant advantages in safety, flexibility, work efficiency, maintenance costs, logistics management, etc., and is suitable for various complex working environments.

[0003] However, the traditional AGV obstacle avoidance adaptive control system has the following disadvantages:

[0004] The error and noise models of each sensor in the traditional AGV obstacle avoidance adaptive control system are different. For example, the ranging accuracy and azimuth accuracy of ultrasonic sensors in detecting obstacles are much lower than those of lidar. If only sensors are used to sense obstacles, unreasonable sensor combinations will not only increase the cost of use, but may also cause mutual interference between sensors. Summary of the invention

[0005] The purpose of the present invention is to provide an AGV obstacle avoidance adaptive control system and method based on reinforcement learning to solve the problems mentioned in the background technology.

[0006] To achieve the above-mentioned objectives, the present invention provides an AGV obstacle avoidance adaptive control system based on reinforcement learning, including an infrastructure module, a reinforcement learning module and a simulation experiment module, wherein the infrastructure module is electrically connected to the reinforcement learning module, and the reinforcement learning module is electrically connected to the simulation experiment module.

[0007] Preferably, the infrastructure module includes an on-board controller, a navigation unit, a sensor unit, a communication unit, a safety protector, a power supply unit and a motion control unit.

[0008] Preferably, the reinforcement learning module includes a multi-objective optimization unit, a transfer learning unit and a deep learning reinforcement unit, the multi-objective optimization unit is electrically connected to the deep learning reinforcement unit, and the deep learning reinforcement unit is electrically connected to the transfer learning unit.

[0009] Preferably, the simulation experiment module includes a simulation environment construction unit, a simulation experiment design unit and an experiment data analysis unit, the simulation environment construction unit is electrically connected to the simulation experiment design unit, and the simulation experiment design unit is electrically connected to the experiment data analysis unit.

[0010] An AGV obstacle avoidance adaptive control method based on reinforcement learning, the steps comprising:

[0011] S1, virtual modeling, the simulation experiment module builds AGV dynamic and static obstacle models and performs simulation environment modeling;

[0012] S2, input parameters, in the modeled environment, set the environment state input of the reinforcement learning algorithm through the reinforcement learning module;

[0013] S3. Set up a two-layer decision model. According to the input parameters, the simulation experiment module uses the DDPG algorithm as the main decision layer and the DQN algorithm as the secondary decision layer to output actions.

[0014] S4. Setting a reinforcement learning reward value function. The reinforcement learning module initializes the two-layer decision model according to the set reinforcement learning reward value function, and continuously trains the AGV obstacle avoidance capability under the two-layer decision model.

[0015] S5. Use the robot action parameters to control the AGV navigation and obstacle avoidance operation. The infrastructure module inputs the target point location information, obstacle map information and surrounding environment information into the pre-trained reinforcement learning module, obtains the robot action parameters output by the reinforcement learning module, adjusts the movement speed and steering angle parameters according to the calculation results, and uses these parameters to control the AGV's navigation and obstacle avoidance operation at the current moment.

[0016] Preferably, in step S1, an affine nonlinear model is used to construct the AGV dynamic and static obstacle models, and the affine nonlinear model expression is:

[0017]

[0018] Where x∈R n Represents the control system state; f(x):R n →R n represents internal dynamic information, g(x):R n →R nxq represents the control coupling matrix, u(t)∈R q Indicates control input.

[0019] Preferably, the Euclidean distance is used in the AGV dynamic and static obstacle models to represent the distance between the obstacle and the agent, and the Euclidean distance expression is:

[0020]

[0021] Where i represents the i-th obstacle, represents the obstacle coordinates, and x represents the agent coordinates.

[0022] Preferably, the cost function of the agent in the simulation environment modeling is expressed as:

[0023]

[0024] r(x,u)=x T Qx+u T Ru+B(x);

[0025] In the formula, x T Qx,Q=Q T >0 indicates system operation cost, u T Ru,R=R T 〉0 represents the control cost, B(x) represents the barrier function, J(x,u) represents the predicted value, r(x,u) represents the actual value, and γ represents the input parameter.

[0026] Preferably, the reinforcement learning algorithm in step S2 is a Q-learning algorithm, which is expressed as:

[0027] Q(s,a)=(1-α)Q(s,a)+α(r+γmax a' (Q(s',a')));

[0028] Where Q(s,a) represents the value function of executing action a in state s, α represents the learning rate (0<α≤1), r represents the immediate reward after executing action a, γ represents the discount factor (0≤γ≤1), s' represents the new state after executing action a, and a' represents the action selected in the new state s'.

[0029] Preferably, in step S3, the DDPG algorithm is updated using the classic TD algorithm, and target_network is introduced to estimate TD_target, and the expression is:

[0030] Δθq(sj,μ(sj;θ);w)=Δθμ(sj,θ)Δaq(sj,a∧j;w)θ←θ+β·θ;

[0031] Where S represents the state parameter, θ represents the policy network parameter, w represents the value network parameter, μ(sj,θ) represents the action parameter, and q(sj,a^j;w) represents the value parameter.

[0032] Therefore, the present invention adopts the above-mentioned AGV obstacle avoidance adaptive control system and method based on reinforcement learning, which has the following beneficial effects:

[0033] (1) By setting up the infrastructure module, the sensor transmits the required data in real time. Through the data feedback from the sensor, the reinforcement learning module adjusts the movement speed and steering angle parameters according to the calculation results to ensure the safety and accuracy of the transportation task and improve the adaptability of the AGV obstacle avoidance adaptive control system to the environment;

[0034] (2) By setting up simulation experiment modules, through algorithm optimization, simulation and experimental verification, and field testing, its adaptability to the environment and system integration can be improved to achieve safer, more efficient, and intelligent operations.

[0035] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of the control system structure of an embodiment of the present invention;

[0037] Figure 2 A schematic diagram of the architecture of an infrastructure module according to an embodiment of the present invention;

[0038] Figure 3 Schematic diagram of the architecture of a reinforcement learning module according to an embodiment of the present invention;

[0039] Figure 4 A schematic diagram of the architecture of a simulation experiment module according to an embodiment of the present invention;

[0040] Figure 5 It is a front view of the control system assembly of an embodiment of the present invention;

[0041] Figure 6 A rear view of the control system of an embodiment of the present invention after assembly;

[0042] Figure 7 is a flow chart of a method according to an embodiment of the present invention;

[0043] Reference numerals

[0044] 1. Control rack; 2. Mounting plate; 3. Sensor unit. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0046] Example

[0047] like Figure 1-6 As shown, the present invention provides an AGV obstacle avoidance adaptive control system based on reinforcement learning. The control system is installed on a control frame 1, and a mounting plate 2 is fixedly installed on one side of the control frame 1. The control system includes an infrastructure module, a reinforcement learning module and a simulation experiment module. The infrastructure module is electrically connected to the reinforcement learning module, and the reinforcement learning module is electrically connected to the simulation experiment module.

[0048] The infrastructure module includes an on-board controller, a navigation unit, a sensor unit 3, a communication unit, a safety protector, a power supply unit and a motion control unit.

[0049] Specifically, one side of the vehicle controller, one side of the navigation unit, one side of the sensor unit 3, one side of the communication unit, one side of the safety protector, one side of the power supply unit and one side of the motion control unit are all fixedly connected to the mounting plate 2. The vehicle controller is responsible for receiving and processing information from the navigation unit, the sensor unit 3 and other input devices, and making corresponding control decisions. The vehicle controller usually uses a central control unit as the core, and has signal input, processing and output, real-time control, fault diagnosis and fault handling functions. The navigation unit provides the position and heading information of the AGV to help the AGV find the best path in a complex environment. The navigation unit is based on a variety of sensors such as laser radar, ultrasonic sensor, infrared sensor, etc., and learns through a reinforcement learning model to obtain the robot action parameters output by the model, adjust the movement speed and steering angle parameters according to the calculation results, and use these parameters to control the navigation and obstacle avoidance operation of the AGV at the current moment. The sensor unit 3 includes a laser radar, an ultrasonic sensor, and an infrared sensor, which are used to detect obstacles and environmental changes around the AGV. These sensors transmit real-time data to the vehicle controller to help the AGV make obstacle avoidance decisions. The communication unit usually uses wireless communication technology, such as Wi-Fi and Bluetooth, to ensure the real-time transmission of information and the coordinated work of the control system. The safety protector includes protection for the AGV itself and protection measures for people or other equipment. The safety protection device ensures that the AGV will not cause damage to the surrounding environment during the obstacle avoidance process, and ensures the safety of personnel and equipment. The power supply unit provides a stable power supply for the AGV to ensure that the normal operation of the system will not be affected by power problems during the obstacle avoidance process. The power supply unit usually includes a battery and a charging device to ensure that the AGV can be safely docked in the event of a power outage. The motion control unit is responsible for controlling the walking, steering and braking actions of the AGV. The motion control unit adjusts the speed, direction and braking of the AGV according to the instructions of the on-board controller to achieve obstacle avoidance and path planning.

[0050] The reinforcement learning module includes a multi-objective optimization unit, a transfer learning unit and a deep learning reinforcement unit. The multi-objective optimization unit is electrically connected to the deep learning reinforcement unit, and the deep learning reinforcement unit is electrically connected to the transfer learning unit.

[0051] Specifically, the multi-objective optimization unit receives data transmitted by the sensor unit 3, optimizes the multiple sensor data of the sensor unit 3, and helps the AGV find the best trade-off between multiple sensors. For example, the Pareto optimization method is used to simultaneously optimize multiple objectives by defining multiple reward functions. The deep learning reinforcement unit receives the data optimized by the multi-objective optimization unit, processes high-dimensional input data through a deep neural network, and learns complex strategies. In the AGV autonomous obstacle avoidance path planning, deep reinforcement learning processes complex environmental information to achieve obstacle avoidance, for example, using a deep Q network (DQN) or a deep deterministic policy gradient (DDPG) algorithm. The transfer learning unit learns the obstacle avoidance strategy that has been trained by the deep reinforcement learning unit, helping the AGV to quickly adapt to new environments or tasks and reduce learning time.

[0052] The simulation experiment module includes a simulation environment construction unit, a simulation experiment design unit and an experiment data analysis unit. The simulation environment construction unit is electrically connected to the simulation experiment design unit, and the simulation experiment design unit is electrically connected to the experiment data analysis unit.

[0053] Specifically, the simulation environment construction unit uses simulation platforms such as Gazebo and Unity to build a simulation environment, and simulates the behavior of AGV and the dynamic changes of the environment. The simulation experiment design unit designs different scenarios, such as different obstacle distributions, to evaluate the performance of the algorithm under different conditions. The experimental data analysis unit collects experimental data, such as obstacle avoidance success rate, path length, and energy consumption indicators, quantifies the performance of the algorithm, and uses statistical analysis methods, such as variance analysis and correlation analysis, to analyze the data and draw conclusions.

[0054] The present invention also provides an AGV obstacle avoidance adaptive control method based on reinforcement learning, such as Figure 7 As shown, the steps include:

[0055] S1. Virtual modeling, the simulation experiment module builds AGV dynamic and static obstacle models and performs simulation environment modeling.

[0056] Specifically, the affine nonlinear model is used to construct the AGV dynamic and static obstacle models. The expression of the affine nonlinear model is:

[0057]

[0058] Where x∈R n Represents the control system state; f(x):R n →R n represents internal dynamic information, g(x):R n →R nxq represents the control coupling matrix, u(t)∈R q Indicates control input.

[0059] For the convenience of calculation, obstacles are considered as static objects, and the number of obstacles is recorded as 0. In addition, overlapping obstacles are not considered. Once they overlap, they are regarded as a larger obstacle. The maximum radius is used as the obstacle avoidance condition, and the obstacle is regarded as a circular area or a spherical area, and its position center is recorded as Therefore, the distance between the i-th obstacle and the agent is expressed as the Euclidean distance:

[0060]

[0061] Where i represents the i-th obstacle, represents the obstacle coordinates, and x represents the agent coordinates.

[0062] The cost function of the agent is expressed as:

[0063]

[0064] r(x,u)=x T Qx+u T Ru+B(x);

[0065] In the formula, x T Qx,Q=Q T >0 indicates system operation cost, u T Ru,R=R T >0 represents the control cost, B(x) represents the barrier function, J(x,u) represents the predicted value, r(x,u) represents the actual value, and γ represents the input parameter.

[0066] When building a simulation environment, use simulation platforms such as Gazebo and Unity to build a simulation environment, simulate the behavior of AGV and the dynamic changes of the environment, and design different scenarios, such as different obstacle distributions.

[0067] S2. Input parameters. In the modeled environment, the environment state input of the reinforcement learning algorithm is set through the reinforcement learning module.

[0068] Specifically, the reinforcement learning algorithm uses the Q-learning algorithm, which is expressed as:

[0069] Q(s,a)=(1-α)Q(s,a)+α(r+γmax a' (Q(s',a')));

[0070] Where Q(s,a) represents the value function of executing action a in state s, α represents the learning rate (0<α≤1), r represents the immediate reward after executing action a, γ represents the discount factor (0≤γ≤1), s' represents the new state after executing action a, and a' represents the action selected in the new state s'.

[0071] S3. Set up a two-layer decision model. According to the input parameters, the simulation experiment module uses the DDPG algorithm as the main decision layer and the DQN algorithm as the secondary decision layer to output actions.

[0072] Specifically, the DDPG algorithm uses the classic TD algorithm for updating and introduces target_network to estimate TD_target. The expression is:

[0073] Δθq(sj,μ(sj;θ);w)=Δθμ(sj,θ)Δaq(sj,a∧j;w)θ←θ+β·θ;

[0074] Where S represents the state parameter, θ represents the policy network parameter, w represents the value network parameter, μ(sj,θ) represents the action parameter, and q(sj,a^j;w) represents the value parameter.

[0075] S4. Setting a reinforcement learning reward value function. The reinforcement learning module initializes the two-layer decision model according to the set reinforcement learning reward value function, and continuously trains the AGV obstacle avoidance capability under the two-layer decision model.

[0076] S5. Use the robot action parameters to control the AGV navigation and obstacle avoidance operation. The infrastructure module inputs the target point location information, obstacle map information and surrounding environment information into the pre-trained reinforcement learning module, obtains the robot action parameters output by the reinforcement learning module, adjusts the movement speed and steering angle parameters according to the calculation results, and uses these parameters to control the AGV's navigation and obstacle avoidance operation at the current moment.

[0077] In order to verify the effectiveness of the method of the present invention, three groups of simulation experiments were conducted to prove that the intelligent agent can autonomously move to the target point and avoid multiple obstacles.

[0078] The simulation environment is based on MATLAB2020a, and the simulation step is 0.01s. The first group of simulation experiments is carried out on a simplified two-dimensional rotor UAV system. At this time, the UAV is considered to be in level flight and does not involve altitude changes. This group of simulation experiments considers three situations, including method comparison, reward parameter changes, and control constraints; the second group of simulation experiments is carried out on a nonlinear numerical system to test the performance of the proposed method for complex nonlinearity; the third group of simulation experiments is carried out on a simplified three-dimensional rotor UAV system, and the obstacle corresponds to a three-dimensional sphere. The control system provides obstacle avoidance protection for unmanned aircraft on different flight routes. The collision rate between the first group of control systems and obstacles is 5.24%, the collision rate between the second group of control systems and obstacles is 8.77%, and the collision rate between the third group of control systems and obstacles is 10.31%. It is installed on it to provide flight obstacle avoidance protection for the UAV during flight. Similarly, the control system is installed on the unmanned ship to facilitate the unmanned driving of the ship on the sea; the control system is installed on the unmanned vehicle to facilitate the automatic driving of the vehicle on the road.

[0079] Therefore, the present invention adopts the above-mentioned AGV obstacle avoidance adaptive control system and method based on reinforcement learning to ensure the safety and accuracy of transportation tasks, and improves the adaptability of the AGV obstacle avoidance adaptive control system to the environment. By setting up a simulation experiment module, through algorithm optimization, simulation and experimental verification and field testing, its adaptability to the environment and the integration of the system are improved to achieve safer, more efficient and intelligent operations.

[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. An AGV obstacle avoidance adaptive control system based on reinforcement learning, characterized by: It includes an infrastructure module, a reinforcement learning module and a simulation experiment module. The infrastructure module is electrically connected to the reinforcement learning module, and the reinforcement learning module is electrically connected to the simulation experiment module.

2. The AGV obstacle avoidance adaptive control system based on reinforcement learning according to claim 1, characterized in that: The infrastructure module includes an on-board controller, a navigation unit, a sensor unit, a communication unit, a safety protector, a power supply unit and a motion control unit.

3. The AGV obstacle avoidance adaptive control system based on reinforcement learning according to claim 1, characterized in that: The reinforcement learning module includes a multi-objective optimization unit, a transfer learning unit and a deep learning reinforcement unit. The multi-objective optimization unit is electrically connected to the deep learning reinforcement unit, and the deep learning reinforcement unit is electrically connected to the transfer learning unit.

4. The AGV obstacle avoidance adaptive control system based on reinforcement learning according to claim 1, characterized in that: The simulation experiment module includes a simulation environment construction unit, a simulation experiment design unit and an experiment data analysis unit. The simulation environment construction unit is electrically connected to the simulation experiment design unit, and the simulation experiment design unit is electrically connected to the experiment data analysis unit.

5. An AGV obstacle avoidance adaptive control method based on reinforcement learning, using an AGV obstacle avoidance adaptive control system based on reinforcement learning as described in any one of claims 1 to 4, characterized in that the steps include: S1, virtual modeling, the simulation experiment module builds AGV dynamic and static obstacle models and performs simulation environment modeling; S2, input parameters, in the modeled environment, set the environment state input of the reinforcement learning algorithm through the reinforcement learning module; S3. Set up a two-layer decision model. According to the input parameters, the simulation experiment module uses the DDPG algorithm as the main decision layer and the DQN algorithm as the secondary decision layer to output actions. S4. Setting a reinforcement learning reward value function. The reinforcement learning module initializes the two-layer decision model according to the set reinforcement learning reward value function, and continuously trains the AGV obstacle avoidance capability under the two-layer decision model. S5. Use the robot action parameters to control the AGV navigation and obstacle avoidance operation. The infrastructure module inputs the target point location information, obstacle map information and surrounding environment information into the pre-trained reinforcement learning module, obtains the robot action parameters output by the reinforcement learning module, adjusts the movement speed and steering angle parameters according to the calculation results, and uses these parameters to control the AGV's navigation and obstacle avoidance operation at the current moment.

6. The AGV obstacle avoidance adaptive control method based on reinforcement learning according to claim 5, characterized in that: In step S1, an affine nonlinear model is used to construct the AGV dynamic and static obstacle models. The expression of the affine nonlinear model is: x(t)=f(x(t))+g(x(t))u(t); Where x∈R n Represents the control system state; f(x):R n →R n represents internal dynamic information, g(x):R n →R nxq represents the control coupling matrix, u(t)∈R q Indicates control input.

7. The AGV obstacle avoidance adaptive control method based on reinforcement learning according to claim 6 is characterized in that: The Euclidean distance is used in the AGV dynamic and static obstacle models to represent the distance between the obstacle and the intelligent agent.

8. The AGV obstacle avoidance adaptive control method based on reinforcement learning according to claim 7, characterized in that: The cost function of the agent in the simulation environment modeling is expressed as: r(x,u)=x T Qx+u T Ru+B(x); In the formula, x T Qx,Q=Q T >0 indicates system operation cost, u T Ru,R=R T >0 represents the control cost, B(x) represents the barrier function, J(x,u) represents the predicted value, r(x,u) represents the actual value, and γ represents the input parameter.

9. The AGV obstacle avoidance adaptive control method based on reinforcement learning according to claim 8, characterized in that: The reinforcement learning algorithm in step S2 is the Q-learning algorithm, and its expression is: Q(s,a)=(1-α)Q(s,a)+α(r+γmax a' (Q(s',a'))); Where Q(s,a) represents the value function of executing action a in state s, α represents the learning rate, r represents the immediate reward after executing action a, γ represents the discount factor, s' represents the new state after executing action a, and a' represents the action selected in the new state s'.

10. The AGV obstacle avoidance adaptive control method based on reinforcement learning according to claim 5, characterized in that: In step S3, the DDPG algorithm uses the classic TD algorithm for update, and introduces target_network to estimate TD_target. The expression is: Δθq(sj,μ(sj;θ);w)=Δθμ(sj,θ)Δaq(sj,a^j;w)θ←θ+β·θ; Where S represents the state parameter, θ represents the policy network parameter, w represents the value network parameter, μ(sj,θ) represents the action parameter, and q(sj,a^j;w) represents the value parameter.

Images