A dynamic obstacle avoidance method for AGV based on machine learning

Abstract

The present invention belongs to the field of robot-related technology, and discloses an AGV dynamic obstacle avoidance method based on machine learning, comprising the following steps: (1) performing dynamic obstacle delineation on two-dimensional laser radar point cloud data, and marking the coverage range of dynamic obstacles; (2) calculating and obtaining the velocity vector of dynamic obstacles; (3) calculating and obtaining the trend vector of dynamic obstacles; (4) constructing an AGV motion guidance matrix and a motion guidance tensor; (5) using the AGV motion guidance tensor as a neural network input of deep reinforcement learning to construct a neural network; (6) introducing a multi-order dynamic penalty mechanism based on the motion state of dynamic obstacles into deep reinforcement learning training; (7) introducing an AGV forward reward mechanism into deep reinforcement learning training; (8) performing deep reinforcement learning obstacle avoidance training, and then controlling the AGV to avoid obstacles. The present invention has high avoidance safety and avoidance efficiency, and good feasibility and adaptability.

Description

Technical Field

[0001] The present invention belongs to the field of robot-related technologies, and more specifically, relates to an AGV dynamic obstacle avoidance method based on machine learning. Background Art

[0002] With the rapid development of computer technology and robotics technology, and the urgent need to improve the level of industrial production automation, automatic guided vehicles are gradually being used in various industrial production scenarios. Typical application scenarios include material handling in automated production lines, material dumping in large warehouses, and material handling and loading and unloading operations in flexible production workshops. The navigation method of AGV is also constantly changing, from the early magnetic track navigation, to the fixed QR code navigation, and then to the current natural navigation method. Different navigation methods directly determine the operational flexibility of AGV on the one hand, and on the other hand, they also put forward higher requirements on its movement ability. One of the key points is the automatic obstacle avoidance ability of AGV. In AGVs using natural navigation, AGVs have the ability to perceive the environment because they are equipped with sensors such as laser radar, which enables them to perceive scene changes and also provides a prerequisite for their autonomous avoidance of dynamic obstacles. The design of the dynamic obstacle avoidance method of such AGV determines its operating stability and operational flexibility in dynamic scenes. It can be foreseen that AGVs with environmental perception and autonomous obstacle avoidance capabilities will become the development direction of AGVs in the next stage. Therefore, the research and design of its dynamic obstacle avoidance method is of great significance. The existing AGV dynamic obstacle avoidance method has the following main shortcomings:

[0003] (1) In traditional avoidance algorithms, the main implementation logic is to use a fixed algorithm to replan a local path after detecting the obstacle position information, replacing the original path with a new local path to avoid the obstacle. This type of method performs well when facing static obstacles, but because it ignores the movement information of the obstacle, its avoidance safety performance is poor when facing dynamic obstacles.

[0004] (2) In traditional avoidance algorithms, the actual motion state and danger level of different dynamic obstacles are ignored, and the same avoidance strategy is used when facing fast-moving objects and slow-moving objects, which increases the amplitude of unnecessary avoidance actions to a certain extent, resulting in a decrease in avoidance efficiency.

[0005] (3) In traditional avoidance algorithms, the operation of the algorithm usually requires the support of pre-built path information. If the operational requirements change, human intervention is required to update the environmental information, or external equipment is required to collect environmental information. The implementation of the technical solution is not convenient enough. Summary of the invention

[0006] In view of the above defects or improvement needs of the prior art, the present invention provides an AGV dynamic obstacle avoidance method based on machine learning. This method enables an AGV equipped with a two-dimensional laser radar to quickly identify and judge dynamic obstacles and their motion states, and select avoidance actions with higher safety based on the calculation results; the information for the AGV to make avoidance action decisions all comes from the sensors equipped by itself without the assistance of external equipment, so when the AGV service environment changes, there is no need for human intervention to update the environmental information, and the AGV can maintain its original dynamic avoidance performance.

[0007] To achieve the above object, the present invention provides an AGV dynamic obstacle avoidance method based on machine learning, which mainly includes the following steps:

[0008] (1) Processing the point cloud data of the AGV motion scene collected by the two-dimensional laser radar installed on the AGV to mark the dynamic obstacles in the motion scene and the coverage of the dynamic obstacles, and then calculating the velocity vector of the dynamic obstacle based on the obtained position information of the dynamic obstacle; at the same time, performing weighted average calculation on the change of the velocity vector of the obstacle within the current predetermined time period to obtain the motion trend vector of the dynamic obstacle;

[0009] (2) The AGV motion guidance matrix is ​​constructed using the position vector of the AGV and the target point, the position vector of the AGV and the obstacle, the speed vector of the obstacle, and the motion trend vector of the obstacle. Then, a negatively correlated weight is taken according to the straight-line distance between the AGV and each obstacle at the current moment. Then, an AGV motion guidance tensor is constructed based on all the obtained AGV dynamic guidance matrices and their respective weights.

[0010] (3) Constructing a deep neural network based on the AGV motion guidance tensor; the input of the deep neural network is the AGV motion guidance tensor, and the output is the speed and steering angle of the AGV;

[0011] (4) The deep neural network is trained by deep reinforcement learning based on a multi-order dynamic penalty mechanism and an AGV forward reward mechanism, and the trained deep neural network is used to control the AGV to perform dynamic obstacle avoidance.

[0012] Furthermore, static objects in the moving scene are screened out by comparing the changes in the point cloud data in the two frames of data, and then the remaining points are divided and judged by the radius growth method. Finally, the coverage range of each dynamic obstacle in the plane coordinate system is represented by a circle.

[0013] Furthermore, obtaining the velocity vector of the dynamic obstacle includes the following steps:

[0014] a. For AGVs equipped with 2D laser radar, the motion environment point cloud data collected by the 2D laser radar is recorded as point set P1 in plane coordinate format. The sensor accuracy unit is G. Two adjacent frames of data are compared and the data with position fluctuation less than the predetermined coefficient c·ln[1+(x i -x agv ) 2 +(y i -y agv ) 2 ]*G, and keep the remaining points as point set P2. All points in this point set P2 are called dynamic points; where c is a constant, x i ,y i is the coordinate of a point i in the point cloud, x agv ,y agv is the coordinate of AGV;

[0015] b. Get the point M1 whose coordinates are the median of all the points in the point set P2 and call it the median point. Draw a circle C1 with point M1 as the center to satisfy the equation i=1,2,3,..., where η is a constant; i in the formula is given an increasing integer value and used to make a new circle. This step is called radius growth;

[0016] c. If there are dynamic points in the point set P2 except C1 in this radius increase that are located inside or on the new circle C1, then Then it is determined that the point corresponds to the same object and is included in the circle, and then the radius growth continues;

[0017] d. Repeat steps b and c until no dynamic points in the point set P2 are included in the circle in the radius growth. Then, it is determined that all points in the circle correspond to the same object, and the circle is used as the coverage range corresponding to the object in the plane coordinate system.

[0018] e. For the remaining points in point set P2, take point M1 as the horizontal coordinate It is divided into two parts based on and The two parts of the points perform steps b, c, and d respectively, and finally obtain all the circles that meet the conditions and record them as set C. Each circle C in set C i The coverage area that will be judged as a dynamic obstacle;

[0019] f, according to each circle C in the set C i The coordinate changes in the previous and next data frames are combined with the time difference of the sensor data frames to obtain the velocity vector of the dynamic obstacle at time t.

[0020] Furthermore, each circle C in the processed set C i The center coordinate of the circle is marked as x i,t,y i,t According to the coordinate change of the center of the circle in the previous and next data frames, combined with the time difference of the data frames, the linear velocity v of the dynamic obstacle movement is obtained. i,t and the direction of motion θ i,t , combined into a velocity vector

[0021] Furthermore, let the current time be time t, and calculate the obstacle linear velocity v at time t-3 to time t, a total of 4 times i,t and the obstacle moving direction θ i,t , and find v at two adjacent moments i,t and θ i,t The difference, Δv i,t =v i,t -v i,t-1 and Δθ i,t =θ i,t -θ i,t-1 , where i = t-3, t-2, t-1, t, and constant γ = 0.8, and the three differences are weighted averaged according to the following formula:

[0022]

[0023]

[0024] Expressed in vector form as This vector is called the motion trend vector of dynamic obstacle i at time t.

[0025] Furthermore, the coordinate difference between the AGV and the target point at time t is used to construct the vector The coordinate difference between AGV and obstacle constructs the vector and the velocity vector of the obstacle The obstacle's motion trend vector Build the matrix together It is called the motion guidance matrix of the AGV to the dynamic obstacle at time t; agv,t ,y agv,t represents the coordinates of the AGV at time t, x i,t ,y i,t Represents the coordinates of the i-th dynamic obstacle at time t.

[0026] Furthermore, if there are multiple dynamic obstacles within the perception range of the AGV, the corresponding guidance matrices W are established respectively. i,t , where i represents the number of the dynamic obstacle and t represents the time; the weight is then taken according to the straight-line distance between the AGV and each obstacle at the current time. Where c is a constant, x agv,t ,yagv,t represents the coordinates of the AGV at time t, x i,t ,y i,t represents the coordinates of the ith dynamic obstacle at time t, and e is the natural logarithm. i,t and their respective weights λ i,t Together they form a three-dimensional tensor X = (λ 1,t ·W1,λ 2,t ·W2,...,λ i,t ·W i ), called the AGV motion guidance tensor at time t.

[0027] Furthermore, the multi-order dynamic penalty mechanism dynamically changes the penalty area and penalty value according to the motion state of the dynamic obstacle. When the AGV's action causes it to enter the penalty area, the action is punished accordingly. The outline of the penalty area is a number of ellipses D with the same focus and collinear major axes. i , their other focus should be located in the direction of the obstacle relative to this focus; a penalty value is given to the penalty area, and its penalty value i=1,2,...,n;c is a constant, x i ,y i Representing Region D i The coordinates of the point closest to the AGV.

[0028] Furthermore, the AGV forward reward mechanism rewards or punishes based on the degree to which the AGV approaches or moves away from the target point through the current action, thereby dynamically giving different reward values ​​according to the degree to which the AGV approaches the target point.

[0029] Furthermore, a reward value R is given to each step of the AGV. i , R i The formula is:

[0030] R i =c·sig(d i-1 -d i )·ln(1+|d i-1 -d i |), i=1,2,3,...

[0031]

[0032] Among them, x agv ,y agv is the current coordinate of AGV, x des ,y des are the coordinates of the target point, sig() is the sign function, and c is a constant.

[0033] In general, compared with the prior art, the above technical solution conceived by the present invention has the following beneficial effects:

[0034] 1. A dynamic obstacle fast marking method is adopted for the point cloud data collected by the AGV equipped with a two-dimensional laser radar. It is not affected by the specific shape and contour of the obstacle and can quickly and effectively divide the dynamic obstacles and their coverage range in the plane coordinates.

[0035] 2. In view of the shortcoming of insufficient avoidance safety caused by ignoring the specific motion state of dynamic obstacles in traditional obstacle avoidance algorithms, by constructing the dynamic obstacle motion trend vector, it can effectively express the motion change trend of dynamic obstacles in recent times, providing a better prerequisite for AGV to make safer avoidance actions.

[0036] 3. In order to solve the problem of avoiding multiple dynamic obstacles, corresponding motion trend vectors are constructed for AGV and each dynamic obstacle respectively, and then an AGV motion guidance tensor with dynamic weights is constructed, which uniformly represents the motion relationship between AGV and each dynamic obstacle, providing a better prerequisite for AGV to make safer avoidance actions in multiple dynamic obstacle scenarios.

[0037] 4. In order to solve the problem of large avoidance action amplitude and low avoidance efficiency caused by using fixed penalty areas for reinforcement learning training, a multi-order dynamic reward and punishment mechanism based on the motion state of dynamic obstacles is used. Compared with traditional methods, it is more effective to express the degree of danger of dynamic obstacles to AGV in different motion relationships. It can allow AGV to make more appropriate avoidance actions according to the actual motion conditions of dynamic obstacles, thereby improving the avoidance efficiency of AGV. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a flowchart of a step of a machine learning-based AGV dynamic obstacle avoidance method provided by a specific embodiment of the present invention;

[0039] Figure 2 (a), (b), and (c) are schematic diagrams of the screening process of "dynamic points" in the two-dimensional laser radar data involved in the present invention;

[0040] Figure 3 It is a schematic diagram of the "radius growth" step in the dynamic obstacle delineation process involved in the present invention;

[0041] Figure 4 It is a schematic diagram of the effect of rapid dynamic obstacle delineation involved in the present invention;

[0042] Figure 5 It is a schematic diagram of the movement trend of the dynamic obstacle involved in the present invention;

[0043] Figure 6 is a schematic diagram of a multi-dynamic obstacle scene involved in the present invention;

[0044] Figure 7 This is a schematic diagram of the area division of the multi-order dynamic reward and punishment mechanism based on the movement state of obstacles;

[0045] Figure 8 (a) and (b) are schematic diagrams of the change in area size of the multi-order dynamic reward and punishment mechanism based on the motion state of the obstacle. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0047] See also Figure 1 The present invention provides an AGV dynamic obstacle avoidance method based on machine learning, and the AGV dynamic obstacle avoidance method mainly includes the following steps:

[0048] Step 1: Process the point cloud data of the AGV motion scene collected by the two-dimensional laser radar installed on the AGV to mark the dynamic obstacles in the motion scene and the coverage range of the dynamic obstacles, and then calculate the velocity vector of the dynamic obstacle based on the position information of the dynamic obstacle; at the same time, perform weighted average calculation on the change of the speed vector of the obstacle within the current predetermined time period to obtain the motion trend vector of the dynamic obstacle.

[0049] Specifically, a fast processing algorithm for two-dimensional laser radar point cloud data is used to quickly process, screen, and divide the point cloud data of the AGV's motion scene collected by the two-dimensional laser radar installed on the AGV to mark the dynamic obstacles in the motion scene and the coverage range of the dynamic obstacles, and then the obtained obstacle position information is used for calculation to obtain the velocity vector of the dynamic obstacle.

[0050] The fast marking method used screens out static objects in the moving scene by comparing the changes in the point cloud data in the two frames of data, and then divides and determines the remaining points by the radius growth method, and finally represents the coverage of each dynamic obstacle in the plane coordinate system with a circle. Specifically, it includes the following steps:

[0051] a. For AGVs equipped with 2D laser radar, the point cloud data of the motion environment collected by the 2D laser radar is in plane coordinate format, i.e., point set P1. The sensor accuracy unit is G. Two adjacent frames of data are compared and the data with position fluctuation less than the predetermined coefficient c·ln[1+(x i -x agv ) 2 +(y i -y agv ) 2 ]*G, and keep the remaining points as point set P2. All points in this point set P2 are called dynamic points. i ,y i is the coordinate of a point in the point cloud, x agv ,y agv is the coordinate of the AGV; the predetermined coefficient indicates that the greater the distance of the point cloud data from the AGV, the greater the tolerance for fluctuation.

[0052] In this embodiment, for an AGV equipped with a two-dimensional laser radar, the environmental point cloud data collected by a certain frame of the two-dimensional laser radar is recorded in the plane coordinate format as point set P1, and the sensor accuracy unit is recorded as G. Then the sensor data of the previous frame is taken for comparison, and the points whose plane coordinate position fluctuations between the two frames of data are less than G are eliminated, and the remaining points are retained and recorded as point set P2. All points in the point set P2 are called dynamic points, please refer to Figure 2 .

[0053] b. Get the point M1 whose coordinates are the median of all the points in the point set P2 and call it the median point. Draw a circle C1 with point M1 as the center to satisfy the equation i=1,2,3,..., where η is a constant; i in the formula is given an increasing integer value and used to make a new circle. This step is called radius growth.

[0054] In this embodiment, the median of the horizontal and vertical coordinates of all points in the point set P2 is taken, and the point corresponding to the median is recorded as M1, and a circle C1 is drawn with the point M1 as the center and a radius equal to G. Next, the radius of the circle C1 will gradually increase with a step size of G, and this step is called radius growth.

[0055] c. If there are dynamic points in the point set P2 except C1 in this radius increase that are located inside or on the new circle C1, then If i=1,2,3,..., the point is determined to correspond to the same object and included in the circle, and then the radius growth continues.

[0056] In this embodiment, if there are dynamic points in the point set P2 other than C1 located in the new circle C1 or on the circle C1 during a radius growth process, it is determined that these points belong to the same object as the existing points in C1, and all these points are included in the circle, and then the radius growth is continued. Figure 3 .

[0057] d. Repeat steps b and c until no dynamic points in the point set P2 are included in the circle during the radius growth. Then, it is determined that all points in the circle correspond to the same object, and the circle is used as the coverage range corresponding to the object in the plane coordinate system.

[0058] In this embodiment, the above steps are repeated until no dynamic points in the point set P2 are included in the circle C1 in a certain radius increase, then all points in the circle are determined to belong to the same object, and the circle C1 is used as the coverage range corresponding to the object in the actual scene and in the coordinate plane, and the circle will be used in subsequent steps.

[0059] e. For the remaining points in point set P2, take point M1 as the horizontal coordinate It is divided into two parts based on and The two parts of the points perform steps b, c, and d respectively, and finally obtain all the circles that meet the conditions and record them as set C. Each circle C in set C i The coverage area that will be judged as a dynamic obstacle.

[0060] In this embodiment, for the remaining points in the point set P2 that are not included in the circle C1, the horizontal coordinate of the center M1 of the circle C1 is The entire plane coordinate system is divided into two parts based on and The above steps are performed on the two parts of the points respectively, and finally all circles that meet the conditions are obtained, which are recorded as set C. Each circle C in set C i will be judged as dynamic obstacle coverage, and these circles will be used in subsequent steps, see Figure 4 .

[0061] f, according to each circle C in the set C i The coordinate change in the previous and next data frames, combined with the time difference of the sensor data frames, can be used to obtain the velocity vector velocity of the dynamic obstacle at time t.

[0062] In this embodiment, the current time is assumed to be time t, and two frames of sensor data at time t and time t-1 with a time interval of Δt are taken to perform the above steps. i The center coordinate of the circle is marked as x i,t ,y i,tAccording to the coordinate change of the center of the circle in the previous and next data frames, combined with the time difference of the data frames, the linear velocity v of the dynamic obstacle movement is obtained. i,t and the direction of motion θ i,t , combined into a velocity vector

[0063] Obstacle linear speed:

[0064] Obstacle movement direction:

[0065] See also Figure 5 , let the current time be time t, and calculate the obstacle linear velocity v at t-3 to t, a total of 4 times, according to the above steps i,t and the obstacle moving direction θ i,t , and find v at two adjacent moments i,t and θ i,t The difference, Δv i,t =v i,t -v i,t-1 and Δθ i,t =θ i,t -θ i,t-1 , where i = t-3, t-2, t-1, t. Take constant γ = 0.8, and perform weighted average of the above three differences according to the following formula:

[0066]

[0067]

[0068] Expressed in vector form as This vector is called the motion trend vector of the dynamic obstacle i at time t, or trend vector for short, and is recorded as

[0069] The weighted average of the recent changes in the motion state of the dynamic obstacle is used to represent the trend of the dynamic obstacle's motion change, and the farther the change is from the present, the lower the corresponding weight.

[0070] Step 2: Use the position vector of the AGV and the target point, the position vector of the AGV and the obstacle, the speed vector of the obstacle, and the motion trend vector of the obstacle to construct the AGV motion guidance matrix. Then, according to the straight-line distance between the AGV and each obstacle at the current moment, a negatively correlated weight is taken. Finally, based on all the obtained AGV dynamic guidance matrices and their respective weights, an AGV motion guidance tensor is constructed.

[0071] Specifically, the coordinate difference between the AGV and the target point at time t is used to construct the vector The coordinate difference between AGV and obstacle constructs the vector and the velocity vector of the obstacle The obstacle's motion trend vector Build the matrix together It is called the motion guidance matrix of the AGV to the dynamic obstacle at time t, or guidance matrix for short.

[0072] Among them, the AGV motion guidance matrix is ​​a matrix constructed using 4 vectors, which uniformly represents the motion relationship between the AGV and a dynamic obstacle and the target point at a certain moment. The matrix elements all come from the AGV's own sensor data and do not need to be assisted by externally collected information.

[0073] In this embodiment, the current time is set as time t, and the coordinate difference between the AGV and the target point is used to construct the vector The coordinate difference between AGV and obstacle constructs the vector The velocity vector of the obstacle The obstacle's motion trend vector Construct a 2×4 matrix as follows:

[0074]

[0075] This matrix is ​​called the motion guidance matrix of the dynamic obstacle i on the AGV at time t, or guidance matrix for short.

[0076] If there are multiple dynamic obstacles within the perception range of the AGV, the corresponding guidance matrix W is established respectively. i,t , where i represents the number of the dynamic obstacle and t represents the time; the weight is then taken according to the straight-line distance between the AGV and each obstacle at the current time. Where c is a constant, x agv,t ,y agv,t represents the coordinates of the AGV at time t, x i,t ,y i,t represents the coordinates of the ith dynamic obstacle at time t, and e is the natural logarithm. i,t and their respective weights λ i,t Together they form a three-dimensional tensor X = (λ 1,t ·W1,λ 2,t ·W2,...,λ i,t ·W i ), called the AGV motion guidance tensor at time t, or guidance tensor for short. It uniformly represents the motion relationship between the AGV and each dynamic obstacle at time t and the importance of each relationship. For multiple dynamic obstacles, the corresponding guidance matrix is ​​first established, and then the weight is determined according to the distance between the AGV and each obstacle. Finally, the weights are added and formed into a tensor as the input of the deep neural network.

[0077] In this implementation, please refer to Figure 6 , let the current time be time t, and calculate the motion guidance matrix W of each obstacle at time t according to the above steps i,t (i=1,2,...,n). Calculate the straight-line distance D between each obstacle and AGV at time t i,t , then take As a weight coefficient, it is added to the motion guidance matrix of each obstacle. Finally, all the weighted motion guidance matrices are expressed in the form of a three-dimensional tensor, which can be recorded as: X t =(λ 1,t W1,λ 2,t W2,...,λ i,t W i ), which is called the motion guidance tensor of AGV at time t.

[0078] Step 3: Based on the AGV motion guidance tensor, the Actor-Critic type reinforcement learning algorithm is used as the algorithm framework to build a deep neural network that meets the algorithm requirements. The AGV motion guidance tensor is used to define the environment state space, and the AGV speed and steering angle are used to define the action space; the input of the deep neural network is the AGV motion guidance tensor, and the output is the AGV speed and steering angle.

[0079] In this implementation, based on the Actor-Critic algorithm in the field of deep reinforcement learning, the obtained AGV motion guidance tensor is used as the environmental state space, which is used as the input layer of the deep neural network. The action space of the Actor-Critic algorithm is set to a two-dimensional vector (a, b), where a represents the speed of the AGV and b represents the angle of the AGV movement, which is used as the output layer of the deep neural network. The specific parameters of the hidden layer of the deep neural network should be adjusted according to the actual situation.

[0080] Step 4: Perform deep reinforcement learning training on the deep neural network based on the multi-order dynamic penalty mechanism and the AGV forward reward mechanism, and use the trained deep neural network to control the AGV to perform dynamic obstacle avoidance.

[0081] Specifically, for deep reinforcement learning training, a multi-order dynamic penalty mechanism is used to dynamically change the penalty area range and penalty value according to the dynamic obstacle's own motion state. When the AGV's action causes it to enter the penalty range, the action is punished accordingly. By dynamically adjusting the penalty area and penalty value based on the obstacle's speed, obstacles in different motion states with different risk levels can be more appropriately represented, which can guide the deep neural network to train a more efficient action strategy.

[0082] A multi-order dynamic reward and punishment mechanism designed for the dynamic obstacle avoidance problem. The outline of the penalty area is a number of ellipses D with the same focus and collinear major axes. i , their other focus should be located in the direction of the obstacle relative to this focus, and the specific size of the ellipse should satisfy:

[0083] a. One of the two foci is M i

[0084] b. The long axis is collinear and in the same direction as the obstacle's current moving direction

[0085] c. Focal length f n satisfy: c1 is a constant

[0086] d. Long axis l n satisfy: c2 is a constant

[0087] Where v is the dimensionless value of the obstacle’s velocity, M i The center of the circle representing the obstacle coverage, R i M i radius.

[0088] The penalty area is assigned a penalty value. Specifically, for area D i , its penalty value i=1,2,...,n;c is a constant. i The numbering rule is to increase from small to large areas, and x i ,y i Representing Region D i The coordinates of the point closest to the AGV.

[0089] This penalty mechanism gives different penalty values ​​to the actions of the AGV in different directions and at different distances based on the current speed of the dynamic obstacle. This improves the problem of excessive avoidance action and low avoidance efficiency that may occur when the AGV faces a low-speed dynamic obstacle or passes behind a dynamic obstacle when using fixed penalty values ​​and penalty areas.

[0090] For deep reinforcement learning training, an AGV forward reward mechanism is designed. This mechanism rewards or punishes the AGV based on the degree to which it approaches or moves away from the target point through the current action. Different reward values ​​are dynamically given according to the degree to which the AGV approaches the target point, which can better promote the action strategy of the deep neural network training to reach the target point as quickly as possible. In the design of the AGV forward reward mechanism, a reward value R is given to each step of the AGV action. i , R i The value of is defined by:

[0091] Ri =c·sig(d i-1 -d i )·ln(1+|d i-1 -d i |), i=1,2,3,...

[0092]

[0093] Among them, x agv ,y agv is the current coordinate of AGV, x des ,y des is the coordinate of the target point, sig() is the sign function, and c is a constant. This mechanism is based on the degree to which the AGV approaches the target point with each action. It gives greater rewards to more active actions approaching the target point, and gives negative rewards to actions away from the target point, thereby prompting the AGV to take actions that can reach the target point as quickly as possible.

[0094] In this implementation, please refer to Figure 7 and Figure 8 , a multi-stage dynamic reward and punishment mechanism is designed for the training steps of deep reinforcement learning. The mechanism takes the center M of a circle obtained by “obstacle demarcation” as the step i , let the radius of the circle be R i , the dimensionless value of the obstacle's linear velocity is v i ; M i For one of the foci, construct three ellipses satisfying the following constraints:

[0095] a. One of the two foci is M i

[0096] b. The long axis is collinear and in the same direction as the obstacle's current moving direction

[0097] c. Focal lengths f1, f2, and f3 are: f1 = c·v i ,f2=c 2 ·v i ,f3=c 3 ·v i ,c=1.5

[0098] d. The major axes l1, l2, and l3 are: l1 = 2c·R i +f1,l2=2c 2 ·R i +f2,l3=2c 3 ·R i +f3,c=1.25

[0099] The three elliptical areas are assigned increasing penalty values ​​from large to small. When the AGV's action causes itself to enter the area, the corresponding penalty value is given to guide the deep reinforcement learning algorithm to train an action strategy that can avoid dynamic obstacles.

[0100] For deep reinforcement learning training, an AGV reward mechanism is designed, which uses the distance d between the AGV and the target point at two adjacent moments. t and d t-1 As a quantitative indicator, a reward value R is given to each step of the AGV t , R t The value of is defined by:

[0101] R t =sig(d t-1 -d t )·ln(1+|d t-1 -d t |)

[0102]

[0103] Among them, x agv ,y agv is the current coordinate of AGV, x des ,y des are the target point coordinates, and sig() is the sign function.

[0104] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A machine learning-based AGV dynamic obstacle avoidance method, characterized in that: The method comprises the following steps: (1) Processing the point cloud data of the AGV motion scene collected by the two-dimensional laser radar installed on the AGV to mark the dynamic obstacles in the motion scene and the coverage of the dynamic obstacles, and then calculating the velocity vector of the dynamic obstacle based on the obtained position information of the dynamic obstacle; at the same time, performing weighted average calculation on the change of the velocity vector of the obstacle within the current predetermined time period to obtain the motion trend vector of the dynamic obstacle; (2) The AGV motion guidance matrix is ​​constructed using the position vector of the AGV and the target point, the position vector of the AGV and the obstacle, the speed vector of the obstacle, and the motion trend vector of the obstacle. Then, a negatively correlated weight is taken according to the straight-line distance between the AGV and each obstacle at the current moment. Then, an AGV motion guidance tensor is constructed based on all the obtained AGV dynamic guidance matrices and their respective weights. (3) Constructing a deep neural network based on the AGV motion guidance tensor; the input of the deep neural network is the AGV motion guidance tensor, and the output is the speed and steering angle of the AGV; (4) The deep neural network is trained by deep reinforcement learning based on a multi-order dynamic penalty mechanism and an AGV forward reward mechanism, and the trained deep neural network is used to control the AGV to perform dynamic obstacle avoidance.

2. The AGV dynamic obstacle avoidance method based on machine learning as claimed in claim 1, characterized in that: By comparing the changes in the point cloud data between the two frames of data, static objects in the moving scene are screened out, and then the remaining points are divided and judged by the radius growth method. Finally, the coverage of each dynamic obstacle in the plane coordinate system is represented by a circle.

3. The AGV dynamic obstacle avoidance method based on machine learning as claimed in claim 2, characterized in that: The acquisition of the velocity vector of a dynamic obstacle includes the following steps: a. For AGVs equipped with 2D laser radar, the motion environment point cloud data collected by the 2D laser radar is recorded as point set P1 in plane coordinate format. The sensor accuracy unit is G. Two adjacent frames of data are compared and the data with position fluctuation less than the predetermined coefficient c·ln[1+(x i -x agv ) 2 +(y i -y agv ) 2 ]*G, and keep the remaining points as point set P2. All points in this point set P2 are called dynamic points; where c is a constant, x i ,y i is the coordinate of a point i in the point cloud, x agv ,y agv is the coordinate of AGV; b. Get the point M1 whose coordinates are the median of all the points in the point set P2 and call it the median point. Draw a circle C1 with point M1 as the center to satisfy the equation Where η is a constant; i in the formula is taken as an increasing integer value and used to make a new circle. This step is called radius growth; c. If there are dynamic points in the point set P2 except C1 in this radius increase that are located inside or on the new circle C1, then Then it is determined that the point corresponds to the same object and is included in the circle, and then the radius growth continues; d. Repeat steps b and c until no dynamic points in the point set P2 are included in the circle in the radius growth. Then, it is determined that all points in the circle correspond to the same object, and the circle is used as the coverage range corresponding to the object in the plane coordinate system. e. For the remaining points in point set P2, take point M1 as the horizontal coordinate It is divided into two parts based on and The two parts of the points perform steps b, c, and d respectively, and finally obtain all the circles that meet the conditions and record them as set C. Each circle C in set C i The coverage area that will be judged as a dynamic obstacle; f, according to each circle C in the set C i The coordinate changes in the previous and next data frames are combined with the time difference of the sensor data frames to obtain the velocity vector of the dynamic obstacle at time t.

4. The AGV dynamic obstacle avoidance method based on machine learning as claimed in claim 3, characterized in that: Each circle C in the processed set C i The center coordinate of the circle is marked as x i,t ,y i,t According to the coordinate change of the center of the circle in the previous and next data frames, combined with the time difference of the data frames, the linear velocity v of the dynamic obstacle movement is obtained. i,t and the direction of motion θ i,t , combined into a velocity vector 5. The AGV dynamic obstacle avoidance method based on machine learning as claimed in claim 4, characterized in that: Assume the current time is time t, calculate the obstacle linear velocity v at time t-3 to time t, a total of 4 times i,t and the obstacle moving direction θ i,t , and find v at two adjacent moments i,t and θ i,t The difference, Δv i,t =v i,t -v i,t-1 and Δθ i,t =θ i,t -θ i,t-1 , where i = t-3, t-2, t-1, t, and constant γ = 0.8, and the three differences are weighted averaged according to the following formula: Expressed in vector form as This vector is called the motion trend vector of dynamic obstacle i at time t.

6. The AGV dynamic obstacle avoidance method based on machine learning according to any one of claims 1 to 5, characterized in that: Use the coordinate difference between the AGV and the target point at time t to construct a vector The coordinate difference between AGV and obstacle constructs the vector and the velocity vector of the obstacle The obstacle's motion trend vector Build the matrix together It is called the motion guidance matrix of the AGV to the dynamic obstacle at time t; agv,t ,y agv,t represents the coordinates of the AGV at time t, x i,t ,y i,t Represents the coordinates of the i-th dynamic obstacle at time t.

7. The AGV dynamic obstacle avoidance method based on machine learning according to any one of claims 1 to 5, characterized in that: If there are multiple dynamic obstacles within the perception range of the AGV, the corresponding guidance matrix W is established respectively. i,t , where i represents the number of the dynamic obstacle and t represents the time; the weight is then taken according to the straight-line distance between the AGV and each obstacle at the current time. Where c is a constant, x agv,t ,y agv,t represents the coordinates of the AGV at time t, x i,t ,y i,t represents the coordinates of the ith dynamic obstacle at time t, e is the natural logarithm; the matrix W i,t and their respective weights λ i,t Together they form a three-dimensional tensor X = (λ 1,t ·W1,λ 2,t ·W2,...,λ i,t ·W i ), called the AGV motion guidance tensor at time t.

8. The AGV dynamic obstacle avoidance method based on machine learning according to any one of claims 1 to 5, characterized in that: The multi-order dynamic penalty mechanism dynamically changes the penalty area and penalty value according to the motion state of the dynamic obstacle. When the AGV's action causes it to enter the penalty range, the action is punished accordingly. The outline of the penalty area is a number of ellipses D with the same focus and collinear major axes. i , their other focus should be located in the direction of the obstacle relative to this focus; a penalty value is given to the penalty area, and its penalty value c is a constant, x i ,y i Representing Region D i The coordinates of the point closest to the AGV.

9. The AGV dynamic obstacle avoidance method based on machine learning according to any one of claims 1 to 5, characterized in that: The AGV forward reward mechanism rewards or punishes based on the degree to which the AGV approaches or moves away from the target point through the current action, thereby dynamically giving different reward values ​​according to the degree to which the AGV approaches the target point.

10. The AGV dynamic obstacle avoidance method based on machine learning as claimed in claim 9, characterized in that: Give a reward value R to each step of the AGV i , R i The formula is: R i =c·sig(d i-1 -d i )·ln(1+|d i-1 -d i |),i=1,2,3,... Among them, x agv ,y agv is the current coordinate of AGV, x des ,y des are the coordinates of the target point, sig() is the sign function, and c is a constant.

Images