Building robot path planning method and related device

Abstract

The invention belongs to the technical field of robot path planning, and discloses a building robot path planning method and related devices.The building robot path planning method comprises the steps that the current state of a building robot is input into an improved double-delay depth deterministic strategy gradient model to obtain a planning action, executing the planning action to obtain the state of the next moment and obtaining a reward for calculating the planning action; based on the state, rewards and actions at the current moment and the state at the next moment, experience data is formed through a sampling priority experience playback mechanism and stored in an experience pool; sampling from an experience pool based on a sampling priority experience playback mechanism, and training the improved double-delay depth deterministic strategy gradient model to obtain a path planning model; performing path planning by using the path planning model to obtain a building robot path planning result; according to the method, the problems of gradient disappearance and gradient explosion of traditional models such as RNN are effectively solved, the model training time is greatly shortened, and the convergence speed is increased.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot path planning, and in particular relates to a construction robot path planning method and related devices. Background Art

[0002] Construction robots refer to robotic equipment used in the entire life cycle of a building, such as design, construction, operation and maintenance, and demolition processes. They are considered to be one of the effective ways to improve construction efficiency, quality, and safety. At construction sites, the working environment of construction robots is usually an unstructured and unknown environment, which requires robots to have autonomous perception, autonomous path planning, and autonomous obstacle avoidance. Path planning is the key to autonomous navigation of construction robots, and its accuracy affects the efficiency of subsequent work of the construction robots.

[0003] At present, the research on construction robot path planning mostly adopts traditional path planning algorithms; however, with the continuous expansion of application scenarios of construction robots, the environment they face is becoming more and more complex; but traditional path planning algorithms rely heavily on mathematical models and expert experience. When faced with dynamic and complex construction environments, the algorithm has obvious defects and is not enough to generate reliable construction paths for construction robots; secondly, construction sites usually have complex and changeable terrain and environments, including uneven ground, dynamic construction machinery, and stacking of construction materials; the path planning algorithm of construction robots needs to have sufficient adaptability to be able to stably drive and perform tasks in complex environments; research has found that deep reinforcement learning has begun to be applied to construction robot path planning in recent years because of its advantages such as being able to handle high-dimensional state and action space and being suitable for handling nonlinear and complex relationships.

[0004] However, when traditional deep reinforcement learning algorithms are applied to construction environments, there are still many problems with the path planning of construction robots. For example, in complex dynamic construction environments, high-dimensional data input can easily cause dimensionality explosion. Secondly, traditional deep reinforcement learning algorithm models have the defects of single action space and state space and sparse reward function, which leads to long training time and slow convergence. Summary of the invention

[0005] In view of the technical problems existing in the prior art, the present invention provides a construction robot path planning method and related devices to solve the technical problems that the existing construction robot path planning process is prone to dimensionality explosion, long training time and slow convergence speed.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is:

[0007] The present invention provides a construction robot path planning method, comprising:

[0008] Constructing an improved dual-delay deep deterministic policy gradient model; wherein the improved dual-delay deep deterministic policy gradient model is a fusion model of the LSTM algorithm and the dual-delay deep deterministic policy gradient algorithm;

[0009] Inputting the current state of the construction robot into the improved double-delay deep deterministic policy gradient model to obtain a planning action, executing the planning action to obtain the state at the next moment and obtaining a reward for calculating the planning action;

[0010] Based on the current state, reward and action as well as the state at the next moment, experience data is formed through the sampling priority experience playback mechanism and stored in the experience pool;

[0011] Sampling from the experience pool based on the sampling priority experience replay mechanism, and training the improved double-delay deep deterministic policy gradient model to obtain a path planning model;

[0012] The path planning model is used to perform path planning to obtain a construction robot path planning result.

[0013] Furthermore, the improved dual-delay deep deterministic policy gradient model includes an Actor network module and a Critic network module;

[0014] The Actor network module includes an Actor current network and an Actor target network, and the Critic network module includes Critic current network 1, Critic current network 2, Critic target network 1 and Critic target network 2; wherein, the structures of the Actor current network, the Actor target network, the Critic current network 1, Critic current network 2, Critic target network 1 and Critic target network 2 are the same, including an input layer, a first LSTM layer, a second LSTM layer, a fully connected layer and an output layer.

[0015] Furthermore, based on the sampling priority experience replay mechanism, sampling is performed from the experience pool, and the improved dual-delay deep deterministic policy gradient model is trained. In the process of obtaining the path planning model, the network parameters of the Actor network module and the network parameters of the Critic network module are updated by introducing a delayed update strategy of the soft update mechanism.

[0016] Furthermore, random noise is added to the Actor target network; wherein the random noise is specifically:

[0017]

[0018] ε~clip(N(0,σ),-c,c)

[0019] Among them, y is the cumulative reward of target Q; r is the instantaneous reward; γ is the discount factor; is the output value of Critic's current network 1; is the output value of the current network 2 of Critic; s′ is the state at the moment; π φ′ (s′) is the action output by the current Critic network; ε is the added noise based on normal distribution; c is the upper boundary of the noise; -c is the lower boundary of the noise; clip(*) is the truncation function.

[0020] Furthermore, the state space of the current state of the construction robot includes a motion state, a collision state, and a state of successfully reaching the target; wherein the state space of the current state of the construction robot is specifically:

[0021]

[0022] Among them, Continue is the motion state; Collision is the collision state; Goal is the state of successfully reaching the goal; d t is the distance data perceived by the construction robot’s laser radar at time t; d g is the distance between the construction robot and the target point.

[0023] Furthermore, in the process of executing the planned action to obtain the state at the next moment and obtaining the reward for calculating the planned action, the reward for the planned action is calculated using the reinforcement learning reward function;

[0024] The reinforcement learning reward function is specifically:

[0025]

[0026] R r =100+λ1(d-d0)+λ2(t-t0)

[0027] R c =-100

[0028] R g =α·v-β·w+γ·dis

[0029] Among them, r(s t ,a t ) is the reward value function; R r Reward value for reaching the target point; R c is the collision state reward value; R gis the sparse reward; λ1 is the trajectory length adjustment factor; d is the target point distance; d0 is the moving trajectory length; λ2 is the adjustment factor for the planning time; t is the current time; t0 is the start time; α is the linear velocity weight; v is the linear velocity; β is the angular velocity weight; w is the angular velocity; γ is the laser radar c ranging weight; dis is the value calculated by the function specification after the laser radar ranging.

[0030] Furthermore, the sampling probability in the sampling priority experience playback mechanism is:

[0031]

[0032] Among them, P(i) is the probability that the i-th experience is selected; is the priority of the i-th experience; α is a hyperparameter; is the parameter for adjusting the weight during sampling; k is the total number of samples in the experience pool;

[0033] In the sampling priority experience playback mechanism, a priority sampling method with importance sampling weights is introduced to correct the samples, and the importance sampling weights are applied to the experience update formula of the experience priority;

[0034] The importance sampling weight is specifically:

[0035]

[0036] Among them, w i is the importance sampling weight of the i-th experience; N is the total number of samples in the experience replay buffer; β′ is a hyperparameter used to control the degree of importance sampling.

[0037] The present invention also provides a construction robot path planning system, comprising:

[0038] A modeling module, used to construct an improved dual-delay deep deterministic policy gradient model; wherein the improved dual-delay deep deterministic policy gradient model is a fusion model of the LSTM algorithm and the dual-delay deep deterministic policy gradient algorithm;

[0039] A model training module is used to input the current state of the construction robot into the improved double-delay deep deterministic policy gradient model to obtain a planning action, execute the planning action to obtain the state at the next moment and obtain a reward for calculating the planning action; based on the state, reward and action at the current moment and the state at the next moment, experience data is formed through a priority experience replay mechanism and stored in an experience pool; based on the priority experience replay mechanism, samples are taken from the experience pool, and the improved double-delay deep deterministic policy gradient model is trained to obtain a path planning model;

[0040] The path planning module is used to perform path planning using the path planning model to obtain a path planning result for the construction robot.

[0041] The present invention also provides a construction robot path planning device, comprising:

[0042] Memory for storing computer programs;

[0043] A processor is used to implement the steps of the construction robot path planning method when executing the computer program.

[0044] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the construction robot path planning method are implemented.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] The construction robot path planning method provided by the present invention utilizes the LSTM algorithm and the double-delay deep deterministic policy gradient algorithm to construct a fusion model, controls the network's memory and forgetting degree of experience through the LSTM, and dynamically modifies the network weights in a self-loop to adjust the information flow, thereby effectively solving the gradient vanishing and gradient explosion problems of traditional models such as RNN; secondly, by introducing a double critic network and a delayed update mechanism in the double-delay deep deterministic policy gradient algorithm, the problems of overestimation of Q value and unstable training existing in the traditional DDPG algorithm are effectively solved, so that the TD3 algorithm exhibits better performance and stability in continuous control tasks; in the process of constructing experience data and sampling, a sampling priority experience replay mechanism is introduced, which uses the temporal difference error as an indicator to measure the priority, greedily selects the experience data with the highest value, thereby improving the algorithm efficiency and training effect, greatly reducing the model training time, and improving the convergence speed.

[0047] Furthermore, a new action space and state space were designed according to the characteristics of the construction environment. The radar data was shielded and divided, and the motion state of the construction robot was reset and divided into motion state, collision state and target point arrival state, so as to improve the adaptability of the construction robot's path planning in complex construction environments.

[0048] Furthermore, based on the existing design method of reward function, by avoiding the problems of sparse reward design and formal reward design, the advantages of reward coefficient variable reward design method are absorbed. At the same time, factors such as the distance between the construction robot and the target point, movement speed, movement time, running status, whether the target point is reached, and whether a collision occurs are considered when the construction robot is searching for the target. A negative reward mechanism is set according to other possible states. It can be effectively applied to complex construction environments and ensure that the construction robot improves training efficiency under the premise of safe path planning.

[0049] The construction robot path planning system, construction robot path planning device and computer-readable storage medium provided by the present invention have all the advantages of the above-mentioned construction robot path planning method. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to make the technical problems, technical solutions and beneficial effects solved by this application clearer, the technical solutions in the embodiments of this application will be described clearly and completely in combination with the drawings in the embodiments of this application; obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0051] Figure 1 A flowchart of the construction robot path planning method provided in Example 1;

[0052] Figure 2 This is a framework diagram of the improved double-delay deep deterministic policy gradient model in Example 1;

[0053] Figure 3 This is a network structure diagram of the improved double-delay deep deterministic policy gradient model in Example 1;

[0054] Figure 4 is a schematic diagram of an exemplary simulation environment 1 in Embodiment 1;

[0055] Figure 5 is a schematic diagram of an exemplary simulation environment 2 in Example 1;

[0056] Figure 6 A graph showing the reward value and the number of iterations for different model training;

[0057] Figure 7 The graph of the number of times the target point is reached and the number of iterations when training different models;

[0058] Figure 8 A comparison chart of path planning results using different models in the example simulation environment 1;

[0059] Fig. 9 Comparison of path planning results using different models in example simulation environment 2

[0060] Fig.10 A structural block diagram of a construction robot path planning system provided in Example 2;

[0061] Fig.11 This is a structural block diagram of the construction robot path planning device provided in Example 3. DETAILED DESCRIPTION

[0062] In order to make the technical problems, technical solutions and beneficial effects solved by the present invention more clearly understood, the present invention is further described in detail in the following specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0063] Example 1

[0064] As attached Figure 1 As shown, this embodiment 1 provides a construction robot path planning method, comprising the following steps:

[0065] Step 1: Use the laser radar and odometer of the construction robot to obtain the state information of the construction robot; and build a state model of the construction robot according to the state information of the construction robot. The state information of the construction robot includes the starting point position, target point information, robot position state, linear velocity, angular velocity and collision information of the construction robot.

[0066] Step 2: Build an improved double-delayed deep deterministic policy gradient model. Figure 2 As shown, the improved dual-delay deep deterministic policy gradient model includes an Actor network module and a Critic network module, the Actor network module includes an Actor current network and an Actor target network, and the Critic network module includes Critic current network 1, Critic current network 2, Critic target network 1, and Critic target network 2. The structures of the Actor current network, the Actor target network, the Critic current network 1, the Critic current network 2, the Critic target network 1, and the Critic target network 2 are the same, including an input layer, a first LSTM layer, a second LSTM layer, a fully connected layer, and an output layer, as shown in the attached figure. Figure 3 shown.

[0067] In this embodiment 1, taking the Actor current network as an example, the activation function of the first LSTM layer adopts the ReLu function, and the second LSTM layer contains 600 neurons; the fully connected layer is a linear layer, and the output of the second LSTM layer is used as input to capture the information of the time series; the output layer contains 1 neuron for Q value estimation, the connection mode is fully connected, and the Sigmoid function is selected as the activation function.

[0068] It should be noted that the improved double-delay deep deterministic policy gradient model adopts a dual Q evaluation network, with the robot state as input and the output control action as the input of the Critic network module; wherein, the network module designs 6 networks, and in the Critic module, establishes 2 independent Critic networks; when calculating the target value, the Critic network and the Target-Critic network are used to evaluate the action-value Q(s,a), and the one with the smaller Q value is selected as the target value y; wherein, the cumulative reward y of the target Q value t and deterministic policy actions As follows, the details are as follows:

[0069]

[0070]

[0071] Among them, y t is the cumulative reward of the target Q value; r t is the instant reward at time t; γ is the discount factor; is the output value of the current Critic network; s t+1 is the state at time t+1; is the target action after adding noise; π φ′ (s t+1 ) is the action output by the current network of the Critic at time t+1; ε′ is Gaussian noise.

[0072] Step 3: Initialize all states of the construction robot, initialize the sample pool, Actor network module, Critic network module and environment state.

[0073] Step 4. Obtain the current state of the construction robot according to the construction robot state model, and input the current state of the construction robot into the improved dual-delay deep deterministic policy gradient model to execute the planned action a according to the current action network and the noise planning action a, obtain the state s at the next moment and obtain the reward r for the calculated planning action, and then calculate the current error δ and the priority sampling weight w.

[0074] It should be noted that the priority sampling weight w is an importance sampling weight; wherein the importance sampling weight is specifically:

[0075]

[0076] Among them, w i is the importance sampling weight of the ith experience; N is the total number of samples in the experience replay buffer; β is a hyperparameter used to control the degree of importance sampling; P(i) is the probability of the ith experience being selected.

[0077] It should be noted that the current error δ is the weighted time difference error of the i-th experience; wherein the weighted time difference error of the i-th experience is calculated by applying the importance sampling weight to the experience update formula of the experience priority; the current error δ is specifically:

[0078] δ i =w i TD

[0079] Among them, δ i is the weighted time difference error of the i-th experience; w i is the weight of the i-th experience; TD is the time difference error of the i-th experience.

[0080] In this embodiment 1, due to the design limitations of the construction robot, the laser radar is usually installed in front of the construction robot. Therefore, the laser radar will be blocked by the robot body. Only 180° of the laser radar's 360° scanning range data is available, and the remaining radar data is shielded as noise. In addition, considering the kinematics of the construction robot and the actual safety application requirements, the construction robot does not perform reverse movement. Therefore, only the radar detection data between -90° and +90° in the forward direction of the construction robot is processed during training.

[0081] In order to ensure the safety and feasibility of the construction robot's path planning, in this embodiment 1, the construction robot is judged to have hit an obstacle when the distance between the construction robot and the obstacle is less than 0.2m, and it is judged to have reached the target point when the distance between the construction robot and the target point is less than 0.1m. Through this setting, the motion state of the construction robot is divided into three states: one is the motion state, one is the collision state, and the other is the state of successfully reaching the target; among them, the state space of the current state of the construction robot is specifically:

[0082]

[0083] Among them, Continue is the motion state; Collision is the collision state; Goal is the state of successfully reaching the goal; d t is the distance data perceived by the construction robot’s laser radar at time t; d g is the distance between the construction robot and the target point.

[0084] In this embodiment 1, in order to make the target strategy smooth, random noise is added to the Actor target network part to improve the algorithm stability; wherein the random noise is specifically:

[0085]

[0086] ε~clip(N(0,σ),-c,c)

[0087] Among them, y is the cumulative reward of target Q; r is the instantaneous reward; γ is the discount factor; is the output value of Critic's current network 1; is the output value of the current network 2 of Critic; s′ is the state at the moment; π φ′ (s′) is the action output by the current Critic network; ε is the added noise based on normal distribution; c is the upper boundary of the noise; -c is the lower boundary of the noise; clip(*) is the truncation function.

[0088] It should be noted that the reward function is the standard for the construction robot to evaluate the quality of action selection. The rationality of the reward function design directly affects the algorithm training efficiency and the final effect. In the entire obstacle avoidance task, there will be a large number of collisions and unreachable targets in the early stage of network structure training. If only the task success reward +100 and the task failure reward -100 are set, it will lead to very slow convergence and even failure to learn in the early stage.

[0089] In this embodiment 1, in the process of executing the planned action to obtain the state at the next moment and obtaining the reward for calculating the planned action, the reward for the planned action is calculated using the reinforcement learning reward function;

[0090] The reinforcement learning reward function is specifically:

[0091]

[0092] R r =100+λ1(d-d0)+λ2(t-t0)

[0093] R c =-100

[0094] R g =α·v-β·w+γ·dis

[0095] Among them, r(s t ,a t ) is the reward value function; R r Reward value for reaching the target point; R c is the collision state reward value; R gis the sparse reward; λ1 is the trajectory length adjustment factor; d is the target point distance; d0 is the moving trajectory length; λ2 is the adjustment factor for the planning time; t is the current time; t0 is the start time; α is the linear velocity weight; v is the linear velocity; β is the angular velocity weight; w is the angular velocity; γ is the laser radar c ranging weight; dis is the value calculated by the function specification after the laser radar ranging.

[0096] It should be noted that in this embodiment 1, based on the existing design method of reward function, the problems of sparse reward design and formal reward design are avoided, the advantages of the reward coefficient variation reward design method are absorbed, and factors such as the distance between the construction robot and the target point, the movement speed, the movement time, the operating status, whether the target point is reached, and whether a collision occurs are considered when the construction robot is searching for the target. The construction environment is carefully studied, and a negative reward mechanism is set according to other possible states, thereby obtaining a reward function mechanism suitable for complex construction environments.

[0097] Step 5: Based on the current state, reward and action and the state at the next moment, experience data is formed and stored in the experience pool through the sampling priority experience replay mechanism. In order to make more efficient use of data and enhance the stability of network training, experience replay adopts the sampling probability in the sampling priority experience replay mechanism; wherein, the sampling probability in the sampling priority experience replay mechanism is:

[0098]

[0099] Among them, P(i) is the probability that the i-th experience is selected; is the priority of the i-th experience; α is a hyperparameter, usually a positive value, used to control the distribution of sampling probability; where α=0 is uniform sampling, α=1 is proportional sampling, and α>1 increases the preference for experiences with higher priority; is the parameter for adjusting the weight during sampling; k is the total number of samples in the experience pool.

[0100] It should be noted that the experience playback allocation method of experience priority will be frequently called because some experiences (Q values) are better, resulting in sample bias, affecting network updates, and further affecting the training convergence speed; therefore, in this embodiment 1, a priority sampling method with importance sampling weights is introduced to correct the samples, and the importance sampling weights are applied to the experience update formula of the experience priority; wherein, the importance sampling weights are specifically:

[0101]

[0102] Among them, w iis the importance sampling weight of the i-th experience; N is the total number of samples in the experience replay buffer; β′ is a hyperparameter used to control the degree of importance sampling.

[0103] The process of applying the importance sampling weights to the experience update formula of the experience priority is as follows:

[0104] δ i =w i TD

[0105] Among them, δ i is the weighted TD error of the i-th experience; w i is the weight of the i-th experience; TD is the time difference error of the i-th experience.

[0106] Step 6: Sample from the experience pool based on the sampling priority experience replay mechanism, and train the improved dual-delay deep deterministic policy gradient model.

[0107] Specifically, the double Q mean square errors are calculated according to the Loss function, and N samples are selected from the experience pool with the latest probability; the Critic network is updated by minimizing the loss function, and the Actor network is updated by the gradient descent method.

[0108] The process of updating the Critic network is as follows:

[0109] Sample N samples from the experience pool and calculate the target Q value; then, calculate the loss function of the Critic network; preferably, the loss function of the Critic network adopts mean square error; finally, use the gradient descent method or its variants to minimize the MSE loss function.

[0110] Specifically, the loss function of the Critic network is:

[0111]

[0112] The process of updating the Actor network is as follows: The goal of updating the Actor network is to maximize the Q value output by the Critic network; the Actor network uses the policy gradient method to maximize the Q value, specifically:

[0113]

[0114] It should be noted that in order to reduce the cumulative error and variance, the network update adopts a double delay strategy. The evaluation network and the action network are updated using the delayed update strategy of the soft update mechanism respectively. The update frequency of the evaluation function is greater than that of the strategy function, and the update strategy no longer directly copies the network parameters. Among them, the delayed update strategy of the soft update mechanism is specifically:

[0115] θ i =τθ i +(1-τ)θ i '

[0116] φ′=τφ+(1-τ)φ′

[0117] Among them, θ i is the current parameter of the target network; τ is the soft update rate; θ i ′ is the parameter of the target network after update; φ′ is the parameter of the action network after update; φ is the current parameter of the action network.

[0118] The traditional double-delayed deep deterministic policy gradient algorithm Loss function calculates the mean square error (MES) loss of the double Q (current_Q1 and current_Q2) respectively, and the Loss function value is the sum of the two MES losses. In the priority experience replay, samples with higher TD errors will be selected frequently, so that samples that need more learning in training will receive more attention, so that in subsequent sampling, the experience with higher TD in the experience pool will be called first.

[0119] Step 7: Set the network structure to run independently for a preset number of times according to the above steps 4-6 to determine whether the end adjustment is satisfied; if not, return to step 4; otherwise, output the path planning model.

[0120] Step 8: Apply the path planning model to the construction robot state model, use the path planning model to perform path planning, and obtain the construction robot path planning result.

[0121] The construction robot path planning method described in this embodiment 1 effectively improves the path planning effect compared to the traditional path planning algorithm.

[0122] Method performance verification, detailed description is as follows:

[0123] In this embodiment 1, two simulation environment examples of construction are built based on the Gazebo simulation platform, and a robot model is designed. Different model training is performed in the designed virtual environment, and the path planning performance is verified to verify the robustness and generalization of the model; wherein the example simulation environment 1 is as shown in the attached figure. Figure 4 As shown in the example simulation environment 2, Figure 5 shown.

[0124] As attached Figure 6-7 As shown, attached Figure 6 The curves of reward value and iteration number for different model training are given in Appendix Figure 7 The curves of the number of times reaching the target point and the number of iterations during training of different models are given in the attached Figure 6It can be seen from the figure that the improved TD3 algorithm combines LSTM and PRE, and has a significant improvement in convergence efficiency; although both the improved TD3 algorithm and the TD3 algorithm show obvious convergence trends in about 100 rounds, the improved TD3 algorithm benefits from the existence of the priority experience replay mechanism and the introduction of LSTM, and the algorithm obtains better experience during training and achieves rapid convergence; from the attached Figure 7 It can be seen that the improved TD3 algorithm has a path planning success rate of more than 86% after about 180 training times. Figure 6 The results show that the average reward value reaches 120, which is more stable and has better overall effect than the unimproved TD3 algorithm and DDPG algorithm.

[0125] As attached Figure 8-9 As shown in the attached Figure 8 The path planning results of different models in the example simulation environment 1 are given in Appendix. Fig. 9 The path planning results of different models in the example simulation environment 2 are given in the attached Figure 8-9 It can be seen that compared with the traditional algorithm, the construction robot path planning method described in this embodiment 1 has a relatively better overall effect and has better performance when applied to complex construction environments.

[0126] As shown in Tables 1-3 below, Table 1 gives a comparison of model training iteration effects, Table 2 gives a comparison of algorithm path planning performance in a normal environment, and Table 3 gives a comparison of algorithm path planning performance in a construction environment with dense obstacles.

[0127] Table 1 Comparison of model training iteration effects

[0128]

[0129] Table 2 Comparison of algorithm path planning performance under normal conditions

[0130]

[0131] Table 3 Comparison of algorithm path planning performance in construction environment with dense obstacles

[0132]

[0133] From Table 1 above, it can be seen that the improved TD3 algorithm has great advantages in average reward value and number of times reaching the target point. From Table 2 above, it can be seen that when the TD3 algorithm performs path planning in a normal environment, the path is shorter and the time taken is second only to the DDPG algorithm, showing better performance. From Table 3 above, it can be seen that both the TD3 algorithm and the DDPG algorithm can reach the target point, but they cannot quickly complete the path planning task in a construction environment with dense obstacles. Combining data analysis with experimental comparison, it can be concluded that the improved TD3 algorithm takes 8% less time to perform path planning in a complex and dense environment than the unimproved TD3 algorithm. When facing dynamic obstacles, compared with the unimproved TD3 algorithm and DDPG algorithm, the number of turning points and collisions is less, and the obstacle avoidance performance is better.

[0134] Example 2

[0135] As attached Fig.10 As shown, this embodiment 2 provides a construction robot path planning system, including a state model module, a modeling module, an initialization module, a model training module and a path planning module; the state model module is used to obtain the state information of the construction robot by using the laser radar and odometer of the construction robot; and to construct a construction robot state model according to the state information of the construction robot; the modeling module is used to construct an improved double-delay deep deterministic policy gradient model; wherein the improved double-delay deep deterministic policy gradient model is a fusion model of the LSTM algorithm and the double-delay deep deterministic policy gradient algorithm; the initialization module is used to initialize all states of the construction robot, initialize the sample pool, and the Actor network Module, Critic network module and environmental state; a model training module, used to input the current state of the construction robot into the improved double-delay deep deterministic policy gradient model to obtain a planning action, execute the planning action to obtain the state at the next moment and obtain the reward for calculating the planning action; based on the current state, reward and action and the state at the next moment, experience data is formed through a priority experience playback mechanism and stored in an experience pool; based on the priority experience playback mechanism, samples are taken from the experience pool, and the improved double-delay deep deterministic policy gradient model is trained to obtain a path planning model; a path planning module, used to use the path planning model to perform path planning and obtain a path planning result for the construction robot.

[0136] Example 3

[0137] As attached Fig.11 As shown, this embodiment 3 provides a construction robot path planning method device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor, such as a construction robot path planning method program; wherein the attached Fig.11The communication interface in is used to access external devices to obtain data.

[0138] When the processor executes the computer program, the steps of the construction robot path planning method are implemented, or when the processor executes the computer program, the functions of each module in the system are implemented.

[0139] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the construction robot path planning method and device.

[0140] The construction robot path planning method and device can be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The construction robot path planning method and device can include, but is not limited to, a processor and a memory. Those skilled in the art can understand that the attached Fig.11 The schematic diagram is merely an example of a construction robot path planning method and device and does not constitute a limitation of the construction robot path planning method and device. It may include more or fewer components than shown in the diagram, or a combination of certain components, or different components. For example, the construction robot path planning method and device may also include input and output devices, network access devices, buses, etc.

[0141] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the construction robot path planning method device, and uses various interfaces and lines to connect the various parts of the entire construction robot path planning method device.

[0142] The memory can be used to store the computer program and / or module, and the processor implements various functions of the construction robot path planning method and device by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory.

[0143] The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SmartMediaCard, SMC), a secure digital (SecureDigital, SD) card, a flash card (FlashCard), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0144] Example 4

[0145] This embodiment 4 also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the construction robot path planning method.

[0146] If the module / unit integrated in the construction robot path planning method device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0147] Based on this understanding, the present embodiment 4 implements all or part of the processes in the above-mentioned construction robot path planning method, and can also be completed by instructing related hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of the above-mentioned construction robot path planning method can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc.

[0148] The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0149] It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media does not include electrical carrier signals and telecommunication signals.

[0150] In the present invention, the LSTM control network's memory and forgetting degree of experience is used, and the network weights are dynamically modified in a self-loop to adjust the information flow, thereby effectively solving the gradient vanishing and gradient exploding problems of traditional models such as RNN. In the double-delay deep deterministic policy gradient algorithm, a double Critic network and a delayed update mechanism are introduced to effectively solve the problems of overestimation of Q value and unstable training in the traditional DDPG algorithm, so that the TD3 algorithm shows better performance and stability in continuous control tasks. In the process of constructing experience data and sampling, a sampling priority experience replay mechanism is introduced. The sampling priority experience replay mechanism uses the temporal difference error as an indicator to measure the priority, greedily selects the experience data with the highest value, improves the algorithm efficiency and training effect, greatly reduces the model training time, and improves the convergence speed.

[0151] The above embodiment is only one of the implementation methods that can realize the technical solution of the present invention. The scope of protection claimed by the present invention is not limited only to this embodiment, but also includes changes, replacements and other implementation methods that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed by the present invention.

Claims

1. A construction robot path planning method, characterized in that: include: Constructing an improved dual-delay deep deterministic policy gradient model; wherein the improved dual-delay deep deterministic policy gradient model is a fusion model of the LSTM algorithm and the dual-delay deep deterministic policy gradient algorithm; Inputting the current state of the construction robot into the improved double-delay deep deterministic policy gradient model to obtain a planning action, executing the planning action to obtain the state at the next moment and obtaining a reward for calculating the planning action; Based on the current state, reward and action as well as the state at the next moment, experience data is formed through the sampling priority experience playback mechanism and stored in the experience pool; Sampling from the experience pool based on the sampling priority experience replay mechanism, and training the improved double-delay deep deterministic policy gradient model to obtain a path planning model; The path planning model is used to perform path planning to obtain a construction robot path planning result.

2. A construction robot path planning method according to claim 1, characterized in that: The improved dual-delay deep deterministic policy gradient model includes an Actor network module and a Critic network module; The Actor network module includes an Actor current network and an Actor target network, and the Critic network module includes Critic current network 1, Critic current network 2, Critic target network 1 and Critic target network 2; wherein, the structures of the Actor current network, the Actor target network, the Critic current network 1, Critic current network 2, Critic target network 1 and Critic target network 2 are the same, including an input layer, a first LSTM layer, a second LSTM layer, a fully connected layer and an output layer.

3. A construction robot path planning method according to claim 2, characterized in that: Based on the sampling priority experience replay mechanism, sampling is performed from the experience pool, and the improved double-delayed deep deterministic policy gradient model is trained. In the process of obtaining the path planning model, the network parameters of the Actor network module and the network parameters of the Critic network module are updated by introducing a delayed update strategy of the soft update mechanism.

4. A construction robot path planning method according to claim 2, characterized in that: Random noise is added to the Actor target network; wherein the random noise is specifically: ε~clip(N(0,σ),-c,c) Among them, y is the cumulative reward of target Q; r is the instantaneous reward; γ is the discount factor; is the output value of Critic's current network 1; is the output value of the current network 2 of Critic; s′ is the state at the moment; π φ′ (s′) is the action output by the current Critic network; ε is the added noise based on normal distribution; c is the upper boundary of the noise; -c is the lower boundary of the noise; clip(*) is the truncation function.

5. A construction robot path planning method according to claim 1, characterized in that: The state space of the current state of the construction robot includes the motion state, the collision state and the state of successfully reaching the target; wherein the state space of the current state of the construction robot is specifically: Among them, Continue is the motion state; Collision is the collision state; Goal is the state of successfully reaching the goal; d t is the distance data perceived by the construction robot’s laser radar at time t; d g is the distance between the construction robot and the target point.

6. A construction robot path planning method according to claim 1, characterized in that: In the process of executing the planned action to obtain the state at the next moment and obtaining the reward for calculating the planned action, the reward for the planned action is calculated using the reinforcement learning reward function; The reinforcement learning reward function is specifically: R r =100+λ1(d-d0)+λ2(t-t0) R c ＝-100 R g =α·v-β·w+γ·dis Among them, r(s t ,a t ) is the reward value function; R r Reward value for reaching the target point; R c is the collision state reward value; R g is the sparse reward; λ1 is the trajectory length adjustment factor; d is the target point distance; d0 is the moving trajectory length; λ2 is the adjustment factor for the planning time; t is the current time; t0 is the start time; α is the linear velocity weight; v is the linear velocity; β is the angular velocity weight; w is the angular velocity; γ is the laser radar c ranging weight; dis is the value calculated by the function specification after the laser radar ranging.

7. A construction robot path planning method according to claim 1, characterized in that: The sampling probability in the sampling priority experience playback mechanism is: Among them, P(i) is the probability that the i-th experience is selected; is the priority of the i-th experience; α is a hyperparameter; is the parameter for adjusting the weight during sampling; k is the total number of samples in the experience pool; In the sampling priority experience playback mechanism, a priority sampling method with importance sampling weights is introduced to correct the samples, and the importance sampling weights are applied to the experience update formula of the experience priority; The importance sampling weight is specifically: Among them, w i is the importance sampling weight of the i-th experience; N is the total number of samples in the experience replay buffer; β′ is a hyperparameter used to control the degree of importance sampling.

8. A construction robot path planning system, characterized in that: include: A modeling module, used to construct an improved dual-delay deep deterministic policy gradient model; wherein the improved dual-delay deep deterministic policy gradient model is a fusion model of the LSTM algorithm and the dual-delay deep deterministic policy gradient algorithm; A model training module is used to input the current state of the construction robot into the improved double-delay deep deterministic policy gradient model to obtain a planning action, execute the planning action to obtain the state at the next moment and obtain a reward for calculating the planning action; based on the state, reward and action at the current moment and the state at the next moment, experience data is formed through a priority experience replay mechanism and stored in an experience pool; based on the priority experience replay mechanism, samples are taken from the experience pool, and the improved double-delay deep deterministic policy gradient model is trained to obtain a path planning model; The path planning module is used to perform path planning using the path planning model to obtain a path planning result for the construction robot.

9. A construction robot path planning device, characterized in that: include: Memory for storing computer programs; A processor, used to implement the steps of the construction robot path planning method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the construction robot path planning method as described in any one of claims 1 to 7 are implemented.

Images