Path planning method and system based on deep reinforcement learning, and electronic equipment

Abstract

The invention relates to the technical field of path planning scheme design, in particular to a path planning method and system based on deep reinforcement learning and electronic equipment. The method comprises the following steps: acquiring a preset priority of each AGV robot located in a congestion area, controlling each AGV robot to start in sequence from high to low according to the preset priority and pass through the congestion area, recording related congestion coupling data corresponding to the congestion area, and further training a mobile strategy automatic planning model. And subsequently, taking the predicted actual moving path output by the model as an optimal path, obtaining the predicted parking duration and the predicted parking place output by the model, and adjusting the preset moving speed of each current AGV robot according to the predicted parking duration and the predicted parking place output by the model, so that each AGV robot does not need to be parked when passing through the predicted parking place, and the efficiency of the AGV robot is improved. According to the invention, effective and safe path planning of the unmanned vehicle is realized, and the safety, reliability and intelligent degree of path planning are improved to a great extent.

Description

Technical Field

[0001] The present invention relates to the technical field of path planning scheme design, and in particular to a path planning method and system based on deep reinforcement learning, and an electronic device. Background Art

[0002] The production line environment is complex and changeable. With the increase in the number of AGV transport robots in the production line, more and more AGV transport robots are in a mobile state, which may cause congestion and may cause transportation paralysis, greatly affecting production efficiency. There is even the possibility of collision, causing damage to production equipment and even threatening the safety of nearby production personnel. How to effectively and safely plan the unmanned vehicle path, while maximizing the running smoothness of each AGV transport robot during the transportation process and reducing the number of stops, is a problem that needs to be solved urgently in the existing technology.

[0003] Therefore, the prior art needs to be further developed. Summary of the invention

[0004] The purpose of the present invention is to overcome the above-mentioned technical deficiencies and provide a path planning method and system, and electronic device based on deep reinforcement learning to solve the problems existing in the prior art.

[0005] To achieve the above technical objectives, according to a first aspect of the present invention, the present invention provides a path planning method based on deep reinforcement learning, comprising: S100. In the MES system, a preset priority is assigned to each AGV transport robot according to a preset priority list, and the location data of each AGV transport robot is obtained by using a GPS sensor arranged on each AGV transport robot. It is determined whether there is a congested area according to the location data of each AGV transport robot. If so, each AGV transport robot located in the congested area is first controlled to stop, and the preset priority of each AGV transport robot located in the congested area is obtained. Each AGV transport robot is controlled to start and pass through the congested area in sequence from high to low according to the preset priority, and the preset priority, preset transport path, transport start time, actual moving path, parking time, and parking location of each AGV transport robot corresponding to the congested area are recorded to form a congested coupling data set, obtain a preset number of congested coupling data sets, and divide the collected congested coupling data sets into a training data set and a verification data set according to a first preset ratio; S200, using the collected training data set to train the mobile strategy automatic planning model, using the verification data set to optimize the mobile strategy automatic planning model, optimize the model parameters, and complete the mobile strategy automatic planning model training; S300. After the model training is completed, the preset transport path, preset priority, and transport start time of each AGV transport robot that will perform the transport task are obtained, and the obtained information is input into the mobile strategy automatic planning model. The predicted actual moving path output by the model is used as the optimal path, and the predicted parking time and predicted parking location output by the model are obtained. According to the predicted parking time and predicted parking location output by the model, the preset moving speed of each current AGV transport robot is adjusted, so that each AGV transport robot does not need to stop when passing the predicted parking location.

[0006] Specifically, assigning a preset priority to each AGV handling robot according to a preset priority list includes: Each AGV handling robot is divided into level one, level two, level three, and level four according to its priority from high to low.

[0007] Specifically, judging whether there is a congested area according to the position data of each AGV transport robot includes: Determine whether the distance between any two AGV robots is greater than or equal to a first preset threshold; If so, it is determined that there is a congested area; If not, it is determined that there is no congested area.

[0008] Specifically, the preset moving speed of each current AGV transport robot is adjusted according to the predicted parking time and predicted parking location output by the model, so that each AGV transport robot does not need to stop when passing the predicted parking location, including: Obtain the preset transport path of each AGV transport robot, and then calculate the length of the preset transport path of each AGV transport robot, and then calculate the first distance between the starting point of the preset transport path of each AGV transport robot and the predicted parking location, obtain the preset moving speed of each AGV transport robot, and calculate the first time for each AGV transport robot to arrive at its respective predicted parking location according to the preset moving speed and the first distance, calculate the sum of the first time and the predicted parking time, record it as the second time, calculate the quotient of the first distance and the second time, calculate the difference between the quotient and the first speed adjustment amount, record the difference as the adjusted speed, and control each AGV transport robot to move at the adjusted speed.

[0009] Specifically, the method of using the collected training data set to train the mobile strategy automatic planning model, using the verification data set to optimize the mobile strategy automatic planning model, optimizing the model parameters, and completing the mobile strategy automatic planning model training includes: The training data set is input into the preset network layer in batches for training, and the preset network layer includes a Transformer network layer. The Transformer network layer is used to predict the next congestion coupling data set based on the forward propagation of the current congestion coupling data set, and then obtain the predicted loss value; the loss value of the preset network layer is calculated and input into the optimizer for optimization to determine the direction in which the parameter gradient of the mobile strategy automatic planning model decreases fastest; the mobile strategy automatic planning model performs back propagation according to the loss value and the parameter gradient of the model to optimize the parameters of the mobile strategy automatic planning model.

[0010] Specifically, the method of using the collected training data set to train the mobile strategy automatic planning model, using the verification data set to optimize the mobile strategy automatic planning model, optimizing the model parameters, and completing the mobile strategy automatic planning model training also includes: After each training, the verification data set is input into the preset network layer of the previous training in batches for model parameter verification, and cyclic training is performed, with the total number of training rounds set to the first preset round; the loss value of the preset network layer is recorded, and it is determined whether the loss value meets the first preset condition, and whether to end the training and output the model parameters based on the judgment result.

[0011] Specifically, the determining whether the loss value satisfies the first preset condition, and determining whether to end the training and output the model parameters according to the determination result, includes: If the loss value meets the first preset condition, the training is terminated and the parameters of the current mobile strategy automatic planning model are output; if the loss value does not meet the first preset condition, the training is continued.

[0012] Specifically, the first preset condition includes: After the second preset round of training, the loss values ​​obtained in the next round of training are greater than or equal to the loss values ​​that occurred during the second preset round of training.

[0013] According to a second aspect of the present invention, there is provided a path planning system based on deep reinforcement learning, comprising: An acquisition module, including a GPS sensor on each AGV handling robot, is used to acquire the position data of each AGV handling robot; The control module is used to assign a preset priority to each AGV handling robot according to a preset priority list in the MES system; and is used to determine whether there is a congested area according to the position data of each AGV handling robot. If so, first control each AGV handling robot located in the congested area to stop, obtain the preset priority of each AGV handling robot located in the congested area, control each AGV handling robot to start and pass through the congested area in sequence from high to low according to the preset priority, record the preset priority, preset handling path, handling start time, actual moving path, parking duration, and parking location of each AGV handling robot corresponding to the congested area, form a congested coupling data set, obtain a preset number of congested coupling data sets, and divide the collected congested coupling data sets into training data sets according to a first preset ratio. The invention relates to a mobile strategy automatic planning model, a data set and a verification data set; the mobile strategy automatic planning model is trained by using the collected training data set, and the mobile strategy automatic planning model is optimized by using the verification data set, the model parameters are optimized, and the mobile strategy automatic planning model training is completed; the preset handling path, preset priority, and handling start time of each AGV handling robot that will perform the handling task are obtained after the model training is completed, and the obtained information is input into the mobile strategy automatic planning model, and the predicted actual moving path output by the model is used as the optimal path, and the predicted parking time and predicted parking location output by the model are obtained, and the preset moving speed of each current AGV handling robot is adjusted according to the predicted parking time and predicted parking location output by the model, so that each AGV handling robot does not need to stop when passing the predicted parking location.

[0014] According to a third aspect of the present invention, there is provided an electronic device, comprising: a memory; and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the above-mentioned path planning method based on deep reinforcement learning is implemented.

[0015] Beneficial effects: The present invention assigns a preset priority to each AGV transport robot according to a preset priority list in the MES system, obtains the position data of each AGV transport robot by using a GPS sensor arranged on each AGV transport robot, and determines whether there is a congested area according to the position data of each AGV transport robot. If so, firstly, each AGV transport robot located in the congested area is controlled to stop, the preset priority of each AGV transport robot located in the congested area is obtained, and each AGV transport robot is controlled to start and pass through the congested area in sequence from high to low according to the preset priority, and records the relevant congestion coupling data corresponding to the congested area, and then trains the mobile strategy automatic planning model, and subsequently uses the predicted actual moving path output by the model as the optimal path, and obtains the predicted parking time and predicted parking location output by the model, and adjusts the preset moving speed of each current AGV transport robot according to the predicted parking time and predicted parking location output by the model, so that each AGV transport robot does not need to stop when passing the predicted parking location, thereby realizing effective and safe planning of the unmanned vehicle path, and at the same time, it can maximize the running smoothness of each AGV transport robot in the transport process, avoid unnecessary parking, and greatly improve the safety, reliability and intelligence of path planning. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart of a path planning method based on deep reinforcement learning provided in a specific embodiment of the present invention; Figure 2 is a schematic diagram of the system composition of a path planning system based on deep reinforcement learning provided in a specific embodiment of the present invention; Figure 3 It is a structural diagram of a mobile strategy automatic planning model provided in a specific embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to enable those skilled in the art to better understand the technical solution of the present invention, the technical solution of the present invention is clearly and completely described below in conjunction with the accompanying drawings of the present invention. Based on the embodiments in this application, other similar embodiments obtained by ordinary technicians in this field without making creative work should all fall within the scope of protection of this application. In addition, the directional words mentioned in the following embodiments, such as "up", "down", "left", "right", etc., are only reference to the directions of the accompanying drawings. Therefore, the directional words used are used to illustrate rather than limit the invention.

[0018] The present invention will be further described below in conjunction with the accompanying drawings and preferred embodiments.

[0019] See also Figure 1 and Figure 3 The present invention provides a path planning method based on deep reinforcement learning, comprising: S100. In the MES system, a preset priority is assigned to each AGV transport robot according to a preset priority list, and the location data of each AGV transport robot is obtained by using a GPS sensor arranged on each AGV transport robot. It is determined whether there is a congested area based on the location data of each AGV transport robot. If so, each AGV transport robot located in the congested area is first controlled to stop, and the preset priority of each AGV transport robot located in the congested area is obtained. Each AGV transport robot is controlled to start and pass through the congested area in sequence from high to low according to the preset priority, and the preset priority, preset transport path, transport start time, actual moving path, parking duration, and parking location of each AGV transport robot corresponding to the congested area are recorded to form a congested coupling data set, obtain a preset number of congested coupling data sets, and divide the collected congested coupling data sets into a training data set and a verification data set according to a first preset ratio.

[0020] Specifically, assigning a preset priority to each AGV handling robot according to a preset priority list includes: Each AGV handling robot is divided into level one, level two, level three, and level four according to its priority from high to low.

[0021] It can be understood that each preset priority in the preset priority list can be increased or decreased according to the actual needs of the user of the present invention, which further improves the usability and scope of application of the present invention.

[0022] It can be understood that the control of each AGV transport robot to start in order from high to low according to the preset priority and pass through the congested area includes: Control each AGV transport robot to start in sequence from high to low according to the preset priority and pass through the congested area according to the preset automatic obstacle avoidance strategy.

[0023] Specifically, passing through the congested area according to the preset automatic obstacle avoidance strategy includes: When the AGV transport robot moves along the preset transport path, the depth camera installed on the AGV transport robot collects the depth image data in front, processes the collected data, and uses the processed depth camera data to judge the passable window according to the preset judgment rules. The travel rules are formulated based on the judgment results, and the AGV transport robot moves according to the preset transport path and the formulated travel rules.

[0024] Specifically, the processing of the collected data includes: First, we obtain the set of depth arrays {Ad1, Ad2, …, Adn} and the corresponding angles and distances {d1, d2, …, dn} in the depth image data, where 125° of the laser radar corresponds to 0° of the depth camera's viewing angle. We count the points in the image within the range of d1±dσ and store the coordinates in array Ad1. We obtain the maximum and minimum values ​​of the horizontal coordinate and the maximum and minimum values ​​of the vertical coordinate in array Ad1. We establish a target search box M1 based on the maximum and minimum values ​​of the horizontal coordinate and the maximum and minimum values ​​of the vertical coordinate in Ad1. We count the points in the image within the range of d2±dσ and store the coordinates in array Ad2, …, and so on. The depth image is divided into a series of target search boxes Mn.

[0025] Specifically, the processing of the depth camera data to determine the passable window according to the preset determination rules, and formulating the travel rules based on the determination results, include: Calculate the distance difference of adjacent target search boxes respectively, and determine whether the maximum distance difference is greater than or equal to the second preset threshold. If so, determine whether the width of the target search box with a larger distance between the two target search boxes that produce the maximum distance difference is greater than or equal to the third preset threshold, and at the same time determine whether the height of the target search box is greater than or equal to the fourth preset threshold. If so, the area indicated by the target search box is a passable window, then calculate the deviation angle between the center of the target search box and the driving direction of the AGV transport robot, and calculate the distance that the AGV transport robot needs to move left or right based on the distance and the deviation angle. The AGV transport robot moves this distance left or right so that the forward direction of the AGV transport robot and the center of the target search box are located in the same vertical plane, and then the AGV transport robot enters the passable window along the intersection of the vertical plane and the horizon.

[0026] Specifically, judging whether there is a congested area according to the position data of each AGV transport robot includes: Determine whether the distance between any two AGV robots is greater than or equal to a first preset threshold; If so, it is determined that there is a congested area; If not, it is determined that there is no congested area.

[0027] It should be noted here that in the MES system, the preset transport path of each AGV transport robot is stored.

[0028] Specifically, before step S100, the following steps are included: A first preset threshold, a second preset threshold, a third preset threshold, a fourth preset threshold, a preset number, a first preset ratio, a first preset round, and a second preset round are preset in the control module.

[0029] It can be understood that the first preset threshold, the second preset threshold, the third preset threshold, the fourth preset threshold, the first speed adjustment amount, the preset number, the first preset ratio, the first preset round, and the second preset round can be specifically set according to the actual needs of the user of the present invention. The present invention does not limit the specific values ​​of the above parameters, as long as it is applicable to the path planning method based on deep reinforcement learning proposed in the present invention.

[0030] Preferably, the present invention sets the first preset threshold to 1 meter, and the present invention sets the second preset threshold to 1.5 meters; the present invention sets the third preset threshold to be greater than the width of the AGV transport robot, and the present invention sets the fourth preset threshold to be greater than the height of the AGV transport robot. The settings of the third preset threshold and the fourth preset threshold are used to ensure that the AGV transport robot can pass through the passable channel; the present invention sets the first speed adjustment amount to 0.1 meters per second, the present invention sets the preset number to 500, the present invention sets the first preset ratio to 9:1, the present invention sets the first preset round to 20,000 times, and the present invention sets the second preset round to 200. The above settings can further improve the performance of the model and the efficiency of model training, and greatly improve the accuracy and reliability of the prediction of the mobile strategy automatic planning model of the present invention.

[0031] It is understandable that since the paths of multiple identical handling tasks are often related when performed simultaneously, the impact on the passing AGV handling robots is also similar. That is, after the congested area disappears, there is a certain coupling relationship between the preset priority, preset handling path, handling start time, actual moving path, parking time, and parking location of each AGV handling robot corresponding to the congested area. This coupling relationship can provide information for the subsequent path planning of the robot. The present invention designs an automatic planning model for mobile strategies based on this.

[0032] S200: Use the collected training data set to train the mobile strategy automatic planning model, use the verification data set to optimize the mobile strategy automatic planning model, optimize the model parameters, and complete the mobile strategy automatic planning model training.

[0033] Specifically, the method of using the collected training data set to train the mobile strategy automatic planning model, using the verification data set to optimize the mobile strategy automatic planning model, optimizing the model parameters, and completing the mobile strategy automatic planning model training includes: The training data set is input into the preset network layer in batches for training, and the preset network layer includes a Transformer network layer. The Transformer network layer is used to predict the next congestion coupling data set based on the forward propagation of the current congestion coupling data set, and then obtain the predicted loss value; the loss value of the preset network layer is calculated and input into the optimizer for optimization to determine the direction in which the parameter gradient of the mobile strategy automatic planning model decreases fastest; the mobile strategy automatic planning model performs back propagation according to the loss value and the parameter gradient of the model to optimize the parameters of the mobile strategy automatic planning model.

[0034] It can be understood that Transformer is a deep learning model based on the self-attention mechanism, which consists of a multi-head attention mechanism and a feedforward neural network. It can process sequence data and capture long-distance dependencies. The Transformer model does not contain traditional convolutional layers, but processes each element in the input sequence through an attention layer. The typical structure of Transformer includes an encoder and a decoder, wherein the encoder is composed of multiple identical layers stacked together, and the decoder adds a self-attention layer on the basis of the encoder to process sequence generation tasks. The present invention uses Transformer to supplement global context information, enhances the model's capabilities in feature extraction and global understanding, greatly improves the performance of the model, and greatly reduces the computational cost.

[0035] Specifically, the preset network layer also includes a ResNet network layer, and the ResNet network layer is used to predict the output result according to the collected mobile strategy automatic planning model.

[0036] It should be noted here that ResNet is a deep convolutional neural network that solves the gradient vanishing problem in deep network training by introducing residual connections. The core component of ResNet is the residual block, which usually contains two or three convolutional layers and a jump connection, which allows the gradient to bypass these convolutional layers directly. The network structure of ResNet can be very deep. These networks extract the preset priority, preset transport path, transport start time, actual moving path, parking time, and parking location of each AGV transport robot corresponding to the congested area by stacking multiple residual blocks, and predict the output results through global average pooling and fully connected layers. The prediction results of the model include predicted parking time, predicted parking location, and predicted actual moving path.

[0037] See also Figure 3The present invention combines the ResNet network layer and the Transformer network layer. ResNet is good at extracting the local features of the preset priority, preset transport path, transport start time, actual moving path, parking duration, and parking location of each AGV transport robot corresponding to the congested area, while Transformer can supplement the global context information. This combination can enhance the model's capabilities in feature extraction and global understanding, greatly improve the performance of the model, and greatly reduce the computational cost.

[0038] It can be understood that the ResNet network layer is used to predict the output result through the mobile strategy automatic planning model collected in the training data set and the verification data set. The specific steps are: (1) First, the training data set is put into the Backbones network layer of the model. Backbone refers to a series of convolutional layers that constitute the skeleton of the neural network. The main function of these layers is to extract the features of the input data. The Backbone network is usually composed of multiple convolutional layers, pooling layers, and activation functions. It can extract meaningful feature representations from the original data. Backbones is composed of multiple Backbones, and the core network layer in Backbone is the Resnet network layer. After the data set is input into the model, the mobile strategy automatic planning model will be input into Backbones for training; (2) When building Backbone, the Resnet network layer was added. After the training data set was input, Resnet began to train layer by layer. The training extracted the features in the mobile strategy automatic planning model and assisted Transformer in predicting the optimal path.

[0039] Specifically, the method of using the collected training data set to train the mobile strategy automatic planning model, using the verification data set to optimize the mobile strategy automatic planning model, optimizing the model parameters, and completing the mobile strategy automatic planning model training also includes: After each training, the verification data set is input into the preset network layer of the previous training in batches for model parameter verification, and cyclic training is performed, with the total number of training rounds set to the first preset round; the loss value of the preset network layer is recorded, and it is determined whether the loss value meets the first preset condition, and whether to end the training and output the model parameters based on the judgment result.

[0040] Specifically, the determining whether the loss value satisfies the first preset condition, and determining whether to end the training and output the model parameters according to the determination result, includes: If the loss value meets the first preset condition, the training is terminated and the parameters of the current mobile strategy automatic planning model are output; if the loss value does not meet the first preset condition, the training is continued.

[0041] Specifically, the first preset condition includes: After the second preset round of training, the loss values ​​obtained in the next round of training are greater than or equal to the loss values ​​that occurred during the second preset round of training.

[0042] Specifically, after each training, the verification data set is input into the preset network layer of the previous training in batches to verify the model parameters, and the cyclic training is performed, including the following methods: When the number of training times is not enough and the model needs to be optimized, the training can only be performed from the beginning. Therefore, the present invention develops a method for continuing the following training based on the previously trained model parameter file; (1) Provide custom parameters, which can be set according to user needs to continue training based on any training parameters; (2) Load the parameters of the custom model file; (3) Put the validation data set into the custom parameter network layer for validation, obtain the loss value and record the model parameters of the model corresponding to the minimum loss value during the training process; (4) Based on this set of model parameters, forward propagation training and back propagation are performed, and the optimizer optimizes the parameters to obtain a new set of data; (5) Put the verification data into the new data network layer for inference to obtain the loss value; (6) Perform cyclic training to obtain the best model and complete model optimization.

[0043] It is understandable that the training method adopted by the present invention is batch training, which means that each time the model parameters are updated, only a part of the samples in the validation data set is used, which is called a batch. The advantages of batch training are that it can reduce memory consumption, speed up training, increase randomness, and facilitate the generalization of the model.

[0044] It should be noted here that, in the present invention, batch_size=4, and the batch_size is the batch size. For example, when batch_size=4, it means that two mobile strategy automatic planning models are selected from the verification data set each time and put into the model for verification.

[0045] Specifically, the determining whether the loss value satisfies the first preset condition, and determining whether to end the training and output the model parameters according to the determination result, includes: If the loss value satisfies the first preset condition, the training is terminated and the parameters of the current mobile strategy automatic planning model are output; if the loss value does not satisfy the first preset condition, the training is continued; The first preset condition is that after the second preset round of training, the loss values ​​obtained in the next round of training are greater than or equal to the loss values ​​that occurred during the second preset round of training.

[0046] It can be understood that the present invention preferably sets the maximum round of cyclic training to 20,000 times, which can effectively ensure the accuracy of the model prediction of the present invention and record the loss value of the model training. The present invention sets the second preset round to 200, that is, when the loss value of the training no longer decreases within 200 rounds of training, the round with the smallest loss value in the 200 rounds of training is saved as the best verification round parameter to generate the best model file. The above setting makes the training rounds no longer rely on manual judgment, but uses the deep learning model for automatic training, which improves the accuracy of the model prediction, and automatically ends the training when the accuracy meets the requirements, which greatly saves the training time, and can also effectively prevent overfitting, which greatly improves the intelligence level of the present invention and the efficiency of model training.

[0047] Furthermore, the specific process of training the mobile strategy automatic planning model of the present invention is as follows: (1) Obtain the potential vector mean mu and logarithmic variance logvar during the forward propagation process of the mobile strategy automatic planning model; (2) Calculate a reasonable loss value representing the training effect: Calculate the divergence: k1=-0.5*(1+logvar-mu^2-(e^logvar)); Calculate the loss between the mobile strategy automatic planning model and the predicted value in the validation data set: loss=L1+k1*k1_weight; Among them, loss is the loss value, k1 is the divergence, k1_weight is the proportion of k1, the value is 10, L1 is the absolute difference between the predicted value and the true value, and the calculation formula of L1 is as follows: ; Among them, A is the L1 loss in pytorch, which calculates the average or sum of the absolute differences between two tensors; the shape of A is N*C*H*W, which indicates that it is a four-dimensional tensor, N represents the number of samples (Batch Size), indicating that there are N data points, C represents the number of channels (Channels), H and W represent the height and width respectively, which are usually the two-dimensional spatial dimensions of the input features; P represents the padding value, which is a Boolean type, and the shape of P is c*h*w*1. The dimensions of P are the same as those of A in terms of channel, height, and width, but there is no sample number N, which means that P is a mask or padding information for a feature map of each sample, and the Boolean value of P may be used to mark whether the elements in A are filled (True) or valid (False).

[0048] (3) The loss value when using the validation data set to validate the current model network; (4) If the validation loss of this round is less than that of the previous round, the best model parameters are replaced with the training model of this round; (5) Put the rounds and loss values ​​into the dictionary one by one; (6) Compare the loss value of the current round with the loss values ​​of the 200 rounds before this round (parameters are provided and can be set according to actual needs, the default is 200), that is, traverse the loss values ​​from epoch-200 to epoch-1 (epoch is the training round, and the comparison is performed when the epoch is greater than 200). If the loss values ​​of the current round of epoch are all greater than the loss values ​​from (epoch-200) to (epoch-1), stop training.

[0049] (7) Save the best model parameters in the cyclic validation as a ckpt model file.

[0050] S300. After the model training is completed, the preset transport path, preset priority, and transport start time of each AGV transport robot that will perform the transport task are obtained, and the obtained information is input into the mobile strategy automatic planning model. The predicted actual moving path output by the model is used as the optimal path, and the predicted parking time and predicted parking location output by the model are obtained. According to the predicted parking time and predicted parking location output by the model, the preset moving speed of each current AGV transport robot is adjusted, so that each AGV transport robot does not need to stop when passing the predicted parking location.

[0051] Specifically, the preset moving speed of each current AGV transport robot is adjusted according to the predicted parking time and predicted parking location output by the model, so that each AGV transport robot does not need to stop when passing the predicted parking location, including: Obtain the preset transport path of each AGV transport robot, and then calculate the length of the preset transport path of each AGV transport robot, and then calculate the first distance between the starting point of the preset transport path of each AGV transport robot and the predicted parking location, obtain the preset moving speed of each AGV transport robot, and calculate the first time for each AGV transport robot to arrive at its respective predicted parking location according to the preset moving speed and the first distance, calculate the sum of the first time and the predicted parking time, record it as the second time, calculate the quotient of the first distance and the second time, calculate the difference between the quotient and the first speed adjustment amount, record the difference as the adjusted speed, and control each AGV transport robot to move at the adjusted speed.

[0052] It is understandable that by recording the difference as the adjusted speed and controlling each AGV transport robot to move according to the adjusted speed, it can be achieved that when each AGV transport robot passes through the predicted parking location of each AGV transport robot, it does not need to stop. The possibility of collision between AGV robots is further prevented, and the safety and reliability of the path planning of the present invention are further improved.

[0053] It should be noted here that the present invention assigns a preset priority to each AGV transport robot according to a preset priority list in the MES system, uses a GPS sensor arranged on each AGV transport robot to obtain the position data of each AGV transport robot, and determines whether there is a congested area according to the position data of each AGV transport robot. If so, first control each AGV transport robot located in the congested area to stop, obtain the preset priority of each AGV transport robot located in the congested area, control each AGV transport robot to start and pass through the congested area in sequence from high to low according to the preset priority, record the relevant congestion coupling data corresponding to the congested area, and then train the mobile strategy automatic planning model, and then use the predicted actual moving path output by the model as the optimal path, and obtain the predicted parking time and predicted parking location output by the model, and adjust the preset moving speed of each current AGV transport robot according to the predicted parking time and predicted parking location output by the model, so that each AGV transport robot does not need to stop when passing the predicted parking location, thereby realizing effective and safe planning of the unmanned vehicle path, and at the same time, it can maximize the running smoothness of each AGV transport robot during the transportation process, avoid unnecessary parking, and greatly improve the safety, reliability and intelligence of path planning.

[0054] See also Figure 2 The present invention provides another embodiment, which provides a path planning system based on deep reinforcement learning, and the path planning system based on deep reinforcement learning includes: The acquisition module 100 includes a GPS sensor on each AGV transport robot, which is used to acquire the position data of each AGV transport robot; The control module 200 is used to assign a preset priority to each AGV transport robot in the MES system according to a preset priority list; and is used to determine whether there is a congested area based on the position data of each AGV transport robot. If so, first control each AGV transport robot located in the congested area to stop, obtain the preset priority of each AGV transport robot located in the congested area, control each AGV transport robot to start and pass through the congested area in sequence from high to low according to the preset priority, record the preset priority, preset transport path, transport start time, actual moving path, parking duration, and parking location of each AGV transport robot corresponding to the congested area, form a congested coupling data set, obtain a preset number of congested coupling data sets, and divide the collected congested coupling data sets into training sets according to a first preset ratio. The invention relates to a data set and a verification data set; used to train the mobile strategy automatic planning model using the collected training data set, and to optimize the mobile strategy automatic planning model using the verification data set, optimize the model parameters, and complete the mobile strategy automatic planning model training; used to obtain the preset handling path, preset priority, and handling start time of each AGV handling robot that will perform the handling task after the model training is completed, and input the obtained information into the mobile strategy automatic planning model, and use the predicted actual moving path output by the model as the optimal path, and obtain the predicted parking time and predicted parking location output by the model, and adjust the preset moving speed of each current AGV handling robot according to the predicted parking time and predicted parking location output by the model, so that each AGV handling robot does not need to stop when passing the predicted parking location.

[0055] It should be noted here that the present invention assigns a preset priority to each AGV transport robot according to a preset priority list in the MES system, uses a GPS sensor arranged on each AGV transport robot to obtain the position data of each AGV transport robot, and determines whether there is a congested area according to the position data of each AGV transport robot. If so, first control each AGV transport robot located in the congested area to stop, obtain the preset priority of each AGV transport robot located in the congested area, control each AGV transport robot to start and pass through the congested area in sequence from high to low according to the preset priority, record the relevant congestion coupling data corresponding to the congested area, and then train the mobile strategy automatic planning model, and then use the predicted actual moving path output by the model as the optimal path, and obtain the predicted parking time and predicted parking location output by the model, and adjust the preset moving speed of each current AGV transport robot according to the predicted parking time and predicted parking location output by the model, so that each AGV transport robot does not need to stop when passing the predicted parking location, thereby realizing effective and safe planning of the unmanned vehicle path, and at the same time, it can maximize the running smoothness of each AGV transport robot during the transportation process, avoid unnecessary parking, and greatly improve the safety, reliability and intelligence of path planning.

[0056] In a preferred embodiment, the present application further provides an electronic device, the electronic device comprising: A memory; and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the path planning method based on deep reinforcement learning is implemented. The computer device can be broadly a server, a terminal, or any other electronic device with necessary computing and / or processing capabilities. In one embodiment, the computer device may include a processor, a memory, a network interface, a communication interface, etc. connected via a system bus. The processor of the computer device can be used to provide necessary computing, processing and / or control capabilities. The memory of the computer device may include a non-volatile storage medium and an internal memory. An operating system, a computer program, etc. may be stored in or on the non-volatile storage medium. The internal memory may provide an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface and the communication interface of the computer device can be used to connect and communicate with external devices through a network. When the computer program is executed by the processor, the steps of the method of the present invention are performed.

[0057] The present invention may be implemented as a computer-readable storage medium having a computer program stored thereon, which causes the steps of the method of an embodiment of the present invention to be executed when executed by a processor. In one embodiment, the computer program is distributed on a plurality of computer devices or processors coupled to a network so that the computer program is stored, accessed and executed in a distributed manner by one or more computer devices or processors. A single method step / operation, or two or more method steps / operations, may be performed by a single computer device or processor or by two or more computer devices or processors. One or more method steps / operations may be performed by one or more computer devices or processors, and one or more other method steps / operations may be performed by one or more other computer devices or processors. One or more computer devices or processors may perform a single method step / operation, or perform two or more method steps / operations.

[0058] It will be appreciated by a person skilled in the art that the method steps of the present invention may be performed by instructing related hardware such as a computer device or a processor through a computer program, and the computer program may be stored in a non-temporary computer-readable storage medium, which causes the steps of the present invention to be performed when the computer program is executed. Depending on the circumstances, any reference to memory, storage, database, or other media herein may include non-volatile and / or volatile memory. Examples of non-volatile memory include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk, etc. Examples of volatile memory include random access memory (RAM), external cache memory, etc.

[0059] It can be understood that the present invention assigns a preset priority to each AGV transport robot according to a preset priority list in the MES system, uses a GPS sensor arranged on each AGV transport robot to obtain the position data of each AGV transport robot, and determines whether there is a congested area according to the position data of each AGV transport robot. If so, first control each AGV transport robot located in the congested area to stop, obtain the preset priority of each AGV transport robot located in the congested area, control each AGV transport robot to start and pass through the congested area in sequence from high to low according to the preset priority, record the relevant congestion coupling data corresponding to the congested area, and then train the mobile strategy automatic planning model, and then use the predicted actual moving path output by the model as the optimal path, and obtain the predicted parking time and predicted parking location output by the model, and adjust the preset moving speed of each current AGV transport robot according to the predicted parking time and predicted parking location output by the model, so that each AGV transport robot does not need to stop when passing the predicted parking location, thereby realizing effective and safe planning of the unmanned vehicle path, and at the same time, it can maximize the running smoothness of each AGV transport robot during the transportation process, avoid unnecessary parking, and greatly improve the safety, reliability and intelligence of path planning.

[0060] The various technical features described above can be combined arbitrarily. Although all possible combinations of these technical features are not described, any combination of these technical features should be considered to be covered by this specification as long as there is no contradiction in such combination.

[0061] The specific implementation of the present invention described above does not constitute a limitation on the protection scope of the present invention. Any other corresponding changes and modifications made based on the technical concept of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A path planning method based on deep reinforcement learning, characterized in that: The method comprises: S100. In the MES system, a preset priority is assigned to each AGV transport robot according to a preset priority list, and the location data of each AGV transport robot is obtained by using a GPS sensor arranged on each AGV transport robot. It is determined whether there is a congested area according to the location data of each AGV transport robot. If so, each AGV transport robot located in the congested area is first controlled to stop, and the preset priority of each AGV transport robot located in the congested area is obtained. Each AGV transport robot is controlled to start and pass through the congested area in sequence from high to low according to the preset priority, and the preset priority, preset transport path, transport start time, actual moving path, parking time, and parking location of each AGV transport robot corresponding to the congested area are recorded to form a congested coupling data set, obtain a preset number of congested coupling data sets, and divide the collected congested coupling data sets into a training data set and a verification data set according to a first preset ratio; S200, using the collected training data set to train the mobile strategy automatic planning model, using the verification data set to optimize the mobile strategy automatic planning model, optimize the model parameters, and complete the mobile strategy automatic planning model training; S300. After the model training is completed, the preset transport path, preset priority, and transport start time of each AGV transport robot that will perform the transport task are obtained, and the obtained information is input into the mobile strategy automatic planning model. The predicted actual moving path output by the model is used as the optimal path, and the predicted parking time and predicted parking location output by the model are obtained. According to the predicted parking time and predicted parking location output by the model, the preset moving speed of each current AGV transport robot is adjusted, so that each AGV transport robot does not need to stop when passing the predicted parking location.

2. The path planning method based on deep reinforcement learning according to claim 1, characterized in that: The step of assigning a preset priority to each AGV handling robot according to a preset priority list includes: Each AGV handling robot is divided into level one, level two, level three, and level four according to its priority from high to low.

3. The path planning method based on deep reinforcement learning according to claim 1, characterized in that: The determining whether there is a congested area according to the position data of each AGV transport robot includes: Determine whether the distance between any two AGV robots is greater than or equal to a first preset threshold; If so, it is determined that there is a congested area; If not, it is determined that there is no congested area.

4. The path planning method based on deep reinforcement learning according to claim 1, characterized in that: The method of adjusting the preset moving speed of each current AGV transport robot according to the predicted parking time and the predicted parking location output by the model, so that each AGV transport robot does not need to stop when passing the predicted parking location, includes: Obtain the preset transport path of each AGV transport robot, and then calculate the length of the preset transport path of each AGV transport robot, and then calculate the first distance between the starting point of the preset transport path of each AGV transport robot and the predicted parking location, obtain the preset moving speed of each AGV transport robot, and calculate the first time for each AGV transport robot to arrive at its respective predicted parking location according to the preset moving speed and the first distance, calculate the sum of the first time and the predicted parking time, record it as the second time, calculate the quotient of the first distance and the second time, calculate the difference between the quotient and the first speed adjustment amount, record the difference as the adjusted speed, and control each AGV transport robot to move at the adjusted speed.

5. The path planning method based on deep reinforcement learning according to claim 1, characterized in that: The method of using the collected training data set to train the mobile strategy automatic planning model, using the verification data set to optimize the mobile strategy automatic planning model, optimizing the model parameters, and completing the mobile strategy automatic planning model training includes: The training data set is input into the preset network layer in batches for training, and the preset network layer includes a Transformer network layer. The Transformer network layer is used to predict the next congestion coupling data set based on the forward propagation of the current congestion coupling data set, and then obtain the predicted loss value; the loss value of the preset network layer is calculated and input into the optimizer for optimization to determine the direction in which the parameter gradient of the mobile strategy automatic planning model decreases fastest; the mobile strategy automatic planning model performs back propagation according to the loss value and the parameter gradient of the model to optimize the parameters of the mobile strategy automatic planning model.

6. The path planning method based on deep reinforcement learning according to claim 5, characterized in that: The method of using the collected training data set to train the mobile strategy automatic planning model, using the verification data set to optimize the mobile strategy automatic planning model, optimizing the model parameters, and completing the mobile strategy automatic planning model training also includes: After each training, the verification data set is input into the preset network layer of the previous training in batches for model parameter verification, and cyclic training is performed, with the total number of training rounds set to the first preset round; the loss value of the preset network layer is recorded, and it is determined whether the loss value meets the first preset condition, and whether to end the training and output the model parameters based on the judgment result.

7. The path planning method based on deep reinforcement learning according to claim 6, characterized in that: The determining whether the loss value satisfies the first preset condition, and determining whether to end the training and output the model parameters according to the determination result, includes: If the loss value meets the first preset condition, the training is terminated and the parameters of the current mobile strategy automatic planning model are output; if the loss value does not meet the first preset condition, the training is continued.

8. The path planning method based on deep reinforcement learning according to claim 6, characterized in that: The first preset condition includes: After the second preset round of training, the loss values ​​obtained in the next round of training are greater than or equal to the loss values ​​that occurred during the second preset round of training.

9. A path planning system based on deep reinforcement learning, characterized in that: include: An acquisition module, including a GPS sensor on each AGV handling robot, is used to acquire the position data of each AGV handling robot; A control module is used to assign a preset priority to each AGV transport robot in the MES system according to a preset priority list; and is used to determine whether there is a congested area according to the position data of each AGV transport robot. If so, first control each AGV transport robot located in the congested area to stop, obtain the preset priority of each AGV transport robot located in the congested area, control each AGV transport robot to start and pass through the congested area in sequence from high to low according to the preset priority, record the preset priority, preset transport path, transport start time, actual moving path, parking time, and parking location of each AGV transport robot corresponding to the congested area, form a congested coupling data set, obtain a preset number of congested coupling data sets, and divide the collected congested coupling data sets into a training data set and a verification data set according to a first preset ratio; It is used to train the mobile strategy automatic planning model using the collected training data set, and to optimize the mobile strategy automatic planning model using the verification data set, optimize the model parameters, and complete the mobile strategy automatic planning model training; it is used to obtain the preset handling path, preset priority, and handling start time of each AGV handling robot that will perform the handling task after the model training is completed, and input the obtained information into the mobile strategy automatic planning model, use the predicted actual moving path output by the model as the optimal path, and obtain the predicted parking time and predicted parking location output by the model, and adjust the preset moving speed of each current AGV handling robot according to the predicted parking time and predicted parking location output by the model, so that each AGV handling robot does not need to stop when passing the predicted parking location.

10. An electronic device, characterized in that: include: Memory; and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the path planning method based on deep reinforcement learning according to any one of claims 1 to 8 is implemented.

Images