A path exploration parameter optimization system and method based on deep reinforcement learning

Abstract

The application relates to the technical field of path planning, and provides a path exploration parameter optimization system based on deep reinforcement learning, which comprises the following modules: a variable parameter path planning module, which is used for node exploration based on a deep reinforcement learning network, collision detection of sub-nodes in a sub-node set, calculation of the generation value of all the sub-nodes, and finally generation of a loading parking-in path through a Reeds-Shepp curve; an environment state space modeling module, which is used for regional division of surrounding obstacles of a current node and environment state space modeling; and a deep learning parameter optimization module, which is used for constructing a deep learning network to calculate an optimal step length and an optimal steering angle, constructing a reward function to optimize the deep learning network, and simultaneously executing a training process of the deep learning network. The exploration parameters in the path planning method are optimized through the deep reinforcement learning network, and a state space considering obstacle distribution is established by analyzing obstacle information in the environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of path planning, in particular to a path exploration parameter optimization system and method based on deep reinforcement learning. BACKGROUND

[0002] The application of automatic driving technology for mine cars will significantly improve the efficiency of mining and transportation of mine resources, and enhance safety. In order to realize the unmanned driving task of mine cars, an efficient path planning system is essential. The system should be able to generate safe and efficient driving paths according to the work requirements combined with the mine environment.

[0003] The mine operation area is usually complex in terrain, rugged in road, and has a large number of obstacles and irregular terrain, resulting in a relatively limited feasible region, which brings many challenges to path planning. In addition, the orientation angle of the mine car must be strictly controlled during loading operation to ensure the safety and efficiency of the loading process, so the path planning needs to consider the final state pose constraint, which further increases the difficulty of problem solving. In the complex and variable mine operation scene, the existing path planning method still faces the problems of low search efficiency and poor path quality.

[0004] The method based on graph search, such as hybrid A-star algorithm, although it can generate a path with accurate final state pose, but its calculation speed is slow and the planning efficiency is low. The sampling-based method, such as random search tree algorithm, generates a path that often does not meet the motion characteristics of unmanned mine cars, making it difficult to be directly applied. The core reason for the insufficient search efficiency of the above search algorithms is that the path exploration parameters are fixed and difficult to adapt to complex and variable mine operation scenes. In addition, although the exploration parameters can be changed by setting artificial rules, the distribution of mine gravel is irregular, and the set rules are difficult to accurately adapt to each scene, and there are a large number of parameters that need to be debugged, with high maintenance cost.

[0005] Based on the limitations of the above methods, there is an urgent need for a new path planning algorithm that can not only improve the planning speed but also ensure good generalization ability to adapt to complex mine environments. SUMMARY

[0006] In view of the above problems, the purpose of the present application is to provide a path exploration parameter optimization system and method based on deep reinforcement learning, which optimizes the exploration parameters in the path planning method through a deep reinforcement learning network, thereby realizing the adaptive optimization of exploration parameters in complex scenes, reducing the parameter debugging and maintenance cost, and improving the search efficiency and path quality. By analyzing the obstacle information in the environment, a state space considering obstacle distribution is established, thereby improving the understanding of the environment by deep reinforcement learning, improving the generalization in complex and variable scenes, and reducing the dependence on the map and overfitting.

[0007] The above invention purpose of the present application is realized by the following technical solutions:

[0008] A path exploration parameter optimization system based on deep reinforcement learning, comprising:

[0009] A variable parameter path planning module, configured to generate an optimal step length and an optimal steering angle based on a deep reinforcement learning network for a current node and environment information, and to construct a fixed steering angle set, and to generate a child node set in a manner of combining the optimal step length and the fixed steering angle set, to generate a steering angle optimization child node in a manner of combining the optimal step length and the optimal steering angle and add it to the child node set, to perform node exploration, to perform collision detection on the child nodes in the child node set, and to calculate the generation value of all the child nodes, and finally to generate a loading parking-in path through a Reeds-Shepp curve;

[0010] An environment state space modeling module, configured to divide the surrounding obstacles of the current node into regions, and to model the environment state space;

[0011] A deep learning parameter optimization module, configured to construct a deep learning network to calculate the optimal step length and the optimal steering angle, and to construct a reward function to optimize the deep learning network, while performing a training process of the deep learning network.

[0012] Further, in the variable parameter path planning module, the optimal step length and the optimal steering angle are generated based on a deep reinforcement learning network for a current node and environment information, and a fixed steering angle set is constructed, and node exploration is performed in a manner of generating a child node set by combining the optimal step length and the fixed steering angle set, generating a steering angle optimization child node by combining the optimal step length and the optimal steering angle and adding it to the child node set, specifically:

[0013] Suppose the current node is , is a position coordinate, is an orientation angle, and the following node exploration process is established according to the motion characteristics of the unmanned mine truck:

[0014]

[0015] wherein is the next child node explored from the current node, is a position coordinate, is an orientation angle, represents an extended direction of the current node including backward or forward, and represent the steering angle and the step length of node expansion, respectively. The front and rear axle distance of the unmanned mine car;

[0016] The optimal step and the optimal steering angle for the current node and the environment information are generated by the deep reinforcement learning network, and the fixed steering angle set is constructed in a uniform sampling manner For the fixed steering angle The calculation method is as follows:

[0017]

[0018] Wherein, The maximum steering angle that can be performed by the mine car, The number of constructed steering angles;

[0019] The node exploration is performed in two steps including step optimization and steering angle optimization. In the first step, the optimal step And all sampled fixed steering angles in the fixed steering angle set are substituted into formula (1), thereby generating the sub-node set of fixed steering angle exploration, the number of sub-nodes in the sub-node set is the same as the number of sampled angles in the fixed steering angle set , which is In the second step, the optimal step And the optimal steering angle are substituted into formula (1) to generate the steering angle optimization sub-node and added to the sub-node set .

[0020] Further, in the variable parameter path planning module, collision detection is performed on the sub-nodes in the sub-node set, and the generation value of all the sub-nodes is calculated, specifically:

[0021] Collision detection is performed on all sub-nodes in the sub-node set , the mine car is covered by two envelope circles, and sampling is performed on the path from the current node to the explored sub-node, to determine whether the distance of the obstacle grid is less than the envelope circle radius. If it is less than, it is considered that the sub-node is not feasible, and is deleted from the sub-node set ;

[0022] The generation value of all explored sub-nodes is calculated by using , wherein represents the actual consumption cost of the starting point to the explored sub-node mine car in the moving process, represents the predicted cost of the explored sub-node to the destination, is the weight of the predicted cost;

[0023] wherein the actual consumption cost is:

[0024]

[0025] In the above formula, is the cost basis of the current node contains five indicators, is the distance from the current node to the child node in the iterative search, is the backtracking cost, is the change cost, is the turning cost, turning change cost, is the weight coefficient.

[0026] Further, in the variable parameter path planning module, the loading parking path is generated by the Reeds-Shepp curve, specifically:

[0027] When the distance between the current node and the end point is less than the threshold , a plurality of candidate loading parking path curves from the current node to the end point are generated by the Reeds-Shepp curve, the node cost on the curve is calculated by formula (3), and the cost is sorted according to the cost, the path with the minimum cost is selected and reverse lookup is performed to obtain the global path;

[0028] If all the candidate loading parking path curves collide, the node exploration step is continued.

[0029] Further, in the environment state space modeling module, the surrounding obstacles of the current node are regionally divided, specifically:

[0030] The space around the current node is divided into 8 regions according to the angle, and represents the minimum distance between the obstacles in the i-th region and the mine car.

[0031] Further, in the environment state space modeling module, the environment state space modeling is performed, specifically:

[0032] The state space is designed as follows:

[0033] ​

[0034] in, represents the coordinates of the current node, Indicates the distance from the starting point to the current node. Indicates that the starting point is at the current node as the origin and the heading angle direction is The relative angular orientation of the axis in the coordinate system, Indicates the distance of the end point relative to the current node. Indicates the relative angle of the end point in the coordinate system with the current node as the origin and the heading angle as the x-axis. Indicates that the terminal loading posture is with the current node as the origin and the heading angle direction as The direction of the axis in the coordinate system, Indicates the number of obstacles within a given range of the current node. Indicates the The minimum distance between obstacles and minecarts in a region.

[0035] Furthermore, in the deep learning parameter optimization module, a deep learning network is constructed to calculate the optimal step length and the optimal steering angle, specifically:

[0036] The DQN algorithm is used to train the deep learning network, and the action space is expanded by the optimal step size to be selected Combination of the optimal steering angle The action space is composed of all possible combination of;

[0037] The DQN algorithm uses two networks with exactly the same structure but different parameters for training. Used to calculate strategy selection Value and perform Iterative update of values; target network Used to calculate the next state in the time difference target TDTarget Value, the loss function Loss of the DQN algorithm is designed as follows:

[0038]

[0039] in, A set of state transition data obtained during the training process, including the current state , Current Action , the reward obtained after taking this action And the state obtained by interacting with the environment after taking this action and actions , is an adjustable attenuation coefficient;

[0040] Target network And training network Both are built with three layers of fully connected layers, with 32 neurons each. The output of the first two layers of fully connected layers is input into the activation function, and then output to the next fully connected layer. The activation function is set to PReLu. The last layer of fully connected layer directly outputs the Q value of each action, including the steering angle and the step length. The steering angle and step length with the highest Q value are finally selected as the final optimized exploration parameters including the optimal step length and the optimal steering angle.

[0041] Further, in the deep learning parameter optimization module, a reward function is constructed to optimize the deep learning network, specifically:

[0042] The reward function involves the destination-oriented reward , obstacle avoidance reward , exploration cost , and smoothness reward :

[0043] The destination-oriented reward is defined as follows:

[0044]

[0045] wherein, is an adjustable weight, is the Euclidean distance from the current node to the end point in this iteration round, is the Euclidean distance from the steering angle optimization sub-node to the end point , is a fixed reward for reaching the destination when the Reeds-Shepp curve in this round successfully connects to the destination, indicating that the mine car has reached the destination.

[0046] The obstacle avoidance reward is defined as follows:

[0047]

[0048] wherein, represents the obstacle avoidance reward in the i-th region, and are adjustable weight coefficients, and the distance threshold is designed as , which is considered as collision and returns a large penalty constant when When the risk is considered, a smaller penalty function is returned, when the safety is considered, no penalty is returned, and the overall obstacle avoidance reward satisfies the following formula:

[0049]

[0050] The exploration cost is defined as follows:

[0051]

[0052] wherein, the fixed penalty cost constant set for each step, guiding the mine car to approach the destination more quickly, preventing meaningless exploration, and the cost is set to be negative;

[0053] The smoothness reward is defined as follows:

[0054]

[0055] wherein, the current search iteration round in which the current node is generated corresponding to the steering angle, corresponding to the best steering angle generated by the deep reinforcement learning network in the current search iteration round, are respectively adjustable coefficients;

[0056] The reward function is finally:

[0057] .

[0058] Further, in the deep learning parameter optimization module, the training process of the deep learning network is performed, specifically:

[0059] First, a suitable starting point and ending point are randomly selected on the map according to actual production data, and path planning is performed. In the planning, the path planning parameters are optimized through reinforcement learning, thereby forming a plurality of groups of state transition sampling data, and the data is added to the replay buffer. In the training process, a batch of data is randomly selected from the replay buffer, and the parameters of the estimation network are updated according to the loss function . After a certain number of iterations, the parameters of the training network are copied to the target network . In this way, a learning process is completed. ​​​​​​​

[0060] A deep reinforcement learning-based path exploration parameter optimization method performed by the deep reinforcement learning-based path exploration parameter optimization system as described above, comprising the following steps:

[0061] S1: generating an optimal step length and an optimal steering angle based on a deep reinforcement learning network for a current node and environment information, constructing a fixed steering angle set, and performing node exploration in the following manner: generating a sub-node set by combining the optimal step length and the fixed steering angle set, generating a steering angle optimization sub-node by combining the optimal step length and the optimal steering angle, and adding it to the sub-node set;

[0062] S2: performing collision detection on the sub-nodes in the sub-node set and calculating the generation value of all the sub-nodes;

[0063] S3: obtaining the sub-node with the lowest generation value in each round of iterative search as the next sub-node of the current node selected finally;

[0064] S4: when the distance from the current node to the terminal point is less than a set threshold, generating a loading parking path through a Reeds-Shepp curve and generating a planned path through node backtracking.

[0065] Compared with the prior art, the present application has at least one of the following beneficial effects:

[0066] (1) Improving search efficiency and path quality

[0067] The mine operation has irregularly distributed gravel and mountains, which causes the exploration difficulty and feasible region of different areas to change, and the existing path planning method has fixed exploration parameters, so there is a problem of mismatching search parameters and causing search efficiency to decline. The present application optimizes the path exploration parameters through a deep reinforcement learning network, automatically optimizes and adjusts the exploration parameters according to the environment information, and generates nodes with lower cost, thereby improving the search efficiency and path quality.

[0068] (2) Adapt to complex and variable mine operation scenarios

[0069] Mining operation scenarios change dynamically as operations progress. While existing planning algorithms have designed exploration rules, they do not consider the impact of environmental characteristics such as gravel on planning and exploration, making them difficult to apply to a variety of mining scenarios. The present invention establishes a state space that takes into account the distribution of obstacles and inputs it into a deep learning network to optimize exploration parameters, thereby establishing a mapping relationship from gravel obstacles to exploration parameters, which is more suitable for complex and changeable mining operation scenarios. In addition, this algorithm also has potential application value for path planning in general unstructured scenarios, and the exploration parameter optimization ideas can also be borrowed from different search algorithms.

[0070] (3) Reduce parameter debugging and maintenance costs

[0071] In order to cope with different scenarios, existing path planning methods need to set different path exploration parameters and establish parameter tables for maintenance, which makes parameter tuning difficult and the maintenance cost high. In particular, the mining operation scenarios are changeable, which further increases the difficulty of parameter debugging and maintenance. The present invention realizes the adaptive optimization of path exploration parameters by constructing a deep learning network, which does not require manual parameter adjustment and reduces debugging costs. On the other hand, the deep learning network takes into account the characteristics of obstacles, so it is more generalizable to scenarios, and when changing scenarios, only offline training on the new map is required, without the need for manual maintenance of a large parameter table, which reduces maintenance costs.

[0072] (4) Experimental verification

[0073] In experiments, the proposed solution was compared with existing methods in path planning performance in a mining operation scenario. The results showed that the proposed solution achieved better performance in both solution efficiency and path planning length, with higher search efficiency and path quality.

[0074] The specific data are as follows:

[0075] On the grid map of the actual mine shovel loading platform ( ) were tested using start and end points from actual production data and compared with the existing Hybrid A-Star algorithm for unmanned mining truck path planning. The proposed algorithm achieved a solution time of 0.43 seconds, an 80% reduction compared to Hybrid A-Star, and a path length of 437 meters, an 11% reduction compared to Hybrid A-Star, demonstrating significant superiority.

[0076] In summary, the present invention proves through theoretical analysis and experimental data that the path exploration parameter optimization method based on deep reinforcement learning significantly improves the search efficiency and path quality, reduces parameter debugging and maintenance costs, is more suitable for changeable mining environments, and has important practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 The overall structure diagram of the path exploration parameter optimization system based on deep reinforcement learning of the present application is shown in the figure;

[0078] Figure 2 The node exploration rule flowchart of the present application with variable parameters is shown in the figure;

[0079] Figure 3 The schematic diagram of the present application for dividing the obstacle region around the mine card is shown in the figure;

[0080] Figure 4 The DQN network training flowchart of the present application is shown in the figure;

[0081] Figure 5 The overall flowchart of the path exploration parameter optimization method based on deep reinforcement learning of the present application is shown in the figure;

[0082] Figure 6 The path exploration parameter optimization algorithm flowchart based on deep reinforcement learning of the present application is shown in the figure. DETAILED DESCRIPTION

[0083] To make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0084] Those skilled in the art can understand that, unless specifically stated, the singular forms "a", "an" and "the" used herein also include the plural forms. It should be further understood that the phrase "comprising" used in the specification of the present application means that the features, integers, steps, operations, elements and / or components exist, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0085] In order to realize efficient path planning of unmanned mine cards, the present application proposes a path exploration parameter optimization system and method based on deep reinforcement learning. Firstly, a hybrid A-star path planning framework with variable exploration parameters is constructed, an environment representation model considering obstacle region division is established, and on this basis, an exploration parameter optimization strategy based on deep reinforcement learning is established, as follows:

[0086] 1. Path planning framework with variable exploration parameters

[0087] 1.1 Node exploration rule with variable parameters

[0088] By analyzing the kinematic characteristics of unmanned mine trucks, the iterative exploration rules of path nodes are established, and key exploration parameters are extracted to construct variable parameter node exploration.

[0089] 1.2 Node evaluation method

[0090] Based on the child nodes generated by the node exploration in 1.1, the effectiveness of the child nodes is analyzed through collision detection, and the evaluation method is established accordingly, so as to set the generation value for each child node and obtain the child node set.

[0091] 1.3 Loading parking route generation method based on Reeds-Shepp curve

[0092] In the iterative exploration process, the end point constraint needs to be considered, and multiple candidate parking routes are generated based on the Reeds-Shepp curve, and are screened and sorted through the evaluation function, and finally the appropriate loading parking route is selected and the search is ended.

[0093] 2. Environment representation model considering obstacle region division

[0094] 2.1 Obstacle region division

[0095] According to the information of the current node, the space around the node is divided based on the mine truck model, the occupancy state of the obstacle is judged, and the modeling of the distribution characteristics of the obstacle is realized.

[0096] 2.2 Environment state space modeling

[0097] Based on the region division in 2.1, the environment state space for deep reinforcement learning is constructed, which is used to represent the state obtained by the agent from the environment and input to the neural network of deep reinforcement learning.

[0098] 3. Exploration parameter optimization method based on deep reinforcement learning

[0099] 3.1 Deep learning network construction

[0100] Based on the state space in 2.2 and the exploration parameter rule in 1.1, the neural network is constructed to realize the mapping from the state space to the exploration parameter.

[0101] 3.2 Reward function construction

[0102] The necessary indicators of path planning are analyzed, and the reward function of the agent in the iterative training is constructed, so as to guide the strategy training of deep reinforcement learning.

[0103] 3.3 Deep reinforcement learning training process

[0104] Based on the design of the above deep reinforcement learning module, the offline training process is constructed to realize the optimization of the exploration parameter optimization network.

[0105] The following is illustrated by specific embodiments:

[0106] First embodiment

[0107] As Figure 1 shown, the embodiment provides a path exploration parameter optimization system based on deep reinforcement learning, comprising: establishing a path planning framework with variable exploration parameters, analyzing the obstacle distribution characteristics on this basis, establishing an environment state space, and finally constructing a deep reinforcement learning network to realize adaptive optimization of path exploration parameters.

[0108] I. Variable parameter path planning module 1 for path planning framework with variable exploration parameters

[0109] The variable parameter path planning module 1 is used to generate an optimal step length and an optimal steering angle based on a deep reinforcement learning network for a current node and environment information, and to construct a fixed steering angle set. The steps of generating a sub-node set by combining the optimal step length and the fixed steering angle set, generating a steering angle optimization sub-node by combining the optimal step length and the optimal steering angle and adding it to the sub-node set for node exploration, performing collision detection on the sub-nodes in the sub-node set, and calculating the generation value of all the sub-nodes are performed. Finally, a loading parking path is generated by a Reeds-Shepp curve.

[0110] In this embodiment, the variable parameter path planning module 1 specifically comprises:

[0111] Variable parameter exploration rule

[0112] Node exploration should meet the motion characteristic constraints of the unmanned mine car, otherwise the mine car cannot track the generated path, resulting in a large risk. Therefore, it is necessary to first model the motion characteristics of the mine car. In the mine car working scene of this project, since the mine car usually runs at a low speed, a two-degree-of-freedom vehicle kinematics model can be used to represent the motion characteristics of the unmanned mine car.

[0113] Specifically, the vehicle pose state at any time can be represented as: , the coordinate origin of the vehicle is located at the center position of the rear axle, and the coordinate axis is parallel to the vehicle body. represents the vehicle speed, represents the vehicle orientation angle, represents the steering angle of the vehicle, represents the distance between the front and rear axles of the vehicle. The kinematics model of the vehicle can be represented as follows:

[0114]

[0115] Assuming that the current node is , is a position coordinate, is an orientation angle, the motion characteristics of the unmanned mine car according to the above formula establishes the following node exploration process:

[0116]

[0117] wherein is a next child node explored with the current node, is a position coordinate, is an orientation angle, represents an expansion direction of the current node including back or forward, and respectively represent a turning angle and a step length of node expansion, is a front-rear axle distance of the unmanned mine car. It can be seen that the position and orientation of the child node are related to the expansion direction, the turning angle and the step length. The existing algorithm often uses a fixed turning angle and step length, and thus is difficult to be applied to a complex and changeable mine operation environment.

[0118] In order to realize variable turning angles and step lengths, the turning angles and step lengths can be sampled, and corresponding turning angle set and step length set are formed, wherein and are the number of samplings. However, since the step length and the turning angle can be combined to form combinations, a large amount of additional calculation time is caused, and thus the present application optimizes the path exploration parameters through deep reinforcement learning, and establishes a variable parameter exploration rule.

[0119] Specifically, as shown in Figure 2 , the deep reinforcement learning network generates the optimal step length and the optimal turning angle for the current node and the environment information, and since the left and right turning abilities of the mine car are symmetrical, the fixed turning angle set is constructed in a uniform sampling manner, wherein, is the number of turning angle samplings in the fixed turning angle exploration set is less than , so as to reduce the calculation amount and save time), and for the fixed turning angle , the calculation manner is:

[0120]

[0121] wherein, is the maximum turning angle that can be performed by the mine car, the number of steering angles to be constructed;

[0122] Based on the above parameters, the node exploration is carried out in two steps, i.e. step length optimization and steering angle optimization. In the first step, the optimal step length and all sampled fixed steering angles in the fixed steering angle set are substituted into formula (1) to generate the sub-node set of fixed steering angle exploration , the number of sub-nodes in the sub-node set is the same as the number of sampled angles in the fixed steering angle set , which is However, the sub-nodes of fixed steering angle exploration are likely to be suboptimal and have a higher value. In the second step, the optimal step length and the optimal steering angle are substituted into formula (1) to generate the steering angle optimization sub-node and add it to the sub-node set .

[0123] Node evaluation method

[0124] After obtaining the sub-node set , the sub-nodes in the set need to be evaluated. Specifically, first, collision detection is performed on all sub-nodes in the sub-node set , the mine car is covered by two envelope circles, and sampling is performed on the path (smooth circular arc generated by the turning radius corresponding to the steering angle) from the current node to the explored sub-node to determine whether the distance of the obstacle grid is less than the envelope circle radius. If it is less than, it is considered that the sub-node is infeasible, and is deleted from the sub-node set .

[0125] On this basis, the value of all explored sub-nodes is calculated using , wherein represents the actual consumption cost of the starting point to the moving mine car of the explored sub-node, represents the predicted cost of the explored sub-node to the destination, is the weight of the predicted cost; when designing the cost function , it is considered that the operations such as reversing and switching direction of the mine car during movement usually consume more time and energy, and this paper comprehensively considers the factors such as reversing penalty, direction switching penalty and path length in the cost function to evaluate the pros and cons of the node.

[0126] , wherein the actual consumption cost is:

[0127]

[0128] In the above formula, is the cost of the current node based on the cost of the previous node contains five indicators, is the distance from the current node in the iterative search to the child node is the reverse cost, is the change cost, is the steering cost, is the steering change cost, is the weight coefficient.

[0129] If the child node is obtained by reverse exploration of the vehicle, the reverse cost , generally a large cost constant, is added to the cost function; when the moving direction of the vehicle is opposite to the moving direction of the previous round of search, the change cost , generally a large constant, needs to be added to the cost function; if the steering angle used at this time is not zero, the steering cost , which is proportional to the absolute value of the steering angle , needs to be added; when the steering angle used in the current round of search is different from that of the previous round, the steering angle change cost , which is proportional to the absolute value of the difference of the steering angle change, needs to be added. The heuristic function is the predicted value of the distance from the current node to the destination. In this paper, a heuristic function considering obstacles is used, i.e., the distance from the current node to the destination is calculated by the method.

[0130] (3) Loading berthing route generation method based on Reeds-Shepp curve

[0131] When the distance between the current node and the end point is less than the threshold , a plurality of candidate loading berthing path curves from the current node to the end point are generated by the Reeds-Shepp curve, the cost of the nodes on the curve is calculated by formula (3), and the costs are sorted to select the path with the minimum cost and perform reverse lookup to obtain the global path. If all the candidate loading berthing path curves collide, the node exploration step is continued.

[0132] II. Environment state space modeling module 2 for environment state space modeling considering obstacle region division

[0133] The environment state space modeling module 2 is used to divide the obstacles around the current node into regions and perform environment state space modeling.

[0134] In this embodiment, the environmental state space modeling module 2 is specifically:

[0135] Obstacle area division method

[0136] like Figure 3 As shown, in order to characterize the impact of environmental obstacles on planning, the present invention sets the current node The surrounding space is divided into 8 areas according to angles , in each region, use Indicates the The minimum distance between obstacles and minecarts in a region.

[0137] Environmental state space modeling

[0138] Deep reinforcement learning determines the best action based on the state space input. Therefore, in order to make deep reinforcement learning more generalized, it is necessary to consider the distance information between the current node and the surrounding obstacles, as well as the relative position information between the current node and the starting point and the end point. Specifically, the state space The design is as follows:

[0139]

[0140] in, represents the coordinates of the current node, Indicates the distance from the starting point to the current node. Indicates that the starting point is at the current node as the origin and the heading angle direction is The relative angular orientation of the axis in the coordinate system. Indicates the distance of the end point relative to the current node. Indicates the relative angular orientation of the endpoint in a coordinate system with the current node as the origin and the heading angle as the x-axis. Indicates that the terminal loading posture is with the current node as the origin and the heading angle direction as The direction of the axis in the coordinate system. Indicates the number of obstacles within the given range of the current node. Specifically, this paper divides the space around the minecart into 8 areas according to angles, namely .use Indicates the The minimum distance between obstacles and minecarts in a region.

[0141] 3. Deep Learning Parameter Optimization Module for Exploratory Parameter Optimization of Deep Reinforcement Learning 3

[0142] The deep learning parameter optimization module 3 is configured to build a deep learning network to calculate the optimal step length and the optimal steering angle, and to optimize the deep learning network by building a reward function while performing a training process of the deep learning network.

[0143] In this embodiment, the deep learning parameter optimization module 3 is specifically configured to:

[0144] (1) Deep learning network construction

[0145] The deep learning network is trained by using a DQN algorithm, and the action space is composed of combinations of the candidate optimal step lengths and combinations of the candidate optimal steering angles , i.e., the action space is composed of all possible combinations of .

[0146] For example, , , and are the minimum and maximum exploration step lengths, which can be adjusted. The action space is composed of all possible combinations of and , i.e., there are 8 possible actions.

[0147] The DQN algorithm uses two networks with the same structure but different parameters for training. The training network is used to calculate the value of the policy selection and to perform iterative updates of the value, and the target network is used to calculate the value of the next state in the time difference target TDTarget. The loss function Loss of the DQN algorithm is designed as follows:

[0148]

[0149] wherein is a set of state transition data obtained during the training process, including the current state , the current action , the reward obtained after the action is taken , and the state and the action obtained after the interaction with the environment, is an adjustable decay coefficient;

[0150] The target network and the training network They are all constructed using a three-layer fully connected layer network with 32 neurons. The outputs of the first two fully connected layers are input into the activation function and then output to the next fully connected layer. The activation function is set using PReLu. The last fully connected layer directly outputs the Q value of each action, including the steering angle and step length. Finally, the steering angle and step length with the highest Q value are selected as the final optimized exploration parameters including the optimal step length and the optimal steering angle.

[0151] (2) Reward function construction

[0152] In order to train and optimize the deep reinforcement learning network, it is necessary to design a reasonable reward function and optimize the strategy through rewards. Specifically, since path planning is an iterative search, the deep reinforcement learning network will output actions and receive rewards during each round of exploration. When designing the reward function, the main consideration is to ensure that the minecart reaches the destination as quickly as possible, reduce the number of iterations, and maintain a safe distance from obstacles. The reward function is designed to include rewards for reaching the destination. , obstacle avoidance reward , explore the cost , smoothness reward :

[0153] Is the reward for moving towards the destination, set The purpose of the round is to guide the minecart towards the destination. Therefore, when the minecart approaches the destination, it receives a positive reward; when it moves away from the destination, it receives a penalty. In addition, when the Reeds-Shepp curve in this round successfully connects to the destination, the minecart is considered to have reached the destination and is given a fixed reward for reaching the destination. .

[0154] Define the destination reward as follows:

[0155]

[0156] in, is an adjustable weight, The current node in this iteration round To the end The Euclidean distance of The steering angle is optimized by the child node To the end The Euclidean distance of When the Reeds-Shepp curve in this round successfully connects to the destination, the minecart is considered to have reached the destination and a fixed reward for reaching the destination is given; when the Reeds-Shepp curve is triggered, it means that the path is successfully generated, so a larger reward will be obtained. When the Reeds-Shepp curve fails to trigger, it means that exploration is still needed. In node exploration, we hope that the node generated by the optimized exploration parameters is as close to the end point as possible, so we add a method to calculate the distance from the current node to the end point. The distance from the child node to the end point If the child node generated by the exploration parameter optimization is far away from the end point, the reward is negative; otherwise, the reward is positive.

[0157] Is the obstacle avoidance reward, set The purpose is to avoid collisions between the minecart and surrounding obstacles due to the small distance between them. When designing the obstacle avoidance reward function, the child nodes are optimized according to the obstacles and the generated steering angle. distance The safety status of the minecart is divided into four situations: collision, danger, risk, and safety. In addition, to better guide strategy training, a potential field function is used to ensure the continuity of reward output at different distances.

[0158] Define the obstacle avoidance reward as follows:

[0159]

[0160] in, represents the obstacle avoidance reward in the i-th region, and are adjustable weight coefficients and design distance thresholds , , considered a collision, returns a larger penalty constant ,when When , it is considered a dangerous situation and returns a larger penalty function. When , it is considered to be risky and a smaller penalty function is returned. When it is considered safe, no penalty will be returned, and the overall obstacle avoidance reward Satisfy the following formula:

[0161]

[0162] Define the exploration cost as follows:

[0163]

[0164] in, A fixed penalty cost constant is set for each step, which guides the mine car to approach the destination more quickly and prevents meaningless exploration. The cost is set to a negative number;

[0165] Since turning the mine car will incur additional travel costs, this section sets a smoothness reward to encourage avoiding turning the steering wheel. The smoothness reward is defined as follows:

[0166]

[0167] wherein, represents the current node generated in the current search iteration round corresponding to the turning angle, corresponds to the best turning angle generated by the deep reinforcement learning network in the current search iteration round, represents the best step length generated by the deep reinforcement learning network in the current search iteration round, and are adjustable coefficients, respectively;

[0168] The reward function is ultimately:

[0169] .

[0170] (3) Deep reinforcement learning training process

[0171] As Figure 4 shown, for the training of deep reinforcement learning, first randomly select appropriate starting points and endpoints on the map according to actual production data, and perform path planning. In the planning, the path planning parameters are optimized through reinforcement learning, thereby forming multiple sets of state transition sampling data, which are added to the replay buffer. In the training process, a batch of data is randomly selected from the replay buffer, and the parameters of the estimation network are updated according to the loss function. After a certain number of iterations, the parameters of the training network are copied to the target network . In this way, a learning process is completed. Using two networks reduces the correlation between the current Q value and the target Q value to some extent, and improves the stability of the algorithm. In the training, in order to improve the generalization, the starting point and the endpoint can be randomly fine-tuned, thereby enriching the training data and further improving the generalization. As shown in Table 1 below, the pseudo code of the DQN algorithm used in the training of deep reinforcement learning is shown.

[0172] Table 1 DQN algorithm

[0173] Algorithm 1 DQN algorithm

[0174] 1. Initialize the training network with random network parameters​

[0175] 2. Copy the same parameters to initialize the target network

[0176] 3. Initialize the experience replay pool

[0177] 4. for sequence do

[0178] 5. Get the initial state of the environment

[0179] 6. for time step do

[0180] 7. Choose an action according to the current network in a greedy policy

[0181] 8. Perform the action and obtain the reward , the state of the environment becomes

[0182] 9. Store in the replay pool

[0183] 10. If there is enough data, sample data in batches from it

[0184] 11. For each data, calculate

[0185] 12. Update the parameters of the network so that as close as possible to

[0186] 13. Every C times, make

[0187] Second embodiment

[0188] As shown in FIGS. Figure 5 and 6 , the embodiment provides a deep reinforcement learning-based path exploration parameter optimization method performed by a deep reinforcement learning-based path exploration parameter optimization system as in the first embodiment, comprising the following steps:

[0189] S1: The deep reinforcement learning network generates an optimal step length and an optimal steering angle for the current node and environment information, and constructs a fixed steering angle set, and performs node exploration in the following steps: generating a sub-node set by combining the optimal step length and the fixed steering angle set, and generating a steering angle optimization sub-node by combining the optimal step length and the optimal steering angle and adding it to the sub-node set.

[0190] S2: performing collision detection on the child nodes in the child node set and calculating the generation value of all the child nodes.

[0191] S3: obtaining the child node with the lowest generation value in each round of iterative search process as the next child node of the current node selected finally.

[0192] S4: when the distance from the current node to the terminal point is less than a set threshold, generating a loading parking-in path through a Reeds-Shepp curve and generating a planned path through node backtracking.

[0193] A computer readable storage medium stores computer code, when the computer code is executed, the above method is executed. A person of ordinary skill in the art can understand that all or part of the steps in the above method can be completed by a program instructing related hardware, and the program can be stored in a computer readable storage medium, and the storage medium can include a read only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0194] The above is only the preferred embodiment of the present application, and the protection scope of the present application is not limited to the above embodiment. Any technical solution falling within the concept of the present application is within the protection scope of the present application. It should be noted that, for ordinary skilled in the art, some improvements and refinements without departing from the principles of the present application are also considered as the protection scope of the present application.

[0195] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combinations of the technical features do not exist contradictory, they should be considered as the scope of the present application.

[0196] It should be noted that the above embodiments can be freely combined as needed. The above is only the preferred embodiment of the present application, and it should be noted that, for ordinary skilled in the art, some improvements and refinements without departing from the principles of the present application are also considered as the protection scope of the present application.

Claims

1. A path exploration parameter optimization system based on deep reinforcement learning, characterized in that: include: a variable parameter path planning module for generating an optimal step length and an optimal steering angle for the current node and environmental information based on a deep reinforcement learning network, constructing a fixed steering angle set, performing node exploration by combining the optimal step length and the fixed steering angle set to generate a subnode set, combining the optimal step length and the optimal steering angle to generate a steering angle optimization subnode and adding it to the subnode set, performing collision detection on subnodes in the subnode set, calculating the cost of all subnodes, and finally generating a loading and docking path using a Reeds-Shepp curve; Assume that the current node is x c ,y c is the position coordinate, For the orientation angle, the following node exploration process is established based on the motion characteristics of the unmanned mining truck: in is the next child node to be explored with the current node, x s ,y s is the position coordinate, is the direction angle, d∈{-1,1} represents the expansion direction of the current node, including backward or forward, δ and l represent the steering angle and step size of the node expansion respectively, L w is the distance between the front and rear axles of the unmanned mining truck; Generate the optimal step size and the optimal steering angle for the current node and the environmental information through the deep reinforcement learning network, and construct the fixed steering angle set by uniform sampling For a fixed steering angle δ i , i=1,2,..,N3, the calculation method is: Among them, δ max is the maximum steering angle that the minecart can perform, and N3 is the number of steering angles constructed; The node exploration is carried out in two steps including step size optimization and steering angle optimization. In the first step, the optimal step size L best And all sampled fixed steering angles in the fixed steering angle set Δ1 are substituted into formula (1) to generate the subnode set of fixed steering angle exploration The child node set The number of neutron nodes is the same as the number of sampling angles in the fixed steering angle set Δ1, which is N3. In the second step, the optimal step length L is best and the optimal steering angle δ best Substitute into formula (1) to generate the steering angle optimization subnode N best And add it to the child node set middle; An environment state space modeling module, used to divide the surrounding obstacles of the current node into regions and perform environment state space modeling; A deep learning parameter optimization module is used to construct a deep learning network to calculate the optimal step size and the optimal steering angle, and to construct a reward function to optimize the deep learning network, while executing the training process of the deep learning network.

2. The path exploration parameter optimization system based on deep reinforcement learning according to claim 1, characterized in that: In the variable parameter path planning module, collision detection is performed on the sub-nodes in the sub-node set and the cost values ​​of all the sub-nodes are calculated, specifically: For the child node set Collision detection is performed on all child nodes of the node by covering the minecart with two enveloping circles and sampling on the path from the current node to the explored child node to determine whether the distance of the obstacle grid is less than the radius of the enveloping circle. If it is less than the distance, the child node is considered infeasible and the node is removed from the child node set. Delete in; Using f(N s )=g(N s )+w h ×h(N s ) calculates the cost value of all child nodes explored, where g(N s ) represents the actual cost of the minecart in the process of moving from the starting point to the explored subnode, h(N s ) represents the predicted cost from the explored child node to the destination, w h is the weight of the predicted cost; Among them, the actual consumption cost g(N s )for: g(N s )=g(N c )+w1g dis (N s )+w2g back (N s )+w3g switch (N s )+w4g steer (N s )+w5g chage (N s ) (3) In the above formula, g(N s ) is the current node g(N c ) includes 5 indicators based on the cost, g dis (N s ) is the current node N from the iterative search c To child node N s The distance, g back (N s ) is the cost of reversing, g switch (N s ) is the change cost, g steer (N s ) is the turning cost, g change (N s ) Turn to change cost, w i , i=1,…,5 is the weight coefficient.

3. The path exploration parameter optimization system based on deep reinforcement learning according to claim 1, characterized in that: In the variable parameter path planning module, the loading and docking path is generated by the Reeds-Shepp curve, specifically: When the current node and end point N g The distance is less than the threshold L t When the Reeds-Shepp curve is generated from the current node To the end point N g Several candidate loading path curves are obtained, and the node costs on the curve are calculated by formula (3). They are sorted according to the costs, and the path with the minimum cost is selected and reverse search is performed to obtain the global path. If all candidate loading docking path curves collide, the node exploration step is continued.

4. The path exploration parameter optimization system based on deep reinforcement learning according to claim 1, characterized in that: In the environment state space modeling module, the obstacles around the current node are divided into regions, specifically: The current node The surrounding space is divided into 8 areas according to angles In each area, use Indicates the minimum distance between the obstacle and the minecart in the i-th area.

5. The path exploration parameter optimization system based on deep reinforcement learning according to claim 4 is characterized in that: In the environment state space modeling module, environment state space modeling is performed, specifically: The state space The design is as follows: Among them, S position Indicates the coordinates of the current node, d start Indicates the distance from the starting point to the current node, φ start Indicates the relative angle orientation of the starting point in the coordinate system with the current node as the origin and the heading angle as the x-axis, d goal Indicates the distance of the end point relative to the current node, φ goal Indicates the relative angular orientation of the end point in a coordinate system with the current node as the origin and the heading angle as the x-axis. Indicates the direction of the terminal loading posture in the coordinate system with the current node as the origin and the heading angle as the x-axis, N obs Indicates the number of obstacles within a given range of the current node. , i=1,2,…,8 represents the minimum distance between the obstacle and the minecart in the i-th area.

6. The path exploration parameter optimization system based on deep reinforcement learning according to claim 1, characterized in that: In the deep learning parameter optimization module, a deep learning network is constructed to calculate the optimal step size and the optimal steering angle, specifically: The DQN algorithm is used to train the deep learning network, and the action space is expanded by the optimal step length δ rl The combination of the optimal steering angle l rl The action space is composed of all possible (δ rl ,l rl ) The DQN algorithm uses two networks with exactly the same structure but different parameters for training. π Used to calculate the Q value of the strategy selection and iteratively update the Q value; the target network Q π 'Used to calculate the Q value of the next state in the temporal difference target TD Target, the loss function Loss of the DQN algorithm is designed as follows: Among them, (s i ,a i ,r i ,s' i ) is a set of state transition data obtained during the training process, including the current state s i 、Current action a i , the reward r obtained after taking this action i And the state s' obtained by interacting with the environment after taking this action i and action a' i , γ is the adjustable attenuation coefficient; Target network Q π ' and train the network Q π They are all constructed using a three-layer fully connected layer network with 32 neurons. The outputs of the first two fully connected layers are input into the activation function and then output to the next fully connected layer. The activation function is set using PReLu. The last fully connected layer directly outputs the Q value of each action, including the steering angle and step length. Finally, the steering angle and step length with the highest Q value are selected as the final optimized exploration parameters including the optimal step length and the optimal steering angle.

7. The path exploration parameter optimization system based on deep reinforcement learning according to claim 1, characterized in that: In the deep learning parameter optimization module, a reward function is constructed to optimize the deep learning network, specifically: The reward function includes the reward r for moving toward the destination g , obstacle avoidance reward r o , exploration cost r t , smoothness reward r s : Define the reward r for approaching the destination g as follows: Among them, w g is an adjustable weight, l c The current node N in this iteration round c To the end point N g The Euclidean distance, l b The steering angle is optimized by the subnode N best To the end point N g The Euclidean distance, r success When the Reeds-Shepp curve in this round successfully connects to the destination, the minecart is considered to have reached the destination and a fixed reward for reaching the destination is given; Define the obstacle avoidance reward r o as follows: in, represents the obstacle avoidance reward in the i-th area, w1 and w2 are adjustable weight coefficients, and the distance threshold d is designed c , Considered as a collision, a larger penalty constant r is returned collision ,when When , it is considered a dangerous situation and returns a larger penalty function. When , it is considered to be risky and a smaller penalty function is returned. When , it is considered safe and no penalty is returned. The total obstacle avoidance reward is r o Satisfy the following formula: Define the exploration cost r t as follows: r t =-TimeConstant (9) Among them, TimeConstant is a fixed penalty cost constant set for each step, which guides the mine cart to approach the destination faster and prevents meaningless exploration. The cost is set to a negative number; Define the smoothness reward r s as follows: Among them, δ c Indicates that the current node N is generated in the current search iteration round c The corresponding steering angle, δ rl Corresponding to the best turning angle generated by the deep reinforcement learning network in the current search iteration, l rl Indicates the optimal step size generated by the deep reinforcement learning network in the current search iteration round, w3 and w4 are adjustable coefficients; The reward function is finally: R＝r g +r o +r t +r s (12)。 8. The path exploration parameter optimization system based on deep reinforcement learning according to claim 6, characterized in that: In the deep learning parameter optimization module, the training process of the deep learning network is executed, specifically: First, the appropriate starting point and end point are randomly selected on the map according to the actual production data, and the path planning is carried out. During the planning, the path planning parameters are optimized through reinforcement learning to form multiple groups of state transition sampling data, which are added to the playback buffer. During the training process, batches of data are randomly selected from the playback buffer, and the estimation network Q is updated according to the loss function. π The parameters of the network Q will be trained after a certain number of iterations. π The parameters are copied to the target network Q π ', thus completing a learning process.

9. A path exploration parameter optimization method based on deep reinforcement learning, performed by the path exploration parameter optimization system based on deep reinforcement learning according to any one of claims 1 to 8, characterized in that: The following steps are involved: S1: Based on the deep reinforcement learning network, an optimal step length and an optimal steering angle are generated for the current node and environmental information, and a fixed steering angle set is constructed. Node exploration is performed by combining the optimal step length and the fixed steering angle set to generate a subnode set, combining the optimal step length and the optimal steering angle to generate a steering angle optimization subnode, and adding the node to the subnode set. S2: performing collision detection on the child nodes in the child node set and calculating the cost values ​​of all the child nodes; S3: in each round of iterative search, obtaining the child node with the lowest cost as the next child node of the current node finally selected; S4: When the distance from the current node to the end point is less than a set threshold, a loading path is generated by a Reeds-Shepp curve, and a planned path is generated by node backtracking.

Images