Rule and data hybrid driven unmanned mine truck path planning system and method

Abstract

The invention relates to the technical field of path planning, and provides a rule and data hybrid-driven unmanned mine truck path planning system, which comprises the following steps of: constructing a node guide search rule, node exploration is carried out on any current child node by adopting a fixed step length and a steering angle after uniform discrete sampling to obtain an exploration child node set of a next child node with the fixed step length, and a guide node is generated through deep reinforcement learning and added into the exploration child node set; evaluating the cost of each exploration sub-node in the exploration sub-node set so as to obtain the exploration sub-node with the lowest cost in each subsequent iterative search process as the finally selected next sub-node, and generating a parking curve from the current sub-node to the terminal point when the distance from the current sub-node to the terminal point is smaller than a set threshold value; and a final global curve is obtained. According to the technical scheme, a rule-data hybrid driving path planning framework is constructed, and advantage complementation of rules and learning methods is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of path planning, and in particular to a rule- and data-hybrid driven unmanned mining truck path planning system and method. Background Art

[0002] The implementation of unmanned mining trucks can effectively reduce labor costs and improve the safety and transportation efficiency of mining production. Among them, the path planning module plans a collision-free path for the unmanned mining truck based on environmental information, which is an important part of the unmanned mining truck automatic driving system. In the transportation process of existing unmanned mining trucks, the routes from the parking lot to various shoveling platforms, dumping areas, unloading areas and other areas are generally fixed routes, and the roads are relatively flat. The driving path can be recorded and automated transportation can be achieved by tracking. However, after arriving at the shoveling platform, dumping area and other operating areas, the end point of the mine car will change with the changes in operating needs, so the tracking idea is no longer applicable, and an efficient and safe production path needs to be planned through the path planning module.

[0003] The working area of ​​the mine is unstructured, with irregular distribution of mountains and various gravels and puddles, which makes the boundary of the feasible domain blurred and reduces the efficiency of path search. In addition, the mine car must ensure high-precision position and posture when loading and unloading, otherwise it will cause safety risks. Therefore, the end point orientation constraint needs to be considered in path planning, which further increases the difficulty of path planning.

[0004] Existing path search methods can be mainly divided into two technical routes: rule-based and learning-based. Among them, the rule-based type adopts artificially set search rules. However, due to the complex and changeable mining environment and the presence of irregularly distributed gravel obstacles, the set rules are difficult to accurately adapt to each scene, so there will be a problem of insufficient search efficiency. The learning-based technical route realizes path planning by constructing a deep learning network. Although it can fit a good effect under a specific data set, the network is not interpretable enough, does not have a clear physical meaning, and faces serious stability and generalization problems. It is difficult to apply to unmanned mining truck path planning with strict safety requirements.

[0005] Therefore, how to integrate the parameter self-evolution characteristics based on learning methods into the rule-based path planning framework, and use rule constraints to ensure the stability and generalization of the neural network, so as to improve the search efficiency while ensuring the quality and stability of the generated path, and be suitable for variable mining operation scenarios, is the problem that this technology needs to solve. Summary of the invention

[0006] In view of the above problems, the purpose of the present invention is to provide a rule- and data-driven unmanned mining truck path planning system and method, construct a rule-data hybrid driven path planning framework, and realize the complementary advantages of rules and learning methods, which are specifically manifested in: by constructing a search guidance strategy based on deep reinforcement learning, the search efficiency and path quality are improved, and the problem of difficult parameter debugging and complex maintenance of rule-based algorithms is compensated. The optimal node is selected through a hybrid A-star framework, and a reward feedback mechanism for optimal node guidance is established, thereby guiding and constraining the strategy of deep reinforcement learning, and compensating for the lack of stability and generalization of learning algorithms.

[0007] The above-mentioned object of the present invention is achieved through the following technical solutions: A rule- and data-driven unmanned mining truck path planning system, including: A hybrid A-star planning module is used to construct node guidance search rules. For any current sub-node, a fixed step size and a steering angle after uniform discrete sampling are used to perform node exploration to obtain a set of exploration sub-nodes of the next sub-node with a fixed step size. A guide node is generated through deep reinforcement learning and added to the exploration sub-node set. The cost of each exploration sub-node in the exploration sub-node set is evaluated to obtain the exploration sub-node with the lowest cost as the next sub-node finally selected in each subsequent round of iterative search. At the same time, when the distance from the current sub-node to the end point is less than a set threshold, a berth curve from the current sub-node to the end point is generated to obtain the final global curve. A guide node generation module, used to design the state space and action space of the current child node, and to design a neural network structure for generating the guide node. In each round of iteration, the state space is input into the neural network structure, the Q value of the steering angle and the step length in each group of the action space is output, the steering angle and the step length with the highest Q value are selected to generate the guide node, and a reward function is designed to guide the training of the neural network structure; The path exploration framework design module is used to design the path exploration framework and the training strategy for deep reinforcement learning.

[0008] Furthermore, in the hybrid A-star planning module, the node guidance search rule is constructed, and for any current sub-node, a fixed step size and a steering angle after uniform discrete sampling are used to perform node exploration to obtain the exploration sub-node set of the next sub-node with a fixed step size, and the guidance node is generated by deep reinforcement learning and added to the exploration sub-node set, specifically: After obtaining the current child node, the steering angle Perform uniform discrete sampling, for , calculated as follows: in, is the boundary of the steering angle, corresponding to the maximum front wheel steering angle that the mine car can perform, i is the sequence number of the sampled steering angle, and n is the number of sampled steering angles; Considering the forward and backward exploration cases, the fixed step size and the sampled steering angle Substitute into the following formula (2) to obtain the set of explored subnodes for the next subnode: ; Among them, the current child node is , is the global coordinate of the current child node on the map, is the heading angle of the minecart at the current subnode, and any exploration subnode in the exploration subnode set is , To explore the global coordinates of the child nodes under the map, is the heading angle of the minecart at the exploration subnode, is the fixed step size of exploration, is the wheelbase of the mining truck, is the exploration direction of the node, i.e., backward or forward; Input the environment information and the state of the current child node into the deep reinforcement learning network, and output the guided step length and the guide steering angle , and substitute it into formula (2) to generate the guide node , the guiding node has a larger step size and a lower cost, and is added to the exploration subnode set, that is, .

[0009] Furthermore, in the hybrid A-star planning module, the cost of each exploration sub-node in the exploration sub-node set is evaluated to obtain the exploration sub-node with the lowest cost as the next sub-node finally selected in each subsequent iterative search process, specifically: First, the effectiveness of each exploration sub-node in the exploration sub-node set is evaluated, that is, the mine cart cannot collide with environmental obstacles when it is located at the location of the exploration sub-node, and the arc path from the current sub-node to each exploration sub-node is sampled. At the corresponding discrete point position and orientation, the mine cart is simulated with a rectangle, and the mine cart is finely divided into grids. The corresponding grid on the global map is calculated for each subdivided grid, and the corresponding grid on the global map is queried whether it is occupied. If it is occupied, it means that there is a collision, that is, the exploration sub-node is an invalid sub-node, and it is removed from the exploration sub-node set. If no collision occurs at any discrete point on the arc path, it means that the exploration subnode is valid; On this basis, the cost evaluation function is established To evaluate each exploration subnode , specifically: in, is the current cost function, which represents the actual cost from the starting point to the explored subnode. is the heuristic cost function, which represents the cost of exploring the subnode to the end point. is the weight coefficient, For the current child node The cost basis includes 5 indicators, namely, distance cost , the cost of reversing , forward and backward motion switching cost , Turning cost and the cost of turning change , is the predicted value of the distance from the explored child node to the destination.

[0010] Further, in the hybrid A-star planning module, when the distance from the current subnode to the end point is less than the set threshold, the berth curve from the current subnode to the end point is generated, thereby obtaining the final global curve, specifically: If the current child node To the end The distance is less than the set threshold , generated from the current child node by the Reeds-Shepp curve To the end Several candidate loading berth curves are obtained, and the node costs on the berth curves are calculated by formula (3), and sorted according to the costs, and the path with the minimum cost is selected and reverse search is performed to obtain the final global path; If the berth curves of all candidate loads collide, the step of node exploration is continued.

[0011] Furthermore, in the guiding node generation module, the state space and action space of the current child node are designed, specifically: The state space Designed for: (4) in, is the current child node, is the starting point node information, is the terminal node information, is the position and heading angle of the current child node relative to the starting node, is the position and heading angle of the end node relative to the current child node, is the distance between the position of the current child node and the end node, is the minimum distance between the obstacle and the minecart in the ith area; The exploration parameters of the guiding node include step size and steering angle, and the corresponding action space is , deep reinforcement learning adopts the DQN algorithm, which adopts a discrete action space and performs discrete sampling on the steering angle and step size.

[0012] Furthermore, in the guiding node generation module, the neural network structure of the guiding node is designed and generated, specifically: The DQN algorithm uses two networks with exactly the same structure but different parameters for training, estimating the network To calculate the strategy selection Value and perform Iterative update of the value, the target network Used to calculate the next state in the time difference target TDTarget Value, the loss function of the DQN algorithm is designed as follows: Where N is the number of samples, is the target Q value, through the target network Calculate the time difference target to get, The neural network structure is calculated in the state Take action Q value; The target network and the estimation network All of them are constructed using a three-layer fully connected network, where the number of neurons is 32. The outputs of the first two fully connected layers are input into the activation function and then output to the next fully connected layer. The activation function is set using PReLu. The last fully connected layer directly outputs the Q value of each action, including the steering angle and step length. Finally, the steering angle and step length with the highest Q value are selected as the final exploration parameters of the guide node and are substituted into formula (2) to generate the guide node.

[0013] Furthermore, in the guiding node generation module, a reward function is designed to guide the training of the neural network structure, specifically: The reward function includes a reward for reaching the destination , obstacle avoidance reward , explore the cost , Smoothness Reward and node selection rewards ; The destination reward The definition is as follows: in, is a large fixed reward constant, is the manually adjusted weight coefficient, The current child node in this iteration To the end node The Euclidean distance of is the guide node generated by the neural network structure To the end node The Euclidean distance of The obstacle avoidance reward The definition is as follows: in, and is an adjustable weight coefficient, is the distance threshold, when , considered a collision, returns a larger penalty constant ,when When , it is considered a dangerous situation and returns a larger penalty function. When , it is considered to be risky and a smaller penalty function is returned. When it is considered safe, no penalty will be returned, Represents the obstacle avoidance reward in the i-th area, with a total of n areas, and the overall obstacle avoidance reward Satisfy the following formula: The exploration cost The definition is as follows: in, is an adjustable penalty constant; The smoothness reward The definition is as follows: in, and is an adjustable weight coefficient, The boot node generated The corresponding steering angle, The current child node in the search iteration The corresponding steering angle; The node selection reward The definition is as follows: in, is an adjustable fixed reward constant, is the set of explored subnodes in this round of iterative search The steering angle corresponding to the best node finally selected in is an adjustable weight coefficient; The reward function is: .

[0014] Furthermore, in the path exploration framework design module, a path exploration framework is designed, specifically: Get the global grid map and the start and end point pose information, initialize the open set and closed set, set the start point as a node and set the cost to 0, add it to the open set, and then enter the iterative search phase; In each round of iterative search, firstly, the node with the lowest cost is selected from the open set as the current child node, and then the current child node is removed from the open set and added to the closed set; Determine whether the distance between the current child node and the end point is not greater than a threshold If the conditions are met, try to use the Reeds-Shepp curve to directly generate the loading path. If successful, add the path to the closed set and end the iterative search. If unsuccessful, or the distance to the end point is greater than the threshold, , then the current child node is explored, and the next child node and the guide node with a fixed step length are generated by using the node-guided search rule and the deep reinforcement learning network respectively, and added to the exploration child node set, and then the exploration child nodes in the exploration child node set are subjected to collision detection to remove invalid child nodes and update the child node costs through the evaluation function, and finally added to the open set, ending this round of iterative search; After the path planning is successful and the iterative search is completed, all nodes of the planned path can be obtained through node backtracking.

[0015] Furthermore, in the path exploration framework design module, a training strategy for deep reinforcement learning is designed, specifically: For deep reinforcement learning training, appropriate starting and ending points are randomly selected on the map based on actual production data, and path planning is performed; In planning, the path planning parameters are optimized through reinforcement learning to form multiple sets of state transition sampling data and add them to the playback buffer; During the training process, batches of data are randomly selected from the playback buffer and the estimation network is updated according to the loss function. After a certain number of iterations, the estimated network The parameters are copied to the target network Complete a learning process.

[0016] A method for unmanned mining truck path planning driven by a rule and data hybrid drive using the unmanned mining truck path planning system driven by a rule and data hybrid drive as described above comprises the following steps: S1: Constructing node guided search rules, for any current child node, using a fixed step size and a steering angle after uniform discrete sampling to perform node exploration to obtain a set of explored child nodes of the next child node with a fixed step size, and generating a guiding node through deep reinforcement learning to add to the set of explored child nodes; S2: Evaluate the cost of each exploration sub-node in the exploration sub-node set, and obtain the exploration sub-node with the lowest cost as the next sub-node finally selected in each round of iterative search process; S3: When the distance from the current subnode to the end point is less than a set threshold, a berth curve from the current subnode to the end point is generated, thereby obtaining a final global curve.

[0017] Compared with the prior art, the beneficial effects of the present invention include: (1) Improving the timeliness of path planning The distribution of gravel and mountains in the mining environment is irregular, so different exploration strategies need to be adopted according to environmental information. However, the existing planning method has a single exploration strategy and fixed parameters, which leads to a decrease in search efficiency and low path quality. The present invention constructs a path planning framework based on rule-data hybrid drive, establishes a node exploration method based on hybrid A-star-deep reinforcement learning fusion, and can effectively adjust the exploration strategy according to different environments, thereby accelerating the efficiency of path search. On this basis, since the generated node cost is lower, the generated path quality will also be higher.

[0018] (2) Reduced reliance on fixed exploration rules Existing rule-based path planning methods deal with path planning for different operating scenarios by establishing different rules. However, simple rule establishment will lead to limited applicable scenarios, and complex rule design will introduce debugging problems that bring a large number of parameters, making the rules difficult to maintain. Based on the construction of basic iterative exploration rules, the present invention generates adaptive guidance nodes by establishing a deep reinforcement learning network, thereby accelerating the exploration efficiency. In addition, the present invention takes into account obstacle information in the state space of deep reinforcement learning, which can effectively cope with complex and changeable mining scenarios. It no longer requires a large amount of manual debugging of exploration parameters, reducing the reliance on fixed exploration rules.

[0019] (3) Providing guidance for strategy convergence of deep reinforcement learning Existing learning planning algorithms often directly establish the fitting relationship between the environment and the path, lack clear physical meaning, and lack guidance during training, resulting in the generated path not meeting the movement characteristics of mining trucks. The present invention constructs node selection reward feedback on the rule-based path planning framework, and rewards the exploration of paths that meet the rules during the training process, which can effectively guide the strategy convergence of deep reinforcement learning, so that it can generate nodes that are more in line with the rules and have lower costs.

[0020] (4) Ensure the stability and generalization of path planning The environment of mining operation scenes is complex and changeable, and existing planning algorithms all face stability and generalization problems. The rule construction of rule-based algorithms is difficult and the parameter adjustment cost is high. The output of learning-based algorithms is unstable and lacks generalization, all of which have difficulties in practical application. In contrast, the present invention retains the rule exploration of hybrid A-star during the exploration process, and generates guide nodes based on adaptive exploration parameters through deep reinforcement learning to accelerate the node exploration process; on the other hand, in the event of failure of the guide node, the path can still be generated through rule exploration, which has higher redundancy and ensures the output stability and scene generalization of the algorithm.

[0021] (5) Experimental verification On the grid map of the actual mine loading platform ( ), and tested it according to the starting point and end point in the actual production data, and compared it with the existing hybrid A-star algorithm used for unmanned mining truck path planning. The solution time of the present invention is 0.68 seconds, which is 80% less than that of the hybrid A-star, and the path length is 579m, which is 4% less than that of the hybrid A-star, which has significant superiority.

[0022] The specific data are as follows: On the grid map of the actual mine loading platform ( ), the solution time of the present invention is 0.68 seconds and the path length is 579m. The solution time is reduced by 80% and the path length is reduced by 5% compared with the hybrid A-star; the solution time is reduced by 60% and the path length is reduced by 2% compared with DQN (pure data-driven), which is significantly better than other existing path planning methods.

[0023] In summary, the present invention proves through theoretical analysis and experimental data that the path planning method based on rule-data hybrid drive can better realize the complementary advantages of rule-driven and data-driven methods, significantly improve the search efficiency and path quality, and ensure stability and generalization. It is more suitable for changeable mining environments and has important practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 The overall structure diagram of the unmanned mining truck path planning system driven by rules and data hybrid of the present invention; Figure 2 It is a schematic diagram of the obstacle area division of the present invention; Figure 3 This is a flow chart of the unmanned mining truck path planning framework and training framework driven by rule-data hybrid of the present invention; Figure 4 This is a schematic diagram of the node guidance exploration rule flow of the present invention; Figure 5 The overall flow chart of the unmanned mining truck path planning method driven by hybrid rules and data of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0027] The present invention proposes a path planning method based on rule-data hybrid drive. The method first establishes a node-guided hybrid A-star planning module, and on this basis, constructs a guidance node generation module based on deep reinforcement learning, and finally establishes a path exploration framework and training strategy based on rule-data hybrid drive, as follows: 1. Hybrid A-star planning module based on node guidance 1.1 Node-guided exploration rules Firstly, a node-guided path exploration rule is constructed and parameterized to generate candidate child nodes for subsequent iterative search.

[0028] 1.2 Node evaluation method For the child nodes generated in 1.1, the validity of the child nodes is judged based on collision detection, and a node evaluation method is constructed to evaluate the node cost.

[0029] 1.3 Parking route generation method based on Reeds-Shepp curve Since the loading operation path has strict constraints on the final posture, a number of candidate docking paths are generated through the Reeds-Shepp curve, and they are sorted and screened through the evaluation function to select the optimal path, thereby accelerating the search.

[0030] 2. Guided node generation module based on deep reinforcement learning 2.1 State Space Design The relative positions of the starting and ending points and the obstacle distribution characteristics are considered in the state space, and parameter modeling is performed to represent the agent's environmental information and input it into the neural network.

[0031] 2.2 Neural Network Structure Design Based on the action modeling in 1.1 and the state space modeling in 2.1, the corresponding neural network structure is constructed to realize the mapping from state to action.

[0032] 2.3 Reward Function Design A reward function is constructed based on safety and efficiency, and an optimal node-guided reward feedback mechanism is established to guide and constrain the agent strategy training through rule-based path exploration.

[0033] 3. Rule-data hybrid driven path exploration framework and training strategy 3.1 Path Exploration Framework Design A rule-data hybrid driven path exploration framework is established, in which the guide nodes are generated through the DQN network to accelerate the path search efficiency.

[0034] 3.2 Training strategy design Based on 3.1, in offline training, the results of node selection in each search iteration are mapped to the reward function to guide the training of the DQN network.

[0035] The following is described by specific examples: First embodiment like Figure 1 As shown, this embodiment provides a rule- and data-driven unmanned mining truck path planning system, including: The hybrid A-star planning module 1 is used to construct a node guidance search rule, and for any current sub-node, a fixed step size and a steering angle after uniform discrete sampling are used to perform node exploration to obtain a set of exploration sub-nodes of the next sub-node with a fixed step size, and a guide node is generated through deep reinforcement learning to be added to the exploration sub-node set, and the cost of each exploration sub-node in the exploration sub-node set is evaluated to obtain the exploration sub-node with the lowest cost as the next sub-node finally selected in each subsequent round of iterative search process, and at the same time, when the distance from the current sub-node to the end point is less than a set threshold, a berth curve from the current sub-node to the end point is generated, thereby obtaining the final global curve; A guide node generation module 2 is used to design the state space and action space of the current child node, and to design a neural network structure for generating the guide node. In each round of iteration, the state space is input into the neural network structure, and the Q value of the steering angle and step length in each group of the action space is output. The steering angle and step length with the highest Q value are selected to generate the guide node, and a reward function is designed to guide the training of the neural network structure. Path exploration framework design module 3 is used to design the path exploration framework and design the training strategy for deep reinforcement learning.

[0036] The following is a description of each module: 1. Hybrid A-star planning module based on node guidance 1 Node-guided exploration rules Existing hybrid A-stars often use fixed step sizes and steering angles for searching. However, in complex and changing environments, fixed parameters cannot guarantee that the cost of the sub-nodes generated in each iterative search is minimal. However, directly using learning methods to replace iterative search can easily lead to path planning failures due to the instability of network output. Therefore, by constructing node-guided search rules, fixed step sizes and steering angles are used for node exploration to obtain fixed sub-nodes, and deep reinforcement learning is used to generate guide nodes, which generally have longer step sizes and smaller costs, thereby reducing the number of iterations, accelerating path planning searches, and improving path quality.

[0037] Specifically, since the mine cart travels at a low speed, the motion characteristic constraints of the mine cart can be represented by a bicycle model in the hybrid A-star, clear boundary restrictions can be established for the steering angle, and circular arc trajectories can be used for node exploration.

[0038] After obtaining the current child node, the steering angle Perform uniform discrete sampling, for , calculated as follows: in, is the boundary of the steering angle, corresponding to the maximum front wheel steering angle that the mine car can perform, i is the sequence number of the sampled steering angle, and n is the number of sampled steering angles; Considering the forward and backward exploration cases, the fixed step size and the sampled steering angle Substitute into the following formula (2) to obtain the set of explored subnodes for the next subnode: ; The exploration subnodes in this exploration subnode set are generated by exploration rules, and the step size is relatively fixed.

[0039] Among them, the current child node is , is the global coordinate of the current child node on the map, is the heading angle of the minecart at the current subnode, and any exploration subnode in the exploration subnode set is , To explore the global coordinates of the child nodes under the map, is the heading angle of the minecart at the exploration subnode, is the fixed step size of exploration, is the wheelbase of the mining truck, is the exploration direction of the node, i.e., backward or forward; Input the environment information and the state of the current child node into the deep reinforcement learning network, and output the guided step length and the guide steering angle , and substitute it into formula (2) to generate the guide node , the guiding node has a larger step size and a lower cost, and is added to the exploration subnode set, that is, .

[0040] (2) Node evaluation method Get the exploration child node collection After that, the cost of the child node needs to be evaluated, so as to obtain the child node with the lowest cost as the current node in each subsequent iterative search. Specifically, firstly, the effectiveness of each exploration subnode in the exploration subnode set is evaluated, that is, the mine cart cannot collide with environmental obstacles when it is located at the location of the exploration subnode. The arc path from the current subnode to each exploration subnode is sampled, and the mine cart is simulated with a rectangle at the corresponding discrete point position and orientation. The mine cart is finely divided into grids, and the corresponding grid on the global map is calculated for each subdivided grid, and the corresponding grid on the global map is queried whether it is occupied. If it is occupied, it means that there is a collision, that is, the exploration subnode is an invalid subnode, and it is removed from the exploration subnode set. If no collision occurs at any discrete point on the arc path, it means that the exploration subnode is valid; On this basis, the cost evaluation function is established To evaluate each exploration subnode , specifically: in, is the current cost function, which represents the actual cost from the starting point to the explored subnode. is the heuristic cost function, which represents the cost of exploring the subnode to the end point. is the weight coefficient, For the current child node The cost basis includes 5 indicators, namely, distance cost , the cost of reversing , forward and backward motion switching cost , Turning cost and the cost of turning change , is the predicted value of the distance from the explored child node to the destination.

[0041] Specifically, the distance cost is the distance from the current node to the child node. is a fixed value. If the exploration direction from the current node to the child node is backward ( ), then accumulate; if it is a positive exploration It is considered as 0. The cost of switching forward and backward motion is a fixed value, the current node exploration direction and the child node exploration direction ( ), a fixed value is accumulated; if the exploration direction is the same ( ), it is considered as 0. Turn cost The steering angle with respect to the child node The absolute value of is proportional to, that is, . Turn to change cost is a fixed value, the exploration steering angle of the current node and the exploration steering angle of the child node If they are different, a fixed value is accumulated, otherwise it is considered to be 0. Heuristic function It is the predicted value of the distance from the current node to the destination. Due to obstacles, it cannot be directly predicted by Euclidean distance, but is used The method plans to calculate the distance between the grid where the current node is located and the grid where the destination is located.

[0042] (3) Parking route generation method based on Reeds-Shepp curve If the current child node To the end The distance is less than the set threshold , generated from the current child node by the Reeds-Shepp curve To the end Several candidate loading berth curves are obtained, and the node costs on the berth curves are calculated by formula (3), and sorted according to the costs, and the path with the minimum cost is selected and reverse search is performed to obtain the final global path; If the berth curves of all candidate loads collide, the step of node exploration is continued.

[0043] 2. Guided node generation module based on deep reinforcement learning 2 (1) State space and action space design Deep reinforcement learning determines the best action based on the state space input. Therefore, in order to make deep reinforcement learning more generalized, it is necessary to design the state space and fully consider the impact of environmental obstacles and the relative position of the end point on the planning. Specifically, the state space Designed for: (4) in, is the current child node, is the starting point node information, is the terminal node information, is the position and heading angle of the current child node relative to the starting node, is the position and heading angle of the end node relative to the current child node, is the distance between the position of the current child node and the end node, is the minimum distance between the obstacle and the minecart in the i-th area.

[0044] For example, in this embodiment, if Figure 2 As shown in Figure 1, in order to ensure the safe driving of the mine car, the distance between the mine car and the surrounding obstacles must be monitored. Specifically, this paper divides the space around the mine car into 8 areas according to the angle, namely .use Indicates the minimum distance between the obstacle and the minecart in the i-th area.

[0045] The output of deep reinforcement learning is the action of the agent, which is reflected in the exploration parameters of the guide node in the present invention. The exploration parameters of the guide node include step size and steering angle, and the corresponding action space is , deep reinforcement learning uses the DQN algorithm, which uses a discrete action space to discretely sample the steering angle and step length. For example, in this embodiment, the sampling of the steering angle and step length is: , , a total of 105 combinations.

[0046] (2) Neural network structure design The DQN algorithm uses two networks with exactly the same structure but different parameters for training, estimating the network To calculate the strategy selection Value and perform Iterative update of the value, the target network Used to calculate the next state in the time difference target TDTarget Value, the loss function of the DQN algorithm is designed as follows: Where N is the number of samples, is the target Q value, through the target network Calculate the time difference target to get. The neural network structure is calculated in the state Take action Q value; The target network and the estimation network All of them are constructed using a three-layer fully connected network, where the number of neurons is 32. The outputs of the first two fully connected layers are input into the activation function and then output to the next fully connected layer. The activation function is set using PReLu. The last fully connected layer directly outputs the Q value of each action, including the steering angle and step length. Finally, the steering angle and step length with the highest Q value are selected as the final exploration parameters of the guide node and are substituted into formula (2) to generate the guide node.

[0047] The Q value is the output of the DQN network, which is related to the state and action and is used to evaluate the quality of a set of action states. In simple terms, the deep reinforcement learning network will evaluate each action based on the current state. This evaluation is equivalent to the quality of taking the action in the current state, that is, whether it is more likely to obtain a greater reward. A high Q value indicates that the strategy of taking the action in the current state has a greater probability of obtaining a better reward, or is less likely to have negative effects such as penalties. The present invention is based on the current state space. Determine the steering angle and step size corresponding to the guide node. In order to better guide the search and improve the search efficiency, each action in the discrete action space is evaluated and the corresponding Q value is obtained. The steering angle and step size corresponding to the maximum Q value are selected for output.

[0048] (3) Reward function design The path planning framework of the present invention is an iterative search, so in each iteration round, the intelligent body will transform the state space The information is input into the deep reinforcement learning network and the action space is output The Q value of each set of steering angle and step length action in is selected, and the steering angle and step length with the highest Q value are output, so as to generate the guiding node according to formula (2) In order to effectively guide the training of the agent strategy, it is necessary to evaluate the quality of the action through the environmental feedback after the agent completes the action, design the corresponding reward return, and thus guide the training of the deep reinforcement learning network. During the training process, the agent will generate a guide node according to the process of each round of iterative search. The corresponding reward is obtained based on the effect, thus guiding the training of the network.

[0049] Specifically, when designing the reward function, it is necessary to ensure that the mine car reaches the target point as quickly as possible while maintaining a safe distance from obstacles. Therefore, the reward function designed by the present invention includes a reward for approaching the destination. , obstacle avoidance reward , explore the cost , Smoothness Reward and node selection rewards ; Is a reward for heading to the destination, set The purpose is to guide the mine cart to the destination. Therefore, when the mine cart approaches the destination, it will receive a positive reward; when it moves away from the destination, it will be punished. In addition, when the mine cart is successfully connected to the destination directly through the Reeds-Shepp curve in this round of search iteration, it is considered that the mine cart has reached the destination and a fixed reward for reaching the destination is given. The definition is as follows: in, is a large fixed reward constant, is the manually adjusted weight coefficient, The current child node in this iteration To the end node The Euclidean distance of is the guide node generated by the neural network structure To the end node The Euclidean distance.

[0050] When the Reeds-Shepp curve is triggered, it means that the path is generated successfully, so a larger reward will be obtained. When the Reeds-Shepp curve fails to be triggered, it means that exploration is still needed. In node exploration, it is hoped that the generated guide node As close to the end point as possible, so add by calculating the distance from the current node to the end point The distance from the guide node to the end point The difference between If the distance is far from the end point, the reward is negative; otherwise, the reward is positive.

[0051] is the obstacle avoidance reward, set The purpose is to avoid collisions between the minecart and surrounding obstacles due to the small distance between the minecart and the surrounding obstacles. When designing the obstacle avoidance reward function, the distance between the obstacle and the minecart is calculated. The safety status of the mine car is divided into four situations: collision, danger, risk and safety. The definition is as follows: in, and is an adjustable weight coefficient, is the distance threshold, when , considered a collision, returns a larger penalty constant ,when When , it is considered a dangerous situation and returns a larger penalty function. When , it is considered to be risky and a smaller penalty function is returned. When it is considered safe, no penalty will be returned, Represents the obstacle avoidance reward in the i-th area, with a total of n areas, and the overall obstacle avoidance reward Satisfy the following formula: In order to guide the mine cart to approach the destination more quickly and prevent meaningless exploration, this section sets a fixed exploration cost for each step. The cost is set to a negative number. The definition is as follows: in, is an adjustable penalty constant; Since turning the minecart will bring additional driving cost, this section sets a smoothness bonus , to guide the agent to reduce the steering wheel rotation, the smoothness reward The definition is as follows: in, and is an adjustable weight coefficient, The boot node generated The corresponding steering angle, The current child node in the search iteration The corresponding steering angle; In addition, in order to enable deep reinforcement learning to generate the optimal steering angle, that is, the generated steering angle cost is the smallest and is preferentially selected as the current node for the next round of search, this paper sets the node selection reward , by mixing A star to choose whether to select the guide node to feedback guide deep reinforcement learning strategy training. Specifically, when the node generated by reinforcement learning is the best node in this expansion, a reward is given; otherwise, a penalty is given. The node selection reward The definition is as follows: in, is an adjustable fixed reward constant, is the set of explored subnodes in this round of iterative search The steering angle corresponding to the best node finally selected in is an adjustable weight coefficient.

[0052] Specifically, the hybrid A star is evaluated by the evaluation function The quality of each node in the simulation is evaluated, and the optimal node with the minimum cost is selected. The minimum cost means that it is more likely to be selected as the current node (i.e., the point of the final path) in each round of iterative exploration in actual exploration, so it will be directly given a fixed reward. On the contrary, if exist If it is not the node with the minimum cost, it means that it is more difficult to select this node as a path point in actual exploration. Therefore, it is necessary to guide the deep learning network to select and generate a better guide point, so the steering angle is calculated. With the selected node (i.e. The steering angle corresponding to the node with the minimum actual cost The difference is used as penalty.

[0053] In summary, the reward function is: .

[0054] 3. Path exploration framework design module of rule-data hybrid driven path exploration framework and training strategy 1 (1) Path exploration framework design Path exploration framework Figure 3As shown, the global grid map and the starting and ending point pose information are obtained, and the open set and closed set are initialized. The starting point is set as a node and the cost is set to 0, added to the open set, and then the iterative search phase is entered; In each round of iterative search, firstly, the node with the lowest cost is selected from the open set as the current child node, and then the current child node is removed from the open set and added to the closed set; Determine whether the distance between the current child node and the end point is not greater than a threshold If the conditions are met, try to use the Reeds-Shepp curve to directly generate the loading path. If successful, add the path to the closed set and end the iterative search. If unsuccessful, or the distance to the end point is greater than the threshold, , then explore through the current child node, first use Figure 4 As shown, the node-guided search rule and the deep reinforcement learning network generate the next child node and the guide node of a fixed step size respectively, and add them to the exploration child node set, then perform collision detection on the exploration child nodes in the exploration child node set to remove invalid child nodes and update the child node costs through the evaluation function, and finally add them to the open set to end this round of iterative search; after the path planning is successful and the iterative search ends, all nodes of the planned path can be obtained through node backtracking.

[0055] (2) Training strategy design For deep reinforcement learning training, appropriate starting and ending points are randomly selected on the map based on actual production data, and path planning is performed; In planning, the path planning parameters are optimized through reinforcement learning to form multiple sets of state transition sampling data and add them to the playback buffer; During the training process, batches of data are randomly selected from the playback buffer and the estimation network is updated according to the loss function. After a certain number of iterations, the estimated network The parameters are copied to the target network Complete a learning process.

[0056] Using two networks for training reduces the correlation between the current Q value and the target Q value to a certain extent, and improves the stability of the algorithm. During training, in order to improve generalization, the starting point and the end point can be randomly fine-tuned to enrich the training data and further improve generalization. The pseudo code of the DQN algorithm is shown in Table 1 below.

[0057] Table 1 DQN algorithm 1. Initialize the prediction network with random network parameters 2. Copy the same parameters to initialize the target network 3. Initialize the experience replay pool 4. for sequence do 5. Get the initial state of the environment 6.for time step do 7. According to the current network Selecting actions with a greedy strategy 8. Perform actions and get rewards , the environment state becomes 9. Store in the playback pool 10. If there is enough data in the data, sample data in batches 11. For each data, use the target network to calculate 12. Update the parameters of the network so that As close as possible 13. Every C updates make Here is the specific training algorithm of DQN, because the final training and fitting of the DQN algorithm is to The output of this function is the Q value, which is a good or bad evaluation of taking action a in the current state s (i.e., whether it is possible to obtain more rewards or reduce penalties). In the present invention, the actions are steering angles and step lengths. Therefore, in the specific implementation process, the corresponding Q value is generated for each permutation and combination of steering angles and step lengths according to the current state s, and the steering angle and step length with the highest Q value are selected and used as the steering angle and step length output by the deep reinforcement learning network to generate the guide node .

[0058] The algorithm here is the training algorithm of the DQN network. For specific usage of DQN, please refer to Figure 3 and Figure 4 In short, the status here Contains state space is the combination of steering angle and step length, that is The reward is in the corresponding state space Next action In this training algorithm, the greedy strategy is used to output the action first, and the reverse path planning training is carried out. According to the results of each round of search iteration in the training, Store them in the experience replay pool, perform batch sampling training, update network parameters, and learn the optimal strategy.

[0059] Second embodiment like Figure 5 As shown, this embodiment provides a rule- and data-hybrid-driven unmanned mine truck path planning method executed by the rule- and data-hybrid-driven unmanned mine truck path planning system as in the first embodiment, comprising the following steps: S1: Constructing node guided search rules, for any current child node, using a fixed step size and a steering angle after uniform discrete sampling to perform node exploration to obtain a set of explored child nodes of the next child node with a fixed step size, and generating a guiding node through deep reinforcement learning to add to the set of explored child nodes; S2: Evaluate the cost of each exploration sub-node in the exploration sub-node set, and obtain the exploration sub-node with the lowest cost as the next sub-node finally selected in each round of iterative search process; S3: When the distance from the current subnode to the end point is less than a set threshold, a berth curve from the current subnode to the end point is generated, thereby obtaining a final global curve.

[0060] A computer-readable storage medium stores computer code. When the computer code is executed, the above method is executed. A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program. The program can be stored in a computer-readable storage medium. The storage medium can include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0061] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

[0062] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0063] It should be noted that the above embodiments can be freely combined as needed. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention.

Claims

1. A rule- and data-driven unmanned mining truck path planning system, characterized in that: include: A hybrid A-star planning module is used to construct node guidance search rules. For any current sub-node, a fixed step size and a steering angle after uniform discrete sampling are used to perform node exploration to obtain a set of exploration sub-nodes of the next sub-node with a fixed step size. A guide node is generated through deep reinforcement learning and added to the exploration sub-node set. The cost of each exploration sub-node in the exploration sub-node set is evaluated to obtain the exploration sub-node with the lowest cost as the next sub-node finally selected in each subsequent round of iterative search. At the same time, when the distance from the current sub-node to the end point is less than a set threshold, a berth curve from the current sub-node to the end point is generated to obtain the final global curve. A guide node generation module, used to design the state space and action space of the current child node, and to design a neural network structure for generating the guide node. In each round of iteration, the state space is input into the neural network structure, the Q value of the steering angle and the step length in each group of the action space is output, the steering angle and the step length with the highest Q value are selected to generate the guide node, and a reward function is designed to guide the training of the neural network structure; The path exploration framework design module is used to design the path exploration framework and the training strategy for deep reinforcement learning.

2. The rule- and data-driven unmanned mining truck path planning system according to claim 1 is characterized in that: In the hybrid A-star planning module, the node guidance search rule is constructed, and for any current sub-node, a fixed step size and a steering angle after uniform discrete sampling are used to perform node exploration to obtain the exploration sub-node set of the next sub-node with a fixed step size, and the guidance node is generated by deep reinforcement learning and added to the exploration sub-node set, specifically: After obtaining the current child node, the steering angle Perform uniform discrete sampling, for , calculated as follows: in, is the boundary of the steering angle, corresponding to the maximum front wheel steering angle that the mine car can perform, i is the sequence number of the sampled steering angle, and n is the number of sampled steering angles; Considering the forward and backward exploration cases, the fixed step size and the sampled steering angle Substitute into the following formula (2) to obtain the set of explored subnodes for the next subnode: ; Among them, the current child node is , is the global coordinate of the current child node on the map, is the heading angle of the minecart at the current subnode, and any exploration subnode in the exploration subnode set is , To explore the global coordinates of the child nodes under the map, is the heading angle of the minecart at the exploration subnode, is the fixed step size of exploration, is the wheelbase of the mining truck, is the exploration direction of the node, i.e., backward or forward; Input the environment information and the state of the current child node into the deep reinforcement learning network, and output the guided step length and the guide steering angle , and substitute it into formula (2) to generate the guide node , the guiding node has a larger step size and a lower cost, and is added to the exploration subnode set, that is, .

3. The rule- and data-driven unmanned mining truck path planning system according to claim 2 is characterized in that: In the hybrid A-star planning module, the cost of each exploration sub-node in the exploration sub-node set is evaluated to obtain the exploration sub-node with the lowest cost as the next sub-node finally selected in each subsequent iterative search process, specifically: First, the effectiveness of each exploration sub-node in the exploration sub-node set is evaluated, that is, the mine cart cannot collide with environmental obstacles when it is located at the location of the exploration sub-node, and the arc path from the current sub-node to each exploration sub-node is sampled. At the corresponding discrete point position and orientation, the mine cart is simulated with a rectangle, and the mine cart is finely divided into grids. The corresponding grid on the global map is calculated for each subdivided grid, and the corresponding grid on the global map is queried whether it is occupied. If it is occupied, it means that there is a collision, that is, the exploration sub-node is an invalid sub-node, and it is removed from the exploration sub-node set. If no collision occurs at any discrete point on the arc path, it means that the exploration subnode is valid; On this basis, the cost evaluation function is established To evaluate each exploration subnode , specifically: in, is the current cost function, which represents the actual cost from the starting point to the explored subnode. is the heuristic cost function, which represents the cost of exploring the subnode to the end point. is the weight coefficient, For the current child node The cost basis includes 5 indicators, namely, distance cost , the cost of reversing , forward and backward motion switching cost , Turning cost and the cost of turning change , is the predicted value of the distance from the explored child node to the destination.

4. The rule- and data-driven unmanned mining truck path planning system according to claim 3 is characterized in that: In the hybrid A-star planning module, when the distance from the current subnode to the end point is less than the set threshold, the berth curve from the current subnode to the end point is generated, thereby obtaining the final global curve, specifically: If the current child node To the end The distance is less than the set threshold , generated from the current child node by the Reeds-Shepp curve To the end Several candidate loading berth curves are obtained, and the node costs on the berth curves are calculated by formula (3), and sorted according to the costs, and the path with the minimum cost is selected and reverse search is performed to obtain the final global path; If the berth curves of all candidate loads collide, the step of node exploration is continued.

5. The rule- and data-driven unmanned mining truck path planning system according to claim 4 is characterized in that: In the guiding node generation module, the state space and action space of the current child node are designed, specifically: The state space Designed for: （4） in, is the current child node, is the starting point node information, is the terminal node information, is the position and heading angle of the current child node relative to the starting node, is the position and heading angle of the end node relative to the current child node, is the distance between the position of the current child node and the end node, is the minimum distance between the obstacle and the minecart in the ith area; The exploration parameters of the guiding node include step size and steering angle, and the corresponding action space is , deep reinforcement learning adopts the DQN algorithm, which adopts a discrete action space and performs discrete sampling on the steering angle and step size.

6. The rule- and data-driven unmanned mining truck path planning system according to claim 5 is characterized in that: In the guiding node generation module, the neural network structure of the guiding node is designed and generated, specifically: The DQN algorithm uses two networks with exactly the same structure but different parameters for training, estimating the network To calculate the strategy selection Value and perform Iterative update of the value, the target network Used to calculate the next state in the time difference target TD Target Value, the loss function of the DQN algorithm is designed as follows: Where N is the number of samples, is the target Q value, through the target network Calculate the time difference target to get, The neural network structure is calculated in the state Take action Q value; The target network and the estimation network All of them are constructed using a three-layer fully connected network, where the number of neurons is 32. The outputs of the first two fully connected layers are input into the activation function and then output to the next fully connected layer. The activation function is set using PReLu. The last fully connected layer directly outputs the Q value of each action, including the steering angle and step length. Finally, the steering angle and step length with the highest Q value are selected as the final exploration parameters of the guide node and are substituted into formula (2) to generate the guide node.

7. The rule- and data-driven unmanned mining truck path planning system according to claim 6 is characterized in that: In the guide node generation module, a reward function is designed to guide the training of the neural network structure, specifically: The reward function includes a reward for reaching the destination , obstacle avoidance reward , explore the cost , Smoothness Reward and node selection rewards ; The destination reward The definition is as follows: in, is a large fixed reward constant, is the manually adjusted weight coefficient, The current child node in this iteration To the end node The Euclidean distance of is the guide node generated by the neural network structure To the end node The Euclidean distance of The obstacle avoidance reward The definition is as follows: in, and is an adjustable weight coefficient, is the distance threshold, when , considered a collision, returns a larger penalty constant ,when When , it is considered a dangerous situation and returns a larger penalty function. When , it is considered to be risky and a smaller penalty function is returned. When it is considered safe, no penalty will be returned, Represents the obstacle avoidance reward in the i-th area, with a total of n areas, and the overall obstacle avoidance reward Satisfy the following formula: The exploration cost The definition is as follows: in, is an adjustable penalty constant; The smoothness reward The definition is as follows: in, and is an adjustable weight coefficient, The boot node generated The corresponding steering angle, The current child node in the search iteration The corresponding steering angle; The node selection reward The definition is as follows: in, is an adjustable fixed reward constant, is the set of explored subnodes in this round of iterative search The steering angle corresponding to the best node finally selected in is an adjustable weight coefficient; The reward function is: 。 8. The rule- and data-driven unmanned mining truck path planning system according to claim 7 is characterized in that: In the path exploration framework design module, the path exploration framework is designed, specifically: Get the global grid map and the start and end point pose information, initialize the open set and closed set, set the start point as a node and set the cost to 0, add it to the open set, and then enter the iterative search phase; In each round of iterative search, firstly, the node with the lowest cost is selected from the open set as the current child node, and then the current child node is removed from the open set and added to the closed set; Determine whether the distance between the current child node and the end point is not greater than a threshold If the conditions are met, the loading path is directly generated by using the Reeds-Shepp curve. If successful, the loading path is added to the closed set and the iterative search ends. If unsuccessful, or the distance to the end point is greater than the threshold, , then the current child node is explored, and the next child node and the guide node with a fixed step length are generated by using the node-guided search rule and the deep reinforcement learning network respectively, and added to the exploration child node set, and then the exploration child nodes in the exploration child node set are subjected to collision detection to remove invalid child nodes and update the child node costs through the evaluation function, and finally added to the open set, ending this round of iterative search; After the path planning is successful and the iterative search is completed, all nodes of the planned path can be obtained through node backtracking.

9. The rule- and data-driven unmanned mining truck path planning system according to claim 8, characterized in that: In the path exploration framework design module, a deep reinforcement learning training strategy is designed, specifically: For deep reinforcement learning training, appropriate starting and ending points are randomly selected on the map based on actual production data, and path planning is performed; In planning, the path planning parameters are optimized through reinforcement learning to form multiple sets of state transition sampling data and add them to the playback buffer; During the training process, batches of data are randomly selected from the playback buffer and the estimation network is updated according to the loss function. The parameters of the estimated network are The parameters are copied to the target network Complete a learning process.

10. A method for unmanned mining truck path planning driven by a rule and data hybrid drive, which is executed by the unmanned mining truck path planning system driven by a rule and data hybrid drive according to any one of claims 1 to 9, characterized in that: The following steps are involved: S1: Constructing node guided search rules, for any current child node, using a fixed step size and a steering angle after uniform discrete sampling to perform node exploration to obtain a set of explored child nodes of the next child node with a fixed step size, and generating a guiding node through deep reinforcement learning to add to the set of explored child nodes; S2: Evaluate the cost of each exploration sub-node in the exploration sub-node set, and obtain the exploration sub-node with the lowest cost as the next sub-node finally selected in each round of iterative search process; S3: When the distance from the current subnode to the end point is less than a set threshold, a berth curve from the current subnode to the end point is generated, thereby obtaining a final global curve.

Images