An unmanned intelligent vehicle path planning method and unmanned intelligent vehicle

Abstract

The application relates to an unmanned intelligent vehicle path planning method and an unmanned intelligent vehicle, and belongs to the technical field of path planning, wherein the method comprises the following steps: acquiring state information of the unmanned intelligent vehicle at each time step; the state information comprises position information and traffic congestion information of an area where the unmanned intelligent vehicle is located; a moving action is determined according to the state information, and an instant reward of the unmanned intelligent vehicle after executing the moving action is determined; the instant reward comprises a traffic congestion negative reward; a Q value is calculated according to the instant reward through a Q-Learning algorithm; a path planning strategy is learned according to the Q value; and a target unmanned intelligent vehicle to be path planned is path planned according to the learned target path planning strategy. When the path planning strategy is used for path planning, real-time traffic congestion information can be combined for path planning, hot areas of traffic congestion can be avoided, balanced distribution of road network load can be realized, and the utilization efficiency of the whole road network is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of path planning, in particular to a path planning method for an unmanned intelligent vehicle and the unmanned intelligent vehicle. BACKGROUND

[0002] With the rapid development of intelligent manufacturing and automated logistics, the deployment scale of unmanned intelligent vehicle systems in industrial workshops, warehouse logistics parks and unmanned production lines continues to expand. In a complex dynamic environment, unmanned intelligent vehicles need to coordinate to complete tasks such as material transfer, equipment inspection and cargo sorting, and the rationality of their path planning directly affects system efficiency and safety.

[0003] Existing path planning methods for unmanned intelligent vehicles, such as the A-Star Algorithm, Dijkstra's algorithm and the Rapidly-exploring Random Tree (RRT) algorithm, have solved the problem of path planning for unmanned intelligent vehicles to some extent, but in the context of unmanned intelligent vehicle coordination, they still have high computational complexity and can cause local congestion while leaving other areas relatively idle, making it difficult to meet the high-efficiency coordination needs of unmanned intelligent vehicles in complex dynamic environments. SUMMARY

[0004] Therefore, it is necessary to provide a path planning method for an unmanned intelligent vehicle and the unmanned intelligent vehicle to solve the problem of high computational complexity of existing path planning algorithms and the inability of planned paths to effectively utilize the overall road network.

[0005] To solve the above problems, in a first aspect, the present application provides a path planning method for an unmanned intelligent vehicle, comprising:

[0006] obtaining state information of the unmanned intelligent vehicle at each time step; the state information includes position information and traffic congestion information of the area where the unmanned intelligent vehicle is located;

[0007] determining a movement action based on the state information, and determining an immediate reward after the unmanned intelligent vehicle performs the movement action;

[0008] calculating a Q value based on the immediate reward by a Q-Learning algorithm;

[0009] learning a path planning strategy based on the Q value;

[0010] planning a path for a target unmanned intelligent vehicle to be planned based on the learned target path planning strategy;

[0011] wherein the immediate reward is calculated by the following formula:

[0012]

[0013]

[0014] In the formula, t For time steps, For autonomous intelligent vehicles in status Perform movement actions at time The immediate reward afterwards For autonomous intelligent vehicles in status Perform movement actions at time The subsequent positive reward, Negative incentives for traffic congestion. Let the traffic congestion cost function be... The traffic congestion penalty coefficient is a non-negative hyperparameter. and As a weighting factor, For autonomous intelligent vehicles in status Perform movement actions at time The local traffic density of the area where the autonomous vehicle is located. For autonomous intelligent vehicles in status Perform movement actions at time The path overlap rate after that.

[0015] In one possible implementation, calculating the Q-value based on the reward using the Q-Learning algorithm includes:

[0016] The Q value is calculated using the following formula:

[0017]

[0018] In the formula, In the state Perform movement actions at time Q-value estimation at time, To The Q-value estimate obtained by updating For learning rate, As a discount factor, From state The maximum Q value among all moving actions initiated.

[0019] In one possible implementation, the immediate reward also includes at least one of the following: a negative reward for encountering an obstacle, a conflict negative reward, and a deadlock negative reward.

[0020] In one possible implementation, the conflict negative reward includes negative rewards for oncoming conflicts and intersection conflicts.

[0021] In a possible implementation, the state information of the unmanned intelligent vehicle at each time step is acquired, including:

[0022] A grid environment map is constructed according to the moving environment of the unmanned intelligent vehicle.

[0023] Position information of the unmanned intelligent vehicle in the grid environment map is determined, and traffic congestion information of an area where the unmanned intelligent vehicle is located is determined according to the position information.

[0024] In a possible implementation, the moving action is determined according to the state information, including:

[0025] The moving action is determined according to the state information by using a greedy strategy.

[0026] In a possible implementation, the moving action is determined according to the state information, including:

[0027] The moving action is determined according to the state information by using a DQN model.

[0028] The Q value is calculated according to the instant reward by using a Q-Learning algorithm, including:

[0029] The Q value is calculated according to the instant reward by using the DQN model.

[0030] The path planning strategy is learned according to the Q value, including:

[0031] A loss value is determined according to the Q value, and parameters of the DQN model are adjusted in a reverse direction according to the loss value until the DQN model is trained completely; wherein the target path planning strategy is included in the DQN model trained completely.

[0032] In a possible implementation, the method further includes:

[0033] Samples are extracted from an experience replay pool to train the DQN model.

[0034] In a second aspect, the present application further provides an unmanned intelligent vehicle, including a memory and a processor, wherein the memory is used to store a program; the processor is coupled with the memory and is used to execute the program stored in the memory to realize steps in the unmanned intelligent vehicle path planning method.

[0035] The present application has the following advantages:

[0036] The application adopts a reinforcement learning method based on a Q-Learning algorithm to learn a target path planning strategy, in the learning process, traffic congestion information is introduced into the state information of the unmanned intelligent vehicle, after the unmanned intelligent vehicle selects a moving action according to the traffic congestion information, the immediate reward of the unmanned intelligent vehicle after executing the moving action is calculated, wherein the immediate reward includes a traffic congestion negative reward, then the Q value is calculated according to the immediate reward, and the path planning strategy is learned according to the Q value, so that when the target path planning strategy learned is used for path planning, the real-time traffic congestion information can be combined for path planning, the traffic congestion hotspot area can be avoided, the balanced distribution of the road network load can be realized, and the overall utilization efficiency of the road network is improved. In addition, the Q-Learning algorithm is used to learn the path planning strategy, while ensuring the calculation efficiency, the optimal path planning strategy, i.e., the target path planning strategy, can be learned and found more quickly. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0038] Figure 1 A flowchart of an embodiment of the unmanned intelligent vehicle path planning method provided by the present application;

[0039] Figure 2 An embodiment structure diagram of the unmanned intelligent vehicle provided by the present application. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings of the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0041] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more. The "first", "second", and the like referred to in the embodiments of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence, nor to indicate or imply that the relative importance of the objects indicated or implied. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are usually a class, and do not limit the number of objects, for example, the first object can be one or more.

[0042] Reference herein to "embodiments" means that the specific features, structures, or characteristics described in connection with the embodiments can be included in at least one embodiment of the present application. The phrase appears at various places in the specification does not necessarily all refer to the same embodiments, nor is it necessarily independent or alternative embodiments to other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0043] Reference Figure 1 , shows a flowchart of an embodiment of the path planning method of the unmanned intelligent vehicle provided by the present application, the method comprising:

[0044] S101, obtaining the state information of the unmanned intelligent vehicle at each time step; the state information includes position information and traffic congestion information of the area where the unmanned intelligent vehicle is located.

[0045] The unmanned intelligent vehicle is an intelligent mobile vehicle based on artificial intelligence (AI), automatic driving, sensor fusion and decision control technology, which can autonomously perceive the environment, plan the path and execute the task without or with limited human intervention. The unmanned intelligent vehicle can include unmanned logistics delivery vehicles, unmanned industrial operation vehicles, unmanned special operation vehicles, etc.

[0046] The position information can include the positioning information of the unmanned intelligent vehicle corresponding to the time step, and the traffic congestion information can include the number of vehicles in the area where the unmanned intelligent vehicle is located, the degree of traffic congestion in the area where the unmanned intelligent vehicle is located, etc.

[0047] S102, determining a moving action according to the state information, and determining the immediate reward of the unmanned intelligent vehicle after executing the moving action.

[0048] The moving action can be selected based on the state information by a greedy strategy, and then the immediate reward of the unmanned intelligent vehicle after executing the moving action is determined according to a preset reward function, which includes a traffic congestion negative reward obtained by comparing and evaluating the traffic congestion information before and after executing the moving action.

[0049] Specifically, the instant reward is calculated using the following formula:

[0050]

[0051]

[0052] In the formula, t For time steps, For autonomous intelligent vehicles in status Perform movement actions at time The immediate reward afterwards For autonomous intelligent vehicles in status Perform movement actions at time The subsequent positive reward, Negative incentives for traffic congestion. Let the traffic congestion cost function be... The traffic congestion penalty coefficient is a non-negative hyperparameter. and As a weighting factor, For autonomous intelligent vehicles in status Perform movement actions at time The local traffic density of the area where the autonomous vehicle is located. For autonomous intelligent vehicles in status Perform movement actions at time The path overlap rate after that.

[0053] In this embodiment, positive rewards can reflect the movement actions performed by the autonomous vehicle. The positive benefits afterward include approaching the target location and successfully avoiding obstacles, where the target location can be the endpoint of the path planning. Local traffic density can be calculated by the number of vehicles within a preset area centered on the autonomous vehicle. Path overlap rate can be calculated based on the number of vehicles within the autonomous vehicle's movement actions. Then, the path overlap rate between the path already traveled by the autonomous vehicle and the paths already traveled by other autonomous vehicles is calculated. When there are multiple other autonomous vehicles, the path overlap rate between the autonomous vehicle and each of the other autonomous vehicles can be calculated separately, and then the path overlap rates can be added together and divided by the number of other autonomous vehicles.

[0054] S103 calculates the Q value based on immediate rewards using the Q-Learning algorithm.

[0055] Q-Learning is a classic model-free reinforcement learning algorithm, belonging to the temporal difference learning method. It learns an action-value function (Q-function), enabling the agent to find the optimal policy through trial and error in an unknown environment.

[0056] The calculated Q value can be used to update the Q value in the Q table (path planning strategy).

[0057] S104, learning the path planning strategy according to the Q value.

[0058] When the algorithm converges, the optimal path planning strategy, i.e., the target path planning strategy, can be obtained.

[0059] S105, path planning for the target unmanned intelligent vehicle to be path planned according to the learned target path planning strategy.

[0060] The unmanned intelligent vehicle path planning method provided by the embodiment can be applied to an unmanned intelligent vehicle path planning system, which can be a software system running on a terminal device. The terminal device can be a tablet computer, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a mobile phone, or the like. The embodiment does not limit the specific type of the terminal device.

[0061] To sum up, the embodiment adopts a reinforcement learning method based on the Q-Learning algorithm to learn the target path planning strategy. In the learning process, traffic congestion information is introduced into the state information of the unmanned intelligent vehicle. After the unmanned intelligent vehicle selects a moving action according to the traffic congestion information, the reward value of the unmanned intelligent vehicle after executing the moving action is calculated. The reward value includes a traffic congestion negative reward. Then, the Q value is calculated according to the reward value, and the path planning strategy is learned according to the Q value. This can enable the target path planning strategy learned through learning to perform path planning in combination with real-time traffic congestion information, avoid hot areas of traffic congestion, achieve balanced distribution of road network load, and improve the overall utilization efficiency of the road network. In addition, the embodiment learns the path planning strategy through the Q-Learning algorithm, which can quickly learn and find the optimal path planning strategy, i.e., the target path planning strategy, while ensuring the calculation efficiency.

[0062] The existing Q value update strategy is shown in the following formula:

[0063]

[0064] In the formula, t is the time step, is the state , and Q-value estimation of the time, is an update of Q-value estimation, is a learning rate, is a discount factor, is a maximum Q-value of all moving actions starting from state , is an immediate reward.

[0065] In some embodiments of the present application, an improved reward mechanism is introduced in the above Q-value update strategy, and local congestion information is integrated into the reward and punishment value of the Q-Learning algorithm, to obtain the following Q-value update strategy:

[0066]

[0067] In the formula, is an immediate reward calculated according to the negative reward of traffic congestion.

[0068] In some embodiments of the present application, the immediate reward can further include at least one of a negative reward when encountering an obstacle, a conflict negative reward, and a deadlock negative reward.

[0069] The conflict can include head-on conflict and intersection conflict. The head-on conflict refers to two or more unmanned intelligent vehicles moving in opposite directions in the same lane or narrow space. The intersection conflict refers to a conflict caused by path intersection of multiple unmanned intelligent vehicles at an intersection without traffic lights or a complex intersection. The conflict negative reward can include negative rewards of head-on conflict and intersection conflict.

[0070] Deadlock refers to a state in which multiple unmanned intelligent vehicles are trapped in an infinite blocking state and cannot continue to travel because they are waiting for each other to release resources (such as lanes and intersections). For example, four vehicles each occupy one side at a crossroads and wait for other vehicles to go first (similar to a “four-way parking deadlock”). For example, resource competition deadlock: multiple vehicles simultaneously apply for the same narrow passage and do not yield to each other.

[0071] In an example of the present embodiment, the immediate reward can be calculated by the following formula:

[0072]

[0073] In the formula, represents a positive reward of executing a moving action to reach a target position, represents a negative reward when encountering an obstacle, represents a traffic congestion negative reward.

[0074] The embodiment can make the target path planning strategy obtained through learning adjust the path in real time to cope with dynamic obstacles, conflicts and deadlocks and the like while ensuring global optimality when the path planning is performed by the target path planning strategy.

[0075] In some embodiments of the present application, S101 comprises:

[0076] constructing a grid environment map according to the moving environment of the unmanned intelligent vehicle;

[0077] determining position information of the unmanned intelligent vehicle in the grid environment map, and determining traffic congestion information of an area where the unmanned intelligent vehicle is located according to the position information.

[0078] In some embodiments of the present application, the moving environment of the unmanned intelligent vehicle comprises a workshop, a logistics park or a production line.

[0079] When the Q-Learning algorithm is used for path planning of the unmanned intelligent vehicle in the grid environment, the core principle is to use the powerful search capability of a computer to find an optimal solution through continuous trial and error of the agent and the environment. However, when the number of unmanned intelligent vehicles is large, the search speed of the algorithm decreases, the time consumption becomes long, and it is difficult to achieve efficient path planning in a complex scene. Therefore, a path planning algorithm based on a deep reinforcement learning deep Q network (DQN) is considered to achieve path planning of the unmanned intelligent vehicle. The DQN model is an extended version of Q-Learning. Under the premise of retaining the core idea of Q-Learning, the algorithm introduces a deep neural network to approximate the Q function, so that reinforcement learning can be applied to a high-dimensional state space. At the same time, the experience replay mechanism improves the data utilization rate, reduces the training time, and improves the sample processing efficiency and training stability.

[0080] Therefore, in some embodiments of the present application, the step of determining a moving action according to the state information can comprise determining the moving action according to the state information through a DQN model. S103 can comprise calculating a Q value according to the immediate reward through the DQN model. S104 can comprise determining a loss value according to the Q value, and reversely adjusting parameters of the DQN model according to the loss value until the DQN model is trained completely; wherein the DQN model trained completely comprises a target path planning strategy.

[0081] The DQN model trained completely learns an optimal mapping relationship (target path planning strategy) between the state information and the moving action. Through the mapping relationship, the unmanned intelligent vehicle can make corresponding moving actions according to the state information of each time step.

[0082] In the embodiment, the loss value is calculated by the following formula:

[0083]

[0084] In the formula, is the Q value estimation of the action network of the DQN model for the state and action , represents that the target network calculates the Q values of all possible actions corresponding to the state , and then selects the maximum Q value among them.

[0085] In some embodiments of the application, first, a regular grid environment map is constructed as the running scene of the unmanned intelligent vehicle, the state of the unmanned intelligent vehicle is defined as the position in the grid, and the action space includes static and movement in four directions. A reward function is designed to guide the unmanned intelligent vehicle to learn, for example, a positive reward is given when reaching the target position, and a negative reward is given when encountering obstacles or conflicts or congestion.

[0086] Then, the Q table is initialized, at each time step, the unmanned intelligent vehicle perceives the current state, and according to - the greedy strategy selects the movement action, obtains the immediate reward after executing the action and moves to the new state, and then adjusts the Q value in the Q table according to the Q-Learning update rule. After multiple training and parameter optimization, the algorithm converges, forming an optimal path planning strategy (finally determined Q table) considering load balancing, which can make the unmanned intelligent vehicle avoid load hotspot areas and achieve balanced distribution of road network load, improving the overall operation efficiency of the system.

[0087] Further, the DQN model can be used instead of the Q table, and the process of learning the optimal path planning strategy through the DQN model is as follows: initialize the DQN model, the input of the DQN model is the state information, and the output is the Q value of the corresponding movement action; at each time step, the DQN model obtains the current state, and according to - the greedy strategy selects the action, the unmanned intelligent vehicle obtains the immediate reward after executing the movement action and moves to the new state, and the state transition data is put into the experience replay pool; samples are randomly extracted from the experience replay pool for training the DQN model, updating the Q value; the parameters of the target network of the DQN model are frozen regularly to stabilize the learning process. After multiple training and parameter optimization, the algorithm converges, forming the optimal path planning strategy of the unmanned intelligent vehicle, ensuring that the unmanned intelligent vehicle system can operate efficiently and stably.

[0088] The embodiment constructs a gridding model of the environment map, and combines a Q-Learning reinforcement learning method to realize accurate path initialization planning and ensure global feasibility of path planning. Compared with a traditional path planning method, the embodiment uses Q-Learning to search and optimize the path, continuously updates a state-action value function (Q value), ensures calculation efficiency, improves the accuracy of path planning, and enables the intelligent unmanned vehicle to learn and find a feasible path more quickly.

[0089] In terms of global path optimization, the Q-Learning can gradually learn an optimal path strategy by designing a reasonable state space, action space and reward function, and can minimize path conflicts in the intelligent unmanned vehicle collaborative scenario. Compared with a traditional path planning method, the Q-Learning can continuously optimize the path planning strategy through autonomous learning in a dynamic environment, reduce the calculation complexity, improve the adaptability of path planning, and enable the intelligent unmanned vehicle system to achieve better task execution efficiency in a complex environment. The scheme has moderate calculation load, wider applicability and stronger robustness, and is particularly suitable for intelligent manufacturing, warehouse logistics and disaster rescue and other application scenarios with high requirements for real-time and collaboration of path planning.

[0090] Reference Figure 2 , an intelligent unmanned vehicle 300 provided by the application is shown. The intelligent unmanned vehicle 300 includes a processor 301, a memory 302 and a display 303. Figure 2 Only part of the components of the intelligent unmanned vehicle 300 are shown, but it should be understood that all the shown components are not required to be implemented, and more or fewer components can be alternatively implemented.

[0091] The processor 301 can be a central processing unit (CPU), a microprocessor or other data processing chip in some embodiments, used to run program codes or process data stored in the memory 302, such as the intelligent unmanned vehicle path planning method in the application.

[0092] In some embodiments, the processor 301 can be a single server or a server group. The server group can be centralized or distributed. In some embodiments, the processor 301 can be local or remote. In some embodiments, the processor 301 can be implemented in a cloud platform. In an embodiment, the cloud platform can include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, a multiple cloud, etc., or any combination thereof.

[0093] The memory 302 can be an internal storage unit of the unmanned intelligent vehicle 300 in some embodiments, such as a hard disk or a memory of the unmanned intelligent vehicle 300. The memory 302 can also be an external storage device of the unmanned intelligent vehicle 300 in other embodiments, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, and the like.

[0094] Further, the memory 302 can include both an internal storage unit and an external storage device of the unmanned intelligent vehicle 300. The memory 302 is used to store application software and various data installed in the unmanned intelligent vehicle 300.

[0095] The display 303 can be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, and the like in some embodiments. The display 303 is used to display information of the unmanned intelligent vehicle 300 and to display a visualized user interface. The components 301-303 of the unmanned intelligent vehicle 300 communicate with each other through a system bus.

[0096] In an embodiment, when the processor 301 executes the unmanned intelligent vehicle path planning program in the memory 302, the following steps can be implemented:

[0097] Obtaining state information of the unmanned intelligent vehicle at each time step; the state information includes position information and traffic congestion information of an area where the unmanned intelligent vehicle is located;

[0098] Determining a movement action according to the state information, and determining an immediate reward after the unmanned intelligent vehicle performs the movement action;

[0099] Calculating a Q value according to the immediate reward through a Q-Learning algorithm;

[0100] Learning a path planning strategy according to the Q value;

[0101] Planning a path for a target unmanned intelligent vehicle to be planned according to the learned target path planning strategy;

[0102] Wherein, the immediate reward is calculated through the following formula:

[0103]

[0104]

[0105] In the formula, t is a time step, t is a time step, is a state of the unmanned intelligent vehicle when the mobile action is performed the immediate reward after is a negative reward for traffic congestion, when the mobile action is performed the positive reward after is a negative reward for traffic congestion, is a cost function for traffic congestion, is a penalty coefficient for traffic congestion, which is a non-negative hyperparameter, and is a weighting factor, when the mobile action is performed the local traffic density of the area where the unmanned intelligent vehicle is located after when the mobile action is performed the path overlap rate after when the mobile action is performed .

[0106] It should be understood that, in addition to the above functions, the processor 301 can also implement other functions when executing the unmanned intelligent vehicle path planning program in the memory 302. For details, please refer to the description of the corresponding method embodiments.

[0107] The embodiment adopts a reinforcement learning method based on a Q-Learning algorithm to learn a target path planning strategy. In the learning process, traffic congestion information is introduced into the state information of the unmanned intelligent vehicle. After the unmanned intelligent vehicle selects a mobile action according to the traffic congestion information, the reward value of the unmanned intelligent vehicle after performing the mobile action is calculated. The reward value includes a negative reward for traffic congestion. Then, the Q value is calculated according to the reward value, and the path planning strategy is learned according to the Q value. The target path planning strategy learned in this way can combine real-time traffic congestion information for path planning, avoid hot areas of traffic congestion, achieve balanced distribution of road network load, and improve the overall utilization efficiency of the road network. In addition, the present application can learn and find the optimal path planning strategy, i.e., the target path planning strategy, more quickly while ensuring the calculation efficiency through the Q-Learning algorithm for path planning strategy learning.

[0108] Those skilled in the art can understand that all or part of the processes of the above embodiments can be completed by a computer program instructing related hardware. The program can be stored in a computer readable storage medium, such as a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0109] The above merely describes preferred specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. An unmanned intelligent vehicle path planning method, characterized in that, The method comprises: acquiring state information of the unmanned intelligent vehicle at each time step; the state information comprises position information and traffic congestion information of an area where the unmanned intelligent vehicle is located; determining a moving action according to the state information, and determining an immediate reward after the unmanned intelligent vehicle performs the moving action; calculating a Q value according to the immediate reward through a Q-Learning algorithm; learning a path planning strategy according to the Q value; planning a path for a target unmanned intelligent vehicle to be planned according to the learned target path planning strategy; wherein the immediate reward is calculated through the following formula: In the formula, t For time steps, For autonomous intelligent vehicles in status Perform movement actions at time The immediate reward afterwards For autonomous intelligent vehicles in status Perform movement actions at time The subsequent positive reward, Let the traffic congestion cost function be... The traffic congestion penalty coefficient is a non-negative hyperparameter. and As a weighting factor, For autonomous intelligent vehicles in status Perform movement actions at time The local traffic density of the area where the autonomous vehicle is located. For autonomous intelligent vehicles in status Perform movement actions at time The path overlap rate after the intersection; wherein the Q value is calculated according to the immediate reward through a Q-Learning algorithm, comprising: calculating the Q value through the following formula: wherein is the Q-value estimate at state when performing a move is the Q-value estimate at state is the Q-value estimate obtained by updating is the Q-value estimate obtained by updating is the learning rate, is the discount factor, is the maximum Q-value among all move actions starting from state is the maximum Q-value among all move actions starting from state the acquiring of the state information of the unmanned intelligent vehicle at each time step comprises: constructing a grid environment map according to a moving environment of the unmanned intelligent vehicle; determining position information of the unmanned intelligent vehicle in the grid environment map, and determining traffic congestion information of an area where the unmanned intelligent vehicle is located according to the position information; the determining of the moving action according to the state information comprises: determining the moving action according to the state information through a DQN model; the calculating of the Q value according to the immediate reward through the Q-Learning algorithm comprises: calculating the Q value according to the immediate reward through the DQN model; the learning of the path planning strategy according to the Q value comprises: determining a loss value according to the Q value, and reversely adjusting parameters of the DQN model according to the loss value until the DQN model is trained completely; wherein the DQN model trained completely comprises the target path planning strategy. 2.The path planning method of the intelligent unmanned vehicle according to claim 1, characterized in that, The immediate reward further comprises at least one of a negative reward when an obstacle is encountered, a conflict negative reward, and a deadlock negative reward. 3.The path planning method of the intelligent unmanned vehicle according to claim 2, characterized in that, The conflict negative reward comprises negative rewards of opposite conflict and intersection conflict. 4.The path planning method of the intelligent unmanned vehicle according to claim 1, characterized in that, The determining of the moving action according to the state information comprises: determining the moving action according to the state information through a greedy strategy. 5.The path planning method of the intelligent unmanned vehicle according to claim 1, characterized in that, The method further comprises: extracting samples from an experience replay pool to train the DQN model.

6. An unmanned intelligent vehicle, characterized by The device comprises a memory and a processor, wherein: the memory is used to store a program; the processor is coupled with the memory, and is used to execute the program stored in the memory to realize steps in the unmanned intelligent vehicle path planning method in any one of claims 1 to 5.

Images