Unmanned vehicle control method and device based on deep reinforcement learning, equipment and medium

Abstract

The invention belongs to the technical field of unmanned vehicle control, and relates to an unmanned vehicle control method and device based on deep reinforcement learning, equipment and a medium. The method comprises the steps of obtaining environment information of an unmanned aerial vehicle, and constructing an initial state space; according to the initial state space, a deep reinforcement learning method is adopted, values of different actions in the initial state space are calculated through the initial first network, strategy selection is carried out, and an initial optimal action is obtained; a next first network is obtained according to the initial optimal action, and the unmanned vehicle moves at the same time; obtaining a next state space according to the environment information after the unmanned vehicle moves, obtaining a next optimal action through a next first network, and generating an action value set until a preset condition is met; and the environment information of the unmanned aerial vehicle is obtained again, the execution action of the unmanned vehicle is output according to the action value set, and unmanned vehicle control is realized. According to the invention, unmanned vehicle control without SLAM mapping can be realized.

Description

Technical Field

[0001] The present application relates to the field of unmanned vehicle control technology, and in particular to unmanned vehicle control methods, devices, equipment and media based on deep reinforcement learning. Background Art

[0002] With the development of the times and the advancement of technology, the control methods of unmanned vehicles are also changing with each passing day.

[0003] In the existing technology, the control of unmanned vehicle robots relies on SLAM technology. SLAM technology realizes unmanned vehicle navigation by simultaneously performing self-localization and map construction. Typical SLAM algorithms include EKF-SLAM (Extended Kalman Filter SLAM), FastSLAM and Graph-Based SLAM. The advantage of SLAM technology is that it can provide high-precision maps and positioning information.

[0004] However, the disadvantages of SLAM are high computational complexity, the need for a large amount of storage memory and computing resources, strong dependence on sensors, limited performance in dynamic and complex environments, inability to use SLAM to build maps during the first navigation, and limited recognition of the environment without preset knowledge. Summary of the invention

[0005] Based on this, it is necessary to provide an unmanned vehicle control method, device, equipment and medium based on deep reinforcement learning to address the above-mentioned technical problems, which can realize unmanned vehicle control without SLAM mapping.

[0006] Unmanned vehicle control method based on deep reinforcement learning, including: Obtain the drone’s environmental information and construct the initial state space; According to the initial state space, a deep reinforcement learning method is used to calculate the values ​​of different actions in the initial state space through the initial first network, and a strategy is selected to obtain an initial optimal action; According to the initial optimal action, the corresponding loss function is calculated to update the initial first network to obtain the next first network; according to the initial optimal action, the unmanned vehicle moves; According to the environmental information after the unmanned vehicle moves, the initial state space is updated to obtain the next state space, and the value of different actions in the next state space is calculated through the next first network to obtain the next optimal action. When the preset conditions are met, an action value set is generated according to the value of different actions in different state spaces; Re-acquire the environmental information of the drone, output the execution action of the unmanned vehicle according to the action value set, and realize the control of the unmanned vehicle.

[0007] In one embodiment, obtaining the environmental information of the drone and constructing the initial state space includes: Obtain environmental information of the drone; Environmental information includes: the distance between the unmanned vehicle and the target point, the angle between the unmanned vehicle and the target point, the linear speed of the unmanned vehicle, the angular speed of the unmanned vehicle, and the distance between the unmanned vehicle and each obstacle; The initial state space is constructed based on the distance between the unmanned vehicle and the target point, the angle between the unmanned vehicle's direction and the line connecting the target point, the unmanned vehicle's linear speed, the unmanned vehicle's angular speed, and the distance between the unmanned vehicle and each obstacle.

[0008] In one embodiment, the values ​​of different actions in the initial state space are calculated by the initial first network, including: Through the initial first network, the state value function and the action value function of different actions in the initial state space are calculated; according to the state value function and the action value function, the value of different actions in the initial state space is obtained.

[0009] In one embodiment, strategy selection is performed to obtain an initial optimal action, including: Through the initial second network, strategy selection is performed, and the action with the highest value is taken as the initial optimal action.

[0010] In one embodiment, the second network is an Actor network. In one embodiment, the Actor network is updated via policy gradients and gradient clipping.

[0011] In one embodiment, the first network is a Critic network.

[0012] The unmanned vehicle control device based on deep reinforcement learning includes: The acquisition module is used to obtain the environmental information of the UAV and construct the initial state space; A calculation module, used to calculate the value of different actions in the initial state space through the initial first network according to the initial state space, adopt a deep reinforcement learning method, and select a strategy to obtain an initial optimal action; An updating module is used to calculate a corresponding loss function according to the initial optimal action to update the initial first network and obtain a next first network; the unmanned vehicle moves according to the initial optimal action; The training module is used to update the initial state space according to the environmental information after the unmanned vehicle moves, obtain the next state space, and calculate the value of different actions in the next state space through the next first network to obtain the next optimal action, until the preset conditions are met, and generate an action value set according to the value of different actions in different state spaces; The control module is used to reacquire the environmental information of the UAV, output the execution action of the unmanned vehicle according to the action value set, and realize the control of the unmanned vehicle.

[0013] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented: Obtain the drone’s environmental information and construct the initial state space; According to the initial state space, a deep reinforcement learning method is used to calculate the values ​​of different actions in the initial state space through the initial first network, and a strategy is selected to obtain an initial optimal action; According to the initial optimal action, the corresponding loss function is calculated to update the initial first network to obtain the next first network; according to the initial optimal action, the unmanned vehicle moves; According to the environmental information after the unmanned vehicle moves, the initial state space is updated to obtain the next state space, and the value of different actions in the next state space is calculated through the next first network to obtain the next optimal action. When the preset conditions are met, an action value set is generated according to the value of different actions in different state spaces; Re-acquire the environmental information of the drone, output the execution action of the unmanned vehicle according to the action value set, and realize the control of the unmanned vehicle.

[0014] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps: Obtain the drone’s environmental information and construct the initial state space; According to the initial state space, a deep reinforcement learning method is used to calculate the values ​​of different actions in the initial state space through the initial first network, and a strategy is selected to obtain an initial optimal action; According to the initial optimal action, the corresponding loss function is calculated to update the initial first network to obtain the next first network; according to the initial optimal action, the unmanned vehicle moves; According to the environmental information after the unmanned vehicle moves, the initial state space is updated to obtain the next state space, and the value of different actions in the next state space is calculated through the next first network to obtain the next optimal action. When the preset conditions are met, an action value set is generated according to the value of different actions in different state spaces; Re-acquire the environmental information of the drone, output the execution action of the unmanned vehicle according to the action value set, and realize the control of the unmanned vehicle.

[0015] The above-mentioned unmanned vehicle control method, device, equipment and medium based on deep reinforcement learning, the unmanned vehicle uses the deep reinforcement learning algorithm to learn the optimal movement strategy. In an unknown environment, the unmanned vehicle makes decisions based on the environmental information obtained in real time, including selecting a suitable path and taking obstacle avoidance actions. The deep reinforcement learning algorithm can be trained in advance through interactive learning with the environment, and can quickly identify environmental information for control, and continuously optimize the movement strategy of the unmanned vehicle, so that it can safely and efficiently traverse complex environments, reduce the computational complexity, and reduce the required storage memory and computing resources. The deep reinforcement learning method can learn the dynamic characteristics of the environment and the distribution of obstacles through interaction with the environment, and continuously optimize its passing ability through continuous learning, so that the unmanned vehicle can accurately decide how to move next in the face of different similar environments, so as to achieve rapid deployment and continuous progress. The unmanned vehicle can make decisions autonomously based on the learned strategy, and can adapt to different environments and scenarios, with a certain degree of autonomy and robustness. In addition, the present application is a SLAM-free technology, in which the unmanned vehicle can perceive, make decisions and act in an unknown environment in real time, and build an environmental map in real time without the need to explicitly build a map of the environment in advance, so that the navigation algorithm of the unmanned vehicle during movement does not rely on SLAM mapping, and can perform positioning and mapping in an unknown environment, providing the necessary information support for path planning and obstacle avoidance, and realizing navigation, movement, obstacle avoidance and exploration in unknown environments, with higher real-time and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is an application scenario diagram of an unmanned vehicle control method based on deep reinforcement learning in one embodiment; Figure 2 1 is a flow chart of an unmanned vehicle control method based on deep reinforcement learning in one embodiment; Figure 3 A training framework diagram of an unmanned vehicle control method based on deep reinforcement learning in one embodiment; Figure 4 A physical picture of an unmanned vehicle in one embodiment; Figure 5 This is one of the unmanned vehicle reinforcement learning training scene diagrams in one embodiment; Figure 6 This is a second diagram of a reinforcement learning training scenario for an unmanned vehicle in an embodiment; Figure 7 Figure 3 is a training scenario of unmanned vehicle reinforcement learning in one embodiment; Figure 8 is a structural block diagram of an unmanned vehicle control device based on deep reinforcement learning in one embodiment; Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment.

[0017] Reference numerals: Unmanned vehicle A, obstacle B, radar beam line C. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0019] It should be noted that all directional indications in the embodiments of the present application (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0020] In addition, the descriptions of "first", "second", etc. in this application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "multiple groups" means at least two groups, such as two groups, three groups, etc., unless otherwise clearly and specifically defined.

[0021] In this application, unless otherwise clearly specified and limited, the terms "connection", "fixation", etc. should be understood in a broad sense. For example, "fixation" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, an electrical connection, a physical connection, or a wireless communication connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0022] In addition, the technical solutions between the various embodiments of the present application can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0023] The unmanned vehicle control method based on deep reinforcement learning provided in this application can be applied to Figure 1In the application scenario diagram shown, the terminal 102 communicates with the server 104 through the network, and the terminal 102 may include but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices, and the server 104 may be various portal websites, servers corresponding to the backend of the work system, etc.

[0024] This application provides a method for controlling an unmanned vehicle based on deep reinforcement learning. Figure 2 The flowchart shown in FIG. 1 shows a method for applying the method to Figure 1 The following is an example of a terminal in the example, including: Step 202, obtain the environmental information of the UAV and construct an initial state space.

[0025] Specifically: Obtaining the environmental information of the UAV; the environmental information includes: the distance between the UAV and the target point, the angle between the UAV’s direction and the line connecting the target point, the UAV’s linear speed, the UAV’s angular speed, and the distance between the UAV and each obstacle; The initial state space is constructed based on the distance between the unmanned vehicle and the target point, the angle between the unmanned vehicle's direction and the line connecting the target point, the unmanned vehicle's linear speed, the unmanned vehicle's angular speed, and the distance between the unmanned vehicle and each obstacle.

[0026] For example: the state space [distance, theta, action[0], action[1], laserdat[0], …, laserdat

[19] ], where distance represents the distance between the unmanned vehicle and the target point, theta represents the angle between the unmanned vehicle and the line connecting the target point, action[0] represents the linear velocity of the unmanned vehicle, action[1] represents the angular velocity of the unmanned vehicle, and laserdat[0], …, laserdat

[19] represent the distance between the unmanned vehicle and each obstacle.

[0027] In this step, for indoor multi-obstacle scenes, deep learning technology is used to process and analyze sensor data to obtain environmental information state. Sensor data includes: unmanned vehicle odometer information, camera images, and lidar scanning data. The distance between the unmanned vehicle and the target point is obtained based on the unmanned vehicle odometer information, the angle between the unmanned vehicle's direction and the target point is obtained based on the camera image, and the distance between the unmanned vehicle and each obstacle is obtained based on the lidar scanning data. Through the deep learning model, the unmanned vehicle can more accurately understand key information such as objects and road conditions in the environment. The state input uses the unmanned vehicle's forward 180° lidar scanning data, which is divided into an interval of every 9°. The maximum value in this interval is selected as a 20-dimensional data, the angle between the unmanned vehicle's forward direction and its destination, and the speed output by the unmanned vehicle are used as parameter inputs.

[0028] Step 204, based on the initial state space, a deep reinforcement learning method is used to calculate the values ​​of different actions in the initial state space through the initial first network, and a strategy is selected to obtain the initial optimal action.

[0029] Specifically: According to the initial state space, the deep reinforcement learning method is used to calculate the state value function and action value function of different actions in the initial state space through the initial first network; according to the state value function and action value function, the value of different actions in the initial state space is obtained; According to the value of different actions in the initial state space, the strategy is selected through the initial second network, and the action with the highest value is taken as the initial optimal action.

[0030] In this step, in the training environment, after receiving the environmental information state, the deep reinforcement learning method generates an action value action through a random or deterministic strategy to train the first network and the second network. The action will act on the unmanned vehicle and interact with the surrounding environment, mainly completing path planning and obstacle avoidance functions, collecting experience and rewards through interaction, and storing the experience data in the playback buffer, from which batch data is regularly collected for learning.

[0031] The first network is the Critic network.

[0032] The Critic network is responsible for estimating the value of taking different actions in different states, that is, estimating the value function. It collects data through interaction with the environment and uses this data to adjust its own parameters to approximate the real value function. The value function includes the state value function or the action value function. The state value function represents the long-term cumulative reward that can be obtained by taking a certain action in the current state, and the action value function represents the expected cumulative reward starting from the next state after taking a certain action in the current state. Collect a batch of data from the replay buffer (the storage space for reinforcement learning action samples), calculate the minimum value function estimate of each sample taking different actions in different states, update the value function to the target point (the final value function), and then use the loss function to update the Critic network parameters. The action selection here is obtained by the Actor network. The Critic network learns a value function through the data collected by the Actor network and the environment. This function is used to judge the quality of the action, and then helps the Actor network to update the strategy.

[0033] The state value function can represent the The immediate reward after taking an action Plus the next state after considering the discount factor The expected value of the value function under is:

[0034] in, For the current state The state value after taking an action, For the next moment in state The state value after taking an action, For status, Update the reward for the algorithm, which means averaging the random variables to reduce the impact of environmental uncertainty in reinforcement learning; The state of the driverless car at the next moment The immediate reward after taking an action. For example, in order to prevent the driverless car from deliberately staying in the same place, a reward is set for each driverless car that will be punished continuously if it stays in the same place. is the discount factor, which represents the weighted addition of the state value of the previous step to the state value of the next step. It is a parameter between 0 and 1, which is used to determine the importance of future rewards relative to immediate rewards. If is close to 1, then future rewards are as important as immediate rewards. Close to 0, then only the immediate reward matters; For the next moment ( t +1 time) is the next state, For the current moment ( t moment); The action value function can represent the Take action The immediate reward after plus the next state after considering the discount factor The expected cumulative reward value of the best action value among all possible actions is:

[0035] in, For the current state Take action The action value that can be obtained by following the optimal strategy; For the next moment in state Take action The action value that can be obtained by following the optimal strategy; It is the maximum value operator, which means to select the action with the largest expected reward among all possible actions, which reflects the "greedy" strategy, that is, to select the best action in a given state; For the next moment ( t +1 moment) action, For the current moment ( t moment) action; The second network is the Actor network.

[0036] The Actor network is responsible for learning the policy function and the probability distribution of actions selected when a state is given. That is, in a given state, through the combination with the Critic network, the Actor network can output an action that is most likely to bring high value based on the current state. Collect a batch of data from the playback buffer, calculate the policy gradient for each batch, and then perform gradient clipping and update the Actor network parameters. The output of the Actor network is a probability distribution of selected actions, which represents the probability of each action being selected. In the discrete action space, the softmax function can be used to convert the network output into a probability distribution; in the continuous action space, the mean of the action can be directly output, and then appropriate constraints can be added as needed to ensure the legitimacy of the action. The Actor network interacts with the environment and uses policy gradients to learn a better policy network under the guidance of the Critic network value function.

[0037] In deep reinforcement learning training, learning is done through path planning and obstacle avoidance, and interaction with the environment, which involves randomly selecting a series of actions and observing the feedback from the environment. The selection of these actions is based on a value function approach, which aims to evaluate the potential value of each action to guide decision-making. The value function can help the agent make the best action decision by estimating the long-term cumulative reward of taking an action in the current state. During the training process, by constantly interacting with the environment and adjusting the estimate of the value function based on feedback, the agent gradually improves its decision-making ability, allowing it to act more intelligently and efficiently in complex environments.

[0038] The reward function is:

[0039] in, For the current moment ( t The state of the moment, For the current moment ( t moment) action, As a reward for reaching your destination, is the distance to the destination, is the distance threshold, For the reward of collision, If a collision occurs, For speed, is the angular velocity, For situations other than reaching the destination and collision; When the unmanned vehicle reaches the destination, a positive maximum reward is returned; when the unmanned vehicle encounters an obstacle, a negative maximum reward is returned; in other cases, the reward is based on the difference between the unmanned vehicle's speed and direction and the angle between the unmanned vehicle's position and the destination's position; when the unmanned vehicle moves towards the destination, a reward is given, and the greater the speed, the greater the reward. This training can ensure that the unmanned vehicle runs as fast as possible.

[0040] Step 206, according to the initial optimal action, the corresponding loss function is calculated to update the initial first network to obtain the next first network; according to the initial optimal action, the unmanned vehicle moves.

[0041] Specifically: According to the initial optimal action, a loss function corresponding to the initial optimal action is calculated; according to the loss function, the parameters of the initial first network are updated to obtain the next first network; Based on the initial optimal action, the unmanned vehicle executes and thus completes the movement.

[0042] In this step, deep reinforcement learning (DRL) is a machine learning method that combines deep learning technology with reinforcement learning technology (RL), uses neural networks to process complex high-dimensional state and action spaces, and the agent learns the optimal strategy through interaction with the environment to maximize the cumulative reward. In the RL framework, the agent perceives the state of the environment, takes action, and obtains reward signals from the environment to adjust its strategy, enabling the agent to autonomously navigate and avoid obstacles in dynamic and complex environments, overcoming the limitations of traditional SLAM methods.

[0043] Step 208, based on the environmental information after the unmanned vehicle moves, the initial state space is updated to obtain the next state space, and the values ​​of different actions in the next state space are calculated through the next first network to obtain the next optimal action, until the preset conditions are met, and an action value set is generated based on the values ​​of different actions in different state spaces.

[0044] Specifically: According to the environmental information after the unmanned vehicle moves, the initial state space is updated to obtain the next state space; According to the next state space, the deep reinforcement learning method is used to calculate the value of different actions in the next state space through the next first network, and the strategy is selected to obtain the next optimal action; According to the next optimal action, the unmanned vehicle moves to obtain new environmental information for iterative looping until the preset conditions are met, at which point the iteration stops and the values ​​of different actions in different state spaces are obtained. The training process is completed using the values ​​of different actions in different state spaces as the action value set.

[0045] In this step, after the unmanned vehicle moves, the environmental information changes, and the unmanned vehicle interacts with the new environmental information to update the initial state space.

[0046] Step 210, reacquire the environmental information of the UAV, output the execution action of the unmanned vehicle according to the action value set, and realize the control of the unmanned vehicle.

[0047] In this step, the environmental information of the UAV is input into the action value set obtained after training, and the execution action of the unmanned vehicle is output. The execution action of the unmanned vehicle includes path planning and obstacle avoidance planning, thereby realizing the control of the unmanned vehicle.

[0048] like Figure 3 As shown in the training framework diagram, this application adopts a perception system based on deep learning. The unmanned vehicle is responsible for receiving observation data of the external environment through the perception system and the environment, and passing this data to the deep reinforcement learning method. The training model can be used to more accurately understand and interpret sensor data, thereby improving the unmanned vehicle's ability to perceive the environment; the deep reinforcement learning algorithm is used, and the optimal action strategy can be learned in the process of continuous interaction with the environment, so that the unmanned vehicle can flexibly respond to various complex road conditions and obstacles, and realize path planning and action decision-making. Compared with the existing technology, sensor data processing is usually time-consuming and easily interfered by environmental noise, and path planning is usually based on pre-defined map information. It performs poorly in the face of dynamic environments. This application has great advantages.

[0049] The above-mentioned unmanned vehicle control method based on deep reinforcement learning solves the problem that navigation must rely on SLAM mapping and cannot navigate and move to avoid obstacles in unknown environments. The unmanned vehicle uses the deep reinforcement learning algorithm to learn the optimal mobile strategy. In an unknown environment, the unmanned vehicle makes decisions based on the real-time environmental information, including choosing a suitable path and taking obstacle avoidance actions. The deep reinforcement learning algorithm can be trained in advance through interactive learning with the environment, and can quickly identify environmental information for control, and continuously optimize the mobile strategy of the unmanned vehicle, so that it can safely and efficiently traverse complex environments, reducing the computational complexity, and reducing the required storage memory and computing resources. The deep reinforcement learning method can learn the dynamic characteristics of the environment and the distribution of obstacles through interaction with the environment, and continuously optimize its passing ability through continuous learning, so that the unmanned vehicle can accurately decide how to move next in the face of different similar environments, so as to achieve rapid deployment and continuous progress. The unmanned vehicle can make decisions autonomously based on the learned strategy, and can adapt to different environments and scenarios, with a certain degree of autonomy and robustness. In addition, the present application is a SLAM-free technology, which allows unmanned vehicles to perceive, make decisions and act in real time in unknown environments, and build environmental maps in real time during movement without the need to explicitly build maps of the environment in advance, so that the navigation algorithm of the unmanned vehicle during movement does not rely on SLAM mapping, and can perform positioning and mapping in unknown environments, providing the necessary information support for path planning and obstacle avoidance, and realizing navigation, mobile obstacle avoidance and exploration in unknown environments, with higher real-time and adaptability.

[0050] In a specific embodiment, migration control of an unmanned vehicle is performed.

[0051] Hardware requirements: The actual picture of the four-wheel drive unmanned vehicle is as follows Figure 4 As shown, it is equipped with a lidar, odometer, and Nvidia NX computing board.

[0052] Model deployment: Deploy the trained model files on the Nvidia NX computing board.

[0053] Sensor data acquisition: Ensure that sensor devices can normally obtain observation data of the environment and pass this data to the deployed model.

[0054] Training environment: including tables, random obstacles, various special-shaped obstacles, and various corner simulations.

[0055] Real-time control: Publish reachable target points and start the unmanned vehicle.

[0056] At the beginning of training: the unmanned vehicle will start at a fixed position and move towards the target point; random actions will be selected to calculate rewards and stored in the Critic network.

[0057] like Figure 5 In the training scenario shown, when encountering an obstacle, the unmanned vehicle will choose the direction of movement based on the current radar information and the distance information between the target point and the currently unknown one.

[0058] like Figure 6 The training scenario shown increases the redundancy of the environment by designing random boxes so that the trained driverless car will not overfit.

[0059] like Figure 7 The training scenario shown demonstrates the ability of the unmanned vehicle to reach the target point under different positions, where the starting point and the target point are randomly generated (not shown), A is the unmanned vehicle, B is the obstacle, and C is the radar beam line.

[0060] It should be understood that although Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0061] This application also provides an unmanned vehicle control device based on deep reinforcement learning, such as Figure 8 As shown, in one embodiment, it includes: an acquisition module 802, a calculation module 804, an update module 806, a training module 808 and a control module 810, wherein: The acquisition module 802 is used to obtain the environmental information of the UAV and construct an initial state space; A calculation module 804 is used to calculate the value of different actions in the initial state space by using a deep reinforcement learning method according to the initial state space through an initial first network, and to select a strategy to obtain an initial optimal action; An updating module 806 is used to calculate a corresponding loss function according to the initial optimal action to update the initial first network and obtain a next first network; the unmanned vehicle moves according to the initial optimal action; The training module 808 is used to update the initial state space according to the environmental information after the unmanned vehicle moves, obtain the next state space, and calculate the value of different actions in the next state space through the next first network to obtain the next optimal action, until the preset conditions are met, and generate an action value set according to the value of different actions in different state spaces; The control module 810 is used to reacquire the environmental information of the drone, output the execution action of the unmanned vehicle according to the action value set, and realize the control of the unmanned vehicle.

[0062] For the specific definition of the unmanned vehicle control device based on deep reinforcement learning, please refer to the definition of the unmanned vehicle control method based on deep reinforcement learning above, which will not be repeated here. Each module in the above device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0063] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for controlling an unmanned vehicle based on deep reinforcement learning is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0064] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0065] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in the above embodiment when executing the computer program.

[0066] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in the above embodiment are implemented.

[0067] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0068] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0069] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. The unmanned vehicle control method based on deep reinforcement learning is characterized by: include: Obtain the drone’s environmental information and construct the initial state space; According to the initial state space, a deep reinforcement learning method is used to calculate the values ​​of different actions in the initial state space through the initial first network, and a strategy is selected to obtain an initial optimal action; According to the initial optimal action, the corresponding loss function is calculated to update the initial first network to obtain the next first network; according to the initial optimal action, the unmanned vehicle moves; According to the environmental information after the unmanned vehicle moves, the initial state space is updated to obtain the next state space, and the value of different actions in the next state space is calculated through the next first network to obtain the next optimal action. When the preset conditions are met, an action value set is generated according to the value of different actions in different state spaces; Re-acquire the environmental information of the drone, output the execution action of the unmanned vehicle according to the action value set, and realize the control of the unmanned vehicle.

2. The unmanned vehicle control method based on deep reinforcement learning according to claim 1, characterized in that: Obtain the drone's environmental information and construct the initial state space, including: Obtain environmental information of the drone; Environmental information includes: the distance between the unmanned vehicle and the target point, the angle between the unmanned vehicle and the target point, the linear speed of the unmanned vehicle, the angular speed of the unmanned vehicle, and the distance between the unmanned vehicle and each obstacle; The initial state space is constructed based on the distance between the unmanned vehicle and the target point, the angle between the unmanned vehicle's direction and the line connecting the target point, the unmanned vehicle's linear speed, the unmanned vehicle's angular speed, and the distance between the unmanned vehicle and each obstacle.

3. The unmanned vehicle control method based on deep reinforcement learning according to claim 2 is characterized in that: The initial first network calculates the value of different actions in the initial state space, including: Through the initial first network, the state value function and the action value function of different actions in the initial state space are calculated; according to the state value function and the action value function, the value of different actions in the initial state space is obtained.

4. The unmanned vehicle control method based on deep reinforcement learning according to any one of claims 1 to 3, characterized in that: Select a strategy and get the initial optimal action, including: Through the initial second network, strategy selection is performed, and the action with the highest value is taken as the initial optimal action.

5. The unmanned vehicle control method based on deep reinforcement learning according to claim 4 is characterized in that: The second network is the Actor network.

6. The unmanned vehicle control method based on deep reinforcement learning according to claim 5 is characterized in that: The actor network is updated via policy gradients and gradient clipping.

7. The unmanned vehicle control method based on deep reinforcement learning according to any one of claims 1 to 3, characterized in that: The first network is the Critic network.

8. An unmanned vehicle control device based on deep reinforcement learning, characterized in that: include: The acquisition module is used to obtain the environmental information of the UAV and construct the initial state space; A calculation module, used to calculate the value of different actions in the initial state space through the initial first network according to the initial state space, adopt a deep reinforcement learning method, and select a strategy to obtain an initial optimal action; An updating module is used to calculate a corresponding loss function according to the initial optimal action to update the initial first network and obtain a next first network; the unmanned vehicle moves according to the initial optimal action; The training module is used to update the initial state space according to the environmental information after the unmanned vehicle moves, obtain the next state space, and calculate the value of different actions in the next state space through the next first network to obtain the next optimal action, until the preset conditions are met, and generate an action value set according to the value of different actions in different state spaces; The control module is used to reacquire the environmental information of the UAV, output the execution action of the unmanned vehicle according to the action value set, and realize the control of the unmanned vehicle.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images