An intelligent decision-making system for drones based on imitation learning and reinforcement learning

Abstract

The present invention discloses an intelligent decision-making system for unmanned aerial vehicles based on imitation learning and reinforcement learning, including an environmental perception module, an expert behavior demonstration module, a decision-making learning module, and a control execution module. The system can make accurate decisions for complex real-time scenarios, achieve the goal of flying from a starting point to a set end point, and can make autonomous decisions during the entire flight process, select unmanned aerial vehicle control strategies based on environmental information, real-time events, etc., avoid obstacles, and reach the target point safely and efficiently. Compared with the decision-making system of traditional expert rule design, the present invention clones the expert behavior by imitation learning to obtain the initial value of the decision control network, and then uses deep reinforcement learning to set the reward function according to the events and states in the process of executing the flight mission, and learns to obtain a more accurate and better generalized decision control Q network, which plays an important role in improving the autonomous control performance of unmanned aerial vehicles.

Description

Technical Field

[0001] The present invention belongs to the intersection field of unmanned aerial vehicle intelligent control decision-making and computer technology, and particularly relates to an unmanned aerial vehicle intelligent decision-making system based on imitation learning and reinforcement learning. Background Art

[0002] Drones are becoming more and more widely used in various fields due to their relatively low cost, good maneuverability and high safety factor. However, traditional drone control technology still requires a lot of human participation, and autonomous decision-making technology is still a bottleneck problem that limits the autonomous capabilities of drones. At present, the challenges faced by autonomous decision-making of drones are mainly: 1) The decision model requires a large amount of calculation and requires high real-time decision results; 2) It is difficult to establish an accurate mathematical model of drones in practical applications; 3) The environment is complex during the actual flight process, and there is a lot of missing and erroneous information, which increases the difficulty of decision-making.

[0003] The existing intelligent decision-making methods include expert systems, differential games, and dynamic programming. Expert systems can output decision information based on situation information and expert rules, but their over-reliance on rules makes the decision process lack flexibility and is not adaptable to complex environments. Differential game methods solve decision problems from the perspective of numerical optimization calculations, but they are based on accurate mathematical models of drones and are difficult to obtain in practical applications. Using approximate value function dynamic programming methods to solve decision problems has difficulties in establishing a unified standard decision model, and it is easy to cause dimensionality disaster when using numerical methods to solve. Unlike traditional decision-making methods, reinforcement learning is a learning method in which intelligent agents learn in a "trial and error" manner, and the rewards obtained by interacting with the environment guide actions, with the goal of making the intelligent agents obtain the maximum reward. Compared with traditional decision-making methods, reinforcement learning systems need to rely mainly on their own experience to learn, and the reinforcement signals provided by the environment evaluate the quality of the generated actions. The best model is determined by executing the actions that can obtain the maximum reward a limited number of times. This decision-making mechanism has shown great potential in the field of drone control. However, reinforcement learning is essentially a data-driven optimization algorithm that relies heavily on information provided by the external environment. How to obtain high-quality control decision data has become the main bottleneck limiting its application in the field of UAV measurement and control. Summary of the invention

[0004] In order to overcome the problems in the process of autonomous decision-making and control of UAVs, such as the weak adaptability of traditional decision-making methods and the difficulty in using reinforcement learning methods due to data acquisition issues, the present invention provides a UAV intelligent decision-making system based on imitation learning and reinforcement learning. It combines the imitation learning of human control knowledge with the adaptive reinforcement learning of complex environments, and implements the UAV decision-making model training in steps. It has strong generalization ability and high robustness when dealing with complex dynamic scenes.

[0005] The technical solution adopted by the present invention is as follows:

[0006] An intelligent decision-making system for unmanned aerial vehicles based on imitation learning and reinforcement learning, including an environment perception module, an expert behavior demonstration module, a decision learning module and a control execution module;

[0007] Environmental perception module: extracts features from environmental information collected by various sensors during the flight of the drone and fuses them to form a state space vector, which is output to the demonstration module and decision learning module; the environmental information includes images, directions and distances;

[0008] Expert behavior demonstration module: collects drone control instructions made by experts in various environments and events based on domain knowledge and operating experience. The drone control instructions and data provided by the environmental perception module together form an expert demonstration data set, which is output to the decision learning module;

[0009] Decision learning module: takes the state space vector as the input of the network structure, pre-learns the network structure according to the expert demonstration data set, obtains the pre-learning model, and conducts training based on the pre-learning model to learn the decision control actions that should be taken in different scenarios and different events, obtains the final decision learning model, and outputs the action vector and its probability to the control execution module;

[0010] Control execution module: After obtaining the actions and their probabilities output by the decision learning module, the action instruction with the highest probability is selected for execution by the UAV, and the new environmental state of the UAV after the action is executed is obtained.

[0011] The specific processing process of the environment perception module is as follows: the input is the environmental data collected by various sensors during the flight of the drone, the image data is extracted using the Resnet18 network to obtain high-dimensional features, and then the location information and distance information of the drone are fused to form a fused state space vector as the output. The state space vector s t =(l t , d t , x t，1 , x t，2 , ..., x t，m ), where S t is the state space vector of the UAV at time t, l t is its own position, d t is the distance between the current position and the target position, (x t，1 , x t，2 , ..., x t，m ) is the high-dimensional information of the image in the current field of view.

[0012] The specific processing process of the expert behavior demonstration module is as follows: the input is the state space vector output by the environmental perception module, and the expert knowledge is used to control the UAV in different scenes and at different times in the virtual reality simulation platform, and the control instruction sequence is output. The input state space vector and the output control instruction sequence together constitute the expert demonstration data set.

[0013] Among them, the decision learning module includes an imitation learning module and a reinforcement learning module;

[0014] The imitation learning module and the reinforcement learning module have the same network structure, which is implemented by the network structure Q. The input of the network structure is the state space vector obtained by the environment perception module. The input layer and the hidden layer are fully connected, and the hidden layer and the output layer are fully connected. The final output is the action space and its probability. The action space is the action vector a taken at time t. t =(a t，f , a t，b , a t，w , a t，e , a t，u , a t，d );

[0015] The imitation learning module performs pre-learning based on the expert demonstration dataset, and is trained using the environmental images, orientation, distance information in the expert demonstration dataset as input, as well as the drone control instructions output by the experts, to obtain the pre-learning model of the decision learning module; the reinforcement learning module performs reinforcement training based on the initialization parameters of the pre-learning model, learns the decision control actions that should be taken in different scenarios and different events, and obtains the final decision learning model.

[0016] Among them, the specific processing process of the control execution module is:

[0017] The action vector a output by the decision learning module t =(a t，f , a t，b , a t，w , a t，e , a t，u , a t，d ), select the action instruction with the highest action probability to execute, get the new environment state of the UAV after executing the action, and then iterate the entire process from environmental perception to control execution to realize autonomous control decision of the UAV.

[0018] The beneficial effects of the present invention are:

[0019] The present invention can make accurate decisions for complex real-time scenarios and enable the UAV to fly from the starting point to the set end point: 1) The entire flight process is completely autonomous, and the UAV control strategy can be selected according to environmental information, real-time events, etc. to avoid obstacles such as buildings and pedestrians; 2) There is no need to design expert rules, but the expert behavior is cloned by imitation learning to obtain the rough initial value of the decision control network and complete the initialization of the decision network; 3) The deep reinforcement learning method is adopted to set the reward function for the events and states during the flight process, so that the decision generation is more accurate and more generalized. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a design diagram of the decision network architecture of the system of the present invention.

[0021] Figure 2 It is a schematic diagram of the system composition and information interaction of the present invention.

[0022] Figure 3 It is a flow chart of decision network training of the system of the present invention. DETAILED DESCRIPTION

[0023] The present invention will be further described in detail below with reference to the accompanying drawings.

[0024] An intelligent decision-making system for unmanned aerial vehicles based on imitation learning and reinforcement learning includes an environmental perception module, an expert behavior demonstration module, a decision-making learning module (including an imitation learning module and a reinforcement learning module), and a control execution module. It should be noted that this system generally requires a virtual reality simulation platform to provide typical scenarios for simulating environmental data collection and the operation of unmanned aerial vehicle virtual models. This solution uses Microsoft's AirSim platform, which only provides flight scenarios and virtual unmanned aerial vehicle models for the system and does not affect the independence of the decision-making system. Therefore, it is not a component of this decision-making system. The construction and implementation methods of each module of this system are as follows:

[0025] 1. Environmental perception module: The environmental information collected by various sensors during the flight of the drone, including images, directions and distances, is extracted and integrated to form a state space vector to provide input data for the subsequent expert behavior demonstration module and decision learning module.

[0026] The extraction of environmental information obtained by various sensors mainly uses the Resnet18 network to extract image information to obtain high-dimensional features. The network extracts high-dimensional information from the environmental image through the input layer, convolution layer, pooling layer and fully connected layer; then, a fully connected layer is used to fuse the image features, orientation features and distance features into an environmental state vector s t =(l t , d t , x t，1 , xt，2 , ..., x t，m ), where s t is the state of the drone at time t, and the state vector contains its own position l t , the distance d between the current position and the target position t , and the high-dimensional information of the image in the current field of view (x t，1 , x t，2 , …, x t，m ).

[0027] The entire network structure and parameter design reference Figure 1 The input layer of the network uses the ResNet18 network, which contains 256 neurons. The input image is in 64*64 pixel 3-channel format. There are two hidden layers, with 128 and 64 neurons respectively, and the output layer has 6 neurons. The input layer and the hidden layer are fully connected, and the hidden layer and the output layer are fully connected. All neuron activation functions use the relu function, and the learning rate α is set to 0.01.

[0028] 2. Expert behavior demonstration module: By collecting the drone control instructions made by experts in various environments and various events based on their domain knowledge and operating experience, the drone control instructions and the data provided by the environmental perception module together constitute the expert demonstration data set, providing instruction learning samples for the imitation learning part of the subsequent decision learning module.

[0029] The input of this module is the state space vector s output by the environment perception module. t =(l t , d t , x t，1 , x t，2 , ..., x t，m ), the UAV operator (i.e., flight control expert) uses expert knowledge to control the handle controller (Xbox controller is used in this solution) to control the UAV in a large number of different scenarios and times in a virtual reality simulation platform (AirSim is used in this solution), and outputs a sequence of control instructions. The input state space vector and the output control instruction sequence constitute an expert demonstration dataset. The data of the expert demonstration dataset is the <state, action> matching pairs of all expert demonstrations for subsequent imitation learning.

[0030] 3. Decision learning module: including imitation learning module and reinforcement learning module. The two modules have the same network structure and are implemented by network structure Q. The input layer of the network structure is the state space vector obtained by the environment perception module. The input layer and the hidden layer are fully connected, and the hidden layer and the output layer are fully connected. The final output is the action space. The imitation learning module performs pre-learning based on the expert demonstration data set to obtain the pre-learning model of the decision learning module; reinforcement learning is trained on the basis of the pre-learning model to learn the decision control actions that should be taken in different scenarios and events to obtain the final decision learning model.

[0031] 3.1 The imitation learning module mainly performs pre-learning based on the expert demonstration data set, that is, imitation learning is performed through behavior cloning to obtain the initial learning model of the decision learning module as the initialization parameter of the Q network. The training process of imitation learning is as follows: the domain expert provides decision data {τ1, τ2, ..., τ m}, each decision data includes state and action sequence, that is, Then extract all <state, action> pairs to get a new data set:

[0032] D={(s1, a1), (s2, a2), (s3, a3),...}

[0033] The state is used as a feature and the action is used as a label for classification or regression to obtain the optimal decision-making strategy model. By collecting a large amount of operational flight data from human experts, the state s of the flight data is the environmental scene during flight, and the action A is the action in this scene. These data are input into the neural network Q network to make the output of the network as close as possible to the actual action taken by humans to complete the task. That is, the decision-making control strategy is learned based on the state-action pairs provided by human experts. This process is called behavioral cloning. Using imitation learning, the search space for free exploration of reinforcement learning can be reduced, providing a pre-training model for deep reinforcement learning.

[0034] 3.2 The reinforcement learning module mainly performs reinforcement training based on the initialization parameters of the Q network obtained in pre-learning, so as to reduce the search space and accelerate convergence. The implementation method is to further learn through deep reinforcement learning after the imitation learning module completes the initialization of the Q network. Specifically, the DQN algorithm is adopted. The DQN algorithm is an algorithm based on the value function. DQN updates the current state s t The Q value Q(s) of the next action A t , A), first execute action A, update one step to reach state s t+1 ; then put s t+1 Input Q network and calculate s t+1 The Q value of all actions is calculated, and the maximum Q value plus the reward R is taken as the update target; finally, Q(s t , A) and max(Q(st+1 , the difference between A))+R is used as the loss, and the loss is used to update the Q network, that is, the DQN update is performed according to the following formula:

[0035]

[0036] Among them, Q(s t , A) indicates that the agent is in state s t After selecting action A, the expected total reward until the final state, the reward function is set to the reward R related to the distance to the destination d and the reward R related to flight time t sum.

[0037] R=R d +R t

[0038] Destination-related rewards d The reward is: reaching the destination will get a larger reward, the closer the distance d to the destination is, the larger the reward will be, and hitting an obstacle will get a negative reward. That is:

[0039]

[0040] Time-related negative reward (penalty) R t It increases with the increase of flight time, so that the flight mission can be completed in a shorter time. The termination condition for reaching the final state is set as reaching the destination, flying to the map boundary, or completing the set number of explorations. Meeting any of these conditions means reaching the final state. After continuous iterative training, an accurate Q network can be obtained as the core network of the decision learning module of the UAV intelligent decision-making system.

[0041] 3.3 Q network model training process reference for decision learning module Figure 3At the beginning of training, the weights of the decision learning network are initialized by the parameters obtained by imitation learning. Step 3-1, parameter setting, including the initial value N of the number of cyclic training, the set flight target point, and the boundary setting of the map. Step 3-2, set the maximum number of explorations n in one training. Step 3-3, obtain the current environmental information and input it into the environmental perception module. Step 3-4, extract the features of the environmental information to obtain the state space vector of the environment. Step 3-5, input the state space vector into the Q network to obtain the probability of each action in the current state. Step 3-6, calculate the error δ between the actual value of the network and the updated target. Step 3-7, use the error δ to update the weights in the network according to the back propagation algorithm. Step 3-8, according to the output action probability, select the action with the highest probability to execute and obtain the next state. Step 3-9, determine whether the current state is the final state, based on whether the maximum number of explorations n for one training has been reached, whether the target point has been reached, and whether the map boundary has been exceeded. If the final state has not been reached, repeat the above steps to continue exploring. If the final state has been reached, execute step 3-10, and the number of trainings is increased by 1. Step 3-11, determine whether the number of trainings has reached the set number of cyclic trainings. If not, continue with a new training. If reached, the training process ends.

[0042] 4. Control execution module: responsible for the specific execution of the decision learning module. After obtaining the probability of each action in the action space output by the decision learning module, the action instruction with the highest probability is selected for execution by the UAV, thereby obtaining the new environmental state of the UAV after executing the action.

[0043] The action vector a output by the decision learning module t =(a t，f , a t，b , a t，w , a t，e , a t，u , a t，d ) The probability of each action is calculated, and the action instruction with the highest probability is selected for execution, so as to obtain the new environmental state of the UAV after executing the action. Then, the entire process from environmental perception to control execution is iterated in a loop to realize autonomous control decision-making of the UAV.

Claims

1. A UAV intelligent decision-making system based on imitation learning and reinforcement learning, characterized in that: It includes environment perception module, expert behavior demonstration module, decision learning module and control execution module; Environmental perception module: extracts features from environmental information collected by various sensors during the flight of the drone and fuses them to form a state space vector, which is output to the demonstration module and decision learning module; the environmental information includes images, directions and distances; Expert behavior demonstration module: collects drone control instructions made by experts in various environments and events based on domain knowledge and operating experience. The drone control instructions and data provided by the environmental perception module together form an expert demonstration data set, which is output to the decision learning module; Decision learning module: takes the state space vector as the input of the network structure, pre-learns the network structure according to the expert demonstration data set, obtains the pre-learning model, and conducts training based on the pre-learning model to learn the decision control actions that should be taken in different scenarios and different events, obtains the final decision learning model, and outputs the action vector and its probability to the control execution module; Control execution module: After obtaining the actions and their probabilities output by the decision learning module, the action instruction with the highest probability is selected for execution by the UAV, and the new environmental state of the UAV after the action is executed is obtained; Among them, the decision learning module includes an imitation learning module and a reinforcement learning module; The imitation learning module and the reinforcement learning module have the same network structure, which is implemented by the network structure Q. The input of the network structure is the state space vector obtained by the environment perception module. The input layer and the hidden layer are fully connected, and the hidden layer and the output layer are fully connected. The final output is the action space and its probability. The action space is the action vector a taken at time t. t =(a t,f ,a t,b ,a t,w ,a t,e ,a t,u ,a t,d ); The imitation learning module performs pre-learning based on the expert demonstration data set, and is trained with the environmental images, orientation, and distance information in the expert demonstration data set as input, as well as the drone control instructions output by the experts, to obtain the pre-learning model of the decision learning module; the reinforcement learning module performs reinforcement training based on the initialization parameters of the pre-learning model, learns the decision control actions that should be taken in different scenarios and different events, and obtains the final decision learning model; Among them, the specific processing process of the control execution module is: The action vector a output by the decision learning module t =(a t,f ,a t,b ,a t,w ,a t,e ,a t,u ,a t,d ), select the action instruction with the highest action probability to execute, get the new environment state of the UAV after executing the action, and then iterate the entire process from environmental perception to control execution to realize autonomous control decision of the UAV.

2. The UAV intelligent decision-making system based on imitation learning and reinforcement learning according to claim 1 is characterized in that: The specific processing process of the environment perception module is as follows: the input is the environmental data collected by various sensors during the flight of the drone, the image data is extracted using the Resnet18 network to obtain high-dimensional features, and then the location information and distance information of the drone are fused to form a fused state space vector as the output. The state space vector s t =(l t ,d t ,x t,1 ,x t,2 ,…,x t,m ), where s t is the state space vector of the UAV at time t, l t is its own position, d t is the distance between the current position and the target position, (x t,1 ,x i,2 ,…,x t,m ) is the high-dimensional information of the image in the current field of view.

3. The UAV intelligent decision-making system based on imitation learning and reinforcement learning according to claim 1 is characterized in that: The specific processing process of the expert behavior demonstration module is as follows: the input is the state space vector output by the environmental perception module, and the expert knowledge is used to control the UAV in different scenes and at different times in the virtual reality simulation platform, and the control instruction sequence is output. The input state space vector and the output control instruction sequence together constitute the expert demonstration data set.

Images