Unmanned aerial vehicle target sensing and tracking simulation method based on deep reinforcement learning

Abstract

The invention discloses an unmanned aerial vehicle target sensing and tracking simulation method based on deep reinforcement learning. The method comprises the following steps: constructing a virtual environment through an Airsim simulation platform; designing a protocol connection layer; constructing an algorithm control module; the algorithm control module comprises an environment module, a target sensing and tracking module, an intelligent agent training module and an experience playback pool module; the environment module is used for receiving environment observation information of the virtual environment; the target sensing and tracking module is used for controlling the flight of the unmanned aerial vehicle in the virtual environment and executing a target sensing and tracking task; the agent training module is used for generating an action control instruction of a flight control action; the experience playback pool module is used for receiving experience data generated by interaction of the agent training module and the virtual environment, and packaging and storing the experience data for training of the agent training module; model training; and after training is completed, the target sensing and tracking module loads model parameters, executes a target sensing and tracking task in a virtual environment, and verifies the performance and stability of an algorithm.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of unmanned aerial vehicle target autonomous control and simulation, and particularly relates to a simulation method for unmanned aerial vehicle target perception and tracking based on deep reinforcement learning. BACKGROUND

[0002] With the increasing application of unmanned aerial vehicles in public safety, agriculture and logistics, the performance requirements of unmanned aerial vehicle target perception and tracking algorithms in various fields have also increased accordingly. In order to verify the performance of unmanned aerial vehicle target perception and tracking algorithms, simulation is needed. The existing simulation methods for unmanned aerial vehicle target perception and tracking play an important role in technology verification, algorithm development and system evaluation, but still have the following defects and limitations: (1) distortion of simulation environment and physical characteristics; most existing simulation platforms are based on simplified or idealized environment models, and lack sufficient modeling of real-world complexity and uncertainty. Specifically, the sensor noise model is too regular, and it is difficult to simulate the nonlinear distortion of real optical, radar and other sensors under weather changes and electromagnetic interference; the scene dynamic elements and physical interaction coupling are insufficient; in addition, the material reflection characteristics and kinematic response of the target and background are mostly static parameters, which leads to significant domain differences between simulation data and real collected data, affecting the reliability of model migration. (2) Limitations of scalability and open ecosystem; most simulation platforms rely on specific software frameworks or private tool chains, have low modularity, and the coupling between sensor models, environment engines and algorithm interfaces is tight, which is not conducive to the rapid integration of new sensors or perception algorithms. At the same time, open source simulation resources often have difficulty in balancing fidelity, completeness and real-time performance, limiting their application in industrial-level research and development. SUMMARY

[0003] The purpose of the present application is to provide a simulation method for unmanned aerial vehicle target perception and tracking based on deep reinforcement learning, which aims to overcome the shortcomings of existing simulation methods in terms of environment fidelity, system scalability and algorithm migration reliability.

[0004] In order to achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0005] A simulation method for unmanned aerial vehicle target perception and tracking based on deep reinforcement learning, comprising the following steps:

[0006] S1, build an experimental system architecture, the experimental system architecture includes an Airsim simulation platform, a protocol connection layer and an algorithm control module; the Airsim simulation platform is used to build a simulation training environment; the protocol connection layer is used to realize the data flow interaction between the Airsim platform and the algorithm control module; the algorithm control module is used to control the interaction between the simulation object and the virtual environment, make action decisions, train models and complete the simulation target perception and tracking task;

[0007] S2, constructing a virtual environment through the Airsim simulation platform;

[0008] S3, designing the protocol connection layer;

[0009] S4, constructing the algorithm control module; the algorithm control module includes an environment module, a target perception and tracking module, an agent training module, and an experience replay pool module;

[0010] The environment module interacts with the Airsim simulation platform through the protocol connection layer, and is configured to receive environment observation information of the virtual environment;

[0011] The target perception and tracking module is configured to control the flight of a UAV in the virtual environment of the Airsim simulation platform, and perform an end-to-end target perception and tracking task;

[0012] The agent training module is configured to generate an action control instruction of a flight control action;

[0013] The experience replay pool module is configured to receive experience data generated by the interaction between the agent training module and the virtual environment, and encapsulate and store the experience data for use by the agent training module for training;

[0014] S5, model training;

[0015] The algorithm control module performs model training under different scenarios of the Airsim simulation platform, collects sequences of states, actions, rewards, and next states, stores the sequences in the experience replay pool module, saves model parameters obtained by training after completing the model training, and

[0016] S6, task implementation effect evaluation and verification; after training is completed, the target perception and tracking module loads the model parameters, executes a target perception and tracking task in the virtual environment, and verifies the performance and stability of the algorithm.

[0017] Further, the environment observation information includes UAV coordinate data received by a GPS sensor in the virtual environment and a depth image containing a target UAV obtained by a front camera of the tracking UAV in the virtual environment.

[0018] Further, the agent training module is based on a deep neural network, takes the obtained depth image containing the target UAV as input, performs feature extraction and action evaluation by the deep neural network, makes an action selection, and outputs a corresponding action control instruction, and the UAV in the virtual environment is controlled to fly according to the action control instruction by the target perception and tracking module.

[0019] Further, the target perception and tracking module loads the corresponding environment module and the trained model, controls the flight of the unmanned aerial vehicle in the virtual environment through the flight control API interface of the Airsim simulation platform, and saves the unmanned aerial vehicle flight formation queue data to create a new thread to synchronize the tracking trajectory drawing.

[0020] Further, the protocol connection layer transmits the environment observation information in the virtual environment to the algorithm control module through the API interface, so that the algorithm control module receives the required environment state information, and transmits the action value output by the algorithm control module back to the virtual environment, so that the virtual environment receives the decision information of the algorithm control module and performs corresponding updating.

[0021] Further, the environment module calculates the state reward in each state, packs the state, action and reward information based on time sequence and specified format, and puts them into the experience replay pool module for model training; after the model training of the agent training module is completed, the model parameters are saved, and the model parameters are loaded during performance verification to complete the target perception and tracking task.

[0022] From the above technical solutions, the simulation method is based on the AirSim simulation platform, a virtual environment containing various complex scenes is constructed, and a separate protocol connection layer is designed to realize efficient and decoupled data interaction between the simulation platform and the external algorithm control module; the algorithm control module adopts a deep reinforcement learning control architecture, takes the depth image obtained by the unmanned aerial vehicle in real time as the main input, directly outputs the flight control instruction through the deep neural network, realizes the integrated decision from the original perception to the tracking control, and contains the environment interaction, experience storage, network training and task execution modules, supports the model training by using the deep learning algorithm, trains and tests the model in multiple differentiated virtual scenes, evaluates the algorithm performance by using the tracking trajectory, error curve and other multi-dimensional indexes, and verifies the generalization ability and robustness. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 The flowchart of the simulation method of the application is shown in the figure;

[0024] Figure 2 The framework diagram of the experimental system architecture of the embodiment of the application is shown in the figure

[0025] Figure 3 The flowchart of the virtual environment built by the embodiment of the application is shown in the figure;

[0026] Figure 4 The block diagram of the algorithm control module of the embodiment of the application is shown in the figure;

[0027] Figure 5a And Figure 5bThe average loss and average reward curve diagrams of the simulation method based on the DQN algorithm and the Dueling DQN algorithm model trained in different environments, respectively;

[0028] Figure 5c and Figure 5d The tracking error diagram and flight trajectory diagram of the simulation method based on the DQN algorithm and the Dueling DQN algorithm model trained in different environments, respectively, performing the unmanned aerial vehicle target perception and tracking task.

[0029] The specific embodiments of the present application will be further described in detail below with reference to the accompanying drawings. DETAILED DESCRIPTION

[0030] The technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0031] With the development of unmanned aerial vehicle technology, unmanned aerial vehicles have been widely used in military, public safety, agriculture and logistics fields. In actual application, unmanned aerial vehicles need to realize real-time and accurate perception and tracking of targets to ensure the smooth completion of tasks such as autonomous navigation, target tracking, damage assessment and communication relay, which requires efficient, accurate and robust algorithms as support. At present, the performance of unmanned aerial vehicle target perception and tracking control algorithm is generally verified through a simulation platform. The mainstream simulation platforms at present are Airsim, FightGear, Jmavsim, Gazebo, Hecter and XPlan, etc. The performance comparison of these simulation platforms is shown in Table 1.

[0032] Table 1 Performance comparison table of mainstream simulation platforms

[0033]

[0034] From Table 1, it can be seen that the Airsim simulation platform can provide better environment and situation simulation functions compared with other simulation platforms, and can meet the research needs of unmanned aerial vehicle target perception and tracking simulation. Specifically, the AirSim simulation platform is based on Unreal Engine 4, has a highly restored physical engine, strong 3D rendering capability, realistic vehicle model, flexible control method and excellent scalability, etc. In addition, as a plug-in of Unreal Engine 4, Airsim can be flexibly applied to various Unreal scenarios.

[0035] To address the shortcomings of existing simulation technologies, such as insufficient environmental realism, poor platform scalability, and inefficiency caused by the separation of perception and tracking processes, this invention provides a deep reinforcement learning-based simulation method for UAV target perception and tracking, based on the Airsim simulation platform. The core of this method lies in constructing a simulation training and testing system that integrates high-fidelity environment simulation, open architecture, and end-to-end intelligent decision-making to simulate UAV target perception and tracking control algorithms.

[0036] Figure 1 The flowchart of the simulation method of the present invention is as follows: Figure 1 As shown, the simulation method in this embodiment includes the following steps:

[0037] S1. Build the experimental system architecture, such as Figure 2 As shown, the experimental system architecture consists of three parts: the Airsim simulation platform, the protocol connection layer, and the algorithm control module;

[0038] Among them, the Airsim simulation platform is used to build a high-fidelity, scalable simulation training environment. By building a virtual environment through the open-source simulation platform AirSim, and utilizing its high-quality physics engine and 3D rendering capabilities based on Unreal Engine, it can generate complex 3D scenes containing diverse terrains, dynamic lighting and realistic obstacle models.

[0039] The protocol connection layer acts as a bridge to enable data flow interaction between the Airsim platform and the algorithm control module outside the virtual environment. The protocol connection layer is mainly responsible for the interface connection between the Airsim simulation platform and the algorithm control module. It transmits environmental observation information from the virtual environment to the algorithm control module through the API interface, so that the algorithm control module can receive the required environmental state information. At the same time, it transmits the action values ​​output by the algorithm control module back to the virtual environment, so that the virtual environment can receive the decision information of the algorithm control module and make corresponding updates.

[0040] The algorithm control module is used to control the interaction between the simulated object (UAV) and the virtual environment, to make action decisions, train models, and complete simulated target perception and tracking tasks; the algorithm control module receives real-time status information transmitted by the UAV in the Airsim virtual environment through the protocol connection layer interface;

[0041] S2. Construct a virtual environment using the Airsim simulation platform;

[0042] The Airsim simulation platform provides a highly realistic physics engine, various virtual scenes with strong 3D rendering, and highly realistic vehicle models, enabling drones to conduct trial-and-error flights in near-realistic environments, collect a large number of high-quality experience samples, and improve the safety and efficiency of model training. Through different types of virtual scenes such as Blocks, Landscape Mountains, Coastline, and Small urban neighborhood in the Airsim simulation platform, model training and performance testing can be carried out to verify the generalization ability in complex environments in multiple scenarios.

[0043] Figure 3 The flowchart for setting up the virtual environment of the Airsim simulation platform in this embodiment is as follows: Figure 3 As shown, the steps for constructing a virtual environment in this embodiment are as follows: The simulation system environment layer of the Airsim simulation platform selects the corresponding scene according to the requirements, returns the corresponding task flow and environment settings, generates the corresponding simulation environment (virtual environment), and then renders the created simulation environment according to the environment rendering file, compiles the color tuner, and achieves a highly realistic reproduction of the 3D scene. At the same time, the corresponding simulation simulator has physical characteristics similar to the real world. The generated simulation environment interacts with the algorithm control module through the corresponding port, receives the corresponding control commands, makes the corresponding state updates in the virtual environment, and presents them in real time. Based on the corresponding sensors (such as airborne cameras), the observed environmental information is provided to the algorithm control module in real time for decision-making and model training.

[0044] S3. Design the protocol connection layer; through the protocol connection layer, realize stable and efficient data interaction between the virtual environment and the external algorithm control module, and ensure that the system has the scalability to flexibly integrate different sensor models and algorithm control modules;

[0045] S4. Construct the algorithm control module; such as... Figure 4 As shown, the algorithm control module in this embodiment includes an environment module, a target perception and tracking module, an agent training module, and an experience replay pool module;

[0046] The environment module interacts with the Airsim simulation platform through the protocol connection layer. The environment module receives environmental observation information from the virtual environment, including UAV coordinate data received from GPS sensors in the virtual environment and depth images containing the target UAV acquired by the front-facing camera tracking the UAV in the virtual environment. Specifically, based on an instantiated AirSim MultirotorClient object, the environment module receives target UAV coordinate data from the GPS sensors in the Airsim simulation platform's virtual environment through the corresponding API interface (protocol connection layer interface). This data serves as the ground truth for calculating the reward function. The module also receives information from the front-facing camera tracking the UAV. The system acquires a depth map containing the target UAV and uses it as input to the state space of the agent training module for training. The environment module receives action commands output from the algorithm control module and connects to the flight control API interface of the Airsim simulation platform via IP communication. The flight control API interface controls the UAV in the virtual environment to perform corresponding actions. The environment module is also used to interpret and calculate position offsets. Based on the state information between the tracking UAV and the target UAV read from the depth image acquired from the tracking UAV, it calculates new state information of the UAV. The reward value generated by the interaction between the agent training module and the environment, together with the round end judgment flag, is output as an observation value to the target perception and tracking module and the agent training module.

[0047] The target perception and tracking module is used to control the flight of UAVs in the virtual environment of the Airsim simulation platform and execute end-to-end target perception and tracking tasks. After loading the corresponding environment module and the model trained by the agent training module, the target perception and tracking module controls the flight of UAVs in the virtual environment through the flight control API interface of the Airsim simulation platform, saves the UAV flight formation queue data, and creates a new thread to synchronously draw the tracking trajectory.

[0048] The agent training module is used to generate flight control actions. In this embodiment, the agent training module is based on a deep neural network. Taking the depth image of the target UAV as input, the deep neural network performs feature extraction and action evaluation, makes action selection, and outputs corresponding action control commands. The target perception and tracking module controls the UAV in the virtual environment of the Airsim simulation platform to fly according to the action control commands and interact with the virtual environment. At the same time, the agent training module receives the experience sequence sampled from the experience replay pool module, performs experience learning training, calculates the loss function, and performs gradient descent optimization to update the parameters of the deep neural network, thereby realizing the training and optimization of the agent model.

[0049] The experience replay pool module receives experience data generated by the interaction between the agent training module and the virtual environment, and encapsulates and stores it according to a specified format for use by the agent training module. Specifically, the environment module calculates the state reward for each state, packages the state, action, and reward information based on time series and a specified format, and puts it into the experience replay pool for model training of the agent training module. After the model training of the agent training module is completed, the model parameters are saved and loaded during performance verification to complete the target perception and tracking tasks.

[0050] S5, Model Training;

[0051] After confirming that the Airsim simulation platform and the algorithm control module are connected and can communicate normally, the algorithm control module trains the model in different scenarios on the Airsim simulation platform and collects the sequence of state, action, reward and next state, and stores it in the experience replay pool module; then, it randomly and uniformly samples batch data from the experience replay pool module to update the neural network parameters; after completing the model training for a specified number of rounds, it saves the trained model parameters for testing the task implementation effect.

[0052] S6. Task Implementation Performance Evaluation and Verification: After training, the target perception and tracking module loads the model parameters and executes the target perception and tracking task in a virtual environment to verify the algorithm's performance and stability. During the verification process, multiple methods can be used, such as tracking trajectory visualization, quantitative analysis of tracking errors, and model training curve evaluation, to verify the algorithm's performance.

[0053] The agent training module in this embodiment employs a deep neural network, using depth images acquired by the UAV's onboard camera as the primary state input. The deep neural network directly maps and generates flight control actions, forming an end-to-end closed loop of "perception-decision-control." Specifically, the agent training module uses deep reinforcement learning algorithms such as DQN and Dueling DQN, leveraging empirical data collected from multi-scenario simulation interactions to optimize the policy network through a combination of offline and online training. The deep reinforcement learning agent training module is sequentially or hybridly trained in multiple pre-set virtual scenarios of varying complexity to improve the model's generalization ability and robustness under different environmental conditions.

[0054] This invention reduces the difference between the simulation and real domains through a high-fidelity simulation environment, improving the transfer reliability of the training model; the modular design enhances the openness and scalability of the system, facilitating rapid algorithm iteration; the end-to-end control framework achieves tight coupling between perception and tracking, simplifies the system process, and exhibits higher tracking accuracy and stronger environmental adaptability in complex scenarios, providing an efficient and safe simulation verification platform for the research and development of UAV autonomous tracking technology.

[0055] Figures 5a to 5d The results of the simulation method of this invention are shown in the figure. The simulation objects used are two isomorphic quadcopter drones, one as the target drone and the other as the tracking drone. The sensor is the onboard camera on the tracking drone, which provides real-time depth image information for the action decision-making of the agent training module. The agent training module trains the model using both the DQN algorithm and the Dueling DQN algorithm. During the simulation, the trained model parameters are loaded in four different scenarios on the Airsim simulation platform to perform target perception and tracking tasks. The three most recent frames of depth images captured by the onboard camera of the tracking drone are displayed in real time and saved in the image folder. Simultaneously, the formation queue data of the two drones is saved. After the perception and tracking tasks are completed, the flight trajectories, tracking error curves, and model training index curves of the two drones are plotted to evaluate the task completion effect and the effectiveness and stability of the model training.

[0056] from Figure 5a and Figure 5b It can be seen that in Blocks and Landscape Mountains scenarios, where the scene construction is relatively simple and the background interference of the UAV performing perception and tracking tasks is relatively small, both algorithms can achieve good training convergence. The Dueling DQN algorithm has a shorter convergence time and a better convergence value. However, in Coastline and Small urban neighborhood scenarios with more scenery and stronger background interference, the training convergence of the DQN algorithm deteriorates significantly and even shows a divergence trend in the later stage of training. Although the Dueling DQN algorithm is also affected by scene conditions, its overall training convergence is still good.

[0057] from Figure 5c and Figure 5d It can be seen that the tracking error of the UAV under the control of the Dueling DQN algorithm is significantly smaller than that of the traditional DQN algorithm when performing perception and tracking tasks. The tracking error is always within 0.06m, the tracking accuracy is higher, and it can complete the predetermined target perception and tracking tasks better.

[0058] The simulation method of the present invention has the following advantages:

[0059] 1. Effectively overcomes the distortion problem of simulation environment and improves the reliability of model transfer: By adopting the AirSim high-fidelity simulation platform, it is possible to simulate complex visual scenes and physical interactions that are close to the real world. In particular, by using its high-quality sensor (such as binocular camera) model to generate realistic depth image data, the domain difference between simulation data and real data is significantly reduced. This makes it more likely that the deep reinforcement learning strategy trained in this environment can be directly or after a few minor adjustments to the real drone platform, thus improving R&D efficiency and algorithm reliability.

[0060] 2. Breaking the limitations of platform closure and enhancing system scalability and openness: This invention is built on the open-source AirSim platform and a modular system architecture. The separation design of the protocol connection layer and the algorithm control module makes algorithm development no longer limited to specific, closed simulation frameworks. Researchers can easily replace or upgrade reinforcement learning algorithms, neural network structures, or integrate new virtual sensor models, which greatly improves the flexibility of the simulation system and its ability to respond quickly to cutting-edge technologies, and is conducive to building an open R&D ecosystem.

[0061] 3. Achieving tight coupling between perception and tracking, simplifying the algorithm process and improving tracking accuracy: The algorithm control module adopts an end-to-end deep reinforcement learning control method, abandoning the complex serial process of first performing target detection separately and then designing an independent tracker in traditional methods. By learning the optimal tracking strategy directly from the original image perception information through a deep neural network, deep coupling between target perception and motion control is achieved.

[0062] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A simulation method for UAV target perception and tracking based on deep reinforcement learning, characterized in that, Includes the following steps: S1. Establish the experimental system architecture, which includes an Airsim simulation platform, a protocol connection layer, and an algorithm control module; the Airsim simulation platform is used to build a simulation training environment; the protocol connection layer is used to realize data flow interaction between the Airsim platform and the algorithm control module; The algorithm control module is used to control the interaction between the simulation object and the virtual environment, to make action decisions, train models, and complete the tasks of simulating target perception and tracking. S2. Construct a virtual environment using the Airsim simulation platform; S3. Design the protocol connection layer; S4. Construct the algorithm control module; The algorithm control module includes an environment module, a target perception and tracking module, an agent training module, and an experience replay pool module. The environment module interacts with the Airsim simulation platform through the protocol connection layer to receive environmental observation information from the virtual environment; The target perception and tracking module is used to control the flight of the UAV in the virtual environment of the Airsim simulation platform and to perform end-to-end target perception and tracking tasks. The intelligent agent training module is used to generate action control commands for flight control actions; The experience replay pool module is used to receive experience data generated by the interaction between the agent training module and the virtual environment, and encapsulate and store it for use by the agent training module during training. S5, Model Training; The algorithm control module trains the model in different scenarios on the Airsim simulation platform, collects the sequence of state, action, reward and next state, stores it in the experience replay pool module, and saves the trained model parameters after the model training is completed. S6. Evaluation and verification of task implementation effectiveness; After training, the target perception and tracking module loads the model parameters and executes the target perception and tracking task in a virtual environment to verify the performance and stability of the algorithm.

2. The UAV target perception and tracking simulation method as described in claim 1, characterized in that: The environmental observation information includes UAV coordinate data received by GPS sensors in the virtual environment and depth images containing the target UAV obtained by the front-facing camera of the UAV tracking UAV in the virtual environment.

3. The UAV target perception and tracking simulation method as described in claim 2, characterized in that: The intelligent agent training module is based on a deep neural network. It takes the acquired depth image containing the target UAV as input, performs feature extraction and action evaluation, makes action selection, and outputs corresponding action control commands. The target perception and tracking module controls the UAV in the virtual environment to fly according to the action control commands.

4. The UAV target perception and tracking simulation method as described in claim 1, characterized in that: After loading the corresponding environment module and trained model, the target perception and tracking module controls the flight of the UAV in the virtual environment through the flight control API interface of the Airsim simulation platform, saves the UAV flight formation queue data, and creates a new thread to synchronously draw the tracking trajectory.

5. The UAV target perception and tracking simulation method as described in claim 1, characterized in that: The protocol connection layer transmits environmental observation information from the virtual environment to the algorithm control module via an API interface, enabling the algorithm control module to receive the required environmental state information. Simultaneously, the action values ​​output by the algorithm control module are transmitted back to the virtual environment, allowing the virtual environment to receive the decision information from the algorithm control module and update accordingly.

6. The UAV target perception and tracking simulation method as described in claim 1, characterized in that: The environment module calculates the state reward for each state, packages the state, action, and reward information based on time series and a specified format, and puts it into the experience replay pool module for model training. After the model training of the agent training module is completed, it saves the model parameters and loads the model parameters during performance verification to complete the target perception and tracking task.

Images