Target tracking method and system based on unmanned aerial vehicle

Abstract

The invention provides a target tracking method and system based on an unmanned aerial vehicle, and the method comprises the steps: collecting a target image based on a camera of the unmanned aerial vehicle, and detecting a target position in the image through a Center Net network model; controlling and adjusting a current unmanned shooting visual angle according to the target position, planning an unmanned aerial vehicle path through a reinforcement learning model according to the target moving direction, and controlling the unmanned aerial vehicle to track and shoot; and calculating the relative position of the detection target and the unmanned aerial vehicle in real time, marking the target position on the local map based on the current unmanned aerial vehicle positioning, and displaying a target tracking picture. Through the scheme, unmanned aerial vehicle tracking visual field adjustment can be realized, operation time delay in target tracking is reduced, and unmanned aerial vehicle automatic tracking control is realized.

Description

Technical Field

[0001] The present invention belongs to the field of unmanned aerial vehicles, and in particular relates to a target tracking method and system based on unmanned aerial vehicles. Background Art

[0002] As the application scenarios of drones continue to expand, drones are playing an important role in both military and civilian fields, such as unmanned photography, daily patrols, field searches, target reconnaissance, etc. Due to the diversity of usage scenarios, higher requirements are placed on drone target recognition and tracking.

[0003] Currently, target tracking of UAVs is usually limited to keeping the target within the field of view of the UAV, or maintaining a specific observation distance and observation angle. For target tracking in complex scenes and multi-target tracking, UAV control under traditional logic is difficult to guarantee the best observation field of view. Moreover, in complex scenes, this type of target tracking will have a large logic operation delay, which will cause the loss of tracking targets for some fast-moving targets. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a target tracking method and system based on a drone, which is used to solve the problem that the existing target tracking has a large logical operation delay and easily leads to target loss.

[0005] In a first aspect of an embodiment of the present invention, a target tracking method based on a drone is provided, comprising: The target image is collected based on the drone camera, and the target position in the image is detected through the CenterNet network model; Adjust the current unmanned shooting angle according to the target position control, and plan the drone path through the reinforcement learning model according to the target movement direction, and control the drone to track and shoot; The relative position between the detection target and the UAV is calculated in real time. Based on the current UAV positioning, the target position is marked on the local map and the target tracking screen is displayed.

[0006] In a second aspect of an embodiment of the present invention, a target tracking system based on a drone is provided, comprising: The target detection module is used to collect target images based on the drone camera and detect the target position in the image through the CenterNet network model; The target tracking module is used to control and adjust the current unmanned shooting angle according to the target position, and plan the drone path through the reinforcement learning model according to the target movement direction, and control the drone to track and shoot; The target display module is used to calculate the relative position of the detection target and the UAV in real time, mark the target position on the local map based on the current UAV positioning, and display the target tracking screen.

[0007] In a third aspect of an embodiment of the present invention, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the first aspect of the embodiment of the present invention when executing the computer program.

[0008] In a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method provided in the first aspect of the embodiment of the present invention are implemented.

[0009] In the embodiment of the present invention, the target position in the image is detected by the CenterNet network model, and the UAV path is planned by the reinforcement learning model, the UAV is controlled to track and shoot, the detected target position is calculated in real time, and the target tracking screen is displayed, thereby realizing the tracking of the detected target. It can not only adjust the observation field of the UAV, but also plan the UAV tracking path in real time, avoiding the problem of large calculation delay of traditional target tracking methods in complex scenes, and at the same time, it can effectively improve the target tracking display effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0011] Figure 1 A schematic diagram of a flow chart of a target tracking method based on a drone provided by an embodiment of the present invention; Figure 2 A schematic diagram of a target tracking system based on a drone provided by an embodiment of the present invention; Figure 3 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0012] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0013] It should be understood that the term "including" and other similar expressions in the specification or claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions, such as a process, method, system, or device including a series of steps or units is not limited to the listed steps or units. In addition, "first" and "second" are used to distinguish different objects, not to describe a specific order.

[0014] See also Figure 1 , a flowchart of a target tracking method based on a drone provided by an embodiment of the present invention includes: S101, collect target images based on drone cameras, and detect the target position in the image through the CenterNet network model; The drone camera is an angle-adjustable camera, the rotation angle of which is controlled by an independent motor to keep the target within the camera's field of view.

[0015] The CenterNet network is a target detector that uses the CenterNet network and is based on CenterNet as an anchor-free detector. It removes the inefficient and complex Anchors operation, improves the performance of the detection algorithm, and further improves the running speed of the entire algorithm after removing the time-consuming non-maximum suppression NMS. The accuracy of detecting the center of the target is higher than that of the anchor-based method.

[0016] The target detected in each frame is added to the matching container, and the target detected in the current frame is matched with the target in the matching container through the Hungarian algorithm to calculate the target feature similarity; When the target feature similarity exceeds the set value, the current frame target and the matching target are marked the same.

[0017] S102, adjusting the current unmanned shooting angle according to the target position control, and planning the drone path through the reinforcement learning model according to the target moving direction, and controlling the drone to track and shoot; When detecting a single target, the drone camera angle can be automatically adjusted based on the position of the target detected in the two-dimensional image to keep the target at the center of the image or within the center range; when detecting multiple targets, the camera position can be adjusted to minimize the distance variance based on the variance of the target distance from the image center. It can be understood that each time the camera angle is adjusted, the data storage in the drone will record the camera angle in order to calculate the relative position of the target.

[0018] Optionally, a target movement state equation and a drone observation equation are constructed respectively, and according to the target movement state equation, the drone observation equation and the target state vector of the current frame, the observation equation is updated through an adaptive Kalman filter to obtain an estimated value of the target state of the next frame; The solution of the target state estimation value is converted into a linear regression model, the linear regression model is solved by the Gauss-Newton iteration method, and the noise covariance is updated by the variational Bayes method.

[0019] The target state estimation value includes at least the moving speed and direction of the target. The state of the moving target in the next frame is estimated by Kalman filtering, so as to timely control the drone to track the corresponding target. At the same time, when the target is blocked by other objects, the target state in the next frame can be inferred.

[0020] Specifically, based on tracking the target movement trajectory and target movement direction, the reinforcement learning model is used to plan the flight path of the UAV under the set constraints; Among them, the reinforcement learning model is a drone tracking model based on Markov decision making. The current state, current action, updated state and current reward of the drone are used as empirical sample data, the Actor network is updated by batch sample policy gradient, and the Critic network is updated by minimizing the loss function to obtain the trained drone tracking model.

[0021] The reinforcement learning model takes the shortest moving path as the optimization goal, and realizes the flight control of the drone based on the set constraints, such as the drone's flight altitude, target distance range, speed range and other conditional constraints. By constructing an intelligent agent, taking a series of flight actions, and recording the rewards obtained for each step of the action, the goal is to obtain the most cumulative rewards, and continuously learn and feedback to obtain the final control strategy. The model includes an Actor network and a Critic network. The Actor network is used to output actions, and the Critic network is used to output the Q value of the evaluation state-action pair. The Actor network and the Critic network are continuously learned and updated based on the empirical data generated by batch simulation.

[0022] S103, calculating the relative position of the detection target and the UAV in real time, marking the target position on the local map based on the current UAV positioning, and displaying the target tracking screen.

[0023] By measuring the distance between the detection target and the drone, combined with the camera angle, the drone’s own GPS positioning and altitude, the relative position of the two can be obtained.

[0024] The relative distance between the UAV and the detection target is measured by at least one of laser ranging, infrared ranging or binocular camera ranging, and the GPS position of the detection target is calculated by combining the current UAV camera angle, UAV GPS positioning and altitude.

[0025] It can be understood that if the camera shooting angle is known and combined with the straight-line distance of the target, the offset distance of the target in the horizontal direction can be calculated and its GPS position can be inferred. When the drone attitude is offset (not in a horizontal flight attitude), the drone flight attitude needs to be considered to infer the target position.

[0026] Preferably, when the tracked target is blocked, the target position in the current frame is predicted based on the historical motion trajectory of the target and the target state in the previous frame, and the target is identified at the blocked position in the tracking picture.

[0027] Since the target state can be obtained through Kalman filtering and combined with the target's historical movement trajectory, the target position can be inferred. Even if the target is blocked for a long time, the target position can still be predicted.

[0028] In this embodiment, not only can the UAV tracking field of view be adjusted, but also the automatic planning and control of the UAV tracking path can be realized, thereby reducing the logical operation delay of the UAV control in target tracking and improving the target tracking display effect.

[0029] It should be understood that the serial numbers of the steps in the above embodiments do not imply a sequence of execution, and the execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0030] Figure 2 A schematic diagram of a target tracking system based on a drone provided by an embodiment of the present invention, the system includes: The target detection module 210 is used to collect target images based on the drone camera and detect the target position in the image through the CenterNet network model; Wherein, the target detection module 210 includes: The target matching unit is used to add the target detected in each frame to the matching container, match the target detected in the current frame with the target in the matching container through the Hungarian algorithm, calculate the target feature similarity, and when the target feature similarity exceeds the set value, set the same mark for the current frame target and the matching target.

[0031] The target tracking module 220 is used to control and adjust the current unmanned shooting angle according to the target position, and plan the drone path through the reinforcement learning model according to the target movement direction, and control the drone to track and shoot; Wherein, the target tracking module 220 includes: The target prediction unit is used to construct the target movement state equation and the drone observation equation respectively. According to the target movement state equation, the drone observation equation and the current frame target state vector, the observation equation is updated through adaptive Kalman filtering to obtain the target state estimation value of the next frame; The solution of the target state estimation value is converted into a linear regression model, the linear regression model is solved by the Gauss-Newton iteration method, and the noise covariance is updated by the variational Bayes method.

[0032] Specifically, based on tracking the target movement trajectory and target movement direction, the reinforcement learning model is used to plan the flight path of the UAV under the set constraints; Among them, the reinforcement learning model is a drone tracking model based on Markov decision making. The current state, current action, updated state and current reward of the drone are used as empirical sample data, the Actor network is updated by batch sample policy gradient, and the Critic network is updated by minimizing the loss function to obtain the trained drone tracking model.

[0033] The target display module 230 is used to calculate the relative position of the detection target and the drone in real time, mark the target position on the local map based on the current drone positioning, and display the target tracking screen.

[0034] Wherein, the target display module 230 includes: The target positioning unit is used to measure the relative distance between the drone and the detection target by at least one of laser ranging, infrared ranging or binocular camera ranging, and calculate the GPS position of the detection target in combination with the current drone camera angle, drone GPS positioning and altitude.

[0035] Optionally, marking the target location on the local map and displaying the target tracking screen includes: When the tracked target is blocked, the target position in the current frame is predicted based on the target's historical motion trajectory and the target state in the previous frame, and the target is identified at the blocked position in the tracking image.

[0036] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the above-described system and module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0037] Figure 3 FIG. 1 is a schematic diagram of a structure of an electronic device provided by an embodiment of the present invention. The electronic device is used for target tracking. Figure 3 As shown, the electronic device 3 of this embodiment includes: a memory 310, a processor 320 and a system bus 330, and the memory 310 includes an executable program 3101 stored thereon. Those skilled in the art can understand that Figure 3 The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0038] Combine the following Figure 3 A detailed introduction to the various components of electronic equipment: The memory 310 can be used to store software programs and modules. The processor 320 executes various functional applications and data processing of the electronic device by running the software programs and modules stored in the memory 310. The memory 310 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the electronic device (such as cache data), etc. In addition, the memory 310 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0039] The memory 310 includes an executable program 3101 of the network request method, and the executable program 3101 can be divided into one or more modules / units, which are stored in the memory 310 and executed by the processor 320 to perform target recognition and tracking, etc. The one or more modules / units can be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the executable program 3101 in the electronic device 3. For example, the executable program 3101 can be divided into functional modules such as a target detection module and a target tracking module.

[0040] The processor 320 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 310, and calling data stored in the memory 310, it performs various functions of the electronic device and processes data, thereby monitoring the overall status of the electronic device. Optionally, the processor 320 may include one or more processing units; preferably, the processor 320 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 320.

[0041] The system bus 330 is used to connect the various functional components inside the computer, and can transmit data information, address information, and control information. Its types can be, for example, PCI bus, ISA bus, CAN bus, etc. The instructions of the processor 320 are transmitted to the memory 310 through the bus, and the memory 310 feeds back data to the processor 320. The system bus 330 is responsible for the data and instruction exchange between the processor 320 and the memory 310. Of course, the system bus 330 can also be connected to other devices, such as network interfaces, display devices, etc.

[0042] In the embodiment of the present invention, the executable program executed by the processor 320 included in the electronic device includes: The target image is collected based on the drone camera, and the target position in the image is detected through the CenterNet network model; Adjust the current unmanned shooting angle according to the target position control, and plan the drone path through the reinforcement learning model according to the target movement direction, and control the drone to track and shoot; The relative position between the detection target and the UAV is calculated in real time. Based on the current UAV positioning, the target position is marked on the local map and the target tracking screen is displayed.

[0043] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0044] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0045] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A target tracking method based on a drone, characterized in that: include: The target image is collected based on the drone camera, and the target position in the image is detected through the CenterNet network model; Adjust the current unmanned shooting angle according to the target position control, and plan the drone path through the reinforcement learning model according to the target movement direction, and control the drone to track and shoot; The relative position between the detection target and the UAV is calculated in real time. Based on the current UAV positioning, the target position is marked on the local map and the target tracking screen is displayed.

2. The method according to claim 1, characterized in that The detecting the target position in the image by using the CenterNet network model also includes: Add the detected target in each frame to the matching container, match the target detected in the current frame with the target in the matching container through the Hungarian algorithm, and calculate the target feature similarity; When the target feature similarity exceeds the set value, the current frame target and the matching target are marked the same.

3. The method according to claim 1, characterized in that The method of planning the drone path through the reinforcement learning model according to the target moving position and controlling the drone to track and shoot includes: The target movement state equation and the drone observation equation are constructed respectively. According to the target movement state equation, the drone observation equation and the current frame target state vector, the observation equation is updated through adaptive Kalman filtering to obtain the target state estimation value of the next frame. The solution of the target state estimation value is converted into a linear regression model, the linear regression model is solved by the Gauss-Newton iteration method, and the noise covariance is updated by the variational Bayes method.

4. The method according to claim 1, characterized in that: The method of planning the drone path through the reinforcement learning model according to the target moving position and controlling the drone to track and shoot includes: Based on tracking the target's moving trajectory and moving direction, the UAV's flight path is planned using a reinforcement learning model under set constraints. Among them, the reinforcement learning model is a drone tracking model based on Markov decision making. The current state, current action, updated state and current reward of the drone are used as empirical sample data, the Actor network is updated by batch sample policy gradient, and the Critic network is updated by minimizing the loss function to obtain the trained drone tracking model.

5. The method according to claim 1, characterized in that The real-time calculation of the relative position of the detection target and the drone includes: The relative distance between the UAV and the detection target is measured by at least one of laser ranging, infrared ranging or binocular camera ranging, and the GPS position of the detection target is calculated by combining the current UAV camera angle, UAV GPS positioning and altitude.

6. The method according to claim 1, characterized in that The marking of the target location on the local map and displaying the target tracking screen includes: When the tracked target is blocked, the target position in the current frame is predicted based on the target's historical motion trajectory and the target state in the previous frame, and the target is identified at the blocked position in the tracking image.

7. A target tracking system based on drone, characterized in that: include: The target detection module is used to collect target images based on the drone camera and detect the target position in the image through the CenterNet network model; The target tracking module is used to control and adjust the current unmanned shooting angle according to the target position, and plan the drone path through the reinforcement learning model according to the target movement direction, and control the drone to track and shoot; The target display module is used to calculate the relative position of the detection target and the drone in real time, mark the target position on the local map based on the current drone positioning, and display the target tracking screen.

8. The system according to claim 7, characterized in that The target detection module comprises: The target matching unit is used to add the target detected in each frame to the matching container, match the target detected in the current frame with the target in the matching container through the Hungarian algorithm, calculate the target feature similarity, and when the target feature similarity exceeds the set value, set the same mark for the current frame target and the matching target.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the target tracking method based on a drone as described in any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed, the steps of a target tracking method based on a drone as described in any one of claims 1 to 6 are implemented.

Images