Multi-thread unmanned aerial vehicle multi-target tracking method and device

Abstract

The invention relates to a multi-thread unmanned aerial vehicle multi-target tracking method and device, and the method comprises the steps: obtaining two adjacent frames of images shot by an unmanned aerial vehicle-mounted high-speed camera, carrying out the target detection and feature matching, obtaining the position information of the same target in the two adjacent frames of images, and fitting the motion trail of each target according to the position information; a tracking thread is distributed for each target, and the tracking thread predicts the position of the target in the image at the next moment according to the motion state information of the unmanned aerial vehicle and the motion trail of the corresponding target; acquiring a next frame of image, and performing extended search on the next frame of image according to the predicted position of the target to obtain an actual position of the target in the next frame of image; cutting the next frame of image based on the actual position of the target according to a preset cutting rule, and storing the cut image; and updating the motion track of the target, and predicting the position of the target in the next frame of image. The method is high in tracking efficiency and small in required storage resources.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle search and tracking, and in particular to a multi-threaded unmanned aerial vehicle multi-target tracking method and device. Background Art

[0002] In recent years, with the continuous improvement of the level of science and technology such as automation technology and computer vision technology, drones have developed rapidly in military, industrial and civilian fields. As an important branch of drone application technology, drone target tracking technology has broad application prospects in many aspects, has received great attention, and has become one of the most active research directions in this field. The main purpose of target tracking is to calculate the two-dimensional coordinate position of the moving target on each frame of the image sequence obtained from the imaging sensor through processing and analysis, associate the same moving target in consecutive frames in the image sequence, and obtain the complete motion trajectory of the moving target. Simply put, it is to find the determined position of the target in the next frame of the image and feed it back to the tracking system for tracking. In the prior art, the commonly used methods for target tracking of drone videos include Kalman filter tracking algorithm, meanshift tracking algorithm, target tracking algorithm combined with Kalman and meanshift, particle filter tracking algorithm, and related filter tracking algorithm, etc., but the above methods are usually used to track a single target and cannot meet the tracking requirements of multiple targets. Moreover, the image sequence used for target tracking in the prior art is usually stored in full frame, which occupies a large amount of storage resources, which is contrary to the miniaturization requirements of drones. Summary of the invention

[0003] The present invention aims at solving the technical problems existing in the prior art and provides a multi-threaded unmanned aerial vehicle multi-target tracking method and device.

[0004] The technical solution of the present invention to solve the above technical problems is as follows:

[0005] In a first aspect, the present invention provides a multi-threaded UAV multi-target tracking method, comprising:

[0006] S1, obtaining two adjacent frames of images taken by a high-speed camera on a drone, and performing target detection and feature matching respectively, obtaining the position information of the same target in the two preceding and succeeding frames of images, and fitting the motion trajectory of each target according to the position information;

[0007] S2, assigning a tracking thread to each target, wherein the tracking thread predicts the position of the target in the image at the next moment according to the motion state information of the UAV and the motion trajectory of the corresponding target;

[0008] S3, obtaining the next frame image, and performing an extended search on the next frame image according to the predicted position of the target to obtain the actual position of the target in the next frame image;

[0009] S4, cropping the next frame of image based on the actual position of the target according to a preset cropping rule, and storing the cropped image;

[0010] S5, update the target motion trajectory and predict the position of the target in the next frame image, and repeat steps S3 to S5 until the tracking task is completed.

[0011] Further, step S1 includes:

[0012] Obtain target feature information in the tracking task. Note: the task acquisition time is t0;

[0013] Starting from time t0, target detection is performed on the image taken by the high-speed camera on the drone based on the target feature information until the tracking target appears in the image;

[0014] The frame image in which the tracking target appears for the first time is recorded as the first frame image, and the pixel position information and feature information of each target contained in the first frame image are extracted;

[0015] The next frame image captured by the high-speed camera on the drone is obtained, recorded as a sub-frame image, and target detection is performed on the sub-frame image to obtain pixel position information and feature information of each target contained in the sub-frame image;

[0016] Matching the feature information of each target in the first frame image and the second frame image to obtain the pixel position information of the same target in the first frame image and the second frame image;

[0017] The motion trajectory of the corresponding target is fitted according to the pixel position information.

[0018] Furthermore, step S1 further includes:

[0019] Extract the UAV motion status information based on the timestamps of the first frame image and the second frame image;

[0020] The motion trajectory of the corresponding target is fitted according to the UAV motion state information and the pixel position information.

[0021] Furthermore, the UAV motion state information is recorded as: The UAV motion equation is as follows:

[0022]

[0023] Where T is the frame interval, a t is the acceleration of the drone at time t, φ t is the steering angle of the drone, is the position of the drone at time t+1, is the speed of the drone at time t+1, θ t+1is the heading angle of the UAV at time t+1, and g is the acceleration due to gravity.

[0024] Furthermore, the extended search of the next frame of image according to the predicted position of the target to obtain the actual position of the target in the next frame of image includes:

[0025] Generate an initial bounding rectangle according to the size of the target;

[0026] Extending the four sides of the initial circumscribed rectangular frame outward by a distance of multiple pixels to obtain an extended rectangular frame;

[0027] The center position of the extended rectangular box is set as the predicted position of the target, and the target is detected in the extended rectangular box to obtain the actual position of the target.

[0028] Furthermore, if no target is detected in the extended rectangular frame, the four sides of the extended rectangular frame are extended outward by a plurality of pixel distances to obtain a new extended rectangular frame, and target detection is performed.

[0029] Furthermore, if no target is detected in the new extended rectangular frame, it is determined whether the number of expansions of the extended rectangular frame is less than a preset threshold and whether the size of the extended rectangular frame is less than a preset size. If so, the rectangular frame is continued to be expanded and target detection is performed; otherwise, target detection is performed in the full image.

[0030] In a second aspect, the present invention provides a multi-threaded drone multi-target tracking device, comprising:

[0031] The detection and matching module obtains two adjacent frames of images taken by the high-speed camera on the drone, and performs target detection and feature matching respectively to obtain the position information of the same target in the two previous and next frames of images, and fits the motion trajectory of each target according to the position information;

[0032] The target position prediction module allocates a tracking thread to each target. The tracking thread predicts the position of the target in the image at the next moment based on the UAV motion state information and the motion trajectory of the corresponding target.

[0033] The extended search module obtains the next frame of image, and performs an extended search on the next frame of image according to the predicted position of the target to obtain the actual position of the target in the next frame of image;

[0034] An image cropping module crops the next frame of image based on the actual position of the target according to a preset cropping rule, and stores the cropped image;

[0035] The target trajectory update module updates the target motion trajectory and predicts the position of the target in the next frame image.

[0036] In a third aspect, the present invention provides an electronic device, comprising:

[0037] Memory for storing computer software programs;

[0038] The processor is used to read and execute the computer software program, thereby implementing a multi-threaded drone multi-target tracking method as described in the first aspect of the present invention.

[0039] In a fourth aspect, the present invention provides a non-transitory computer-readable storage medium, wherein the storage medium stores a computer software program, and when the computer software program is executed by a processor, the multi-threaded drone multi-target tracking method as described in the first aspect of the present invention is implemented.

[0040] The beneficial effects of the present invention are as follows: the present invention allocates a tracking thread to each tracking target, and multiple targets can be tracked in parallel, effectively improving the target tracking efficiency to adapt to the shooting rate of the drone's high-speed camera. Since drone reconnaissance requires a certain degree of concealment and usually requires long-distance shooting, there is often a large amount of invalid data in a frame of image, which will occupy a large amount of storage resources when stored. After the present invention detects the target, it will crop the image before storing it, thereby greatly reducing storage resources, so that the drone can maximize image storage while meeting the requirements of miniaturization. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A schematic flow chart of a multi-threaded UAV multi-target tracking method provided by an embodiment of the present invention;

[0042] Figure 2 A schematic diagram of the structure of a multi-threaded UAV multi-target tracking device provided by an embodiment of the present invention;

[0043] Figure 3 A schematic diagram of an electronic device according to an embodiment of the present invention;

[0044] Figure 4 A schematic diagram of an embodiment of a computer-readable storage medium provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0046] In the description of the present application, “plurality” means two or more than two, unless otherwise clearly and specifically defined.

[0047] In the description of the present application, the term "for example" is used to mean "used as an example, illustration or description". Any embodiment described as "for example" in the present application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid unnecessary details to obscure the description of the present invention. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present application.

[0048] like Figure 1 As shown, an embodiment of the present invention provides a multi-threaded drone multi-target tracking method, comprising:

[0049] S1, obtain two adjacent frames of images taken by a high-speed camera onboard an unmanned aerial vehicle, and perform target detection and feature matching respectively to obtain the position information of the same target in the two preceding and succeeding frames of images, and fit the motion trajectory of each target according to the position information.

[0050] Before executing a tracking mission, the drone will generally receive a tracking mission from the control center, which contains information about multiple targets to be tracked. The drone control module parses the mission information and obtains the target feature information in the tracking mission. Note: the mission acquisition time is t0.

[0051] Starting from time t0, the UAV's motion status information is recorded in real time. Based on the target feature information obtained from the tracking task, target detection is performed on the images taken by the UAV's high-speed camera until any tracking target appears in the image. In this process, images without tracking targets are discarded to reduce the storage pressure of the UAV.

[0052] The frame image in which the tracked target appears for the first time is recorded as the first frame image and stored, and the pixel position information and feature information of each target contained in the first frame image are extracted; the next frame image taken by the high-speed camera on the drone is obtained and recorded as the second frame image, and target detection is performed on the second frame image to obtain the pixel position information and feature information of each target contained in the second frame image.

[0053] The feature information of each target in the first frame image and the second frame image is matched to obtain the pixel position information of the same target in the first frame image and the second frame image; the UAV motion state information is extracted according to the timestamps of the first frame image and the second frame image; and the motion trajectory of the corresponding target is fitted according to the UAV motion state information and the pixel position information.

[0054] The UAV motion status information is recorded as: The UAV motion equation is as follows:

[0055]

[0056] Where T is the frame interval, a t is the acceleration of the drone at time t, φ t is the steering angle of the drone, is the position of the drone at time t+1, is the speed of the drone at time t+1, θ t+1 is the heading angle of the UAV at time t+1, and g is the acceleration due to gravity.

[0057] According to the UAV motion state information and the pixel position information of the target in the image, the target motion information such as the moving speed, heading angle, and the distance of the target relative to the UAV can be calculated through coordinate transformation and other methods, so as to fit the target motion trajectory.

[0058] It should be understood that multiple tracking targets may not appear in one frame of image at the same time. For a certain tracking target, the image in which the target appears for the first time is recorded as the first frame of image corresponding to the target, and the next frame of image is recorded as the second frame of image corresponding to the target. Therefore, when multiple targets appear in one frame of image at the same time, the multiple targets share one first frame of image.

[0059] In this step, it is also necessary to determine whether all tracking targets in the tracking task have appeared. If not all of them appear, continue to perform target detection in the full image.

[0060] S2, a tracking thread is assigned to each target, and the tracking thread predicts the position of the target in the image at the next moment according to the UAV motion state information and the motion trajectory of the corresponding target.

[0061] It should be noted here that when predicting the target position, there is no need to calculate the true position of the predicted tracking target, only the pixel coordinates of the target need to be predicted.

[0062] S3, obtaining the next frame of image, and performing an extended search on the next frame of image according to the predicted position of the target to obtain the actual position of the target in the next frame of image. It should be understood that the actual position of the target in the next frame of image is also the pixel coordinates of the target.

[0063] Specifically, this step first generates an initial circumscribed rectangular frame according to the size of the target; the four sides of the initial circumscribed rectangular frame are extended outward by a plurality of pixel distances, for example, 2 to 3 pixel distances, to obtain an extended rectangular frame. Assuming that the size of the initial circumscribed rectangular frame is 5*5 pixels and the extended distance is 2 pixels, the size of the extended rectangular frame is 9*9 pixels. It should be understood here that the outward extended pixel distance can be manually set according to actual needs and is not limited to 2 to 3 pixel distances.

[0064] The center position of the extended rectangular box is set as the predicted position of the target, and the target is detected in the extended rectangular box to obtain the actual position of the target.

[0065] If no target is detected in the extended rectangular frame, the four sides of the extended rectangular frame are extended outward by a plurality of pixel distances to obtain a new extended rectangular frame, and target detection is performed.

[0066] If no target is detected in the new extended rectangular frame, it is determined whether the number of expansions of the extended rectangular frame is less than a preset threshold and whether the size of the extended rectangular frame is less than a preset size. If so, the rectangular frame is continued to be expanded and target detection is performed; otherwise, target detection is performed in the full image.

[0067] S4, according to the preset cropping rules, the next frame of the image is cropped based on the actual position of the target, and the cropped image is stored. Cropping the image before storing it greatly reduces the storage resources, so that the image storage of the drone is maximized while meeting the requirements of miniaturization.

[0068] S5, update the target motion trajectory and predict the position of the target in the next frame image, and repeat steps S3 to S5 until the tracking task is completed.

[0069] like Figure 2 As shown, the embodiment of the present invention also provides a multi-threaded drone multi-target tracking device, comprising:

[0070] The detection and matching module obtains two adjacent frames of images taken by the high-speed camera on the drone, and performs target detection and feature matching respectively to obtain the position information of the same target in the two previous and next frames of images, and fits the motion trajectory of each target according to the position information;

[0071] The target position prediction module allocates a tracking thread to each target. The tracking thread predicts the position of the target in the image at the next moment based on the UAV motion state information and the motion trajectory of the corresponding target.

[0072] The extended search module obtains the next frame of image, and performs an extended search on the next frame of image according to the predicted position of the target to obtain the actual position of the target in the next frame of image;

[0073] An image cropping module crops the next frame of image based on the actual position of the target according to a preset cropping rule, and stores the cropped image;

[0074] The target trajectory update module updates the target motion trajectory and predicts the position of the target in the next frame image.

[0075] See also Figure 3 , Figure 3 Schematic diagram of an electronic device provided by an embodiment of the present invention. Figure 3 As shown, an embodiment of the present invention provides an electronic device 500, including a memory 510, a processor 520, and a computer program 511 stored in the memory 510 and executable on the processor 520. When the processor 520 executes the computer program 511, the following steps are implemented:

[0076] S1, obtaining two adjacent frames of images taken by a high-speed camera on a drone, and performing target detection and feature matching respectively, obtaining the position information of the same target in the two preceding and succeeding frames of images, and fitting the motion trajectory of each target according to the position information;

[0077] S2, assigning a tracking thread to each target, wherein the tracking thread predicts the position of the target in the image at the next moment according to the motion state information of the UAV and the motion trajectory of the corresponding target;

[0078] S3, obtaining the next frame image, and performing an extended search on the next frame image according to the predicted position of the target to obtain the actual position of the target in the next frame image;

[0079] S4, cropping the next frame of image based on the actual position of the target according to a preset cropping rule, and storing the cropped image;

[0080] S5, update the target motion trajectory and predict the position of the target in the next frame image, and repeat steps S3 to S5 until the tracking task is completed.

[0081] See also Figure 4 , Figure 4 A schematic diagram of an embodiment of a computer-readable storage medium provided in an embodiment of the present invention. Figure 4 As shown, this embodiment provides a computer-readable storage medium 600, on which a computer program 611 is stored. When the computer program 611 is executed by a processor, the following steps are implemented:

[0082] S1, obtaining two adjacent frames of images taken by a high-speed camera on a drone, and performing target detection and feature matching respectively, obtaining the position information of the same target in the two preceding and succeeding frames of images, and fitting the motion trajectory of each target according to the position information;

[0083] S2, assigning a tracking thread to each target, wherein the tracking thread predicts the position of the target in the image at the next moment according to the motion state information of the UAV and the motion trajectory of the corresponding target;

[0084] S3, obtaining the next frame image, and performing an extended search on the next frame image according to the predicted position of the target to obtain the actual position of the target in the next frame image;

[0085] S4, cropping the next frame of image based on the actual position of the target according to a preset cropping rule, and storing the cropped image;

[0086] S5, update the target motion trajectory and predict the position of the target in the next frame image, and repeat steps S3 to S5 until the tracking task is completed.

[0087] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0088] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0089] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0090] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1A function specified in one or more boxes.

[0091] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0092] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0093] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A multi-threaded UAV multi-target tracking method, characterized in that: include: S1, obtaining two adjacent frames of images taken by a high-speed camera on a drone, and performing target detection and feature matching respectively, obtaining the position information of the same target in the two preceding and succeeding frames of images, and fitting the motion trajectory of each target according to the position information; S2, assigning a tracking thread to each target, wherein the tracking thread predicts the position of the target in the image at the next moment according to the motion state information of the UAV and the motion trajectory of the corresponding target; S3, obtaining the next frame image, and performing an extended search on the next frame image according to the predicted position of the target to obtain the actual position of the target in the next frame image; S4, cropping the next frame of image based on the actual position of the target according to a preset cropping rule, and storing the cropped image; S5, update the target motion trajectory and predict the position of the target in the next frame image, and repeat steps S3 to S5 until the tracking task is completed.

2. The method according to claim 1, characterized in that Step S1 includes: Obtain target feature information in the tracking task. Note: the task acquisition time is t0; Starting from time t0, target detection is performed on the image taken by the high-speed camera on the drone based on the target feature information until the tracking target appears in the image; The frame image in which the tracking target appears for the first time is recorded as the first frame image, and the pixel position information and feature information of each target contained in the first frame image are extracted; The next frame image captured by the high-speed camera on the drone is obtained, recorded as a sub-frame image, and target detection is performed on the sub-frame image to obtain pixel position information and feature information of each target contained in the sub-frame image; Matching the feature information of each target in the first frame image and the second frame image to obtain the pixel position information of the same target in the first frame image and the second frame image; The motion trajectory of the corresponding target is fitted according to the pixel position information.

3. The method according to claim 2, characterized in that Step S1 further includes: Extract the UAV motion status information based on the timestamps of the first frame image and the second frame image; The motion trajectory of the corresponding target is fitted according to the UAV motion state information and the pixel position information.

4. The method according to claim 1 or 3, characterized in that: The UAV motion status information is recorded as: The UAV motion equation is as follows: Where T is the frame interval, a t is the acceleration of the drone at time t, φ t is the steering angle of the drone, is the position of the drone at time t+1, is the speed of the drone at time t+1, θ t+1 is the heading angle of the UAV at time t+1, and g is the acceleration due to gravity.

5. The method according to claim 1, characterized in that The step of extending the search for the next frame of image according to the predicted position of the target to obtain the actual position of the target in the next frame of image includes: Generate an initial bounding rectangle according to the size of the target; Extending the four sides of the initial circumscribed rectangular frame outward by a distance of multiple pixels to obtain an extended rectangular frame; The center position of the extended rectangular box is set as the predicted position of the target, and the target is detected in the extended rectangular box to obtain the actual position of the target.

6. The method according to claim 5, characterized in that If no target is detected in the extended rectangular frame, the four sides of the extended rectangular frame are extended outward by a plurality of pixel distances to obtain a new extended rectangular frame, and target detection is performed.

7. The method according to claim 6, characterized in that If no target is detected in the new extended rectangular frame, it is determined whether the number of expansions of the extended rectangular frame is less than a preset threshold and whether the size of the extended rectangular frame is less than a preset size. If so, the rectangular frame is continued to be expanded and target detection is performed; otherwise, target detection is performed in the full image.

8. A multi-threaded drone multi-target tracking device, characterized in that: include: The detection and matching module obtains two adjacent frames of images taken by the high-speed camera on the drone, and performs target detection and feature matching respectively to obtain the position information of the same target in the two previous and next frames of images, and fits the motion trajectory of each target according to the position information; The target position prediction module allocates a tracking thread to each target. The tracking thread predicts the position of the target in the image at the next moment based on the UAV motion state information and the motion trajectory of the corresponding target. The extended search module obtains the next frame of image, and performs an extended search on the next frame of image according to the predicted position of the target to obtain the actual position of the target in the next frame of image; An image cropping module crops the next frame of image based on the actual position of the target according to a preset cropping rule, and stores the cropped image; The target trajectory update module updates the target motion trajectory and predicts the position of the target in the next frame image.

9. An electronic device, characterized in that: include: Memory for storing computer software programs; A processor is used to read and execute the computer software program, thereby implementing a multi-threaded UAV multi-target tracking method as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium, characterized in that: The storage medium stores a computer software program, and when the computer software program is executed by the processor, it implements the multi-threaded drone multi-target tracking method as described in any one of claims 1 to 7.

Images