Real-time multi-unmanned-aerial-vehicle multi-target tracking method for target dense scene, storage medium and electronic equipment

Abstract

The invention provides a real-time multi-unmanned aerial vehicle multi-target tracking method for a target dense scene, a storage medium and electronic equipment, and relates to the technical field of unmanned aerial vehicle target tracking, and the method comprises the steps: firstly, enabling all targets in a video first frame to be projected into a same coordinate system through a first projection transformation matrix obtained through calculation of the video first frame; euclidean distances between center point coordinates of all targets in video first frames of every two unmanned aerial vehicles are calculated in the coordinate system, bipartite graph matching is carried out by using the Euclidean distances, and a multi-target tracker is generated by using a target matching result of the video first frames in a target tracking process of subsequent frames of the videos. And carrying out target tracking on subsequent frames of the video by using the multi-target tracker, and if a new target is found, carrying out matching again, and finally realizing tracking of a plurality of targets in the video shot by the unmanned aerial vehicle. According to the method provided by the invention, while multi-target tracking is realized, the target tracking precision is also improved.

Description

Technical Field

[0001] The present application relates to the technical field of drone target tracking, and in particular to a real-time multi-drone multi-target tracking method, storage medium and electronic device for target-intensive scenes. Background Art

[0002] With the rapid improvement of computer computing power, fields such as computer vision, artificial intelligence, and machine perception have developed rapidly, and research on drone target tracking is becoming increasingly important.

[0003] Most existing UAV target tracking methods track targets by extracting image features, but this method has the problem of inaccurate tracking. Summary of the invention

[0004] The embodiments of the present application provide a real-time multi-UAV multi-target tracking method, storage medium and electronic device for target-dense scenarios to improve the accuracy of multi-target tracking by UAVs.

[0005] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0006] According to a first aspect of an embodiment of the present application, a real-time multi-UAV multi-target tracking method for a target-intensive scene is provided, characterized in that it includes:

[0007] Determine the first projection transformation matrix between two drones based on the first frame of the video taken by the drone;

[0008] Determine the center point coordinates of each target in the first frame of the video based on the first frame of the video taken by the drone;

[0009] Based on the first projection transformation matrix and the coordinates of the center point of each target in the first frame of the video, a bipartite graph matching algorithm is used to match two targets, and the two successfully matched targets are assigned the same ID, and the unmatched targets are assigned new unique IDs until all targets in the first frame of the video taken by all drones are assigned;

[0010] Generate a multi-target tracker for each UAV using the target assignment results in the first frame of the video;

[0011] Send the subsequent frames of the video taken by each drone to the corresponding multi-target tracker for target tracking and detection;

[0012] If a new target without an assigned ID appears in the subsequent frames of the video, the second projection transformation matrix between the two drones is calculated based on the center points of all successfully matched targets in the current frame of the video;

[0013] Based on the second projection transformation matrix and the center point coordinates of each target in the subsequent frames of the video, a bipartite graph matching algorithm is used to perform pairwise target matching. The two successfully matched targets are assigned the same ID, and the unmatched targets are assigned a new unique ID until all targets in the subsequent frames of the video taken by all drones are assigned.

[0014] In some embodiments of the present application, based on the above solution, determining the first projection transformation matrix between two drones based on the first frame of the video shot by the drone includes:

[0015] Get the first frame of the video taken by each drone at the same time;

[0016] Extract the key points and key point features of the first frame of all videos;

[0017] The first projection transformation matrix between each pair of UAVs is calculated based on the key points and key point features.

[0018] In some embodiments of the present application, based on the above solution, determining the center point coordinates of each target in the first frame of the video based on the first frame of the video taken by the drone includes:

[0019] Perform target detection on the first frames of videos shot by multiple drones to obtain target detection results;

[0020] Determine the center point coordinates of each target based on the target detection results.

[0021] In some embodiments of the present application, based on the aforementioned solution, the pairwise target matching is performed using a bipartite graph matching algorithm based on the first projection transformation matrix and the center point coordinates of each target in the first frame of the video, including:

[0022] Projecting the center point coordinates of multiple targets between two UAVs into the same position coordinate system based on the first projection transformation matrix;

[0023] Based on the projected center point coordinates, the Euclidean distances between the center point coordinates of all targets in the first frame of the video shot by one of the two drones and the center point coordinates of all targets in the first frame of the video shot by the other drone are calculated;

[0024] The Euclidean distance is used as a cost matrix and sent to a bipartite graph matching algorithm to perform pairwise target matching.

[0025] In some embodiments of the present application, based on the above solution, the process of performing pairwise target matching further includes:

[0026] The matching results whose cost matrix is ​​less than the set threshold are restricted to be valid matches.

[0027] In some embodiments of the present application, based on the above solution, the second projection transformation matrix between two drones is calculated according to the center points of all successfully matched targets in the current frame of the video, including:

[0028] The center points of all successfully matched targets in the current frame of the video shot by each drone are used as feature points;

[0029] A second projection transformation matrix between each pair of UAVs is calculated based on the feature points.

[0030] In some embodiments of the present application, based on the aforementioned solution, the pairwise target matching is performed using a bipartite graph matching algorithm based on the second projection transformation matrix and the center point coordinates of each target in subsequent frames of the video, including:

[0031] Get the center point coordinates of each target in subsequent frames of the video;

[0032] Projecting the center point coordinates of multiple targets between two UAVs into the same position coordinate system based on the second projection transformation matrix;

[0033] Based on the projected center point coordinates, the Euclidean distances between the center point coordinates of all targets in the subsequent frames of the video shot by one of the two drones and the center point coordinates of all targets in the subsequent frames of the video shot by the other drone are calculated;

[0034] The Euclidean distance is used as a cost matrix and sent to a bipartite graph matching algorithm to perform pairwise target matching.

[0035] In some embodiments of the present application, based on the above solution, obtaining the coordinates of the center point of each target in the subsequent frames of the video includes:

[0036] Perform target detection on subsequent frames of videos shot by multiple drones to obtain target detection results;

[0037] The center point coordinates of each target in subsequent frames of the video are determined based on the target detection results.

[0038] According to a second aspect of an embodiment of the present application, a computer-readable storage medium is provided, wherein the storage medium stores computer instructions, and when the computer instructions are executed on a computer, the computer executes the method according to the first aspect.

[0039] According to a third aspect of an embodiment of the present application, there is provided an electronic device, including: a memory and a processor;

[0040] The memory is used to store computer instructions;

[0041] The processor is used to call the computer instructions stored in the memory so that the electronic device executes the method as described in the first aspect.

[0042] The technical solution of the present application is to first project all targets in the first frame of the video into the same coordinate system through the first projection transformation matrix calculated through the first frame of the video, and then calculate the Euclidean distance between the center point coordinates of all targets in the first frame of the video of each drone in this coordinate system, and then use the Euclidean distance to perform bipartite graph matching. In the process of target tracking in subsequent frames of the video, a multi-target tracker is generated using the target matching result of the first frame of the video, and the multi-target tracker is used to track targets in subsequent frames of the video. If a new target is found, it is matched again, and finally the tracking of multiple targets in the video taken by the drone is achieved, thereby improving the tracking accuracy.

[0043] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0045] Figure 1 A schematic diagram of a real-time multi-UAV multi-target tracking method for a target-intensive scene according to an embodiment of the present application is shown;

[0046] Figure 2 A block diagram of an electronic device according to an embodiment of the present application is shown;

[0047] Figure 3 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0048] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concept of the example embodiments to those skilled in the art.

[0049] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0050] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0051] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0052] It should be noted that the “plurality” mentioned in this article refers to two or more.

[0053] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the objects used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those shown or described.

[0054] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all the embodiments. Based on the embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] Some embodiments of the present application will be described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments may be combined with each other.

[0056] See also Figure 1 , shows a schematic diagram of a real-time multi-UAV multi-target tracking method for a target-dense scene according to an embodiment of the present application.

[0057] like Figure 1 As shown, a real-time multi-UAV multi-target tracking method for target-dense scenes is demonstrated, which specifically includes steps S100 to S700.

[0058] refer to Figure 1 , step S100, determining the first projection transformation matrix between each pair of drones based on the first frame of the video taken by the drone.

[0059] It should be noted that, in the technical solution of the present application, it is necessary to ensure that each drone has an overlapping shooting area with at least one other drone, so as to ensure that the videos shot by the two drones have the same parts.

[0060] In some feasible embodiments, based on the above solution, determining the first projection transformation matrix between two drones based on the first frame of the video shot by the drone includes:

[0061] Get the first frame of the video taken by each drone at the same time;

[0062] Extract the key points and key point features of the first frame of all videos;

[0063] The first projection transformation matrix between each pair of UAVs is calculated based on the key points and key point features.

[0064] It can be understood that, through this step, the first projection transformation matrix between all drones can be determined to prepare for subsequent target matching.

[0065] For example, assuming there are drone A and drone B, obtain the first frame A and the first frame B of the video taken simultaneously by drone A and drone B, extract the key points and key point features of the first frame A and the first frame B respectively, and use the extracted key points and key point features to calculate the first projection transformation matrix AB between drone A and drone B.

[0066] Continue to refer Figure 1 , step S200, determining the center point coordinates of each target in the first frame of the video based on the first frame of the video taken by the drone.

[0067] In some feasible embodiments, based on the above solution, determining the center point coordinates of each target in the first frame of the video based on the first frame of the video shot by the drone includes:

[0068] Perform target detection on the first frames of videos shot by multiple drones to obtain target detection results;

[0069] Determine the center point coordinates of each target based on the target detection results.

[0070] It can be understood that, through this step, the center point coordinates of all targets in the first frame of the video shot by each drone can be determined.

[0071] Exemplarily, based on the foregoing example, the first frame A of the video taken by drone A contains an image of a person and an image of a car, and the first frame B of the video taken by drone B contains an image of a person, an image of a car, and an image of a dog. Through the target detection method, the center point coordinates of the image of the person and the image of the car in the first frame A of the video, as well as the center point coordinates of the image of the person, the image of the car, and the image of the dog in the first frame B of the video are obtained respectively.

[0072] Continue to refer Figure 1 , step S300, based on the first projection transformation matrix and the center point coordinates of each target in the first frame of the video, use the bipartite graph matching algorithm to perform pairwise target matching, assign the same ID to the two successfully matched targets, and assign a new unique ID to the unmatched targets until all targets in the first frame of the video taken by all drones are assigned.

[0073] It should be noted that, through this step, the same targets in the first frames of the videos shot by two drones can be matched, and the unmatched targets can be given unique IDs for distinction.

[0074] In some feasible embodiments, based on the above solution, the pairwise target matching is performed using a bipartite graph matching algorithm based on the first projection transformation matrix and the center point coordinates of each target in the first frame of the video, including:

[0075] Projecting the center point coordinates of multiple targets between two UAVs into the same position coordinate system based on the first projection transformation matrix;

[0076] Based on the projected center point coordinates, the Euclidean distances between the center point coordinates of all targets in the first frame of the video shot by one of the two drones and the center point coordinates of all targets in the first frame of the video shot by the other drone are calculated;

[0077] The Euclidean distance is used as a cost matrix and sent to a bipartite graph matching algorithm to perform pairwise target matching.

[0078] Exemplarily, based on the foregoing example, the center point coordinates of the image person and the image car in the first frame A of the video are projected into the same coordinate system as the center point coordinates of the image person, the image car and the image dog in the first frame B of the video through the first projection transformation matrix AB, and then the Euclidean distances between the image person in the first frame A of the video and the image person in the first frame B of the video, the image person in the first frame A of the video and the image car in the first frame B of the video, the image person in the first frame A of the video and the image dog in the first frame B of the video, the image car in the first frame A of the video and the image person in the first frame B of the video, the image car in the first frame A of the video and the image car in the first frame B of the video, and the image car in the first frame A of the video and the image dog in the first frame B of the video are calculated, and these Euclidean distances are used as the cost The matrix is ​​sent to the bipartite graph matching algorithm for calculation, and the image person and image car in the first frame A of the video are matched with the image person, image car and image dog in the first frame B of the video. If the match is successful, the same ID is assigned, and the unmatched target is assigned a new unique ID. For example, the image person in the first frame A of the video and the image person in the first frame B of the video are matched successfully, and the image car in the first frame A of the video and the image car in the first frame B of the video are matched successfully, then the image person in the first frame A of the video and the image person in the first frame B of the video are assigned the same ID: person, and the image car in the first frame A of the video and the image car in the first frame B of the video are assigned the same ID: car, and the image dog in the first frame B of the video is assigned a unique ID: dog.

[0079] In some feasible embodiments, based on the above solution, the process of pairwise target matching further includes:

[0080] The matching results whose cost matrix is ​​less than the set threshold are restricted to be valid matches.

[0081] It should be noted that the effective matching mentioned here refers to successful matching. In this way, it is possible to prevent two targets that are far apart from each other from matching, which would affect the matching accuracy.

[0082] Continue to refer Figure 1 ,Step S400, generating a multi-target tracker for each UAV using the target allocation result in the first frame of the video.

[0083] It can be understood that the generated multi-target tracker can identify the targets that have been successfully matched in the first frame to avoid repeated tracking and detection.

[0084] Continue to refer Figure 1 , step S500, sending the subsequent frames of the video taken by each drone to the corresponding multi-target tracker for target tracking detection.

[0085] It should be noted that the function of the multi-target tracker is to continuously assign the same ID to the same target that appears continuously. For example, if the image person and image car that have been successfully matched continue to appear, the multi-target tracker will continue to assign the ID person to the image person and the ID car to the image car.

[0086] It should be noted that when the multi-target tracker tracks and detects subsequent frames of the video, it will divide the targets in the subsequent frames of the video into matched targets and unmatched targets. The matched targets do not need to be processed, while the unmatched targets need to be re-matched.

[0087] Continue to refer Figure 1 , Step S600, if a new target without an assigned ID appears in a subsequent frame of the video, the second projection transformation matrix between the two drones is calculated based on the center points of all the targets successfully matched in the current frame of the video.

[0088] In some feasible embodiments, based on the above solution, the second projection transformation matrix between two drones is calculated according to the center points of all successfully matched targets in the current frame of the video, including:

[0089] The center points of all successfully matched targets in the current frame of the video shot by each drone are used as feature points;

[0090] A second projection transformation matrix between each pair of UAVs is calculated based on the feature points.

[0091] It should be noted that, since the drone is moving, the first projection transformation matrix calculated in the first frame of the video cannot be used, and the transformation matrix of multiple drones in the current frame needs to be re-estimated. Thanks to the dense target scene, in this embodiment, the center points of all targets matched in the current frame are used as feature points to estimate the second projection transformation matrix, thereby avoiding the use of time-consuming key point detection algorithms.

[0092] Exemplarily, based on the foregoing example, the second projection transformation matrix may be estimated using the center points of the successfully matched target A1 and target B1 as feature points.

[0093] Continue to refer Figure 1 , step S700, based on the second projection transformation matrix and the center point coordinates of each target in the subsequent frames of the video, use the bipartite graph matching algorithm to perform pairwise target matching, assign the same ID to the two successfully matched targets, and assign a new unique ID to the unmatched targets, until all targets in the subsequent frames of the video taken by all drones are assigned.

[0094] In some feasible embodiments, based on the above solution, the pairwise target matching is performed using a bipartite graph matching algorithm based on the second projection transformation matrix and the center point coordinates of each target in subsequent frames of the video, including:

[0095] Get the center point coordinates of each target in subsequent frames of the video;

[0096] Projecting the center point coordinates of multiple targets between two UAVs into the same position coordinate system based on the second projection transformation matrix;

[0097] Based on the projected center point coordinates, the Euclidean distances between the center point coordinates of all targets in the subsequent frames of the video shot by one of the two drones and the center point coordinates of all targets in the subsequent frames of the video shot by the other drone are calculated;

[0098] The Euclidean distance is used as a cost matrix and sent to a bipartite graph matching algorithm to perform pairwise target matching.

[0099] In some feasible embodiments, based on the above solution, obtaining the coordinates of the center point of each target in the subsequent frames of the video includes:

[0100] Perform target detection on subsequent frames of videos shot by multiple drones to obtain target detection results;

[0101] The center point coordinates of each target in subsequent frames of the video are determined based on the target detection results.

[0102] Exemplarily, based on the foregoing example, for example, a person, a car, and an image dog are detected in a subsequent frame A of a video shot by drone A, and a person, a car, a dog, and an image tree are detected in a subsequent frame B of a video shot by drone B. The center point coordinates of the image dog in the subsequent frame A of the video, the dog, and the image tree in the subsequent frame B of the video are obtained respectively, and the center point coordinates of the image dog in the subsequent frame A of the video are projected to the same coordinate system as the center point coordinates of the dog and the image tree in the subsequent frame B of the video through the second projection transformation matrix, and then the image dog in the subsequent frame A of the video and the dog and the image tree in the subsequent frame B of the video are calculated. The Euclidean distance between the image dog in the subsequent frame A and the image tree in the subsequent frame B of the video is calculated, and these Euclidean distances are sent to the bipartite graph matching algorithm as the cost matrix for calculation. The image dog in the subsequent frame A of the video is matched with the dog and the image tree in the subsequent frame B of the video. If the match is successful, the same ID is assigned, and the unmatched target is assigned a new unique ID. For example, if the image dog in the subsequent frame A of the video and the dog in the subsequent frame B of the current video are matched successfully, the same ID is assigned: dog; if the image tree in the subsequent frame B of the current video is not matched successfully, the ID: tree is assigned for distinction.

[0103] In summary, the multi-target tracking method provided in the embodiment of the present application improves the tracking accuracy of the targets while achieving multi-target tracking.

[0104] like Figure 2As shown, an embodiment of the present application also provides an electronic device 200, including a memory 210, a processor 220, and a computer program 211 stored in the memory 210 and executable on the processor. When the processor 220 executes the computer program 211, the steps of the above-mentioned real-time multi-UAV multi-target tracking method for target-dense scenes are implemented.

[0105] Since the electronic device introduced in this embodiment is a device used to implement a real-time multi-UAV multi-target tracking method for target-dense scenes in the embodiment of the present application, based on the method introduced in the embodiment of the present application, technical personnel in this field can understand the specific implementation mode of the electronic device of this embodiment and its various variations. Therefore, how the electronic device implements the method in the embodiment of the present application is not introduced in detail here. As long as the equipment used by technical personnel in this field to implement the method in the embodiment of the present application is within the scope of protection of this application.

[0106] During the specific implementation process, when the computer program 211 is executed by a processor, any implementation method in the embodiments corresponding to the first aspect can be implemented.

[0107] Figure 3 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown.

[0108] It should be noted that Figure 3 The computer system 300 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0109] like Figure 3 As shown, the computer system 300 includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 302 or the program loaded from the storage part 308 to the random access memory (RAM) 303, such as executing the method described in the above embodiment. In the RAM 303, various programs and data required for system operation are also stored. The CPU 301, ROM 302 and RAM 303 are connected to each other through a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0110] The following components are connected to the I / O interface 305: an input section 306 including a keyboard, a mouse, etc.; an output section 307 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 310 as needed so that a computer program read therefrom is installed into the storage section 308 as needed.

[0111] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 309, and / or installed from a removable medium 311. When the computer program is executed by a central processing unit (CPU) 301, various functions defined in the system of the present application are executed.

[0112] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0113] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0114] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0115] As another aspect, the present application further provides a computer program product or a computer program, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes a real-time multi-UAV multi-target tracking method for target-intensive scenes as described in the above embodiment.

[0116] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the real-time multi-UAV multi-target tracking method for target-intensive scenes described in the above embodiment.

[0117] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0118] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation methods of the present application.

[0119] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary techniques in the art that are not disclosed in the present application. It should be understood that the present application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A real-time multi-UAV multi-target tracking method for target-intensive scenes, characterized in that: include: Determine the first projection transformation matrix between two drones based on the first frame of the video taken by the drone; Determine the center point coordinates of each target in the first frame of the video based on the first frame of the video taken by the drone; Based on the first projection transformation matrix and the coordinates of the center point of each target in the first frame of the video, a bipartite graph matching algorithm is used to match two targets, and the two successfully matched targets are assigned the same ID, and the unmatched targets are assigned new unique IDs until all targets in the first frame of the video taken by all drones are assigned; Generate a multi-target tracker for each UAV using the target assignment results in the first frame of the video; Send the subsequent frames of the video taken by each drone to the corresponding multi-target tracker for target tracking and detection; If a new target without an assigned ID appears in the subsequent frames of the video, the second projection transformation matrix between the two drones is calculated based on the center points of all successfully matched targets in the current frame of the video; Based on the second projection transformation matrix and the center point coordinates of each target in the subsequent frames of the video, a bipartite graph matching algorithm is used to perform pairwise target matching. The two successfully matched targets are assigned the same ID, and the unmatched targets are assigned a new unique ID until all targets in the subsequent frames of the video taken by all drones are assigned.

2. The method according to claim 1, characterized in that: The method of determining a first projection transformation matrix between two drones based on the first frame of the video shot by the drone includes: Get the first frame of the video taken by each drone at the same time; Extract the key points and key point features of the first frame of all videos; The first projection transformation matrix between each pair of UAVs is calculated based on the key points and key point features.

3. The method according to claim 1, characterized in that: The method of determining the center point coordinates of each target in the first frame of the video based on the first frame of the video shot by the drone includes: Perform target detection on the first frames of videos shot by multiple drones to obtain target detection results; Determine the center point coordinates of each target based on the target detection results.

4. The method according to claim 1, characterized in that: The performing pairwise target matching using a bipartite graph matching algorithm based on the first projection transformation matrix and the center point coordinates of each target in the first frame of the video includes: Projecting the center point coordinates of multiple targets between two UAVs into the same position coordinate system based on the first projection transformation matrix; Based on the projected center point coordinates, the Euclidean distances between the center point coordinates of all targets in the first frame of the video shot by one of the two drones and the center point coordinates of all targets in the first frame of the video shot by the other drone are calculated; The Euclidean distance is used as a cost matrix and sent to a bipartite graph matching algorithm to perform pairwise target matching.

5. The method according to claim 4, characterized in that The process of pairwise target matching also includes: The matching results whose cost matrix is ​​less than the set threshold are restricted to be valid matches.

6. The method according to claim 1, characterized in that The second projection transformation matrix between two drones is calculated based on the center points of all targets successfully matched in the current frame of the video, including: The center points of all successfully matched targets in the current frame of the video shot by each drone are used as feature points; A second projection transformation matrix between each pair of UAVs is calculated based on the feature points.

7. The method according to claim 1, characterized in that The step of performing pairwise target matching based on the second projection transformation matrix and the center point coordinates of each target in subsequent frames of the video using a bipartite graph matching algorithm includes: Get the center point coordinates of each target in subsequent frames of the video; Projecting the center point coordinates of multiple targets between two UAVs into the same position coordinate system based on the second projection transformation matrix; Based on the projected center point coordinates, the Euclidean distances between the center point coordinates of all targets in the subsequent frames of the video shot by one of the two drones and the center point coordinates of all targets in the subsequent frames of the video shot by the other drone are calculated; The Euclidean distance is used as a cost matrix and sent to a bipartite graph matching algorithm to perform pairwise target matching.

8. The method according to claim 7, characterized in that The step of obtaining the center point coordinates of each target in subsequent frames of the video includes: Perform target detection on subsequent frames of videos shot by multiple drones to obtain target detection results; The center point coordinates of each target in subsequent frames of the video are determined based on the target detection results.

9. A computer-readable storage medium, characterized in that: The storage medium stores computer instructions, and when the computer instructions are executed on a computer, the computer executes the method according to any one of claims 1 to 8.

10. An electronic device, characterized in that: include: Memory and processor; The memory is used to store computer instructions; The processor is used to call the computer instructions stored in the memory so that the electronic device executes the method as described in any one of claims 1-8.

Images