Unmanned aerial vehicle ground multi-target tracking method and system

Through the combination of optical flow point tracking and target detection, the problem of multi-target tracking of drones during oil and gas pipeline inspection is solved, and the stable tracking and matching of ground targets is achieved, repeated alarms and false alarms are reduced, and the improvement of improvements is improved. The system's response speed and adaptability are achieved.

CN119027461APending Publication Date: 2024-11-26XIAN INNO AVIATION TECH CO LTD

Patent Information

Application Number
CN202411002763.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-24
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The existing drone multi-target tracking algorithms have problems such as target matching failure, pseudo-target generation and repeated alarms in oil and gas pipeline inspection scenarios, especially when the targets are sparse and the movement is complex, there is a lack of suitable general algorithms.

Method used

Using a combination of optical flow point tracking and object detection, the motion and static targets are decoupled through initialization processing, optical flow point tracking, object detection and multi-object tracking frameworks, and the pyramid Lucas-Kanade optical flow algorithm and object detection algorithm are used. Achieve stable tracking and matching of multiple targets to reduce GPU resource usage.

Benefits of technology

It significantly improves the tracking effect and scene adaptability of the drone to ground targets, reduces the repeated alarms and false alarms of target detection, and improves the system's response speed and detection rate.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027461A_ABST
    Figure CN119027461A_ABST
Patent Text Reader

Abstract

The invention discloses an unmanned aerial vehicle over-the-ground multi-target tracking method, and the method comprises the steps: carrying out the initialization processing of a video image shot in the inspection process of an unmanned aerial vehicle, and reading a first frame image in the video image; extracting optical flow points in the first frame of image, and meanwhile, endowing each optical flow point with different PIDs (Proportion Integration Differentiation); tracking the optical flow points, setting the same PID for the same optical flow points, deleting the optical flow points which are not tracked in the current frame, and adding new optical flow points according to judgment; using a target detection algorithm to detect all supported targets in the current video frame, including the pixel positions, sizes and categories of the targets; a sort algorithm architecture is adopted, and the motion of a target in a picture is divided into decoupling of aircraft motion and target motion and decoupling of a static target and moving target tracking strategy; and counting the targets of the same TID, the categories of all detection frames and static and dynamic types, and updating the category and static and dynamic identifiers of the target by using the target with the highest occurrence frequency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle (UAV) applications, and in particular to a method and system for tracking multiple ground targets by an unmanned aerial vehicle (UAV). Background Art

[0002] With the development and progress of drone technology, the use of drone-mounted cameras to identify ground targets has been widely used in the field of drone inspections. In promoting the informatization transformation and digital upgrade of oil and gas pipeline networks, using drones to conduct daily inspections according to planned routes and identify and warn of abnormal targets appearing in the video has become an important part.

[0003] Multi-target tracking technology assigns a unique number to each target identified in each frame, which is the prerequisite for accurate warning of abnormal targets. Since some detection targets are also moving when the drone is flying, this undoubtedly brings difficulties to target tracking. In recent years, some tracking algorithms based on deep learning, although they can achieve good continuous tracking effects, have high requirements on equipment and require large-scale data training to derive a generalized model. Classic algorithms dedicated to multi-target tracking, such as sort, deepsort, etc., can be better applied to general scenarios, but when the target matching fails, the target will be automatically lost and a new pseudo target will be generated, resulting in multiple repeated alarms.

[0004] When drones are inspecting oil and gas pipelines, they usually plan flight routes, and then conduct routine inspections along fixed routes. A group company is responsible for multiple inspection routes and performs operations simultaneously during a fixed time period. The abnormal targets of drone oil and gas pipeline inspections are mainly: personnel, vehicles, illegal buildings, waterlogged areas, etc., and generally there are few people and vehicles near the pipelines. When an abnormal target alarm is issued, relevant staff are required to go to the scene to handle it. There are stricter practical requirements for repeated alarms and false alarms of abnormal target alarms. At present, there is no general algorithm suitable for this proprietary scenario. Summary of the invention

[0005] In view of the above-mentioned defects or deficiencies in the prior art, it is desired to provide a method and system for tracking multiple targets on the ground by a UAV.

[0006] In a first aspect, an embodiment of the present application provides a method for tracking multiple ground targets by a drone, the method comprising:

[0007] S1: Initialize the video images captured during the drone inspection process, read the first frame of the video, and record the image width and height as subsequent boundary restrictions; extract the optical flow points in the first frame of the image, and assign different PIDs to each optical flow point;

[0008] S2: Track the optical flow points, set the same PID for the same optical flow points, delete the optical flow points that are not tracked in the current frame, and add new optical flow points based on the judgment;

[0009] S3: Use the target detection algorithm to detect all supported targets in the current video frame, including: pixel position, size and category of the target;

[0010] S4: Construct a multi-target tracking framework based on the pixel position, size and category of the target;

[0011] S5: Set the target marked as TS::Lost in the multi-target tracking framework to TS::Deleted and delete it from the target chain to be tracked; at the same time, the category and static and dynamic attributes of the target are statistically analyzed, and the highest value of the category and static and dynamic attributes of the target in all detection frames is used as the category and static and dynamic attributes of the target;

[0012] S6: Each time a new video image frame is read, steps S2 to S5 are repeated to achieve multi-target tracking of the video.

[0013] In one embodiment, extracting optical flow points in the first frame of image includes:

[0014] Grayscale the first frame image, extract the corner points in the image, and perform non-maximum suppression on the corner points whose distance is within the threshold;

[0015] The reserved corner points are used as the system initialization optical flow points, and the state of each optical flow point is recorded as PT::Init, and a globally unique PID is assigned.

[0016] In one embodiment, tracking the optical flow points, setting the same PID for the same optical flow points, and deleting the optical flow points that are not tracked in the current frame include:

[0017] Use the pyramid Lucas-Kanade optical flow algorithm to search for the optical flow points marked in the previous frame in the current frame. The state of the optical flow points that are not tracked and matched is recorded as PT::Lost, and the state of the tracked optical flow is recorded as PT::Match, and the PID of the optical flow points matching the current frame is updated;

[0018] When the position of the optical flow point in the current frame is less than 2 pixels away from the image boundary, its state is set to PT::Lost;

[0019] Delete all optical flow points marked as PT::Lost in the current frame.

[0020] In one embodiment, adding a new optical flow point according to the judgment includes:

[0021] Divide the image into M*N grids of equal size and construct a mask image of the same size as the image. Too fine a division will make the system optical flow point tracking very time-consuming; where M and N are positive integers;

[0022] Every preset time or when the number of optical flow points in the current frame is less than M*N*0.4, sparse optical flow points are added, otherwise this step is skipped;

[0023] Set a mask with the same size as the current frame image and initialize it to all 1s; traverse all optical flow points in the current frame, set all pixels in the grid where the optical flow points are located to 0, and form a mask image for corner point extraction;

[0024] Corner points are extracted in the area with a value of 1 in the mask. Corner points whose distance is smaller than the grid size are suppressed. The retained corner points are then assigned a globally unique PID and the state is set to PT::Init as newly added optical flow points.

[0025] In one embodiment, the S3 includes:

[0026] Taking the initialization frame as the benchmark, it is executed every T frames, using the target detection algorithm to detect all supported target types in the current video frame.

[0027] The present application also provides a UAV-to-ground multi-target tracking system, comprising:

[0028] The initialization module is used to initialize the video images captured during the drone inspection process, read the first frame of the video image, and record the image width and height as subsequent boundary restrictions; extract the optical flow points in the first frame of the image, and at the same time, assign different PIDs to each optical flow point;

[0029] The optical flow tracking module is used to track the optical flow points. The same optical flow points are assigned the same PID. The optical flow points that are not tracked in the current frame are deleted, and new optical flow points are added based on the judgment.

[0030] The template detection module is used to detect all supported targets in the current video frame using the target detection algorithm, including: the pixel position, size and category of the target;

[0031] The construction module is used to build a multi-target tracking framework based on the pixel position, size and category of the target;

[0032] The post-processing module is used to set the target marked as TS::Lost in the multi-target tracking framework to TS::Deleted and delete it from the target chain to be tracked; at the same time, a statistical method is used to determine the category and static and dynamic attributes of the target, and the highest value of the category and static and dynamic attributes in all detection frames of the target is used as the category and static and dynamic attributes of the target.

[0033] In one embodiment, extracting optical flow points in the first frame of image includes:

[0034] Grayscale the first frame image, extract the corner points in the image, and perform non-maximum suppression on the corner points whose distance is within the threshold;

[0035] The reserved corner points are used as the system initialization optical flow points, and the state of each optical flow point is recorded as PT::Init, and a globally unique PID is assigned.

[0036] In one embodiment, tracking the optical flow points, setting the same PID for the same optical flow points, and deleting the optical flow points that are not tracked in the current frame include:

[0037] Use the pyramid Lucas-Kanade optical flow algorithm to search for the optical flow points marked in the previous frame in the current frame. The state of the optical flow points that are not tracked and matched is recorded as PT::Lost, and the state of the tracked optical flow is recorded as PT::Match, and the PID of the optical flow points matching the current frame is updated;

[0038] When the position of the optical flow point in the current frame is less than 2 pixels away from the image boundary, its state is set to PT::Lost;

[0039] Delete all optical flow points marked as PT::Lost in the current frame.

[0040] The beneficial effects of this application include:

[0041] The present application provides a method for tracking multiple ground targets by unmanned aerial vehicles, which significantly improves the tracking effect and scene adaptability of unmanned aerial vehicles by decoupling the aircraft motion from the target motion and the static target from the moving target tracking strategy. At the same time, this method can detect targets every other frame, reducing the occupation of GPU resources by target detection, so that the system can use idle GPU resources to support other algorithms that rely on deep learning models for oil and gas event alarms and support the processing of multi-channel videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0043] Figure 1 A schematic diagram of a flow chart of a method for tracking multiple ground targets by a drone provided in an embodiment of the present application is shown;

[0044] Figure 2 A schematic diagram of the main links of multi-target tracking provided by an embodiment of the present application is shown;

[0045] Figure 3 A target matching flow chart provided in an embodiment of the present application is shown;

[0046] Figure 4 The IOU schematic diagram provided in the embodiment of the present application is shown;

[0047] Figure 5 An example diagram of moving target matching provided by an embodiment of the present application is shown;

[0048] Figure 6 A target mismatch processing flow chart provided in an embodiment of the present application is shown;

[0049] Figure 7 A logic block diagram provided by an embodiment of the present application is shown;

[0050] Figure 8 A target tracking effect diagram provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0051] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant application, rather than to limit the application. It is also necessary to explain that, for ease of description, only the parts related to the application are shown in the accompanying drawings.

[0052] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0053] Please refer to Figure 1 As shown, Figure 1 A method for tracking multiple ground targets by a drone provided in an embodiment of the present application is shown, the method comprising:

[0054] S1: Initialize the video images captured during the drone inspection process, read the first frame of the video, and record the image width and height as subsequent boundary restrictions; extract the optical flow points in the first frame of the image, and assign different PIDs to each optical flow point;

[0055] S2: Track the optical flow points, set the same PID for the same optical flow points, delete the optical flow points that are not tracked in the current frame, and add new optical flow points based on the judgment;

[0056] S3: Use the target detection algorithm to detect all supported targets in the current video frame, including: pixel position, size and category of the target;

[0057] S4: Construct a multi-target tracking framework based on the pixel position, size and category of the target;

[0058] S5: Set the target marked as TS::Lost in the multi-target tracking framework to TS::Deleted and delete it from the target chain to be tracked; at the same time, the category and static and dynamic attributes of the target are statistically analyzed, and the highest value of the category and static and dynamic attributes of the target in all detection frames is used as the category and static and dynamic attributes of the target;

[0059] S6: Each time a new video image frame is read, steps S2 to S5 are repeated to achieve multi-target tracking of the video.

[0060] Using the above technical solution:

[0061] a) By establishing the relationship between the optical flow points of the previous and next frames, the target motion in the image captured during the drone inspection is decoupled, and the motion of the detected target is decomposed into two parts: camera motion and target motion, which are estimated separately to improve the accuracy of target position prediction and the matching effect;

[0062] b) In the process of matching the detection results with the targets, the static targets and the moving targets are decoupled and matched in two steps. For the static targets, fast matching is performed, and for the moving targets, a global cost matrix is ​​constructed for optimal matching, which decouples and improves the matching effect and efficiency of passing through the static target gathering area;

[0063] c) For unmatched targets, they are decomposed into static targets and moving targets and the loss logic is judged separately. When static targets such as puddles, construction, holes, buildings and other immovable ground targets are not detected, optical flow prediction is performed until the target is out of the picture, ensuring long-term tracking of targets with low detection rate;

[0064] d) The system supports alternate-frame target detection, and the intermediate frames are predicted, which can free up more GPU resources for the system to be used for other more complex algorithms or to support the analysis and processing of multiple videos.

[0065] Specifically, the method includes:

[0066] (1) Initialization

[0067] (1.1) Image width and height initialization. Read the first frame of the extracted image, record the image width and height as ImgWidth and ImgHeight, which are used to constrain the target boundary and optical flow point position in subsequent processing;

[0068] (1.2) Initialization of optical flow points. Grayscale the first frame image, extract the corner points in the image, and perform non-maximum suppression on the corner points with close distances (taking 1080P video as an example, the distance threshold is 60 pixels, and a maximum of 200 corner points are extracted). After that, the retained corner points are used as the system initialization optical flow points, and the state of each optical flow point is recorded as PT::Init, and a globally unique PID is assigned. The state identification of the optical flow point is shown in Table 1.

[0069] (2) Optical flow tracking

[0070] (2.1) Search Match

[0071] (2.1.1) Use the classic pyramid Lucas-Kanade optical flow algorithm to search for the optical flow points marked in the previous frame in the current frame. The state of the optical flow points that are not tracked and matched is recorded as PT::Lost, and the state of the tracked optical flow is recorded as PT::Match, and the PID of the matched optical flow points in the current frame is updated;

[0072] (2.1.2) When the position of the optical flow point in the current frame is less than 2 pixels away from the image boundary, its state is set to PT::Lost, and the boundary point is considered unreliable;

[0073] (2.1.3) Delete all optical flow points marked as PT::Lost in the current frame.

[0074] Table 1. Optical flow point status identification

[0075] Serial number Logo meaning 1 KP::Unknown Unknown, default initial value 2 KP::Initial Newly added points 3 KP::Match Matching points 4 KP::Lost Missing, unmatched points

[0076] (2.2) Add sparse optical flow points

[0077] (2.2.1) Divide the image into M*N grids of equal size (the default setting is M=N=10) and construct a mask image of the same size as the image. Too fine a division will make the system optical flow point tracking very time-consuming;

[0078] (2.2.2) Every once in a while (default is 2 seconds) or when the number of optical flow points in the current frame is less than M*N*0.4, add sparse optical flow points, otherwise skip step (2.2);

[0079] (2.2.3) Set a mask with the same size as the current frame image and initialize it to all 1s. Traverse all optical flow points in the current frame and set all pixels in the grid where the optical flow points are located to 0 to form a mask image for corner point extraction;

[0080] (2.2.4) Corner points are extracted only in the area where the value in the mask is 1. Corner points whose distance is smaller than the grid size are suppressed. The retained corner points are then assigned a globally unique PID and the state is set to PT::Init as newly added optical flow points.

[0081] (3) Object Detection

[0082] (3.1) Taking the initialization frame as the benchmark, it is executed every T frames, using the mainstream target detection algorithm (such as yolov5) to detect all supported target types in the current video frame. The main research point of this invention is the multi-target tracking and matching strategy under the video shot by the drone, which is applicable to the existing general target detector as long as it contains the pixel position, size and category of the target.

[0083] (4) Multi-target tracking

[0084] (4.1) Multi-target tracking framework

[0085] (4.1.1) Each target is assigned a unique target TID, and the target status identification is defined as shown in Table 2. During the multi-target tracking process, a target chain is maintained, which contains all the target information that has not been deleted.

[0086] Table 2. Target status identification

[0087] Serial number Logo meaning 1 TS::Unknown Unknown, default initial value 2 TS::Initial Newly added goals 3 TS::Match Matching target 4 TS::Lost Target lost 5 TS::Uncertain Unsure, waiting for the next frame to match 6 TS::Deleted Target deletion

[0088] (4.1.2) The overall multi-target tracking framework is similar to sort, such as Figure 2 The detection result is matched with the current frame prediction result of the existing trajectory. When the prediction result of the trajectory does not match the current frame detection result, it is considered that the target may be lost and is considered to be deleted; the current frame detection result does not match the prediction result of the trajectory, it is considered that it may be a new target and a new trajectory is considered to be added; the current frame detection result matches the prediction result of a certain trajectory, it is considered to match, and the current frame position of the trajectory is updated.

[0089] (4.1.3) Core step target matching process, such as Figure 3 The detection results are estimated to offset the target position changes caused by camera motion. Then, rough matching and secondary matching are performed based on the distance relationship. If there are still unmatched objects, the optical flow points of the current detection are marked, and the optical flow points are tracked during the next matching to determine whether they are the same target.

[0090] (4.1.4) The detection result of the current frame is:

[0091] A={[x n ,y i1 , x i2 ,y i2 ,cls],0≤i<N obj , i is an integer}, the number of detected targets is N obj Each detected target contains x i1 ,y i1 , x i2 ,y i2The upper left x-axis image coordinate, upper left y-axis image coordinate, lower right x-axis image coordinate, and lower right y-axis image coordinate of the i-th detection result are respectively identified, and cls represents the category of the detection.

[0092] (4.1.5) The target chains maintained in the system are denoted as set B,

[0093] B={[x j1 ,y j1 , x j2 ,y j2 , cls, tid, status], 0≤j<N target , j is an integer}, where the number of targets in the target chain is N target Each target contains x j1 ,y j1 , x j2 ,y j2 They respectively identify the upper left x-axis image coordinate, upper left y-axis image coordinate, lower right x-axis image coordinate, and lower right y-axis image coordinate of the j-th target. cls is the category, tid is the unique identifier of the target, and status is the target state described in Table 2, initialized to TS::Unknown.

[0094] (4.2) Target Matching

[0095] (4.2.1) Position estimation

[0096] (4.2.1.1) The optical flow point set in the previous frame image is pts0, and the optical flow point set in the current frame image is pts1. The target in the target chain to be tracked is denoted as vTargets.

[0097] (4.2.1.2) Traverse the targets in the target chain and calculate the N closest to the target in pts0 po The specific implementation process is as follows:

[0098] a) Calculate the pixel distance between each optical flow point in pts0 and the target center;

[0099] b) Sort the optical flow points in pts0 by distance from small to large;

[0100] c) Take the first N po The optical flow points that are not in the target are taken as the neighboring point set of the target;

[0101] d) Repeat steps a) to c) until all targets are traversed.

[0102] (4.2.1.3) Restore the box position prediction. For each target, calculate the average difference between the motion of the same PID optical flow points in the current frame and the previous frame around the target as the pixel offset of the target caused by the drone motion. Then, superimpose the corresponding offset on the target as motion compensation to achieve position estimation.

[0103] (4.2.2) Static Target Matching

[0104] (4.2.2.1) Calculate the distance between the detection box and the target. Here the distance is expressed using the standard IOU, such as Figure 4 The red box A and the green box B are two target detection boxes, and the IOU calculation formula between them is:

[0105]

[0106] Among them, S A∩B is the area of ​​the intersection of A and B, S A∪B is the union of the areas of A and B.

[0107] (4.2.2.2) Traverse each detection result in turn, and compare the IOU with the targets in all unmatched target chains. When IOU>0.6, it is considered a match, and the current detection result is no longer compared with the targets in the subsequent target chain. After traversing in turn, the unmatched detections and targets enter the moving target matching stage.

[0108] (4.2.3) Moving Target Matching

[0109] (4.2.3.1) Calculate the distance between the detection box and the target. Here the distance is defined as:

[0110]

[0111] The calculation of dis2 can be divided into two parts. When the two compared rectangles A and B are close in size, length and width, that is:

[0112]

[0113] Among them, dis c is the straight-line distance between the center points of the two rectangular boxes, bw a , bh a 、bw b , bh b are the width and height of rectangular boxes A and B respectively. If they are not close, dis2 is directly assigned a large number, such as 10. The actual physical meaning of dis2 indicates how many times the distance between two similar boxes is their long sides. The basis for judging similarity can be simply defined as:

[0114]

[0115] Among them, S a , S b are the areas of rectangular boxes A and B respectively, ths h 、ths l are high and low thresholds, which are 1.2 and 0.8 respectively.

[0116] (4.2.3.2) Calculate the cost matrix. Calculate the distance between the detection result and the target in the target chain, and construct a cost matrix in the form of Table 3.

[0117] Table 3. Cost Matrix

[0118]

[0119] This is a typical assignment problem. After minimum cost and using the Hungarian algorithm (a commonly used method), the results are: target 1 matches detection 2, target 3 matches detection 3, target 4 matches detection 1, and target 2 does not match, reaching the global optimum.

[0120] (4.2.3.3) Corresponding to the match under the global optimality, a secondary threshold judgment is performed. When the distance is greater than a certain threshold, the match is considered unreasonable and determined to be a mismatch. The threshold here is an empirical value, which is related to the fastest moving speed of the target in the picture and the number of matching interval frames. It is selected according to the scene-specific test. When the scene target moves fast, 1.5 target 1 is matched with detection 2; when the scene target moves slowly, 1.2 is taken, then it matches.

[0121] (4.2.4) Target not matched

[0122] (4.2.4.1) If there is no matching detection result, the detection result is considered to be a new target, the value of the global target tid is increased by 1 as the unique identifier of the new target, and the state of the new target is initialized to TS::Initial.

[0123] (4.2.4.2) Unmatched targets are marked as TS::Uncertain, which can be divided into two parts: static target unmatched and moving target unmatched. Figure 6 As shown in the target mismatch processing flow, according to the static and dynamic identifiers added to the target in step (5), the target mismatch is divided into two cases: static target mismatch and dynamic target mismatch. The specific processing method is as follows:

[0124] a) Unmatched moving targets, mainly people and vehicles, if the current frame is not matched, Kalman prediction and position estimation are performed, and the next frame continues to match the detection result. If there is no match after a certain period of time, the target is considered lost and marked as TS::Lost;

[0125] b) Unmatched static targets can be divided into ground objects and people / vehicles. Since ground objects do not move, if they are not matched, they will be predicted and matched until they are out of the picture, ensuring continuous tracking even when the detection rate is low, reducing target loss and the generation of false targets; for static people / vehicles, the same processing method is used as for unmatched moving targets, except that the time threshold for loss judgment can be slightly longer. Targets judged to be lost are marked as TS::Lost.

[0126] (4.2.5) Target Matching

[0127] (4.2.5.1) If the target matches, the Kalman update and prediction of position and size are performed, and the state of the target is set to TS::Match.

[0128] (5) Post-processing

[0129] (5.1) Set the target marked as TS::Lost to TS::Deleted and delete it from the target chain to be tracked;

[0130] (5.2) For the category and static and dynamic attributes of the target, a statistical method is used, and the highest value of the category and static and dynamic attributes of the target in all detection frames is used as the category and static and dynamic attributes of the target.

[0131] (6) Each time a new video frame is read, steps (2) to (5) are repeated to achieve multi-target tracking in the video.

[0132] For example, a drone or a drone equipped with a drone library can respond to route missions immediately, issue route inspection instructions through the control platform, and start the drone to extend the route. The drone collects oil and gas pipeline video data streams in real time through the onboard camera, and performs intelligent analysis of the video data through the onboard smart box, including: target detection, multi-target tracking, abnormal alarm and other logic. Afterwards, the analysis results are transmitted to the data management platform and the mobile platform in the form of video streams, pictures, and data streams through the aircraft-side data transmission and image transmission modules for management and control, such as Figure 7 .

[0133] In a specific embodiment of the present invention, the onboard smart box can be replaced with a cloud server, and the video stream can be pushed to the cloud computing server through the image transmission module on the drone. The computing server analyzes the video stream and then pushes the result to the data management platform and the mobile platform for management and control. The computing server method can simultaneously support the needs of multi-channel drone video processing.

[0134] Figure 8The input resolution is 1080P video, the drone is flying at an altitude of 120 meters, the flight speed is 6m / s, the pitch angle is 45°, and the flight time is 15 minutes. The green line is the optical flow trajectory, and the yellow dot on the line is the position of the optical flow point in the current frame. It can be seen that after the patented method, the optical flow points are evenly distributed in the image, and the points are very stable during the tracking process. The above points ensure the stability of static target tracking in the actual multi-target tracking when the detection is unstable.

[0135] Table 4 and Table 5 are comparisons between the method of the present invention and other methods. In the time consumption comparison, when deepsort triggers the target matching logic, it is necessary to call an additional model to distinguish the target ID. In the oil and gas pipeline inspection scenario, the probability of target overlap is small, so the overall time consumption is not significantly increased compared to sort; compared to sort, the method of this paper simplifies the matching strategy and improves the response time when matching static targets, but the optical flow tracking module itself increases the time consumption by an additional 2ms per frame. In the actual target matching link, there is no obvious speed advantage. When the target is considered lost, a new tid will be assigned to the target. In Table 5, the optimized matching strategy for static targets enables the method of this paper to reduce the duplication of static targets. For moving targets, the decoupling strategy of the present invention improves the accuracy of position prediction. The improvement of the moving target matching distance function also enables matching when there is tree occlusion. There is little occlusion between targets in the oil and gas inspection scenario, so deepsort does not have a clear advantage over sort.

[0136] Table 4. Comparison of target matching time

[0137] method sort deepsort Methods Speed ​​(ms) 1.3 2.4 1.1

[0138] Table 5. Repeated target situation

[0139]

[0140] In some embodiments, extracting optical flow points in the first frame of image includes:

[0141] Grayscale the first frame image, extract the corner points in the image, and perform non-maximum suppression on the corner points whose distance is within the threshold;

[0142] The reserved corner points are used as the system initialization optical flow points, and the state of each optical flow point is recorded as PT::Init, and a globally unique PID is assigned.

[0143] In some embodiments, tracking the optical flow points, setting the same PID for the same optical flow points, and deleting the optical flow points that are not tracked in the current frame include:

[0144] Use the pyramid Lucas-Kanade optical flow algorithm to search for the optical flow points marked in the previous frame in the current frame. The state of the optical flow points that are not tracked and matched is recorded as PT::Lost, and the state of the tracked optical flow is recorded as PT::Match, and the PID of the optical flow points matching the current frame is updated;

[0145] When the position of the optical flow point in the current frame is less than 2 pixels away from the image boundary, its state is set to PT::Lost;

[0146] Delete all optical flow points marked as PT::Lost in the current frame.

[0147] In some embodiments, the adding of new optical flow points according to the judgment includes: dividing the image into M*N grids of equal size, constructing a mask image of the same size as the image, and too fine division will make the system optical flow point tracking very time-consuming; wherein M and N are positive integers;

[0148] Every preset time or when the number of optical flow points in the current frame is less than M*N*0.4, sparse optical flow points are added, otherwise this step is skipped;

[0149] Set a mask with the same size as the current frame image and initialize it to all 1s; traverse all optical flow points in the current frame, set all pixels in the grid where the optical flow points are located to 0, and form a mask image for corner point extraction;

[0150] Corner points are extracted in the area with a value of 1 in the mask. Corner points whose distance is smaller than the grid size are suppressed. The retained corner points are then assigned a globally unique PID and the state is set to PT::Init as newly added optical flow points.

[0151] In some embodiments, S3 includes: taking the initialization frame as a reference, executing once every T frames, using a target detection algorithm to detect all supported target types in the current video frame.

[0152] In some embodiments, the S4 includes: assigning a unique target TID to each target;

[0153] When the prediction result of the trajectory does not match the detection result of the current frame, it is considered that the target is lost and deleted; when the detection result of the current frame does not match the prediction result of the trajectory, it is considered a new target and a new trajectory is added; when the detection result of the current frame matches the prediction result of a certain trajectory, it is considered a match and the current frame position of the trajectory is updated;

[0154] The detection results are estimated to offset the target position changes caused by camera motion. Then, the distance relationship is used for primary and secondary matching. If there are still unmatched targets, the optical flow points of the current detection are marked, and the optical flow points are tracked during the next matching to determine whether they are the same target.

[0155] A={[x i1 ,y i1 , x i2 ,y i2 ,cls],0≤i<N obj , i is an integer}, the number of detected targets is N obj Each detected target contains x i1 ,y i1 , x i2 ,y i2 Respectively identify the upper left corner x-axis image coordinate, upper left corner y-axis image coordinate, lower right corner x-axis image coordinate, and lower right corner y-axis image coordinate of the i-th detection result, and cls represents the category of the detection;

[0156] B={[x j1 ,y j1 , x j2 ,y j2 , cls, tid, status], 0≤j<N target , j is an integer}, where the number of targets in the target chain is N target Each target contains x j1 ,y j1 , x j2 ,y j2 Identify the upper left x-axis image coordinate, upper left y-axis image coordinate, lower right x-axis image coordinate, and lower right y-axis image coordinate of the j-th target respectively. cls is the category, tid is the unique identifier of the target, and status is the target status, which is initialized to TS::Unknown.

[0157] In some embodiments, S6 includes: setting the target marked as TS::Lost to TS::Deleted, and deleting it from the target chain to be tracked;

[0158] For the category, static and dynamic attributes of the target, a statistical method is used, and the highest value of the category, static and dynamic attributes of the target in all detection frames is used as the category, static and dynamic attributes of the target.

[0159] It should be noted that although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0160] Furthermore, the present application also provides a UAV-to-ground multi-target tracking system, comprising:

[0161] The initialization module is used to initialize the video images captured during the drone inspection process, read the first frame of the video image, and record the image width and height as subsequent boundary restrictions; extract the optical flow points in the first frame of the image, and at the same time, assign different PIDs to each optical flow point;

[0162] The optical flow tracking module is used to track the optical flow points. The same optical flow points are assigned the same PID. The optical flow points that are not tracked in the current frame are deleted, and new optical flow points are added based on the judgment.

[0163] The template detection module is used to detect all supported targets in the current video frame using the target detection algorithm, including: the pixel position, size and category of the target;

[0164] The construction module is used to build a multi-target tracking framework based on the pixel position, size and category of the target;

[0165] The post-processing module is used to set the target marked as TS::Lost in the multi-target tracking framework to TS::Deleted and delete it from the target chain to be tracked; at the same time, a statistical method is used to determine the category and static and dynamic attributes of the target, and the highest value of the category and static and dynamic attributes in all detection frames of the target is used as the category and static and dynamic attributes of the target.

[0166] It should be understood that the units or modules recorded in the system are the same as those in the reference Figure 1 The various steps in the described method correspond to each other. Therefore, the operations and features described above for the method are also applicable to the system and the units contained therein, and will not be repeated here. The system can be pre-implemented in a browser or other security application of an electronic device, or loaded into a browser or its security application of an electronic device by downloading or the like. The corresponding units in the system can cooperate with the units in the electronic device to implement the solution of the embodiment of the present application.

[0167] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of application involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the aforementioned application concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A method for tracking multiple ground targets by a UAV, characterized in that: The method includes: S1: Initialize the video images captured during the drone inspection process, read the first frame of the video, and record the image width and height as subsequent boundary restrictions; extract the optical flow points in the first frame of the image, and assign different PIDs to each optical flow point; S2: Track the optical flow points, set the same PID for the same optical flow points, delete the optical flow points that are not tracked in the current frame, and add new optical flow points based on the judgment; S3: Use the target detection algorithm to detect all supported targets in the current video frame, including: pixel position, size and category of the target; S4: Construct a multi-target tracking framework based on the pixel position, size and category of the target; S5: Set the target marked as TS::Lost in the multi-target tracking framework to TS::Deleted and delete it from the target chain to be tracked; at the same time, the category and static and dynamic attributes of the target are statistically analyzed, and the highest value of the category and static and dynamic attributes of the target in all detection frames is used as the category and static and dynamic attributes of the target; S6: Each time a new video image frame is read, steps S2 to S5 are repeated to achieve multi-target tracking of the video.

2. The method for tracking multiple ground targets by a drone according to claim 1, characterized in that: The extracting of optical flow points in the first frame image includes: Grayscale the first frame image, extract the corner points in the image, and perform non-maximum suppression on the corner points whose distance is within the threshold; The reserved corner points are used as the system initialization optical flow points, and the state of each optical flow point is recorded as PT::Init, and a globally unique PID is assigned.

3. The method for tracking multiple ground targets by a drone according to claim 2, characterized in that: The optical flow points are tracked, the same optical flow points are set to the same PID, and the optical flow points that are not tracked in the current frame are deleted, including: Use the pyramid Lucas-Kanade optical flow algorithm to search for the optical flow points marked in the previous frame in the current frame. The state of the optical flow points that are not tracked and matched is recorded as PT::Lost, and the state of the tracked optical flow is recorded as PT::Match, and the PID of the optical flow points matching the current frame is updated; When the position of the optical flow point in the current frame is less than 2 pixels away from the image boundary, its state is set to PT::Lost; Delete all optical flow points marked as PT::Lost in the current frame.

4. The method for tracking multiple ground targets by a drone according to claim 3, characterized in that: The adding of new optical flow points according to the judgment includes: Divide the image into M*N grids of equal size and construct a mask image of the same size as the image. Too fine a division will make the system optical flow point tracking very time-consuming; where M and N are positive integers; Every preset time or when the number of optical flow points in the current frame is less than M*N*0.4, sparse optical flow points are added, otherwise this step is skipped; Set a mask with the same size as the current frame image and initialize it to all 1s; traverse all optical flow points in the current frame, set all pixels in the grid where the optical flow points are located to 0, and form a mask image for corner point extraction; Corner points are extracted in the area with a value of 1 in the mask. Corner points whose distance is smaller than the grid size are suppressed. The retained corner points are then assigned a globally unique PID and the state is set to PT::Init as newly added optical flow points.

5. The method for tracking multiple ground targets by a drone according to claim 4, characterized in that: The S3 includes: Taking the initialization frame as the benchmark, it is executed every T frames, using the target detection algorithm to detect all supported target types in the current video frame.

6. A UAV ground multi-target tracking system, characterized in that: include: The initialization module is used to initialize the video images captured during the drone inspection process, read the first frame of the video image, and record the image width and height as subsequent boundary restrictions; extract the optical flow points in the first frame of the image, and at the same time, assign different PIDs to each optical flow point; The optical flow tracking module is used to track the optical flow points. The same optical flow points are assigned the same PID. The optical flow points that are not tracked in the current frame are deleted, and new optical flow points are added based on the judgment. The template detection module is used to detect all supported targets in the current video frame using the target detection algorithm, including: the pixel position, size and category of the target; The construction module is used to build a multi-target tracking framework based on the pixel position, size and category of the target; The post-processing module is used to set the target marked as TS::Lost in the multi-target tracking framework to TS::Deleted and delete it from the target chain to be tracked; At the same time, a statistical method is used to determine the category and static and dynamic attributes of the target, and the highest value of the category and static and dynamic attributes of the target in all detection frames is used as the category and static and dynamic attributes of the target.

7. The UAV ground multi-target tracking system according to claim 6, characterized in that: The extracting of optical flow points in the first frame image includes: Grayscale the first frame image, extract the corner points in the image, and perform non-maximum suppression on the corner points whose distance is within the threshold; The reserved corner points are used as the system initialization optical flow points, and the state of each optical flow point is recorded as PT::Init, and a globally unique PID is assigned.

8. The UAV ground multi-target tracking system according to claim 7, characterized in that: The optical flow points are tracked, the same optical flow points are set to the same PID, and the optical flow points that are not tracked in the current frame are deleted, including: Use the pyramid Lucas-Kanade optical flow algorithm to search for the optical flow points marked in the previous frame in the current frame. The state of the optical flow points that are not tracked and matched is recorded as PT::Lost, and the state of the tracked optical flow is recorded as PT::Match, and the PID of the optical flow points matching the current frame is updated; When the position of the optical flow point in the current frame is less than 2 pixels away from the image boundary, its state is set to PT::Lost; Delete all optical flow points marked as PT::Lost in the current frame.

Citation Information

Patent Citations

  • Method for controlling automatically video camera moving according image position of objects

    CN101043616A

  • Multi-target tracking method based on improved TLD (tracking-learning-detected)

    CN106709938A

  • Dynamic pixel point distribution identification method, system and equipment based on sparse optical flow

    CN116740477A

  • Anti-jitter real-time target tracking method based on LK optical flow method

    CN117440246A

  • Public place multi-camera target tracking and re-identification algorithm and system

    CN118057479A

Cited By

  • Satellite video multi-target tracking method based on optical flow and state space model

    CN121236121A

  • Unmanned aerial vehicle video multi-target tracking method based on motion decoupling and related device

    CN121438170A