A multi-target long-term tracking method for unmanned aerial vehicles

Abstract

The application provides a multi-target long-term tracking method for a UAV, which comprises the following steps: a detection module identifies target detection frame images and target relative position information of all monitoring targets from environmental image information, and obtains a target detection frame image sequence and a target relative position information sequence; a cross-lens tracking module extracts multi-dimensional apparent features of each target detection frame image; for each frame in the target detection frame image sequence, a matching module performs a shake-proof operation thereon, divides the frame into high-score target frames and low-score target frames according to a confidence degree, obtains a fusion loss matrix based on the multi-dimensional apparent features and the target relative position information, and then performs track matching through a Hungarian algorithm to complete target tracking. The application fuses a lens shake-proof strategy in the matching process, and introduces a cross-lens tracking module, thereby solving the problem of tracking errors caused by shaking of the UAV, and solving the problem of ID addition caused by rotation of the camera.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of target tracking, and particularly relates to a multi-target long-term tracking method for an unmanned aerial vehicle. BACKGROUND

[0002] Multi-target tracking has developed rapidly in the past ten years. Early typical methods include MeanShift and particle filtering methods, but the overall accuracy is relatively low, and single-target tracking is mainly used. With the continuous improvement of the detection target accuracy, a detection-based tracking framework appears, and SORT, DeepSORT, BYTETrack and BoTSORT are typical algorithms.

[0003] The detection-based tracking framework process is mainly divided into detection and matching two parts. However, due to the flexibility of the framework, the detection part can be continuously updated with the development of the target detection algorithm, and the detection result also affects the accuracy of the subsequent matching process. The SORT algorithm is the most initial classic algorithm in detection tracking. The core of the matching process is the combination of Kalman filtering and the Hungarian algorithm. The main process is to first perform Kalman filtering prediction on the position of the previous frame detection target frame in the current frame, and then use the Hungarian matching algorithm to perform optimal matching between the current detection target frame and the predicted target frame. The DeepSORT algorithm is an improved version of SORT. The main improvement is to increase the CNN network to extract the appearance features of the target frame, increase the robustness to occlusion, and propose a cascade matching strategy to realize the correlation measurement between the real target frame and the predicted target frame by combining the motion information and the appearance information. The BYTETrack algorithm proposes a new data association method. Unlike the SORT algorithm, the algorithm takes into account the low-confidence detection target frame due to occlusion. The algorithm uses the similarity between the detection frame and the track to retain high-score detection results while removing the background from low-score results, and to improve the track continuity. First, the high-score frame is matched with the previous track, then the low-score frame and the high-score track frame that fails to match for the first time are matched, and for the track that has not been matched, the detection frame with high score is used to construct a new tracking track. The BoTSORT algorithm is a more robust tracker. The overall matching process is basically the same as that of BYTETrack, but it adds a ReID module and a camera motion compensation method.

[0004] In the prior art multi-target tracking technical solutions, like SORT and BYTETrack, only position information is used for matching, and the premise for accurate matching is that the background cannot change greatly between frames, and the target moves at a relatively slow speed. If the premise is not met, the matching result has a serious ID Switch (identity conversion) problem, and even short-term continuous tracking of a target cannot be achieved. When the camera is shaken or rotated, it is almost impossible to continuously track the target only by using position information. Although DeepSORT and BoTSORT add ReID (cross-camera tracking) to re-identify and recognize the objects under the monitoring camera, to a certain extent, the ID Switch situation is alleviated. However, when the target is blocked for a long time or the camera rotates greatly, the position information added in the matching process cannot play a role, and thus the same target cannot be associated. SUMMARY

[0005] To solve the above technical problems, the present application provides a method for long-term tracking of multiple targets by an unmanned aerial vehicle.

[0006] The first aspect of the present application discloses a method for long-term tracking of multiple targets by an unmanned aerial vehicle. The method is based on a tracking system to locate and track multiple targets obtained by the unmanned aerial vehicle. The tracking system includes a detection module, a cross-camera tracking module, and a matching module.

[0007] The method comprises the following steps:

[0008] S1, the detection module identifies the target detection frame image and the target relative position information of all monitored targets from the environmental image information, and obtains the target detection frame image sequence and the target relative position information sequence.

[0009] S2, the cross-camera tracking module extracts the multi-dimensional apparent feature of each target detection frame image.

[0010] S3, for each frame in the target detection frame image sequence, the matching module performs a de-jittering operation, and then divides it into high-score target frames and low-score target frames according to the confidence. Based on the multi-dimensional apparent feature and the target relative position information, a fusion loss matrix is obtained, and then the Hungarian algorithm is used to perform trajectory matching on the scored target frames to complete target tracking.

[0011] According to the method of the first aspect of the present application, in the step S3, the step of de-jittering operation comprises:

[0012] An homographic transformation matrix is obtained based on the target detection frame image of the current frame and the target detection frame image of the previous frame, and then the current detection target frame after perspective transformation is obtained according to the product of the target detection frame image of the previous frame and the homographic transformation matrix.

[0013] According to the method of the first aspect of the application, the step of obtaining the homographic transformation matrix based on the target bounding box image of the current frame and the target bounding box image of the last frame comprises:

[0014] The feature points and the corresponding descriptors of each feature point of the target bounding box image of the current frame and the target bounding box image of the last frame are extracted respectively by using the ORB operator;

[0015] The matched feature point pairs are obtained by matching the descriptors of the feature points;

[0016] After the abnormal matching filtering of the feature point pairs, the correct feature point pairs are obtained;

[0017] The optimal 8 feature point pairs are selected based on the random sample consensus algorithm, and the homographic transformation matrix is calculated based on the 8 feature point pairs.

[0018] According to the method of the first aspect of the application, in the step S3, the step of obtaining the fusion loss matrix based on the multi-dimensional appearance feature and the target relative position information comprises:

[0019] The position of the last frame detected target box in the current detected target box in the trajectory pool is predicted based on the Kalman filter, and a predicted target box is obtained;

[0020] The current detected target box and the predicted target box are respectively modeled into 2D Gaussian distribution based on the target relative position information;

[0021] The distance between the Gaussian distributions corresponding to the current detected target box and the predicted target box is calculated;

[0022] The calculation formula of the distance between the Gaussian distributions is:

[0023]

[0024] wherein, represents the distance between the Gaussian distributions of the current detected target box a and the predicted target box b, represents the Gaussian distribution of the current detected target box a, represents the Gaussian distribution of the predicted target box b, , represents the center point coordinates of the current detected target box a, represents the width of the current detected target box a, represents the height of the current detected target box a, , represents the center point coordinates of the predicted target box b, represents the width of the predicted target box b, represents the height of the predicted target box b.

[0025] normalizing the distance between the Gaussian distributions to obtain a position similarity between the current detection target frame and the predicted target frame;

[0026] The calculation formula of the position similarity is:

[0027] wherein, represents the similarity between the current detection target frame a and the predicted target frame b, and C represents a data set related constant;

[0028] According to the position similarity, an NWD loss matrix is calculated;

[0029] The formula of the NWD loss matrix is:

[0030] The similarity between the multi-dimensional apparent features corresponding to the current detection target frame and the predicted target frame is calculated by using the cos similarity;

[0031] The calculation formula of the similarity between the multi-dimensional apparent features is:

[0032]

[0033] wherein, represents the apparent feature of the i-th target frame, represents the apparent feature of the j-th target frame, represents the value of the i-th target frame in the k-th dimension, represents the value of the j-th target frame in the k-th dimension, represents the dimension of the apparent feature;

[0034] The similarity between the multi-dimensional apparent features is normalized to obtain an apparent feature loss matrix;

[0035] The apparent feature loss matrix is:

[0036] According to the NWD loss matrix and the apparent feature loss matrix, a fusion loss matrix is obtained;

[0037] The fusion loss matrix is:

[0038]

[0039] wherein, represents the threshold value of the position distance, and the current value is 0.3, represents the threshold value of the apparent feature.

[0040] According to the method of the first aspect of the present application, the track pool matched with the high-score target frame is a track currently associated, and the track pool matched with the low-score target frame is a track not successfully matched with the high-score target frame.

[0041] According to the method of the first aspect of the present application, after step S3, the method further comprises:

[0042] The cosine loss matrix is calculated by using the target appearance features not matched with the high-score target frame and the target appearance features in the deleted track, and the association matching is performed by using the Hungarian algorithm based on the cosine loss matrix.

[0043] According to the method of the first aspect of the present application, the successfully matched track is updated.

[0044] The updating is specifically performed by the following formula:

[0045]

[0046] wherein, indicates a weight parameter, the current value is 0.9, fea is the target appearance feature of the previous frame, and cur_fea is the current target appearance feature.

[0047] According to the method of the first aspect of the present application, step S2 comprises:

[0048] The image preprocessing is performed on the each target detection frame image.

[0049] Restnet50 is used as the backbone network to capture the feature map [1, 2048, 16, 8] of the preprocessed each target detection frame image.

[0050] The aggregation network is used to reduce the dimension and normalize the feature map [1, 2048, 16, 8] to obtain the multi-dimensional appearance feature [1, 2048, 1, 1].

[0051] The second aspect of the present application discloses an electronic device. The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor executes the computer program to realize the steps in the method for long-term tracking of multiple targets of a UAV according to any one of the first aspect of the present application.

[0052] The third aspect of the present application discloses a computer readable storage medium. The computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the steps in the method for long-term tracking of multiple targets of a UAV according to any one of the first aspect of the present application.

[0053] In summary, the scheme provided in the present application has the following technical effects: the present application combines lens anti-shake strategy in the matching process, solves the problem of tracking error when the unmanned aerial vehicle is affected by weather and shakes, and introduces the ReID model, adds the post-processing strategy after matching, solves the ID addition problem of the same person disappearing and reappearing in the camera when the camera rotates, and further realizes long-term stable target tracking of multiple targets under the view angle of the unmanned aerial vehicle. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the drawings needed in the specific embodiments or the prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0055] Figure 1 The flow chart of the unmanned aerial vehicle multi-target long-term tracking method according to an embodiment of the present application;

[0056] Figure 2 The structure diagram of the unmanned aerial vehicle multi-target long-term tracking system according to an embodiment of the present application;

[0057] Figure 3 The structure diagram of the electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical scheme and advantages of the embodiments of the present application more clear, the technical scheme in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0059] With the development of unmanned aerial vehicle technology, the application scenarios of unmanned aerial vehicles in the military field are also increasing, and target tracking is the basic ability construction when the unmanned aerial vehicle is applied, and long-term stable tracking target is one of the research difficulties. The main differences between multi-target tracking under the view angle of the unmanned aerial vehicle and other view angle target tracking are as follows: 1) most of the targets are small targets, generally less than Pixels; 2) is seriously affected by weather, when the abnormal weather such as strong wind, the camera stability is poor; 3) when the camera rotates, the same target disappears and reappears, and ID switch (identity conversion) is more likely to occur, and these factors are the main factors causing the failure of long-term stable tracking target. Therefore, in order to realize long-term stable tracking target of the unmanned aerial vehicle, a long-term trajectory tracking method of multiple targets is proposed. The method provided by the present application can solve the above technical problems.

[0060] Please refer to Figure 1 The first aspect of the present application discloses a long-term tracking method of multiple targets of an unmanned aerial vehicle; the method is based on a tracking system for positioning, tracking and monitoring multiple targets obtained by the unmanned aerial vehicle. Please refer to Figure 2 The tracking system comprises a detection module, a cross-mirror tracking module and a matching module.

[0061] The method comprises the following steps:

[0062] S1, the detection module identifies the target detection frame image and the target relative position information of all monitoring targets from the environmental image information, obtains the target detection frame image sequence and the target relative position information sequence;

[0063] The detection module is the basis of the early stage of the tracking algorithm, the input of the tracking is the output of the detection module, and the accuracy of the detection module also has a certain influence on the tracking effect. The current detection model uses YOLOX algorithm, and the input end uses the relatively mainstream Mosaic, MixUp and random flip technology, the backbone network is Darknet53 model, and the SPP layer is increased at the same time. In the present application, the training data set is composed of self-labeled data and open source data in a mixed manner, wherein the self-labeled data contains 10w+ images, the open source data is extracted from the extracted person images of COCO data, and the test uses 5000 images of COCO2017 test set for verification.

[0064] Suppose represents the kth image, the detection module detects N targets of the kth frame, and the output result is represented as

[0065] ,

[0066] Among them represents the center point of the target frame, width represents the width of the target frame, and height represents the height of the target frame.

[0067] S2, the cross-mirror tracking module extracts the multi-dimensional apparent feature of each target detection frame image;

[0068] The cross-mirror tracking (ReID) module is mainly divided into image preprocessing, backbone network, aggregation network, Head layer and Loss, and there are multiple implementation methods for each layer module sub-method, and the application does not make specific limitations. Step S2 comprises:

[0069] The image preprocessing is performed on each target detection frame image through image preprocessing;

[0070] Restnet50 is adopted as the backbone network to capture the feature map [1, 2048, 16, 8] of each target detection frame image after preprocessing;

[0071] The aggregation network is adopted to reduce the dimension and normalize the feature map [1, 2048, 16, 8] to obtain a multi-dimensional apparent feature [1, 2048, 1, 1].

[0072] The input of ReID is the detected target image, which can be represented as .

[0073] Image preprocessing is one of the common enhancement strategies in deep learning. In the preprocessing process, the application mainly includes resize, random flip, random erasing, random patch and cutout. Through the cleaning, standardization and expansion of data, the robustness and generalization ability of the model are enhanced.

[0074] The picture input size affects the feature map size, and large images can make the model learn more clear high-dimensional features. The application resizes the image to a uniform size of 384x128. This size is a fixed scale, and it is related to the use of resnet50 in the ReID model backbone. The target features are unified to 2048 dimensions.

[0075] The picture is randomly flipped, including random horizontal flipping and random vertical flipping. Random erasing is to remove a rectangular block of a predefined size range from the original image at different epochs during training. A rectangular block of [1 / 6, 1 / 2] of the original image height and width is randomly generated in the input image to perform erasing, that is, the RGB values in the random rectangular block range are set to random values.

[0076] Random patch and random erasing operation are similar, except that the RGB values of the random rectangular block of the image are obtained from other images. The cutout operation sets the removed position to [0, 0, 0].

[0077] ReID model backbone network: the classic Resnet50 is used as the backbone network. Considering that ResNet50 adds a residual structure, the network structure will not appear gradient disappearance when it is continuously deepened. The network structure contains 4 blocks. In order to reduce the feature dimension, reduce the calculation amount of the model, and increase the nonlinear expression ability of the model, different numbers of bottlenecks, B, are added in each block. Each bottleneck structure is convolution, followed by convolution, and finally convolution, which retains sufficient representation ability to capture the detailed information of the image.

[0078] In order to further improve the diversity and robustness of the features, an aggregation network is added to the backbone network and the head layer. The feature map [1, 2048, 16, 8] generated by Restnet50 is output as [1, 2048, 1, 1] after the pooling layer.

[0079] The dimension of the feature generated by the head layer is relatively large, and the feature needs to be reduced and normalized, including a BN layer and a decision layer.

[0080] The loss layer uses a cross-entropy loss function. The cross-entropy loss function is a commonly used loss function for classification problems. In the ReID model, each target is considered as a class, and the probability of the target belonging to the class is output.

[0081] In order to increase the robustness of the model, self-collection and multi-source data integration are performed on the data. The training data set comes from mark1501, DukeMTMC, MSMT17 and self-labeled data. The number of data categories reaches 10903. The self-labeled data is obtained by professional labeling team through target labeling according to real scene and unmanned aerial vehicle collected video data. The pedestrians with less than 100 frames are deleted and filtered.

[0082] ReID generates a 2048-dimensional appearance feature for each target. The appearance feature of all targets in the kth frame can be expressed as .

[0083] S3, for each frame in the target detection frame image sequence, the matching module performs anti-shake operation on it, and divides it into high-score target frame and low-score target frame according to the confidence. Based on the multi-dimensional appearance feature and the relative position information of the target, a fusion loss matrix is obtained, and then the Hungarian algorithm is used to match the target frame after scoring to complete the target tracking.

[0084] In the step S3, the step of performing anti-shake operation includes:

[0085] Obtain a homographic transformation matrix based on the target detection frame image of the current frame and the target detection frame image of the previous frame, and then obtain a perspective transformed current detection target frame according to a product of the target detection frame image of the previous frame and the homographic transformation matrix.

[0086] According to the method of the first aspect of the present application, the step of obtaining the homographic transformation matrix based on the target detection frame image of the current frame and the target detection frame image of the previous frame comprises:

[0087] Extract feature points and corresponding descriptors of each feature point of the target detection frame image of the current frame and the target detection frame image of the previous frame respectively by using an ORB operator;

[0088] Obtain matched feature point pairs by matching the descriptors of the feature points;

[0089] Obtain correct feature point pairs after performing abnormal matching filtering on the feature point pairs;

[0090] Select the optimal 8 feature point pairs based on a random sample consensus algorithm, and calculate the homographic transformation matrix through the 8 feature point pairs.

[0091] When the unmanned aerial vehicle is in the air, it is more susceptible to weather, and when the wind is strong, the lens will shake severely, which may cause ID to be incorrectly matched. In order to reduce IDSwitch caused by such reasons, an anti-shake process is added. The core point of the entire anti-shake process is to search for 8 correct feature points of the current frame and the previous frame image, and for this purpose, parallelism matching is added. The feature descriptor and feature point of the target detection frame image of the previous frame are represented as , and the feature descriptor and feature point of the target detection frame image of the current frame are represented as ;

[0092] In order to reduce the possibility of incorrect matching, the matching feature points are subjected to abnormal matching filtering by calculating the parallelism with the virtual line segment. Most of the feature points can be normally matched, and there is a parallel relationship. When a small part of the matching points is incorrect, the parallel relationship will disappear, so the included angle matrix between the matching point pairs is calculated. When the parallel relationship of a certain point pair with other point pairs is less than 50% of all point pairs, the point pair is deleted.

[0093] According to the method of the first aspect of the present application, in the step S3, the step of obtaining a fusion loss matrix based on the multi-dimensional appearance feature and the target relative position information comprises:

[0094] Obtain a predicted target frame based on the Kalman filter to predict the position of the previous frame detection target frame in the current detection target frame in the trajectory pool.

[0095] model the current detection target frame and the predicted target frame into 2D Gaussian distribution respectively based on the target relative position information;

[0096] For small targets, there will always be some background pixels in the target frame, because the real object cannot be exactly a rectangle. In the target frame, the foreground pixels are generally concentrated in the middle, and the background pixels are generally concentrated on the side, and the importance of the pixels decreases from the center to the boundary. In order to better weight each pixel in the target frame, the target frame can be modeled as a 2D Gaussian distribution. Specifically, for a horizontal target frame, it can be represented by an inscribed ellipse as follows:

[0097]

[0098] is the center point of the ellipse, is the x and y axis radius, respectively corresponding to the width and height of the target frame, , represents the center point of the target frame, represents the width of the target frame, represents the height of the target frame.

[0099] This ellipse is a distribution contour of a 2D Gaussian distribution. Therefore, the target frame can be modeled as a 2D Gaussian distribution, and the similarity between two target frames can be represented by the distance between the two Gaussian distributions.

[0100] calculate the distance between the Gaussian distributions corresponding to the current detection target frame and the predicted target frame;

[0101] The formula for calculating the distance between the Gaussian distributions is as follows:

[0102]

[0103] wherein, represents the distance between the Gaussian distributions of the current detection target frame a and the predicted target frame b, represents the Gaussian distribution of the current detection target frame a, represents the Gaussian distribution of the predicted target frame b, , represents the center point coordinates of the current detection target frame a, represents the width of the current detection target frame a, represents the height of the current detection target frame a, , represents the center point coordinates of the predicted target frame b, represents the width of the predicted target frame b, represents the height of the predicted target frame b.

[0104] ​The distance between the Gaussian distributions is normalized to obtain a position similarity between the current detection target frame and the predicted target frame;

[0105] The calculation formula of the position similarity is:

[0106]

[0107] Wherein, represents the similarity between the current detection target frame a and the predicted target frame b, and C represents a data set related constant;

[0108] An NWD loss matrix is calculated according to the position similarity;

[0109] The NWD loss matrix formula is:

[0110]

[0111] The similarity between the multi-dimensional apparent features corresponding to the current detection target frame and the predicted target frame is calculated by using the cos similarity;

[0112] The calculation formula of the similarity between the multi-dimensional apparent features is:

[0113]

[0114] Wherein, represents the apparent feature of the i-th target frame, represents the apparent feature of the j-th target frame, represents the value of the i-th target frame in the k-th dimension, represents the value of the j-th target frame in the k-th dimension, represents the dimension of the apparent feature;

[0115] The similarity between the multi-dimensional apparent features is normalized to obtain an apparent feature loss matrix;

[0116] The apparent feature loss matrix is:

[0117]

[0118] A fusion loss matrix is obtained according to the NWD loss matrix and the apparent feature loss matrix;

[0119] The fusion loss matrix is:

[0120] Wherein, represents the threshold value of the position distance, and the current value is 0.3, represents the threshold value of the apparent feature.

[0121] The position loss calculation method of small targets in the current tracking algorithm is based on the IoU measurement method, and the IoU is very sensitive to the change of the position of small targets. When the unmanned aerial vehicle shakes or the lens rotates, the position information contributes almost equal to 0 to the matching of the tracking process. From the scene requirement, the current position loss IOU method is not applicable in the unmanned aerial vehicle scene, because the distance between targets becomes zero due to slight lens shaking, and the position information between the targets cannot be accurately measured. In order to reduce this influence, the position loss of the application adopts the Normalized Gaussian Wasserstein Distance to calculate the similarity between the current target frame and the previous target frame.

[0122] According to the first aspect of the application, the trajectory pool matched with the high-score target frame is the current associated trajectory, and the trajectory pool matched with the low-score target frame is the trajectory that is not successfully matched with the high-score target frame.

[0123] Specifically, the high-score target matching is as follows:

[0124] The high-score matching process is as shown in the part 2-1 in the Figure 2 , the target frame with a detection result confidence greater than 0.7 is defined as a high-score target frame

[0125] The trajectory is a series of target frames formed by the same target, the trajectory pool is the current associated trajectory, mainly including the activated trajectory and the lost trajectory, and the position of the last target frame in the trajectory pool in the current frame is predicted based on the Kalman filter, that is, the predicted target frame The fusion loss matrix of the current target frame and the predicted target frame is calculated , and then the loss matrix Hungarian algorithm is used for association matching. When the target frame cannot be associated with any trajectory in the trajectory pool, that is, the association fails, the target is defined as a lost trajectory when the current target is associated for 30 consecutive frames, and the target is defined as a deleted trajectory when the association fails for more than 30 frames. The output result of the high-score target matching is the matched high-score target frame and the unmatched high-score target frame , the trajectory pool is divided into the matched trajectory and the unmatched trajectory .

[0126] Low-score target matching: the low-score target frame process is as shown in the part 2-2 in the Figure 2 , the target frame with a detection result output confidence less than 0.7 is defined as a low-score target frame , and the trajectory pool is the trajectory that is not successfully matched with the high-score target frame , the Kalman filter is used to predict the position of the target frame in the previous frame in the current frame, that is, to predict the target frame , the low-score target frame and the predicted target frame of the unmatched trajectory Calculate the loss matrix , based on the loss matrix, use the Hungarian algorithm for association matching, and the low-score target matching output result is the matched low-score target frame and the unmatched low-score target frame , the current trajectory pool trajectory is divided into matched trajectory and unmatched trajectory .

[0127] Step S3 also includes long-term missing target matching, and the long-term missing target process is shown in part 2-3 of Figure 2 , which is a step of finding the high-score target frame again after the long-term target loss, and the specific steps include:

[0128] Calculate the cosine loss matrix using the target appearance feature of the high-score unmatched target and the target appearance feature in the deleted trajectory, and use the Hungarian algorithm for association matching based on the cosine loss matrix.

[0129] The long-term missing target matching output result is the matched high-score target frame and the unmatched high-score target frame, and a new ID is assigned to the unmatched high-score target frame, which is considered as a new target. The deleted trajectory is divided into matched deleted trajectory and unmatched deleted trajectory, and the matched deleted trajectory is updated to active trajectory.

[0130] According to the method of the first aspect of the application, the matched trajectory is updated;

[0131] The update is specifically performed by the following formula:

[0132] wherein, represents the updated appearance feature, represents the weight parameter, the current value is 0.9, represents the target appearance feature of the previous frame, represents the current target appearance feature.

[0133] The second aspect of the application discloses an electronic device. The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor executes the computer program to realize the steps of any one of the multi-target long-term tracking method of the unmanned aerial vehicle in the first aspect of the application.

[0134] Figure 3 The structure diagram of the electronic device according to the embodiment of the application is as follows: Figure 3As shown, the electronic device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the electronic device is used for wired or wireless communication with external terminals. Wireless communication can be achieved through WIFI, operator network, near field communication (NFC) or other technologies. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen. The input device of the electronic device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad provided on the shell of the electronic device, or an external keyboard, touchpad or mouse, etc.

[0135] Those skilled in the art can understand that, Figure 3 The structure shown in the figure is only a structure diagram of part of the technical solution of the present application, and does not constitute a limitation on the electronic device to which the technical solution of the present application is applied. The specific electronic device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0136] The third aspect of the present application discloses a computer readable storage medium. The computer readable storage medium stores a computer program. When the computer program is executed by a processor, the steps in the steps of any one of the unmanned aerial vehicle multi-target long-term tracking method of the first aspect of the present application are implemented.

[0137] In summary, the scheme proposed by the present application has the following technical effects: the present application fuses the lens anti-shake strategy in the matching process, solves the problem of tracking error when the unmanned aerial vehicle shakes due to weather influence, and introduces the ReID model, adds the post-processing strategy after matching, solves the ID addition problem when the same person disappears and reappears in the camera, and further realizes the multi-target long-term stable target tracking under the unmanned aerial vehicle view.

[0138] Please note that the technical features of the above embodiments can be combined in any manner, and for the sake of brevity, not all possible combinations of the technical features in the above embodiments are described, however, as long as the combinations of the technical features do not contradict each other, they should be considered as within the scope of the present disclosure. The above embodiments only express several implementation manners of the present application, which are described in a more specific and detailed manner, but should not be understood as a limitation on the patent scope of the present application. It should be noted that for those skilled in the art, some modifications and improvements can be made without departing from the concept of the present application, and these are within the scope of the present application. Therefore, the patent protection scope of the present application should be subject to the appended claims.

Claims

1. A method for multi-target long-term tracking of unmanned aerial vehicles, characterized in that the method is based on a tracking system for positioning, tracking and monitoring of multiple targets acquired by unmanned aerial vehicles, the tracking system comprising a detection module, a cross-camera tracking module and a matching module; the method comprising: S1, the detection module identifies target detection frame images and target relative position information of all monitored targets from environmental image information, obtaining a target detection frame image sequence and a target relative position information sequence; S2, the cross-camera tracking module extracts multi-dimensional appearance features of each target detection frame image; S3, for each frame in the target detection frame image sequence, the matching module divides it into high-score target frames and low-score target frames according to the confidence after anti-shake operation, obtains a fusion loss matrix based on the multi-dimensional appearance features and the target relative position information, and then performs trajectory matching on the scored target frames by the Hungarian algorithm to complete target tracking; in step S3, the step of obtaining a fusion loss matrix based on the multi-dimensional appearance features and the target relative position information comprises: predicting the position of the last frame detection target frame in the current detection target frame based on Kalman filtering to obtain a predicted target frame; modeling the current detection target frame and the predicted target frame into 2D Gaussian distribution based on the target relative position information; calculating the distance between the Gaussian distributions corresponding to the current detection target frame and the predicted target frame; the formula for calculating the distance between the Gaussian distributions is: the position similarity between the current detection target frame and the predicted target frame is obtained by normalizing the distance between the Gaussian distributions; the formula for calculating the position similarity is: an NWD loss matrix is calculated according to the position similarity; the formula for calculating the NWD loss matrix is: the similarity between the multi-dimensional appearance features corresponding to the current detection target frame and the predicted target frame is calculated by using cos similarity; the formula for calculating the similarity between the multi-dimensional appearance features is: an appearance feature loss matrix is obtained by normalizing the similarity between the multi-dimensional appearance features; the appearance feature loss matrix is: a fusion loss matrix is obtained according to the NWD loss matrix and the appearance feature loss matrix; the fusion loss matrix is: in step S3, the step of performing anti-shake operation comprises: obtaining a homography transformation matrix based on the target detection frame image of the current frame and the target detection frame image of the last frame, and then obtaining the current detection target frame after perspective transformation according to the product of the target detection frame image of the last frame and the homography transformation matrix. the step of obtaining a homography transformation matrix based on the target detection frame image of the current frame and the target detection frame image of the last frame comprises: extracting feature points and descriptors corresponding to each feature point of the target detection frame image of the current frame and the target detection frame image of the last frame by using an ORB operator; obtaining matched feature point pairs by matching the descriptors of the feature points; obtaining correct feature point pairs after filtering the abnormal matching of the feature point pairs; ​ ​ ​ ​ ​ ​ ​ ​ ​ ， wherein, represents a distance between the Gaussian distribution of the current detection target frame a and the predicted target frame b, represents a Gaussian distribution of the current detection target frame a, represents a Gaussian distribution of the predicted target frame b, , represents a center point coordinate of the current detection target frame a, represents a width of the current detection target frame a, represents a height of the current detection target frame a, , represents a center point coordinate of the predicted target frame b, represents a width of the predicted target frame b, represents a height of the predicted target frame b; ​ ​ ， wherein, represents the similarity between the current detection target frame a and the predicted target frame b, and C represents a dataset-dependent constant. ​ ​ ， ​ ​ ， wherein, represents the appearance feature of the i-th target frame, represents the appearance feature of the j-th target frame, represents the value of the k-th dimension of the i-th target, represents the value of the k-th dimension of the j-th target, represents the dimension of the appearance feature; ​ ​ ， ​ ​ ， wherein, is a threshold value representing the distance of the position, currently taking the value 0.3, is a threshold value representing the apparent feature. 2.The method of claim 1, wherein, ​ ​ 3.The method of claim 2, wherein, ​ ​ ​ ​ Based on the random sampling consistent algorithm, 8 optimal feature points are selected, and a homographic transformation matrix is calculated through the 8 feature points. 4.The method of claim 1, wherein, The track pool matched with the high-score target frame is the current track that can be associated, and the track pool matched with the low-score target frame is the track that is not matched with the high-score target frame.

5. The method of claim 4, wherein, After step S3, the following steps are further included: A cosine loss matrix is calculated using the target appearance features that are not matched with the high-score target frame and the target appearance features in the deleted track, and Hungarian algorithm is used for association matching based on the cosine loss matrix.

6. The method of claim 5, wherein, The matched track is updated; The update is performed by the following formula: ， wherein, represents the updated appearance feature, represents the weight parameter, the current value is 0.9, represents the previous frame target appearance feature, represents the current target appearance feature.

7. The method of claim 1, wherein, Step S2 includes: Image preprocessing is performed on the target detection frame image; Restnet50 is used as the backbone network to capture the feature map [1, 2048, 16, 8] of the preprocessed target detection frame image; An aggregation network is used to reduce the dimension and normalize the feature map [1, 2048, 16, 8] to obtain a multi-dimensional appearance feature [1, 2048, 1, 1].

8. An electronic device, comprising: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the steps of the multi-target long-term tracking method of the unmanned aerial vehicle in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the multi-target long-term tracking method of the unmanned aerial vehicle in any one of claims 1 to 7.

Images