Multi-target vehicle tracking method and device under unmanned aerial vehicle perspective, equipment and medium

Abstract

A multi-target vehicle tracking method, device and equipment under the perspective of a UAV, and a medium, comprising inputting an image sequence captured by a UAV, performing target detection to obtain a target detection result in the image; extracting the appearance features of the target detection result, extracting the appearance features of each target in each target frame; using a Kalman filter to estimate and predict the state of each target, including the position, velocity and direction information of the target, to obtain the motion features of each target; combining the extracted appearance features and motion features to obtain the target feature vector of each target; matching the target feature vector in the current frame captured by the UAV with the known target feature vector in the previous frame to ensure the stability of tracking between the previous and current frames; outputting the position information of each target to the result to form a tracking trajectory. The present application can improve the stability and accuracy of the UAV in detecting and tracking targets of varying scales.

Description

TECHNICAL FIELD

[0001] The present application mainly relates to the technical field of target tracking, and in particular to a multi-target vehicle tracking method, device and equipment under the visual angle of a UAV and a medium. BACKGROUND

[0002] Vehicle tracking has a wide range of applications in civilian and military fields, such as intelligent transportation and environmental monitoring in civilian fields, and information support and medical assistance in military fields. Due to the high flexibility and safety of UAVs, vehicle tracking from the perspective of a UAV has attracted widespread research interest in recent years. When multiple vehicles appear in the field of view of a UAV, the vehicle tracking task becomes a multi-target tracking (MOT) task. In the actual target tracking process, both the UAV and the vehicle target are moving, and the vehicle travels at a high speed. When the distance between the UAV and the vehicle target changes, the target scale captured by the sensor on the UAV will also change, resulting in missed detection and identity switching of the vehicle target, which greatly increases the difficulty of multi-target tracking.

[0003] At present, most tracking algorithms adaptively update the target scale through multi-scale estimation to realize variable-scale target detection tracking. However, this method needs to select multiple scale coefficients and generate different target boxes, which has a large amount of computation and affects the real-time performance of tracking. How to quickly and accurately adapt to the scale change of the target is still a key problem in variable-scale target detection tracking. SUMMARY

[0004] In order to solve the problem of missed detection and identity switching caused by target scale change in vehicle tracking from the perspective of a UAV, the present application provides a multi-target vehicle tracking method, device and equipment under the visual angle of a UAV and a medium to improve the stability and accuracy of variable-scale target detection tracking by a UAV.

[0005] To achieve the above purpose, the technical solution adopted by the present application is as follows:

[0006] On the one hand, the present application provides a multi-target vehicle tracking method under the visual angle of a UAV, comprising:

[0007] inputting an image sequence captured by a UAV, performing vehicle target detection on the image to obtain vehicle target detection results in the image;

[0008] extracting appearance features of each vehicle target in each target box based on the vehicle target detection results;

[0009] using a Kalman filter based on the vehicle target detection results to estimate and predict the state of each vehicle target, including the position, speed and direction information of the vehicle target, to obtain the motion features of each vehicle target;

[0010] combining the appearance features and the motion features of each vehicle target to obtain a target feature vector of each vehicle target;

[0011] matching the target feature vector of the vehicle target in the current frame photographed by the unmanned aerial vehicle with the target feature vector of the known vehicle target in the previous frame, and performing optimal assignment using the Hungarian algorithm to ensure stability of keeping track between the previous frame and the current frame;

[0012] outputting the position information of each vehicle target to the result to form a tracking trajectory of each vehicle target.

[0013] In another aspect, an unmanned aerial vehicle perspective multi-target vehicle tracking device is provided, comprising:

[0014] A first module is configured to input an image sequence photographed by an unmanned aerial vehicle, perform vehicle target detection on the images, and obtain vehicle target detection results in the images.

[0015] A second module is configured to extract appearance features of each vehicle target in each target frame based on the vehicle target detection results.

[0016] A third module is configured to perform state estimation and prediction on each vehicle target based on the vehicle target detection results using a Kalman filter, including position, speed and direction information of the vehicle target, and obtain motion features of each vehicle target.

[0017] A fourth module is configured to combine the appearance features and the motion features of each vehicle target to obtain a target feature vector of each vehicle target.

[0018] A fifth module is configured to match the target feature vector of the vehicle target in the current frame photographed by the unmanned aerial vehicle with the target feature vector of the known vehicle target in the previous frame, and perform optimal assignment using the Hungarian algorithm to ensure stability of keeping track between the previous frame and the current frame.

[0019] A sixth module is configured to output the position information of each vehicle target to the result to form a tracking trajectory of each vehicle target.

[0020] In another aspect, the present application provides a computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps of the above-mentioned unmanned aerial vehicle perspective multi-target vehicle tracking method when executing the computer program.

[0021] In another aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the steps of the above-mentioned unmanned aerial vehicle perspective multi-target vehicle tracking method when executed by a processor.

[0022] In another aspect, the present application provides a computer program product stored on a computer readable storage medium and comprising computer instructions which, when executed by a processor, cause a computer device to implement the steps of the above-mentioned method for tracking multiple vehicle targets from the perspective of a UAV.

[0023] Compared with the prior art, the technical effects of the present application are:

[0024] In order to solve the problem of missing detection and identity switching caused by target scale change in vehicle tracking from the perspective of a UAV, the present application proposes a method for tracking multiple vehicle targets from the perspective of a UAV, which processes scale change vehicle targets from the perspective of a UAV through multi-scale feature fusion. Specifically, in the vehicle template detection stage, unlike other detection methods that use feature splicing operations for feature fusion, the present application proposes an improved YOLOv8 target detection network (FB-YOLOv8 target detection network) for target detection of video frames, outputting vehicle target detection results. The task of this stage is to identify vehicle targets in the video and label their positions. This improved target detection method can improve the detection accuracy of vehicle targets. The task of this stage is to identify vehicle targets in the video and label their positions. Specifically, the FB-YOLOv8 target detection network introduces a feature alignment and aggregation module (FAAM) in the neck of the network to solve the problem of feature misalignment, and proposes a bidirectional path aggregation network (BPAN) to enhance the multi-scale feature fusion capability. First, the input image is preprocessed, including normalization and size adjustment, to ensure that the image is suitable for network input. Then, the image passes through the backbone network of the YOLOv8 network to generate multi-scale feature maps. The generated adjacent feature maps of different scales are input into the feature alignment and aggregation module for multi-scale feature alignment operation. The feature maps processed by the multi-scale feature alignment operation are input into the bidirectional path aggregation network, and the deep and shallow features are further fused to obtain the final feature. Then, the network outputs the vehicle target detection result.

[0025] In the vehicle re-identification process, in order to extract more fine-grained appearance features, the present application introduces a feature pyramid network (FPN) based on the OSNet re-identification network to capture the pixel dependency between multiple feature maps. The feature pyramid network aggregates features from different levels to generate features that contain both low-level information and high-level semantics. In the feature fusion process, a convolution block attention module (CBAM) is introduced to help the model focus on image regions containing important information, thereby extracting more discriminative appearance features and improving the accuracy of matching and association.

[0026] Meanwhile, in the present application, the Kalman filter is used for state estimation and prediction of each vehicle target based on the vehicle target detection result, the motion feature of the vehicle target is extracted, and the motion feature of the vehicle target describes the dynamic trajectory of the vehicle target. The appearance feature and the motion feature of the vehicle target are combined to obtain the target feature vector of each vehicle target. The target feature vector of the vehicle target in the current frame photographed by the unmanned aerial vehicle is matched with the target feature vector of the known vehicle target in the previous frame, and the Hungarian algorithm is used for optimal allocation to ensure the stability of the tracking between the previous frame and the current frame. Thus, the same target in different time frames is associated according to the appearance feature and the motion feature of the vehicle target, efficient tracking is realized through the matching algorithm, and the complete trajectory of all targets in the video is obtained. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor based on the drawings shown.

[0028] Figure 1 FIG. 1 is a framework diagram of a multi-target vehicle tracking scheme under the view of an unmanned aerial vehicle according to an embodiment;

[0029] Figure 2 FIG. 2 is a flowchart of a multi-target vehicle tracking method under the view of an unmanned aerial vehicle according to an embodiment;

[0030] Figure 3 FIG. 3 is a detection network structure diagram provided by an embodiment;

[0031] Figure 4 FIG. 4 is a schematic diagram of a bidirectional path aggregation network according to an embodiment;

[0032] Figure 5 FIG. 5 is a re-identification network structure diagram provided by an embodiment;

[0033] Figure 6 FIG. 6 is an experimental comparison effect diagram of the method of the present application and the benchmark method BoT-SORT in a daytime scene;

[0034] Figure 7 FIG. 7 is an experimental comparison effect diagram of the method of the present application and the benchmark method BoT-SORT in a nighttime scene. DETAILED DESCRIPTION

[0035] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be clearly and completely described below, obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0036] In order to improve the tracking accuracy and stability of the scale change target, the present application first proposes a FB-YOLOv8 target detection network for a multi-scale target scene to realize more accurate vehicle target detection and positioning. Then, the vehicle target detection result is input into a multi-scale feature fusion appearance feature extraction network (MSFF-OSNet) to extract more discriminative appearance features. At the same time, the extracted appearance features are combined with motion features to promote the matching and association of trajectories, and the reference Figure 1 The framework diagram of the multi-target vehicle tracking scheme under the perspective of the unmanned aerial vehicle of an embodiment.

[0037] In the detection stage, the present application proposes a FB-YOLOv8 target detection network, introduces a feature alignment and aggregation module (FAAM) in the neck of the network to solve the feature misplacement problem, and proposes a bidirectional path aggregation network (BPAN) to enhance the multi-scale feature fusion capability. In the vehicle re-identification process, in order to extract more fine-grained appearance features, the present application introduces a feature pyramid network (FPN) based on the OSNet re-identification network to capture the pixel dependency relationship between multiple feature maps. The feature pyramid network (FPN) aggregates features from different levels to generate features containing low-level information and high-level semantics at the same time, and introduces a convolution block attention module (CBAM) in the feature fusion process to help the model focus on the image area containing important information, so as to extract more discriminative appearance features and improve the accuracy of matching and association.

[0038] Specifically, referring to Figure 2 An embodiment provides a multi-target vehicle tracking method under the perspective of an unmanned aerial vehicle, comprising:

[0039] Inputting an image sequence photographed by the unmanned aerial vehicle, performing vehicle target detection on the image to obtain vehicle target detection results in the image;

[0040] Performing appearance feature extraction based on the vehicle target detection results to extract appearance features of each vehicle target in each target frame;

[0041] Using a Kalman filter based on the vehicle target detection results to perform state estimation and prediction on each vehicle target, including position, speed and direction information of the vehicle target, to obtain motion features of each vehicle target;

[0042] Combine the appearance features and motion features of each extracted vehicle target to obtain a target feature vector of each vehicle target;

[0043] Match the target feature vector of the vehicle target in the current frame photographed by the unmanned aerial vehicle with the target feature vector of the known vehicle target in the previous frame, and perform optimal assignment using the Hungarian algorithm to ensure the stability of keeping track between the previous and subsequent frames;

[0044] Output the position information of each vehicle target to the result to form a tracking trajectory of each vehicle target.

[0045] Further, the present application provides an FB-YOLOv8 target detection network, which inputs an image sequence photographed by an unmanned aerial vehicle into the FB-YOLOv8 target detection network for target detection to obtain a target detection result in the image. Figure 3 is the detection network structure provided by an embodiment, which shows the FB-YOLOv8 target detection network structure provided by the present application. In order to align the features of different resolution feature maps, the present application integrates a feature alignment and aggregation module (FAAM) in the neck of the network, and proposes a bidirectional path aggregation network (BPAN) to further enhance the multi-scale fusion capability of the network. At the same time, a shallow detection head is added at the prediction stage to improve the detection capability of small targets.

[0046] The FB-YOLOv8 target detection network provided by the present application is used for vehicle target detection of images, which includes:

[0047] Before inputting into the FB-YOLOv8 target detection network, the images in the image sequence photographed by the unmanned aerial vehicle are sequentially preprocessed, and the preprocessing includes normalization processing and size adjustment;

[0048] Different resolution feature maps in the preprocessed image are extracted through a multi-layer convolutional network. As shown in Figure 3 The preprocessed image extracts different resolution feature maps through a multi-layer convolutional network, Figure 3 The feature maps P1, P2, P3, P4, and P5 are included in the feature maps P1, P2, P3, P4, and P5, the resolution of each layer of feature maps gradually decreases (i.e. the resolution of the feature maps P1, P2, P3, P4, and P5 decreases in turn), and the semantic information gradually increases.

[0049] The generated feature maps of different resolutions are subjected to multi-scale feature alignment operation to obtain fusion feature maps.

[0050] The fusion feature maps obtained after the multi-scale feature alignment operation are input into the bidirectional path aggregation network, and the deep and shallow features are further fused to obtain the final feature;

[0051] Based on the final features, multi-scale target detection is performed, and a vehicle target detection result is output, including a target box of the vehicle target and a confidence thereof.

[0052] The vehicle target detection result is further output, including: removing target boxes with high overlap and low confidence; the vehicle target detection result includes a target box of the vehicle target and a confidence thereof, and the target boxes with high overlap are filtered out by a non-maximum suppression (NMS) algorithm; a confidence threshold is set, and the target boxes below the confidence threshold will be discarded directly; after the target boxes with high overlap and low confidence are removed, the remaining target boxes are subjected to post-processing, including target box coordinate conversion and confidence sorting, and the final vehicle target detection result is output.

[0053] The application provides a feature alignment and aggregation module (FAAM) for performing multi-scale feature alignment operation on generated feature maps with different resolutions to obtain a fusion feature map, including:

[0054] For the generated feature maps with different resolutions, the feature maps with adjacent resolutions form a group, one of the feature maps in each group is a low-resolution feature map, and the other is a high-resolution feature map.

[0055] For any group of feature maps, the low-resolution feature map is up-sampled based on a bilateral interpolation, and then the up-sampled low-resolution feature map and the high-resolution feature map are connected in series to obtain a connected feature; the connected feature is processed through a depth separable convolution module to obtain a spatially aligned high-resolution feature map; the spatially aligned high-resolution feature map is added to the up-sampled low-resolution feature map to obtain a fusion feature map, so that a fusion feature map corresponding to each group of feature maps is obtained, and the fusion feature map contains detailed information and global information. The depth separable convolution module is composed of a depth convolution layer and a point-wise convolution layer, a depth convolution layer with a kernel size of 3x3 is used to capture local spatial information around each sampling position and transmit the local spatial information to the point-wise convolution layer to generate a two-dimensional offset; the two-dimensional offset and the high-resolution feature map are input into an alignment function to obtain a spatially aligned high-resolution feature map.

[0056] The application introduces the feature alignment and aggregation module (FAAM) for aligning adjacent features with different resolutions. The alignment operation makes the features with different resolutions correspond in space, thereby improving the ability of the model to process detailed and global information and the ability of the detection algorithm to accurately locate small targets.

[0057] The application provides a bidirectional path aggregation network (BPAN) to further enhance the multi-scale fusion capability of the target detection network. The input of the bidirectional path aggregation network comes from multiple scale feature maps extracted by a multi-layer convolutional network, which have different resolutions: low-layer feature maps have high resolution and contain more details; high-layer feature maps have low resolution but stronger semantic information. Low-layer feature maps focus on small targets and local details, and high-layer feature maps focus on large targets and global context. The bidirectional path aggregation network includes cross-layer connections between feature maps of different resolutions, multi-scale feature splicing of the fusion feature maps processed by the multi-scale feature alignment operation, and the combination of the attention mechanism, and finally the spliced feature maps contain multi-scale and multi-semantic information. The bidirectional path aggregation network makes the feature fusion more comprehensive and can capture target information at different levels. In the cross-layer connection between feature maps of different resolutions, the high-resolution feature map is first down-sampled to align with the low-resolution feature map. In this way, the fusion between feature maps of different layers will not be affected by the mismatch of resolutions. Through the combination of cross-layer connection, multi-scale feature splicing and attention mechanism, the bidirectional path aggregation network effectively integrates the detailed information of low-layer feature maps and the semantic information of high-layer feature maps, is more accurate for small target detection and more precise for large target positioning, and ensures that the FB-YOLOv8 model can realize robust target detection in various complex scenes, especially for multi-scale target detection. The bidirectional path aggregation network finally outputs the fused feature map to the detection head for vehicle target classification and target box prediction.

[0058] Reference Figure 4Fig. 1 is a schematic diagram of a bidirectional path aggregation network according to an embodiment. The input of the bidirectional path aggregation network comes from the feature maps P2, P3, P4 and P5 extracted by the multi-layer convolutional network. The feature maps P2, P3, P4 and P5 are input into the feature alignment aggregation module (FAAM) to obtain a fused feature map. Concat represents feature concatenation, which realizes the splicing operation of the feature maps from different paths and integrates them into an enhanced feature representation. The spliced features contain multi-scale information and can better cope with vehicle target detection tasks of different sizes. The purple arrow in the figure represents cross-layer feature fusion, which realizes information supplementation by transferring features from low-resolution layers to high-resolution layers. The high-resolution layer features are transferred to the low-resolution layer through upsampling to provide global context information for the low-layer features. The blue arrow represents the flow direction of down-sampled features, which is used to adjust the feature scale to match the features of different layers. The orange arrow represents the flow direction of up-sampled features, which is used to transfer high-resolution features to low-resolution layers to realize feature sharing. After each layer of features interacts with the cross-layer features, they are spliced with the current layer of features. The spliced feature maps combine multi-layer information and contain both details and global semantics. The features after the FAAM and Concat operations are used as the output of the BPAN for subsequent vehicle target detection tasks.

[0059] The FB-YOLOv8 target detection network improves the detection accuracy and robustness by adding the feature alignment aggregation module (FAAM) and the bidirectional path aggregation network (BPAN) module based on YOLOv8. This architecture is particularly friendly to small target and multi-scale target detection and is suitable for precise detection of multiple vehicle targets in a traffic scene.

[0060] Meanwhile, the Kalman filter is used to estimate and predict the state of each vehicle target based on the vehicle target detection results, including the position, speed and direction information of the vehicle target, to obtain the motion characteristics of each vehicle target. Kalman filter (KF) is a state optimal estimation algorithm that can be used for state estimation in multi-target tracking. This algorithm is based on the linear system state equation and optimally estimates the state of the vehicle target through the prediction and update stages. In the prediction process, the Kalman filter predicts the position information of the current time frame through the system state equation and the state transition matrix. In the update process, the Kalman filter updates and corrects the position information of the previous prediction process through the observation equation and the matrix.

[0061] In order to enhance the distinguishing features of the vehicle target, the vehicle target detection result is input into the re-identification network MSFF-OSNet for appearance feature extraction, and the features are then used for appearance feature matching and association, effectively reducing the ID switching and other problems caused by the change of vehicle scale. The re-identification network MSFF-OSNet adopts a feature pyramid structure network in the backbone, and extracts feature information representing different semantics and scales from each different stage of the backbone. In addition, the CBAM attention mechanism is introduced in the multi-level feature fusion process to focus on the image regions containing important information, so as to extract more discriminative appearance features. These extracted appearance features are then used for appearance feature matching, effectively reducing the identity switching and other problems caused by the change of vehicle scale.

[0062] Figure 5Figure 1 is a structural diagram of a re-identification network according to an embodiment, which shows the specific structure of the re-identification network MSFF-OSNet, and adopts the OSNet network as the backbone network. The feature pyramid network captures different scale information of the target frame image through the convolution blocks Conv1, Conv2, Conv3, Conv4 to Conv5, and extracts different scale feature maps C1, C2, C3, C4 and C5 of the target frame. Under the framework of the feature pyramid network, the features of different levels are integrated to make full use of multi-scale information. The feature maps C2 and C3 are adjusted in the channel number through upsampling and 1x1 convolution layer to match the feature maps C1 and C2 respectively. The feature maps C4 and C5 are adjusted in the channel number through 1x1 convolution layer to match the feature maps C3 and C4 respectively. Specifically, the feature map C2 is adjusted in the channel number through upsampling and 1x1 convolution layer to match the feature map C1 to obtain the feature map D2. The enhanced feature of the feature map C1 obtained through the CBAM attention module is then added to the feature map D2 in an element manner to obtain the output feature map F1, which is transmitted to the next layer. The feature map C3 is adjusted in the channel number through upsampling and 1x1 convolution layer to match the feature map C2 to obtain the feature map D3, and the enhanced feature of the output feature map F1 obtained through the CBAM attention module is then added to the feature map D3 in an element manner to obtain the output feature map F2, which is transmitted to the next layer. The feature map C4 is adjusted in the channel number through 1x1 convolution layer to match the feature map C3 to obtain the feature map D4, and the enhanced feature of the output feature map F2 obtained through the CBAM attention module is then added to the feature map D4 in an element manner to obtain the output feature map F3, which is transmitted to the next layer. The feature map C5 is adjusted in the channel number through 1x1 convolution layer to match the feature map C4 to obtain the feature map D5, and the enhanced feature of the output feature map F3 obtained through the CBAM attention module is then added to the feature map D5 in an element manner to obtain the final feature map F4, which is the appearance feature of the vehicle target. In this way, the features are propagated to the low layer through upsampling step by step, and are fused with the low layer features step by step. The finally fused feature is used as the output feature, which can take into account the global semantic information and the detail information.

[0063] The CBAM attention module is composed of two independent sub-modules: the channel attention module CAM and the spatial attention module SAM, which reconstructs the feature map to highlight the key information. The channel attention module CAM focuses on the relationship between the channels of the feature map and evaluates the importance of each pixel. Take the enhanced feature of the feature map C1 obtained through the CBAM attention module as an example: the input of the channel attention module CAM outputs the channel attention which is expressed as:

[0064]

[0065] Spatial attention output by the spatial attention module SAM is denoted as:

[0066]

[0067] In the above two equations, C1 represents the input feature map C1, Avg represents the average value, Max represents the global max pooling, MLP represents the multi-layer perceptron, and σ {} represents the Sigmoid activation function, denotes a convolution kernel with a side length of 5.

[0068] Therefore, the overall expression of the CBAM attention module is:

[0069]

[0070] that is, the enhanced features of the feature map C1 obtained through the CBAM attention module. Based on the same method, the enhanced features of the output feature map F1, the output feature map F2, and the output feature map F3 obtained through the CBAM attention module can be obtained.

[0071] The feature map C2 is up-sampled and adjusted in the number of channels using a 1x1 convolutional layer to match the feature map C1, obtaining a feature map D2, and the expression of the feature map D2 is as follows:

[0072]

[0073] The re-identification network MSFF-OSNet introduces a feature pyramid network (FPN) in the backbone to capture the pixel dependency between multiple feature maps. The feature pyramid network aggregates features from different levels to generate features that contain both low-level information and high-level semantics. In the feature fusion process, a convolutional block attention module (CBAM) is introduced to help the model focus on image regions containing important information, thereby extracting more discriminative appearance features and improving the accuracy of matching and association.

[0074] In the process of multi-target tracking, it is necessary to match the detected targets and the predicted target trajectories, but due to changes in object position and other information, there are multiple predicted trajectories. Therefore, an algorithm needs to be selected to complete the tracking task. The Hungarian algorithm is an algorithm based on the sufficiency proof in Hall's theorem, which is commonly used in graph matching problems. The core idea of this theorem is to find an augmenting path. By applying this theorem, the Hungarian algorithm can use the augmenting path to find an algorithm that matches the maximum matching of the bipartite graph. The input of the matching algorithm is the cost matrix C(m x n). The steps of the matching algorithm are as follows:

[0075] Step1: find the minimum value of each column in the cost matrix C col min , make A = C - col min ;

[0076] Step2: find the minimum value of each row in the cost matrix C raw min , make B = C - raw min ;

[0077] Step3: select the horizontal line of vertical Y axis or vertical X axis to change all zero elements in B with the least number;

[0078] Step4: if the number of horizontal lines is m, return the optimal matching, otherwise enter step5;

[0079] Step5: find the minimum value in B which is not changed by any horizontal line, add the value to any column which is not covered by horizontal line, subtract the value from any row which is not covered by horizontal line, return to step3.

[0080] The present application proposes a multi-target tracking method for solving the multi-scale feature fusion of vehicles under the variable scale of the unmanned aerial vehicle perspective. First, the present application proposes an FB-YOLOv8 framework to solve the missed detection problem. The framework combines a feature alignment aggregation module (FAAM) and a bidirectional path aggregation network (BPAN) to enhance multi-scale feature fusion. Secondly, the present application proposes a multi-scale feature fusion network (MSFF-OSNet) to extract appearance features, thereby solving the identity switching problem. The framework integrates a feature pyramid network (FPN) and a convolution block attention module (CBAM) into the OSNet to capture multi-level pixel dependencies and combine low-level and high-level features. By effectively integrating the FB-YOLOv8 and MSFF-OSNet modules into the tracking process, the accuracy and stability of tracking are improved. The experimental results are shown in Figure 6 and Figure 7 , wherein Figure 6 is the experimental comparison effect diagram of the method of the present application and the reference method BoT-SORT in the daytime scene, Figure 7 is the experimental comparison effect diagram of the method of the present application and the reference method BoT-SORT in the night scene. In the two scenes, when the vehicle is at the intersection and the scale changes, the original reference method BoT-SORT method has missed detection (FN) and ID switching (IDSW) when tracking (marked with a yellow square in Figure 6 and Figure 7 ), while the method of the present application can avoid these problems, proving that the method can realize accurate and stable tracking of vehicle targets in the scale change scene.

[0081] In another aspect, the present application provides a multi-target vehicle tracking device under the perspective of a UAV, comprising:

[0082] A first module for inputting an image sequence captured by a UAV, performing vehicle target detection on the images, and obtaining vehicle target detection results in the images;

[0083] A second module for performing appearance feature extraction based on the vehicle target detection results, and extracting appearance features of each vehicle target in each target frame;

[0084] A third module for performing state estimation and prediction on each vehicle target based on the vehicle target detection results using a Kalman filter, including position, speed and direction information of the vehicle target, and obtaining motion features of each vehicle target;

[0085] A fourth module for combining the extracted appearance features and motion features of each vehicle target to obtain a target feature vector of each vehicle target;

[0086] A fifth module for matching the target feature vector of the vehicle target in the current frame captured by the UAV with the target feature vector of the known vehicle target in the previous frame, and performing optimal allocation using the Hungarian algorithm to ensure the stability of the tracking between the previous and subsequent frames;

[0087] A sixth module for outputting the position information of each vehicle target to the results to form a tracking trajectory of each vehicle target.

[0088] In another aspect, the present application provides a computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps of the multi-target vehicle tracking method under the perspective of a UAV as provided in any of the above embodiments when executing the computer program. The computer device can be a server. The computer device comprises a processor, a memory, a network interface and a database connected by a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store sample data. The network interface of the computer device is used to communicate with external terminals through network connection.

[0089] In another aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the steps of the multi-target vehicle tracking method under the perspective of a UAV as provided in any of the above embodiments.

[0090] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, database or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0091] The details of the present application are as follows.

[0092] The technical features of the above embodiments can be combined in any way. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, but as long as the combinations of the technical features do not exist, they should be considered as the scope of the present application.

[0093] The above embodiments only express several embodiments of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

[0094] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for tracking multiple targets of vehicles from a UAV perspective, characterized in that, The method comprises the following steps: inputting an image sequence captured by a UAV, performing vehicle target detection on the images to obtain vehicle target detection results in the images, wherein performing vehicle target detection on the images comprises the following steps: performing preprocessing on the images in the image sequence captured by the UAV in sequence, the preprocessing comprising normalization processing and size adjustment; extracting feature maps of different resolutions in the preprocessed images; performing multi-scale feature alignment on the generated feature maps of different resolutions to obtain fusion feature maps, comprising the following steps: for the generated feature maps of different resolutions, feature maps of adjacent resolutions are a group, one of the feature maps in each group is a low-resolution feature map, and the other is a high-resolution feature map; for any group of feature maps, upsample the low-resolution feature map based on bilateral interpolation, then concatenate the upsampled low-resolution feature map and the high-resolution feature map to obtain a connected feature; process the connected feature through a depth separable convolution module to obtain a spatially aligned high-resolution feature map; add the spatially aligned high-resolution feature map to the upsampled low-resolution feature map to obtain a fusion feature map, thus obtaining a fusion feature map corresponding to each group of feature maps; inputting the fusion feature maps obtained after the multi-scale feature alignment processing into a bidirectional path aggregation network to further fuse deep and shallow features to obtain final features; performing multi-scale target detection based on the final features and outputting vehicle target detection results; extracting appearance features of each vehicle target in each target box based on the vehicle target detection results; performing state estimation and prediction on each vehicle target based on the vehicle target detection results using a Kalman filter, including position, speed and direction information of the vehicle target, to obtain motion features of each vehicle target; combine the extracted appearance features and motion features of each vehicle target to obtain a target feature vector of each vehicle target; match the target feature vector of each vehicle target in the current frame captured by the UAV with the target feature vector of the known vehicle target in the current frame, and use the Hungarian algorithm for optimal allocation to ensure the stability of tracking between frames; output the position information of each vehicle target to the result to form a tracking trajectory of each vehicle target. 2.The method of claim 1, wherein, Output the vehicle target detection results, including: removing target boxes with high overlap and low confidence: the vehicle target detection results include target boxes of vehicle targets and their confidence, and the non-maximum suppression algorithm is used to filter out target boxes with high overlap; set a confidence threshold, and target boxes below the confidence threshold will be discarded directly; after removing target boxes with high overlap and low confidence, the remaining target boxes are subjected to post-processing, including target box coordinate conversion and confidence sorting, and the final vehicle target detection results are output. 3.The method of claim 1 or 2, wherein, The deep separable convolution module is composed of a deep convolution layer and a point-wise convolution layer, a deep convolution layer with a kernel size of 3*3 is used to capture local spatial information around each sampling position and pass it to the point-wise convolution layer to generate a two-dimensional offset; the two-dimensional offset and the high-resolution feature map are input into an alignment function to obtain a spatially aligned high-resolution feature map. 4.The method of claim 3, wherein, The vehicle target detection result is input into the re-identification network MSFF-OSNet for appearance feature extraction, the re-identification network MSFF-OSNet adopts a feature pyramid structure network in the backbone, extracts different levels of feature information representing different semantics and scales from different stages of the backbone, and introduces a CBAM attention mechanism in the multi-level feature fusion process to focus on image regions containing important information, thereby extracting more discriminative appearance features. 5.The method of claim 4, wherein, The appearance feature extraction includes: The feature pyramid structure network sequentially passes through the convolution blocks Conv1, Conv2, Conv3, Conv4 to Conv5 to capture different scale information of the target frame image, and extracts feature maps C1, C2, C3, C4 and C5 of different scales of the target frame; The feature map C2 is adjusted in the channel number by upsampling and using a 1*1 convolution layer to match the feature map C1, to obtain a feature map D2, and the enhanced feature of the feature map C1 obtained through the CBAM attention module is then added to the feature map D2 in an element manner to obtain an output feature map F1 and pass it to the next layer; The feature map C3 is adjusted in the channel number by upsampling and using a 1*1 convolution layer to match the feature map C2, to obtain a feature map D3, and the enhanced feature of the output feature map F1 obtained through the CBAM attention module is then added to the feature map D3 in an element manner to obtain an output feature map F2 and pass it to the next layer; The feature map C4 is adjusted in the channel number by a 1*1 convolution layer to match the feature map C3, to obtain a feature map D4, and the enhanced feature of the output feature map F2 obtained through the CBAM attention module is then added to the feature map D4 in an element manner to obtain an output feature map F3 and pass it to the next layer; The feature map C5 is adjusted in the channel number by a 1*1 convolution layer to match the feature map C4, to obtain a feature map D5, and the enhanced feature of the output feature map F3 obtained through the CBAM attention module is then added to the feature map D5 in an element manner to obtain a final feature map F4, which is the appearance feature of the vehicle target.

6. A multi-target vehicle tracking device under the perspective of a drone, characterized by, It includes: The first module is used for inputting an image sequence photographed by a UAV, performing vehicle target detection on the image to obtain a vehicle target detection result in the image, wherein the vehicle target detection on the image includes: The images in the image sequence photographed by the UAV are sequentially preprocessed, and the preprocessing includes normalization processing and size adjustment; Different resolution feature maps in the preprocessed images are extracted; The generated feature maps of different resolutions are subjected to a multi-scale feature alignment operation to obtain a fused feature map, including that for the generated feature maps of different resolutions, the feature maps of adjacent resolutions are a group, one of the feature maps in each group is a low-resolution feature map, and the other is a high-resolution feature map; for any group of feature maps, the low-resolution feature map is up-sampled based on a bilateral interpolation, and then the up-sampled low-resolution feature map and the high-resolution feature map are concatenated to obtain a connected feature; the connected feature is processed through a depth separable convolution module to obtain a spatially aligned high-resolution feature map; the spatially aligned high-resolution feature map is added to the up-sampled low-resolution feature map to obtain a fused feature map, and thus a fused feature map corresponding to each group of feature maps is obtained; The fused feature map obtained after the multi-scale feature alignment operation is input into a bidirectional path aggregation network, and deep and shallow features are further fused to obtain a final feature; Based on the final feature, multi-scale target detection is performed, and a vehicle target detection result is output; The second module is configured to extract appearance features of each vehicle target in each target frame based on the vehicle target detection result; The third module is configured to use a Kalman filter to estimate and predict the state of each vehicle target based on the vehicle target detection result, including position, speed and direction information of the vehicle target, to obtain motion features of each vehicle target; The fourth module is configured to combine the appearance features and motion features of each vehicle target to obtain a target feature vector of each vehicle target; The fifth module is configured to match the target feature vector of the vehicle target in the current frame captured by the unmanned aerial vehicle with the target feature vector of the known vehicle target in the previous frame, and use the Hungarian algorithm for optimal allocation to ensure the stability of tracking between the previous and current frames. The sixth module is configured to output the position information of each vehicle target to the result to form a tracking trajectory of each vehicle target. 7.A computer device, comprising a memory and a processor, the memory storing a computer program, and the computer device is characterized in that, The processor executes the computer program to implement the steps of the multi-target vehicle tracking method under the view angle of the unmanned aerial vehicle according to claim 1.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the multi-target vehicle tracking method under the view angle of the unmanned aerial vehicle according to claim 1.

Images