Multi-target vehicle tracking method, device and equipment under view angle of unmanned aerial vehicle and medium

Abstract

A multi-target vehicle tracking method, device, equipment and medium under a view angle of an unmanned aerial vehicle comprises the steps of inputting an image sequence shot by the unmanned aerial vehicle, and performing target detection to obtain a target detection result in an image; performing appearance feature extraction on the target detection result, and extracting the appearance feature of each target in each target frame; performing state estimation and prediction on each target by using a Kalman filter, including position, speed and direction information of the target, and obtaining motion characteristics of each target; combining the extracted appearance features and motion features to obtain a target feature vector of each target; matching the target feature vector in the current frame shot by the unmanned aerial vehicle with the known target feature vector in the previous frame to ensure that the tracking stability is kept between the previous frame and the next frame; and outputting the position information of each target to a result to form a tracking trajectory. According to the invention, the stability and accuracy of detecting and tracking the variable-scale target by the unmanned aerial vehicle can be improved.

Description

Technical Field

[0001] The present invention mainly relates to the field of target tracking technology, and in particular to a method, device, equipment and medium for tracking multiple targets from the perspective of an unmanned aerial vehicle. Background Art

[0002] Vehicle tracking has a wide range of applications in both civil and military fields, such as intelligent transportation and environmental monitoring in the civil field, and information support and medical assistance in the military field. Due to the high flexibility and safety of drones, vehicle tracking from the perspective of drones has attracted widespread research interest in recent years. When multiple vehicles appear in the field of view of the drone, the vehicle tracking task is transformed into a multi-target tracking (MOT) task. In the actual target tracking process, both the drone and the vehicle target are moving, and the vehicle is moving at a high speed. When the distance between the drone and the vehicle target changes, the scale of the target captured by the sensor on the drone will also change, resulting in missed detection and identity switching of the vehicle target, which greatly increases the difficulty of multi-target tracking.

[0003] At present, most tracking algorithms adaptively update the target scale through multi-scale estimation to achieve variable-scale target detection and tracking. However, this method requires selecting multiple scale coefficients and generating different target frames, which requires a large amount of computation and affects the real-time tracking performance. How to quickly and accurately adapt to the scale changes of the target is still a key issue in variable-scale target detection and tracking. Summary of the invention

[0004] In order to solve the problems of missed detection and identity switching caused by target scale changes in vehicle tracking from a drone perspective, the present invention proposes a multi-target vehicle tracking method, device, equipment and medium from a drone perspective to improve the stability and accuracy of drone detection and tracking of variable-scale targets.

[0005] To achieve the above purpose, the technical solution adopted by the present invention is as follows: On the one hand, the present invention provides a multi-target vehicle tracking method from the perspective of an unmanned aerial vehicle, comprising: Input the image sequence taken by the drone, perform vehicle target detection on the image, and obtain the vehicle target detection result in the image; Appearance features are extracted based on the vehicle target detection results to extract the appearance features of each vehicle target in each target frame; Based on the vehicle target detection results, the Kalman filter is used to estimate and predict the state of each vehicle target, including the position, speed and direction information of the vehicle target, and the motion characteristics of each vehicle target are obtained; Combining the extracted appearance features and motion features of each vehicle target to obtain a target feature vector of each vehicle target; Match the target feature vector of the vehicle target in the current frame taken by the drone with the target feature vector of the known vehicle target in the previous frame, and use the Hungarian algorithm to optimize the allocation to ensure the stability of tracking between the previous and next frames; The position information of each vehicle target is output to the result to form the tracking trajectory of each vehicle target.

[0006] On the other hand, a multi-target vehicle tracking device from the perspective of a drone is provided, comprising: The first module is used to input the image sequence taken by the drone, perform vehicle target detection on the image, and obtain the vehicle target detection result in the image; The second module is used to extract appearance features based on the vehicle target detection results, and extract the appearance features of each vehicle target in each target frame; The third module is used to use the Kalman filter to estimate and predict the state of each vehicle target based on the vehicle target detection result, including the position, speed and direction information of the vehicle target, and obtain the motion characteristics of each vehicle target; The fourth module is used to combine the extracted appearance features and motion features of each vehicle target to obtain a target feature vector of each vehicle target; The fifth module is used to match the target feature vector of the vehicle target in the current frame taken by the drone with the target feature vector of the known vehicle target in the previous frame, and use the Hungarian algorithm for optimal allocation to ensure the stability of tracking between the previous and next frames; The sixth module is used to output the position information of each vehicle target into the result to form a tracking track of each vehicle target.

[0007] On the other hand, the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the multi-target vehicle tracking method from the perspective of a drone are implemented.

[0008] On the other hand, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned multi-target vehicle tracking method from the perspective of a drone.

[0009] On the other hand, the present invention provides a computer program product, which is stored on a computer-readable storage medium and includes computer instructions, which, when executed by a processor, enable a computer device to implement the steps of the above-mentioned multi-target vehicle tracking method from the perspective of a drone.

[0010] Compared with the prior art, the technical effects of the present invention are: In order to solve the problem of missed detection and identity switching caused by the change of target scale in vehicle tracking under the perspective of drone, the present invention proposes a multi-target vehicle tracking method under the perspective of drone, and processes the scale-changing vehicle targets under the perspective of drone by multi-scale feature fusion. Specifically, in the vehicle template detection stage, unlike other detection methods that use feature splicing operations for feature fusion, the present invention proposes an improved YOLOv8 target detection network (FB-YOLOv8 target detection network), performs target detection on video frames, and outputs vehicle target detection results. The task of this stage is to identify vehicle targets in the video and mark their positions. The improved target detection method can improve the detection accuracy of vehicle targets. The task of this stage is to identify vehicle targets in the video and mark their positions. Specifically, the FB-YOLOv8 target detection network introduces a feature alignment aggregation module (FAAM) at the neck of the network to solve the problem of feature misalignment, and proposes a bidirectional path aggregation network (BPAN) to enhance the multi-scale feature fusion capability. First, the input image is preprocessed, including normalization and resizing, to ensure that the image is suitable for network input. Subsequently, the image is passed through the backbone network of the YOLOv8 network to generate a multi-scale feature map. The generated adjacent feature maps of different scales are input into the feature alignment aggregation module to perform a multi-scale feature alignment operation. The feature map processed by the multi-scale feature alignment operation is input into the bidirectional path aggregation network to further fuse the deep and shallow features to obtain the final feature. After that, the network outputs the vehicle target detection result.

[0011] In the vehicle re-identification process, in order to extract more fine-grained appearance features, the present invention introduces a feature pyramid network (FPN) based on the OSNet re-identification network to capture the pixel dependencies between multiple feature maps. The feature pyramid network aggregates features from different levels to generate features that contain both low-level information and high-level semantics. In the feature fusion process, a convolutional block attention module (CBAM) is introduced to help the model focus on image areas containing important information, thereby extracting more discriminative appearance features and improving the accuracy of matching associations.

[0012] At the same time, in the present invention, the vehicle target detection result uses a Kalman filter to perform state estimation and prediction on each vehicle target, and extracts the motion characteristics of the vehicle target. The motion characteristics of the vehicle target describe the dynamic trajectory of the vehicle target. The appearance characteristics and motion characteristics of the vehicle target are combined to obtain a target feature vector for each vehicle target. The target feature vector of the vehicle target in the current frame taken by the drone is matched with the target feature vector of the known vehicle target in the previous frame, and the Hungarian algorithm is used for optimal allocation to ensure that the tracking stability is maintained between the previous and next frames. In this way, the present invention associates the same target in different time frames according to the appearance characteristics and motion characteristics of the vehicle target, realizes efficient tracking through a matching algorithm, and obtains the complete trajectory of all targets in the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0014] Figure 1 It is a framework diagram of a multi-target vehicle tracking solution from the perspective of a drone in one embodiment; Figure 2 is a flow chart of a method for tracking multiple target vehicles from the perspective of a drone according to an embodiment; Figure 3 is a diagram of a detection network structure provided by an embodiment; Figure 4 is a schematic diagram of a bidirectional path aggregation network in one embodiment; Figure 5 is a diagram of a re-identification network structure provided by an embodiment; Figure 6 The experimental comparison effect diagram of the method of the present invention and the benchmark method BoT-SORT in the daytime scene; Figure 7 The experimental comparison effect diagram of the method of the present invention and the benchmark method BoT-SORT in night scenes is shown. DETAILED DESCRIPTION

[0015] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0016] In order to improve the tracking accuracy and stability of scale-varying targets, the present invention first proposes an FB-YOLOv8 target detection network for multi-scale target scenes to achieve more accurate vehicle target detection and positioning. Subsequently, the vehicle target detection results are input into a multi-scale feature fusion appearance feature extraction network (MSFF-OSNet) to extract more discriminative appearance features. At the same time, the extracted appearance features are combined with motion features to facilitate the matching and association of trajectories. Figure 1 A framework diagram of a multi-target vehicle tracking solution from the perspective of a drone according to an embodiment.

[0017] In the detection stage, the present invention proposes an FB-YOLOv8 target detection network, introduces a feature alignment aggregation module (FAAM) at the neck of the network to solve the problem of feature misalignment, and proposes a bidirectional path aggregation network (BPAN) to enhance the multi-scale feature fusion capability. In the vehicle re-identification process, in order to extract more fine-grained appearance features, the present invention introduces a feature pyramid network (FPN) based on the OSNet re-identification network to capture the pixel dependencies between multiple feature maps. The feature pyramid network (FPN) aggregates features from different levels to generate features that contain both low-level information and high-level semantics. In the feature fusion process, a convolutional block attention module (CBAM) is introduced to help the model focus on image areas containing important information, thereby extracting more discriminative appearance features and improving the accuracy of matching associations.

[0018] Specifically, refer to Figure 2 An embodiment provides a method for tracking multiple target vehicles from a drone perspective, including: Input the image sequence taken by the drone, perform vehicle target detection on the image, and obtain the vehicle target detection result in the image; Appearance features are extracted based on the vehicle target detection results to extract the appearance features of each vehicle target in each target frame; Based on the vehicle target detection results, the Kalman filter is used to estimate and predict the state of each vehicle target, including the position, speed and direction information of the vehicle target, and the motion characteristics of each vehicle target are obtained; Combining the extracted appearance features and motion features of each vehicle target to obtain a target feature vector of each vehicle target; Match the target feature vector of the vehicle target in the current frame taken by the drone with the target feature vector of the known vehicle target in the previous frame, and use the Hungarian algorithm to optimize the allocation to ensure the stability of tracking between the previous and next frames; The position information of each vehicle target is output to the result to form the tracking trajectory of each vehicle target.

[0019] Furthermore, the present invention proposes an FB-YOLOv8 target detection network, and inputs the image sequence taken by the drone into the FB-YOLOv8 target detection network for target detection to obtain the target detection result in the image. Figure 3 This is a detection network structure diagram provided by an embodiment, showing the FB-YOLOv8 target detection network structure proposed by the present invention. In order to align the features of feature maps with different resolutions, the present invention integrates a feature alignment aggregation module (FAAM) at the neck of the network, and proposes a bidirectional path aggregation network (BPAN) to further enhance the multi-scale fusion capability of the network. At the same time, a shallow detection head is added in the prediction stage to improve the detection capability of small targets.

[0020] The FB-YOLOv8 target detection network proposed in the present invention is used to detect vehicle targets in images, including: Before inputting into the FB-YOLOv8 target detection network, the images in the image sequence taken by the drone are preprocessed in sequence, including normalization and size adjustment; The feature maps of different resolutions in the preprocessed image are extracted through a multi-layer convolutional network. Figure 3 As shown in , after preprocessing, the image is passed through a multi-layer convolutional network to extract feature maps of different resolutions. Figure 3 It includes feature map P1, feature map P2, feature map P3, feature map P4, and feature map P5. The resolution of each layer of feature maps gradually decreases (that is, the resolution of feature map P1, feature map P2, feature map P3, feature map P4, and feature map P5 decreases successively), and the semantic information is gradually enhanced.

[0021] The generated feature maps of different resolutions are subjected to multi-scale feature alignment to obtain a fused feature map.

[0022] The fused feature map obtained after the multi-scale feature alignment operation is input into the bidirectional path aggregation network to further fuse the deep and shallow features to obtain the final feature; Multi-scale target detection is performed based on the final features, and the vehicle target detection results are output, including the target box of the vehicle target and its confidence.

[0023] The vehicle target detection results are further output, including: removing target frames with high overlap and low confidence: the vehicle target detection results include the target frame of the vehicle target and its confidence, and the target frame with high overlap is filtered out by the non-maximum suppression (NMS) algorithm; the confidence threshold is set, and the target frame below the confidence threshold will be directly discarded; after removing the target frame with high overlap and low confidence, the remaining target frame is post-processed, including target frame coordinate conversion and confidence sorting, and the final vehicle target detection result is output.

[0024] The present invention proposes a feature alignment aggregation module (FAAM) for performing a multi-scale feature alignment operation on the generated feature maps of different resolutions to obtain a fused feature map, including: For the generated feature maps of different resolutions, feature maps with adjacent resolutions are grouped together, and in each group of feature maps, one is a low-resolution feature map and the other is a high-resolution feature map; For any set of feature maps, the low-resolution feature map is upsampled based on bilateral interpolation, and then the upsampled low-resolution feature map and the high-resolution feature map are connected in series to obtain the connected features; the connected features are processed by a depthwise separable convolution module to obtain a spatially aligned high-resolution feature map; the spatially aligned high-resolution feature map is added to the upsampled low-resolution feature map to obtain a fused feature map, so as to obtain a fused feature map corresponding to each set of feature maps, and the fused feature map contains detailed information and global information. The depthwise separable convolution module is composed of a depthwise convolution layer and a pointwise convolution layer, and a depthwise convolution layer with a kernel size of 3×3 is used to capture the local spatial information around each sampling position, and it is passed to the pointwise convolution layer to generate a two-dimensional offset; the two-dimensional offset and the high-resolution feature map are input into the alignment function to obtain a spatially aligned high-resolution feature map.

[0025] The present invention introduces a feature alignment aggregation module (FAAM) to align adjacent features of different resolutions. The alignment operation makes the features of different resolutions correspond in space, thereby improving the model's ability to process details and global information, and the detection algorithm's ability to accurately locate small targets.

[0026] The present invention proposes a bidirectional path aggregation network (BPAN) to further enhance the multi-scale fusion capability of the target detection network. The input of the bidirectional path aggregation network comes from feature maps of multiple scales extracted by a multi-layer convolutional network. These feature maps have different resolutions: low-level feature maps have high resolution and contain more details; high-level feature maps have low resolution but stronger semantic information. Low-level feature maps focus on small targets and local details, while high-level feature maps focus on large targets and global context. The bidirectional path aggregation network includes cross-layer connections between feature maps of different resolutions, and multi-scale feature splicing of fused feature maps obtained after multi-scale feature alignment operations. At the same time, combined with the attention mechanism, the final spliced ​​feature map contains multi-scale and multi-semantic information. The bidirectional path aggregation network makes feature fusion more comprehensive and can capture target information at different levels. When cross-layer connections are made between feature maps of different resolutions, the high-resolution feature map is first downsampled so as to align with the low-resolution feature map. This ensures that the fusion between feature maps of different layers will not be affected by resolution mismatch. The bidirectional path aggregation network effectively integrates the detailed information of low-level feature maps with the semantic information of high-level feature maps through the combination of cross-layer connections, multi-scale feature concatenation and attention mechanism, making small target detection more accurate and large target positioning more precise, ensuring that the FB-YOLOv8 model can achieve robust target detection in various complex scenarios, especially for multi-scale target detection. The bidirectional path aggregation network finally outputs the fused feature map to the detection head for vehicle target classification and target box prediction.

[0027] Reference Figure 4, is a schematic diagram of a bidirectional path aggregation network in an embodiment, the input of the bidirectional path aggregation network comes from feature maps P2, P3, P4, and P5 extracted by a multi-layer convolutional network. Feature maps P2, P3, P4, and P5 are input to the feature alignment aggregation module (FAAM) to obtain a fused feature map. Concat represents feature concatenation, which realizes the concatenation operation of feature maps from different paths and integrates them into an enhanced feature representation. The concatenated features contain multi-scale information and can better cope with vehicle target detection tasks of different sizes. The purple arrow in the figure: represents cross-layer feature fusion, and the features are transferred from the low-resolution layer to the high-resolution layer to achieve information supplementation. The high-resolution layer features are transferred to the low-resolution layer through upsampling to provide global context information for the low-layer features. The blue arrow: represents the flow direction of the downsampled features, which is used to adjust the feature scale so that the features of different layers match. The orange arrow: represents the flow direction of the upsampled features, which is used to transfer the high-resolution features to the low-resolution layer to achieve feature sharing. After the cross-layer feature interaction, the features of each layer are concatenated with the features of the current layer. The concatenated feature map combines multiple layers of information, including details and global semantics. The features after FAAM and Concat operations are used as the output of BPAN for subsequent vehicle target detection tasks.

[0028] The FB-YOLOv8 target detection network adds the Feature Alignment Aggregation Module (FAAM) and the Bidirectional Path Aggregation Network (BPAN) modules based on YOLOv8, which improves the accuracy and robustness of detection. This architecture is particularly friendly to small and multi-scale target detection, and is suitable for accurate detection of multi-target vehicles in traffic scenes.

[0029] At the same time, the present invention uses a Kalman filter to perform state estimation and prediction on each vehicle target based on the vehicle target detection result, including the position, speed and direction information of the vehicle target, and obtains the motion characteristics of each vehicle target. Kalman filter (KF) is a state optimal estimation algorithm that can be used for state estimation in the process of multi-target tracking. The algorithm is based on the linear system state equation and performs optimal estimation of the vehicle target state through the two stages of prediction and update. In the prediction process, the Kalman filter predicts the position information of the current time frame through the system state equation and the state transfer matrix. In the update process, the Kalman filter updates and corrects the position information of the previous prediction process through the observation equation and matrix.

[0030] In order to enhance the distinguishing features of vehicle targets, the vehicle target detection results are input into the re-identification network MSFF-OSNet for appearance feature extraction, which are then used for appearance feature matching and association, effectively reducing problems such as ID switching caused by vehicle scale changes. The re-identification network MSFF-OSNet uses a feature pyramid structure network in the backbone to extract feature information representing different semantics and scales from different stages of the backbone. In addition, the CBAM attention mechanism is introduced in the multi-level feature fusion process to focus on image areas containing important information, thereby extracting more discriminative appearance features. The extracted appearance features are then used for appearance feature matching, effectively reducing problems such as identity switching caused by vehicle scale changes.

[0031] Figure 5This is a re-identification network structure diagram provided by an embodiment, which shows the specific structure of the re-identification network MSFF-OSNet, and uses the OSNet network as the backbone network. The feature pyramid structure network captures the different scale information of the target frame image through convolution blocks Conv1, Conv2, Conv3, Conv4 to Conv5, and extracts feature maps C1, feature map C2, feature map C3, feature map C4, and feature map C5 of different scales of the target frame. Under the framework of the feature pyramid network, features at different levels are integrated to make full use of multi-scale information. Feature maps C2 and C3 are adjusted by upsampling and using a 1×1 convolution layer on the number of channels to match feature maps C1 and feature map C2, respectively. Feature maps C4 and feature maps C5 are adjusted on the number of channels by a 1×1 convolution layer to match feature maps C3 and feature maps C4, respectively. Specifically, feature map C2 is adjusted on the number of channels by upsampling and using a 1×1 convolution layer to match feature map C1, and feature map D2 is obtained. The enhanced features of feature map C1 obtained by the CBAM attention module are then element-wise added to feature map D2 to obtain output feature map F1, which is passed to the next layer. Feature map C3 is adjusted to match feature map C2 by upsampling and using a 1×1 convolution layer on the number of channels to obtain feature map D3, while the enhanced features of output feature map F1 obtained by the CBAM attention module are then element-wise added to feature map D3 to output feature map F2, which is passed to the next layer. Feature map C4 is adjusted to match feature map C3 by a 1×1 convolution layer on the number of channels to obtain feature map D4, while the enhanced features of output feature map F2 obtained by the CBAM attention module are then element-wise added to feature map D4 to output feature map F3, which is passed to the next layer. The feature map C5 is adjusted in terms of the number of channels through a 1×1 convolutional layer to match the feature map C4, and the feature map D5 is obtained. At the same time, the enhanced features of the output feature map F3 obtained by the CBAM attention module are then added to the feature map D5 in an element-wise manner, and the final feature map F4, which is the appearance feature of the vehicle target, is output. In this way, it is propagated to the lower layers step by step through upsampling, and is fused step by step with the lower layer features. The final fused features are used as the output features, and the output features can take into account both global semantic information and detail information.

[0032] The CBAM attention module consists of two independent submodules: the channel attention module CAM and the spatial attention module SAM, which reconstructs the feature map to highlight key information. The channel attention module CAM focuses on the relationship between the channels of the feature map and evaluates the importance of each pixel. Take the enhanced features of the feature map C1 obtained by the CBAM attention module as an example: Input channel attention module CAM output channel attention module SAM It is expressed as:

[0033] Spatial attention of the output of the spatial attention module SAM It is expressed as:

[0034] In the above two equations, C1 represents the input feature map C1, Avg represents the average value, Max represents the global maximum pooling, MLP represents the multi-layer perceptron, σ{} represents the Sigmoid activation function, Represents a convolution kernel with a side length of 5.

[0035] Therefore, the overall expression of the CBAM attention module is:

[0036] That is, the enhanced features of the feature map C1 obtained by the CBAM attention module. Based on the same method, the enhanced features of the output feature map F1, the output feature map F2, and the output feature map F3 can be obtained by the CBAM attention module.

[0037] The feature map C2 is upsampled and adjusted in the number of channels using a 1×1 convolutional layer to match the feature map C1 to obtain the feature map D2. The expression of the feature map D2 is as follows:

[0038] The re-identification network MSFF-OSNet introduces a feature pyramid network (FPN) in the backbone to capture the pixel dependencies between multiple feature maps. The feature pyramid network aggregates features from different levels to generate features that contain both low-level information and high-level semantics. In the feature fusion process, a convolutional block attention module (CBAM) is introduced to help the model focus on image areas containing important information, thereby extracting more discriminative appearance features and improving the accuracy of matching associations.

[0039] In the process of multi-target tracking, it is necessary to match the detected targets and the predicted target trajectories, but due to the changes in the object position and other information, there are multiple predicted trajectories. Therefore, it is necessary to select an algorithm to complete the tracking task. The Hungarian algorithm is an algorithm based on the idea of ​​sufficiency proof in Hall's theorem. It is often used in graph matching problems. The core idea of ​​the theorem is to find augmenting paths. By applying this theorem, the Hungarian algorithm can use augmenting paths to find the maximum matching algorithm with the bipartite graph. The input of the matching algorithm is the cost matrix C(mⅹn). The matching algorithm steps are as follows: Step 1: Find the minimum value col in each column of the cost matrix C min , so that A = C - col min ; Step 2: Find the minimum value of each row in the cost matrix C min , so that B = C - raw min ; Step 3: Select a horizontal line perpendicular to the Y axis or perpendicular to the X axis to change all zero elements in B by the minimum number; Step 4: If the number of horizontal lines is m, return the optimal match, otherwise go to step 5; Step 5: Find the minimum value in B that is not changed by any horizontal line, add this value to any column that is not covered by any horizontal line, subtract this value from any row that is not covered by any horizontal line, and return to step 3. This paper proposes a multi-target tracking method that solves the problem of multi-scale feature fusion of variable-scale vehicle targets from the perspective of a drone. First, this paper proposes a FB-YOLOv8 framework to solve the problem of missed detection. The framework combines the feature alignment aggregation module (FAAM) and the bidirectional path aggregation network (BPAN) to enhance multi-scale feature fusion. Secondly, this paper proposes a multi-scale feature fusion network (MSFF-OSNet) to extract appearance features to solve the identity switching problem. The framework integrates the feature pyramid network (FPN) and the convolutional block attention module (CBAM) into OSNet to capture multi-level pixel dependencies and combine low-level and high-level features. By effectively integrating the FB-YOLOv8 and MSFF-OSNet modules into the tracking process, the tracking accuracy and stability are improved. The experimental results are shown in Figure 6 and Figure 7 As shown, Figure 6 The experimental comparison results of the method of the present invention and the benchmark method BoT-SORT in the daytime scene are shown in the figure. Figure 7 The experimental comparison results of the proposed method and the benchmark method BoT-SORT in night scenes are shown in the figure. In both scenes, when the vehicle changes scale at the intersection, the original benchmark method BoT-SORT method has missed detection (FN) and ID switching (IDSW) during tracking ( Figure 6 and Figure 7 The method of the present invention can avoid these problems, which proves that the method can achieve accurate and stable tracking of vehicle targets in scale-changing scenarios.

[0040] On the other hand, the present invention provides a multi-target vehicle tracking device from the perspective of a drone, comprising: The first module is used to input the image sequence taken by the drone, perform vehicle target detection on the image, and obtain the vehicle target detection result in the image; The second module is used to extract appearance features based on the vehicle target detection results, and extract the appearance features of each vehicle target in each target frame; The third module is used to use the Kalman filter to estimate and predict the state of each vehicle target based on the vehicle target detection result, including the position, speed and direction information of the vehicle target, and obtain the motion characteristics of each vehicle target; The fourth module is used to combine the extracted appearance features and motion features of each vehicle target to obtain a target feature vector of each vehicle target; The fifth module is used to match the target feature vector of the vehicle target in the current frame taken by the drone with the target feature vector of the known vehicle target in the previous frame, and use the Hungarian algorithm for optimal allocation to ensure the stability of tracking between the previous and next frames; The sixth module is used to output the position information of each vehicle target into the result to form a tracking track of each vehicle target.

[0041] On the other hand, the present invention provides a computer device, including a memory and a processor, the memory storing a computer program, and the processor implementing the steps of the multi-target vehicle tracking method from the perspective of a drone provided in any of the above embodiments when executing the computer program. The computer device may be a server. The computer device includes a processor, a memory, a network interface, and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store sample data. The network interface of the computer device is used to communicate with an external terminal via a network connection.

[0042] On the other hand, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the multi-target vehicle tracking method from the perspective of a drone provided in any of the above embodiments are implemented.

[0043] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0044] Matters not covered by the present invention are known technologies.

[0045] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0046] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

[0047] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multi-target vehicle tracking method from the perspective of a drone, characterized in that: include: Input the image sequence taken by the drone, perform vehicle target detection on the image, and obtain the vehicle target detection result in the image; Appearance features are extracted based on the vehicle target detection results to extract the appearance features of each vehicle target in each target frame; Based on the vehicle target detection results, the Kalman filter is used to estimate and predict the state of each vehicle target, including the position, speed and direction information of the vehicle target, and the motion characteristics of each vehicle target are obtained; Combining the extracted appearance features and motion features of each vehicle target to obtain a target feature vector of each vehicle target; Match the target feature vector of the vehicle target in the current frame taken by the drone with the target feature vector of the known vehicle target in the previous frame, and use the Hungarian algorithm to optimize the allocation to ensure the stability of tracking between the previous and next frames; The position information of each vehicle target is output to the result to form the tracking trajectory of each vehicle target.

2. The multi-target vehicle tracking method under the perspective of a drone according to claim 1, characterized in that: Perform vehicle target detection on the image, including: Preprocessing the images in the image sequence taken by the drone in sequence, including normalization and size adjustment; Extract feature maps of different resolutions from the preprocessed image; Perform multi-scale feature alignment on the generated feature maps of different resolutions to obtain a fused feature map; The fused feature map obtained after the multi-scale feature alignment operation is input into the bidirectional path aggregation network to further fuse the deep and shallow features to obtain the final feature; Multi-scale target detection is performed based on the final features, and the vehicle target detection results are output.

3. The multi-target vehicle tracking method under the perspective of a drone according to claim 2, characterized in that: Output vehicle target detection results, including: removing target frames with high overlap and low confidence: vehicle target detection results include the target frame of the vehicle target and its confidence, and filter out target frames with high overlap through the non-maximum suppression algorithm; set the confidence threshold, and target frames below the confidence threshold will be directly discarded; after removing target frames with high overlap and low confidence, the remaining target frames are post-processed, including target frame coordinate conversion and confidence sorting, and the final vehicle target detection results are output.

4. The multi-target vehicle tracking method from the perspective of a drone according to claim 2 or 3, characterized in that: The generated feature maps of different resolutions are subjected to multi-scale feature alignment to obtain a fused feature map, including: For the generated feature maps of different resolutions, feature maps with adjacent resolutions are grouped together, and in each group of feature maps, one is a low-resolution feature map and the other is a high-resolution feature map; For any group of feature maps, the low-resolution feature map is upsampled based on bilateral interpolation, and then the upsampled low-resolution feature map and the high-resolution feature map are connected in series to obtain the connected features; the connected features are processed by a depthwise separable convolution module to obtain a spatially aligned high-resolution feature map; the spatially aligned high-resolution feature map is added to the upsampled low-resolution feature map to obtain a fused feature map, thus obtaining the fused feature map corresponding to each group of feature maps.

5. The multi-target vehicle tracking method under the perspective of a drone according to claim 4 is characterized in that: The depthwise separable convolution module consists of a depthwise convolution layer and a pointwise convolution layer. A depthwise convolution layer with a kernel size of 3×3 is used to capture the local spatial information around each sampling position and pass it to the pointwise convolution layer to generate a two-dimensional offset; the two-dimensional offset and the high-resolution feature map are input into the alignment function to obtain a spatially aligned high-resolution feature map.

6. The multi-target vehicle tracking method under the perspective of a drone according to claim 4, characterized in that: The vehicle target detection results are input into the re-identification network MSFF-OSNet for appearance feature extraction. The re-identification network MSFF-OSNet adopts a feature pyramid structure network in the backbone, extracts different levels of feature information representing different semantics and scales from different stages of the backbone, and introduces the CBAM attention mechanism in the multi-level feature fusion process to focus on the image areas containing important information, thereby extracting more discriminative appearance features.

7. The multi-target vehicle tracking method under the perspective of a drone according to claim 3, characterized in that: Appearance feature extraction includes: The feature pyramid structure network captures information of different scales of the target frame image through convolution blocks Conv1, Conv2, Conv3, Conv4 to Conv5 in turn, and extracts feature maps C1, C2, C3, C4, and C5 of different scales of the target frame; Feature map C2 is upsampled and adjusted in the number of channels using a 1×1 convolutional layer to match feature map C1 to obtain feature map D2. The enhanced features of feature map C1 obtained by the CBAM attention module are then added to feature map D2 in an element-wise manner to obtain the output feature map F1, which is passed to the next layer. Feature map C3 is upsampled and adjusted to match feature map C2 using a 1×1 convolutional layer on the number of channels to obtain feature map D3. At the same time, the enhanced features of the output feature map F1 obtained by the CBAM attention module are then added to the feature map D3 in an element-wise manner to output feature map F2 and pass it to the next layer. Feature map C4 is adjusted in terms of the number of channels to match feature map C3 through a 1×1 convolutional layer, resulting in feature map D4. At the same time, the enhanced features of the output feature map F2 obtained by the CBAM attention module are then added to feature map D4 in an element-wise manner, outputting feature map F3 and passing it to the next layer. The feature map C5 is adjusted in terms of the number of channels through a 1×1 convolutional layer to match the feature map C4, and the feature map D5 is obtained. At the same time, the enhanced features of the output feature map F3 obtained by the CBAM attention module are then added to the feature map D5 in an element-wise manner, and the final feature map F4, which is the appearance feature of the vehicle target, is output.

8. A multi-target vehicle tracking device from the perspective of a drone, characterized in that: include: The first module is used to input the image sequence taken by the drone, perform vehicle target detection on the image, and obtain the vehicle target detection result in the image; The second module is used to extract appearance features based on the vehicle target detection results, and extract the appearance features of each vehicle target in each target frame; The third module is used to use the Kalman filter to estimate and predict the state of each vehicle target based on the vehicle target detection result, including the position, speed and direction information of the vehicle target, and obtain the motion characteristics of each vehicle target; The fourth module is used to combine the extracted appearance features and motion features of each vehicle target to obtain a target feature vector of each vehicle target; The fifth module is used to match the target feature vector of the vehicle target in the current frame taken by the drone with the target feature vector of the known vehicle target in the previous frame, and use the Hungarian algorithm for optimal allocation to ensure the stability of tracking between the previous and next frames; The sixth module is used to output the position information of each vehicle target into the result to form a tracking track of each vehicle target.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the multi-target vehicle tracking method from the perspective of the drone as claimed in claim 1 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, the steps of the multi-target vehicle tracking method from the perspective of the drone as claimed in claim 1 are implemented.

Images