Target detection-tracking integrated method and system based on AFPN and LSK-Net

Abstract

The invention discloses a target detection-tracking integrated method and system based on AFPN and LSK-Net, and the method comprises the steps: inputting a video image frame into a CSPDarknet backbone network, and obtaining feature maps of multiple scales; the method comprises the following steps: fusing feature maps of multiple scales by utilizing an AFPN (Progressive Feature Pyramid Network), carrying out adaptive screening and feature enhancement on the fused feature maps through a space attention mechanism of a large-scale selective kernel network (LSK-Net) in an ELAN (Efficient Layer Aggregation Network), and generating a target detection result based on the enhanced feature maps; the target detection result comprises detection frame data, detection confidence and appearance features; and based on the detection frame data, the detection confidence and the appearance features, performing state estimation and tracking association on the detection target to obtain a target tracking result. According to the method, target detection-tracking end-to-end integrated optimization can be realized, information redundancy of target detection and tracking is reduced, and real-time performance is improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, specifically to an integrated object detection-tracking method and system based on AFPN and LSK-Net. Background Technology

[0002] In practical applications of vehicle speed measurement methods based on monocular vision, in order to improve the model's ability to identify medium and small targets and reduce false detections and missed detections, it is necessary to optimize the target detection algorithm and the target tracking algorithm.

[0003] In recent years, deep learning-based object detection algorithms have developed rapidly, mainly divided into two-stage detectors (such as Faster R-CNN and Mask R-CNN) and single-stage detectors (such as YOLO and SSD). YOLOv7, as one of the current mainstream single-stage detectors, has achieved a good balance between speed and accuracy, but its detection accuracy for small and medium-sized targets in complex scenes is still insufficient. Existing improvement methods include: using Path Aggregation Network (PANet) to enhance multi-scale feature fusion capabilities through path enhancement networks, and introducing a weighted bidirectional feature pyramid in Bi-Directional Feature Pyramid Network (BiFPN), but these methods have not solved the problem of excessively large gaps between high-level and low-level feature semantics.

[0004] Furthermore, in specific application research such as vehicle speed measurement based on monocular vision, the target detection module and the target tracking module in the existing methods are operated and optimized independently. The fragmented design of detection and tracking results in low feature reuse rate and high computational redundancy, leading to information redundancy and insufficient real-time performance between the target detection module and the target tracking module. Summary of the Invention

[0005] In view of this, this application provides an integrated target detection-tracking method based on AFPN and LSK-Net, the method comprising: The video image frames to be processed are input into the CSPDarknet backbone network to obtain feature maps of multiple scales corresponding to the video image frames. The feature maps of multiple scales are fused using AFPN to obtain the first feature map. The first feature map is adaptively filtered and enhanced using the spatial attention mechanism of LSK-Net in ELAN to obtain the second feature map, and the target detection result is generated based on the second feature map. The target detection result includes detection box data, detection confidence, and appearance features. Based on the detection box data, detection confidence, and appearance features, the state of the detected target is estimated and tracked to obtain the target tracking result. The target tracking result is obtained by cascading matching based on the updated appearance features and predicted trajectory state data.

[0006] Optionally, fusing feature maps at multiple scales to obtain a first feature map includes: performing resolution alignment on the feature maps at multiple scales to obtain multiple first reference feature maps with the same spatial scale; generating a first spatial weight map corresponding to the spatial size of each of the multiple first reference feature maps; and performing weighted summation on the multiple first reference feature maps according to their respective first spatial weight maps to obtain the first feature map.

[0007] Optionally, adaptive filtering and feature enhancement are performed on the first feature map to obtain a second feature map, including: decoupling the first convolutional kernel to obtain multiple second convolutional kernels; each second convolutional kernel has a different size and dilation rate, and the size of the first convolutional kernel is larger than the size of each second convolutional kernel; convolving the first feature map with the multiple second convolutional kernels to obtain multiple second reference feature maps; fusing the spatial feature vectors of the multiple second reference feature maps with a 1×1 convolutional kernel to obtain a third reference feature map; performing max pooling and average pooling on the third reference feature map respectively, and converting it into multiple reference spatial attention feature maps through convolution operations; the number of reference spatial attention feature maps is the same as the number of second reference feature maps; performing a non-linear transformation on each reference spatial attention feature map using the Sigmoid activation function to generate a second spatial weight map; performing a weighted summation of the second spatial weight map and multiple second reference feature maps, and fusing them through a convolutional layer to obtain a target spatial attention feature; and multiplying the target spatial attention feature and the first feature map element-wise to obtain the second feature map.

[0008] Optionally, state estimation and tracking association are performed on the detected target, including: updating the appearance features using the EMA strategy to obtain the updated appearance features; estimating the motion state of the detected target using the NSA Kalman filtering method based on the detection box data and detection confidence to obtain the predicted trajectory state data of the detected target; and performing cascade matching based on the updated appearance features and predicted trajectory state data to obtain the target tracking result.

[0009] Optionally, the appearance features are updated using an EMA strategy to obtain updated appearance features, including: creating an appearance feature update formula based on the appearance features, the exponential moving average smoothing coefficient, and historical appearance features determined based on historical video image frames; and updating the appearance features based on the appearance feature update formula to obtain updated appearance features.

[0010] Optionally, the appearance feature update formula is expressed as: ; Where t is greater than 1 and less than or equal to the total number of video frames, and i is an integer greater than 0. This represents the appearance feature corresponding to trajectory segment i from the initial video image frame to the t-th video image frame. This represents the appearance feature corresponding to trajectory segment i from the initial video image frame to the (t-1)th video image frame. Exponential moving average smoothing coefficient, The appearance features of the target are detected in the trajectory segment i extracted from the t-th video image frame.

[0011] Optionally, the motion state of the detected target is estimated using the NSA Kalman filtering method based on the detection box data and detection confidence, resulting in predicted trajectory state data of the detected target, including: The detection confidence level and baseline observation noise are substituted into the observation noise calculation formula to obtain adaptive observation noise. Based on the state data and motion model of the detected target in the previous video image frame, the state data of the detected target in the current video image frame is predicted. The state data of the detected target is updated based on the adaptive observation noise and the state data of the detected target in the current video image frame to obtain updated state data. The predicted trajectory state data of the detected target is created based on the updated state data.

[0012] Optionally, the formula for calculating observation noise is expressed as: ; in, Indicates adaptive observation noise, Indicates the reference observation noise. This indicates the confidence level of the detection.

[0013] Optionally, the object detection-tracking integration method based on AFPN and LSK-Net is applied to the object detection-tracking integration model based on AFPN and LSK-Net. The training process of the object detection-tracking integration model based on AFPN and LSK-Net includes: creating a joint loss function associated with the object detection loss function, the appearance feature recognition loss function, and the motion state estimation loss function; the object detection loss function is used to calculate the prediction loss of the detection box data, the detection confidence, and the category of the detected object; the appearance feature recognition loss function is used to calculate the prediction loss of the appearance features; and the motion state estimation loss function is used to calculate the motion state estimation loss of the NSA Kalman filtering method; the network parameters of the object detection-tracking integration model based on AFPN and LSK-Net are continuously optimized through the joint loss function until the convergence condition is met, resulting in the trained object detection-tracking integration model based on AFPN and LSK-Net.

[0014] This application provides an integrated target detection-tracking system based on AFPN and LSK-Net, including a CSPDarknet backbone network, AFPN, an ELAN incorporating the LSK-Net spatial attention mechanism, and an improved DeepSORT network based on EMA policy and NSAKalman filtering. Its features include... The CSPDarknet backbone network is used to extract features from the video image frames to be processed, and obtain feature maps of multiple scales corresponding to the video image frames. AFPN is used to fuse feature maps of multiple scales to obtain the first feature map; ELAN, which includes the LSK-Net spatial attention mechanism, is used to adaptively filter and enhance the first feature map to obtain the second feature map, and to generate the object detection result based on the second feature map; the object detection result includes detection box data, detection confidence and appearance features; An improved DeepSORT network is used to perform state estimation and tracking association of detected targets based on detection box data, detection confidence, and appearance features, thereby obtaining target tracking results.

[0015] According to the scheme provided in the embodiments of this application, the video image frame to be processed is input into the CSPDarknet backbone network to obtain feature maps of multiple scales corresponding to the video image frame; the feature maps of multiple scales are fused using AFPN to obtain a first feature map; the first feature map is adaptively filtered and enhanced using the spatial attention mechanism of LSK-Net in ELAN to obtain a second feature map, and a target detection result is generated based on the second feature map; the target detection result includes detection box data, detection confidence, and appearance features; based on the detection box data, detection confidence, and appearance features, state estimation and tracking association are performed on the detected target to obtain the target tracking result. In this process, the fusion of feature maps of multiple scales extracted by the backbone network through AFPN can alleviate the gap between high-level feature semantics and low-level feature semantics, thereby improving the multi-scale target detection capability; the adaptive filtering and feature enhancement of the first feature map using the spatial attention mechanism of LSK-Net in ELAN enables the model to automatically learn the receptive field size corresponding to the feature map at different spatial locations, thereby dynamically selecting different convolution kernels for feature extraction, reducing false detections and false negatives, and enhancing the model's adaptability and discrimination ability in complex scenes. Furthermore, this application obtains detection box data, detection confidence, and appearance features through target detection, and can directly perform state estimation and tracking association on the detected target based on the detection box data, detection confidence, and appearance features. This enables end-to-end integrated optimization of target detection and tracking, reduces information redundancy in target detection and tracking, and improves real-time performance. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein: Figure 1 A flowchart illustrating an integrated target detection-tracking method based on AFPN and LSK-Net, provided for an embodiment of this application; Figure 2 This is a schematic diagram of an integrated target detection-tracking system based on AFPN and LSK-Net, provided as an embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0019] It should be noted that the terms "first, second, and third" used in the embodiments of the present invention are only used to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, and third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present invention described herein can be implemented in an order other than that illustrated or described herein.

[0020] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiments of this application pertain. It should also be understood that terms such as those defined in general dictionaries should be understood to have a meaning consistent with their meaning in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0021] For multi-object tracking (MOT) algorithms, the mainstream frameworks currently include SORT and DeepSORT. SORT is based on Kalman filtering and the Hungarian algorithm, which is fast but suffers from severe ID switching in occluded scenarios. DeepSORT introduces appearance features and cascaded matching to improve occlusion robustness, but the feature library update uses a fixed sliding window, which is susceptible to detection noise.

[0022] Figure 1 This is a flowchart illustrating an integrated target detection and tracking method based on AFPN and LSK-Net provided in an embodiment of this application. The integrated target detection and tracking method based on AFPN and LSK-Net provided in this embodiment can be executed by an electronic device, such as a computer or server. Figure 1 As shown, the object detection-tracking integrated method based on AFPN and LSK-Net provided in this application includes: S101. Input the video image frame to be processed into the CSPDarknet backbone network to obtain feature maps of multiple scales corresponding to the video image frame.

[0023] It should be noted that the video image frames to be processed can be any image frames obtained after segmenting the video used for target detection and target tracking. The video can be captured by roadside cameras, vehicle cameras, etc., and includes the detected targets (such as pedestrians and vehicles); the CSPDarknet (Cross Stage Partial Network) backbone network can be the backbone network of the YOLOv7 model.

[0024] In some embodiments, feature extraction of different levels of semantics is performed on the input video image frames through the CSPDarknet backbone network, which can obtain feature maps of multiple scales. The number and richness of features (texture features, semantic features, etc.) contained in the feature maps of different scales are different.

[0025] For example, the feature maps at multiple scales may include high-resolution detail feature maps, mid-level semantic feature maps, deep semantic feature maps, and ultra-deep feature maps. The spatial size (scale) of each feature map may be 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the video image frame to be processed, respectively. For example, for a video image frame with a spatial size of 640x640, the spatial sizes of the corresponding multiple feature maps may be 160x160, 80x80, 40x40, and 20x20, respectively.

[0026] S102. Use AFPN to fuse feature maps of multiple scales to obtain the first feature map.

[0027] In some embodiments, the Asymptotic Feature Pyramid Network (AFPN) can generate a weight map corresponding to the feature map at each scale, and then fuse the feature maps at different scales based on the weight map to obtain the fused first feature map.

[0028] S103. Adaptively filter and enhance the first feature map using the spatial attention mechanism of LSK-Net in ELAN to obtain the second feature map, and generate the target detection result based on the second feature map.

[0029] Here, the target detection results include bounding box data, detection confidence, and appearance features.

[0030] In some embodiments, the spatial attention mechanism of the Large Selective Kernel Network (LSK-Net) can use multiple convolutional kernels of different sizes (representing receptive fields of different sizes) in parallel to scan the first feature map. Then, through a selection mechanism, the LSK-Net network learns on its own which region of the current scene should use a large or small receptive field, thereby giving feature extraction dynamic selection capabilities, thus enabling more accurate localization and recognition of small targets.

[0031] In some embodiments, adaptive filtering and feature enhancement can be achieved by the model learning to determine the weights corresponding to each feature in the feature map, and based on these weights, the corresponding features can be filtered or enhanced, thereby improving the quality and richness of the Chinese medicine features in the first feature map.

[0032] S104. Based on the detection box data, detection confidence, and appearance features, perform state estimation and tracking association on the detected target to obtain the target tracking result.

[0033] In some embodiments, the appearance features can be updated to obtain updated appearance features, and motion state estimation can be performed based on the detection box data and detection confidence to obtain the trajectory state data of the detected target. Then, cascade matching can be performed based on the updated appearance features and trajectory state data to obtain the target tracking result of the detected target.

[0034] It should be noted that this application constructs an integrated target detection-tracking network architecture with AFPN and LSK-Net as core feature enhancement modules. Through structural modifications to AFPN and LSK-Net, the quality of the backbone network output features (second feature map) is fundamentally improved, giving it both excellent detection discriminativeness and tracking robustness. This provides unprecedented high-quality input for any subsequent tracking logic (including but not limited to using EMA and NSA-Kalman), achieving end-to-end collaborative optimization from the feature layer to the tracking output, and solving the fundamental contradiction of the disconnect between detection features and tracking task requirements in traditional schemes.

[0035] In this embodiment, the video image frames to be processed are input into the CSPDarknet backbone network to obtain feature maps of multiple scales corresponding to the video image frames. The AFPN is used to fuse the feature maps of multiple scales to obtain a first feature map. The spatial attention mechanism of LSK-Net in ELAN is used to adaptively filter and enhance the first feature map to obtain a second feature map, and a target detection result is generated based on the second feature map. The target detection result includes detection box data, detection confidence, and appearance features. Based on the detection box data, the detection confidence, and the appearance features, state estimation and tracking association are performed on the detected target to obtain the target tracking result. In this process, fusing the feature maps of multiple scales extracted by the backbone network through AFPN can alleviate the gap between high-level and low-level feature semantics, thereby improving the multi-scale target detection capability. The spatial attention mechanism of LSK-Net in ELAN is used to adaptively filter and enhance the first feature map, enabling the model to automatically learn the receptive field size corresponding to the feature map at different spatial locations, thereby dynamically selecting different convolutional kernels for feature extraction, reducing false detections and false negatives, and enhancing the model's adaptability and discrimination ability in complex scenes. Furthermore, this application obtains detection box data, detection confidence, and appearance features through target detection, and can directly perform state estimation and tracking association for the detected target based on the detection box data, detection confidence, and appearance features. This enables end-to-end integrated optimization of target detection and tracking, reduces information redundancy in target detection and tracking, and improves real-time performance.

[0036] In some embodiments of this application, the fusion of feature maps at multiple scales in step S102 to obtain the first feature map can be achieved through the following steps S1021 to S1023, and each step will be described below.

[0037] S1021. Align the feature maps at multiple scales to obtain multiple first reference feature maps with the same spatial scale.

[0038] In some embodiments, to facilitate the fusion of feature maps at different scales, the dimensions of multiple feature maps can be adjusted to obtain multiple first reference feature maps with the same spatial scale. For feature maps requiring resolution upsampling, 1×1 convolution operations and bilinear interpolation techniques can be used for upsampling. For feature maps requiring resolution downsampling, the appropriate convolution kernel size and stride can be selected based on the specific downsampling rate. For example, when 2x downsampling is required, a 2×2 convolution kernel with a stride of 2 is used; for 4x and 8x downsampling, a 4×4 convolution kernel with a stride of 4 and an 8×8 convolution kernel with a stride of 8 are used respectively, ensuring the accuracy of feature fusion.

[0039] S1022. Generate a first spatial weight map corresponding to the spatial size of each of the multiple first reference feature maps.

[0040] It should be noted that the first spatial weight map and the first reference feature map have the same spatial scale.

[0041] In some embodiments, AFPN further optimizes the feature fusion process by introducing an Adaptive Spatial Feature Fusion (ASFF) algorithm. ASFF can dynamically adjust the weights of different first feature maps, highlighting the importance of key layers, thereby alleviating the feature conflict problem that may occur in multi-object detection.

[0042] In some embodiments, multiple first reference feature maps can be concatenated, and the number of channels can be compressed to N (N is the number of first reference feature maps, for example, 3) through a 1×1 convolutional layer. The spatial dimension is normalized by the Softmax function, and the weight corresponding to each pixel position in the first reference feature map is calculated. The sum of the weights of the N first reference feature maps at that pixel position is 1, and finally, N spatial weight feature maps can be obtained.

[0043] S1023. The multiple first reference feature maps are weighted and summed according to their respective first spatial weight maps to obtain the first feature map.

[0044] In some embodiments, each resolution-aligned first reference feature map and its corresponding spatial weight feature map can be multiplied element-wise, and then all the results of the multiplication can be summed to obtain the fused first feature map.

[0045] Understandably, by generating a first spatial weight map corresponding to the spatial size of the first reference feature map obtained after resolution alignment, the network can adaptively learn which layer's features should be relied upon more at which location. For example, for the core region of a large target, the network may assign higher weights to deep semantic features; while for the edge of the target, it may rely more on shallow detail features, thus better preserving important features and improving the accuracy of target detection.

[0046] In some embodiments of this application, the adaptive filtering and feature enhancement of the first feature map in step S103 to obtain the second feature map can be achieved by the following steps S1031 to S1037, and each step is described below.

[0047] S1031. Decouple the first convolution kernel to obtain multiple second convolution kernels.

[0048] It should be noted that the size and expansion rate of each second convolution kernel are different, and the size of the first convolution kernel is larger than the size of each second convolution kernel.

[0049] In some embodiments, to reduce the computational complexity of the large convolutional kernel (first convolutional kernel), the large convolutional kernel can be decoupled to obtain multiple smaller convolutional kernels (second convolutional kernels). Assuming that the first convolutional kernel is decoupled to obtain M second convolutional kernels, the... The size of the second convolutional kernel is expansion rate The corresponding receptive field is Then the following conditions are met: .

[0050] Understandably, by decoupling the large first convolutional kernel into a smaller second convolutional kernel, rich features with multiple receptive field sizes can be generated, facilitating subsequent dynamic kernel selection operations. Furthermore, by decoupling the first convolutional kernel instead of directly calculating based on it, the number of model parameters can be reduced, thereby improving the model's computational efficiency.

[0051] S1032. The first feature map is convolved using multiple second convolution kernels to obtain multiple second reference feature maps.

[0052] In some embodiments, multiple second convolution kernels can be sequentially convolved on the first feature map to obtain multiple second reference feature maps. Since the size and dilation rate of each second convolution kernel are different, the size of each second reference feature map obtained after convolution is also different.

[0053] S1033. Channel fusion of spatial feature vectors of multiple second reference feature maps is performed using a 1×1 convolution kernel to obtain a third reference feature map.

[0054] In some embodiments, each second reference feature map can be convolved with a 1×1 convolution kernel, and the resulting feature maps can be concatenated along the channel dimension to obtain a third reference feature map, thereby forming a multi-scale feature pool to facilitate subsequent feature selection and feature fusion.

[0055] S1034 performs max pooling and average pooling on the third reference feature map, and then converts it into multiple reference space attention feature maps through convolution operations.

[0056] Here, the number of reference spatial attention feature maps is the same as the number of second reference feature maps.

[0057] In some embodiments, by performing max pooling and average pooling on the third reference feature map respectively, the spatial information of each channel can be compressed into a single value. Max pooling can capture the most prominent features, while average pooling can capture the overall feature distribution, thereby obtaining global spatial statistical information from two different perspectives.

[0058] In some embodiments, the spatial features obtained after max pooling and average pooling can be concatenated using a 1×1 convolution kernel, and then... The operation transforms channel-level information into spatial-level weights, thereby obtaining a reference spatial attention feature map composed of multiple weights.

[0059] S1035. Use the Sigmoid activation function to perform a non-linear transformation on each reference space attention feature map to generate a second space weight map.

[0060] In some embodiments, by performing a non-linear transformation on each reference space attention feature map using the Sigmoid activation function, the elements (weights) in the reference space attention feature map can be normalized to the range (0,1), that is, the elements in the generated second space weight map represent the normalized weights.

[0061] S1036. The second spatial weight map and multiple second reference feature maps are weighted and summed, and then fused through a convolutional layer to obtain the target space attention features.

[0062] In some embodiments, the number of second spatial weight maps is the same as the number of second reference feature maps. The second spatial weight maps and the second reference feature maps can be multiplied element-wise, and the corresponding elements of each feature map obtained after element-wise multiplication can be added together. Then, they can be further fused through a convolutional layer to obtain the target spatial attention features.

[0063] S1037. Multiply the target space attention features and the first feature map element by element to obtain the second feature map.

[0064] In some embodiments, the target space attention features can be multiplied by each element (feature) in the first feature map to obtain the fused second feature map.

[0065] Understandably, by performing max pooling and average pooling on the third reference feature map and converting it into multiple reference spatial attention feature maps through convolution, the contribution of features from all different receptive fields can be controlled, enabling adaptive selection of spatial features. Furthermore, the spatial weight map generated based on the reference spatial attention feature map is weighted and summed with the second reference feature map, and then fused through a convolutional layer. The resulting target spatial attention feature is then multiplied element-wise with the first feature map, allowing the network to dynamically mix contextual information of different scales at each pixel location, thereby improving the richness of the fused features.

[0066] In some embodiments of this application, the state estimation and tracking association of the detected target in step S104 can be achieved by the following steps S1041 to S1042, and each step will be described below.

[0067] S1041. The appearance features are updated using the EMA strategy to obtain the updated appearance features; and the motion state of the detected target is estimated using the NSAKalman filtering method based on the detection box data and detection confidence to obtain the predicted trajectory state data of the detected target.

[0068] In some embodiments, the Exponential Moving Average (EMA) strategy can smooth appearance features. Instead of directly replacing old appearance features with new ones, it gradually updates the appearance features in the appearance feature library according to a smoothing coefficient. The smoothing coefficient can adjust the contribution of old and new appearance features to the current appearance feature library, and it can be adaptively adjusted so that the update intensity can be dynamically determined according to the quality of the new features or the drastic change in the appearance of the detected target.

[0069] It should be noted that the observation noise of traditional Kalman filtering is fixed, but in practice, the reliability of detection boxes varies. High-confidence detection boxes are more reliable, while low-confidence detection boxes may be inaccurate or background noise. Therefore, the noise-adaptive NSA Kalman filtering method can calculate the observation noise based on the detection confidence of the detection box, so that the observation noise and detection confidence are dynamically correlated. This can improve the robustness of motion state estimation of the detected target and improve the accuracy of predicted trajectory state data.

[0070] In some embodiments, the NSA Kalman filtering method can calculate the corresponding adaptive observation noise based on the detection confidence, and combine the adaptive observation noise and the detection box data to estimate the motion state of the detected target, thereby obtaining the predicted trajectory state data of the detected target. The predicted trajectory state data may include the motion speed, position, etc. of the detected target.

[0071] It should be noted that the performance of the tracking module in this application benefits from the higher quality feature base provided by the front-end AFPN and LSK-Net modules. Under this premise, the appearance feature update strategy using EMA can more effectively accumulate stable and discriminative historical appearance information, reducing ID switching. Simultaneously, based on more reliable detection confidence, adaptive filtering methods such as NSA-Kalman can more accurately adjust the motion estimation process, improving robustness to occlusion and noise. These optimization techniques work together to produce performance gains superior to their application in traditional separate models (where the detection and tracking models are separate).

[0072] S1042. Based on the updated appearance features and predicted trajectory state data, cascade matching is performed to obtain the target tracking result.

[0073] In some embodiments, the trajectory matching of the detected target can be performed based on the updated appearance features and predicted trajectory state data, and a cascaded matching strategy can be adopted, such as performing IoU (Intersection over Union) matching first and then appearance matching, so as to obtain the target tracking result. The target tracking result may include the trajectory shape of the detected target, the position of the detected target, the movement speed, etc.

[0074] In some embodiments, the Intersection over Union (IoU) between each detection box and the predicted box (obtained from the predicted trajectory state data) of each video image frame can be calculated. The Hungarian algorithm (also known as the KM algorithm) is used to solve the assignment problem. The matching principle is to minimize the total IoU distance. If the calculated IoU distance is greater than a certain threshold (e.g., 0.7, i.e., IoU is less than 0.3), the match is rejected (i.e., the match fails). For trajectories and detections that do not match after IoU matching, appearance features are used for matching. The detected appearance features are extracted and cosine distances are calculated with the appearance feature library of the trajectory (which stores the most recent associated appearance features of the trajectory, including the updated appearance features). The Hungarian algorithm is used for matching, while setting an appearance distance threshold (e.g., 0.4). Only matches with a distance less than the appearance distance threshold are accepted.

[0075] In some embodiments, successfully matched detection boxes can be assigned the correct ID and the output is the final tracking trajectory. Unmatched detection boxes can be initialized as new trajectories, and unmatched trajectories can be retrieved in subsequent frames.

[0076] Understandably, updating appearance features through the EMA strategy allows the model to dynamically determine the update intensity of appearance features based on the quality of the new appearance features or the drastic change in the appearance of the detected target. This ensures the stability and reliability of the appearance feature library, reduces ID switching, and improves the continuity of trajectory tracking. Furthermore, estimating the motion state of the detected target using the NSA Kalman filtering method on the detection box data and detection confidence dynamically correlates observation noise with detection confidence, thereby improving the robustness of motion state prediction and reducing the impact of detection noise.

[0077] In some embodiments of this application, the use of the EMA strategy to update the appearance features in step S1041 to obtain the updated appearance features can be achieved through the following steps S201 to S202, and each step is described below.

[0078] S201. Create an appearance feature update formula based on appearance features, exponential moving average smoothing coefficient, and historical appearance features determined based on historical video image frames.

[0079] In some embodiments, appearance features refer to a numerical representation that uniquely characterizes and distinguishes the visual appearance of different detection targets. For example, the appearance information (color, texture, shape, clothing, etc.) of the detection target can be encoded into a compact numerical vector, which constitutes the appearance features of the detection target.

[0080] In some embodiments, the appearance features of the detected target can be extracted from multiple consecutive video image frames. Therefore, the historical appearance features determined based on historical video image frames can be considered as a set of appearance features or an appearance feature library.

[0081] In some embodiments, the appearance feature update formula can be expressed by the following formula (1): (1); Where t is greater than 1 and less than or equal to the total number of video frames, and i is an integer greater than 0. This represents the appearance feature corresponding to trajectory segment i from the initial video image frame to the t-th video image frame. This represents the appearance feature corresponding to trajectory segment i from the initial video image frame to the (t-1)th video image frame. Detect the appearance features of the target in the trajectory segment i extracted in the t-th video image frame. The exponential moving average smoothing coefficient has a value range of (0, 1) and can be adaptively adjusted.

[0082] S202. Update the appearance features based on the appearance feature update formula to obtain the updated appearance features.

[0083] In some embodiments, the exponential moving smoothing coefficient, appearance features determined based on historical video image frames, and appearance features of the target detected from the current video image frame can be substituted into the appearance feature update formula for calculation to obtain the updated appearance features.

[0084] Understandably, by creating an appearance feature update formula based on appearance features, exponential motion smoothing coefficient, and appearance features determined based on historical video image frames, and updating appearance features based on the appearance feature update formula, the update intensity of appearance features can be dynamically determined according to the quality of the new appearance features or the drastic change in the appearance of the detected target. This makes the update of the appearance feature library more stable and reliable. Even after the target has experienced brief occlusion or noise interference, it can still be correctly associated with it through high-quality historical features, thereby reducing ID switching.

[0085] In some embodiments of this application, the motion state estimation of the detected target by using the NSA Kalman filtering method on the detection box data and detection confidence in step S1041 to obtain the predicted trajectory state data of the detected target can be achieved by the following steps S301 to S303, and each step will be described below.

[0086] S301. Substitute the detection confidence and the baseline observation noise into the observation noise calculation formula to obtain the adaptive observation noise; and predict the state data of the detected target in the current video image frame based on the state data and motion model of the detected target in the previous video image frame.

[0087] In some embodiments, the calculation formula for observation noise can be expressed by the following formula (2): (2); in, Indicates adaptive observation noise, Indicates the reference observation noise. This indicates the confidence level of the detection.

[0088] In some embodiments, the reference observation noise can be a preset observation noise, which can be a diagonal matrix. The detection confidence and the baseline observation noise can be substituted into formula (2) for calculation to obtain the adaptive observation noise.

[0089] In some embodiments, the state data of the detected target in the previous video image frame may include data such as the position and velocity of the detected target in the previous video image frame. A state vector and an observation vector may be constructed based on the position and velocity data of the detected target in the previous video image frame. Then, an observation model and a state transition model are constructed by combining the motion model. The state transition model is used to predict the state data of the detected target in the current video image frame.

[0090] S302. Update the state data of the detected target based on the adaptive observation noise and the state data of the detected target in the current video image frame to obtain the updated state data.

[0091] In some embodiments, the Kalman gain can be calculated based on adaptive observation noise and the state data of the detected target in the current video image frame. Then, based on the Kalman gain, the observation model, and the state data of the detected target in the current video image frame, the state data of the detected target can be updated to obtain the updated state data.

[0092] S303. Create predicted trajectory status data for the detected target based on the updated status data.

[0093] In some embodiments, the updated state data may include data such as the detection bounding box, position, and velocity of the detected target, and the predicted trajectory state data of the detected target can be created by combining the detection bounding box, position, and velocity data.

[0094] It is understandable that by substituting the detection confidence and the baseline observation noise into the observation noise calculation formula, adaptive observation noise is obtained. Based on the adaptive observation noise and the state data of the detected target in the current video image frame, the state data of the detected target can be updated. This allows the update of the state data of the detected target to rely more on the detection results of the current video image frame, so as to significantly correct the predicted value. This results in a smoother and more stable motion trajectory, reduces tracking jitter caused by false detection, and improves the accuracy of target tracking.

[0095] In some embodiments of this application, the object detection-tracking integration method based on AFPN and LSK-Net provided in this application can be applied to the object detection-tracking integration model based on AFPN and LSK-Net. The training process of the object detection-tracking integration model based on AFPN and LSK-Net may include: S401. Create a joint loss function associated with the target detection loss function, appearance feature recognition loss function, and motion state estimation loss function.

[0096] It should be noted that the target detection loss function is used to calculate the prediction loss of the detection box data, detection confidence, and the category of the detected target; the appearance feature recognition loss function is used to calculate the prediction loss of appearance features; and the motion state estimation loss function is used to calculate the motion state estimation loss of the NSA Kalman filtering method.

[0097] In some embodiments, the joint loss function It can be expressed by the following formula (3): (3); in, Let represent the target detection loss function, the appearance feature recognition loss function, and the motion state estimation loss function, respectively.

[0098] In some embodiments, the object detection loss function can be used to optimize the performance of the object detection model, ensuring that the model can accurately identify objects (such as pedestrians, vehicles, etc.) in an image and precisely locate the bounding boxes of the objects. The object detection loss typically consists of two parts: classification loss and regression loss, used to simultaneously optimize the prediction of the "presence" and "location" of the object.

[0099] In some embodiments, the appearance feature recognition loss function can be used to optimize the appearance feature extraction model of the detected target, ensuring that the features of the same target are as similar as possible in different video image frames, different viewpoints, and different poses, while the features of different detected targets are as distinct as possible. The appearance feature recognition loss function usually combines classification loss and measurement loss, and is a weighted sum of classification loss and measurement loss.

[0100] In some embodiments, the motion state estimation loss function can be used to optimize the motion model of the detected target, enabling the model to more accurately predict the target's position or motion state in subsequent frames. The motion state estimation loss function is typically calculated based on the difference between the predicted and observed values ​​of the detected target's motion state, and can be calculated using mean squared error (MSE).

[0101] S402. Continuously optimize the network parameters of the object detection-tracking integrated model based on AFPN and LSK-Net through the joint loss function until the convergence condition is met, and obtain the trained object detection-tracking integrated model based on AFPN and LSK-Net.

[0102] In some embodiments, the convergence condition can be that the joint loss function value is less than or equal to a preset loss value. Training sample images can be input into the initial object detection-tracking integrated model based on AFPN and LSK-Net. The joint loss value is calculated using the joint loss function, as well as the object detection results, appearance feature extraction results, and running state prediction results predicted by the model. If the joint loss function value is less than the preset loss value, the network parameters of the integrated object detection-tracking model based on AFPN and LSK-Net are continuously adjusted until the joint loss function value calculated during the iteration process is less than or equal to the preset loss value. Then, training can be stopped, and the trained object detection-tracking integrated model based on AFPN and LSK-Net is obtained.

[0103] Understandably, the joint loss function integrates the optimization objectives of detection, appearance features, and motion model. By simultaneously minimizing these three losses, it ensures the accuracy of the AFPN and LSK-Net-based integrated target detection-tracking model in target detection, the discriminativeness and consistency of target appearance features, and the accuracy of motion prediction. This allows the trained AFPN and LSK-Net-based integrated target detection-tracking model to simultaneously address the three core questions of "where is the target," "who is the target," and "where will the target go," reducing issues such as identity switching, missed detections, and false detections, thereby improving the overall performance of target detection and tracking.

[0104] In this embodiment, the video image frames to be processed are input into the CSPDarknet backbone network to obtain feature maps of multiple scales corresponding to the video image frames. The AFPN is used to fuse the feature maps of multiple scales to obtain a first feature map. The spatial attention mechanism of LSK-Net in ELAN is used to adaptively filter and enhance the first feature map to obtain a second feature map, and a target detection result is generated based on the second feature map. The target detection result includes detection box data, detection confidence, and appearance features. Based on the detection box data, detection confidence, and appearance features, state estimation and tracking association are performed on the detected target to obtain the target tracking result. In this process, fusing the feature maps of multiple scales extracted by the backbone network through AFPN can alleviate the gap between high-level and low-level feature semantics, thereby improving the multi-scale target detection capability. The spatial attention mechanism of LSK-Net in ELAN is used to adaptively filter and enhance the first feature map, enabling the model to automatically learn the receptive field size corresponding to the feature map at different spatial locations, thereby dynamically selecting different convolutional kernels for feature extraction, reducing false detections and false negatives, and enhancing the model's adaptability and discrimination ability in complex scenes. Furthermore, this application obtains detection box data, detection confidence, and appearance features through target detection, and can directly perform state estimation and tracking association on the detected target based on the detection box data, detection confidence, and appearance features. This enables end-to-end integrated optimization of target detection and tracking, reduces information redundancy in target detection and tracking, and improves real-time performance.

[0105] This application provides an integrated target detection-tracking system based on AFPN and LSK-Net, such as Figure 2 As shown, the target detection-tracking integrated system 500 based on AFPN and LSK-Net includes a CSPDarknet backbone network 501, AFPN 502, ELAN 503 which includes the LSK-Net spatial attention mechanism, and an improved DeepSORT network 504 based on the EMA strategy and NSA Kalman filtering.

[0106] The CSPDarknet backbone network 501 is used to extract features from the video image frames to be processed, and obtain feature maps of multiple scales corresponding to the video image frames. AFPN 502 is used to fuse feature maps from multiple scales to obtain the first feature map; ELAN 503, which incorporates the LSK-Net spatial attention mechanism, is used to adaptively filter and enhance the first feature map to obtain the second feature map, and to generate the object detection result based on the second feature map; the object detection result includes detection box data, detection confidence, and appearance features; An improved DeepSORT network 504 is used to perform state estimation and tracking association of detected targets based on detection box data, detection confidence, and appearance features, so as to obtain target tracking results.

[0107] The target detection-tracking integrated method and system based on AFPN and LSK-Net provided in this application can be applied to vehicle speed measurement scenarios. By improving the accuracy and real-time performance of target detection and tracking, the system can better detect and track vehicles being measured, thereby improving the accuracy of vehicle speed measurement.

[0108] The description of the above system embodiments is similar to that of the above method embodiments, and has the same beneficial effects as the method embodiments. For technical details not disclosed in the system embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0109] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.

[0110] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be processed by software stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.

[0111] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0112] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.

Claims

1. A target detection-tracking integrated method based on AFPN and LSK-Net, characterized in that, include: The video image frame to be processed is input into the CSPDarknet backbone network to obtain feature maps of multiple scales corresponding to the video image frame; The first feature map is obtained by fusing the feature maps of the multiple scales using AFPN; The first feature map is adaptively filtered and enhanced using the spatial attention mechanism of LSK-Net in ELAN to obtain the second feature map, and the target detection result is generated based on the second feature map. The target detection results include detection box data, detection confidence, and appearance features; Based on the detection box data, the detection confidence level, and the appearance features, the state estimation and tracking association of the detected target are performed to obtain the target tracking result.

2. The method according to claim 1, characterized in that, The fusion of the feature maps at multiple scales to obtain the first feature map includes: The feature maps at the multiple scales are aligned by resolution to obtain multiple first reference feature maps with the same spatial scale. Generate a first spatial weight map corresponding to each spatial scale for multiple first reference feature maps; The first reference feature maps are weighted and summed according to their respective first spatial weight maps to obtain the first feature map.

3. The method according to claim 1, characterized in that, The adaptive filtering and feature enhancement of the first feature map to obtain the second feature map includes: The first convolutional kernel is decoupled to obtain multiple second convolutional kernels; each second convolutional kernel has a different size and dilation rate, and the size of the first convolutional kernel is larger than the size of each second convolutional kernel; The first feature map is convolved using the plurality of second convolution kernels to obtain a plurality of second reference feature maps; A third reference feature map is obtained by channel fusion of spatial feature vectors of the multiple second reference feature maps using a 1×1 convolution kernel. The third reference feature map is subjected to max pooling and average pooling respectively, and then converted into multiple reference spatial attention feature maps through convolution operations; the number of the reference spatial attention feature maps is the same as the number of the second reference feature maps. The Sigmoid activation function is used to perform a non-linear transformation on the attention feature map of each reference space to generate a second space weight map. The second spatial weight map and the plurality of second reference feature maps are weighted and summed, and then fused through a convolutional layer to obtain the target spatial attention features; The second feature map is obtained by multiplying the target space attention features and the first feature map element by element.

4. The method according to claim 1, characterized in that, The process of estimating the state of the detected target and associating it with tracking includes: The appearance features are updated using the EMA strategy to obtain the updated appearance features; and the motion state of the detected target is estimated using the NSA Kalman filtering method based on the detection box data and the detection confidence to obtain the predicted trajectory state data of the detected target. The target tracking result is obtained by cascading matching based on the updated appearance features and the predicted trajectory state data.

5. The method according to claim 4, characterized in that, The step of updating the appearance feature using the EMA strategy to obtain the updated appearance feature includes: An appearance feature update formula is created based on the appearance features, the exponential moving average smoothing coefficient, and the historical appearance features determined based on historical video image frames. The appearance feature is updated based on the appearance feature update formula to obtain the updated appearance feature.

6. The method according to claim 5, characterized in that, The formula for updating the appearance feature is expressed as follows: ； Where t is greater than 1 and less than or equal to the total number of video frames, and i is an integer greater than 0. This represents the appearance feature corresponding to trajectory segment i from the initial video image frame to the t-th video image frame. This represents the appearance feature corresponding to trajectory segment i from the initial video image frame to the (t-1)th video image frame. Exponential moving average smoothing coefficient, The appearance features of the target are detected in the trajectory segment i extracted from the t-th video image frame.

7. The method according to claim 4, characterized in that, The step of estimating the motion state of the detected target using the NSA Kalman filtering method on the detection box data and the detection confidence to obtain the predicted trajectory state data of the detected target includes: The detection confidence and baseline observation noise are substituted into the observation noise calculation formula to obtain the adaptive observation noise; and the state data of the detected target in the current video image frame is predicted based on the state data and motion model of the detected target in the previous video image frame. The state data of the detected target is updated based on the adaptive observation noise and the state data of the detected target in the current video image frame to obtain the updated state data; The predicted trajectory state data of the detected target is created based on the updated state data.

8. The method according to claim 7, characterized in that, The formula for calculating the observation noise is expressed as follows: ； in, Indicates adaptive observation noise, Indicates the reference observation noise. This indicates the confidence level of the detection.

9. The method according to any one of claims 4 to 8, characterized in that, The method is applied to an integrated object detection-tracking model based on AFPN and LSK-Net. The training process of the integrated object detection-tracking model based on AFPN and LSK-Net includes: Create a joint loss function associated with the target detection loss function, the appearance feature recognition loss function, and the motion state estimation loss function; the target detection loss function is used to calculate the prediction loss of the detection box data, the detection confidence, and the category of the detected target; the appearance feature recognition loss function is used to calculate the prediction loss of the appearance features; and the motion state estimation loss function is used to calculate the motion state estimation loss of the NSA Kalman filtering method. The network parameters of the object detection-tracking integrated model based on AFPN and LSK-Net are continuously optimized through the joint loss function until the convergence condition is met, thus obtaining the trained object detection-tracking integrated model based on AFPN and LSK-Net.

10. A target detection-tracking integrated system based on AFPN and LSK-Net, comprising a CSPDarknet backbone network, AFPN, an ELAN incorporating the LSK-Net spatial attention mechanism, and an improved DeepSORT network based on EMA strategy and NSA Kalman filtering, characterized in that, The CSPDarknet backbone network is used to extract features from the video image frames to be processed, and obtain feature maps of multiple scales corresponding to the video image frames. The AFPN is used to fuse the feature maps of the multiple scales to obtain a first feature map; The ELAN, which includes the LSK-Net spatial attention mechanism, is used to adaptively filter and enhance the first feature map to obtain a second feature map, and to generate a target detection result based on the second feature map; the target detection result includes detection box data, detection confidence, and appearance features; The improved DeepSORT network is used to perform state estimation and tracking association on the detected target based on the detection box data, the detection confidence, and the appearance features, so as to obtain the target tracking result.

Images