A Vehicle Multi-Object Tracking Method Optimized by Cross-Frame and Local Feature Strategies

Abstract

The present invention discloses a vehicle multi-target tracking method with cross-border head and local feature strategy optimization, which relates to the field of intelligent video surveillance and security technology. The present invention has the following steps: adjusting the YOLOv7 model detection anchor frame and the height and depth of the network to obtain a detection frame based on the traffic scene picture inference and give a confidence score for the detection frame, and the position of the vehicle can be known according to the coordinate position of the detection frame; tracking the target by discriminating the occlusion module, motion estimation, association matching and state update. The present invention obtains a vehicle multi-target tracking method by proposing local feature re-identification. Local feature re-identification utilizes the multiple attributes of the vehicle to enhance the extraction of the vehicle's visual features, and specifically re-identifies the local features in the same vehicle, such as re-identifying local objects such as the annual inspection mark and tissue box in the same vehicle, thereby achieving local tracking of the vehicle, and then tracking the target vehicle through cross-border head.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent video surveillance and security, and in particular relates to a vehicle multi-target tracking method with cross-lens and local feature strategy optimization. Background Art

[0002] MOT is particularly important in the field of video surveillance. It aims to detect and estimate the spatiotemporal trajectory of targets in video streams. Multi-target tracking and traffic flow monitoring of vehicles play a vital role in traffic security applications.

[0003] Multi-object tracking (MOT) typically involves two parts: target detection and tracking. The tracking part consists of three steps: state estimation, data association, and target localization. Currently, most detection-based MOT methods are based on SORT and JDE methods. While these methods have achieved good results in MOT algorithms, they still have the following drawbacks:

[0004] (1) The SORT-based method uses Kalman filtering as a motion state estimation model to predict the motion estimation frame of the next frame, associate it with the detection frame of the next frame obtained by the detection model, and use it to predict the trajectory state in closed-loop or missed detection situations. Since the Kalman filter is linear, while the real scene is nonlinear and the tracked target is easily affected by occlusion and motion speed, the width and size of the estimated frame are inaccurate. Similar to the SORT-based IOU method, it tends to track the quality of the predicted frame. Therefore, in complex real-world scenes, due to the movement of the camera, the predicted frame will get an incorrect position, and the low overlap rate between the detection frame and the predicted frame will lead to a decrease in tracking performance.

[0005] (2) Due to the vehicle's speed, camera resolution, and shooting angle, it is usually impossible to obtain high-quality vehicle images. When vehicle recognition fails, vehicle re-identification becomes a very important alternative technology. Current re-identification is affected by low image resolution, human occlusion, perspective, posture changes, lighting changes, visual ambiguity, etc., resulting in increased differences within the same ID and reduced differences between different IDs, resulting in the loss of vehicle target tracking. Summary of the Invention

[0006] The purpose of the present invention is to provide a vehicle multi-target tracking method with cross-lens and local feature strategy optimization to solve the technical problems raised in the above background technology.

[0007] To solve the above technical problems, the present invention is achieved through the following technical solutions:

[0008] The present invention is a vehicle multi-target tracking method based on cross-lens and local feature strategy optimization, which includes the following steps:

[0009] Adjust the height and depth of the yolov7 model detection anchor frame and the network to infer the detection frame based on the traffic scene image and give the confidence score of the detection frame. The position of the vehicle can be known according to the coordinate position of the detection frame;

[0010] Track the target through occlusion detection module, motion estimation, association matching and state update;

[0011] Track vehicles across multiple shots with cross-shot tracking;

[0012] Count the IDs that have been successfully tracked, then set a time threshold to count the number of vehicles that pass within the specified time, monitor the vehicle flow in the traffic scene, and provide timely feedback.

[0013] Furthermore, the coordinate position of the detection frame is obtained as follows:

[0014]

[0015] Among them, Dect_Model represents the yolov7 detection model, Represents the detection result of the i-th target in the t-th frame, x, y, w, h represent the position of the detection box, x, y are the coordinates of the upper left corner of the box, w, h represent the length and width of the box; confidence represents the confidence score of the box.

[0016] Furthermore, the occlusion module identifies the visualization degree of the current object by sampling the image region of each object and inputting it into the Yolov7 detection network.

[0017] Furthermore, local feature extraction takes the reference frame and the current frame as input, generates visual features through a weight-sharing backbone, and then uses the feature interaction model to establish pixel correspondence between the two frames, and then generates local features for each object to enhance the robustness of the target appearance features.

[0018] Furthermore, the motion estimation aligns the same target in adjacent frames by adding information on the detection noise scale that is ignored during ECC camera motion compensation Kalman filter detection. The Kalman filter is used to learn the target operation state in the t-1th frame, and the target detection frame of the next frame is estimated, recorded as the initial target frame (x, y, w, h). Camera compensation is added on the basis of the initial target frame.

[0019] Furthermore, the camera motion compensation process first uses the ECC algorithm to generate a 3×3 homography matrix I. Through homography association, the target of the subsequent frame image is aligned with the target of the previous frame image to improve the tracking accuracy. The calculation formula of the homography matrix I is as follows:

[0020]

[0021]

[0022] in, represents the i-th target image in the t-1-th frame, Represents the i-th target image in the t-th frame, and the calculated x and y are the coordinate information of the upper left and lower right corners of the target box.

[0023] Furthermore, the association matching steps are as follows:

[0024] Using both appearance and motion information to solve the association assignment problem;

[0025] Use generalized IOU as the motion cost matrix and compare the detection box Dtc And the target prediction box target Generalized IOU distance;

[0026] The local reference target is propagated to the current frame as the target prior, and then the target prior features are fused with the visual features and sent to the local feature detection head to obtain the local tracking target of the vehicle.

[0027] Furthermore, the motion cost matrix D uses the weighted sum of appearance and motion information, and is calculated as follows:

[0028] D=λD a +(1-λ)D m

[0029] Among them, the weight factor λ is 0.95, D a Indicates appearance information, D m Represents motion information, and determines the matched vehicle ID and the unmatched vehicle ID through the distance function;

[0030] Under the constraints of the spatiotemporal range, targets within the IOU distance range are still tracked to improve the detection rate of the tracked targets. The generalized IOU function formula is as follows:

[0031]

[0032] Use local feature reid for matching, based on the similarity cosine distance D cos For the local feature to be re-identified frame F part and vehicle local feature base F all To make a judgment, the calculation formula is as follows:

[0033]

[0034] Furthermore, the state update adopts the exponential moving average method to update the appearance state of the i-th track at the t-th frame. The EMA update strategy not only improves the matching quality but also reduces the time consumption. The formula is as follows:

[0035]

[0036] in, represents the state of the tth frame of the i-th trajectory, Indicates the current appearance state;

[0037] The tracking state of the target object in the current frame is judged and updated. If the target state is judged as state_delete within 30 consecutive frames, the tracking is considered lost.

[0038] Furthermore, cross-shot tracking includes offline tracking and real-time online tracking;

[0039] The offline tracking is to generate the target's motion trajectory from multiple shots and achieve cross-shot target tracking by matching the trajectory to the trajectory;

[0040] The real-time online tracking uses local ReID and dynamic clustering algorithms as motion cost matrices to construct connections between local and target, target and trajectory, and trajectory and trajectory, thereby achieving cross-lens vehicle target tracking.

[0041] The present invention has the following beneficial effects:

[0042] 1. The present invention proposes a method for multi-target vehicle tracking by proposing local feature re-identification. Local feature re-identification utilizes the multiple attributes of a vehicle to enhance the extraction of its visual features. Specifically, it re-identifies local features within the same vehicle. For example, it re-identifies local objects such as annual inspection labels and tissue boxes within the same vehicle, thereby achieving local vehicle tracking and further tracking the target vehicle across multiple lenses.

[0043] 2. The present invention proposes a trajectory dynamic clustering algorithm based on spatiotemporal constraints. The spatiotemporal constraints model the spatiotemporal relationship between related cameras through the trajectory clustering algorithm, dynamically cluster the cross-track features of the target vehicle to constrain the cross-shot matching of the vehicle, and enhance the cross-shot vehicle tracking capability.

[0044] 3. The present invention estimates the single-frame motion trajectory of the vehicle by introducing camera motion compensation and occlusion judgment methods; camera motion can best capture local motion in an uncontrolled environment, and the camera's sensitivity to moving parts can well compensate for the estimation of nonlinear or variable speed motion of the target vehicle; the occlusion module performs local feature re-identification (ReID) extraction on the features under the occlusion module, and improves the accuracy of the estimation through the local ReID method.

[0045] 4. The present invention proposes a three-stage association matching strategy. In the first and second stages, weighted appearance motion information and a generalized IOU distance function are used to construct a motion cost matrix between detection and trajectory. In the third stage, a local feature matching method is introduced to construct a motion cost matrix between local and trajectory, thereby improving the accuracy of the target to be tracked and reducing the possibility of tracking the wrong target. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0047] Figure 1 This is a flow chart of the MOT network framework of the present invention;

[0048] Figure 2 A schematic diagram of the initialization state of the present invention is provided. DETAILED DESCRIPTION

[0049] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0050] See also Figure 1 and Figure 2 As shown, the present invention is a vehicle multi-target tracking method with cross-lens and local feature strategy optimization.

[0051] 1. Target detection part

[0052] YOLOv7 uses module reparameterization and dynamic label allocation strategies to surpass currently known algorithms in both speed and accuracy. This paper targets vehicle detection in vehicle scenes, calls the existing yolov7 pre-trained model, and fine-tunes the detection anchor frame and the height and depth of the network to achieve the best detection effect while ensuring that the model occupies as little memory as possible. Given a traffic scene image, the detection frame is obtained through inference by the detection model and the confidence score of the detection frame is given. The position of the vehicle can be determined based on the coordinate position of the detection frame. The detection result is shown in formula (1):

[0053]

[0054] Among them, Dect_Model represents the yolov7 detection model, Indicates the detection result of the i-th object in the t-th frame. x, y, w, and h are the position of the detection box. x and y are the coordinates of the upper left corner of the box, and w and h are the height and width of the box. confidence is the confidence score of the box.

[0055] 2. Target tracking part

[0056] Step 1: Identify occlusion modules

[0057] The occlusion module uses the detection bounding box B of multiple targets r ={b1, b2, ..., b N}, where N represents the number of targets. The visualization degree of the current target is identified by sampling the image area of each target and inputting it into the Yolov7 detection network, and the occlusion of the target at each moment is identified.

[0058] Local feature extraction takes the reference frame and the current frame as input, generates visual features through a weight-sharing backbone (using the same model weights through two identical networks to ensure process consistency), and then uses a feature interaction model to establish pixel correspondence between the two frames by comparing the pixel-level distance between frames. In order to improve the accuracy of the algorithm, the image area of the target is Gaussian filtered and interpolated to 128×96, and sent to the backbone network to generate local features for each object, enhancing the robustness of the target appearance features. Robustness refers to the generalization ability, which describes that each target has a relatively good detection effect.

[0059] In this embodiment, the backbone network is an existing network, specifically a Resnet network, which is an existing mature technology.

[0060] Step 2: Motion Estimation

[0061] Since the Kalman filter is suitable for unobstructed scenes and objects moving at a constant speed, it ignores the information on the detection noise scale. By adding ECC camera motion compensation and aligning the same target in adjacent frames, it makes up for the shortcomings of Kalman estimation for objects with variable speed.

[0062] First, a Kalman filter is used to learn the target's motion state in frame t-1, and the target detection frame for the next frame is estimated, recorded as the initial target frame (x, y, w, h). Camera compensation is then applied to the initial target frame. The camera motion compensation process first uses the existing ECC algorithm to generate a 3×3 homography matrix I. Then, an image registration function is used to construct a homography association, aligning the target in the subsequent frame with the target in the previous frame, thereby improving tracking accuracy.

[0063] In this embodiment, the homography association is realized by constructing and implementing the image registration function in the prior art OpenCV.

[0064] The calculation process is shown in formula (2):

[0065]

[0066]

[0067] in, represents the i-th target image in the t-1-th frame, Represents the i-th target image in the t-th frame, and the calculated x and y are the coordinate information of the upper left and lower right corners of the target box.

[0068] After obtaining the target frame, the current state is initialized. The pseudo code process is shown in Figure (2).

[0069] Step 3: Association matching stage

[0070] The present invention proposes a three-stage matching strategy. In the first stage, both appearance and motion information are used to solve the allocation problem. The motion cost matrix D uses the weighted sum of appearance and motion information, as shown in formula (3):

[0071] D=λD a +(1-λ)D m (3)

[0072] Among them, the weight factor λ is 0.95, D a Indicates appearance information, D m Represents motion information, and the distance function is used to determine the matched vehicle IDs and unmatched vehicle IDs. Because the unmatched IDs and the unmatched vehicle IDs of the previous frame contain IDs with low confidence but can be detected.

[0073] The second stage uses the generalized IOU as the motion cost matrix and compares the detection box Dtc And the target prediction box target The generalized IOU distance of , under the constraints of the spatiotemporal range, targets within the IOU distance range are still tracked, further improving the detection rate of the tracked target. The generalized IOU function is shown in formula (4):

[0074]

[0075] In the third stage, the local reference target is propagated to the current frame as the target prior, and then the target prior features are fused with the visual features and sent to the local feature detection head to obtain the local tracking target of the vehicle. Finally, the local feature reid is used for matching. According to the similarity cosine distance D cosFor the local feature to be re-identified frame F part and vehicle local feature base F all The judgment process is shown in formula (5):

[0076]

[0077] Step 4: Status Update Phase

[0078] The EMA update strategy uses the exponential moving average method to update the appearance state of the i-th track at the t-th frame. The EMA update strategy not only improves the matching quality but also reduces the time consumption, as shown in Equation (6):

[0079]

[0080] in, represents the state of the tth frame of the i-th trajectory, Represents the current appearance state, α is a hyperparameter, usually 0.8.

[0081] In addition, the tracking state of the target object in the current frame is judged and updated. If the target state is judged as state_delete within 30 consecutive frames, it is considered that tracking is lost.

[0082] 3. Cross-camera tracking

[0083] Cross-shot tracking includes two mechanisms: offline tracking and real-time online tracking. Offline tracking generates target motion trajectories from multiple shots and achieves cross-shot target tracking through trajectory-to-trajectory matching. Real-time online tracking uses local ReID and a dynamic clustering algorithm as a motion cost matrix to construct connections between local parts and targets, targets and trajectories, and trajectories, thereby achieving cross-shot vehicle target tracking.

[0084] 4. Traffic flow monitoring part

[0085] First, count the IDs that have been successfully tracked, then set a time threshold, count the number of vehicles that pass within the specified time, monitor the vehicle flow in the traffic scene, and provide timely feedback.

[0086] This embodiment achieves the following benefits when used:

[0087] First, we propose a method for multi-target vehicle tracking using local feature re-identification. Local feature re-identification leverages multiple vehicle attributes to enhance the extraction of visual features. Specifically, it re-identifies local features within a vehicle. For example, it re-identifies local objects such as annual inspection labels and tissue boxes within the same vehicle. This enables local vehicle tracking and further allows for cross-camera tracking of the target vehicle.

[0088] Second, a trajectory dynamic clustering algorithm based on spatiotemporal constraints is proposed. The spatiotemporal constraints model the spatiotemporal relationship between related cameras through the trajectory clustering algorithm, dynamically cluster the cross-track features of the target vehicle to constrain the cross-shot matching of the vehicle, and enhance the cross-shot vehicle tracking capability.

[0089] Third, to improve estimation accuracy, we introduce camera motion compensation and occlusion detection methods to estimate the vehicle's single-frame trajectory. Camera motion is best able to capture local motion in uncontrolled environments, and the camera's sensitivity to moving parts effectively compensates for the estimation of nonlinear or variable speed motion of the target vehicle. The occlusion module performs local feature re-identification (ReID) on the features under occlusion, improving estimation accuracy through local ReID.

[0090] Fourth, to increase the number of target vehicles tracked, a three-stage correlation matching strategy is proposed. In the first and second stages, weighted appearance-motion information and a generalized IoU distance function are used to construct a motion cost matrix between detections and trajectories. In the third stage, a local feature matching method is introduced to construct a motion cost matrix between local features and trajectories, improving the accuracy of the target to be tracked and reducing the possibility of tracking the wrong target.

[0091] Throughout this specification, references to terms such as "one embodiment," "example," or "specific example" indicate that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0092] The preferred embodiments of the present invention disclosed above are intended only to help illustrate the present invention. The preferred embodiments do not describe all details in detail, nor do they limit the present invention to the specific embodiments described. Obviously, many modifications and variations are possible based on the contents of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the present invention, thereby enabling those skilled in the art to better understand and utilize the present invention. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A vehicle multi-target tracking method based on cross-lens and local feature strategy optimization, characterized in that: Here are the steps: Adjust the height and depth of the Yolov7 model detection anchor frame and the network. Infer the detection frame based on the traffic scene image and give the confidence score of the detection frame. The position of the vehicle can be known based on the coordinate position of the detection frame. Track the target through occlusion detection module, motion estimation, association matching and state update; Track vehicles across multiple shots with cross-shot tracking; Count the IDs that have been successfully tracked, set time thresholds, count the number of vehicles that pass within the specified time, monitor vehicle flow in traffic scenarios, and provide timely feedback; The motion estimation aligns the same target in adjacent frames by adding information on the detection noise scale that is ignored during ECC camera motion compensation Kalman filter detection. The Kalman filter is used to learn the target motion state in the t-1th frame to obtain the target detection frame of the next frame, which is recorded as the initial target frame (x, y, w, h). Camera compensation is added to the initial target frame. The association matching steps are as follows: Using both appearance and motion information to solve the association assignment problem; Use generalized IOU as the motion cost matrix and compare the detection box Dtc And the target prediction box target Generalized IOU distance; The local reference target is propagated to the current frame as the target prior, and then the target prior features are fused with the visual features and sent to the local feature detection head to obtain the local tracking target of the vehicle; The motion cost matrix D uses the weighted sum of appearance and motion information and is calculated as follows: D=λD a +(1-λ)D m Among them, the weight factor λ is 0.95, D a Indicates appearance information, D m Represents motion information, and determines the matched vehicle ID and the unmatched vehicle ID through the distance function; Under the constraints of the spatiotemporal range, targets within the IOU distance range are still tracked to improve the detection rate of the tracked targets. The generalized IOU function formula is as follows: Use local feature reid for matching, based on the similarity cosine distance D cos For the local feature to be re-identified frame F part and vehicle local feature base F all To make a judgment, the calculation formula is as follows: The state update adopts the exponential moving average method to update the appearance state of the i-th track at the t-th frame. The EMA update strategy not only improves the matching quality but also reduces the time consumption. The formula is as follows: in, represents the state of the tth frame of the i-th trajectory, represents the current appearance state, α is a hyperparameter; The tracking state of the target object in the current frame is judged and updated. If the target state is judged as state_delete within 30 consecutive frames, the tracking is considered lost.

2. The vehicle multi-target tracking method with cross-lens and local feature strategy optimization according to claim 1, characterized in that: The formula for obtaining the coordinate position of the detection frame is as follows: Among them, Dect_Model represents the yolov7 detection model, Indicates the detection result of the i-th target in the t-th frame, x, y, w, h indicate the position of the detection box, x, y are the coordinates of the upper left corner of the box, w, h indicate the length and width of the box; confidence indicates the confidence score of the box.

3. The vehicle multi-target tracking method with cross-lens and local feature strategy optimization according to claim 1, characterized in that: The occlusion module identifies the degree of visualization of the current object by sampling the image region of each object and inputting it into the Yolov7 detection network.

4. The vehicle multi-target tracking method with cross-lens and local feature strategy optimization according to claim 3, characterized in that: Local feature extraction takes the reference frame and the current frame as input, generates visual features through a weight-sharing backbone, and then uses the feature interaction model to establish pixel correspondence between the two frames to generate local features for each object and enhance the robustness of the target appearance features.

5. The vehicle multi-target tracking method with cross-lens and local feature strategy optimization according to claim 1, characterized in that: The camera motion compensation process first uses the ECC algorithm to generate a 3×3 homography matrix I. Through homography association, the target of the next frame image is aligned with the target of the previous frame image to improve the tracking accuracy. The calculation formula of the homography matrix I is as follows: in, represents the i-th target image in the t-1-th frame, Represents the i-th target image in the t-th frame, and the calculated x and y are the coordinate information of the upper left and lower right corners of the target box.

6. The vehicle multi-target tracking method with cross-lens and local feature strategy optimization according to claim 1, characterized in that: Cross-lens tracking includes offline tracking and real-time online tracking; The offline tracking is to generate the target's motion trajectory from multiple shots and achieve cross-shot target tracking by matching the trajectory to the trajectory; The real-time online tracking uses local ReID and dynamic clustering algorithms as motion cost matrices to construct connections between local and target, target and trajectory, and trajectory and trajectory, thereby achieving cross-lens vehicle target tracking.

Images