A highway cross-camera multi-vehicle tracking method and system combining multiple models

Through the fusion model of mobile window self-attention and multi-ReID feature, combined with reordering technology, the problem of cross-camera multi-vehicle tracking on the highway is solved, and more accurate and stable vehicle detection and trajectory tracking is achieved, and the highway environment is overcome. Difficulty in detection and tracking.

CN117218580BActive Publication Date: 2024-11-22HANGZHOU PIXEL TECH CO LTD
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202311180228.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-09-13
Publication Date
2024-11-22
Estimated Expiration
2043-09-13

AI Technical Summary

Technical Problem

The existing cross-camera multi-vehicle tracking technology is difficult to work effectively in highway environments. It is mainly due to the fast vehicle speed, long roads, long camera distance, light changes and jitter, which lead to difficulties in vehicle detection and tracking, especially small vehicles being blocked and false positives. The noise problem is serious.

Method used

The vehicle object detection model is optimized by mobile window self-attention, combined with multi-ReID feature fusion model and reordering technology, and the multi-camera collaborative processing is used to achieve accurate tracking of vehicle trajectories across cameras. The specific steps include: the vehicle object detection model uses the mobile window self-attention to extract multi-scale features, the multi-ReID feature extraction fusion model uses different bockbones to fuse the vehicle's appearance features, and uses temporary storage-fusion strategies and post-processing techniques to reduce false positives.

Benefits of technology

在高速公路环境下显著提高了车辆目标检测的准确性和重识别率,减少了车辆轨迹缺失和假阳性噪声,实现了更精准的跨摄像头多车辆跟踪。

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117218580B_ABST
    Figure CN117218580B_ABST
Patent Text Reader

Abstract

The present invention provides a highway cross-camera multi-vehicle tracking method and application combined with multiple models, performs vehicle target detection of multi-scale features of moving window self-attention on target vehicle images of tracking videos, extracts multi-fusion ReID features for detected vehicles, and performs single-camera vehicle tracking trajectories of a temporary storage-fusion strategy based on the box of target detection and the ReID of feature extraction, uses multiple post-processing technologies to filter the single-camera vehicle tracking trajectory and associate and match the single-camera vehicle tracking trajectories between multiple cameras to form a final cross-camera vehicle tracking trajectory. The highway cross-camera multi-vehicle tracking method combined with multiple models can realize accurate cross-camera vehicle tracking on highways, and reduce the occurrence of tracking failures caused by the change of ID identities of multiple vehicles tracked across cameras.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of vehicle tracking, and in particular to a method and system for tracking multiple vehicles across cameras on a highway using moving window self-attention, multi-fusion ReID features and reordering combined with multiple models. Background Art

[0002] Cross-camera multi-vehicle tracking has always been an important research direction in visual tasks and has also attracted extensive research attention in real life. It has been widely used in areas including urban traffic flow statistics, traffic road monitoring, and unmanned driving. Most of the existing cross-camera multi-vehicle tracking technologies are mainly used to track multi-vehicle targets on urban streets and intersections, and there is little research on cross-camera multi-vehicle tracking on highways.

[0003] Compared with traditional cross-camera multi-vehicle tracking in urban blocks, cross-camera multi-vehicle tracking on highways is more challenging. The challenges are mainly as follows:

[0004] 1) Vehicles on highways are faster than those in urban areas, which makes it difficult for cameras to capture them clearly or the vehicle images captured by cameras are blurry, which brings more difficulties to target detection;

[0005] 2) Highways are longer than city blocks and the camera is higher from the ground, which makes it more difficult to distinguish similar vehicles of the same brand and color. At the same time, there are more large vehicles on highways than in city blocks, and small vehicles are easily blocked, resulting in missed detection of vehicles and loss of tracking trajectories.

[0006] 3) The intensity of lights in highway tunnels varies more significantly, which can easily interfere with vehicle ReID and affect the correct identification of target vehicle features;

[0007] 4) The distance between the tracking camera and the vehicle target on the highway is relatively far and there is camera shaking, which makes it easy for some vehicles to leave the camera's field of view and fail to track the vehicle accurately.

[0008] In summary, the current cross-camera multi-vehicle tracking technology used in urban streets is difficult to be directly applied to the cross-camera multi-vehicle tracking scenario on highways. Summary of the invention

[0009] This solution provides a method and system for tracking multiple vehicles across cameras on highways that combines multiple models. The method uses moving window self-attention to optimize the vehicle target detection model, uses multi-feature ReID fusion to optimize the vehicle target re-identification model, and uses a re-ranking method to optimize the tracked vehicle trajectories to achieve multi-vehicle tracking on highways using multiple cameras.

[0010] To achieve the above objectives, this solution provides a highway cross-camera multi-vehicle tracking method combining multiple models, including the following steps:

[0011] Get tracking video from multiple cameras on the highway;

[0012] The video frame of the tracking video of each camera is input into the pre-trained vehicle target detection model to input the vehicle target detection frame, wherein the vehicle target detection model performs block processing on the input image to obtain multiple image blocks, and the image blocks are subjected to multiple layers of swin-transformer to obtain the feature map corresponding to each layer, and the multiple feature maps are adaptively fused after feature extraction through moving window self-attention to obtain fused features, and the fused features are input into the classification network and the bounding box prediction network to predict the vehicle target detection frame;

[0013] Input the vehicle image corresponding to the vehicle target detection frame into the pre-trained multi-ReID feature extraction and fusion model to output the vehicle appearance features, wherein the multi-ReID feature extraction and fusion model includes a ReID feature extraction module composed of multiple ReIDs of different bonebones and a ReID feature fusion module, the vehicle image is input into the ReID feature extraction module to obtain the ReID features corresponding to different focus points, and all the ReID features are input into the ReID feature fusion module to fuse and obtain the vehicle appearance features;

[0014] Input the vehicle target detection frame and the vehicle appearance features into the multi-target tracking module to obtain the single-camera vehicle tracking trajectories of multiple vehicles in each camera;

[0015] The single-camera vehicle tracking trajectories of multiple cameras are filtered, reordered and clustered to obtain the cross-camera vehicle tracking trajectories under multiple cameras.

[0016] In the second aspect, the present invention provides a highway cross-camera multi-vehicle tracking system combining multiple models, including:

[0017] A vehicle detection module, wherein a pre-trained vehicle target detection model is provided in the vehicle detection model, tracking videos of multiple cameras on the highway are obtained, and video frames of the tracking videos of each camera are input into the pre-trained vehicle target detection model to input a vehicle target detection frame, wherein the vehicle target detection model performs block processing on the input image to obtain multiple image blocks, and the image blocks are subjected to multiple layers of swin-transformer to obtain feature maps corresponding to each layer, and multiple feature maps are adaptively fused after feature extraction through moving window self-attention to obtain fused features, and the fused features are input into the classification network and the border prediction network to predict the vehicle target detection frame;

[0018] A multi-ReID feature extraction and fusion module, wherein a pre-trained multi-ReID feature extraction and fusion model is provided in the multi-ReID feature extraction and fusion module, a vehicle image corresponding to a vehicle target detection frame is input into the pre-trained multi-ReID feature extraction and fusion model to output vehicle appearance features, wherein the multi-ReID feature extraction and fusion model includes a ReID feature extraction module composed of ReIDs of multiple different bonebones and a ReID feature fusion module, a vehicle image is input into the ReID feature extraction module to obtain ReID features corresponding to different focus points, and all ReID features are input into the ReID feature fusion module to be fused to obtain vehicle appearance features;

[0019] A multi-target tracking module, used to input the vehicle target detection frame and the vehicle appearance features into the multi-target tracking module to obtain the single-camera vehicle tracking trajectories of multiple vehicles in each camera;

[0020] The cross-camera correlation trajectory module is used to filter, reorder and cluster the single-camera vehicle tracking trajectories of multiple cameras to obtain the cross-camera vehicle tracking trajectories under multiple cameras.

[0021] In a third aspect, the present application provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute a highway cross-camera multi-vehicle tracking method combined with multiple models.

[0022] In a fourth aspect, the present application scheme provides a readable storage medium, in which a computer program is stored, wherein the computer program includes a program code for controlling a process to execute a process, wherein the process includes a highway cross-camera multi-vehicle tracking method combined with multiple models.

[0023] Compared with the existing technology, this solution has the following characteristics and beneficial effects:

[0024] 1. This scheme uses moving window self-attention to optimize the vehicle target detection model, and adaptively fuses multi-scale features of different window sizes, so that the vehicle target detection model can pay more attention to the relationship between features of different scales, and obtain more refined vehicle target detection results through adaptive fusion, which is suitable for accurately detecting vehicles in low-quality video images on highways.

[0025] 2. This scheme adopts a multi-ReID feature extraction fusion model composed of ReIDs of multiple different bonebones, which can be applied to the processing of large quantities of video image frames across cameras, and obtains a globally refined appearance feature representation through fusion; not only that, the model is trained using a joint loss function of the joint cross entropy loss and the triplet loss, so that the multi-ReID feature extraction fusion model can better identify vehicle appearance features and distinguish similar vehicles with similar appearances.

[0026] 3. This solution uses a temporary storage-fusion strategy to divide vehicles into high-confidence groups and low-confidence groups. Different tracking and matching methods are used for vehicles in different confidence groups. For vehicles that fail to match, the tracking threshold and continuous frame judgment are calculated to reduce the loss of single-camera vehicle tracking trajectories caused by low detection of some vehicles due to occlusion;

[0027] 4. This solution uses multiple post-processing methods such as time filtering, reordering and clustering to post-process the single-camera vehicle tracking trajectory of multiple cameras to obtain a more accurate cross-camera vehicle tracking trajectory. The reordering method not only keeps similar vehicles at a distance from each other, but also ensures that different vehicle trajectories with similarities keep a distance from each other, reducing the integration of some false positive vehicle noise. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a schematic diagram of a baseline flow of a highway cross-camera multi-vehicle tracking method combined with multiple models according to the present solution;

[0029] Figure 2 It is a logical flow diagram of a highway cross-camera multi-vehicle tracking method combined with multiple models according to the present solution;

[0030] Figure 3 This is a schematic diagram of the architecture of the vehicle target detection model;

[0031] Figure 4 It is a schematic diagram of the architecture of the multi-ReID feature extraction fusion model;

[0032] Figure 5 It is a visual diagram of the detection effect of the vehicle target detection model;

[0033] Figure 6 It is a schematic diagram of tracking success and failure caused by ID changes;

[0034] Figure 7 This is a schematic diagram of the challenges of traditional cross-camera multi-vehicle tracking on highways;

[0035] Figure 8 It is a schematic diagram of the cross-camera multi-vehicle tracking trajectory of the highway cross-camera multi-vehicle tracking system combined with multiple models on a real highway;

[0036] Fig. 9 It is a schematic diagram of the hardware structure of an electronic device for implementing a highway cross-camera multi-vehicle tracking method combined with multiple models. DETAILED DESCRIPTION

[0037] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0038] Those skilled in the art should understand that, in the disclosure of the present invention, the orientation or position relationship indicated by the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc. are based on the orientation or position relationship shown in the drawings, which are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the above terms should not be understood as limiting the present invention.

[0039] It is to be understood that the term "one" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of an element may be one, while in another embodiment, the number of the element may be multiple, and the term "one" should not be understood as a limitation on the quantity.

[0040] The challenges of traditional multi-vehicle tracking technology when applied to highways include Figure 7 As shown, Figure 7 As shown in (a) in the figure, the vehicles on the highway are traveling too fast, which causes the vehicles in the image captured by the camera to be blurred. Figure 7 As shown in (b), since the highway road is longer than the city block and the camera is higher from the ground, some similar vehicles of the same brand and color are more difficult to distinguish; Figure 7As shown in (c), the light intensity in the highway tunnel will change and affect the effect of vehicle re-identification; Figure 7 As shown in (d), the distance between the camera and the vehicle is too large. These challenges make it difficult for traditional multi-vehicle tracking technology to meet the actual needs of multi-vehicle tracking when applied to highways.

[0041] The scheme proposes a highway cross-camera multi-vehicle tracking method and system that is particularly suitable for highways and combines multiple models. The video frame of the camera is input into a vehicle target detection model optimized by moving window self-attention, and multi-scale features are extracted by moving window self-attention for prediction by classification network and bounding box prediction network to obtain accurate vehicle target detection frame, thereby improving the accuracy of vehicle target detection in blurred images; the vehicle target detection frame is then batch-transferred into multiple ReID feature extraction models to extract ReID features of different focus points, and then the multiple ReID features are fused to obtain global refined features, thereby improving the accuracy of vehicle re-identification; the vehicle target detection frame obtained by the base vehicle target detection model and the fused ReID features obtained by the ReID feature extraction model are then transmitted to the multi-target tracker to track the trajectory of a single camera vehicle, and the temporary storage-fusion strategy is used to re-track the vehicle trajectory of some low-confidence and unmatched vehicles. Finally, a variety of post-processing technologies are used to match and associate the vehicle trajectories under multiple camera tracking between adjacent cameras to form the final cross-camera multi-vehicle tracking result.

[0042] Embodiment 1

[0043] Figure 1 The baseline flowchart of the highway cross-camera multi-vehicle tracking method combined with multiple models is provided. Figure 2 A logical flow diagram of a highway cross-camera multi-vehicle tracking method combining multiple models of the present solution is provided. The highway cross-camera multi-vehicle tracking method combining multiple models comprises the following steps:

[0044] Get tracking video from multiple cameras on the highway;

[0045] The video frame of the tracking video of each camera is input into the pre-trained vehicle target detection model to input the vehicle target detection frame, wherein the vehicle target detection model performs block processing on the input image to obtain multiple image blocks, and the image blocks are subjected to multiple layers of swin-transformer to obtain the feature map corresponding to each layer, and the multiple feature maps are adaptively fused after feature extraction through moving window self-attention to obtain fused features, and the fused features are input into the classification network and the bounding box prediction network to predict the vehicle target detection frame;

[0046] Input the vehicle image corresponding to the vehicle target detection frame into the pre-trained multi-ReID feature extraction and fusion model to output the vehicle appearance features, wherein the multi-ReID feature extraction and fusion model includes a ReID feature extraction module composed of multiple ReIDs of different bonebones and a ReID feature fusion module, the vehicle image is input into the ReID feature extraction module to obtain the ReID features corresponding to different focus points, and all the ReID features are input into the ReID feature fusion module to fuse and obtain the vehicle appearance features;

[0047] Input the vehicle target detection frame and the vehicle appearance features into the multi-target tracking module to obtain the single-camera vehicle tracking trajectories of multiple vehicles in each camera;

[0048] The single-camera vehicle tracking trajectories of multiple cameras are filtered, reordered and clustered to obtain the cross-camera vehicle tracking trajectories under multiple cameras.

[0049] Specifically, the architecture of the vehicle target detection model involved in this solution is as follows: Figure 3 As shown. This solution crops the video frames of the tracking video of each camera into a uniform specification according to a predetermined size and inputs them into the pre-trained vehicle target detection model to input the vehicle target detection frame. Specifically, the vehicle target detection model outputs the box information of the predicted frame, the corresponding target category class and the detected vehicle image. The vehicle target detection frame is the predicted frame with the target category of "vehicle", and the vehicle image is the area image where the target category is the vehicle target detection frame.

[0050] The vehicle target detection model comprises a multi-layer swin-transformer layer and a moving window self-attention module, a classification network and a frame prediction network, wherein an image input into the vehicle target detection model is processed into blocks to obtain a plurality of image blocks, an image block sequence composed of the image blocks is transmitted to a multi-layer swin-transformer layer arranged in series, each swin-transformer layer recombines the image block sequence according to the window size of the current layer to obtain a feature map of the current layer, each feature map is processed by the moving window self-attention module to obtain a corresponding multi-scale feature, and the multi-scale features are adaptively fused to obtain a fused feature; the fused feature is transmitted to the classification network and the frame prediction network to perform target frame prediction and target category classification operations to obtain a vehicle target detection frame.

[0051] In some embodiments, the vehicle target detection model includes three layers of swin-transformer layers, each swin-transformer layer has a different window size, and the image sequence is recombined according to the predetermined window size after feature extraction in the corresponding swin-transformer layer to obtain feature maps P1, P2, and P3 corresponding to the current swin-transformer layer. Correspondingly, the moving window self-attention module includes multi-head self-attention corresponding to the window size of the three feature maps, and each feature map is extracted and paid attention to within the feature map through multi-head self-attention to obtain the corresponding multi-scale features, and the multi-scale features are adaptively fused to obtain fused features. Specifically, the multi-scale features of different scales are downsampled and fused to form more refined fused features.

[0052] It should be noted that this solution performs adaptive fusion processing on multi-scale features to obtain more refined vehicle features. The formula for adaptive fusion is as follows:

[0053] where w i is the multi-scale feature f of the i-th layer i The weight value of n is the number of feature layers of the fused multi-scale features.

[0054] About the classification network: The classification network uses a fully connected layer to reduce the number of channels of the fused features to N (where N is the number of known categories. In highway target tracking, there are mainly the following target categories: car, truck, bus, others). The classification network is used to obtain the target category class contained in the current area image. Car, truck, and bus all correspond to vehicle target detection boxes.

[0055] Regarding the bounding box prediction network: The predicted bounding box information of the corresponding predicted target is obtained through the corresponding bounding box prediction network. This solution uses NMS and IOU to detect and remove repeated predicted targets, and obtains the box information of the vehicle target detection box and the corresponding class information of the final vehicle target detection.

[0056] Regarding the training of the vehicle target detection model: the model is trained using images marked with target categories, and the training is iterated until the set conditions are met.

[0057] like Figure 5 As shown in Figure 1, this scheme compares the detection results of the traditional target detection model and the vehicle target detection model of this scheme for the same video frame image. Figure 5 (b) shows the detection results of the traditional target detection model. Figure 5(c) in the figure shows the detection result of the vehicle target detection model of this scheme. It can be seen that the vehicle target frame detected by the vehicle target detection model provided by this scheme is more refined.

[0058] Specifically, the architecture of the multi-ReID feature extraction fusion model involved in this solution is as follows: Figure 4 Considering that the large number of cameras across the camera leads to too many video frame images, this solution adopts batch-by-batch transmission into different bockbones in the multi-ReID feature extraction fusion model for learning and fusion of ReID features of different focus points to obtain the overall vehicle appearance features.

[0059] The ReID feature extraction module in the multi-ReID feature extraction fusion model is composed of ReIDs of multiple different bonebones. The vehicle image corresponding to the vehicle target detection frame is transferred into the ReID of different bonebones in the ReID feature extraction module in batches according to the batch size to obtain the ReID features of the corresponding focus points. In some embodiments, the ReID feature extraction module includes ResNet50, ResNet101, ResNeXt101, and Se-ResNet101. In order to make the extracted ReID features more refined, the final stride of the backbone in each ReID is 1 and the standardized ReID features are obtained by passing through the BN layer.

[0060] The ReID feature fusion module fuses the ReID features of different focus points of multiple backbones using the L2 criterion to obtain the vehicle appearance features. In some embodiments, due to the inconsistency of the vehicle image size corresponding to the vehicle target detection frame, the original vehicle image is scaled to 384×384 at this stage, and the scaled image is input into the corresponding ReID backbone for corresponding ReID extraction. The vehicle appearance feature is a 2048-dimensional global refined appearance feature representation f reid , to focus on distinguishing vehicles with more similar appearance.

[0061] In addition, in order to better identify vehicle appearance features and distinguish similar vehicles with similar appearances, this scheme uses a joint cross entropy loss and triplet loss to train a multi-ReID feature extraction fusion model, so that vehicles with similar appearance features are kept at a close distance in the feature space, and vehicles with obviously different appearances are kept at a farther distance.

[0062] Specifically, the multi-ReID feature extraction fusion model uses the joint loss function of the cross entropy loss function and the triple loss function as the loss function:

[0063] L=L entropy +βLtriplet

[0064] Where L is the joint loss function, L entropy is the cross entropy loss function, L triplet is a triplet loss function, β is a hyperparameter (balance weight), which is learned and updated during model training;

[0065]

[0066]

[0067] Among them, y i is the true vehicle label of the i-th instance, is the predicted value of the i-th instance vehicle, is the target box of the i-th instance vehicle, is the positive target box of the i-th instance vehicle, is the negative target box of the i-th instance vehicle and θ is a hyperparameter (boundary).

[0068] In other words, regarding the training of the multi-ReID feature extraction fusion model: the model is trained using images marked with the target category, and the training is iterated until the joint loss function is minimized or the number of iterations is reached.

[0069] After obtaining the vehicle target detection frame and vehicle appearance features, this solution will input the vehicle target detection frame and vehicle appearance features into the multi-target tracking module to obtain the single-camera vehicle tracking trajectory of multiple vehicles in each camera. It should be noted that in order to retain more credible vehicles and avoid the phenomenon that some vehicles are missed (missing) due to the abandonment of vehicles with short-term low confidence due to partial occlusion due to low detection at a certain moment, this solution adopts a temporary storage-fusion strategy to generate a single-camera vehicle tracking trajectory.

[0070] Specifically, in the step of "inputting the vehicle target detection frame and the vehicle appearance features into the multi-target tracking module to obtain the single-camera vehicle tracking trajectories of multiple vehicles in each camera", the vehicles are divided into a high-confidence group and a low-confidence group according to the detection threshold, and the vehicle appearance features of the vehicles in the high-confidence group are similarly matched. If they match, the single-camera vehicle tracking trajectory of the current vehicle is tracked and generated. If they do not match, the unmatched vehicles and the vehicles in the low-confidence group are matched again according to the IOU of the vehicle target detection frame between adjacent video frames. If they match, the single-camera vehicle tracking trajectory of the current vehicle is generated. If they do not match again, the tracking threshold of the unmatched vehicle is calculated. If the tracking threshold is greater than the threshold and the vehicle appears in continuous video frames, the vehicle is re-included in the target tracking set for the next round of tracking.

[0071] The reason why this solution generates single-camera vehicle tracking trajectories for vehicles in a hierarchical manner is to reduce the loss of single-camera vehicle tracking trajectories caused by low detection of some vehicles due to occlusion. The detection threshold of this solution refers to the average confidence of each camera detecting the vehicle target frame, and the calculation formula is as follows:

[0072]

[0073] Where N is the number of cameras across the camera, m k is the number of vehicle images corresponding to the vehicle target box under the kth camera, C k,i is the detection confidence of the vehicle target detection box of the i-th instance under the k-th camera.

[0074] The detection threshold of this scheme is divided into a high detection threshold and a low detection threshold. When the confidence of the vehicle target detection frame of a vehicle is greater than the high detection threshold, the vehicle target detection frame of the vehicle is included in the high confidence group. For the target detection frames of the remaining vehicles between the high detection threshold and the low detection threshold, the vehicle is included in the low confidence group. For the vehicles in the high confidence group, the vehicle appearance features of the vehicle are used for similarity matching. If they match, they are represented as the same vehicle and a single camera vehicle tracking trajectory is generated; if they do not match, the IOU of the unmatched vehicle target detection frame between adjacent video frames is calculated for secondary matching, and the IOU of the vehicle target detection frame of the vehicle in the low confidence group is calculated for secondary matching between adjacent video frames. If they match, the corresponding single camera vehicle tracking trajectory is generated; if the secondary matching still does not match, the tracking threshold of the remaining unmatched vehicle target detection frame is calculated, where the tracking threshold is the center offset value of the vehicle target frame of the next frame calculated based on the tracking frame predicted by the Kalman filter. If the tracking threshold is greater than the threshold and the vehicle does appear in the continuous video frames, the vehicle target detection is re-included in the target tracking set for subsequent tracking.

[0075] In the embodiment of this solution, the tracking threshold T is set e is 0.4, detecting high confidence threshold τ high 0.5, low confidence threshold τ low 0.2, matching threshold 0.5 and cosine distance threshold 0.5.

[0076] After obtaining the single-camera vehicle tracking trajectory, this solution uses a variety of post-processing techniques to match and fuse the single-camera vehicle tracking trajectories of multiple cameras to obtain the cross-camera vehicle tracking trajectory.

[0077] The post-processing techniques of this solution include filtering, reordering and clustering:

[0078] Filtering the single-camera vehicle tracking trajectory refers to: obtaining the passing time of the single-camera vehicle tracking trajectory of the same vehicle passing through the camera, judging the false positive single-camera vehicle tracking trajectory based on the passing time and the camera distribution order, and filtering the false positive single-camera vehicle tracking trajectory.

[0079] In other words, the cameras are distributed in the order of the vehicle's forward direction. If the time when a vehicle passes two adjacent cameras is inconsistent with the order of the cameras, it is considered that the single-camera vehicle tracking trajectory of the vehicle has a false positive. For example, the single-camera vehicle tracking trajectory of a vehicle passes through camera A at 10:00 and camera B at 9:50, and camera B is set in the forward direction relative to camera A. In this case, the single-camera vehicle tracking trajectory has a false positive.

[0080] Specifically, this solution is to initialize the time conflict table of vehicles crossing two cameras, and record the time conflicting single-camera vehicle tracking trajectories and passing time into the time conflict table through the passing time of the single-camera vehicle tracking trajectories of the same vehicle ID passing through the camera. According to the time conflict table, it is determined whether the single-camera vehicle tracking trajectories of vehicles with the same ID passing through two cameras are false positives, that is, due to the unreasonable matching of some single-camera vehicle tracking trajectories caused by the switching of IDs, and the false positive vehicles are removed, further reducing the number of cross-camera trajectory matches and the space for subsequent re-ordering searches. Figure 6 As shown, Figure 6 Figure (a) shows an example of tracking failure across cameras after an ID change. Figure 6 Figure (b) shows an example of successful tracking without ID change.

[0081] Reordering the single-camera vehicle tracking trajectory refers to:

[0082] Based on the vehicle appearance features, the similarity between different single-camera vehicle tracking trajectories under the same camera is calculated, and the single-camera vehicle tracking trajectories with high similarity to the target vehicle are reordered to the front position. According to the similarity between different single-camera vehicle tracking trajectories, a similarity matrix between adjacent cameras is established, and the k complementary nearest neighbor algorithm is used to update the similarity matrix.

[0083] Specifically, the formula for calculating the similarity between different single-camera vehicle tracking trajectories under the same camera based on vehicle appearance features is as follows:

[0084]

[0085] Among them, Sim(T i , T j) is the similarity between the i-th single-camera vehicle tracking trajectory and the j-th single-camera vehicle tracking trajectory, T i refers to the i-th single-camera vehicle tracking trajectory, T j refers to the jth single-camera vehicle tracking trajectory, f i Refers to T i The Re-ID features of the corresponding instance vehicle, A f Refers to the average characteristics of the calculated trajectory, f j Refers to T j The Re-ID features of the corresponding instance vehicle.

[0086] This scheme reorders vehicle matching pairs according to the size of similarity, so that vehicles with similarity to the target vehicle are placed in the front position and different vehicles are kept at a distance.

[0087] The calculation formula for establishing the similarity matrix between adjacent cameras based on the similarity between vehicle tracking trajectories of different single cameras is as follows:

[0088]

[0089] Where S m is the similarity matrix.

[0090] However, since the similarity matrix obtained by reordering can only keep the target vehicle and similar vehicles at a distance from each other, but cannot guarantee the distance between different vehicles, this scheme uses the k complementary nearest neighbor algorithm to refine the similarity matrix, which can ensure that different vehicle trajectories with similarities are kept at a distance from each other. Compared with the traditional k nearest neighbor, it is more accurate and reduces the integration of some false positive vehicle noise.

[0091] The clustering of single-camera vehicle tracking trajectories refers to: using a hierarchical clustering method to cluster the single-camera vehicle tracking trajectories within the same camera after time filtering, and then clustering the single-camera vehicle tracking trajectories between different cameras according to the similarity matrix to obtain cross-camera vehicle tracking trajectories.

[0092] Figure 8 This is a schematic diagram of the cross-camera multi-vehicle tracking trajectory of the highway cross-camera multi-vehicle tracking system combined with multiple models on a real highway. It can be seen that this scheme can accurately obtain the cross-camera vehicle tracking trajectory of multiple vehicles across lenses on the highway.

[0093] Embodiment 2

[0094] This solution provides a highway cross-camera multi-vehicle tracking system combining multiple models, including:

[0095] A vehicle detection module, wherein a pre-trained vehicle target detection model is provided in the vehicle detection model, tracking videos of multiple cameras on the highway are obtained, and video frames of the tracking videos of each camera are input into the pre-trained vehicle target detection model to input a vehicle target detection frame, wherein the vehicle target detection model performs block processing on the input image to obtain multiple image blocks, and the image blocks are subjected to multiple layers of swin-transformer to obtain feature maps corresponding to each layer, and multiple feature maps are adaptively fused after feature extraction through moving window self-attention to obtain fused features, and the fused features are input into the classification network and the border prediction network to predict the vehicle target detection frame;

[0096] A multi-ReID feature extraction and fusion module, wherein a pre-trained multi-ReID feature extraction and fusion model is provided in the multi-ReID feature extraction and fusion module, a vehicle image corresponding to a vehicle target detection frame is input into the pre-trained multi-ReID feature extraction and fusion model to output vehicle appearance features, wherein the multi-ReID feature extraction and fusion model includes a ReID feature extraction module composed of ReIDs of multiple different bonebones and a ReID feature fusion module, a vehicle image is input into the ReID feature extraction module to obtain ReID features corresponding to different focus points, and all ReID features are input into the ReID feature fusion module to be fused to obtain vehicle appearance features;

[0097] A multi-target tracking module, used to input the vehicle target detection frame and the vehicle appearance features into the multi-target tracking module to obtain the single-camera vehicle tracking trajectories of multiple vehicles in each camera;

[0098] The cross-camera correlation trajectory module is used to filter, reorder and cluster the single-camera vehicle tracking trajectories of multiple cameras to obtain the cross-camera vehicle tracking trajectories under multiple cameras.

[0099] For the technical contents involved in the second embodiment, please refer to the technical description of the first embodiment, which will not be elaborated here.

[0100] Embodiment 3

[0101] This embodiment also provides an electronic device, referring to Fig. 9 , including a memory 304 and a processor 302, wherein the memory 304 stores a computer program, and the processor 302 is configured to run the computer program to execute the steps in any of the above-mentioned embodiments of the highway cross-camera multi-vehicle tracking method combined with multiple models.

[0102] Specifically, the processor 302 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0103] Among them, the memory 304 may include a large capacity memory 304 for data or instructions. By way of example and not limitation, the memory 304 may include a hard disk drive (HDD), a floppy disk drive, a solid state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 304 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 304 may be inside or outside the data processing device. In a specific embodiment, the memory 304 is a non-volatile memory. In a specific embodiment, the memory 304 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (Programmable Read-Only Memory, PROM for short), an erasable PROM (Erasable Programmable Read-Only Memory, EPROM for short), an electrically erasable PROM (Electrically Erasable Programmable Read-Only Memory, EEPROM for short), an electrically alterable ROM (Electrically Alterable Read-Only Memory, EAROM for short) or a flash memory (FLASH) or a combination of two or more of these. Under appropriate circumstances, the RAM can be a static random access memory (Static Random-Access Memory, abbreviated as SRAM) or a dynamic random access memory (Dynamic Random Access Memory, abbreviated as DRAM), wherein the DRAM can be a fast page mode dynamic random access memory 304 (Fast Page Mode Dynamic Random Access Memory, abbreviated as FPMDRAM), an extended data output dynamic random access memory (Extended Date Out Dynamic Random Access Memory, abbreviated as EDODRAM), a synchronous dynamic random access memory (Synchronous Dynamic Random-Access Memory, abbreviated as SDRAM), etc.

[0104] The memory 304 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 302 .

[0105] The processor 302 reads and executes the computer program instructions stored in the memory 304 to implement any one of the highway cross-camera multi-vehicle tracking methods combined with multiple models in the above embodiments.

[0106] Optionally, the electronic device may further include a transmission device 306 and an input / output device 308 , wherein the transmission device 306 is connected to the processor 302 , and the input / output device 308 is connected to the processor 302 .

[0107] The transmission device 306 can be used to receive or send data via a network. Specific examples of the above-mentioned network may include a wired or wireless network provided by a communication provider of the electronic device. In one example, the transmission device includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 306 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0108] The input and output device 308 is used to input or output information. In this embodiment, the input information may be videos from multiple cameras on a highway, and the output information may be tracking trajectories.

[0109] Optionally, in this embodiment, the processor 302 may be configured to perform the following steps through a computer program:

[0110] Get tracking video from multiple cameras on the highway;

[0111] The video frame of the tracking video of each camera is input into the pre-trained vehicle target detection model to input the vehicle target detection frame, wherein the vehicle target detection model performs block processing on the input image to obtain multiple image blocks, and the image blocks are subjected to multiple layers of swin-transformer to obtain the feature map corresponding to each layer, and the multiple feature maps are adaptively fused after feature extraction through moving window self-attention to obtain fused features, and the fused features are input into the classification network and the bounding box prediction network to predict the vehicle target detection frame;

[0112] Input the vehicle image corresponding to the vehicle target detection frame into the pre-trained multi-ReID feature extraction and fusion model to output the vehicle appearance features, wherein the multi-ReID feature extraction and fusion model includes a ReID feature extraction module composed of multiple ReIDs of different bonebones and a ReID feature fusion module, the vehicle image is input into the ReID feature extraction module to obtain the ReID features corresponding to different focus points, and all the ReID features are input into the ReID feature fusion module to fuse and obtain the vehicle appearance features;

[0113] Input the vehicle target detection frame and the vehicle appearance features into the multi-target tracking module to obtain the single-camera vehicle tracking trajectories of multiple vehicles in each camera;

[0114] The single-camera vehicle tracking trajectories of multiple cameras are filtered, reordered and clustered to obtain the cross-camera vehicle tracking trajectories under multiple cameras.

[0115] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0116] In general, various embodiments may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. Some aspects of the invention may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. Although various aspects of the invention may be shown and described as block diagrams, flow charts, or using some other graphical representation, it should be understood that, as non-limiting examples, the boxes, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0117] Embodiments of the present invention can be implemented by computer software, which is executable by a data processor of a mobile device, such as in a processor entity, or implemented by hardware, or implemented by a combination of software and hardware. Computer software or programs (also referred to as program products) including software routines, applets and / or macros can be stored in any device readable data storage medium, and they include program instructions for performing specific tasks. Computer program products can include one or more computer executable components configured to perform embodiments when the program is running. One or more computer executable components can be at least one software code or a part thereof. In addition, at this point, it should be noted that any box of the logic flow in the figure can represent a program step, or interconnected logic circuits, boxes and functions, or a combination of program steps and logic circuits, boxes and functions. Software can be stored in physical media such as memory chips or storage blocks implemented in processors, magnetic media such as hard disks or floppy disks, and optical media such as, for example, DVDs and data variants thereof, CDs. Physical media are non-transient media.

[0118] Those skilled in the art should understand that the technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0119] The present invention is not limited to the above-mentioned optimal implementation mode. Anyone can derive other various forms of products under the inspiration of the present invention. However, no matter what changes are made in the shape or structure, all technical solutions that are the same or similar to those of the present application fall within the protection scope of the present invention.

Claims

1. A highway cross-camera multi-vehicle tracking method combining multiple models, characterized in that: The following steps are involved: Get tracking video from multiple cameras on the highway; The video frame of the tracking video of each camera is input into the pre-trained vehicle target detection model to output the vehicle target detection frame, wherein the vehicle target detection model performs block processing on the input image to obtain multiple image blocks, and the image blocks are subjected to multiple layers of swin-transformer to obtain the feature map corresponding to each layer, and the image block sequence composed of the image blocks is passed into the multi-layer swin-transformer layer set in series, and each layer of swin-transformer layer recombines the image block sequence according to the window size of the current layer to obtain the feature map of the current layer, and each feature map is processed by the moving window self-attention module to obtain the corresponding multi-scale feature, and the multi-scale features are adaptively fused to obtain the fused feature, and the fused feature is passed into the classification network and the bounding box prediction network to predict the vehicle target detection frame; Input the vehicle image corresponding to the vehicle target detection frame into the pre-trained multi-ReID feature extraction and fusion model to output the vehicle appearance features, wherein the multi-ReID feature extraction and fusion model includes a ReID feature extraction module composed of multiple ReIDs of different bonebones and a ReID feature fusion module, the vehicle image is input into the ReID feature extraction module to obtain the ReID features corresponding to different focus points, and all the ReID features are input into the ReID feature fusion module to fuse and obtain the vehicle appearance features; The vehicle target detection frame and the vehicle appearance features are input into the multi-target tracking module to obtain the single-camera vehicle tracking trajectories of the multiple vehicles in each camera; in the step of "inputting the vehicle target detection frame and the vehicle appearance features into the multi-target tracking module to obtain the single-camera vehicle tracking trajectories of the multiple vehicles in each camera", the vehicles are divided into a high confidence group and a low confidence group according to the detection threshold, wherein the detection threshold is divided into a high detection threshold and a low detection threshold, and when the confidence of the vehicle target detection frame of the vehicle is greater than the high detection threshold, the vehicle target detection frame of the vehicle is included in the high confidence group, and the target detection frames of the remaining vehicles are involved in the high detection threshold and the low detection threshold, and the vehicle is included in the low confidence group. In the high confidence group, the vehicle appearance features of the vehicles in the high confidence group are matched for similarity. If they match, the single-camera vehicle tracking trajectory of the current vehicle is tracked and generated. If they do not match, the unmatched vehicles and the vehicles in the low confidence group are matched again according to the IOU of the vehicle target detection frame between adjacent video frames. If they match, the single-camera vehicle tracking trajectory of the current vehicle is generated. If they do not match again, the tracking threshold of the unmatched vehicle is calculated, where the tracking threshold is the center offset value of the target frame of the next vehicle calculated based on the tracking frame predicted by the Kalman filter. If the tracking threshold is greater than the threshold and the vehicle appears in consecutive video frames, the vehicle is re-included in the target tracking set for the next round of tracking; The single-camera vehicle tracking trajectories of multiple cameras are filtered, reordered and clustered to obtain the cross-camera vehicle tracking trajectories under multiple cameras, wherein filtering the single-camera vehicle tracking trajectories refers to: obtaining the passing time of the single-camera vehicle tracking trajectories of the same vehicle passing through the camera, judging the false positive single-camera vehicle tracking trajectories based on the passing time and the camera distribution order, and filtering the false positive single-camera vehicle tracking trajectories; Reordering the single-camera vehicle tracking trajectories means calculating the similarity between different single-camera vehicle tracking trajectories under the same camera based on the vehicle appearance features, reordering the single-camera vehicle tracking trajectories with high similarity to the target vehicle to the front position, and establishing a similarity matrix between adjacent cameras based on the similarity between different single-camera vehicle tracking trajectories, and using the k-complementary nearest neighbor algorithm to update the similarity matrix; Clustering the single-camera vehicle tracking trajectories refers to: clustering the single-camera vehicle tracking trajectories within the same camera using a hierarchical clustering method for the single-camera vehicle tracking trajectories after time filtering, and then clustering the single-camera vehicle tracking trajectories between different cameras according to the similarity matrix to obtain the cross-camera vehicle tracking trajectories.

2. The highway cross-camera multi-vehicle tracking method combined with multiple models according to claim 1 is characterized in that: In the step of "extracting features from multiple feature maps through moving window self-attention and adaptively fusing them to obtain fused features", each feature map is processed by the moving window self-attention module to obtain the corresponding multi-scale features, and the multi-scale features are adaptively fused to obtain the fused features. The formula for adaptive fusion is as follows: ;in is the multi-scale feature of the i-th layer The weight value of n is the number of feature layers of the fused multi-scale features.

3. The highway cross-camera multi-vehicle tracking method combined with multiple models according to claim 1 is characterized in that: In the step of "inputting the vehicle image into the ReID feature extraction module to obtain the ReID features corresponding to different focus points", the final stride of the backbone in each ReID is 1 and passes through the BN layer to obtain the standardized ReID features.

4. The highway cross-camera multi-vehicle tracking method combined with multiple models according to claim 1 is characterized in that: The multi-ReID feature extraction fusion model uses the joint loss function of the cross entropy loss function and the triple loss function as the loss function: ; Where L is the joint loss function, is the cross entropy loss function, is the triplet loss function, is a hyperparameter, which is learned and updated during model training: ; ; in, It is The real vehicle label of the instance, It is The predicted value of the instance vehicle, It is The target box of the instance vehicle, It is The positive target box of the instance vehicle, It is The negative target box of the instance vehicle, is a hyperparameter.

5. A highway cross-camera multi-vehicle tracking system combining multiple models, characterized in that: include: A vehicle detection module, wherein a pre-trained vehicle target detection model is provided in the vehicle detection model, tracking videos of multiple cameras on the highway are obtained, and the video frames of the tracking videos of each camera are input into the pre-trained vehicle target detection model to output a vehicle target detection frame, wherein the vehicle target detection model performs block processing on the input image to obtain multiple image blocks, and the image blocks are subjected to multiple layers of swin-transformer to obtain feature maps corresponding to each layer, and multiple feature maps are adaptively fused after feature extraction through moving window self-attention to obtain fused features, and the fused features are input into the classification network and the border prediction network to predict the vehicle target detection frame; A multi-ReID feature extraction and fusion module, wherein a pre-trained multi-ReID feature extraction and fusion model is provided in the multi-ReID feature extraction and fusion module, a vehicle image corresponding to a vehicle target detection frame is input into the pre-trained multi-ReID feature extraction and fusion model to output vehicle appearance features, wherein the multi-ReID feature extraction and fusion model includes a ReID feature extraction module composed of ReIDs of multiple different bonebones and a ReID feature fusion module, a vehicle image is input into the ReID feature extraction module to obtain ReID features corresponding to different focus points, and all ReID features are input into the ReID feature fusion module to be fused to obtain vehicle appearance features; The multi-target tracking module is used to input the vehicle target detection frame and the vehicle appearance features into the multi-target tracking module to obtain the single-camera vehicle tracking trajectories of multiple vehicles in each camera: the vehicles are divided into a high-confidence group and a low-confidence group according to the detection threshold, wherein the detection threshold is divided into a high detection threshold and a low detection threshold. When the confidence of the vehicle target detection frame of the vehicle is greater than the high detection threshold, the vehicle target detection frame of the vehicle is included in the high-confidence group. For the target detection frames of the remaining vehicles, the high detection threshold and the low detection threshold are involved, and the vehicles are included in the low-confidence group. The vehicle appearance features of the vehicles in the high-confidence group are compared. Similarity matching, if it matches, the single-camera vehicle tracking trajectory of the current vehicle is tracked and generated. If it does not match, the unmatched vehicles and vehicles in the low confidence group are matched again according to the IOU of the vehicle target detection frame between adjacent video frames. If it matches, the single-camera vehicle tracking trajectory of the current vehicle is generated. If it does not match again, the tracking threshold of the unmatched vehicle is calculated, where the tracking threshold is the center offset value of the tracking frame calculated according to the Kalman filter prediction and the target frame of the next vehicle. If the tracking threshold is greater than the threshold and the vehicle appears in consecutive video frames, the vehicle is re-included in the target tracking set for the next round of tracking; The cross-camera association trajectory module is used to filter, reorder and cluster the single-camera vehicle tracking trajectories of multiple cameras to obtain the cross-camera vehicle tracking trajectories under multiple cameras, wherein filtering the single-camera vehicle tracking trajectory refers to: obtaining the passing time of the single-camera vehicle tracking trajectory of the same vehicle passing through the camera, judging the false positive single-camera vehicle tracking trajectory based on the passing time and the camera distribution order, and filtering the false positive single-camera vehicle tracking trajectory; Reordering the single-camera vehicle tracking trajectories means calculating the similarity between different single-camera vehicle tracking trajectories under the same camera based on the vehicle appearance features, reordering the single-camera vehicle tracking trajectories with high similarity to the target vehicle to the front position, and establishing a similarity matrix between adjacent cameras based on the similarity between different single-camera vehicle tracking trajectories, and using the k-complementary nearest neighbor algorithm to update the similarity matrix; Clustering the single-camera vehicle tracking trajectories refers to: clustering the single-camera vehicle tracking trajectories within the same camera using a hierarchical clustering method for the single-camera vehicle tracking trajectories after time filtering, and then clustering the single-camera vehicle tracking trajectories between different cameras according to the similarity matrix to obtain the cross-camera vehicle tracking trajectories.

6. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the highway cross-camera multi-vehicle tracking method combined with multiple models as described in any one of claims 1 to 4.

7. A readable storage medium, characterized in that: The readable storage medium stores a computer program, which includes a program code for controlling a process to execute a process, wherein the process includes the highway cross-camera multi-vehicle tracking method combined with multiple models according to any one of claims 1 to 4.

Citation Information

Patent Citations

  • Target detection method, system and device and readable storage medium

    CN111144407A

  • Target tracking method and device

    CN115908489A

  • Illegal parking vehicle detection method, system and device based on Transform and super-resolution remote sensing image and medium

    CN115909187A

  • Vehicle position estimation method in traffic road network

    WO2011147101A1

  • Model compression method and apparatus, computer device, and storage medium

    WO2021159748A1