Real-time multi-camera multi-target tracking method based on attention mechanism

Abstract

The invention discloses a real-time multi-camera multi-vehicle tracking system based on an attention mechanism. The real-time multi-camera multi-vehicle tracking system comprises vehicle detection, single-camera multi-vehicle tracking, vehicle re-identification Re-ID and multi-camera multi-vehicle tracking. And vehicle detection: detecting and identifying the vehicle in the image to obtain the position information of the vehicle. A channel attention mechanism (CBAM) is added to a vehicle detector to extract semantic information of a higher level and feature information of a small target; according to the single-camera multi-vehicle tracking, continuous video frames are tracked on the basis of vehicle detection, and continuously changing pixel positions of vehicles are obtained; the vehicle re-identification runs through the whole process, and robust features are generated by expanding a training data set and introducing an attention mechanism; in order to improve the matching accuracy, a similarity matrix formed by space-time information and traffic rules obtained through camera monocular positioning and feature information obtained in the re-recognition stage is filtered, the matching accuracy is improved, and the time cost needed by matching is reduced.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and more particularly to an attention mechanism-based real-time multi-camera multi-target tracking method. Background Art

[0002] As one of the main research areas of intelligent traffic monitoring systems, cross-camera multi-vehicle tracking has a wide range of application scenarios. Cross-camera multi-vehicle tracking aims to solve the problem of continuous tracking of multiple vehicles under multiple cameras. The movement trajectory of the vehicle is obtained by identifying the same vehicle under different cameras at different times. Compared with single-camera multi-vehicle tracking, cross-camera multi-vehicle tracking requires analysis and processing of signals captured by multiple cameras, which requires higher data processing capabilities. Cross-camera multi-vehicle tracking provides important information for traffic monitoring tasks such as detecting whether traffic is abnormal or estimating traffic flow.

[0003] In the process of studying cross-camera multi-vehicle tracking, the following challenges are faced: In an environment with high-density pedestrian and vehicle traffic, vehicles often present small size and blur. Traditional vehicle detection algorithms, such as Yolo and SSD, focus on improving accuracy, but ignore small targets, making it difficult to accurately capture the location and features of small targets and improve the accuracy of small target detection. Vehicle re-identification runs through single-camera multi-vehicle tracking and multi-camera multi-vehicle tracking tasks, and is the key to vehicle matching. However, the small amount of training data will lead to overfitting problems, and due to different camera perspectives, there will be problems of similarity between classes and differences within classes, which will reduce the accuracy of matching. Many recent works have used attention models to extract tiny details in vehicles and obtain robust feature representations, but have not considered time-consuming issues. For multi-camera tracking problems, solving the problem of cross-camera vehicle association is a key issue. Existing cross-camera multi-vehicle tracking mainly uses offline batch processing schemes to associate vehicle trajectories. Offline schemes consider the adjacent frames before and after the current frame, and usually consider the entire video sequence to merge cross-camera vehicle trajectories. Post-processing techniques are also used to improve the final merged trajectory. The offline method improves robustness, and the method that considers the complete trajectory improves the tracking accuracy, but it is time-consuming and limits its application in real-time traffic monitoring scenarios. Therefore, how to ensure the real-time performance of cross-camera multi-target tracking tasks while ensuring accuracy is an urgent task to be solved.

[0004] This study aims to solve the two major problems of accuracy and real-time optimization. By introducing the attention mechanism, small-sized targets can be identified more accurately, and the problem of vehicle re-identification feature extraction can be solved to improve the performance of the multi-vehicle tracking system. While pursuing the accuracy of vehicle detection, real-time optimization is also the focus of attention. By optimizing the computational efficiency of the algorithm, it is ensured that the multi-vehicle tracking system can achieve fast and stable vehicle tracking when facing large-scale video streams. By comprehensively considering the requirements of tracking accuracy and real-time, it meets the growing needs of practical applications. Summary of the invention

[0005] In order to solve the above practical problems, the present invention provides a multi-camera multi-vehicle tracking system based on an attention mechanism, and the technical solution is as follows:

[0006] (1) Camera process: Create multiple camera processes so that they can be processed in parallel and in real time. The main task of the camera process is to read the 10fps video stream, obtain continuous video frames, store them in the frame queue, and continuously send the received frames to the processing process, which is responsible for vehicle detection and re-identification of the video frames. The detected and annotated video frames are returned to the respective camera processes for vehicle tracking.

[0007] (a) Vehicle tracking: In order to ensure real-time and efficient multi-camera multi-vehicle tracking, DeepSORT is selected as the tracker in the single-camera vehicle tracking stage. DeepSORT is an improvement on the SORT algorithm. It combines feature information on the basis of SORT and is more robust. The DeepSORT tracker defines the state of the target in the eight-dimensional state space:

[0008]

[0009] Where (u, v) is the coordinate of the center point of the bounding box of the detected vehicle, r represents the aspect ratio, and h represents the height. Represents the time derivative, that is, the speed in the image. In the vehicle tracking task, the bottom center point of the bounding box is closer to the ground than the center point of the bounding box, and contains less depth information, resulting in less distortion, so (u, v) is defined as the center coordinates of the bottom edge of the bounding box. DeepSORT uses a Kalman filter to predict the state of the vehicle in the current frame based on the previously obtained vehicle trajectory state. The Kalman filter combines prior information with current observations to provide an estimate of the target state. During the vehicle tracking process, the Mahalanobis distance of the vehicle motion information and the cosine distance of the feature information between the current frame and the previous frame are calculated based on the Kalman filter prediction state and the feature information extracted in the re-identification stage:

[0010] d (1) (i,j)=(d j -yi ) T S i -1 (d j -y i )

[0011]

[0012] where d j represents the jth vehicle bounding box detected by the vehicle detector, (y i ,S i ) represents the projection of the i-th vehicle trajectory in the measurement space. j is the appearance descriptor of the vehicle, where ||r j ||=1. In addition, each trajectory k selects the entire running trajectory L k The latest gallery of 100 relevant appearance descriptors R k = Cosine distance d (2) The value range of (i, j) is [0, 2], and the cosine distance between two identical eigenvectors is 0. By weighting the two distances, we get:

[0013] c i，j =λd (1) (i, j)+(1-λ)d (2) (i, j)

[0014] The Hungarian algorithm matches vehicle trajectories based on the constructed distance. If the match is successful, the trajectories are merged and the Kalman filter is updated. If the match fails, the current vehicle is used as the new vehicle trajectory.

[0015] After completing single-camera multi-vehicle tracking, the feature information of the entire trajectory obtained by the re-identification module is weighted summed to obtain a unique feature vector representation for vehicle association under multiple cameras.

[0016] (2) Processing process: The processing process receives the video frames read by the camera process, detects the vehicles in the video frames using the PP-YOLOE detector, and uses the vehicle re-identification model to extract feature information in the video frames for the DeepSORT tracking stage and the construction of the similarity matrix for vehicle association under multiple cameras.

[0017] (a) Vehicle detection: PP-YOLOE, as a single-stage detector, is more accurate and has better real-time performance than YOLOv5. In order to improve the accuracy of small target detection, the training dataset COCO is first subjected to data enhancement processing such as random cropping and small sample oversampling. While detecting small targets, it prevents the model from overfitting on the training set to a certain extent. PP-YOLOE is selected as the basic vehicle detector. There is no need to introduce some hyperparameters for anchors. Instead, an anchor-free method is introduced to make the model faster. Given a video frame, the vehicle detector performs detection and obtains a detection list in the following form:

[0018] [x1, y1, x2, y2, α, β]

[0019] Where (x1, y1) and (x2, y2) show the coordinates of the upper left corner and lower right corner of the bounding box of the detected vehicle. Design b = [x, y, w, h] is the bounding box of a certain vehicle, (x, y) is the pixel coordinates of the upper left corner of the bounding box, (w, h) is the width and height of the bounding box, and B = {b d , d∈[1,D]} is defined as the set of detection bounding boxes for each frame of all cameras, where D is the total number of detection bounding boxes. α and β represent the detection confidence and category label of the detection box, respectively. It is set that only three types of vehicles, cars, buses and trucks, are detected, and the detection confidence is set to 0.25. The backbone network of PP-YOLOE is CSPRepResNet, which consists of three convolutional layers and four subsequent stages of RepResBlock stacking. Each stage adopts cross-stage partial connection to avoid the large number of parameters and computational burden brought by the 3×3 convolutional layer. In the backbone network of the PP-YOLOE model, the channel attention mechanism is changed to CBAM (Convolutional Block AttentionModule). Compared with the original ESE channel attention mechanism, CBAM fully considers spatial information and channel information, uses adaptive attention weight learning instead of just global average pooling, and can extract higher-level semantic information and feature information of small targets to prevent the loss of shallow semantic information.

[0020] The dynamic label assignment algorithm task alignment strategy (TAL) is selected for label assignment. This method calculates the task alignment degree of each anchor to collect training samples (positive or negative samples), and the task alignment loss gradually unifies the best anchors obtained by classification and positioning, and predicts classification and positioning during training:

[0021]

[0022] In order to explicitly increase the score of the aligned anchors and reduce the score of the misaligned anchors, use To replace the label of the positive sample anchor, because of the TAL strategy, the bounding box with the highest classification score and the most accurate positioning can be retained during reasoning.

[0023] Finally, the loss function of PP-YOLOE is:

[0024]

[0025] VFL (varifocal loss) is used as the classification loss. It assigns different weights to positive and negative samples. Positive samples with high IoU contribute relatively more to the loss. This makes the model pay more attention to high-quality samples during training. DFL uses general distribution to predict bounding boxes, regards regression as a distribution prediction task, and solves the problem of inflexible bounding box representation. GIoU loss is used to represent the quality assessment of the detection box. Both use IoU-aware classification score (IACS) as the prediction target. This can effectively learn the joint representation of classification scores and positioning quality estimates, so that there is a high consistency between training and inference.

[0026] (b) Vehicle re-identification: Vehicle re-identification uses the CityFlow-V2 dataset, but the CityFlow-V2 training dataset is insufficient and prone to overfitting. The synthetic dataset VehicleX is used to expand the training dataset. The VehicleX dataset is generated by the publicly available 3D engine VehicleX, which only provides a training dataset of 192,150 images of 1,362 vehicles. There is a domain deviation between the synthetic data VehicleX and the real-world data. The Similarity Preserving Generative Adversarial Network (SPGAN) is introduced to migrate the synthetic data to the real-world images. On the basis of maintaining the original vehicle information, the style of the synthetic images and the real images is similar, which is more conducive to improving accuracy.

[0027] After obtaining sufficient data, we set out to optimize the accuracy and real-time performance of the vehicle re-identification model. Excited Vehicle Re-identification (EVER) is used as the re-identification model. Since different vehicles may be very similar in appearance, with the same brand, model and color, and the appearance of vehicles under different viewing angles varies greatly, the excited vehicle re-identification model introduces a self-supervised attention generation mechanism to generate robust features and focus on small features. EVER uses a variational auto-encoder to generate residual maps in a self-supervised manner. The residual map highlights the small-scale details in the image, as shown in the figure. The intermediate layers are excited by the generated residual maps, and the excitation intensity decreases monotonically as the training continues:

[0028]

[0029] Where T represents the total number of training epochs during the entire training process, and t represents the epoch number. ResNet IBN is used as the backbone architecture of the EVER model. Since only the residual map is used in the training phase to stimulate the intermediate feature map and assist the feature extraction model in learning more discriminative vehicle representations, α(t) = 0 in the inference phase, which reduces the computational complexity. Therefore, it is possible to maintain accuracy without increasing any overhead in inference time.

[0030] When the vehicle is a small target in the video frame, most of the extracted feature information only contains large-scale information, while ignoring the small-scale information that is important for improving matching accuracy. Therefore, after the single-camera multi-vehicle tracking is completed, the feature information extracted from each frame is weighted and averaged to calculate a single feature vector representation of the entire vehicle trajectory for vehicle matching under multiple cameras.

[0031] Train EVER using cross entropy and triplet loss function:

[0032]

[0033] In the triple loss formula, B, b i , a, γ, P(a), and N(a) are the total number of epochs, the i-th epoch, the anchor sample, the distance boundary threshold, and the positive and negative sample sets corresponding to a given anchor, respectively. In addition, f a 、f p 、f n It is the extracted features of anchor, positive sample and negative sample.

[0034]

[0035] In the cross entropy loss formula, It represents the logit (log-odds) value calculated for the jth category of the i-th training sample. Logit is calculated by the model parameters (weight vector W j and bias b j ) and the input feature f i The linear combination is calculated by the softmax function. This value reflects the probability that the model believes that the i-th sample belongs to the j-th category. N and C represent the total number of samples and categories in the training data set.

[0036] (3) Vehicle association under multiple cameras: After successfully implementing vehicle tracking and vehicle re-identification under a single camera, we set out to solve the problem of vehicle trajectory association under a multi-camera environment. The most important thing about this module is to construct a trajectory similarity matrix. The re-identification module extracts deep feature information, uses cosine distance for calculation, and constructs an initial feature similarity matrix. However, the vehicle models and colors are limited, and the lack of highly distinguishable feature information will lead to errors in subsequent matching. In order to improve the real-time requirements, the common post-processing operations of vehicle re-identification are cancelled, and the constraints of vehicle time and space information, traffic rules, etc. are used to optimize the vehicle association results under multiple cameras to improve the accuracy.

[0037] (a) Vehicles are constrained by spatial position information: The CityFlow dataset provides the camera homography matrix between the 2D image plane and the 3D GPS coordinates. The GPS coordinates of the vehicle in the real world can be obtained by detecting the coordinate position of the midpoint of the bottom surface of the vehicle’s bounding box. (latitude and longitude). Perform homography transformation on the detection bounding box set B to obtain the GPS coordinate set in the real world: The distance between cameras can be used to roughly estimate when a vehicle will disappear from one camera and appear in the next. Vehicles with GPS coordinates far apart are unlikely to be from the same vehicle. By setting a suitable distance threshold, we can filter out incompatible matches.

[0038] (b) Filtering based on traffic rules: Vehicles can only leave from designated intersections and reach the next intersection, and are not allowed to go in the opposite direction. Vehicle driving is completely constrained by traffic rules, thereby filtering out matches that are obviously inconsistent with the actual situation.

[0039] (c) Adjacent cameras: According to the given application scenario, the topological structure of the road can be observed and only the adjacent cameras need to be tracked.

[0040] Based on the comprehensive filtering of the above constraints, a new similarity matrix is ​​obtained, which significantly reduces the number of matching candidates and achieves a more robust matching result. Since the number of clustering categories is not known in advance, hierarchical clustering is finally used for vehicle matching to obtain the tracking trajectory of the vehicle under multiple cameras.

[0041] The technical solution provided by the present invention can improve and optimize the existing target detection, tracking algorithms, and re-identification algorithms, introduce an attention mechanism, focus on the feature information of small targets, improve the accuracy, and at the same time perform lightweight processing on the model to ensure real-time performance.

[0042] The solution proposed by the present invention can not only improve the accuracy of multi-camera multi-target tracking, but also solve the real-time demand that has been ignored for a long time. It can be applied to a wide range of traffic scenarios and promote the construction of intelligent transportation. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0044] Figure 1 It is a flow chart of the overall steps of the present invention; DETAILED DESCRIPTION

[0045] In order to enable persons skilled in the art to better understand the solutions of the embodiments of the present invention, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings and implementation modes.

[0046] In this system, the specific steps are as follows:

[0047] (1) Camera process: Create multiple camera processes so that they can be processed in parallel and in real time. The main task of the camera process is to read the 10fps video stream, obtain continuous video frames, store them in the frame queue, and continuously send the received frames to the processing process, which is responsible for vehicle detection and re-identification of the video frames. The detected and annotated video frames are returned to the respective camera processes for vehicle tracking.

[0048] (a) Vehicle tracking: In order to ensure real-time and efficient multi-camera multi-vehicle tracking, DeepSORT is selected as the tracker in the single-camera vehicle tracking stage. DeepSORT is an improvement on the SORT algorithm. It combines feature information on the basis of SORT and is more robust. The DeepSORT tracker defines the state of the target in the eight-dimensional state space:

[0049]

[0050] Where (u, v) is the coordinate of the center point of the bounding box detected by the vehicle detector, r represents the aspect ratio, and h represents the height of the bounding box. Represents the time derivative, that is, the speed in the image. In the vehicle tracking task, the bottom center point of the bounding box is closer to the ground than the center point of the bounding box, and contains less depth information resulting in less distortion, so (u, v) is defined as the center coordinates of the bottom edge of the bounding box of the detected vehicle. DeepSORT uses a Kalman filter to predict the state of the vehicle in the current frame based on the previously obtained vehicle trajectory state. The Kalman filter combines prior information with current observations to provide an estimate of the target state. During the vehicle tracking process, the Mahalanobis distance of the vehicle motion information and the cosine distance of the feature information between the current frame and the previous frame are calculated based on the Kalman filter predicted state and the feature information extracted in the re-identification stage:

[0051] d (1) (i,j)=(d j -y i ) T S i -1 (d j -y i )

[0052]

[0053] where d j represents the jth vehicle bounding box detected by the vehicle detector, (y i ,S i ) represents the projection of the i-th vehicle trajectory in the measurement space. j is the vehicle appearance descriptor, where ||r j ||=1. In addition, each trajectory k selects the entire running trajectory L k The latest gallery of 100 relevant appearance descriptors R k = Cosine distance d (2) The value range of (i, j) is [0, 2], and the cosine distance between two identical eigenvectors is 0. By weighting the two distances, we get:

[0054] c i，j =λd (1) (i, j)+(1-λ)d (2) (i, j)

[0055] The Hungarian algorithm matches vehicle trajectories based on the constructed distance. If the match is successful, the trajectories are merged and the Kalman filter is updated. If the match fails, the current vehicle is used as the new vehicle trajectory.

[0056] After completing single-camera multi-vehicle tracking, the feature information of the entire trajectory obtained by the re-identification module is weighted summed to obtain a unique feature vector representation for vehicle association under multiple cameras.

[0057] (2) Processing process: The processing process receives the video frames read by the camera process, detects the vehicles in the video frames using the PP-YOLOE detector, and uses the vehicle re-identification model to extract feature information in the video frames for the DeepSORT tracking stage and the construction of the similarity matrix for vehicle association under multiple cameras.

[0058] (a) Vehicle detection: PP-YOLOE, as a single-stage detector, is more accurate and has better real-time performance than YOLOv5. In order to improve the accuracy of small target detection, the training dataset COCO is first subjected to data enhancement processing such as random cropping and small sample oversampling. While detecting small targets, it prevents the model from overfitting on the training set to a certain extent. PP-YOLOE is selected as the basic vehicle detector. There is no need to introduce some hyperparameters for anchors. Instead, an anchor-free method is introduced to make the model faster. Given a video frame, the vehicle detector performs detection and obtains a detection list in the following form:

[0059] [x1, y1, x2, y2, α, β]

[0060] Where (x1, y1) and (x2, y2) show the coordinates of the upper left corner and lower right corner of the bounding box of the detected vehicle. Design b = [x, y, w, h] is a vehicle bounding box, (x, y) is the pixel coordinates of the upper left corner of the bounding box, (w, h) is the width and height of the bounding box, and B = {b d , d∈[1,D]} is defined as the set of detection bounding boxes for each frame of all cameras, where D is the total number of detection bounding boxes. α and β represent the detection confidence and category label of the detection box, respectively. Only three types of vehicles, cars, buses and trucks, are set to be detected, and the detection confidence is set to 0.25. The backbone network of PP-YOLOE is CSPRepResNet, which consists of three convolutional layers and four subsequent stages of RepResBlock stacking. Each stage uses cross-stage partial connections to avoid the large number of parameters and computational burden brought by the 3×3 convolutional layer. In the backbone network of the PP-YOLOE model, the channel attention mechanism is changed to CBAM (Convolutional Block Attention Module). Compared with the original ESE channel attention mechanism, CBAM fully considers spatial information and channel information, uses adaptive attention weight learning instead of just global average pooling, and can extract higher-level semantic information and feature information of small targets to prevent the loss of shallow semantic information.

[0061] The dynamic label assignment algorithm task alignment strategy (TAL) is selected for label assignment. This method calculates the task alignment degree of each anchor to collect training samples (positive or negative samples), and the task alignment loss gradually unifies the best anchors obtained by classification and positioning, and predicts classification and positioning during training:

[0062]

[0063] In order to explicitly increase the score of the aligned anchors and reduce the score of the misaligned anchors, use To replace the label of the positive sample anchor, because of the TAL strategy, the bounding box with the highest classification score and the most accurate positioning can be retained during reasoning.

[0064] Finally, the loss function of PP-YOLOE is:

[0065]

[0066] VFL (varifocal loss) is used as the classification loss. It assigns different weights to positive and negative samples. Positive samples with high IoU contribute relatively more to the loss. This makes the model pay more attention to high-quality samples during training. DFL uses general distribution to predict bounding boxes, regards regression as a distribution prediction task, and solves the problem of inflexible bounding box representation. GIoU loss is used to represent the quality assessment of the detection box. Both use IoU-aware classification score (IACS) as the prediction target. This can effectively learn the joint representation of classification scores and positioning quality estimates, so that there is a high consistency between training and inference.

[0067] (b) Vehicle re-identification: Vehicle re-identification uses the CityFlow-V2 dataset, but the CityFlow-V2 training dataset is insufficient and prone to overfitting. The synthetic dataset VehicleX is used to expand the training dataset. The VehicleX dataset is generated by the publicly available 3D engine VehicleX, which only provides a training dataset of 192,150 images of 1,362 vehicles. There is a domain deviation between the synthetic data VehicleX and the real-world data. The Similarity Preserving Generative Adversarial Network (SPGAN) is introduced to migrate the synthetic data to the real-world images. On the basis of maintaining the original vehicle information, the style of the synthetic images and the real images is similar, which is more conducive to improving accuracy.

[0068] After obtaining sufficient data, we set out to optimize the accuracy and real-time performance of the vehicle re-identification model. Excited Vehicle Re-identification (EVER) is used as the re-identification model. Since different vehicles may be very similar in appearance, with the same brand, model and color, and the appearance of vehicles under different viewing angles varies greatly, the excited vehicle re-identification model introduces a self-supervised attention generation mechanism to generate robust features and focus on small features. EVER uses a variational auto-encoder to generate residual maps in a self-supervised manner. The residual map highlights the small-scale details in the image, as shown in the figure. The intermediate layers are excited by the generated residual maps, and the excitation intensity decreases monotonically as the training continues:

[0069]

[0070] Where T represents the total number of training epochs during the entire training process, and t represents the epoch number. ResNet_IBN is used as the backbone architecture of the EVER model. Since only the residual map is used in the training phase to stimulate the intermediate feature map and assist the feature extraction model in learning more discriminative vehicle representations, α(t) = 0 in the inference phase, which reduces the computational complexity. Therefore, it is possible to maintain accuracy without increasing any overhead in inference time.

[0071] When the vehicle is a small target in the video frame, most of the extracted feature information only contains large-scale information, while ignoring the small-scale information that is important for improving matching accuracy. Therefore, after the single-camera multi-vehicle tracking is completed, the feature information extracted from each frame is weighted and averaged to calculate a single feature vector representation of the entire vehicle trajectory for vehicle matching under multiple cameras.

[0072] Train EVER using cross entropy and triplet loss function:

[0073]

[0074] In the triple loss formula, B, b i , a, γ, P(a) and N(a) are the total number of epochs, the i-th epoch, the anchor sample, the distance boundary threshold and the positive and negative sample sets corresponding to the given anchor. In addition, f a 、f p 、f n It is the extracted features of anchor, positive sample and negative sample.

[0075]

[0076] In the cross entropy loss formula, Represents the logit (log-odds) value calculated for the jth category of the i-th training sample. Logit is calculated by adjusting the model parameters (weight vector W j and bias b j ) and the input feature f i The linear combination is calculated by the softmax function. This value reflects the probability that the model believes that the i-th sample belongs to the j-th category. N and C represent the total number of samples and categories in the training data set.

[0077] (3) Vehicle association under multiple cameras: After successfully implementing vehicle tracking and vehicle re-identification under a single camera, we set out to solve the problem of vehicle trajectory association under a multi-camera environment. The most important thing about this module is to construct a trajectory similarity matrix. The re-identification module extracts deep feature information, uses cosine distance for calculation, and constructs an initial feature similarity matrix. However, the vehicle models and colors are limited, and the lack of highly distinguishable feature information will lead to errors in subsequent matching. In order to improve the real-time requirements, the common post-processing operations of vehicle re-identification are cancelled, and the constraints of vehicle time and space information, traffic rules, etc. are used to optimize the vehicle association results under multiple cameras to improve the accuracy.

[0078] (a) Vehicles are constrained by spatial position information: The CityFlow dataset provides the camera homography matrix between the 2D image plane and the 3D GPS coordinates. The GPS coordinates of the vehicle in the real world can be obtained by detecting the coordinate position of the midpoint of the bottom surface of the vehicle’s bounding box. (latitude and longitude). Perform homography transformation on the detection bounding box set B to obtain the GPS coordinate set in the real world: The distance between cameras can be used to roughly estimate when a vehicle will disappear from one camera and appear in the next. Vehicles with GPS coordinates far apart are unlikely to be from the same vehicle. By setting a suitable distance threshold, we can filter out incompatible matches.

[0079] (b) Filtering based on traffic rules: Vehicles can only leave from designated intersections and reach the next intersection, and are not allowed to go in the opposite direction. Vehicle driving is completely constrained by traffic rules, thereby filtering out matches that are obviously inconsistent with the actual situation.

[0080] (c) Adjacent cameras: According to the given application scenario, the topological structure of the road can be observed and only the adjacent cameras need to be tracked.

[0081] Based on the comprehensive filtering of the above constraints, a new similarity matrix is ​​obtained, which significantly reduces the number of matching candidates and achieves a more robust matching result. Since the number of clustering categories is not known in advance, hierarchical clustering is finally used for vehicle matching to obtain the tracking trajectory of the vehicle under multiple cameras.

Claims

1. A real-time multi-camera multi-vehicle tracking method based on an attention mechanism, the method comprising: Step A, vehicle detection, by changing the channel attention mechanism of the vehicle detector PP-YOLOE to CBAM, detect the vehicle appearing in a given video frame and obtain the accurate position of the vehicle in the video frame; Step B, single camera vehicle tracking, based on the vehicle detected in each video frame, the DeepSORT tracker is used to track and match in consecutive video frames to obtain the continuous motion changes of the vehicle over time; Step C, vehicle re-identification, selects the excited vehicle re-identification model based on the self-supervised attention mechanism to extract vehicle feature information for vehicle matching and tracking; Step D, vehicle association under multiple cameras, matches and associates vehicles under different cameras to obtain the complete tracking trajectory of the vehicle under multiple cameras.

2. The real-time multi-camera multi-vehicle tracking method based on attention mechanism according to claim 1 is characterized in that: In step A, PP-YOLOE is used as the basic detection model.

3. The real-time multi-camera multi-vehicle tracking method based on attention mechanism according to claim 1 is characterized in that: In step A, data enhancement processing is performed on the data set, and the channel attention mechanism is changed to CBAM on the PP-YOLOE detection model to improve the accuracy of small target detection.

4. The real-time multi-camera multi-vehicle tracking method based on attention mechanism according to claim 1 is characterized in that: In step B, the detection and tracking paradigm DeepSORT is used as the vehicle tracker.

5. The real-time multi-camera multi-vehicle tracking method based on attention mechanism according to claim 1 is characterized in that: In step C, a major problem faced by vehicle re-identification during training is insufficient training data. A synthetic dataset VehicleX is used to expand the data. In order to solve the domain deviation problem between synthetic data and real-world data, a similarity-preserving generative adversarial network SPGAN is introduced to migrate synthetic data to real-world images.

6. The real-time multi-camera multi-vehicle tracking method based on attention mechanism according to claim 1 is characterized in that: In the step C, an excited vehicle re-identification model that introduces a self-supervised attention mechanism is adopted. In order to retain the small-scale information of the re-identification, the model is equipped with a time weighting module to obtain a unique feature representation of the entire trajectory of the vehicle.

7. The real-time multi-camera multi-vehicle tracking method based on attention mechanism according to claim 1 is characterized in that: In the step D, after achieving the single-camera multi-vehicle tracking and vehicle re-identification tasks, an initial similarity matrix is ​​first constructed based on the feature information extracted by re-identification.

8. The real-time multi-camera multi-vehicle tracking method based on attention mechanism according to claim 1, characterized in that: In the step D, based on the initial similarity matrix, the similarity matrix is ​​filtered and optimized by utilizing the constraints of the vehicle's spatiotemporal information and traffic rules.

9. The real-time multi-camera multi-vehicle tracking method based on attention mechanism according to claim 8, characterized in that: The spatiotemporal information is based on the camera homography matrix provided by the CityFlow dataset. The GPS coordinates of the vehicle are obtained by detecting the coordinate position of the midpoint of the bottom surface of the vehicle's bounding box. According to the distance between the cameras, it is estimated when the vehicle disappears from one camera and appears in the next camera. By setting a distance threshold, non-conforming matches are filtered out.

10. The real-time multi-camera multi-vehicle tracking method based on attention mechanism according to claim 8, characterized in that: The traffic rules are that in the real world, vehicles do not go in the opposite direction and can only leave from a certain intersection to reach the next intersection. The traffic rules formulated in the real world filter out matches that do not meet the situation. In the step D, hierarchical clustering is used according to the constructed similarity matrix to obtain the final vehicle tracking result under multiple cameras.

Images