Cross-camera multi-vehicle tracking method, system and equipment

Abstract

The invention relates to the technical field of image feature extraction, in particular to a cross-camera multi-vehicle tracking method, system and equipment, and the method comprises the steps: obtaining videos of a plurality of cameras, recognizing a target vehicle in each frame of image in the videos, outputting the basic information of the target vehicle, and distributing an image token, a category token and a partial token for the images; the historical features of each part token form a fusion feature; adding the global features to generate combined features; constructing historical track characteristics of the target vehicle before the current frame, and converting the historical track characteristics into a time token through linear transformation; reasoning according to the time token to generate a prediction track of the current frame of the target vehicle, and further reasoning to obtain a complete track; and clustering the complete tracks of the target vehicle of the plurality of cameras to complete cross-camera vehicle tracking. According to the method, the problems of difficulty in cross-camera matching, shielding, motion blur, track repair and vehicle re-identification precision of vehicle tracking in a multi-camera scene can be solved.

Description

Technical Field

[0001] The present invention relates to the technical field of image feature extraction, and in particular to a method, system and device for tracking multiple vehicles across cameras. Background Art

[0002] With the rapid development of intelligent transportation systems and autonomous driving technologies, traffic monitoring and management have become increasingly important in complex and changing urban environments. Vehicle target tracking, as one of the core tasks in intelligent transportation systems, has become a hot topic of research. Traditional vehicle tracking methods usually rely on a single camera to obtain target information, and are prone to misjudgment when dealing with complex situations such as occlusion, motion blur, and target intersection. However, in real-world applications, multi-camera video streams provide richer perspectives and information, which helps to overcome the limitations of single-camera tracking, especially in large-scale urban monitoring.

[0003] Despite this, multi-camera vehicle tracking still faces difficulties in cross-camera target matching, occlusion and motion blur problems, trajectory repair and prediction, and vehicle re-identification accuracy issues.

[0004] Therefore, existing multi-camera vehicle tracking methods still have many shortcomings in accuracy, efficiency and stability. Summary of the invention

[0005] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides a method, system and device for tracking multiple vehicles across cameras.

[0006] The present invention provides a method for tracking multiple vehicles across cameras, and the technical solution is as follows: A cross-camera multi-vehicle tracking method, specifically comprising: S1. Obtain videos from multiple cameras, identify the target vehicle in each frame of the video, and output basic information of the target vehicle, including the category label, category confidence, and bounding box of the target vehicle; S2, splitting the image of the identified target vehicle into N non-overlapping patches according to the size of the preset block, and each patch serves as an image token; According to the category label of the target vehicle, a category token is assigned; Divide the target vehicle into the front, body and rear, and assign corresponding partial tokens; After global attention processing based on category tokens and image tokens, global features are output; Based on the partial tokens and image tokens, local features are output after local attention processing; S3. The historical features of each partial token form a fused feature; compare the fused feature with each local feature, calculate the similarity deviation function value with the local feature, sort them from small to large, start selecting from the fused feature with the smallest similarity deviation function value, and take the first threshold number of fused features as the positive sample set; compare the fused features in the positive sample set with the local features, calculate the similarity deviation function value with the local feature, sort them from small to large, start selecting from the fused feature with the smallest similarity deviation function value, take the second threshold number of fused features as the similarity feature set, and add them to the global feature to generate a combined feature; S4. In different frames of the same video, the target vehicle is matched based on the combined features. If the target vehicle is identified, the coordinate set of the target vehicle's bounding box forms an activity track, and the historical track features of the target vehicle before the current frame are constructed. Each historical track feature is composed of the center of the target vehicle's bounding box and the normalized size of the target vehicle's bounding box in the activity track, and is converted into a time token through a linear transformation; the predicted track of the target vehicle in the current frame is generated based on the time token reasoning, and the deviation between the predicted track of the current frame and the actual activity track of the current frame is calculated. If the deviation is less than the deviation threshold, the predicted track and the activity track of the current frame are summarized to generate the time track of the target vehicle, and further obtain the complete track; S5. Clustering is performed on the complete trajectories of the target vehicle from multiple cameras, and the complete trajectories of the target vehicle are merged to complete the cross-camera vehicle tracking.

[0007] The similarity deviation function is: , in, Represents the similarity deviation function value, m is the fusion feature index, K is the number of samples in the data set, j is the index of the sample, pi represents the local area of ​​the image, is the local eigenvector, is the feature vector, is the temperature coefficient, and n is the number of fused features involved in the calculation.

[0008] In S4, if the target vehicle is not recognized in the image of the current frame, this frame of image is classified as a lost track.

[0009] Based on the time and time trajectory corresponding to the frame where the lost trajectory is located, the lost trajectory is repaired, and the repaired lost trajectory and the time trajectory are spliced ​​into a complete trajectory. The specific method for repairing the lost trajectory is: , in, is the lost trajectory of the target at time t+1, is the activity trajectory, is the activity trajectory of the target at time step t.

[0010] Generate the target vehicle prediction trajectory based on time token reasoning. The specific steps are: , , , , in, is the output of the first inference, For the l -1 layer output, is the output of the second reasoning, is the fusion of the two outputs, For the l The output of the layer, It is the operation module for the first reasoning. It is the operation module for the second reasoning. For multi-layer perceptron operation, It is a normalization operation.

[0011] The fused features form a memory library, and the memory library update formula is: , in, is the fusion feature, pi represents the local area of ​​the image, t is the update round, m is the momentum coefficient, Represents the extraction function of local features, j is the index of the sample, j=1,…,K; K is the number of samples in the data set.

[0012] The specific steps of global attention processing and local attention processing are: Global attention processing: The category token is used as a query and compared with the key of all image tokens to calculate the attention weight, so as to perform weighted summation on all image tokens to obtain the global feature; Local attention processing: Partial tokens are used as queries, and the keys of image tokens corresponding to each partial token in the image are weighted to obtain local features.

[0013] The specific operation steps of S5 are: The center of the video coverage area of ​​multiple cameras is taken as the center of the circle, and the coverage area is divided into four sub-areas at a 90° central angle. The complete trajectory of the target vehicle is clustered according to the same sub-area, and the intersection of the complete trajectory of the target vehicle in every two cameras is searched. The complete trajectories with intersection are merged to complete cross-camera vehicle tracking.

[0014] A cross-camera multi-vehicle tracking system, comprising: Recognition module: obtains videos from multiple cameras, identifies the target vehicle in each frame of the video, and outputs the basic information of the target vehicle, including the category label, category confidence, and bounding box of the target vehicle; Re-identification module: The image of the identified target vehicle is divided into N non-overlapping patches according to the preset block size, and each patch is used as an image token; According to the category label of the target vehicle, a category token is assigned; Divide the target vehicle into the front, body and rear, and assign corresponding partial tokens; After global attention processing based on category tokens and image tokens, global features are output; Based on the partial tokens and image tokens, local features are output after local attention processing; Feature processing module: The historical features of each partial token form a fusion feature; compare the fusion feature with each local feature, calculate the similarity deviation function value with the local feature, sort from small to large, start selecting from the fusion feature with the smallest similarity deviation function value, and take the first threshold number of fusion features as the positive sample set; compare the fusion features in the positive sample set with the local features, calculate the similarity deviation function value with the local feature, sort from small to large, start selecting from the fusion feature with the smallest similarity deviation function value, take the second threshold number of fusion features as the similarity feature set, and add them to the global feature to generate a combined feature; Single-camera trajectory tracking module: In different frame images of the same video, the target vehicle is matched based on the combined features. If the target vehicle is identified, the coordinate set of the target vehicle's bounding box forms an activity trajectory, and the historical trajectory features of the target vehicle before the current frame are constructed. Each historical trajectory feature is composed of the center of the target vehicle's bounding box in the activity trajectory and the normalized size of the target vehicle's bounding box, which are converted into time tokens through linear transformation; the predicted trajectory of the target vehicle in the current frame is generated based on the time token reasoning, and the deviation between the predicted trajectory of the current frame and the actual activity trajectory of the current frame is calculated. If the deviation is less than the deviation threshold, the predicted trajectory and activity trajectory of the current frame are summarized to generate the time trajectory of the target vehicle, and further obtain the complete trajectory; Multi-camera trajectory tracking module: clusters the complete trajectories of target vehicles from multiple cameras, and merges the complete trajectories of target vehicles to complete cross-camera vehicle tracking.

[0015] A cross-camera multi-vehicle tracking device comprises a processor and a memory. When the processor executes a stored computer program, the cross-camera multi-vehicle tracking method described in any one of the above steps can be implemented.

[0016] Beneficial effects of the present invention: By acquiring videos from multiple cameras and identifying the target vehicle in each frame of the video, the basic information of the target vehicle can be accurately output, including category labels, category confidence, and bounding boxes. This provides a reliable data foundation for subsequent vehicle tracking.

[0017] Segmenting the image with the target vehicle identified into non-overlapping patches and assigning category tokens and part tokens helps to capture the features of the vehicle more finely in the subsequent attention processing. This meticulous segmentation and token assignment improves the accuracy and efficiency of feature extraction.

[0018] Global attention is performed based on category tokens and image tokens to output global features; at the same time, local attention is performed based on partial tokens and image tokens to obtain local features. This attention mechanism that combines global and local information can capture vehicle features more comprehensively and improve the accuracy and robustness of vehicle tracking.

[0019] By calculating the similarity deviation function value between the fusion feature and the local feature, the positive sample set and the similar feature set are screened out, and then added to the global feature to generate a combined feature. This intelligent positive sample screening and feature fusion method helps to accurately identify target vehicles in complex environments.

[0020] The target vehicle is matched based on the combined features, the historical trajectory features of the target vehicle are constructed, and the predicted trajectory is generated through time token reasoning. This efficient trajectory construction and prediction method can track the motion trajectory of the target vehicle in real time, providing strong support for cross-camera vehicle tracking.

[0021] The complete trajectories of the target vehicle from multiple cameras are clustered and merged to complete the cross-camera vehicle tracking. This cross-camera tracking method can achieve continuous tracking of the target vehicle between different cameras, providing important technical support for the fields of intelligent transportation systems, security monitoring, etc. DETAILED DESCRIPTION

[0022] The technical solution of the present invention is as follows: A cross-camera multi-vehicle tracking method, specifically comprising: S1. Obtain videos from multiple cameras, identify the target vehicle in each frame of the video, and output the basic information of the target vehicle, including the category label, category confidence, and bounding box of the target vehicle.

[0023] Decode the acquired video stream and split it into consecutive image frames.

[0024] Each frame of image is preprocessed, including denoising and histogram equalization, to improve image quality.

[0025] Use deep learning models to identify targets in each frame.

[0026] Output the bounding box of the target vehicle and the corresponding category label and category confidence, and extract the basic information of the target vehicle from the output, including category label, category confidence and bounding box.

[0027] The present invention uses the YOLOv8 network to achieve target recognition. YOLOv8 is an efficient convolutional neural network designed for fast and accurate real-time recognition tasks. Its high efficiency and high accuracy are suitable for processing vehicle recognition tasks in multi-camera video streams.

[0028] YOLOv8 can extract features at multiple scales and perform object classification and bounding box regression. By training and reasoning on images of different resolutions, YOLOv8 can more accurately detect objects of different sizes. Compared with previous versions, YOLOv8 has been optimized in terms of accuracy and speed, especially in complex backgrounds or multi-target scenes, the detection accuracy has been significantly improved, and the operating memory requirements of the processor are reasonable.

[0029] In practical applications, YOLOv8 first receives each frame of image from the multi-camera video stream, and identifies the target vehicle for each frame. Every time a target vehicle is detected, YOLOv8 outputs the category label, bounding box, and category confidence of the target.

[0030] After preprocessing, the image is input into the YOLOv8 network, and features are extracted through the backbone network; YOLOv8 recognizes the target in the image through its detection head, generates candidate boxes, and uses the non-maximum suppression algorithm to remove duplicate bounding boxes; the target vehicle is identified, and the bounding box coordinates and category confidence of each vehicle are further generated; YOLOv8 will further classify according to category recognition and output the specific category label of the target, including car, truck, bus, motorcycle, and its bounding box coordinates.

[0031] Whenever a bounding box is generated, the YOLOv8 network passes it directly to the re-identification module, which receives the bounding box information and performs re-identification processing on each vehicle target.

[0032] S2, splitting the image of the identified target vehicle into N non-overlapping patches according to the size of the preset block, and each patch serves as an image token; According to the category label of the target vehicle, a category token is assigned; Divide the target vehicle into the front, body and rear, and assign corresponding partial tokens; After global attention processing based on category tokens and image tokens, global features are output; Based on the partial tokens and image tokens, local features are output after local attention processing.

[0033] The core innovation of this step is to effectively capture the local area features of the vehicle by introducing partial tokens, and jointly construct a multi-dimensional feature representation with the category tokens. In order to capture the fine-grained differences of the vehicle, each target vehicle is divided into three main parts: the front, the body, and the rear. This division can help the model focus on the visual features of different parts of the vehicle, so as to better cope with the differences between different models. However, the body designs of different models vary greatly, and even different shooting angles may lead to differences in local features of the front, body, and rear. Therefore, a dynamic adjustment mechanism is introduced during the division to adaptively adjust the selection of local areas according to the structural characteristics of different models.

[0034] Modules that can implement S2 include global attention and local attention. Global attention uses category tokens and all image tokens to perform global attention processing to extract global features. Local perception attention uses partial tokens and image tokens of local areas in the image to perform local attention processing to extract features of specific local areas.

[0035] The specific steps of global attention processing and local attention processing are: Global attention processing: The category token is used as a query and compared with the key of all image tokens to calculate the attention weight, so as to perform weighted summation on all image tokens to obtain the global feature; Local attention processing: Partial tokens are used as queries, and the keys of image tokens corresponding to each partial token in the image are weighted to obtain local features.

[0036] The present invention improves the feature extraction capability of the target vehicle by combining the global and local attention mechanisms through the re-identification module.

[0037] Based on the input image and its corresponding bounding box information, local and global features are extracted, and the appearance features of the vehicle are encoded through a multi-layer attention mechanism to generate a highly recognizable vehicle feature vector. These feature vectors are not only used for vehicle identity recognition, but are also passed to the next step to solve the association problem of the target vehicle under different camera perspectives.

[0038] Through this multi-level feature learning, not only can global information be obtained, but also the fine-grained local features of the target vehicle can be captured in detail, thereby improving the vehicle recognition accuracy in complex scenarios, effectively improving the feature expression ability in re-identification tasks, and reducing the overfitting problem.

[0039] S3. The historical features of each partial token form a fused feature; compare the fused feature with each local feature, calculate the similarity deviation function value with the local feature, sort them from small to large, start selecting from the fused feature with the smallest similarity deviation function value, and take the first threshold number of fused features as the positive sample set; compare the fused features in the positive sample set with the local features, calculate the similarity deviation function value with the local feature, sort them from small to large, start selecting from the fused feature with the smallest similarity deviation function value, take the second threshold number of fused features as the similarity feature set, and add them to the global feature to generate a combined feature.

[0040] In order to maximize the similarity learning of local features, a momentum-updated memory library is maintained for each partial token’s historical features ,The features stored in the memory bank are called fused features. In each similarity training cycle, the ,feature information of different parts is gradually accumulated by updating the ,memory bank with momentum.

[0041] The fused features form a memory library, and the memory library update formula is: , in, is the fusion feature, pi represents the local area of ​​the image, t is the update round, m is the momentum coefficient, Represents the extraction function of local features, j is the index of the sample, j=1,…,K; K is the number of samples in the data set.

[0042] Compare the fused features with each local feature, start from the fused feature with the smallest similarity deviation function value with the local feature, and take the first threshold number of fused features as the positive sample set; compare the fused features in the positive sample set with the local features, start from the fused feature with the smallest similarity deviation function value with the local feature, take the second threshold number of fused features as the similarity feature set, and add them to the global feature to generate a combined feature.

[0043] The similarity deviation function is: , in, Represents the value of the similarity deviation function, m is the fusion feature index selected from the memory library, K is the number of samples in the data set, j is the index of the sample, pi represents the local area of ​​the image, is the local eigenvector, is the feature vector in the memory bank, is the temperature coefficient, and n is the number of memory library fusion features involved in the calculation.

[0044] After S3 processing, a highly recognizable feature vector is output, which contains the combined features of the target vehicle.

[0045] S4. In different frame images of the same video, the target vehicle is matched based on the combined features. If the target vehicle is identified, the coordinate set of the target vehicle's bounding box forms an activity trajectory, and the historical trajectory features of the target vehicle before the current frame are constructed. Each historical trajectory feature is composed of the center of the target vehicle's bounding box and the normalized size of the target vehicle's bounding box in the activity trajectory, and is converted into a time token through a linear transformation; the predicted trajectory of the target vehicle in the current frame is generated based on the time token reasoning, and the deviation between the predicted trajectory of the current frame and the actual activity trajectory of the current frame is calculated. If the deviation is less than the deviation threshold, the predicted trajectory and the activity trajectory of the current frame are summarized to generate the time trajectory of the target vehicle, and further obtain the complete trajectory.

[0046] For the target vehicle, an identity label is further assigned to each bounding box by generating a combined feature of the target vehicle. These identity labels are passed to S4 to ensure that the same target vehicle can be accurately identified and tracked at different times and camera views.

[0047] The specific method for obtaining the target vehicle trajectory is that each trajectory is represented as a sequence of bounding boxes of the target vehicle in continuous time frames. As the video progresses, new targets may enter the scene and existing targets may exit the scene. In order to handle this dynamic change, the trajectory is further divided into active and lost trajectories, which represent the currently active trajectory and the temporarily lost trajectory respectively.

[0048] The calculation of the predicted trajectory of the current frame can be achieved through the Mamba motion predictor, which consists of three main components. The first component is the input embedding layer, which takes the historical dynamics of the object trajectory as input and constructs historical trajectory features to represent the changes in the position of the target vehicle in the past time. Each trajectory feature consists of the normalized changes in the center and size of the bounding box. These features are converted into time tokens through linear transformation; the second component is the encoding layer, which consists of l The Mamba module is the core of the system. In order to better capture the dynamic behavior of the target, the Mamba module adopts a bidirectional encoder structure, which can make full use of the historical trajectory characteristics of the activity trajectory. Each encoding layer contains forward and backward Mamba modules, which process the activity trajectory forward and backward to obtain a comprehensive motion pattern; the last layer is the prediction head, which is responsible for predicting the inter-frame bounding box offset of the object trajectory.

[0049] Generate the target vehicle prediction trajectory based on time token reasoning. The specific steps are: , , , , in, is the output of the first inference, i.e. forward processing, For the l -1 layer output, is the output of the second reasoning, i.e., backward processing, is the fusion of the two outputs, For the l The output of the layer, is the operation module for the first reasoning, It is the operation module for the second reasoning. For multi-layer perceptron operation, It is a normalization operation.

[0050] To solve the problem of lost track caused by occlusion or temporary disappearance of the target vehicle, the motion predictor can be used to restore the track. It can predict the future position of the lost track by modeling the historical motion pattern of the target. During the tracking process, when the target cannot be detected due to occlusion or other reasons, the motion predictor predicts the future track by modeling the dynamic features of the target's past track, ensuring that the track can be reasonably continued and restored even when the target is lost. Through time tokenization and bidirectional encoding, the motion predictor can more accurately capture the dynamic behavior of the target and reduce tracking failures caused by occlusion or blur.

[0051] In real-world scenarios, objects may not be detected at certain points in time due to severe occlusion or motion blur. As a result, the trajectory of the target vehicle may not receive new updates during the matching process, resulting in early termination or breakage of the trajectory. The goal of this invention is to extend the trajectory that does not receive the target vehicle bounding box to enhance the consistency of the trajectory.

[0052] The video frame in which the target vehicle cannot be detected in the image is regarded as a lost track, and the bounding box is calculated starting from the frame in which the target vehicle can be recognized in the previous frame of the lost track. In S4, if the target vehicle is not recognized in the image of the current frame, this frame image is classified as a lost track.

[0053] Based on the time and time trajectory corresponding to the frame where the lost trajectory is located, the lost trajectory is repaired, and the repaired lost trajectory and the time trajectory are spliced ​​into a complete trajectory. The specific method for repairing the lost trajectory is: , in, is the lost trajectory of the target at time t+1, is the activity trajectory, is the activity trajectory of the target at time step t.

[0054] The deviation between the predicted trajectory of the current frame and the actual active trajectory of the current frame is calculated. If the deviation is less than the deviation threshold, the predicted trajectory and the active trajectory of the current frame are summarized to generate the time trajectory of the target vehicle. The time trajectory and the repaired lost trajectory are fused according to the time frame to obtain the complete trajectory of the target vehicle.

[0055] S5. Clustering is performed on the complete trajectories of the target vehicle from multiple cameras, and the complete trajectories of the target vehicle are merged to complete cross-camera vehicle tracking.

[0056] The center of the video coverage area of ​​multiple cameras is taken as the center of the circle, and the coverage area is divided into four sub-areas at a 90° central angle. The complete trajectory of the target vehicle is clustered according to the same sub-area, and the intersection of the complete trajectory of the target vehicle in every two cameras is searched. The complete trajectories with intersection are merged to complete cross-camera vehicle tracking.

[0057] For each pair of cameras containing n complete trajectories, each complete trajectory consists of 2048-dimensional average features of all frames, and an n×n similarity matrix is ​​calculated, where the value filled is the cosine similarity of the same target vehicle in each pair of cameras. This similarity matrix can be used to merge multiple trajectories across cameras. However, as the number of trajectories increases, the search space of the similarity matrix increases dramatically, which may lead to a large number of mismatches, thus affecting the accuracy of subsequent trajectories.

[0058] To improve the matching accuracy, direction and time information are introduced to limit the scope of the matching process. Through the position of each camera, the field of view of multiple cameras is rotated and translated so that the complete trajectory between every two cameras can be connected to form an overall area. The overall area is divided into multiple sub-areas, and the time range between every two cameras is calculated by combining the maximum and minimum speed limits and traffic light information. A trajectory restriction mask is constructed to effectively limit the search range and focus only on specific areas and times, thereby significantly improving the efficiency and accuracy of cross-camera matching.

[0059] Specifically, in a multi-camera environment, the complete trajectory of the vehicle may present a complex form due to differences in viewing angles and shooting times of different cameras.

[0060] Sub-region matching is performed according to the direction of the complete trajectory, and the trajectories of the same target vehicle are clustered through a reliable single camera. Only trajectories heading in the same direction are clustered, thereby effectively reducing the clustering space.

[0061] After the first clustering, the tracks between every two cameras will be merged. The clustering order is very important because the same car may be assigned different categories under different cameras. Therefore, multiple clustering is required to search for the intersection of tracks between every two cameras and merge the tracks with intersection. If there is no intersection, a new track set is created and the iterative search continues.

[0062] The present invention also provides a cross-camera multi-vehicle tracking system, comprising: Recognition module: obtains videos from multiple cameras, identifies the target vehicle in each frame of the video, and outputs the basic information of the target vehicle, including the category label, category confidence, and bounding box of the target vehicle; Re-identification module: The image of the identified target vehicle is divided into N non-overlapping patches according to the preset block size, and each patch is used as an image token; According to the category label of the target vehicle, a category token is assigned; Divide the target vehicle into the front, body and rear, and assign corresponding partial tokens; After global attention processing based on category tokens and image tokens, global features are output; Based on the partial tokens and image tokens, local features are output after local attention processing; Feature processing module: The historical features of each partial token form a fusion feature; compare the fusion feature with each local feature, calculate the similarity deviation function value with the local feature, sort from small to large, start selecting from the fusion feature with the smallest similarity deviation function value, and take the first threshold number of fusion features as the positive sample set; compare the fusion features in the positive sample set with the local features, calculate the similarity deviation function value with the local feature, sort from small to large, start selecting from the fusion feature with the smallest similarity deviation function value, take the second threshold number of fusion features as the similarity feature set, and add them to the global feature to generate a combined feature; Single-camera trajectory tracking module: In different frame images of the same video, the target vehicle is matched based on the combined features. If the target vehicle is identified, the coordinate set of the target vehicle's bounding box forms an activity trajectory, and the historical trajectory features of the target vehicle before the current frame are constructed. Each historical trajectory feature is composed of the center of the target vehicle's bounding box in the activity trajectory and the normalized size of the target vehicle's bounding box, which are converted into time tokens through linear transformation; the predicted trajectory of the target vehicle in the current frame is generated based on the time token reasoning, and the deviation between the predicted trajectory of the current frame and the actual activity trajectory of the current frame is calculated. If the deviation is less than the deviation threshold, the predicted trajectory and activity trajectory of the current frame are summarized to generate the time trajectory of the target vehicle, and further obtain the complete trajectory; Multi-camera trajectory tracking module: clusters the complete trajectories of target vehicles from multiple cameras, and merges the complete trajectories of target vehicles to complete cross-camera vehicle tracking.

[0063] A cross-camera multi-vehicle tracking device comprises a processor and a memory. When the processor executes a stored computer program, any one of the cross-camera multi-vehicle tracking methods described above is implemented.

Claims

1. A method for tracking multiple vehicles across cameras, characterized in that: Specifically include: S1. Obtain videos from multiple cameras, identify the target vehicle in each frame of the video, and output basic information of the target vehicle, including the category label, category confidence, and bounding box of the target vehicle; S2, splitting the image of the identified target vehicle into N non-overlapping patches according to the size of the preset block, and each patch serves as an image token; According to the category label of the target vehicle, a category token is assigned; Divide the target vehicle into the front, body and rear, and assign corresponding partial tokens; After global attention processing based on category tokens and image tokens, global features are output; Based on the partial tokens and image tokens, local features are output after local attention processing; S3. The historical features of each partial token form a fused feature; compare the fused feature with each local feature, calculate the similarity deviation function value with the local feature, sort them from small to large, start selecting from the fused feature with the smallest similarity deviation function value, and take the first threshold number of fused features as the positive sample set; compare the fused features in the positive sample set with the local features, calculate the similarity deviation function value with the local feature, sort them from small to large, start selecting from the fused feature with the smallest similarity deviation function value, take the second threshold number of fused features as the similarity feature set, and add them to the global feature to generate a combined feature; S4. In different frames of the same video, the target vehicle is matched based on the combined features. If the target vehicle is identified, the coordinate set of the target vehicle's bounding box forms an activity track, and the historical track features of the target vehicle before the current frame are constructed. Each historical track feature is composed of the center of the target vehicle's bounding box and the normalized size of the target vehicle's bounding box in the activity track, and is converted into a time token through a linear transformation; the predicted track of the target vehicle in the current frame is generated based on the time token reasoning, and the deviation between the predicted track of the current frame and the actual activity track of the current frame is calculated. If the deviation is less than the deviation threshold, the predicted track and the activity track of the current frame are summarized to generate the time track of the target vehicle, and further obtain the complete track; S5. Clustering is performed on the complete trajectories of the target vehicle from multiple cameras, and the complete trajectories of the target vehicle are merged to complete the cross-camera vehicle tracking.

2. A method for tracking multiple vehicles across cameras according to claim 1, characterized in that: The similarity deviation function is: ， in, Represents the similarity deviation function value, m is the fusion feature index, K is the number of samples in the data set, j is the index of the sample, pi represents the local area of ​​the image, is the local eigenvector, is the feature vector, is the temperature coefficient, and n is the number of fused features involved in the calculation.

3. The method for tracking multiple vehicles across cameras according to claim 1, characterized in that: In S4, if the target vehicle is not recognized in the image of the current frame, this frame of image is classified as a lost track.

4. The method for tracking multiple vehicles across cameras according to claim 3, characterized in that: Based on the time and time trajectory corresponding to the frame where the lost trajectory is located, the lost trajectory is repaired, and the repaired lost trajectory and the time trajectory are spliced ​​into a complete trajectory. The specific method for repairing the lost trajectory is: ， in, is the lost trajectory of the target at time t+1, is the activity trajectory, is the activity trajectory of the target at time step t.

5. The method for tracking multiple vehicles across cameras according to claim 1, characterized in that: The specific steps of generating the target vehicle predicted trajectory based on time token reasoning are as follows: ， ， ， ， in, is the output of the first reasoning, For the l -1 layer output, is the output of the second reasoning, is the fusion of the two outputs, For the l The output of the layer, is the operation module for the first reasoning, It is the operation module for the second reasoning. For multi-layer perceptron operation, It is a normalization operation.

6. The method for tracking multiple vehicles across cameras according to claim 1, characterized in that: The fused features form a memory library, and the memory library update formula is: ， in, is the fusion feature, pi represents the local area of ​​the image, t is the update round, m is the momentum coefficient, Represents the extraction function of local features, j is the index of the sample, j=1,…,K; K is the number of samples in the data set.

7. The method for tracking multiple vehicles across cameras according to claim 1, characterized in that: The specific steps of the global attention processing and the local attention processing are: Global attention processing: The category token is used as a query and compared with the key of all image tokens to calculate the attention weight, so as to perform weighted summation on all image tokens to obtain the global feature; Local attention processing: Partial tokens are used as queries, and the keys of image tokens corresponding to each partial token in the image are weighted to obtain local features.

8. The method for tracking multiple vehicles across cameras according to claim 1, characterized in that: The specific operation steps of S5 are: The center of the video coverage area of ​​multiple cameras is taken as the center of the circle, and the coverage area is divided into four sub-areas at a 90° central angle. The complete trajectory of the target vehicle is clustered according to the same sub-area, and the intersection of the complete trajectory of the target vehicle in every two cameras is searched. The complete trajectories with intersection are merged to complete cross-camera vehicle tracking.

9. A cross-camera multi-vehicle tracking system, characterized in that: include: Recognition module: obtains videos from multiple cameras, identifies the target vehicle in each frame of the video, and outputs the basic information of the target vehicle, including the category label, category confidence, and bounding box of the target vehicle; Re-identification module: The image of the identified target vehicle is divided into N non-overlapping patches according to the preset block size, and each patch is used as an image token; According to the category label of the target vehicle, a category token is assigned; Divide the target vehicle into the front, body and rear, and assign corresponding partial tokens; After global attention processing based on category tokens and image tokens, global features are output; Based on the partial tokens and image tokens, local features are output after local attention processing; Feature processing module: The historical features of each partial token form a fusion feature; compare the fusion feature with each local feature, calculate the similarity deviation function value with the local feature, sort from small to large, start selecting from the fusion feature with the smallest similarity deviation function value, and take the first threshold number of fusion features as the positive sample set; compare the fusion features in the positive sample set with the local features, calculate the similarity deviation function value with the local feature, sort from small to large, start selecting from the fusion feature with the smallest similarity deviation function value, take the second threshold number of fusion features as the similarity feature set, and add them to the global feature to generate a combined feature; Single-camera trajectory tracking module: In different frame images of the same video, the target vehicle is matched based on the combined features. If the target vehicle is identified, the coordinate set of the target vehicle's bounding box forms an activity trajectory, and the historical trajectory features of the target vehicle before the current frame are constructed. Each historical trajectory feature is composed of the center of the target vehicle's bounding box in the activity trajectory and the normalized size of the target vehicle's bounding box, which are converted into time tokens through linear transformation; the predicted trajectory of the target vehicle in the current frame is generated based on the time token reasoning, and the deviation between the predicted trajectory of the current frame and the actual activity trajectory of the current frame is calculated. If the deviation is less than the deviation threshold, the predicted trajectory and activity trajectory of the current frame are summarized to generate the time trajectory of the target vehicle, and further obtain the complete trajectory; Multi-camera trajectory tracking module: clusters the complete trajectories of target vehicles from multiple cameras, and merges the complete trajectories of target vehicles to complete cross-camera vehicle tracking.

10. A cross-camera multi-vehicle tracking device, characterized in that: The method comprises a processor and a memory, and when the processor executes a stored computer program, a cross-camera multi-vehicle tracking method as described in any one of claims 1 to 8 is implemented.

Images