A cross-camera multi-vehicle tracking method, system and device

Abstract

The application relates to the technical field of image feature extraction, in particular to a cross-camera multi-vehicle tracking method, system and device, which comprises the following steps: acquiring videos of multiple cameras, identifying target vehicles in each frame of image in the videos, and outputting basic information of the target vehicles; assigning image tokens, category tokens and part tokens to the images; fusing features of historical features of each part token to generate combined features after adding global features; constructing historical track features of the target vehicles before a current frame, and converting the historical track features into time tokens through linear transformation; generating a predicted track of the target vehicles in the current frame according to the time tokens, and further reasoning to obtain a complete track; and clustering the complete tracks of the target vehicles of the multiple cameras to complete cross-camera vehicle tracking. The application can solve the problems of cross-camera matching difficulty, occlusion and motion blur, track repair and vehicle re-identification precision in vehicle tracking in a multi-camera scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image feature extraction, in particular to a cross-camera multi-vehicle tracking method, system and device. BACKGROUND

[0002] With the rapid development of intelligent transportation systems and autonomous driving technologies, traffic monitoring and management have become increasingly important in complex urban environments. Vehicle target tracking, as one of the core tasks in intelligent transportation systems, has become a research hotspot. Traditional vehicle tracking methods usually rely on a single camera to obtain target information, and are prone to misjudgment when dealing with complex situations such as occlusion, motion blur and target intersection. However, in real-world applications, multi-camera video streams provide more rich perspectives and information, which helps to overcome the limitations of single-camera tracking, especially in large-scale urban monitoring.

[0003] However, multi-camera vehicle tracking still faces difficulties in cross-camera target matching, occlusion and motion blur problems, trajectory repair and prediction, and vehicle re-identification accuracy.

[0004] Therefore, the existing multi-camera vehicle tracking method still has many deficiencies in accuracy, efficiency and stability. SUMMARY

[0005] To solve the technical problems existing in the background art, the present application provides a cross-camera multi-vehicle tracking method, system and device.

[0006] The present application provides a cross-camera multi-vehicle tracking method, and the technical solution is as follows:

[0007] A cross-camera multi-vehicle tracking method, specifically comprising:

[0008] S1, obtaining videos of multiple cameras, identifying target vehicles in each frame of image in the video, and outputting basic information of the target vehicles, including class label, class confidence and bounding box of the target vehicles;

[0009] S2, dividing the image in which the target vehicle is identified into N non-overlapping patches according to the size of the preset block, and taking each patch as an image token;

[0010] According to the class label of the target vehicle, a class token is assigned;

[0011] The target vehicle is divided into a head, a body and a tail, and corresponding part tokens are assigned;

[0012] Based on the class token and the image token, global attention processing is performed, and global features are outputted;

[0013] Based on the partial token and the image token, local attention processing is performed, and local features are output;

[0014] S3, the history features of each partial token form a fusion feature; the fusion feature is compared with each local feature to calculate a similarity deviation function value with the local feature, and the fusion features are sorted from small to large, and the fusion features with the smallest similarity deviation function value are selected as a positive sample set; the fusion features in the positive sample set are compared with the local features to calculate a similarity deviation function value with the local features, and the fusion features are sorted from small to large, and the fusion features with the smallest similarity deviation function value are selected as a similarity feature set, and the similarity feature set is added to the global feature to generate a combined feature;

[0015] S4, in different frame images of the same video, the target vehicle is matched based on the combined feature, if the target vehicle is recognized, the coordinate set of the target vehicle bounding box forms an active track, and a history track feature of the target vehicle before the current frame is constructed, each history track feature is composed of the target vehicle bounding box center and the target vehicle bounding box size normalization in the active track, and is converted into a time token through linear transformation; the prediction track of the target vehicle in the current frame is generated according to the time token, the deviation between the prediction track of the current frame and the actual active track of the current frame is calculated, if the deviation is less than a deviation threshold, the prediction track and the active track of the current frame are summarized to generate a time track of the target vehicle, and a complete track is further obtained;

[0016] S5, the complete tracks of the target vehicles of multiple cameras are clustered and processed, and the complete tracks of the target vehicles are merged to complete cross-camera vehicle tracking.

[0017] The similarity deviation function is:

[0018] ,

[0019] wherein, represents the similarity deviation function value, m is the fusion feature index, K is the number of samples in the data set, j is the index of the sample, pi represents the local area of the image, is a local feature vector, is a feature vector, is a temperature coefficient, and n is the number of fusion features participating in the calculation.

[0020] In S4, if the target vehicle is not recognized in the image of the current frame, the frame image is divided into a missing track.

[0021] Based on the time corresponding to the missing track and the time track, the missing track is repaired, and the repaired missing track and the time track are spliced into a complete track, and the specific method for repairing the missing track is:

[0022] ,

[0023] wherein, is the target lost trajectory at time t+1, is the active trajectory, is the active trajectory of the target at time step t.

[0024] The target vehicle prediction trajectory is generated according to the time token reasoning, and the specific steps are as follows:

[0025] ,

[0026] ,

[0027] ,

[0028] ,

[0029] wherein, is the output of the first reasoning, is the output of the first l -1 layer, is the output of the second reasoning, is the fusion of the two outputs, is the output of the first l layer, is the operation module of the first reasoning, is the operation module of the second reasoning, is the multi-layer perception operation, is the normalization operation.

[0030] The fusion feature constitutes a memory bank, and the memory bank update formula is:

[0031] ,

[0032] wherein, is the fusion feature, pi represents a local region of the image, t is the update round, m is the momentum coefficient, represents the extraction function of the local feature, j is the index of the sample, j=1,…,K; K is the number of samples in the data set.

[0033] The specific steps of global attention processing and local attention processing are as follows:

[0034] Global attention processing: the category token is taken as a query, and the attention weight is calculated by comparing with the key of all image tokens, so as to weight sum all image tokens to obtain the global feature;

[0035] Local attention processing: part tokens are taken as queries, and the keys of the image tokens corresponding to each part token in the image are weighted to obtain local features.

[0036] The specific operation steps of S5 are:

[0037] The centers of the video coverage areas of the multiple cameras are taken as the centers of the circles, and the coverage areas are divided into four sub-areas with a 90° central angle. The complete trajectories of the target vehicles are clustered according to the same sub-areas, the intersection of the complete trajectories of the target vehicles in each two cameras is searched, and the complete trajectories with the intersection are merged to complete the cross-camera vehicle tracking.

[0038] A cross-camera multi-vehicle tracking system, comprising:

[0039] The recognition module: acquires videos of multiple cameras, identifies target vehicles in each frame of image in the video, and outputs basic information of the target vehicles, including the category label, the category confidence and the bounding box of the target vehicles;

[0040] The re-identification module: divides the image in which the target vehicle is recognized into N non-overlapping patches according to the size of the preset block, and each patch is taken as an image token;

[0041] According to the category label of the target vehicle, a category token is assigned;

[0042] The target vehicle is divided into a vehicle head, a vehicle body and a vehicle tail, and corresponding part tokens are assigned;

[0043] Based on the category token and the image token, global attention processing is performed, and global features are output;

[0044] Based on the part token and the image token, local attention processing is performed, and local features are output;

[0045] The feature processing module: the historical features of each part token form a fusion feature; the fusion feature and each local feature are compared to calculate the similarity deviation function value of the local feature, and the fusion features are sorted from small to large, and the fusion features with the smallest similarity deviation function value are selected as a positive sample set; the fusion features in the positive sample set and the local features are compared to calculate the similarity deviation function value of the local feature, and the fusion features are sorted from small to large, and the fusion features with the smallest similarity deviation function value are selected as a similar feature set, and the similar feature set and the global features are added to generate a combined feature;

[0046] Single camera trajectory tracking module: in different frame images of the same video, the target vehicle is matched based on the combined features, if the target vehicle is identified, the coordinate set of the target vehicle bounding box forms the active trajectory, the historical trajectory features before the current frame of the target vehicle are constructed, each historical trajectory feature is composed of the target vehicle bounding box center and the target vehicle bounding box size normalization, and is converted into a time token through linear transformation; the prediction trajectory of the target vehicle current frame is generated according to the time token reasoning, the deviation between the prediction trajectory of the current frame and the actual active trajectory of the current frame is calculated, if the deviation is less than the deviation threshold, the prediction trajectory and the active trajectory of the current frame are summarized to generate the time trajectory of the target vehicle, and the complete trajectory is further obtained;

[0047] Multi-camera trajectory tracking module: the complete trajectories of the target vehicles of multiple cameras are clustered and processed, and the complete trajectories of the target vehicles are merged to complete the cross-camera vehicle tracking.

[0048] A cross-camera multi-vehicle tracking device, comprising a processor and a memory, wherein the processor can implement the cross-camera multi-vehicle tracking method of any one of the above steps when executing the saved computer program.

[0049] The beneficial effects of the present application are:

[0050] By acquiring videos of multiple cameras and identifying the target vehicle in each frame image of the video, the basic information of the target vehicle can be accurately output, including the category label, the category confidence and the bounding box. This provides a reliable data basis for subsequent vehicle tracking.

[0051] The image in which the target vehicle is identified is segmented into non-overlapping patches, and category tokens and part tokens are assigned, which helps to capture the features of the vehicle more finely in subsequent attention processing. This detailed segmentation and token assignment improves the accuracy and efficiency of feature extraction.

[0052] Global attention processing is performed based on the category token and the image token to output global features, and local attention processing is performed based on the part token and the image token to obtain local features. This attention mechanism combining global and local information can more comprehensively capture the features of the vehicle, improving the accuracy and robustness of vehicle tracking.

[0053] By calculating the similarity deviation function value of the fusion features and the local features, the positive sample set and the similar feature set are screened out, and the combined features are generated by adding the global features. This intelligent positive sample screening and feature fusion method helps to accurately identify the target vehicle in a complex environment.

[0054] Based on the combination feature matching target vehicle, the historical trajectory feature of the target vehicle is constructed, and the predicted trajectory is generated through time token reasoning. This efficient trajectory construction and prediction method can track the motion trajectory of the target vehicle in real time, and provides strong support for cross-camera vehicle tracking.

[0055] The complete trajectory of the target vehicle of multiple cameras is clustered and processed, and the complete trajectory is merged to complete the cross-camera vehicle tracking. This cross-camera tracking method can realize continuous tracking of the target vehicle between different cameras, and provides important technical support for intelligent transportation systems, security monitoring and other fields. DETAILED DESCRIPTION

[0056] The technical scheme of the present application is as follows:

[0057] A cross-camera multi-vehicle tracking method, specifically comprising:

[0058] S1, acquiring video of multiple cameras, identifying target vehicles in each frame of image in the video, and outputting basic information of the target vehicles, including class label, class confidence and bounding box of the target vehicles.

[0059] The obtained video stream is decoded and split into continuous image frames.

[0060] Each frame of image is preprocessed, including denoising and histogram equalization, etc., to improve the image quality.

[0061] Each frame of image is identified by using a deep learning model.

[0062] The bounding box of the target vehicle and the corresponding class label and class confidence are outputted, and the basic information of the target vehicle, including the class label, the class confidence and the bounding box, is extracted from the output.

[0063] The present application adopts YOLOv8 network to realize target identification. YOLOv8 is a high-efficiency convolutional neural network specially designed for fast and accurate real-time identification tasks, and its high efficiency and high precision are suitable for processing vehicle identification tasks in multiple camera video streams.

[0064] YOLOv8 can extract features at multiple scales and perform target classification and bounding box regression. By training and reasoning on images of different resolutions, YOLOv8 can more accurately detect targets of different sizes. Compared with previous versions, YOLOv8 has been optimized in terms of accuracy and speed, especially in complex background or multi-target scenarios, the detection accuracy has been significantly improved, and the running memory requirement of the processor is reasonable.

[0065] In practical applications, YOLOv8 first receives each frame of image from a multi-camera video stream, performs target vehicle recognition on each frame of image, and outputs the class label, bounding box, and class confidence of the target vehicle for each detected target vehicle.

[0066] After preprocessing, the image is input into the YOLOv8 network, and the features are extracted through the backbone network. YOLOv8 identifies the target in the image through its detection head, generates candidate boxes, and removes duplicate bounding boxes using the non-maximum suppression algorithm. The target vehicle is recognized, and the bounding box coordinates and class confidence of each vehicle are further generated. YOLOv8 further classifies according to the class recognition and outputs the specific class label of the target, including car, truck, bus, motorcycle, and their bounding box coordinates.

[0067] Whenever a bounding box is generated, the YOLOv8 network directly passes it to the re-identification module, which receives the bounding box information and performs re-identification processing on each vehicle target.

[0068] S2, the image in which the target vehicle is recognized is divided into N non-overlapping patches according to the size of the preset block, and each patch is used as an image token;

[0069] According to the class label of the target vehicle, a class token is assigned;

[0070] The target vehicle is divided into a head, a body, and a tail, and corresponding part tokens are assigned;

[0071] Based on the class token and the image token, global attention processing is performed to output global features;

[0072] Based on the part token and the image token, local attention processing is performed to output local features.

[0073] The core innovation of this step is to effectively capture the local area features of the vehicle by introducing part tokens, and to construct multi-dimensional feature representations together with class tokens. In order to capture the fine-grained differences of vehicles, each vehicle is divided into three main parts: head, body, and tail. This division can help the model focus on the visual features of different parts of the vehicle, thereby better addressing the differences between different vehicle models. However, the body design of different vehicle models varies greatly, and even different shooting angles can cause differences in local features of the head, body, and tail. Therefore, a dynamic adjustment mechanism is introduced during division to adaptively adjust the selection of local areas according to the structural characteristics of different vehicle models.

[0074] The module capable of achieving S2 comprises global attention and local attention. The global attention is processed by the category token and all image tokens, so as to extract global features. The local attention is processed by the part token and the image token of the local region in the image, so as to extract the features of the specific local region.

[0075] The specific steps of the global attention processing and the local attention processing are as follows:

[0076] Global attention processing: the category token is taken as a query, the key of all image tokens is compared to calculate the attention weight, all image tokens are weighted and summed to obtain global features;

[0077] Local attention processing: the part token is taken as a query, the key of the image token corresponding to each part token in the image is weighted to obtain local features.

[0078] The application improves the feature extraction capability of the target vehicle by combining the global and local attention mechanisms of the re-identification module.

[0079] Based on the input image and the corresponding bounding box information, local and global features are extracted, the appearance features of the vehicle are encoded through a multi-layer attention mechanism, and a vehicle feature vector with high recognition is generated. These feature vectors are not only used for vehicle identity recognition, but also passed to the next step to solve the association problem of the target vehicle under different camera perspectives.

[0080] Through this multi-level feature learning, not only global information can be obtained, but also fine-grained local features of the target vehicle can be captured in detail, so as to improve the vehicle recognition accuracy in complex scenes, effectively improve the feature expression capability in the re-identification task, and reduce the overfitting problem.

[0081] S3, the historical features of each part token form a fusion feature; the fusion feature is compared with each local feature to calculate the similarity deviation function value of the local feature, the fusion features are selected from small to large, and the first threshold number of fusion features are taken as a positive sample set; the fusion features in the positive sample set are compared with the local features to calculate the similarity deviation function value of the local features, the fusion features are selected from small to large, and the second threshold number of fusion features are taken as a similar feature set, and the combination feature is generated after adding the global feature.

[0082] In order to maximize the similarity learning of the local features, a momentum updated memory bank is maintained for the historical features of each part token The feature stored in the memory library is called fusion feature, and the feature information of different parts is gradually accumulated by momentum update of the memory library in each similarity training cycle.

[0083] The memory library is composed of fusion features, and the update formula of the memory library is:

[0084] ,

[0085] Wherein, is the fusion feature, pi represents a local region of the image, t is the update round, and m is a momentum coefficient, is an extraction function of the local feature, j is the index of the sample, j = 1,…, K; K is the number of samples in the data set.

[0086] The fusion feature is compared with each local feature, and the first threshold number of fusion features is taken as the positive sample set from the fusion feature with the minimum similarity deviation function value of the local feature. The fusion features in the positive sample set are compared with the local features, and the second threshold number of fusion features is taken as the similar feature set, and the combined feature is generated after being added to the global feature.

[0087] The similarity deviation function is:

[0088] ,

[0089] Wherein, is the similarity deviation function value, m is the index of the fusion feature selected from the memory library, K is the number of samples in the data set, j is the index of the sample, and pi represents a local region of the image, is a local feature vector, is a feature vector in the memory library, is a temperature coefficient, and n is the number of memory library fusion features participating in the calculation.

[0090] After S3 processing, a feature vector with high recognition is output, which contains the combined feature of the target vehicle.

[0091] S4, in different frame images of the same video, a target vehicle is matched based on the combined features, if the target vehicle is identified, a coordinate set of a target vehicle bounding box forms an active track, a historical track feature of the target vehicle before a current frame is constructed, each historical track feature is composed of a target vehicle bounding box center and a target vehicle bounding box size normalization, and is converted into a time token through linear transformation; a predicted track of the target vehicle in the current frame is generated according to the time token reasoning, a deviation between the predicted track of the current frame and an actual active track of the current frame is calculated, if the deviation is less than a deviation threshold, the predicted track of the current frame and the active track are summarized to generate a time track of the target vehicle, and a complete track is further obtained.

[0092] For the target vehicle, a combined feature of the target vehicle is generated, and an identity label is further assigned to each bounding box. The identity label is passed to S4 to ensure that the same target vehicle can be accurately identified and matched in different times and different camera perspectives.

[0093] The specific method for obtaining the target vehicle track is that each track is represented as a bounding box sequence of the target vehicle in consecutive time frames, and as the video proceeds, new targets may enter the scene and existing targets may exit the scene. In order to handle such dynamic changes, the track is further divided into an active track and a lost track, which respectively represent the currently active track and the temporarily lost track.

[0094] The predicted track of the current frame can be achieved by using the Mamba motion predictor, which includes three main components. The first component is an input embedding layer, which takes the historical dynamics of the object track as input to construct historical track features representing the changes in the past positions of the target vehicle. Each track feature is composed of the center and size normalization of the bounding box, and these features are converted into time tokens through linear transformation. The second component is an encoding layer composed of l bidirectional Mamba modules, and the Mamba module is the core. In order to better capture the dynamic behavior of the target, the Mamba module adopts a bidirectional encoder structure, which can fully utilize the historical track features of the active track. Each encoding layer includes forward and backward Mamba modules for forward and backward processing of the active track to obtain a comprehensive motion pattern. The last layer is a prediction head responsible for predicting the inter-frame bounding box offset of the object track.

[0095] The specific steps for generating the predicted track of the target vehicle according to the time token reasoning are as follows:

[0096] ,

[0097] ,

[0098] ,

[0099] ,

[0100] wherein, is the output of the first inference, i.e. the forward processing, is the output of the first inference, l is the output of the first inference, is the output of the second inference, i.e. the backward processing, is the fusion of the two outputs, is the output of the first inference, l is the output of the first inference, is the operation module of the first inference, is the operation module of the second inference, is the multi-layer perception operation, is the normalization operation.

[0101] For the problem of trajectory loss caused by target vehicle occlusion or temporary disappearance, trajectory repair can be achieved through a motion predictor, which can predict the future position of the lost trajectory by modeling the historical motion pattern of the target. During tracking, when the target cannot be detected due to occlusion or other reasons, the motion predictor predicts the future trajectory by modeling the dynamic characteristics of the past trajectory of the target, ensuring that the trajectory can be reasonably continued and repaired even in the case of target loss. Through time tokenization and bidirectional encoding, the motion predictor can more accurately capture the dynamic behavior of the target, reducing tracking failures caused by occlusion or blur.

[0102] In real-world scenarios, due to severe occlusion or motion blur, objects may not be detected at certain time points. Therefore, the trajectory of the target vehicle may not receive new updates during the matching process, resulting in premature termination or breakage of the trajectory. The goal of the present invention is to extend the trajectory that has not received the target vehicle bounding box to enhance the consistency of the trajectory.

[0103] The video frame in which the target vehicle cannot be detected in the image is taken as the lost trajectory, and the bounding box is calculated starting from the frame in which the target vehicle can be identified before the lost trajectory. In S4, if the target vehicle cannot be identified in the current frame of the image, the frame image is divided into a lost trajectory.

[0104] Based on the time corresponding to the lost trajectory and the time trajectory, the lost trajectory is repaired, and the repaired lost trajectory and the time trajectory are spliced into a complete trajectory. The specific method for repairing the lost trajectory is:

[0105] ,

[0106] wherein, is the lost trajectory of the target at time t+1, is the active trajectory, targeting the activity trajectory at time step t.

[0107] calculating the deviation of the predicted trajectory of the current frame and the actual activity trajectory of the current frame, if the deviation is less than the deviation threshold, then the predicted trajectory and the activity trajectory of the current frame are aggregated to generate a time trajectory of the target vehicle, and the time trajectory and the patched missing trajectory are fused according to the time frame to obtain a complete trajectory of the target vehicle.

[0108] S5, clustering the complete trajectories of the target vehicle of the plurality of cameras, and merging the complete trajectories of the target vehicle to complete cross-camera vehicle tracking.

[0109] Taking the center of the video coverage area of the plurality of cameras as the center of the circle, dividing the coverage area into four sub-areas with a 90° central angle, clustering the complete trajectories of the target vehicle according to the same sub-area, searching for the intersection of the complete trajectories of the target vehicle in each two cameras, and merging the complete trajectories with intersection to complete cross-camera vehicle tracking.

[0110] For n complete trajectories contained in each two cameras, each complete trajectory is composed of all frame average features of 2048 dimensions, and an n x n similarity matrix is calculated, wherein the filled value is the cosine similarity of the same target vehicle in each two cameras. The similarity matrix can be used to merge multiple trajectories across cameras. However, as the number of trajectories increases, the search space of the similarity matrix increases dramatically, which may cause a large number of false matches, thereby affecting the subsequent trajectory accuracy.

[0111] In order to improve the matching accuracy, the direction and time information are introduced to limit the range of the matching process. By the position of each camera, the fields of view of multiple cameras are rotated and translated, so that the complete trajectories between each two cameras can be connected to form a total area. The total area is divided into multiple sub-areas, and the maximum and minimum speed limits and traffic signal information are combined to calculate the time range between each two cameras. A trajectory restriction mask is constructed to effectively limit the search range and only focus on specific areas and times, thereby significantly improving the matching efficiency and accuracy across cameras.

[0112] Specifically, in a multi-camera environment, the complete trajectories of vehicles may present complex forms due to the differences in the angles of view and shooting times of different cameras.

[0113] According to the direction of the complete trajectory, the sub-area matching is performed, and the trajectories of the same target vehicle of a reliable single camera are clustered, only the trajectories in the same direction are clustered, thereby effectively reducing the clustering space.

[0114] After the first clustering, the trajectories between every two cameras will be merged, and the clustering order is very important because the same vehicle under different cameras can be assigned different categories. Therefore, multiple clustering is needed to search the intersection of trajectories between every two cameras and merge the trajectories with intersection. If there is no intersection, a new trajectory set is created and the iteration search continues.

[0115] The application also provides a cross-camera multi-vehicle tracking system, comprising:

[0116] The identification module: acquires videos of multiple cameras, identifies target vehicles in each frame of image in the video, and outputs basic information of the target vehicles, including category label, category confidence and bounding box of the target vehicles;

[0117] The re-identification module: splits the image of the identified target vehicle into N non-overlapping patches according to the size of the preset block, and takes each patch as an image token;

[0118] According to the category label of the target vehicle, a category token is assigned;

[0119] The target vehicle is divided into a head, a body and a tail, and corresponding part tokens are assigned;

[0120] Based on the category token and the image token, global attention processing is performed, and global features are outputted;

[0121] Based on the part token and the image token, local attention processing is performed, and local features are outputted;

[0122] The feature processing module: the historical features of each part token form a fusion feature; the fusion feature is compared with each local feature to calculate the similarity deviation function value of the local feature, and the fusion features are sorted from small to large, and the first threshold number of fusion features are selected as a positive sample set from the fusion feature with the smallest similarity deviation function value; the fusion features in the positive sample set are compared with the local features to calculate the similarity deviation function value of the local features, and the fusion features are sorted from small to large, and the second threshold number of fusion features are selected as a similar feature set from the fusion feature with the smallest similarity deviation function value, and the combined features are generated by adding the global features and the similar feature set;

[0123] Single camera track tracking module: in different frame images of the same video, the target vehicle is matched based on combined features, if the target vehicle is identified, the coordinate set of the target vehicle bounding box forms an active track, the historical track features before the current frame of the target vehicle are constructed, each historical track feature is composed of the target vehicle bounding box center and the target vehicle bounding box size normalization in the active track, and is converted into a time token through linear transformation; the prediction track of the target vehicle current frame is generated according to the time token reasoning, the deviation between the prediction track of the current frame and the actual active track of the current frame is calculated, if the deviation is less than the deviation threshold, the prediction track and the active track of the current frame are summarized to generate the time track of the target vehicle, and the complete track is further obtained;

[0124] Multi-camera track tracking module: the complete tracks of the target vehicles of multiple cameras are clustered and processed, and the complete tracks of the target vehicles are merged to complete cross-camera vehicle tracking.

[0125] A cross-camera multi-vehicle tracking device, comprising a processor and a memory, wherein the processor implements any one of the cross-camera multi-vehicle tracking methods as described above when executing the saved computer program.

Claims

1. A cross-camera multi-vehicle tracking method, characterized in that, Specifically comprising: S1, acquiring videos of multiple cameras, identifying target vehicles in each frame of image in the videos, and outputting basic information of the target vehicles, including category label, category confidence and bounding box of the target vehicles; S2, dividing the images in which the target vehicles are identified into N non-overlapping patches according to a preset block size, and taking each patch as an image token; According to the category label of the target vehicle, a category token is assigned; The target vehicle is divided into a vehicle head, a vehicle body and a vehicle tail, and corresponding part tokens are assigned; Based on the category token and the image token, global attention processing is performed, and global features are outputted; Based on the part token and the image token, local attention processing is performed, and local features are outputted; S3, the historical features of each part token form a fusion feature; the fusion feature is compared with each local feature to calculate a similarity deviation function value with the local feature, and the fusion features are sorted from small to large, and the first threshold number of fusion features are taken as a positive sample set from the fusion feature with the smallest similarity deviation function value; the fusion features in the positive sample set are compared with the local features to calculate a similarity deviation function value with the local features, and the fusion features are sorted from small to large, and the second threshold number of fusion features are taken as a similar feature set, and the similar feature set is added to the global features to generate a combined feature; S4, in different frames of images of the same video, the target vehicle is matched based on the combined feature, if the target vehicle is identified, a coordinate set of the target vehicle bounding box forms an active track, a historical track feature of the target vehicle before the current frame is constructed, each historical track feature is composed of a target vehicle bounding box center and a target vehicle bounding box size normalization in the active track, and is converted into a time token through linear transformation; a prediction track of the target vehicle in the current frame is generated according to the time token, a deviation between the prediction track of the current frame and the actual active track of the current frame is calculated, if the deviation is less than a deviation threshold, the prediction track and the active track of the current frame are summarized to generate a time track of the target vehicle, and a complete track is further obtained; S5, the complete tracks of the target vehicles of multiple cameras are clustered and processed, and the complete tracks of the target vehicles are merged to complete cross-camera vehicle tracking.

2. The cross-camera multi-vehicle tracking method of claim 1, wherein, In S4, if the target vehicle is not identified in the image of the current frame, the frame of image is divided into a missing track.

3. The cross-camera multi-vehicle tracking method of claim 2, wherein, Based on the time corresponding to the frame where the missing track is located and the time track, the missing track is repaired, and the repaired missing track and the time track are spliced into a complete track, and the specific method for repairing the missing track is: ， wherein, is the lost trajectory for the target at time t+1, is the active trajectory, is the active trajectory for the target at time step t.

4. The cross-camera multi-vehicle tracking method of claim 1, wherein, The specific steps of generating the prediction track of the target vehicle according to the time token are: ， ， ， ， wherein, is the output of the first inference, is the output of the first inference, l is the output of the -1 layer, is the output of the second inference, is the fusion of the two outputs, is the output of the first inference, l is the output of the -1 layer, is the operation module of the first inference, is the operation module of the second inference, is the multi-layer perception operation, is the normalization operation.

5. The cross-camera multi-vehicle tracking method of claim 1, wherein, The fusion features form a memory bank, and the memory bank updating formula is: ， wherein, pi is the local region of the image, t is the update round, m is the momentum coefficient, is the extraction function of local features, j is the index of the sample, j = 1, …, K; K is the number of samples in the dataset.

6. The cross-camera multi-vehicle tracking method of claim 1, wherein, The specific steps of the global attention processing and the local attention processing are: Global attention processing: the category token is taken as a query, and the attention weight is calculated by comparing with the key of all image tokens, so as to weight sum all image tokens to obtain global features; Local attention processing: the part token is taken as a query, and the key of the image token corresponding to each part token in the image is weighted to obtain local features.

7. The cross-camera multi-vehicle tracking method of claim 1, wherein, The specific operation steps of S5 are: Taking the center of the video coverage area of the multiple cameras as the center of the circle, dividing the coverage area into four sub-areas with a 90° central angle, clustering the complete trajectories of the target vehicles according to the same sub-area, searching for the intersection of the complete trajectories of the target vehicles in each two cameras, and merging the complete trajectories with the intersection to complete the cross-camera vehicle tracking.

8. A cross-camera multi-vehicle tracking system, comprising: Comprise: The identification module: acquires videos of multiple cameras, identifies target vehicles in each frame of image in the videos, and outputs basic information of the target vehicles, including a category label, a category confidence and a bounding box of the target vehicles; The re-identification module: divides the images in which the target vehicles are identified into N non-overlapping patches according to a preset block size, and takes each patch as an image token; According to the category label of the target vehicle, a category token is assigned; The target vehicle is divided into a vehicle head, a vehicle body and a vehicle tail, and corresponding part tokens are assigned; Based on the category token and the image token, global attention processing is performed, and global features are output; Based on the part token and the image token, local attention processing is performed, and local features are output; The feature processing module: the historical features of each part token form fusion features; the fusion features are compared with each local feature to calculate a similarity deviation function value of the local feature, the fusion features are sorted in ascending order of the similarity deviation function value, and a first threshold number of fusion features with the smallest similarity deviation function value are taken as a positive sample set; the fusion features in the positive sample set are compared with the local feature to calculate a similarity deviation function value of the local feature, the fusion features are sorted in ascending order of the similarity deviation function value, and a second threshold number of fusion features with the smallest similarity deviation function value are taken as a similar feature set, and the similar feature set is added to the global features to generate combined features; The single-camera trajectory tracking module: in different frame images of the same video, the target vehicle is matched based on the combined features, if the target vehicle is identified, a coordinate set of the target vehicle bounding box forms an active trajectory, a historical trajectory feature of the target vehicle before the current frame is constructed, each historical trajectory feature is composed of a target vehicle bounding box center and a target vehicle bounding box size normalization, and is converted into a time token through linear transformation; a prediction trajectory of the target vehicle in the current frame is generated according to the time token, a deviation between the prediction trajectory of the current frame and the actual active trajectory of the current frame is calculated, if the deviation is less than a deviation threshold, the prediction trajectory of the current frame and the active trajectory are combined to generate a time trajectory of the target vehicle, and a complete trajectory is further obtained; The multi-camera trajectory tracking module: the complete trajectories of the target vehicles of multiple cameras are clustered and processed, and the complete trajectories of the target vehicles are merged to complete the cross-camera vehicle tracking.

9. A cross-camera multi-vehicle tracking device, comprising: Comprise a processor and a memory, the processor executes the saved computer program to realize the cross-camera multi-vehicle tracking method in any one of claims 1-7.

Images