A cross-view target association tracking method applied to a multi-camera system

Abstract

The present application relates to a kind of cross-view target association tracking method applied to multi-camera system, belong to the research field of multidisciplinary such as computer vision and pattern recognition.For the low target tracking accuracy caused by view difference, occlusion and insufficient feature expression in multi-camera environment, a solution combining deep learning and time-spatial context feature association is proposed.Specifically, it includes: first, obtaining a multi-view target dataset as training and testing samples;Using a pre-trained deep learning model to extract the visual features of the target, and generating a distinctive target description through global feature fusion;Then, the trajectory optimization matching and prediction are carried out through the time-spatial correlation algorithm, realizing the accurate association of cross-time frame and cross-view target;Finally, the matching accuracy and robustness are optimized by combining the target search loss function.This method is widely used in video surveillance, crowd analysis and other scenarios, and promotes the application of multi-camera system in real scenarios.

Description

TECHNICAL FIELD

[0001] The present application relates to a cross-view target association tracking method applied to a multi-camera system, belonging to the research field of computer vision and pattern recognition and other multi-disciplinary fields. The method realizes continuous tracking and identity maintenance of the target by integrating the view information of different cameras, and has important application value in the fields of intelligent monitoring, automatic driving and human-computer interaction. Due to the inherent feature difference between views, the calculation of target similarity becomes complex. Therefore, the present application introduces a collaborative association network combining time and space clues. The method fuses cross-view and cross-frame features to obtain comprehensive global feature representation of each object, and represents data association as a hybrid constraint optimization problem, which significantly enhances the scalability of the method. In addition, we propose a new object search loss function to improve the cross-view and cross-frame data association ability. The method does not need to pre-calibrate and calibrate the camera, greatly improves the deployment efficiency of the multi-camera system, and can be widely applied to intelligent monitoring, crowd analysis and other scenes. It is worth noting that existing multi-target multi-camera tracking methods usually require additional visual annotation information, increasing the labor cost. The present application does not need to pre-calibrate and calibrate the camera, greatly improves the deployment efficiency of the multi-camera system, and can promote the application of multi-target multi-camera tracking system in real scenes. BACKGROUND

[0002] Real-world demands are driving the rapid development of multi-object tracking algorithms. The effectiveness of these algorithms largely depends on the accuracy of the detector. Therefore, many high-precision tracking algorithms adopt the "detect and track" paradigm, relying on object detection to locate objects in each frame. Subsequently, association algorithms such as the Hungarian algorithm are used to connect objects with the same identity between adjacent frames, forming trajectories. However, traditional linear motion model-based algorithms significantly decrease in effectiveness when objects move non-linearly or irregularly. In contrast, non-linear motion model-based algorithms perform better in position prediction. For example, the Observation-Centric SORT method improves tracking performance for non-linear moving objects by redesigning the motion model. Although the above algorithms are effective in most cases, their performance drops dramatically when large-scale occlusion or frame loss occurs. To solve this problem, multi-view algorithms are proposed to capture the representation and motion information of each object from different angles. Studies have shown that objects with the same identity exhibit similar representation information under different perspectives, while objects with different identities exhibit diversity. Therefore, the current challenge is to develop a similarity measure based on global information to distinguish between different objects. In addition, fusing information from multiple perspectives can generate independent features for each object, thereby improving the accuracy of multi-object tracking. For example, the multi-view multi-object collaborative tracking method based on dynamic occlusion threshold effectively solves the problem of target inconsistency and size change in traditional tracking algorithms by introducing an occlusion variable and a dynamic occlusion threshold. Even in the case of occlusion, the method still maintains good tracking performance. In summary, multi-object tracking algorithms that combine non-linear motion models and multi-view information fusion can more effectively handle complex situations in real-world scenarios, such as non-linear motion, occlusion, and frame loss, improving the robustness and accuracy of tracking.

[0003] Cross-view target association tracking of multi-camera systems is to solve the challenge of limited access to object information from independent views. In recent years, there have been various formulations and solutions based on multi-view association and tracking. Liu et al. proposed a unified probabilistic framework and a Markov chain-based method to solve the cross-view target tracking problem. Xu et al. used the consistency of appearance and motion information to solve the target tracking problem in multiple horizontal views. Similarly, Han et al. combined top and horizontal views by creating spatial location relationships and merging appearance information. They formulated the association problem as a mixed integer programming problem. Han also proposed a combination of GNN and RNN to solve the association problem across multiple views. This method uses existing re-identification networks to extract human features to obtain more distinctive identity embeddings. However, it does not consider the fusion of global information, which is crucial to our method. Gan et al. have shown in 2021 that the same object exhibits symmetrical and transitive similarities in different views or frames. This is particularly important when building spatio-temporal relationships between multiple objects in arbitrary views. In addition, Xu et al. require the number and location of cameras to be determined in advance and calibrated. In contrast, the method proposed by us only uses video stream segments from cameras at different angles to achieve multi-view association and tracking. SUMMARY

[0004] The present application aims to provide a cross-view target association tracking method applied to a multi-camera system, reducing the dependence on multi-camera system calibration and being more suitable for dynamic changes in camera layout in actual scenarios. The target tracking and association based on the multi-camera system can be realized, which is suitable for application in the fields of video monitoring, group behavior analysis, public security, and smart city construction, etc.

[0005] The cross-view target association tracking method comprises the following steps:

[0006] 1. Data preparation: The method uses the public dataset MvMHAT (Multi-view Multi-Human Association and Tracking) in the field of cross-view target association tracking, which contains 26 groups of multi-camera video sequences, a total of 98 video sequences, for model training and testing.

[0007] 2. Cross-view feature mining: The publicly pre-trained ResNet50 model on the ImageNet dataset is used to extract preliminary features from video frames, and the object location is located by the target perception device to ensure that the target features can be effectively captured between different view video inputs.

[0008] 3. Global feature fusion: global target representation is achieved through phased feature fusion. In the cross-view fusion stage, different view target feature embeddings are combined; in the cross-time frame fusion stage, features within adjacent time frames are combined; and in the final global fusion stage, a multi-view, multi-frame unified global feature embedding is generated.

[0009] 4. Temporal-spatial context feature association: first, cross-frame association is performed to match the similarity of target trajectories within different time frames. At the same time, cross-view association is performed to associate targets between different views based on trajectory similarity, and a constraint optimization formula is used to solve the data matching problem.

[0010] 5. Target search loss calculation: introduce target search loss, calculate negative log-likelihood loss by matching target features in the current frame and vectors in the stored target feature database, and optimize the accuracy of cross-view and cross-frame data association.

[0011] 6. Parameter tuning and performance evaluation: network parameters are optimized through ablation experiments, and various evaluation indicators (such as IDF1, MOTA, MHAT.F1, etc.) are used to verify the cross-frame and cross-view association tracking effect.

[0012] The beneficial effects of the present application are: the present application can be applied to uncalibrated multi-camera systems, realizing effective cross-view target tracking and association, and can be widely applied to video monitoring, group behavior analysis, public safety and smart city fields. In view of the problem of view difference and inconsistent features, the present application proposes a collaborative feature fusion network and a multi-scale adaptive alignment module, effectively reducing the interference between features and improving the tracking consistency; through spatio-temporal context modeling, the cross-frame association is converted into a constraint optimization problem, enhancing the accuracy of cross-time frame association; reducing the dependence on multi-camera system calibration in dynamic layout scenes, making it more flexible and applicable; introducing target search loss, optimizing global feature information, and improving target identity recognition ability; in parameter tuning, balance information richness and computing efficiency, reduce resource consumption; finally, through various fine evaluation and quantitative experimental indicators, the accuracy, robustness and global performance of the present method are superior to those of the prior art, especially suitable for multi-target tracking requirements in complex dynamic scenes. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 Flowchart for the execution of the present application

[0014] Figure 2 Network model diagram of the present application

[0015] Figure 3 Network prediction result diagram of the present application DETAILED DESCRIPTION

[0016] The application provides a cross-view target association tracking method applied to a multi-camera system, and a specific implementation process is as shown in the figure Figure 1 Firstly, training and test samples are obtained by selecting a multi-view target data set (such as MvMHAT); then, the visual features of the target are extracted by using a pre-trained ResNet50 neural network, and the cross-view and cross-time features are embedded in combination through global feature fusion to generate a more distinctive target description; subsequently, trajectory optimization matching is performed between cross-frames and cross-views through time-space context feature association to ensure the accuracy of target tracking; then, the accuracy and robustness of target association are optimized through a target search loss function; finally, the predicted results of the model are compared with real labels, and indicators such as IDF1 and MOTA are used to evaluate the performance. The method effectively solves the problems of insufficient feature expression and difficult view matching in traditional methods, and realizes high precision and high robustness of the multi-camera target tracking task.

[0017] 1. Cross-view and cross-frame feature mining

[0018] The target sensor obtains the target position in the current frame, denoted as wherein, The feature extractor E(·) adopts a ResNet50 model pre-trained on ImageNet, which is used to preliminarily extract the target features in the data stream video frame wherein, The ResNet50 model pre-trained on the ImageNet data set has learned rich feature representation, which can help us achieve better performance in the cross-view and cross-frame target association tracking task. In the PyTorch framework, the pre-trained ResNet50 model can be called through the torchvision model library.

[0019] 2. Global feature fusion

[0020] Through cross-view feature mining, the feature embedding f of each object is obtained i c，t This feature embedding is a fixed-length feature vector, wherein i represents the unique index of the object, and then, cross-view feature fusion is performed to obtain the cross-view fusion feature and denoted as Then, the cross-frame feature fusion operation is performed to obtain the cross-frame fusion feature and denoted as Finally, the global feature fusion operation is performed to obtain the global multi-view and multi-frame fusion feature wherein, ⊙ represents the tensor splicing operation in Pytorch, and the in the cross-frame fusion represents a preset cross-frame fusion hyperparameter, which is set to 3 as the best after experimental verification, and the global multi-view and multi-frame fusion feature i represents the target index.

[0021] 3. Time-spatial context feature association

[0022] Through the cooperation of the target perceiver and the feature extractor, the proposed cross-view target association tracking method maintains the trajectories of all targets in the current group N c，t Each trajectory element in it saves the association results of the target perceiver and the previous T-frame trajectory, when the view is c and the time frame is t and t-1, the cross-frame data association (tracking) problem is modeled as calculating the similarity association matrix of the trajectory L c,t-1 at view c, t-1 time and the perception result D c，t at view c, t time Similarity association matrix S Its calculation expression is as follows:

[0023]

[0024] At this time, we convert the cross-frame data association (Association of Temporal Relationships, ATR) problem into a constraint optimization problem, and its calculation expression is as follows:

[0025]

[0026] Similarly, when the time frame is t and the view is c and c', the cross-view data association problem is modeled as calculating the similarity association matrix of the trajectory L c，t at view c, t time and the trajectory L c′，t at view c', t time Similarity association matrix S c，t Its calculation expression is as follows:

[0027]

[0028] Similarly, we convert the cross-view data association (Association of Spatial Relationships, ASR) problem into a constraint optimization problem, and its calculation expression is as follows:

[0029]

[0030] 4. Target search loss calculation and result prediction

[0031] 6. After fusing the global information features, we get the global feature information of each object This has a significant contribution to the realization of identity recognition, the proposed cross-view target association tracking method introduces object search loss to utilize fusion features to improve the accuracy of cross-view and cross-frame data association, and creates a tensor database D = {d1, d2, …, d n} for storing the feature information of all labeled targets, a query library Q = {q1, q2, …, q m} for storing the encoding vectors of targets in the current frame, in each data association process, the matching degree between the query library and the tensor database is calculated to obtain the negative log likelihood value minimization loss The calculation expression is as follows:

[0032]

[0033] Wherein, tau = 0.1 is a hyperparameter for controlling the smoothness of the probability curve. The final score is sorted to obtain the final prediction result.

[0034] The present application is based on deep learning and time-space feature association technology, by extracting the visual features of the target and performing global feature fusion, the target features under different camera perspectives are uniformly represented; combined with the trajectory optimization algorithm, the accurate association of cross-time and cross-view targets is realized, and the matching accuracy and robustness are optimized through the target search loss function, thereby solving the problems of insufficient feature expression, large view angle difference and cross-time matching difficulty of traditional methods. The present application can be widely applied to multi-camera monitoring system, public safety, intelligent retail, robot and unmanned driving and other scenes, and significantly improves the efficiency and accuracy of target tracking in intelligent transportation, security monitoring, customer behavior analysis and multi-sensor fusion, and provides a high robustness and high performance target association solution for multi-camera system.

[0035] It should be noted that the above only describes the embodiments of the present application, only explains the present application, and does not limit the scope of the present application. Changes that are only obvious to those skilled in the art and belong to the technical concept of the present application are also within the scope of protection of the present application.

Claims

1. A cross-view target association tracking method applied to a multi-camera system, the method comprising the following steps: (1) Obtain the existing public dataset in the field of cross-view target association tracking: MvMHAT (Multi-view Multi-Human Association and Tracking), MvMHAT comprises 26 groups, a total of 98 video sequences, the total video sequence length is 90900 frames, the average length of each video sequence is 928 frames, and the number of targets appearing in the video is 10. In addition, the training set and the test set are divided in a ratio of 2:1; (2) Cross-view and cross-frame feature mining: the backbone network is a multi-branch structure, the number of branches corresponds to the number of input sequences in each group of multi-view video sequence groups, and the target perceiver obtains the target position in the current frame, denoted as wherein, The feature extractor E(·) adopts a ResNet50 model pre-trained on ImageNet for preliminary extraction of target features F in the data stream video frames c，t ; (3) Global feature fusion: Through cross-view feature mining, the feature embedding of each object is obtained This feature embedding is a fixed-length feature vector, where i represents the unique index of the object, and then, cross-view feature fusion is performed first to obtain cross-view fusion features and denoted as Then, a cross-frame feature fusion operation is performed to obtain cross-frame fusion features and denoted as Finally, a global feature fusion operation is performed to obtain global multi-view multi-frame fusion features (4) Temporal-spatial context feature association: Through the cooperation of the target perceiver and the feature extractor, the proposed cross-view target association tracking method maintains the trajectories of all targets in the current group N c，t Each of the trajectory elements saves the association results of the target perceiver and the previous T-frame trajectory. When the view is c and the time frame is t and t-1, the cross-frame data association (tracking) problem is modeled as calculating the similarity association matrix of the trajectory L c,t-1 at the view c and time t-1 with the perception result D c,t at the view c and time t Similarly, when the time frame is t and the view is c and c', the cross-view data association problem is modeled as calculating the similarity association matrix of the trajectory L c,t at the view c and time t with the trajectory L c′，t at the view c' and time t (5) Target search loss calculation: After fusing global information features, we get the global feature information of each object This has a significant contribution to the realization of identity recognition, the proposed cross-view target association tracking method introduces object search loss to utilize fused features to improve the accuracy of cross-view and cross-frame data association, creates a tensor database D = {d1, d2, …, d n} for storing the feature information of all labeled targets, the query library Q = {q1, q2, …, q m} for storing the encoding vectors of the targets in the current frame, in each data association process, the matching degree between the query library and the tensor database is calculated to get the negative log likelihood value minimization loss and get the final prediction result; (6) Parameter tuning and evaluation index calculation: perform multiple ablation experiments to obtain the best network parameter setting, in addition, the cross-frame tracking effect is evaluated by calculating IDP (Identification Precision), IDR (Identification Recall), IDF1 (Identification F1 Score), MOTP (Multiple Object Tracking Precision) and MOTA (Multiple Object Tracking Accuracy), the cross-view target association effect is evaluated by calculating AIDP (multi-view subject association precision), AIDR (multi-view subject association recall), AIDF1 (multi-view subject association F1 Score) and MHAA (multi-view multi-human association accuracy), and the overall index is evaluated by calculating MHAT.F1 (MvMHAT F1 score) and MHAT.Acc (MvMHAT accuracy score).

2. The cross-view target association tracking method for multi-camera systems of claim 1, wherein, The dataset used in step (1) is the MvMHAT cross-view target association tracking dataset disclosed in the field, in order to ensure data diversity, the dataset contains 26 groups of multi-camera system video sequences (multi-view video sequence groups), each group of video sequences contains 3-4 views, a total of 98 video sequences (single-view video sequences), among which, 46 video sequences are shot by wearable cameras, and 52 fixed multi-camera system video sequences.

3. The cross-view target association tracking method for multi-camera systems of claim 1, wherein, The target feature extracted in step (2) wherein, ResNet50 is a kind of deep convolutional neural network, which is widely used because of its excellent performance in image classification tasks. The pre-trained ResNet50 model on the ImageNet dataset has learned rich feature representations, which can help us achieve better performance in cross-view and cross-frame target association tracking tasks. In the PyTorch framework, the pre-trained ResNet50 model can be called through the torchvision model library.

4. The cross-view target association tracking method for multi-camera systems of claim 1, wherein, The cross-view fusion feature obtained in step (3) wherein, indicates the tensor splicing operation in Pytorch, and indicates the cross-frame fusion indicates the preset cross-frame fusion hyperparameter, which is set to 3 as the best after experimental verification, and the global multi-view multi-frame fusion feature i indicates the target index.

5. The cross-view target association tracking method for multi-camera systems of claim 1, wherein, The similarity correlation matrix in step (4) The computational expression thereof is as follows: At this time, we convert the Association of Temporal Relationships (ATR) problem into a constrained optimization problem, and its calculation expression is as follows: similarity association matrix S c，t whose computational expression is as follows: Similarly, we transform the association of spatial relationships (ASR) problem into a constrained optimization problem, whose computational expression is as follows:

6. The cross-view target association tracking method for multi-camera systems of claim 1, wherein, Step (5) calculates the negative log-likelihood value minimization loss The calculation expression is as follows: where τ = 0.1 is a hyper-parameter that controls the smoothness of the probability curve.

7. The cross-view target association tracking method for multi-camera systems of claim 1, wherein, Three types of evaluation metrics are considered in step (6), namely, cross-view target association, cross-time frame target tracking, and a comprehensive metric, where the cross-view metric contains AIDF1, whose computational expression is as follows: The MHAA metric, whose computational expression is as follows: where MS t , FP t , MM t , N t denote false negative, false positive, mismatched data pair and the total number of targets at time t, respectively, and the comprehensive indicator MHAT.F1 is calculated as follows: MHAT.F1 = mean(IDF1, AIDF1) (8) The comprehensive metric MHAT.Acc, whose computational expression is as follows: MHAT.Acc = mean(MOTA, MHAA) (9).

Images