A multi-camera target association tracking method based on metric learning

By using the metric learning method in a multi-camera environment, the feature vectors of the detection frame are extracted and cosine distance calculations are performed, the occlusion and crossover problems in multi-camera tracking are solved, the tracking accuracy and stability are improved, and the effective correlation of information is achieved.

CN111192297BActive Publication Date: 2024-11-01SHANDONG GUANGYU SCI&TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201911407164.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2019-12-31
Publication Date
2024-11-01
Estimated Expiration
2039-12-31

AI Technical Summary

Technical Problem

In multi-camera environments, it is difficult for the prior art to effectively track target objects, especially in complex environments such as occlusion and crossover, resulting in low tracking accuracy, and single camera occlusion leads to tracking loss, and information sharing and confusion problems are not effective. solve.

Method used

Using a metric learning method, the target object is tracked through Yolo algorithm detection and Deep Sort algorithm, the feature vector of the detection frame is extracted using a deep convolutional network, and the correlation of multi-camera targets is achieved through cosine distance calculation and threshold comparison, and the embedded design is improved Algorithm stability.

Benefits of technology

It improves the accuracy and stability of multi-objective tracking, solves the occlusion and crossover problems in complex environments, realizes effective correlation and tracking of multi-camera information, and improves the practicality and robustness of the algorithm.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111192297B_ABST
    Figure CN111192297B_ABST
Patent Text Reader

Abstract

The present invention belongs to the technical field of computer video image processing, and in particular, to a multi-camera target association tracking method based on metric learning. The purpose of this multi-camera target association tracking method based on metric learning is to overcome the technical defects such as the inability to share information between multiple cameras, information confusion, and tracking loss caused by occlusion of a single camera. At the same time, the tracking algorithm can be further embedded in the hardware device to improve the stability and practicality of the algorithm. The multi-camera target association tracking method includes the following steps: using n cameras to simultaneously collect a video set of a target object in a target place; detecting and tracking the target object to obtain a tracking result; extracting a detection frame from the tracking result; sending the extracted detection frame to a deep convolutional network to extract a feature vector; calculating the cosine distance; judging whether objects in the same frame are similar by threshold comparison; and successfully achieving tracking by association.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the field of computer video image processing technology, and in particular relates to a multi-camera target association tracking method based on metric learning. Background Art

[0002] With the development of computer video image processing technology, multi-target tracking has important applications in intelligent monitoring, motion and behavior analysis, autonomous driving and other fields. Multi-target tracking finds moving objects in a given image sequence, identifies moving objects in different frames, and determines an accurate target identification. These moving objects can be arbitrary, such as pedestrians, vehicles, animals, etc. With the development of target detection in recent years, the tracking-by-detection algorithm has become the mainstream in multi-target tracking. Previously, flow network formulas and probabilistic graphical models for detection appeared, but they deal with the global optimization problem of the entire process and are not suitable for online scenarios. The common tracking algorithms used after detection are the SORT algorithm and the Deep Sort algorithm. The SORT algorithm uses a simple Kalman filter to process the correlation of frame-by-frame data and uses the Hungarian algorithm for correlation measurement. This simple algorithm achieves good performance at high frame rates, but because it ignores the surface features of the detected objects, the SORT algorithm is only accurate when the uncertainty of the object state estimation is low. The Deep Sort algorithm uses a more reliable metric to replace the association metric, and uses a convolutional neural network to train on a large-scale pedestrian dataset to extract features, which increases the network's robustness to loss and obstacles. Although the combination of the Yolo algorithm and the Deep Sort algorithm for multi-target tracking has become one of the mainstream algorithms, in complex environments where people are occluded or crossed, the object identification in the previous frame often corresponds to the same object identification in the next frame; and if the algorithm is used for multi-camera tracking, although there are common areas within the range that multiple cameras can observe, the tracking information of multiple cameras cannot be associated, which will ultimately lead to low precision in multi-target tracking and incorrect information about the target objects. Summary of the invention

[0003] The present invention provides a multi-camera target association tracking method based on metric learning. The purpose of the multi-camera target association tracking method based on metric learning is to overcome technical defects such as the inability to share information between multiple cameras, information confusion, and tracking loss caused by occlusion of a single camera. At the same time, the tracking algorithm can be further embedded into hardware devices to improve the stability and practicability of the algorithm.

[0004] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0005] A multi-camera target association tracking method based on metric learning includes the following steps:

[0006] Step 1: Use n cameras to simultaneously collect a video set V = {V1, V2, ..., V n}; where V i represents the video captured by the camera labeled i, and V i ={img1,img2,…,img n}, img i V i The i-th frame in the video;

[0007] Step 2: Use the Yolo algorithm to calculate V i Detect the target object in the image and track it using the Deep Sort algorithm to obtain V i The object detection box DF = {df1, df2, ..., df n} and specific identification set ID = {id1, id2, ..., id n}; where id i Indicates V i The object named i in the image is then tracked by E = {(df1, id1), (df2, id2), …, (df n ,id n )}Save to local;

[0008] Step 3: Extract the tracking results of p cameras from the tracking result E; assume that p = 2, and the cameras are numbered j and k respectively; extract the same frame of the video captured by cameras j and k, that is, V j and V k The same frame img i ; Extract img separately i The detection boxes of q objects in DF1={d1,d2,…,d q} and DF2={f1,f2,…,f q}; where d i V j img i Where ID is id i The detection box, f i V k img i Where ID is id i Detection box of

[0009] Step 4: Send the extracted detection boxes DF1 and DF2 to the deep convolutional network to extract d i and f iThe characteristic vectors of Vj ={r1 idi ,r2 idi ,…,r n idi},r Vk ={r1 idi ,r2 idi ,…,r n idi}; where r Vj and r Vk Represents cameras j and k in img i Frame ID collected i Appearance characteristics;

[0010] Step 5: Vj ={r1 idi ,r2 idi ,…,r n idi} and r Vk ={r1 idi ,r2 idi ,…,r n idi} Calculate the cosine distance, the calculation formula is shown in formula (1):

[0011] Cos(dis(r Vj ,r Vk ))=1-r vj T r vk (1)

[0012] Among them, r vj T Yes vj The transposed matrix of

[0013] Step 6: Compare and determine V by threshold α j and V k Whether the objects in the same frame are similar, the threshold α is obtained by training; when Cos(dis(r Vj ,r Vk ))≤α, d i and f i The association is successful, and it is judged to be the id in the two detection frames i Similar; otherwise, return to step 5 to continue cosine calculation until all comparisons are completed;

[0014] Step 7, when d i and f i The association is successful, and f i Corresponding id i Change to d i Corresponding idi .

[0015] The present invention provides a multi-camera target association tracking method based on metric learning, and the multi-camera target association tracking method includes the following steps: using n cameras to simultaneously collect a video set of a target object in a target place; detecting and tracking the target object to obtain a tracking result; extracting a detection frame from the tracking result; sending the extracted detection frame to a deep convolutional network to extract a feature vector; calculating the cosine distance; judging whether objects in the same frame are similar by threshold comparison; and tracking is achieved by association. The multi-camera target association tracking method based on metric learning with the above-mentioned step characteristics improves the recording effect by recording multiple cameras at the same time; by using a deep neural network to extract the appearance features of the object in the detection frame, it is simple and highly robust; by using cosine distance as a metric learning method, it has no direct connection with the absolute value, and can better measure the similarity between feature vectors; by using an embedded method, the stability and practicality of the algorithm are improved. And it solves the problems of personnel occlusion and intersection caused by complex environments in the multi-target tracking problem, and can effectively improve the tracking accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of an application scenario of a multi-camera target association tracking method based on metric learning provided by the present invention;

[0017] Figure 2 A schematic diagram of the structure of an embedded device for a multi-camera target association tracking method based on metric learning provided by the present invention. DETAILED DESCRIPTION

[0018] The present invention provides a multi-camera target association tracking method based on metric learning. The purpose of the multi-camera target association tracking method based on metric learning is to overcome technical defects such as the inability to share information between multiple cameras, information confusion, and tracking loss caused by occlusion of a single camera. At the same time, the tracking algorithm can be further embedded into hardware devices to improve the stability and practicability of the algorithm.

[0019] A multi-camera target association tracking method based on metric learning includes the following steps:

[0020] Step 1: Use n cameras to simultaneously collect a video set V = {V1, V2, ..., V n}; where V i represents the video captured by the camera labeled i, and V i ={img1,img2,…,img n}, img i V iThe i-th frame in the video. It is worth noting that Figure 1 As shown, in Figure 1 The technicians set up two cameras in different directions to capture the current scene.

[0021] Step 2: Use the Yolo algorithm to calculate V i Detect the target object in the image and track it using the Deep Sort algorithm to obtain V i The object detection box DF = {df1, df2, ..., df n} and specific identification set ID = {id1, id2, ..., id n}; where id i Indicates V i The object named i in the image is then tracked by E = {(df1, id1), (df2, id2), …, (df n ,id n )}Save it locally.

[0022] Step 3: Extract the tracking results of p cameras from the tracking result E; assume that p = 2, and the cameras are numbered j and k respectively; extract the same frame of the video captured by cameras j and k, that is, V j and V k The same frame img i ; Extract img separately i The detection boxes of q objects in DF1={d1,d2,…,d q} and DF2={f1,f2,…,f q}; where d i V j img i Where ID is id i The detection box, f i V k img i Where ID is id i Detection box.

[0023] Step 4: Send the extracted detection boxes DF1 and DF2 to the deep convolutional network to extract d i and f i The characteristic vectors of Vj ={r1 idi ,r2 idi ,…,r n idi},r Vk ={r1 idi ,r2 idi ,…,r n idi}; where r Vj and r Vk Represents cameras j and k in img i Frame ID collected i appearance characteristics.

[0024] Step 5: Vj ={r1 idi ,r2 idi ,…,r n idi} and r Vk ={r1 idi ,r2 idi ,…,r n idi} Calculate the cosine distance, the calculation formula is shown in formula (1):

[0025] Cos(dis(r Vj ,r Vk ))=1-r vj T r vk (1)

[0026] Among them, r vj T Yes vj The transposed matrix of .

[0027] Step 6: Compare and determine V by threshold α j and V k Whether the objects in the same frame are similar, the threshold α is obtained by training; when Cos(dis(r Vj ,r Vk ))≤α, d i and f i The association is successful, and it is judged to be the id in the two detection frames i Similar; otherwise, return to step 5 and continue the cosine calculation until all comparisons are completed.

[0028] Step 7, when d i and f i The association is successful, and f i Corresponding id i Change to d i Corresponding id i .

[0029] So far, this multi-camera target association tracking method based on metric learning has achieved the association of multi-camera targets.

[0030] One point that needs to be supplemented is that in order to further improve the application scenarios of the multi-camera target association tracking method, the relevant contents of steps 1 to 7 can be embedded into the hardware device, for example: Figure 2 Specifically, the embedded terminal device preferably uses a PCIe (x8) slot, is designed for edge artificial intelligence and machine vision applications, uses an intelligent power supply, has low cost, generates less waste heat, and has a more stable system.

[0031] The present invention provides a multi-camera target association tracking method based on metric learning, and the multi-camera target association tracking method includes the following steps: using n cameras to simultaneously collect a video set of a target object in a target place; detecting and tracking the target object to obtain a tracking result; extracting a detection frame from the tracking result; sending the extracted detection frame to a deep convolutional network to extract a feature vector; calculating the cosine distance; judging whether objects in the same frame are similar by threshold comparison; and tracking is achieved by association. The multi-camera target association tracking method based on metric learning with the above-mentioned step characteristics improves the recording effect by recording multiple cameras at the same time; by using a deep neural network to extract the appearance features of the object in the detection frame, it is simple and highly robust; by using cosine distance as a metric learning method, it has no direct connection with the absolute value, and can better measure the similarity between feature vectors; by using an embedded method, the stability and practicality of the algorithm are improved. And it solves the problems of personnel occlusion and intersection caused by complex environments in the multi-target tracking problem, and can effectively improve the tracking accuracy.

[0032] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A multi-camera target association tracking method based on metric learning, characterized in that: The steps include: Step 1: Use n cameras to simultaneously collect a video set V = {V1, V2, ..., V n }; Among them, V i represents the video captured by the camera labeled i, and V i ={img1,img2,…,img n }, img i V i The i-th frame in the video; Step 2: Use the Yolo algorithm to calculate V i Detect the target object in the image and track it using the Deep Sort algorithm to obtain V i The object detection box DF = {df1, df2, ..., df n } and specific identification set ID = {id1, id2, ..., id n }; where id i Indicates V i The object named i in the image is then tracked by E = {(df1, id1), (df2, id2), …, (df n ,id n )}Save to local; Step 3: Extract the tracking results of p cameras from the tracking result E; where p = 2, and the cameras are numbered j and k respectively; extract the same frame of the video captured by cameras j and k, that is, V j and V k The same frame img i ; Extract img separately i The detection boxes of q objects in DF1={d1,d2,…,d q } and DF2={f1,f2,…,f q }; where d i V j img i Where ID is id i The detection box, f i V k img i Where ID is id i Detection box of Step 4: Send the extracted detection boxes DF1 and DF2 to the deep convolutional network to extract d i and f i The characteristic vectors of Vj ={r1 idi ,r2 idi ,…,r n idi },r Vk ={r1 idi ,r2 idi ,…,r n idi }; where r Vj and r Vk Represents cameras j and k in img i Frame ID collected i Appearance characteristics; Step 5, for r Vj ={r1 idi ,r2 idi ,…,r n idi } and r Vk ={r1 idi ,r2 idi ,…,r n idi } Calculate the cosine distance, the calculation formula is shown in formula (1): Cos(dis(r Vj ,r Vk ))=1-r vj T r vk (1) Among them, r vj T Yes vj The transposed matrix of Step 6: Compare and determine V by threshold α j and V k Whether the objects in the same frame are similar, the threshold α is obtained by training; when Cos(dis(r Vj ,r Vk ))≤α, d i and f i The association is successful, and it is judged to be the id in the two detection frames i Similar; otherwise, return to step 5 to continue cosine calculation until all comparisons are completed; Step 7, when d i and f i The association is successful, and f i Corresponding id i Change to d i Corresponding id i .