Multi-target tracking method based on graph matching

Abstract

The invention relates to the technical field of computer vision research, in particular to a multi-target tracking method based on graph matching. All targets in each frame in a video sequence are regarded as vertexes, edges are constructed among the targets, and each frame is constructed into a graph structure containing the vertexes and the edges, so that a data association problem between the two frames can be modeled into a quadratic graph matching problem, and then the quadratic graph matching problem is solved, so that the matching efficiency of the two frames is improved. And obtaining a solution as a matching result in combination with the trajectory management part. According to the method, high-order structural features are introduced when the target is represented, and a trajectory management strategy is matched, so that even if the target is shielded, continuous tracking of the target can be ensured through structural information between the surrounding target and the shielded target, and the problem of ID wrong transformation is avoided; and the tracking effect under the scenes of camera drift, illumination condition change, scale change and the like is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision research, and in particular to a multi-target tracking method based on graph matching. Background Art

[0002] With the penetration of intelligent industrial information, the intelligent perception and processing of target information have attracted more and more attention in scientific research and practical applications. Multi-target tracking is an important research task in the field of computer vision research. It aims to associate multiple target objects to continuous frames in a video sequence. It is a complex task involving multiple related research fields such as target detection, feature extraction, and data association. In recent years, with the promotion of the wave of intelligent informatization and the emergence of a large number of deep learning-related works, the research on multi-target tracking technology has become increasingly popular and developed rapidly, and has made great progress in theoretical experiments and practical engineering applications. Although the introduction of deep learning methods on the basis of traditional multi-target tracking algorithms has effectively improved the results of multi-target tracking, as the actual application scenarios of multi-target tracking become more and more complex, such as background clutter interference leading to misjudgment, complex scenes leading to target omission and occlusion, camera drift leading to reduced tracking accuracy, and many scenarios that strictly require algorithm operation efficiency, etc., higher performance requirements are put forward for multi-target tracking algorithms, and solving multi-target tracking tasks still faces many challenges.

[0003] Among the many multi-target tracking related research works, most of them focus on how to design or introduce a more sophisticated network structure to obtain target representation, which includes appearance information, motion information, and even depth information. They want to rely on this information to complete the tracking task, and only use simple methods to obtain the final tracking results in the subsequent data association stage. In practice, most video sequences are very complex, and data association will improve the tracking results significantly. There are also some works that build the entire tracking process into an end-to-end network model for training. This treatment emphasizes the integrity of the algorithm. Only one model is used to achieve tasks that can only be completed by combining multiple modules, which greatly reduces the engineering complexity. However, for an end-to-end network model, we cannot clearly see how much it actually optimizes the target detection and feature extraction parts, which further reduces the interpretability of the model. Moreover, because it is designed as a whole, it cannot be as flexible as a multi-module solution. Only a small part of the work considers how to design a more complete data association method to help the multi-target tracking algorithm get better tracking results.

[0004] In existing work, there are similar multi-target tracking methods that use the Hungarian algorithm for data association. The Hungarian algorithm is often used to solve the maximum matching problem in bipartite graph matching. After obtaining the similarity between the target and the trajectory, an iterative method is used to ensure that after matching, each target can be assigned to the trajectory with the highest similarity in general. The Hungarian algorithm only uses first-order information for data association. When the target is well characterized and the tracking scene is single, it can provide very good association results. Therefore, there are still many methods that choose the Hungarian algorithm for data association. However, when the tracking scene becomes complex and the similarity between targets becomes confused, the accuracy of the tracking results cannot be guaranteed.

[0005] Among the existing multi-target tracking methods, the traditional multi-target tracking methods are divided into two types, namely, the extension of the single target tracking method and the construction of multi-target tracking methods based on maximizing probability statistics and deterministic derivation. They are gradually unable to adapt to the current application environment in terms of accuracy, scalability and tracking efficiency. In addition, there are multi-target tracking methods based on deep learning. Most of them rely on designing a complex neural network structure to obtain a deep representation of the target, and do not pay attention to data association issues. When the scene conditions cannot meet their requirements, the tracking effect fluctuates significantly. There are also some works that design the entire tracking process as an end-to-end network structure and train the entire tracking process as a whole. Such a design cannot make each working module reach the optimal working state. Summary of the invention

[0006] Based on the above technical problems, the present invention proposes a multi-target tracking method based on graph matching to solve the problems that the existing multi-target tracking methods are gradually unable to adapt to the current application environment in terms of accuracy, scalability and tracking efficiency, and the multi-target tracking methods based on deep learning do not pay attention to the data association problem. When the scene conditions cannot meet its requirements, the tracking effect fluctuates significantly.

[0007] In order to solve the above technical problems, one of the purposes of the present invention is to provide a multi-target tracking method based on graph matching, and the specific steps are as follows:

[0008] S1: According to the current deep learning method, the detection-tracking paradigm requires the detector and feature extraction parts to be separated, and each part only needs to complete its own task;

[0009] S2: Combine the divided sub-parts with each other and further process data association to achieve tracking.

[0010] Furthermore, the detector is used to obtain target detection in the current frame, and the feature extraction part is used to extract the appearance features of the target. The neural networks used in the detector and the feature extraction part will be independently optimized and trained, and then used together.

[0011] Furthermore, the specific steps of S1 are as follows:

[0012] First, all objects in each frame of the video sequence are regarded as vertices, and edges are constructed between objects. Each frame is constructed into a graph structure containing vertices and edges, and the data association problem between two frames is modeled as a quadratic graph matching problem.

[0013] Secondly, by solving the quadratic graph matching problem and combining it with the trajectory management part, the solution is obtained as the matching result.

[0014] Furthermore, the specific operations of modeling the quadratic graph matching problem are:

[0015] S11: The target in each frame will be constructed as a graph structure, which is specifically defined as G t =(V t , E t ),V t represents the set of vertices in the graph, each vertex is represented by a feature description obtained from detection, E t Represents the edge set in the graph. The edge features are composed of two parts: appearance features and structural features. The appearance features of each edge are represented by the weighted sum of the feature descriptions of the two vertices. The structural features are a set of two-dimensional values, including the Euclidean distance between two targets and the relative angle based on the image. Together, they constitute the edge features. All features are sent to the graph model for the construction and solution of the quadratic graph matching problem. The mathematical modeling method of the quadratic graph matching problem is shown in Formula 1:

[0016]

[0017] stx1 n =1 n , x T 1 n =1 n

[0018] Among them, xE{01} n×m is an assignment matrix, C1∈R n×n , C2ER m×m Represent the weight matrices of the edges in the two graphs, G n ∈R n×m is the affinity matrix of the vertices between the two graphs, 1 n Represents an n-dimensional vector whose values ​​are all 1. When applied to the quadratic graph matching problem, we can consider tr(C1xC2x T ) is expressed as optimizing the matching relationship between edges (objects) in two frames, tr(G n Tx) is expressed as optimizing the matching relationship between vertices (targets) in two frames, and st describes the restriction condition for the above formula to be valid, that is, there must be a one-to-one correspondence between detection and trajectory. The restriction condition meets the basic requirements of multi-target tracking, and the detection subordinate trajectory must be unique;

[0019] S12: Constrain x in equation 1 to a doubly random matrix according to equation 2, where ||·||2 is the L2 norm, and transform C1xC2x T , Equation 1 can be transformed into Equation 3:

[0020]

[0021] Among them, G e2 ∈R (n×m)×(n×m) is the symmetric quadratic affinity matrix between all edges in the two graphs, with dimensions (n×m)×(n×m), represents the Kronecker product, st describes the one-to-one correspondence constraint between detection and trajectory in the matching process, I M and I N are two identity matrices.

[0022] Furthermore, the detection in the S11 image will be adjusted to a size of 3×128×64 and sent to the residual network as input, and the residual network will eventually output a 128-dimensional vector as the appearance feature of the target.

[0023] Furthermore, we solve the convex quadratic optimization problem and use the gradient descent method to obtain the result of convex quadratic optimization. The calculation of the graphical model is divided into two parts:

[0024] Calculate the similarity of the edges, the intermediate matrix G e2 The similarity of the edges in the two frames is calculated by formula 4:

[0025]

[0026] in, and are the edge starting point matrices for detection and trajectory, respectively. and They are the edge termination point matrices of detection and trajectory respectively. For the starting point matrix, only the element at the edge starting point is 1, and the rest are zero. Similarly, for the termination point matrix, only the element at the edge termination point is 1, and the rest are zero. The specific formula is described as follows:

[0027]

[0028] About G e1 First, we calculate the similarity of the detection and trajectory features extracted by the residual network and the position features mentioned above, and then take their weighted sum as Ge1 elements, and finally G e1 Convert to a vector and construct a diagonal matrix from it for subsequent calculations.

[0029] To calculate the similarity and matching results of vertices, we need to calculate the similarity of vertices and obtain the final assignment matrix. To calculate the similarity, we trained a simple twin network to calculate the similarity between different vertices in the two images and obtain the similarity matrix G. n , an assignment matrix is ​​obtained through formula 3 as the final matching result.

[0030] The above one or more technical solutions of the present invention have at least one or more of the following technical effects:

[0031] 1) By introducing high-order structural information into data association, the multi-target tracking task is modeled as a quadratic graph matching problem. The proposed graph model does not involve complex neural network learning, and only uses a simple twin network to improve the accuracy of similarity comparison. All other parts, such as graph construction and solving quadratic graph matching problems, do not require additional iterative learning, which greatly saves computational costs. In addition, when constructing the required intermediate matrix, the dimension of the correlation matrix when solving the quadratic graph matching problem is further optimized, making the entire model more lightweight;

[0032] 2) This method can be used as a data association module in any detection-tracking method and bring improvements.

[0033] In order to better solve the tracking tasks in complex scenes, a multi-target tracking method based on graph matching was invented. High-order structural features were introduced when characterizing the target, and the trajectory management strategy was combined. Even if the target is occluded, the structural information between the surrounding targets and the occluded targets can be used to ensure continuous tracking of the target, avoiding the problem of ID error transformation and ensuring the tracking effect in scenes such as camera drift, changes in lighting conditions and scale changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 :Multi-target tracking method structure based on detection-tracking paradigm;

[0035] Figure 2 : Multi-target tracking model based on quadratic graph matching;

[0036] Figure 3 :Calculation process of the core formula F(*);

[0037] Figure 4 : Calculate the similarity of edges;

[0038] Figure 5 : Calculate the similarity and matching results of vertices. Specific implementation plan

[0039] The present invention provides a multi-target tracking method based on graph matching. When processing data, vertex features are constructed for all targets in each frame in a video sequence, and edge features are constructed between two targets, which are modeled as a graph. The tracking result between the two frames can be obtained by solving the graph matching problem. Solving the multi-target tracking task by solving the graph matching method can further enhance the feature information of the target itself through the high-order structural information between the targets, and improve the tracking effect in complex scenes such as occlusion and lens movement.

[0040] The study invented a multi-target tracking method based on graph matching. All targets in each frame of the video sequence are regarded as vertices, and edges are constructed between targets. Each frame is constructed into a graph structure containing vertices and edges. In this way, the data association problem between two frames can be modeled as a quadratic graph matching problem. Then, by solving the quadratic graph matching problem and combining it with the trajectory management part, the solution is obtained as the matching result. Both theoretical analysis and experimental results show that introducing high-order structural features in the matching function effectively solves the tracking problem in complex scenarios such as imperfect target representation.

[0041] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in combination with the embodiments of the present invention and the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the obtained embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0042] The present invention uses a more classic detection-tracking paradigm. Figure 1 As shown in the figure, it is similar to the traditional processing method, which divides the entire algorithm into multiple sub-parts, each of which only needs to complete its own task, and then combines these sub-parts to achieve tracking. Corresponding to the current deep learning method, the detection-tracking paradigm requires the detector and feature extraction parts to be separated, and then further processed for data association. The detector is used to obtain the target detection in the current frame, and the feature extraction part is used to extract the appearance features of the target. The neural networks used in the detector and feature extraction parts will be independently optimized and trained, and then used together.

[0043] Figure 2 This is the overall framework structure diagram of the algorithm. After the tracking of the t-1th frame is completed, when the tth frame arrives, the detection of the tth frame obtained by the target detector and the historical detection of the t-1th frame will be sent to the model as input, and then the feature extractor is used to obtain the appearance features of the detection. Here, a pre-trained residual network is used for feature extraction. Combined with the structural information between the targets, the data association part can be modeled as a graph model.

[0044] In the modeling of the quadratic graph matching problem, the target in each frame will be constructed as a graph structure, which is specifically defined as G t =(V t , E t ). t Represents the set of vertices in the graph, each vertex is represented by a feature description obtained from the detection. The detection in the image will be resized to 3×128×64 and sent to the residual network as input, and the residual network will eventually output a 128-dimensional vector as the appearance feature of the target. t Represents the edge set in the graph. The edge features consist of two parts: appearance features and structural features. The appearance features of each edge are represented by the weighted sum of the feature descriptions of the two vertices. The result is also a 128-dimensional vector. The structural features are a set of two-dimensional values, including the Euclidean distance between two targets and the relative angle based on the image, which together constitute a 130-dimensional edge feature. All features are then sent to the graph model for the construction and solution of the secondary graph matching problem.

[0045] The mathematical modeling method for the quadratic graph matching problem is as follows:

[0046]

[0047] stx1 n =1 n , x T 1 n =1 n

[0048] Where x∈{01} n×m is an assignment matrix. C1∈R n×n , C2∈R m×m Represent the weight matrices of the edges in the two graphs. n ∈R n×m is the affinity matrix of the vertices between the two graphs. n represents an n-dimensional vector whose values ​​are all 1. Applied to the quadratic graph matching problem, we can consider tr(C1xC2x T ) is expressed as optimizing the matching relationship between edges (objects) in two frames, tr(G n T x) is expressed as optimizing the matching relationship between vertices (targets) in two frames, and st describes the restriction conditions for the above formula to be valid, that is, there must be a one-to-one correspondence between detection and trajectory. The restriction conditions meet the basic requirements of multi-target tracking, and the detection-slave trajectory must be unique.

[0049] Transformation and solution of the quadratic graph matching problem. Constrain x in Equation 1 to a double random matrix according to Equation 2. ||·||2 is the L2 norm, and transform C1xC2x T, Equation 1 can be transformed into Equation 3:

[0050]

[0051] Among them, G e2 ∈R (n×m)×(n×m) is the symmetric quadratic affinity matrix between all edges in the two graphs, with dimensions (n×m)×(n×m). represents the Kronecker product. st describes the constraint condition of one-to-one correspondence between detection and trajectory in the matching process. M and I N are two unit matrices. For solving the convex quadratic optimization problem, the present invention selects to use the gradient descent method to obtain the result of the convex quadratic optimization. Figure 3 The solution process of the core formula F(*) is given.

[0052] In order to make the calculation process of the core formula F(*) easier to understand, the calculation of the graph model is divided into two parts. The first part shows the calculation of the edge similarity, and the second part shows the calculation of the vertex similarity and matching results, which are introduced in detail.

[0053] Figure 4 It is the process of calculating the similarity of edges, and the intermediate matrix G e2 The similarity of the edges in the two frames is calculated by formula 4:

[0054]

[0055] in, and are the edge starting point matrices for detection and trajectory, respectively. and They are the edge termination point matrices of detection and trajectory, respectively. For the starting point matrix, only the element at the edge starting point is 1, and the rest are zero. Similarly, for the termination point matrix, only the element at the edge termination point is 1, and the rest are zero. The specific formula is described as follows:

[0056]

[0057] About G e1 First, we calculate the similarity of the detection and trajectory features extracted by the residual network and the position features mentioned above, and then take their weighted sum as G e1 elements, and finally G e1 Convert to a vector and construct a diagonal matrix from it for subsequent calculations.

[0058] Figure 3 The second part of the process is as follows Figure 5As shown, the similarity of vertices and matching results are calculated. It is necessary to calculate the similarity of vertices and obtain the final assignment matrix. For the calculation of similarity, in order to make the calculated similarity more accurate, a simple twin network is trained to calculate the similarity between different vertices in the two images, and the similarity matrix G can be obtained. n Finally, an assignment matrix can be obtained as the final matching result through Formula 3.

[0059] Table 1 shows the experiment on the number of parameters and floating-point operations. The number of parameters and floating-point operations directly reflects the complexity and computational efficiency of a network structure. The table lists a total of 11 widely used network structures and the comparison of the network structure proposed by this method. Among them, DeepSORT20 is the network structure with the most engineering applications, DAN and AOA are both award-winning network structures, and the ResNet series is a very well-known network structure in related fields. It can be seen from Table 1 that the network structure proposed in this paper is very lightweight, with far fewer parameters than the comparison network, and also performs well in the floating-point operation index.

[0060] Table 1 Experiments on the number of parameters and floating point operations

[0061]

[0062] Table 2 is a multi-target tracking comparison experiment on the MOT17 dataset for data association using IOU, SORT, Hungarian algorithm and the algorithm proposed in this paper. The four methods use the same feature extraction network to ensure the consistency of target feature conditions, and mainly judge the performance of each method in the data association part. It can be seen from the data in the table that MOTA is a comprehensive evaluation indicator. The method proposed in this paper performs best in the comparison. IDs indicates the situation where the target ID is incorrectly exchanged during tracking. The method proposed in this paper also has the least errors.

[0063] Table 2 Comparative experiments of different methods on the MOT17 dataset

[0064]

[0065] A number of horizontal comparison experiments have been conducted to determine the optimal correlation parameters, and a number of vertical comparison experiments have been conducted to prove the feasibility of the invention. Both theoretical analysis and experimental results show that the introduction of high-order structural features in the matching function effectively solves the tracking problem in complex scenarios such as imperfect target representation.

[0066] Obviously, those skilled in the art can make various changes and modifications to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if these modifications and variations of the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A multi-target tracking method based on graph matching, characterized in that: The specific steps are as follows: S1: Corresponding to the current deep learning method, the detection-tracking paradigm requires the detector and feature extraction parts to be separated, and each part only needs to complete its own task; S2: Combine the divided sub-parts with each other and further process data association to achieve tracking.

2. The multi-target tracking method based on graph matching according to claim 1, characterized in that: The detector is used to obtain target detection in the current frame, and the feature extraction part is used to extract the appearance features of the target. The neural networks used in the detector and the feature extraction part will be independently optimized and trained, and then used together.

3. The multi-target tracking method based on graph matching according to claim 1, characterized in that: The specific steps of S1 are as follows: First, all objects in each frame of the video sequence are regarded as vertices, and edges are constructed between objects. Each frame is constructed into a graph structure containing vertices and edges, and the data association problem between two frames is modeled as a quadratic graph matching problem. Secondly, by solving the quadratic graph matching problem and combining it with the trajectory management part, the solution is obtained as the matching result.

4. The multi-target tracking method based on graph matching according to claim 3 is characterized in that: The specific operation of modeling the quadratic graph matching problem is: S11: The target in each frame will be constructed as a graph structure, which is specifically defined as G t =(V t , E t ), V t represents the set of vertices in the graph, each vertex is represented by a feature description obtained from detection, E t Represents the edge set in the graph. The edge features consist of two parts: appearance features and structural features. The appearance features of each edge are represented by the weighted sum of the feature descriptions of the two vertices. Structural features are a set of two-dimensional values, including the Euclidean distance between two targets and the relative angle based on the image. Together, they constitute edge features. All features are sent to the graph model to construct and solve the quadratic graph matching problem. The mathematical modeling method of the quadratic graph matching problem is shown in Formula 1: Where x∈{01} n×m is an assignment matrix, C1∈R n×n , C2∈R m×m Represent the weight matrices of the edges in the two graphs, G n ∈R n×m is the affinity matrix of the vertices between the two graphs, 1 n Represents an n-dimensional vector whose values ​​are all 1; S12: Constrain x in equation 1 to a doubly random matrix according to equation 2, where ||·||2 is the L2 norm, and transform C1xC2x T , Equation 1 can be transformed into Equation 3: Among them, G e2 ∈R (n×m)×(n×m) is the symmetric quadratic affinity matrix between all edges in the two graphs, with dimensions (n×m)×(n×m), represents the Kronecker product, st describes the one-to-one correspondence constraint between detection and trajectory in the matching process, I M and I N are two identity matrices.

5. The multi-target tracking method based on graph matching according to claim 4, characterized in that: The detection in the Sl1 image will be adjusted to a size of 3×128×64 and sent to the residual network as input. The residual network will eventually output a 128-dimensional vector as the appearance feature of the target.

6. The multi-target tracking method based on graph matching according to any one of claims 4 or 5, characterized in that: Solve the convex quadratic optimization problem and use the gradient descent method to obtain the result of convex quadratic optimization. The calculation of the graphical model is divided into two parts: Calculate the similarity of the edges, the intermediate matrix G e2 The similarity of the edges in the two frames is calculated by formula 4: in, and are the edge starting point matrices for detection and trajectory, respectively. and They are the edge termination point matrices of detection and trajectory respectively. For the starting point matrix, only the element at the edge starting point is 1, and the rest are zero. Similarly, for the termination point matrix, only the element at the edge termination point is 1, and the rest are zero. The specific formula is described as follows: About G e1 First, we calculate the similarity of the detection and trajectory features extracted by the residual network and the position features mentioned above, and then take their weighted sum as G e1 elements, and finally G e1 Convert to a vector and construct a diagonal matrix from the vector for subsequent calculations; To calculate the similarity and matching results of vertices, we need to calculate the similarity of vertices and obtain the final assignment matrix. To calculate the similarity, we trained a simple twin network to calculate the similarity between different vertices in the two images and obtain the similarity matrix G. n , an assignment matrix is ​​obtained through formula 3 as the final matching result.

7. The multi-target tracking method based on graph matching according to claim 4, characterized in that: Mathematical modeling method Equation 1 is applied to the quadratic graph matching problem, and it is considered that tr(C1xC2x T ) is expressed as optimizing the matching relationship between edges (objects) in two frames, tr(G n T x) is expressed as optimizing the matching relationship between vertices (targets) in two frames, and st describes the restriction conditions for the above formula to be valid, that is, there must be a one-to-one correspondence between detection and trajectory. The restriction conditions meet the basic requirements of multi-target tracking, and the detection-slave trajectory must be unique.

Images