Cross-camera multi-target tracking method based on heterogeneous graph neural network

Abstract

The invention provides a cross-camera multi-target tracking method based on a heterogeneous graph neural network. According to the method, environment key points are introduced, tracking trajectory nodes are combined, a heterogeneous graph structure is constructed, and the heterogeneous graph structure not only comprises the association of the tracking trajectory nodes in the cameras and between the cameras, but also comprises the association between the tracking trajectory nodes in the cameras and the environment key points, the association between the environment key points and the association between the environment key points between the cameras. The association between the target and the environment is enhanced, the cross-attention graph neural network is designed, and the representation ability of the model to the target features is improved. According to the invention, the problem of large difference of apparent characteristics of observed targets under a multi-camera visual angle is solved, so that the accuracy of cross-camera multi-target tracking is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and particularly relates to a multi-object tracking method across cameras based on a heterogeneous graph neural network. Background Art

[0002] Multi-object tracking technology has been widely applied in fields such as pedestrian monitoring, battlefield environment reconnaissance, and crowd behavior analysis, providing important technical support for analyzing the apparent changes of targets and predicting the movement trends of targets. However, for a single-camera multi-object tracking (MTSCT) system, due to the small visual range of a single camera, the limited tracking scene range, and the fact that targets are easily occluded by other targets or the background in a cluttered and crowded environment, these problems can easily lead to a high target loss rate and poor robustness in the MTSCT system, making the MTSCT system face more and more severe challenges. Compared with the multi-object tracking technology of a single camera, the multi-camera multi-object tracking (MTMCT) technology generates local single-camera trajectories from multiple camera videos, and then considers various information, such as appearance features, geometric information, and camera topology, to calculate the trajectory similarity between cameras. After obtaining the trajectory similarity, cross-camera trajectory matching is completed, and finally the global trajectory is generated. The overlapping multi-camera fields of view and rich multi-view information can better solve the problem of target occlusion. For a target that disappears in a certain camera field of view due to occlusion or leaving the field of view, other cameras can be used to continue monitoring and tracking it. Through this alternating tracking between multiple cameras, the trajectory of the target in the entire monitoring scene can be obtained. Therefore, the multi-camera multi-object tracking system has irreplaceable advantages over the single-camera system in fields such as monitoring the traffic flow of pedestrians and vehicles in an urban closed-loop area and multi-object cooperative tracking based on an unmanned aerial vehicle cluster.

[0003] In 2021, Kha Gia Quach et al. proposed the DyGLIP multi-camera tracking model, which reformulated the multi-camera multi-object tracking problem using a dynamic graph model; Pha Nguyen et al. designed the Global Association Graph model for the intelligent vehicle scenario with a single point and multiple cameras, matched the target three-dimensional motion prediction information with the existing trajectories, and combined with appearance features to achieve target consistency judgment under multiple cameras; in August 2022, Haowen Hu et al. achieved multi-camera multi-object tracking based on trajectory matching by constructing an indoor three-dimensional trajectory tracking graph and establishing an internal ReID method among multiple cameras; in October 2022, Xuanke Shi et al. achieved multi-camera multi-object tracking of similar objects in a large-scale and ultra-long-distance scenario in the curling event, and used the cross-view of 42 cameras in a fixed scenario to perform real-time positioning and trajectory prediction on the curling motion target. In 2024, Yanting Zhang et al. achieved mobile multi-camera tracking by synthesizing the trajectories of multiple mobile cameras using GPS positioning information and camera pose information.

[0004] In 2024, Hao et al. proposed an end-to-end cross-view multi-object tracking baseline framework named CrossMOT. CrossMOT is a unified framework for joint object detection and cross-view multi-object tracking, which uses an integrated embedding model for object detection, single-view tracking, and cross-view tracking. CrossMOT adopts decoupled multi-head embeddings to simultaneously learn object detection, single-view re-identification (Re-ID), and cross-view re-identification features. To solve the conflict problem between cross-view and single-view embeddings, it adopts local perception and conflict-free loss to improve the joint embedding. Specifically, the single-view embedding focuses on learning temporal continuity, while the cross-view embedding focuses on learning the invariant appearance of objects.

[0005] To achieve multi-object collaborative tracking across camera platforms, existing methods mainly focus on developing data association techniques based on the tracking trajectories of individual cameras. However, even the current state-of-the-art trackers still face huge challenges in dealing with complex multi-type datasets. Specifically, it includes:

[0006] On the one hand, it is difficult to measure the similarity of target appearance features in complex environments. The online multi-object tracking task based on multi-camera platforms needs to collect scene information through multiple different camera platforms. The data collection process of multi-camera multi-object tracking is as Figure 1 shown. In complex environments, the appearance features of the targets observed by multiple cameras vary greatly under different observation perspectives and different lighting conditions. Moreover, the changes in the positions and poses of cameras and the movement of targets under multi-camera platforms pose higher requirements for the similarity measurement of targets in multiple cameras. Relying solely on target appearance features is difficult to generate stable local trajectories;

[0007] On the other hand, it is difficult to determine the cycle consistency of the same target under a multi-camera platform. For example Figure 2 and Figure 3 are respectively the schematic diagrams of consistent and inconsistent cycle matching of multi-cameras. The target No. 2 in camera V_1 is matched to the target No. 4 in camera V_2 and the target No. 6 in camera V_3. Then the target No. 4 in camera V_2 should be matched to the target No. 6 in camera V_3. However, due to the complexity of the external environment and the dynamic changes in the internal structures of multiple cameras, it is difficult to ensure that the similarity of the apparent features between the target No. 4 and the target No. 6 is necessarily the maximum value in the calculation of the global feature similarity, resulting in the lack of cycle matching consistency of the same target under different cameras and seriously affecting the matching accuracy. Summary of the Invention

[0008] In view of this, the present invention provides a cross-camera multi-target tracking method based on a heterogeneous graph neural network, which can improve the accuracy of cross-camera multi-target tracking.

[0009] To solve the above technical problems, the present invention is implemented as follows.

[0010] A cross-camera multi-target tracking method based on a heterogeneous graph neural network includes:

[0011] Step 1: For synchronized multi-camera surveillance images, obtain the tracking trajectories of targets under a single camera, the target apparent features as the node features of the tracking trajectories, and the environmental key point features;

[0012] Step 2: Construct the edge relationships between nodes within a camera and the edge relationships between nodes between cameras:

[0013] The edge relationships between nodes within a camera include: the edge relationships between tracking trajectory nodes, the edge relationships between tracking trajectory nodes and environmental key points, and the edge relationships between environmental key points, which form self-cross edge relationships;

[0014] The edge relationships between nodes between cameras include: the edge relationships between two tracking trajectory nodes of different cameras and the edge relationships between two environmental key points of different cameras, which form mutual-cross edge relationships;

[0015] Step 3: Train the heterogeneous graph neural network:

[0016] Input the target apparent features of the tracking trajectory nodes and the environmental key point features into the self-cross attention graph convolution module, and use the self-cross edge relationships obtained in Step 2 as the ideal output to optimize the parameters of the self-cross attention graph convolution module;

[0017] Input the output data of the self-cross attention graph convolution module into the mutual-cross attention graph convolution module, and use the mutual-cross edge relationships obtained in Step 2 as the ideal output to optimize the parameters of the mutual-cross attention graph convolution module;

[0018] Step 4: During actual tracking, obtain the monitoring images of each camera, input them into the multi-object tracking model composed of the self-cross-attention graph convolution module and the mutual-cross-attention graph convolution module, and output the predicted tracking trajectory nodes.

[0019] Step 5: Perform object matching based on the monitoring images and the predicted tracking trajectory nodes to achieve object tracking.

[0020] Preferably, the step 5 is as follows: form a multi-camera hypergraph with the monitoring images of each camera and their predicted tracking trajectory nodes; each camera includes a camera graph, and the camera graph includes a set of tracking trajectory nodes, a set of connection edges of each tracking trajectory node in the camera, and a set of connection edges between different cameras;

[0021] Cluster the multi-camera hypergraph, where the same-class nodes correspond to one object, and complete the matching of the tracking trajectory nodes.

[0022] Preferably, in step 5, divide the multi-camera hypergraph into sub-hypergraphs, calculate the similarity degree of adjacent two-frame and same-position sub-hypergraphs. If the similarity degree is less than the set threshold, it means that the matching relationship of the current frame sub-hypergraph changes little and there is no need to perform re-clustering calculation, and directly reuse the matching result of the previous frame; for sub-hypergraphs with a similarity degree greater than or equal to the set threshold, re-cluster, and then perform the matching of the tracking trajectory nodes.

[0023] Preferably, in the step 2, the construction method of the edge relationship between the tracking trajectory nodes in the camera is as follows: use the Delaunay triangulation method to obtain the edge relationship of the target tracking trajectory nodes.

[0024] Preferably, in the step 2, the construction method of the edge relationship between the tracking trajectory nodes in the camera and the key points and the edge relationship between the key points is as follows: use the K-nearest neighbor method to calculate the Euclidean distance between the node and the other nodes for the node, and take the K nearest nodes to construct the edge relationship.

[0025] Preferably, in the step 1, the method for obtaining the key point features of the environment is as follows: use the BRIEF feature extraction algorithm to extract the corner features of the image area except the target box in the monitoring image as the key point features of the environment.

[0026] Beneficial effects:

[0027] (1) The present invention introduces environmental key points and constructs a heterogeneous graph structure in combination with tracking trajectory nodes. This heterogeneous graph structure not only includes the association of tracking trajectory nodes within and between cameras, but also includes the association between tracking trajectory nodes within a camera and environmental key points, between environmental key points, and between environmental key points between cameras, strengthening the association between the target and the environment. Thus, it builds an information transmission bridge for the tracking trajectory to master local and global information, strengthens the association between the tracking target and the environment, so as to provide environmental perception information and enrich the feature representation ability of nodes in depth map matching.

[0028] (2) A cross-attention graph neural network is designed, which mainly includes a self-cross-attention graph convolution module and a mutual-cross-attention graph convolution module, improving the model's ability to represent target features.

[0029] (3) Aiming at the cycle inconsistency problem of multi-camera matching, the present invention designs a matching strategy based on hypergraph clustering, and designs a hypergraph segmentation strategy for the computing power requirements brought by hypergraph clustering. After a match is completed, the multi-camera matching state of the subsequent frame is segmented and judged. If the matching relationship does not change significantly, the matching structure of the previous frame is directly adopted. If the change is large, hypergraph clustering matching is performed again to achieve cycle-consistent matching between multiple cameras. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of a multi-camera multi-target tracking scenario.

[0031] Figure 2 It is a schematic diagram of cycle-consistent matching.

[0032] Figure 3 It is a schematic diagram of cycle-inconsistent matching.

[0033] Figure 4 It is the principle framework of the cross-camera multi-target tracking method based on the heterogeneous graph neural network of the present invention.

[0034] Figure 5 It is a schematic diagram of visualizing triangularized edge relationships.

[0035] Figure 6 It is a schematic diagram of self-crossed edge relationships.

[0036] Figure 7 It is a schematic diagram of mutual-crossed edge relationships. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0037] The present invention provides a cross-camera multi-object tracking scheme based on heterogeneous graph neural network. The core idea is to introduce environmental key points and combine them with tracking trajectory nodes to construct a heterogeneous graph structure. This heterogeneous graph structure not only includes the association between tracking trajectory nodes within and between cameras, but also includes the association between tracking trajectory nodes within a camera and environmental key points, between environmental key points, and between environmental key points between cameras, strengthening the association between the target and the environment. A cross-attention graph neural network is designed to improve the model's ability to represent target features. It solves the problem that the apparent features of observed targets vary greatly under multi-camera perspectives, improving the accuracy of cross-camera multi-object tracking.

[0038] The present invention further designs a matching strategy based on hypergraph clustering for the cyclic inconsistency problem of multi-camera matching, and designs a hypergraph segmentation strategy for the computing power requirements brought by hypergraph clustering. After a match is completed, the multi-camera matching state of the subsequent frame is segmented and judged. If the matching relationship does not change significantly, the matching structure of the previous frame is directly adopted. If the change is significant, hypergraph clustering matching is performed again to achieve cyclic consistent matching between multi-cameras.

[0039] The following takes embodiments in conjunction with the attached drawings to describe the present invention in detail.

[0040] As Figure 4 shown, the cross-camera multi-object tracking method based on heterogeneous graph neural network of the present invention includes the following steps:

[0041] Step 1: For synchronized multi-camera surveillance images, obtain the target tracking trajectory under a single camera, the target apparent feature as the feature of the trajectory node, and the key point feature of the environment.

[0042] In this step, synchronized multi-camera surveillance images are obtained, and the target tracking trajectory and target apparent feature under a single camera are respectively obtained by using the traditional fast-reid technology. Each node in the target tracking trajectory is represented by the target apparent feature.

[0043] In addition, the present invention also extracts the key point feature of the environment. Specifically, the BRIEF feature extraction algorithm can be used to extract the corner features of the image area (environmental area) except the target box in the surveillance image as the key point feature of the environment.

[0044] In this step, two types of nodes and their features are obtained. One type is the tracking trajectory nodes, and the other type is the environmental key point nodes. To form a heterogeneous graph, the edge relationships between these nodes also need to be constructed. Then, in Step 2 and Step 3 below, three types of edge relationships between the two types of nodes within the camera and two types of edge relationships between the two types of nodes between cameras are extracted respectively. These edge relationships include not only target associations but also the associations between the target and the environment, thus building an information transmission bridge for the tracking trajectory to master local and global information, strengthening the association between the tracking target and the environment, providing environmental perception information, and enriching the feature representation ability of the nodes in depth map matching.

[0045] Step 2: Construct the edge relationships between the two types of nodes within the camera.

[0046] In this step, three types of edge relationships between the two types of nodes are constructed, including: the edge relationship between tracking trajectory nodes; the edge relationship between a tracking trajectory node and an environmental key point; the edge relationship between environmental key points; and these three types of edge relationships form self-crossing edge relationships.

[0047] Since the features of each node within the camera are independent of each other, it is mainly considered to establish edge relationships and connections using spatial position information. Different strategies are used to construct the edge relationships between various types of nodes. The specific implementation is as follows:

[0048] (1) For the edge relationship between tracking trajectory nodes: The Delaunay triangulation method is used to determine which nodes need to establish edge relationships, so as to obtain the edge relationships of the target tracking trajectory nodes. Specifically, first, a super triangle is established to enclose the nodes within the triangle. Then, the nodes within the triangle are analyzed in turn. It is analyzed whether the node is within which smallest triangle. If so, the node is established with the three vertices of the triangle. The edge relationship, and so on, until all nodes are analyzed. The Delaunay triangulation method can effectively capture the local geometric structure between tracking trajectories and helps to extract the local similarity features of the trajectories. As Figure 5 shown in the visualization graph of the triangulated edge relationship construction, all the connecting edges in the figure are bidirectional edges.

[0049] (2) For the edge relationship between a tracking trajectory node and an environmental key point and the edge relationship between environmental key points: They are constructed by the K-nearest neighbor method. Specifically, each node is analyzed in turn, the Euclidean distance between the features of the node and all other nodes is calculated, and then the node is established with the K nearest neighbor nodes. The edge relationship, and so on, until all nodes have established edge relationships. This can help the model understand the interaction between the tracking trajectory and the environmental features.

[0050] Step 3: Construct the edge relationships between the nodes between cameras.

[0051] This step constructs two types of edge relationships for two types of nodes, including: the edge relationship between two tracking trajectory nodes of different cameras; the edge relationship between two environmental key points of different cameras. These two types of edge relationships form a cross-edge relationship.

[0052] In the construction of edge relationships between cameras, feature similarity and historical matching strength are mainly considered. If two feature trajectory nodes have a strong matching relationship in history or their feature similarity is high, an edge is established between them. In order to decouple the learning tasks of the self-cross attention network and the cross-cross attention network, in the construction of edge relationships between cameras, edge relationships within a camera are no longer generated, and the nodes in each camera will only establish edge relationships with the nodes in other cameras. As Figure 6 and Figure 7 are respectively the schematic diagrams for constructing the self-cross edge relationship and the cross-cross edge relationship of the graph structure. Nodes of different colors represent targets in different camera graphs. The self-cross edge relationship is only limited within a camera, and the cross-cross edge relationship is only constructed between nodes of different cameras.

[0053] Step Four: Train the heterogeneous graph neural network.

[0054] The heterogeneous graph neural network of the present invention includes two parts: a self-cross attention graph convolution module and a cross-cross attention graph convolution module. The self-cross attention graph convolution module is used to process the target features within a single camera and learn the edge relationships within a single camera. Its input is two types of node features, and the output is the updated features of the two types of nodes. By learning the relationship features between different nodes through the self-cross attention graph convolution module, the attention weights between nodes are obtained, and the attention weights are adjusted during the learning process. Higher weights are assigned to important target features. After obtaining the attention weights, the features of all nodes are updated. The cross-cross attention graph convolution module is used to learn the edge relationships between cameras. Its input is the output of the self-cross attention graph convolution module, and the output is the node features of each tracking target from different camera perspectives. This is similar to the update mechanism of the self-cross attention graph convolution module, where information interaction is performed on all nodes in multiple cameras, integrating the information from different camera perspectives, and updating the node features of each tracking target from different cameras through weighted cross-camera node features.

[0055] Step 4.1: Train the self-cross attention graph convolution module.

[0056] In this step, the target appearance features and environmental key point features of the tracking trajectory nodes are input into the self-cross attention graph convolution module, and the self-cross edge relationship obtained in Step Two is used as the ideal output to optimize the parameters of the self-cross attention graph convolution module.

[0057] Specifically, the features of the tracking trajectory nodes (i.e., the target appearance features) and the features of the environmental key points are input into the self-cross attention graph convolution module, which mainly processes the target features within a single camera, strengthens the locally connected target features through the graph attention mechanism, fuses the appearance and motion features of the targets by analyzing the mutual relationships between the targets from the perspective of a single camera, emphasizes the correlation between the important features within the camera view through the attention weights, and assigns higher weights to the important target features, which helps to learn the relative position relationships between the tracking trajectories.

[0058] For the edge relationship e between the tracking trajectory node and the environmental key point TK , the edge relationship e between the tracking trajectory node and the tracking trajectory node TT , and the edge relationship e between the environmental key points KK , a self-cross attention graph convolution module is designed to learn the relationship features between different nodes. The message passing methods for the three edge relationships are as follows:

[0059]

[0060] Among them, is the node set of the tracking trajectory nodes within the camera, is the node set of the environmental key points within the camera, h i and h j represent taking a node from a certain set. [·||·] represents concatenating multiple features, and W TK , W KK , W TT as well as a TK , a TT are the learnable parameters in the convolutional network; LeakyReLU represents the LeakyReLU activation function.

[0061] The attention α i,j of node i to node j is:

[0062]

[0063] Among them, softmax() represents an activation function. When i and j take nodes from different sets, e(h i , h j ) is equivalent to e TK , e TT or e KK .

[0064] After obtaining the attention weights between the nodes, the features of all the tracking trajectory nodes are updated as Equation (I):

[0065]

[0066] h in formula (I) i ' represents the updated tracking trajectory node features.

[0067] Among them, represents the set of all neighbor nodes of node i whose category is an environmental key point, represents the set of all neighbor nodes of node i whose category is a tracking trajectory node.

[0068] The features of all environmental key point nodes are updated to formula (II):

[0069]

[0070] h in formula (II) i ' represents the updated environmental key point node features.

[0071] Step 4.2: Train the cross-attention graph convolution module.

[0072] In this step, the output data of the self-cross-attention graph convolution module is input into the cross-attention graph convolution module, and the cross-edge relationship obtained in step two is used as the ideal output to optimize the parameters of the cross-attention graph convolution module.

[0073] Specifically, the output of the self-cross-attention graph convolution module is called the environmental perception feature. The environmental perception feature is input into the cross-attention graph convolution module, which processes the information interaction from different cameras. Through the graph attention mechanism, the global connection and mutual influence between targets from different camera perspectives are analyzed to enhance the cross-camera environmental perception ability of the model. In the cross-attention graph convolution module, the information exchange between the tracking trajectory nodes and the feature point nodes is also concerned, but the node relationship between different cameras is particularly emphasized.

[0074] In a multi-camera setting, the nodes in each camera perspective can be regarded as an independent subgraph, and the subgraphs are connected by cross-camera edges. For two tracking trajectory nodes and from different cameras c1, c2 and as well as the environmental key point nodes,

[0075] The edge relationship can be established by calculating the similarity between them.

[0076] Or

[0077] Among them, a cc and W cc are learnable parameters in the cross-attention graph convolution module.

[0078] Similar to the self-cross attention module, the attention weight α of node i to node j i,j is as follows:

[0079]

[0080] where softmax() represents an activation function, represents the set of all cross-camera same-class neighbor nodes of node i.

[0081] The feature update of the node can be achieved through weighted cross-camera neighbor features, similar to the update mechanism in the self-cross attention module, but the attention and weight calculations here are specifically for cross-camera nodes.

[0082] Through the above steps, the mutual cross attention graph convolution module performs information interaction on all nodes in multiple cameras, integrates information from different camera perspectives, and finally outputs the node feature F corresponding to each tracking target CC .

[0083] Step Five: During actual tracking, obtain the surveillance images of each camera, input them into the multi-object tracking model composed of the self-cross attention graph convolution module and the mutual cross attention graph convolution module, and output the predicted tracking trajectory nodes.

[0084] Step Six: According to the surveillance images and the predicted tracking trajectory nodes, perform target matching to achieve target tracking.

[0085] In this step, image matching between two cameras in the prior art can be adopted. However, to solve the loop inconsistency problem of multi-camera matching, the present invention designs a matching strategy based on hypergraph clustering segmentation, that is, the surveillance images of each camera and the predicted results of their tracking trajectory nodes are combined to form a multi-camera hypergraph. The multi-camera hypergraph is a data packet that includes data of all cameras. Specifically for each camera, it includes a camera graph Camera graph The camera graph

[0086] includes a set of tracking trajectory nodes, a set of connection edges of each tracking trajectory node in the camera, and a set of connection edges between different cameras. Cluster the multi-camera hypergraph, and the nodes of the same class correspond to one target to complete the matching of the tracking trajectory nodes.

[0087] Specifically, the trajectory tracking matching process based on the multi-camera hypergraph includes the following steps:

[0088] Given N cameras, camera i includes a camera graph structure corresponding to the set of tracking trajectory nodes in the camera, is the set of connection edges of each tracking trajectory node in the camera, Denote the set of connected edges of each tracking trajectory node between camera \(i\) and camera \(j\). \(F(u)\) and \(E(u, v)\) respectively represent the node feature \(u\) and edge feature \((u, v)\) in the camera.

[0089] Denote the set of all nodes in the multi-camera hypergraph as Denote the set of all connected edges as The set composed of multiple camera graphs is

[0090] Step 62: Multi-camera hypergraph segmentation strategy. Since the number of connected edges in the complete hypergraph is the number of permutations and combinations of the number of nodes in the graph, if a complete global data association is performed every time with the input of the graph structure, it will generate a huge computational requirement, which will seriously affect the real-time performance of the online multi-camera multi-object tracking task.

[0091] In the temporal multi-graph matching task, usually most of the node relationships in the hypergraph remain stable in consecutive time frames, and only a few nodes need to re-evaluate their matching relationships due to the addition of new tracking targets or incorrect matches. Generally, the matching relationships in the hypergraph input each time do not change drastically.

[0092] To solve this problem, the problem can be segmented according to the existing single-camera matching results to simplify the complexity of the matching algorithm. The present invention proposes a method for segmenting the hypergraph. Through the multi-camera hypergraph segmentation strategy, by judging the change of the multi-camera matching relationship, the hypergraph can be segmented into smaller sub-hypergraphs. For those scenes with a large number of changes in a short time, a corresponding detection mechanism can be designed to ensure that all key changes are captured, while for scenes with fewer changes, the existing matching relationships can be stably used, thereby further reducing the computational cost.

[0093] The multi-camera hypergraph segmentation strategy designed by the present invention includes the following steps:

[0094] Step 62-1: Input the tracking results of each frame and the matching results of the previous frame.

[0095] Step 62-2: Generate assumed matching relationships using the temporal matching results Among them, is the multi-camera temporal graph matching relationship at the given time \(t - 1\), and represent the local matching relationships of individual cameras. The matching relationship is expressed by a matrix, where a match is 1 and no match is 0.

[0096] Step 62-3: Calculate the node similarity \(S\) corresponding to the pairwise matching assumed matching edges ij .

[0097] Step 62-4: For the non-isolated node u composed of nodes with historical matching relationships, traverse its neighboring node set, index the camera number corresponding to node v, and store it in the camera stack Stack c in it.

[0098] Step 62-5: If there is a duplicate camera number in the stack Stack c , move node v to the changed node set

[0099] Step 62-6: If the similarity S(u, v) between the non-isolated node u and its neighboring node v is lower than the set θ a for consecutive t a frames, move nodes u and v to the changed node set.

[0100] Step 62-7: Integrate the set of non-abnormal associated nodes into the steady state set The set of non-abnormal associated nodes refers to the set of nodes that did not show abnormalities and followed cycle consistency during the latest matching association. The steady state set refers to the set of nodes with unchanged matching relationships, and there is no need to perform subsequent hypergraph clustering matching again.

[0101] Step 62-8: Update the non-isolated node set to reduce duplicate searches.

[0102] Step 62-9: For the set of isolated nodes that have not been globally searched, if the tracking duration of node u in this set ≥ τ t , move all nodes with similarity greater than θ s to the changed node set, otherwise store node u in the steady state set. The set of isolated nodes that have not been globally searched refers to the set of nodes without matching relationships that have not been globally searched.

[0103] Step 62-10: The steady state node set and its associated edges form a steady state sub-hypergraph

[0104] Step 62-11: The changed node set and its associated edges form a changed sub-hypergraph

[0105] The multi-camera hypergraph segmentation strategy designed by the present invention, by inputting the tracking results of each frame and the matching results of the previous frame, first generates assumed matching relationships, searches for the cycle consistency of all non-isolated nodes based on the assumed matching relationships and verifies the reliability of the assumed matching relationships, and performs new clustering matching on the uncertain matching relationships and the set of uncertain nodes, where τ a , τ t , θ s , θ a are hyperparameters, and fps is the video frame rate.

[0106] Step 63: Multi-camera hypergraph clustering strategy

[0107] The present invention proposes a multi-graph collaborative matching method based on the clustering of tracking trajectory node features, and gives the following criteria:

[0108] (1) Each node of the multi-camera hypergraph has multiple types of depth features, and the nodes in the same cluster form a tight group in the feature space, that is, they have feature similarity;

[0109] (2) For the nodes within the same camera i Its node feature F u And F v Have exclusivity, and this exclusivity has transitivity.

[0110] For two clusters to be merged And The approximate mean of the cluster features Is:

[0111]

[0112] Where S CC (u, v) is the similarity between nodes u and v calculated based on the node features with global information obtained by the cross-attention graph convolutional network.

[0113] For node And a given threshold θ e , there is an exclusive proximity set

[0114]

[0115] The directed exclusivity value C ep From node u to node v and the directed exclusivity mean Are defined as:

[0116]

[0117] Among them, Idx(v) represents the camera number corresponding to node v.

[0118] For two clusters to be merged And The cluster exclusivity consistency mean Can be expressed as:

[0119]

[0120] Based on the above criteria, the multi-camera hypergraph segmentation strategy designed by the present invention includes the following steps:

[0121] Step 63-1: Input the multi-camera hypergraph Steady-state sub-hypergraph and the changing sub-hypergraph

[0122] Step 63-2: Initialize for each node Initialize

[0123] Step 63-3: Calculate the similarity S of all different camera nodes u and v in the changing sub-hypergraph M (u, v), S CC (u, v), and store [u, v] in the stack Stack s inside

[0124] Step 63-4: Calculate the similarity S of node u in the changing sub-hypergraph and each node v in the scene B (u, v)

[0125] Step 63-5: For all nodes (u, v) in the stack Stack s inside, if, then merge the clusters and

[0126] where, represents the set of camera numbers to which all nodes in the cluster belong, and θ e , θ cc is a hyperparameter

[0127] The design of the present invention determines the state of the multi-camera matching relationship by a quantifiable criterion (i.e., the threshold determination process in the hypergraph segmentation strategy). For example, the similarity degree between two adjacent frames and the sub-hypergraphs at the same position is calculated. If the similarity degree is less than the set threshold, it indicates that the matching relationship of the current frame sub-hypergraph changes little or is considered not to have changed, and the multi-camera matching result of the previous frame can be reused without calling the multi-graph matching algorithm; when a change in the multi-graph matching relationship is captured, that is, for the sub-hypergraph with a similarity degree greater than or equal to the set threshold, the hypergraph is divided into a steady-state sub-hypergraph, denoted as and a changing sub-hypergraph, denoted as For the changing sub-hypergraph, re-cluster, and then perform the matching of the tracking trajectory nodes

[0128] The above specific embodiments only describe the design principle of the present invention. The shapes and names of the components in this description can be different and are not limited. Therefore, those skilled in the art of the present invention can modify or equivalently replace the technical solutions recorded in the foregoing embodiments; and these modifications and replacements do not depart from the purpose and technical solutions of the present invention, and shall all fall within the protection scope of the present invention

Claims

1. A cross-camera multi-object tracking method based on heterogeneous graph neural network, characterized in that Including: Step 1: For the synchronized multi-camera monitoring images, obtain the tracking trajectory of the target under a single camera, the target appearance features serving as the characteristics of the tracking trajectory nodes, and the environmental key point features. Step 2: Construct the edge relationships between nodes within a camera and between cameras: The edge relationships between nodes within a camera include: the edge relationships between tracking trajectory nodes, the edge relationships between tracking trajectory nodes and environmental key points, and the edge relationships between environmental key points, which form self-crossing edge relationships. The edge relationships between nodes between cameras include: the edge relationships between two tracking trajectory nodes from different cameras, and the edge relationships between two environmental key points from different cameras, which form cross-crossing edge relationships. Step 3: Train the heterogeneous graph neural network: Input the target appearance features of the tracking trajectory nodes and the environmental key point features into the self-crossing attention graph convolutional module, and use the self-crossing edge relationships obtained in Step 2 as the ideal output to optimize the parameters of the self-crossing attention graph convolutional module. Input the output data of the self-crossing attention graph convolutional module into the cross-crossing attention graph convolutional module, and use the cross-crossing edge relationships obtained in Step 2 as the ideal output to optimize the parameters of the cross-crossing attention graph convolutional module. Step 4: During actual tracking, obtain the monitoring images of each camera, input them into the multi-object tracking model composed of the self-crossing attention graph convolutional module and the cross-crossing attention graph convolutional module, and output the predicted tracking trajectory nodes. Step 5: According to the monitoring images and the predicted tracking trajectory nodes, perform target matching to achieve target tracking.

2. The method according to claim 1, wherein The said Step 5 is: Compose the monitoring images of each camera and their predicted tracking trajectory nodes into a multi-camera hypergraph; each camera contains a camera graph, and the camera graph includes a set of tracking trajectory nodes, a set of connecting edges of each tracking trajectory node within the camera, and a set of connecting edges between different cameras. Cluster the multi-camera hypergraph, and each set of similar nodes corresponds to one target to complete the matching of the tracking trajectory nodes.

3. The method according to claim 2, wherein In Step 5, divide the multi-camera hypergraph into sub-hypergraphs, calculate the similarity degree between adjacent two-frame and same-position sub-hypergraphs. If the similarity degree is less than the set threshold, it means that the matching relationship of the current frame sub-hypergraph changes little and there is no need to perform re-clustering calculation, and directly reuse the matching result of the previous frame; for sub-hypergraphs with a similarity degree greater than or equal to the set threshold, re-cluster and then perform the matching of the tracking trajectory nodes.

4. The method according to claim 2, wherein In the said Step 2, the construction method of the edge relationships between the tracking trajectory nodes within the camera is: Adopt the Delaunay triangulation method to obtain the edge relationships of the target tracking trajectory nodes.

5. The method according to claim 1, wherein In the said Step 2, the construction methods of the edge relationships between the tracking trajectory nodes within the camera and the key points and between the key points are: Adopt the K-nearest neighbor method, calculate the Euclidean distance between the node and the remaining nodes for the node, and take the K nearest nodes to construct the edge relationships.

6. The method according to claim 1, wherein In the said Step 1, the method for obtaining the environmental key point features is: Use the BRIEF feature extraction algorithm to extract the corner features of the image area except the target box in the monitoring image as the environmental key point features.

Images