Vehicle visual relay tracking method based on spatial-temporal feature fusion

Abstract

The invention discloses a vehicle visual relay tracking method based on spatial-temporal feature fusion, and belongs to the technical field of intelligent traffic. After a multi-target tracking trajectory of each camera on a road is obtained, the method comprises the following steps: establishing a camera space-time constraint based on a topological relation between the cameras and the road, performing trajectory compatibility judgment based on the camera space-time constraint, and filtering out incompatible trajectory pairs; modeling the trajectory of the target vehicle into a graph, and calculating the similarity of compatible trajectory pairs based on a graph convolutional neural network; based on the trajectory similarity, the multi-camera trajectory association problem is decomposed, hierarchical clustering is performed on the trajectory by using a heap data structure, and the global association trajectory of each target vehicle is obtained step by step. According to the method, the efficiency and accuracy of track association are improved, the calculation efficiency is optimized, and vehicle tracking in a complex scene and under multiple cameras can be dealt with.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent transportation technology, relates to a vehicle trajectory analysis technology based on vision, and specifically relates to a vehicle vision relay tracking method based on spatiotemporal feature fusion. Background Art

[0002] Visual relay tracking is a key technology in multi-camera vehicle monitoring systems. It aims to achieve all-round and all-time continuous tracking of vehicles through the collaborative work of multiple cameras. This technology has a wide range of applications in traffic management, public safety, smart cities and other fields. Its importance is mainly reflected in the following aspects: Improving public safety: By tracking the vehicle throughout the entire process, it can effectively prevent and combat criminal acts such as theft, hit-and-run, etc.; Optimizing traffic management: Real-time understanding of vehicle flow is helpful for traffic dispatch and congestion management, and improving traffic efficiency; Smart city construction: Visual relay tracking technology provides basic data support for smart cities, which helps to realize the intelligent and refined management of cities.

[0003] At present, the visual relay tracking system mainly consists of the following modules: target detection, target tracking and trajectory association.

[0004] Object detection: Common methods include deep learning-based object detection algorithms, such as Faster R-CNN, YOLO, and SSD. These methods have high detection accuracy in single-camera scenarios, but in multi-camera scenarios, the detection effect may drop significantly due to differences in viewing angles and occlusion problems.

[0005] Object tracking: Common methods include correlation filter tracking algorithm, KCF (Kernelized Correlation Filters) and deep learning-based tracking algorithms (such as Siamese network). These methods perform well under single camera, but in multi-camera scenarios, since the target may switch frequently between different cameras, the tracking effect will be greatly affected.

[0006] Trajectory association: Existing methods mainly perform trajectory association based on appearance features and spatiotemporal information, such as re-identification (Re-ID) technology and spatiotemporal graph models. However, these methods are easily affected by perspective changes and occlusion problems when dealing with trajectory association between different cameras, resulting in low association accuracy.

[0007] The invention patent with publication number CN116311107A disclosed a cross-camera tracking method and system based on inference optimization and neural network on June 23, 2023. The method detects vehicles in the video stream of a single camera, obtains the detection results of all vehicles therein, generates an identity code for the vehicle, performs single-camera tracking based on the identity code, and then reconstructs the cross-camera tracking trajectory to achieve the tracking of the target vehicle. This technical solution mainly relies on inference optimization and neural networks to detect and track vehicles, and reconstructs cross-camera trajectories through identity codes, but lacks an effective spatiotemporal constraint mechanism in the cross-camera tracking process, resulting in high computational complexity and prone to mismatching when processing large amounts of trajectory data.

[0008] The invention application with publication number CN112257683A disclosed a cross-camera tracking method for vehicle trajectory monitoring on January 22, 2021. It uses a generative adversarial network to expand the vehicle video data set, and uses a lightweight convolutional neural network to detect vehicles. The vehicle detection model is deployed on all cameras, and a position association dictionary between cameras is created. The next group of cameras where the vehicle may appear is predicted based on the direction of movement of the vehicle, and the apparent features of the vehicle captured in the current camera are broadcast to the next group of cameras. The vehicle is tracked in the next group of cameras based on the apparent features to achieve cross-camera tracking of the vehicle. This technical solution relies on generative adversarial networks to expand the data set and lightweight convolutional neural networks for detection. It does not fully utilize the spatiotemporal constraint information between cameras, resulting in a lack of relevance and accuracy of the trajectory in practical applications, especially in complex scenarios. Misjudgment is prone to occur.

[0009] The invention application with publication number CN115294169A disclosed a vehicle tracking method, device, electronic device and storage medium on November 4, 2022, which obtains characteristic parameters used to characterize the motion characteristics of the target vehicle, predicts the trajectory points of the target vehicle in the image acquisition blind spot, and ensures the continuity of the trajectory points, so as to achieve cross-camera tracking of the vehicle when the image acquisition areas between the cameras do not overlap. However, this technical solution mainly predicts the trajectory of the vehicle in the blind spot of the camera through motion characteristics. Although it solves the problem of trajectory continuity, the prediction accuracy is low when the camera coverage is wide or the scene is complex, and it cannot ensure that all trajectory points across cameras can be correctly associated.

[0010] The invention patent with publication number CN112991401A disclosed a vehicle trajectory tracking method, device, electronic device and storage medium on June 18, 2021. The method re-identifies the target vehicle in the video captured by the first camera and the second camera, obtains the first image and the second image containing the target vehicle captured by the first camera and the second camera at the same time, and determines the coordinates of the target vehicle in the reference coordinate system at each moment based on the visual geometric relationship between the two cameras to obtain the target vehicle's trajectory. However, this technology relies on re-identification technology and a reference coordinate system to track vehicle trajectories. When the target vehicle is obstructed, the lighting changes, or deformed, the accuracy of re-identification is low, which can easily cause trajectory association errors and affect the overall tracking effect.

[0011] Although the existing visual relay tracking technology has made some progress, it still faces the following challenges: (1) Differences in perspective between multiple cameras: The difference in perspective between different cameras causes the appearance features of the same target to vary greatly under different cameras, increasing the difficulty of trajectory association. (2) Occlusion and complex background: In practical applications, the target may be occluded by other objects or be in a complex background, affecting the accuracy of detection and tracking. (3) Computational efficiency and real-time performance: Multi-camera systems need to process a large amount of video data. How to improve computational efficiency and real-time performance while ensuring accuracy is an important challenge. Summary of the invention

[0012] In view of the problems and challenges existing in the above-mentioned prior art, the present invention provides a vehicle visual relay tracking method based on spatiotemporal feature fusion, which preliminarily filters the multi-camera vehicle trajectories based on the camera's spatiotemporal constraints, and then calculates the trajectory similarity based on a graph convolutional neural network (GCN) to capture the spatiotemporal feature information between trajectories. A multi-level hierarchical clustering method is used to achieve efficient trajectory association, which improves the efficiency and accuracy of trajectory association, optimizes the computational efficiency, and can cope with vehicle tracking in complex scenes and under multiple cameras.

[0013] The present invention provides a vehicle visual relay tracking method based on spatiotemporal feature fusion. First, a multi-target tracking trajectory of each camera on the road is obtained, and the trajectory is a single camera trajectory SCT. Then, the following steps are performed, including:

[0014] Step 1, the trajectory filtering module performs trajectory compatibility judgment based on the camera spatiotemporal constraints and filters out incompatible trajectory pairs; continue to perform step 2 for compatible trajectory pairs. The camera spatiotemporal constraints are defined by three matrices: compatibility matrix C, minimum time window matrix Tmin and maximum time window matrix Tmax; the compatibility matrix C is based on the road topology and camera position, and the elements in C take values ​​0 or 1, 1 represents that the trajectory between the two cameras can be associated, and 0 represents that the trajectory between the two cameras cannot be associated; the minimum time window matrix Tmin records the shortest time required for the trajectory from one camera to another; the maximum time window matrix Tmax records the longest time required for the trajectory from one camera to another. The trajectory filtering module initially uses the camera spatiotemporal constraints to judge the compatibility of each pair of SCTs; after obtaining the multi-camera trajectory MCT, the compatibility of the MCT pair is judged. As long as there is a pair of SCTs in the MCT pair that are compatible, the two MCTs are compatible.

[0015] Step 2: construct a graph model for the single camera trajectory in chronological order, and calculate the similarity of compatible single camera trajectory pairs through the trajectory similarity calculation module based on the graph convolutional neural network. The trajectory similarity calculation model includes a node embedding module, a graph level embedding module, a graph-graph interaction module, and a prediction similarity module. Suppose a pair of trajectory models are obtained by graph G i and G j , the graph of each trajectory and the node features in the graph are input into the node embedding module for graph convolution, and the embedding vector of each node in the input trajectory graph is obtained and output to the graph-level embedding module and the graph-graph interaction module; the graph-level embedding module aggregates the embedding vectors of the nodes in the trajectory graph based on attention, obtains the global embedding of the trajectory graph and outputs it to the graph-graph interaction module; the neural tensor network model and the pairwise node comparison model are set in the graph-graph interaction module, the neural tensor network model calculates the similarity score of the global embedding vectors of the two trajectories, and the pairwise node comparison model calculates the similarity score of the embedding vectors of the paired nodes of the two trajectory graphs and represents it as a histogram feature; the prediction similarity module uses a fully connected layer to convert the similarity score of the input global embedding vector and the similarity score of the node embedding vector into the final similarity of the two single camera trajectories.

[0016] Step 3, the trajectory hierarchical clustering module uses a hierarchical clustering variant to gradually cluster the trajectories to obtain associated trajectories. The hierarchical clustering variant includes three clustering methods: single link clustering, average link clustering and complete link clustering. Single link clustering is to define the similarity between two MCTs as the similarity of the most similar SCT pairs contained in the two MCTs. Average link clustering is to define the similarity between two MCTs as the average value of the similarities of all SCT pairs contained in the two MCTs. Complete link clustering is to define the similarity between two MCTs as the minimum value of the similarities of all SCT pairs contained in the two MCTs. The stepwise clustering of trajectories includes: step 3.1) placing SCT pairs or MCT pairs into a pile according to similarity, wherein the SCT pairs or MCT pairs at the top of the pile have the highest similarity; step 3.2) taking out SCT pairs or MCT pairs from the top of the pile in turn and merging them into one MCT pair, filtering the MCT pairs through a trajectory filtering module, and recalculating the similarity of the filtered MCT pairs using any clustering method in a hierarchical clustering variant; step 3.3) repeating steps 3.1 and 3.2 until all MCTs of the same target vehicle are merged into one global MCT, or a preset similarity threshold is reached, and the iteration of the target is stopped; and finally the associated trajectory of each target vehicle is output.

[0017] Compared with the prior art, the advantages and positive effects of the present invention are:

[0018] (1) The trajectory filtering based on the spatiotemporal constraints of the camera can effectively filter out trajectory pairs that cannot be associated, and more accurately screen out valid trajectory data. The prior art often ignores the importance of spatiotemporal constraints in trajectory data filtering, which may result in some trajectory data being unable to be correctly associated. By introducing spatiotemporal constraints, the method of the present invention can filter out trajectories that do not meet the association conditions based on the spatiotemporal constraints of the camera, thereby significantly improving the accuracy and effectiveness of trajectory data in a multi-camera system and reducing interference from mismatches and irrelevant data.

[0019] (2) The trajectory similarity calculation based on the graph convolutional neural network in the method of the present invention significantly improves the accuracy of trajectory matching. Traditional methods usually rely on simple distance metrics or shape matching, and it is difficult to handle trajectory similarity calculation problems in complex environments. The method of the present invention utilizes the powerful feature extraction and learning capabilities of the graph convolutional neural network to more effectively capture the complex similarity relationships between trajectories, especially considering the spatiotemporal characteristics of surrounding vehicles, making up for the shortcomings of traditional re-identification features and improving the accuracy and robustness of trajectory matching. Through this method, the method of the present invention can achieve accurate trajectory similarity calculation in more complex traffic environments, greatly improving the practicality of the system.

[0020] (3) The multi-camera trajectory association method based on hierarchical clustering in the present invention provides a more efficient trajectory association solution. The existing technology often faces the problems of high computational complexity and low association accuracy in multi-camera trajectory association. The present invention effectively integrates and associates trajectory data from different cameras by introducing hierarchical clustering association trajectories, improves the accuracy and computational efficiency of multi-camera trajectory association, and ensures the effective tracking and association of global motion trajectories. The trajectory hierarchical clustering module not only simplifies the calculation process, but also enhances the practicality and scalability of the multi-camera system in large-scale data processing scenarios. (4) The present invention has made significant progress in trajectory data filtering, similarity calculation and multi-camera trajectory association. Compared with the existing technology, the present invention not only improves the accuracy and reliability of the overall system, but also significantly improves the processing efficiency, and has stronger practical application value and promotion potential. These beneficial effects reflect the advancement and superiority of the present invention in the field of multi-camera trajectory processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is an overall implementation framework diagram of vehicle tracking using the vehicle visual relay tracking method of the present invention;

[0022] Figure 2 is a pseudo code implementation diagram for performing compatibility judgment on two associated track MCT pairs in an embodiment of the present invention;

[0023] Figure 3 is a vehicle trajectory modeling diagram based on graph convolutional neural network trajectory similarity calculation in an embodiment of the present invention;

[0024] Figure 4 It is a framework diagram for implementing trajectory similarity calculation based on graph convolutional neural network according to an embodiment of the present invention;

[0025] Figure 5 is a schematic diagram of using the CityFlow S06 scene experimental data set to experiment with the method of the present invention;

[0026] Figure 6 It is the indicator evaluation result using the method of the present invention on the experimental data set;

[0027] Figure 7 It is the indicator evaluation result of the traditional method of re-identification ID on the experimental dataset. DETAILED DESCRIPTION

[0028] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0029] In order to solve the problems and challenges of the prior art, the present invention proposes a vehicle visual relay tracking method based on spatiotemporal feature fusion, and the core ideas of its implementation are as follows:

[0030] (1) Implement trajectory filtering based on camera spatiotemporal constraints: By considering the topological relationship between the camera and the road, the vehicle trajectory is initially filtered to reduce the trajectory pairs that cannot be associated, thereby reducing noise and improving the efficiency of subsequent calculations and the accuracy of association. This step, as the basis of the entire tracking method, provides preliminary screening and lays a good foundation for subsequent trajectory similarity calculation and multi-camera trajectory association.

[0031] (2) Implement trajectory similarity calculation based on graph convolutional neural network: Model the trajectory data in the form of a graph, and use a graph convolutional neural network (GCN) to fully utilize the characteristics of the vehicle and its surrounding environment to capture the spatiotemporal feature information between trajectories. This process makes up for the shortcomings of traditional re-identification features and improves the accuracy of trajectory similarity calculation. As the core calculation part of the tracking method, GCN provides a more accurate trajectory similarity measurement and provides high-quality input data for multi-camera trajectory association. Based on the preliminary filtering, the graph convolutional neural network further refines the trajectory similarity calculation to ensure that only highly similar trajectory pairs enter the next step of clustering and association.

[0032] (3) Realizing multi-camera trajectory association based on hierarchical clustering: When performing trajectory association between cameras, traditional methods usually cluster all trajectories on a large scale according to the similarity matrix, but this method is difficult to ensure accurate association of the same vehicle, and may mistakenly cluster different vehicles, resulting in association errors. To solve this problem, the method of the present invention decomposes the multi-camera trajectory association problem into several sub-problems through hierarchical clustering according to the trajectory similarity calculated in the previous step, and gradually clusters and associates the trajectories. In the association process, the overall similarity of multiple trajectories is used for measurement to improve the robustness of the system. For different scenarios, single link clustering, average link clustering and complete link clustering are flexibly adopted to achieve effective tracking and association of global trajectories. This hierarchical clustering method not only improves the accuracy and robustness of trajectory association, but also enhances the adaptability of the system in various application scenarios. As the final execution step of the entire tracking method, it ensures efficient and accurate association of multi-camera trajectories, adapting to the configuration of different camera systems and the needs of different application scenarios.

[0033] like Figure 1 As shown, the embodiment of the present invention specifically implements the vehicle visual relay tracking method based on spatiotemporal feature fusion of the present invention as a multi-camera association module in the figure, Figure 1In the process, the video captured by each camera on the road is obtained through the traffic digital twin data engine, and the captured video data is first processed by a single camera. Vehicle detection, re-identification feature extraction, and single-camera multi-target tracking are performed on the video data of each camera to obtain the trajectory under a single camera. Then, the trajectory under a single camera is associated with multiple cameras, and the global trajectory under multiple cameras is obtained by trajectory filtering based on camera spatiotemporal constraints, trajectory similarity calculation based on graph convolutional neural network, and multi-camera trajectory association based on hierarchical clustering. Finally, the result is output for visualization. The functions realized by a single camera are realized by the existing technology, and the embodiments of the present invention will not be repeated. The implementation of the multi-camera association module involved in the method of the present invention is specifically described below. The multi-camera association module includes a trajectory filtering module, a trajectory similarity calculation module, and a trajectory hierarchical clustering module.

[0034] Step 1: The trajectory filtering module obtains the multi-target tracking trajectories under a single camera on the road, filters the trajectories based on the camera's spatiotemporal constraints, and proceeds to the following steps for compatible trajectory pairs.

[0035] In order to effectively filter out conflicting trajectories and improve the efficiency and accuracy of trajectory association, the present invention proposes a trajectory filtering method based on camera spatiotemporal constraints. This method mainly uses three matrices to define the spatiotemporal constraints between cameras: the compatibility matrix C, the minimum time window matrix Tmin, and the maximum time window matrix Tmax.

[0036] (1) The compatibility matrix C is represented as a k×k binary matrix, where k represents the number of cameras. The compatibility matrix represents the trajectory compatibility between k cameras. Each element C(i,j) in the C matrix can be 0 or 1, with the following meanings: C(i,j) = 1, indicating that the trajectories between camera i and camera j are correlated;

[0037] C(i,j)=0, indicating that the trajectories between camera i and camera j cannot be associated.

[0038] By using the C matrix, some trajectory pairs that are unlikely to be associated can be initially filtered out, thereby reducing the complexity of subsequent calculations.

[0039] In the embodiment of the present invention, the road topology and camera location information of the studied area are obtained in advance, and the compatibility matrix C is set according to the road topology where the camera is located. Generally, the trajectory compatibility value between cameras located on the same section of road (the road between two intersections) can be set to 1, and the trajectory compatibility value between adjacent cameras located at the same intersection can be set to 1; or, it can be directly set that as long as the roads where the two cameras are located can be connected, the trajectory compatibility value of the two cameras is 1.

[0040] (2) The time window matrix is ​​composed of the Tmin matrix and the Tmax matrix, which represent the shortest time and the longest time required for a trajectory to travel from one camera to another. For example, the matrix element Tmin(i,j) represents the shortest time required for a trajectory to travel from camera i to camera j, and Tmax(i,j) represents the longest time required for a trajectory to travel from camera i to camera j. These two matrices are used to define the reasonable time range for a trajectory to appear between different cameras, ensuring the rationality of the trajectory in space and time.

[0041] (3) Perform trajectory filtering according to the camera spatiotemporal constraints of the above three matrices, determine whether each pair of trajectories in the two cameras are compatible, and filter incompatible trajectories. In order to determine whether two trajectories should be filtered, the present invention designs an algorithm for determining the compatibility of two trajectories.

[0042] Suppose there are two cameras i and j, and two trajectories A and B extracted from these two cameras. Trajectory A leaves camera i at time t, and trajectory B appears in camera j at time t′. It is necessary to determine whether these two trajectories should be associated or filtered out. The determination steps are as follows:

[0043] Step 1.1: Calculate the compatibility of two trajectories. Get the compatibility matrix value C(i,j) between camera i and camera j. If C(i,j)=0, directly filter out trajectory A and trajectory B because the trajectories between these two cameras cannot be associated. If C(i,j)=1, continue to check the time window constraint.

[0044] Step 1.2: Check the time window constraint. Get the time t when trajectory A leaves camera i, and the time t′ when trajectory B appears in camera j; get the shortest time Tmin(i,j) and the longest time Tmax(i,j) from camera i to camera j; check whether time t′ is within the time window [t+Tmin(i,j),t+Tmax(i,j)], that is, if t+Tmin(i,j)≤t′≤t+Tmax(i,j), then trajectory A and trajectory B are likely to be related in time and space, and the trajectory pair is retained; if time t′ does not meet the above time window conditions, filter out the trajectory pair.

[0045] Initially, the single camera trajectory (SCT) of multiple targets is obtained through single camera tracking, and the compatibility of the SCT pairs is judged. The subsequent steps are performed for the compatible trajectory pairs. After the multi-camera trajectory (MCT) is obtained through the trajectory hierarchical clustering module, the compatibility of the MCT pairs is judged. As long as one pair of SCTs in two MCTs is compatible, the two MCTs are compatible. MCT is composed of several SCTs. Let two MCTs be T i and T j , each MCT is represented as a collection of multiple SCTs, judging the two trajectories Ti and T j The compatibility of Figure 2 As shown, perform the following steps:

[0046] (1) Determine trajectory T i and T j Is it in the same camera? If so, for the current trajectory pair (T i ,T j ) is not processed;

[0047] (2) According to steps 1.1 and 1.2, traverse and detect T i and T j The compatibility of each pair of SCTs in T i The trajectory segment T a and T j The trajectory segment T b , get T a and T b Corresponding camera c a and c b , and T a On camera c a The starting time of appearance T a .Start and end time T a .end,T b On camera c b The starting time of appearance T b .Start and end time T b .end, then detect:

[0048] If the corresponding element C(c a ,c b )>0 and T b .start≤(T a .end+T max (c a ,c b )) and T b .end≥(T a .end+T min (c a ,c b ), then the trajectory T a and T b Compatible;

[0049] If the corresponding element C(c b ,c a )>0 and T a .start≤(T b .end+T max (c b ,c a)) and T a .end≥(T b .end+T min (c b ,c a ), then the trajectory T b and T a Compatible.

[0050] When a pair of trajectories T is detected b and T a When they are compatible, it means that the trajectory T i and T j Compatible, if T i and T j All pairs of trajectories in are incompatible, which means that trajectory T i and T j Incompatible, filter out the trajectory pairs (T i ,T j ).

[0051] Through the dual filtering of compatibility and time window constraints mentioned above, the number of trajectory pairs that need to be processed can be significantly reduced. First, the trajectories from incompatible camera pairs are filtered out. Secondly, the trajectories that do not meet the requirements in time are further filtered out. These filtering steps reduce the complexity and search space of subsequent trajectory association calculations, and improve the efficiency and accuracy of multi-camera trajectory association. This trajectory filtering method based on camera spatiotemporal constraints ensures that most of the trajectory pairs that cannot be associated are excluded before trajectory similarity calculation and multi-camera trajectory association, thereby optimizing the overall processing flow.

[0052] Step 2: The trajectory similarity calculation module performs trajectory similarity calculation based on graph convolutional neural network.

[0053] The trajectory similarity calculation method based on graph convolutional neural network of the present invention models the trajectory data into a graph form and uses deep learning technology to make full use of the features of other vehicles around the vehicle, effectively capturing the spatiotemporal feature information between trajectories, thereby making up for the deficiency of re-identification features.

[0054] Step 2.1, first model the trajectory into a graph. For each trajectory a, let the vehicle corresponding to the trajectory be v, and model each vehicle trajectory a into a graph in chronological order. This modeling method not only considers the spatial proximity between vehicles, but also captures the temporal characteristics of vehicle movement, so that the trajectory data can be efficiently processed in the form of a graph.

[0055] like Figure 3 As shown in the figure, T and T-1 represent the trajectory time frame, v represents the vehicle node, and the graph model is constructed for the continuous trajectory a to form the trajectory graph G i , including the following:

[0056] (1) Each frame of trajectory a is modeled as a graph in chronological order. The nodes in the graph are vehicles, and edges are established between the current vehicle v and all vehicles in the current frame. The edges between different nodes represent the spatial proximity of vehicles at the same time.

[0057] (2) In each frame, connect the car v in the current frame with the car v in the previous frame.

[0058] (3) The features of each node in the graph include the vehicle’s re-identification features and current location information.

[0059] Step 2.2, design a trajectory similarity calculation model. By designing a learnable embedding function, each modeled graph is mapped to an embedding vector, which provides a global summary of the entire graph, that is, the re-identification feature of the trajectory. This step abstracts and encodes the trajectory at the graph level, so that subsequent similarity calculations can be performed more efficiently.

[0060] like Figure 4As shown in the figure, the trajectory similarity calculation model mainly consists of four parts: node embedding, graph-level embedding, graph-graph interaction, and predicted similarity. In the node embedding module, the input is the features of the graph and the node. Through graph convolution, each node of each graph is converted into a new vector, which encodes the features of the node and the graph structure properties and represents the features of the current node. The purpose of the node embedding stage is to capture the local relationship between nodes within each graph. Then, in the graph-level embedding module, the node embedding vectors generated in the previous stage are integrated through an attention-based aggregation method, namely the attention module ATT, to generate a global embedding vector for the trajectory. This global embedding not only considers the features of each node, but also the spatiotemporal relationship of the nodes, thereby more comprehensively expressing the features of the entire trajectory. The global embedding vector represents the features of the current trajectory. The graph-graph interaction module adopts two strategies: neural tensor network and pairwise node comparison. The neural tensor network model receives two graph-level embeddings and returns an interaction score representing the similarity between the two graph-level embeddings through a series of neural tensor network operations. This process is not just a simple vector comparison, but a complex similarity measurement based on the learned graph-level embedding. The pairwise node comparison model adopts a direct pairwise node comparison method, and directly uses the node-level embedding vector to calculate the similarity score between pairs of nodes. The advantage of the pairwise node comparison method is that it can capture the differences and similarities between each pair of nodes in more detail. In order to further enhance the utilization of node-level information, the method of the present invention introduces histogram features in the pairwise node comparison to represent the pairwise node similarity matrix. This fine-grained node-level comparison strategy provides additional information gain when processing trajectory similarity, enhancing the expressiveness and performance of the model. Finally, the output of the graph-graph interaction vector is converted into a final similarity score by the prediction similarity module. The prediction similarity module of the embodiment of the present invention is implemented using a fully connected layer, and the pairwise node comparison features are combined with the neural tensor network features and input into the fully connected layer to obtain the final similarity score. This similarity score is used to update the model parameters and is compared with the true similarity score, thereby optimizing the accuracy and generalization ability of the trajectory similarity calculation model. The implementation of the trajectory similarity calculation model in the embodiment of the present invention is described in detail below.

[0061] (1) Implementation of the node embedding module. First, we use graph convolutional networks (GCNs) to generate the embedding vector of each node. The core idea of ​​GCNs is to combine the features of each node with the features of its neighboring nodes through neighborhood aggregation to generate the representation of the node. The calculation formula of GCNs is as follows:

[0062]

[0063] in, represents the embedding vector of node n at layer l+1, represents the embedding vector of node m at layer l, is the set of neighbor nodes of node n, d n and d m are the degrees of node n and neighbor m respectively. The statistical degree includes the node itself, W (l) is the weight matrix of the lth layer, and f1 is the activation function. The above formula indicates that the embedding vector of a node is generated by the weighted sum of its neighboring node embeddings. Let the node embedding module finally output the trajectory graph G i , G j The node embedding matrix is ​​U i , U j .

[0064] (2) Implementation of the graph-level embedding module. In order to calculate the similarity between graphs, we first need to generate a global embedding representation for the entire graph. This is achieved through a global context-aware attention mechanism, which generates the overall embedding of the graph by weighted summing up the embeddings of the nodes in the graph.

[0065] First, the global context vector c of the graph is calculated as follows:

[0066]

[0067] Among them, h n is the embedding vector of node n output by the node embedding module, N is the number of nodes in the graph, and W2 is the learnable weight matrix. The global context vector c contains the global structural information of the graph. After that, the attention weight is calculated for each node:

[0068]

[0069] Among them, a n is the attention weight of node n, with a superscript represents transpose, σ(·) is the sigmoid function, which maps the result of the inner product to the range [0,1].

[0070] Finally, the global embedding vector h of the graph graph It is obtained by weighted summing of node embeddings as follows:

[0071]

[0072] By the attention weight a n , we can adaptively select which nodes have a greater impact on the similarity of the graph. Suppose the graph-level embedding module outputs the trajectory graph G i , G j The global embedding vectors are h i and h j .

[0073] (3) Implementation of the neural tensor network model in the graph-graph interaction module. After obtaining the global embedding of the graph, the next step is to calculate the similarity between two graphs. In order to better capture the relationship between graphs, a neural tensor network (NTN) is used to model the interaction between graphs. NTN captures multiple interactions between graphs through tensor operations and generates K similarity scores. NTN calculates the global embedding h of two graphs. i and h j The similarity score g(h i ,h j ) is as follows:

[0074]

[0075] Among them, W3 [1:K] is a weight tensor, b3 is the bias term, V is the weight matrix, and f is the activation function.

[0076] (4) Implementation of the pairwise node comparison model in the graph-graph interaction module. Although the global embedding of a graph provides the overall information of the graph, in some cases, the detailed differences between two graphs may only exist in the local structure. Therefore, the method of the present invention further obtains fine-grained similarity information by pairwise node comparison.

[0077] For each pair of nodes in the two graphs, their similarity score matrix S is calculated as follows:

[0078]

[0079] in, and is the node embedding matrix of the two graphs, N i and N j are the number of nodes in the two graphs, and D is the dimension of the node embedding vector. The similarity between nodes is calculated by inner product, and then the sigmoid function is used to map it to the range of [0,1]. The node similarity matrix S is processed to generate the histogram feature hist(S)∈R B , B is the number of buckets of the histogram. The similarity scores between nodes are assigned to different intervals, and histogram features are generated to supplement the information in the global embedding.

[0080] (5) The prediction similarity module calculates the final similarity of the two trajectories. The similarity score g(h i ,h j ) is combined with the node-level histogram feature hist(S) and input into a fully connected neural network to output the predicted two graphs G i , G j Similarity score as follows:

[0081]

[0082] Among them, FC(·) represents the fully connected layer operation.

[0083] (6) Design of loss function when training trajectory similarity calculation model. The method of the present invention trains the trajectory similarity calculation model through mean square error loss (MSE), with the goal of minimizing the error between the predicted similarity and the true similarity, as follows:

[0084]

[0085] Where M is the training set, |M| is the number of graph pairs in the training set, and s(G i ,G j ) is the graph pair (G i ,G j )’s true similarity.

[0086] Step 3: The trajectory hierarchical clustering module realizes multi-camera trajectory association based on hierarchical clustering, and finally outputs the associated trajectory of each target vehicle.

[0087] The multi-camera trajectory association method based on hierarchical clustering of the present invention is designed for a general camera system, so it can be applied to various camera layouts by modifying the three matrix parameters of the camera spatiotemporal constraints, aiming to effectively integrate vehicle trajectories from different cameras, thereby achieving efficient tracking and association of global vehicle motion trajectories. After obtaining the similarity of the trajectory pairs, a trajectory merging operation is performed, and the merging process is gradually merged based on the heap-based hierarchical clustering. The trajectory hierarchical clustering module associates single-camera trajectories into multi-camera trajectories through hierarchical clustering variants, and further associates MCT. Hierarchical clustering variants include single-link clustering, average-link clustering, and complete-link clustering.

[0088] Single-link clustering: The similarity between two MCTs is defined as the similarity between the most similar SCT pairs contained in the two MCTs. If a pair of SCTs in two MCTs has a high degree of similarity, the two MCTs are considered similar. This method is suitable for discovering local associations.

[0089] Average link clustering: The similarity between two MCTs is defined as the average of the similarities between all SCT pairs contained in the two MCTs. This approach takes into account the average similarity between all SCTs, helps to understand the relationship between MCTs in a broader context, and reduces the impact of a single SCT on the results.

[0090] Complete Linkage Clustering: The similarity between two MCTs is defined as the minimum similarity between all SCT pairs contained in the two MCTs. This approach ensures that two MCTs are considered similar only when there is significant similarity between all SCT pairs. Therefore, this approach is more rigorous and is often used to find high consistency and alignment in global motion patterns.

[0091] In order to achieve high efficiency and effectiveness of hierarchical clustering, the trajectory hierarchical clustering module uses a heap data structure to gradually cluster the trajectories through a hierarchical clustering variant, including the following steps:

[0092] Step 3.1, heap operation: put all possible MCT pairs into a heap according to their similarity, so that the top of the heap always contains the MCT combination with the highest similarity.

[0093] Step 3.2, gradual merging: Take the two MCT combinations with the highest similarity from the top of the pile, merge them into a new MCT, and recalculate the similarity between the new MCT and other MCTs.

[0094] The trajectory hierarchical clustering module initially uses single-link clustering to cluster SCT pairs, performs heap operations on SCT pairs according to similarity, gradually merges them, obtains MCT, and then uses the trajectory filtering module to judge the compatibility of MCT pairs and filter out incompatible MCT pairs; then single-link clustering, average-link clustering, or complete-link clustering is used to recalculate the similarity of MCT pairs.

[0095] Step 3.3, iterative processing: Repeat the above steps 3.1 and 3.2 until all MCTs of the same target are merged into a global MCT, or the preset similarity threshold condition is reached, and the iteration is stopped. Finally, the global MCT or MCT combination of each target vehicle is output. The preset similarity threshold condition here refers to the minimum similarity threshold of the two MCT sets of the target vehicle. If the threshold is not met, stop merging the MCT pairs.

[0096] The method of the present invention is experimented with the CityFlow S06 scene cross-camera vehicle tracking dataset. There are 6 cameras in the S06 scene, distributed as follows: Figure 5 As shown in the figure, looking at the effect of the relay tracking of the two cameras C43 and C42, we can see that the same vehicle is assigned the same tracking target ID 207 by both cameras. Figure 5 shown. Figure 5 The vehicle ID, color, model and speed are marked for each identified vehicle. At the same time, the method of the present invention and the traditional method that only uses the re-identification ID are evaluated and compared on the data set. The evaluation results are as follows: Figure 6 and Figure 7As shown in the figure, MTMC represents the result of multi-camera association, and the evaluation indicators include: identity F1 score IDF1, which is obtained by calculating the harmonic mean of IDP and IDR; identity precision IDP, which measures the consistency of the correct identity in the detected trajectory; identity recall rate IDR, which measures the proportion of target identities correctly matched in the real trajectory; recall rate Rcll, which calculates the proportion of targets successfully tracked in the real target, and measures the target missed detection; precision Prcn, which calculates the proportion of real targets in all detected targets, which is used to measure false detection; the number of real trajectories GT, which is the number of target trajectories actually existing in the statistical data set; the mostly tracked indicator MT, which refers to the number of real target trajectories that are successfully captured in the tracking, and the tracked length exceeds 80%; the partially tracked indicator PT, which refers to the number of real target trajectories that are partially captured in the tracking, and the tracking length is between 20% and 80%; the mostly lost indicator ML, which refers to the number of target trajectories that are mostly lost in the tracking, and the tracking length is less than 20%; the number of false detections FP, which refers to the number of misdetected targets, that is, the situation where there is no real target The number of missed detections FN is the number of real targets that were lost, that is, targets that were not detected; the number of identity switching times IDs is the number of times the same target switches its identity during the tracking process, which is usually a manifestation of tracking failure; the number of trajectory fragmentation times FM is the number of times the tracking trajectory of a single target is interrupted, reflecting the continuity of the trajectory; the multi-target tracking accuracy MOTA comprehensively considers the impact of missed detections, false detections and identity switching; the multi-target tracking precision MOTP measures the spatial accuracy of tracking, mainly the accuracy of the target position; the number of identity transfer times IDt indicates the number of times the identity of a target is converted between different cameras or different time points; the number of identity assignments Ida is the total number of times the tracking system assigns identities to the detected targets; the number of identity merging times IDm is the number of times the system mistakenly merges different targets into the same identity during the tracking process; the number of identity false detections idfp is the number of real targets to which a wrong identity is assigned; the number of identity missed detections idfn is the number of identities of targets missed by the tracking system; the number of identity correct detections idtp is the number of real targets to which the system correctly detects and assigns identities. By comparison, it can be found that the use of the method of the present invention improves the accuracy and effectiveness of trajectory data, reduces interference from mismatches and irrelevant data, and improves the accuracy of trajectory association.

[0097] Except for the technical features described in the specification, all of them are known technologies to those skilled in the art. The present invention omits the description of known components and known technologies to avoid redundancy and unnecessary limitation of the present invention. The implementation methods described in the above embodiments do not represent all implementation methods consistent with the present application. On the basis of the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative labor are still within the protection scope of the present invention.

Claims

1. A vehicle visual relay tracking method based on spatiotemporal feature fusion, characterized in that: The steps include: Step 1: Obtain the multi-target tracking trajectory of each camera on the road. The trajectory is a single camera trajectory SCT. The trajectory filtering module performs trajectory compatibility judgment based on the camera's spatiotemporal constraints and filters out incompatible trajectory pairs. Continue to execute step 2 for compatible trajectory pairs. The camera spatiotemporal constraints are defined by three matrices: a compatibility matrix C, a minimum time window matrix Tmin, and a maximum time window matrix Tmax; the compatibility matrix C is set according to the road topology and the camera position, and the elements in C are 0 or 1, 1 represents that the trajectories between the two cameras can be associated, and 0 represents that the trajectories between the two cameras cannot be associated; The minimum time window matrix Tmin records the shortest time required for a trajectory to go from one camera to another; The maximum time window matrix Tmax records the maximum time required for the trajectory to go from one camera to another; The trajectory filtering module initially uses the camera spatiotemporal constraints to judge the compatibility of each pair of SCTs; after obtaining the multi-camera trajectory MCT, the compatibility of the MCT pair is judged. As long as there is a pair of SCTs in the MCT pair that is compatible, the two MCTs are compatible; Step 2: construct a graph model for the single camera trajectory in chronological order, and calculate the similarity of compatible single camera trajectory pairs based on the graph convolutional neural network through the trajectory similarity calculation module; The trajectory similarity calculation model includes a node embedding module, a graph-level embedding module, a graph-graph interaction module and a prediction similarity module. Suppose a pair of trajectory models are obtained by graph G i and G j , the graph of each trajectory and the node features in the graph are input into the node embedding module for graph convolution, and the embedding vector of each node in the input trajectory graph is obtained and output to the graph-level embedding module and the graph-graph interaction module; the graph-level embedding module aggregates the embedding vectors of the nodes in the trajectory graph based on attention, obtains the global embedding vector of the trajectory and outputs it to the graph-graph interaction module; the graph-graph interaction module sets a neural tensor network model and a pairwise node comparison model, the neural tensor network model calculates the similarity score of the global embedding vectors of the two trajectories, and the pairwise node comparison model calculates the similarity score of the embedding vectors of the paired nodes of the two trajectory graphs and represents it as a histogram feature; The prediction similarity module uses a fully connected layer to convert the similarity scores of the input global embedding vector and the node embedding vector into the final similarity of the two trajectories; Step 3: The trajectory hierarchical clustering module uses a hierarchical clustering variant to gradually cluster the trajectories to obtain associated trajectories. The hierarchical clustering variant includes three clustering methods: single-link clustering, average-link clustering and complete-link clustering; single-link clustering is to define the similarity between two MCTs as the similarity of the most similar SCT pairs contained in the two MCTs; average-link clustering is to define the similarity between two MCTs as the average of the similarities of all SCT pairs contained in the two MCTs; complete-link clustering is to define the similarity between two MCTs as the minimum value of the similarities of all SCT pairs contained in the two MCTs; The stepwise clustering of trajectories includes: step 3.1) placing SCT pairs or MCT pairs into a pile according to similarity, wherein the SCT pairs or MCT pairs at the top of the pile have the highest similarity; step 3.2) taking out SCT pairs or MCT pairs from the top of the pile in turn and merging them into one MCT pair, filtering the MCT pairs through a trajectory filtering module, and recalculating the similarity of the filtered MCT pairs using any clustering method in a hierarchical clustering variant; step 3.3) repeating steps 3.1 and 3.2 until all MCTs of the same target vehicle are merged into one global MCT, or a preset similarity threshold is reached, and the iteration of the target is stopped; and finally the associated trajectory of each target vehicle is output.

2. The method according to claim 1, characterized in that In step 1, the compatible matrix C is a k×k binary matrix, where k represents the number of cameras. The setting method in which the value in the matrix is ​​1 includes: (1) setting the trajectories of cameras located on the same section of road to be associated, and setting the trajectories of adjacent cameras located at the same intersection to be associated; or (2) if the roads where the two cameras are located are connected, setting the trajectories of the cameras to be associated.

3. The method according to claim 1 or 2, characterized in that: In step 1, the method for determining the compatibility of each pair of trajectories includes: assuming that trajectory A is extracted from camera i, trajectory B is extracted from camera j, trajectory A leaves camera i at time t, and trajectory B appears in camera j at time t′, and the following calculation is performed for the trajectory pair (A, B): Step 1.1: Perform trajectory compatibility check, including: obtaining the compatibility value C(i, j) of camera i and camera j from the compatibility matrix C. If C(i, j) = 0, it means that trajectories A and B are incompatible, and the trajectory pair (A, B) is directly filtered out. If C(i, j) = 1, continue to perform time window constraint check; Step 1.2: Perform a time window constraint check, including: obtaining Tmin(i, j) and Tmax(i, j) from the minimum time window matrix Tmin and the maximum time window matrix Tmax respectively; checking whether the time t′ is within the time window [t+Tmin(i, j), t+Tmax(i, j)], if so, it means that trajectories A and B are compatible, otherwise it means that trajectories A and B are incompatible, and the trajectory pair (A, B) is filtered out.

4. The method according to claim 3, characterized in that In step 1, assume that the associated trajectory T of multiple targets is obtained from a single camera i and T j , each associated trajectory is represented as a set of multiple single-camera trajectory segments. If two associated trajectories T i and T j In the same camera, the track pairs (T i , T j );otherwise , traverse detection T i and T j For each trajectory pair in , determine whether it meets the trajectory compatibility and time window constraint requirements. If neither of them is satisfied, it means that the trajectory pair (T i , T j ) are incompatible, filter out the trajectory pair (T i , T j ), otherwise the trajectory pair (T i , T j ) compatible.

5. The method according to claim 1, characterized in that In the step 2, a graph model is constructed for the trajectory of the vehicle v in chronological order, including: the node in each frame of the trajectory is a vehicle, the features of the node include the re-identification features and the current position of the vehicle, for each frame of the vehicle v, an edge is established between v and all other vehicle nodes in the current frame, and a connection edge is established between the vehicle v in the current frame and the previous frame.

6. The method according to claim 1, characterized in that In step 2, the implementation of each module in the trajectory similarity calculation model is as follows: The node embedding module uses a graph convolutional network to generate an embedding vector for each node in the trajectory graph, where the embedding vector of node n at the l+1 layer is represents the embedding vector of node m at the,layer, is the set of neighbor nodes of node n, d n and d m are the degrees of node n and its neighbor node m, respectively, and W (l) is the weight matrix of the lth layer, f1 is the activation function; let the embedding vector of the final output node n of the node embedding module be h n ; Assume that the node embedding module is from the trajectory graph G i and G j Get the node embedding matrix U respectively i and U j ; The graph-level embedding module first calculates the global context vector of the trajectory graph Where N is the number of nodes in the graph, W2 is the learnable weight matrix; then calculate the attention weight of each node in the graph, the attention weight of node n The superscript T represents transposition, and σ(·) is the sigmoid function. Finally, the global embedding vector of the trajectory graph is obtained by weighted summing of the embedding vectors of all nodes in the graph. Assume that the graph-level embedding module obtains the trajectory graph G i and G j The global embedding vectors are h i and h j ; In the graph-graph interaction module, the neural tensor network model is used to i and h j Calculate the similarity score g(h i ,h j ), by comparing the two nodes, the model is i and G j Calculate the similarity score for each pair of nodes in and get the similarity score matrix Generate histogram feature hist(S); The prediction similarity module converts the graph G output by the graph-graph interaction module into i and G j The similarity score g(h i ,h j ) and the histogram feature hist(S) are input into a fully connected neural network to predict the final similarity score of the two trajectories FC(·) represents the fully connected layer operation.

7. The method according to claim 1, characterized in that In step 3, the trajectory hierarchical clustering module initially clusters the SCT pairs using single linkage clustering, performs a heap operation on the SCT pairs according to similarity, and gradually merges them to obtain MCTs.

Images