Multi-camera multi-target trajectory prediction method based on traffic flow and graph convolution

Abstract

The present invention belongs to the field of smart transportation. The present invention discloses a multi-camera multi-target trajectory prediction method based on traffic flow and graph convolution, and utilizes traffic flow to achieve multi-camera multi-target efficient trajectory prediction. The present invention constructs a target tracking demand prediction model, and constructs a target tracking demand prediction based on the spatiotemporal correlation of nodes through a spatiotemporal multi-graph convolutional network and a gated recurrent neural network based on an attention mechanism, thereby obtaining target tracking prediction demand. Based on traffic flow, the next camera spatial position and time window where the target may appear are predicted, so as to call the corresponding camera and time period, and an MCMT fast trajectory prediction algorithm with minimized loss rate is proposed. The present invention utilizes the influence of traffic flow to predict the position and time period where the target to be detected will appear, and realizes target detection by extracting a small number of frames from video data collected by multiple cameras, detecting the target in a small number of frames, and improving the efficiency and accuracy of detection.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent transportation and relates to a multi-camera multi-target (MCMT) trajectory prediction method based on traffic flow and graph convolution. Background Art

[0002] With economic development and increasing public safety concerns, multi-camera surveillance systems are being deployed on a large scale in various public places, including hospitals, train stations, and roads. Traditional multi-camera surveillance systems can already achieve intelligent multi-target monitoring, tracking, and trajectory prediction. Two approaches are used: manually searching for a given target and its appearance time and location, which results in low accuracy and a long time consumption. The other approach uses intelligent algorithms to search for the target in all relevant areas and at all times. These algorithms require monitoring, tracking, and trajectory prediction based on a massive database, resulting in high computational effort and time consumption. In the transportation sector, traditional multi-camera surveillance systems fail to consider the impact of traffic flow on target monitoring, tracking, and trajectory prediction.

[0003] Therefore, a MCMT trajectory prediction method based on traffic flow and graph convolution is proposed. Traffic flow is used to predict the next node and time period where the target may appear, which greatly reduces the target tracking calculation amount and efficiently realizes multi-camera multi-target trajectory prediction. Summary of the Invention

[0004] The present invention aims to provide a MCMT trajectory prediction method based on traffic flow and graph convolution.

[0005] First, a spatiotemporal target tracking prediction model based on the spatial and temporal correlations of nodes is constructed using a spatiotemporal multi-graph convolutional network and an attention-based gated recurrent neural network. This model captures the node target tracking prediction requirements. Then, based on traffic flow, the next possible camera location and time window where the target is likely to appear are predicted, allowing for the appropriate camera and time window to be called. A fast trajectory prediction algorithm, called Multi-Channel Multi-Objective-Transport (MCMT), is designed to minimize the loss rate, effectively implementing MCMT trajectory prediction.

[0006] The present invention specifically comprises the following steps:

[0007] Step 1: Through the spatiotemporal multi-graph convolutional network and the gated recurrent neural network based on the attention mechanism, a target tracking demand prediction model based on node spatial correlation and temporal correlation is constructed to obtain the target tracking demand prediction results.

[0008] First, the spatial correlation of target tracking needs is considered from four dimensions: node proximity, node connectivity, edge traffic flow, and target type to be identified. Then, a global gated recurrent neural network based on an attention mechanism is used to model the correlation in time series. This attention-based gated recurrent neural network is used to extract the impact of different time slots on target tracking demand prediction, and the attention mechanism is used to assign weight coefficients to the predicted values. Finally, a multi-graph convolutional network is used to extract and further fuse features to obtain target tracking demand prediction results.

[0009] Step 2: After determining the target tracking requirements based on the prediction model, we first construct a node topology and define a time consumption model. Next, we calculate the time cost of the target tracking prediction model. Finally, we predict the next camera and time window where the target is likely to appear, allowing us to promptly call the corresponding camera and specify the time period for the call.

[0010] Step 3: Design the MCMT minimum loss rate fast trajectory prediction algorithm and obtain the MCMT minimum loss rate fast trajectory prediction algorithm strategy Achieve MCMT trajectory prediction with minimal computational effort.

[0011] The present invention utilizes the influence of traffic flow to predict the location and time period where the target to be detected will appear, and realizes target detection by extracting a small number of frames from video data collected by multiple cameras. The target can be detected in a small number of frames, thereby improving the efficiency and accuracy of detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 The figure is a diagram of the overall architecture of the method described in the present invention.

[0013] Figure 2 Schematic diagram of the node of the present invention.

[0014] Figure 3 It is a four-branch graph of the node spatial correlation model.

[0015] Figure 4 Schematic diagram of the model of node time correlation. DETAILED DESCRIPTION

[0016] The present invention will be further described below with reference to the accompanying drawings.

[0017] like Figure 1 As shown in Figure 1, a MCMT trajectory prediction method based on traffic flow and graph convolution includes the following steps:

[0018] Step 1: Through the spatiotemporal multi-graph convolutional network and the gated recurrent neural network based on the attention mechanism, a target tracking demand prediction based on the spatial correlation and temporal correlation of nodes is constructed to obtain the target tracking demand prediction result.

[0019] like Figure 2 As shown in , the location of a surveillance camera on the road is abstracted into a point, and the intersection with the surveillance point is represented as a node. The surveillance points are connected by roads, which means that there is an edge between the corresponding two points. The spatial correlation required for target tracking is considered from four dimensions: node proximity, node connectivity, edge traffic flow, and target type to be identified. Figure 3 The graph structure shown is used to represent its nonlinear correlation, and the feature extraction and further fusion of multiple graphs are achieved through a multi-graph convolutional network.

[0020] Graph Structure represents the spatial correlation between nodes, where S is the set of nodes in the graph, It is the adjacency matrix of the graph, which represents the connectivity of the vertices in the graph. Is the degree matrix of the graph, which represents the degree of the nodes in the set S. The Laplacian matrix of the graph is obtained by the adjacency matrix A and the degree matrix D of the graph. The symmetric normalized Laplacian matrix L = ID -1 / 2 AD -1 / 2 , where I is the identity matrix.

[0021] Node proximity Represents the correlation between the spatial locations of nodes, the formula express.

[0022] Connectivity between two nodes means that the two nodes are directly connected through roads, and the formula Indicates that, since the node proximity relationship in the node proximity and the connectivity relationship in the node connectivity graph have overlapping parts, in order to reduce the complexity of the node connectivity graph, the correlation represented in the node proximity graph is excluded, and the above formula can be expressed as

[0023] Different nodes have different spatial positions in the planning, which leads to different traffic flows on the corresponding edges. Therefore, when predicting target tracking needs, the influence of traffic flows on the edges needs to be considered. Indicates that use Represents node v i and v j The traffic flow between the edges, where and Represents node v i and v j The more the traffic flow features between two nodes overlap, the higher the correlation between them.

[0024] The types of targets that need to be identified at different nodes can also indicate the correlation between nodes. This correlation can be determined by judging the degree of overlap of targets at different nodes. The formula is: Indicates. Represents node v i and v j The type correlation of the target needs to be identified, where and Represents node v i and v j The more overlap there is between the type labels of the targets that need to be identified between two nodes, the higher the correlation between them.

[0025] The global features of nodes and the spatial features between nodes are extracted through multi-graph convolutional networks. represents the node observation value at time slot τ, where P represents the dimension of the node feature vector. l+1 =σ(∑ A∈z f(A;θ i )X l W l ),in and Represent the node feature vectors of the l+1th layer and the lth layer respectively. σ represents the activation function ReLU. f(A; θ i ) represents the Laplacian matrix. represents the feature transformation matrix.

[0026] In the time dimension, there are similarities between traffic conditions in different time periods, and the similarities in different time periods are also different. Therefore, a global gated recurrent neural network based on the attention mechanism is used to model the correlation in the time series.

[0027] like Figure 4 The specific steps are as follows:

[0028] The first step is to aggregate the features of the current node and related nodes through the multi-graph convolutional network, that is, through the formula Get, where Θ represents the aggregation function, which aggregates the feature information of the current node and the feature information of the spatial correlation node to form a node feature description. The feature extraction of spatial correlation is achieved through multi-graph convolution operation, where represents multi-graph convolution operation, Represents the maximum degree of the Laplacian matrix of the graph.

[0029] The second step is to use the attention mechanism to assign different weight coefficients to the observation values of different time slot nodes. First, the observation information of each node in the time slot τ is compressed by the global average pooling operation to extract the global features of the region. Perform average global pooling. Represents the global average pooling operation, and |S| represents the number of nodes. The attention mechanism assigns different weights to different observation sequences, investing more attention resources on this node to obtain more detailed information of the target node that needs to be paid attention to, while suppressing other useless information. Through the formula Assign weights, where s represents the weight vector of the time observation sequence, This represents the operation of assigning weights to observation sequences at different times. W1 and W2 are weight parameters, and σ and φ represent the activation functions ReLU and sigmoid, respectively. By assigning weights to the original sequence, a new observation sequence is obtained.

[0030] Step 3: The gated input sequence of each node is aggregated into a single vector Ω through the shared RNN layer. i , as shown in the formula As shown, W3 represents the weight parameter.

[0031] Step 2: Based on the prediction of traffic flow between nodes, in order to meet the information preparation for target tracking, the next node and time window where the target may appear are predicted so that the corresponding camera can be called in time and the calling time period can be clearly defined.

[0032] Construct a network graph G(V, E) to represent the node topology graph, where V represents the node set, that is, V = {v i , i=1,...,N}. E represents the set of edges, e jk Represents the slave node v j to v k Considering the time-varying factors caused by traffic flow, time is discretized, that is, the edge at time t is defined k-1 and t k The time slot between k , k=1,2,...n.

[0033] R i (τ) is the node v i According to step 1, X l The obtained eigenvalue can represent the traffic flow at the intersection. Usually, an intersection has multiple traffic directions. represents the node v at time slot τ i To v j Traffic flow in the direction. Let node v i The road capacity at time slot τ is Q i(τ). When node v i Traffic flow R at time slot τ i (τ) is greater than the road capacity Q i (τ), that is, R i (τ)>Q i (τ), so that R i (τ) = Q i (τ).

[0034] make For the target slave node v i Move to v j The rate of the process in the time slot τ. Then in the time slot τ, the total time cost of target tracking prediction demand By formula calculate.

[0035] Step 3: Design a fast trajectory prediction algorithm that minimizes the loss rate of MCMT.

[0036] The time consumption of moving between two nodes of the task is mainly determined by the traffic flow, so the probability of selecting the next node is where v m Represents the slave node v i The exit node, m = 1, 2, ..., M, Indicates that in time slot τ k From node v i To the exit node v m Movement of traffic flow.

[0037] Finally, the next node with the highest probability of the target going to is selected based on the historical traffic flow.

[0038] The input of the MCMT fast trajectory prediction algorithm with minimum loss rate includes the node target tracking demand historical observation data O |S| (τ1), O |S| (τ2),...,O |S| (τ k-1 ), the output of the algorithm includes MCMT minimization loss rate fast trajectory prediction algorithm strategy

[0039] The specific execution steps of the algorithm are as follows:

[0040] First, input the historical observation data of node target tracking demand, and make predictions through the target tracking demand prediction model to obtain the time slot τ k Node forecast demand O |S| (τ k ).

[0041] Next, the MCMT minimization loss rate fast trajectory prediction algorithm strategy and the set of nodes that do not satisfy the target tracking prediction request Set to an empty set. Track the demand according to the predicted target O |S| (τ k ) Compute node v i Traffic flow R i (τ k ). Node v i Traffic flow R at time slot τ i (τ) is greater than the road capacity Q i (τ), then R i (τ) = Q i (τ). Otherwise, mark the current node and put it into the set For each node v i ∈V, select the node v with the maximum probability that the target will go to m If m≠-1, it means that a node that meets the conditions exists, then make node v m It is v i The next node.

[0042] The final output is the MCMT minimum loss rate fast trajectory prediction algorithm strategy

Claims

1. A multi-camera multi-target trajectory prediction method based on traffic flow and graph convolution, characterized by: The specific steps include: Step 1: Through the spatiotemporal multi-graph convolutional network and the gated recurrent neural network based on the attention mechanism, a target tracking demand prediction model based on node spatial correlation and temporal correlation is constructed to obtain the target tracking demand prediction results; First, the spatial correlation of target tracking requirements is considered from four dimensions: node proximity, node connectivity, edge traffic flow, and target type to be identified; Then, a global gated recurrent neural network based on the attention mechanism models the correlation in the time series. The attention-based gated recurrent neural network is used to extract the impact of different time slots on target tracking demand prediction, and the attention mechanism is used to assign weight coefficients to the predicted values. Finally, a multi-graph convolutional network is used to extract and further fuse features to obtain the target tracking demand prediction results. Step 2: After obtaining the target tracking requirements based on the prediction model, we first construct a node topology and define a time consumption model. Then, we calculate the time cost of the target tracking prediction model. Finally, we predict the next camera and time window where the target may appear, so that we can call the corresponding camera and specify the time period in time. Step 3: Design the MCMT minimum loss rate fast trajectory prediction algorithm and obtain the MCMT minimum loss rate fast trajectory prediction algorithm strategy , achieving multi-camera multi-target trajectory prediction with minimal computational effort.

2. The multi-camera multi-target trajectory prediction method based on traffic flow and graph convolution as claimed in claim 1, characterized in that: In step 1, the location of a surveillance camera on the road is abstracted into a point, and the intersection with the surveillance point is represented as a node. The surveillance points are connected by roads, which means that there is an edge between the corresponding two points; the spatial correlation required for target tracking is considered from four dimensions: node proximity, node connectivity, edge traffic flow, and target type to be identified. The nonlinear correlation is represented by a graph structure, and the feature extraction and further fusion of multiple graphs are achieved through a multi-graph convolutional network.

3. The multi-camera multi-target trajectory prediction method based on traffic flow and graph convolution as claimed in claim 2, characterized in that: The graph structure described in step 1 represents the spatial correlation between nodes, where is the set of nodes in the graph, is the adjacency matrix of the graph, which represents the connectivity of the vertices in the graph; is the degree matrix of the graph, representing the set The degree of the node in the graph; the Laplacian matrix of the graph is obtained through the adjacency matrix of the graph Sum degree matrix Get, symmetric normalized Laplace matrix ,in is the identity matrix; Node proximity Represents the correlation between the spatial locations of nodes, the formula express; Connectivity between two nodes means that the two nodes are directly connected through roads, and the formula Indicates that, since the node proximity relationship in the node proximity and the connectivity relationship in the node connectivity graph have overlapping parts, in order to reduce the complexity of the node connectivity graph, the correlation represented in the node proximity graph is excluded, and the above formula can be expressed as ; Different nodes have different spatial positions in the planning, which leads to different traffic flows on the corresponding edges. Therefore, when predicting target tracking needs, the influence of traffic flows on the edges needs to be considered. Indicates that use Representation node and The traffic flow between the edges, where and Represents nodes respectively and The more the traffic flow features between two nodes overlap, the higher the correlation between them. The types of targets that need to be identified at different nodes can also indicate the correlation between nodes. This correlation can be determined by judging the degree of overlap of targets at different nodes. The formula is: indicates; among them Representation node and The type correlation of the target needs to be identified, where and Representation node and The type vector of the target to be identified; the more overlaps the type labels of the targets to be identified between two nodes, the higher the correlation between them; Extract the global features of nodes and the spatial features between nodes through multi-graph convolutional networks; make Indicates that in the time slot Node observation value of Represents the dimension of the node feature vector; through the formula of multi-graph aggregation ,in and Respectively represent Layer and The node feature vector of the layer; Represents the activation function ReLU; represents the Laplace matrix; represents the feature transformation matrix.

4. The multi-camera multi-target trajectory prediction method based on traffic flow and graph convolution as claimed in claim 1, characterized in that: Modeling the correlation of time series through the global gated recurrent neural network based on the attention mechanism as described in step 1; The specific steps are as follows: The first step is to aggregate the features of the current node and related nodes through the multi-graph convolutional network, that is, through the formula Get, where Represents the aggregation function, which aggregates the feature information of the current node and the feature information of the spatial correlation node to form a node feature description; the feature extraction of spatial correlation is achieved through multi-graph convolution operation, where represents multi-graph convolution operation, Represents the maximum value of the Laplacian matrix degree of the graph; The second step is to use the attention mechanism to assign different weight coefficients to the observation values of different time slot nodes; first, the time slot is averaged by the global average pooling operation. The observation information of each node is compressed to extract the global features of the region; By formula Perform average global pooling; represents the global average pooling operation, Indicates the number of nodes; the attention mechanism assigns different weights to different observation sequences, investing more attention resources on this node to obtain more detailed information of the target node that needs to be paid attention to, while suppressing other useless information; By formula Assign weights, where represents the weight vector of the time observation sequence, Indicates the operation of assigning weights to observation sequences at different times. and is the weight parameter, and Represent the activation functions ReLU and sigmoid respectively; by assigning weights to the original sequence, a new observation sequence is obtained; Step 3: The gated input sequence of each node is aggregated into a single vector through a shared RNN layer. , as shown in the formula As shown, Represents the weight parameter.

5. The multi-camera multi-target trajectory prediction method based on traffic flow and graph convolution as claimed in claim 1, characterized in that: Step 2 of the above describes the construction of a network diagram Represents the node topology graph, where Represents a node set, i.e. represents a set of edges, Represents a slave node arrive Considering the time-varying factors caused by traffic flow, time is discretized, that is, the definition of and The time slot between ; is a node According to step 1 The obtained eigenvalue represents the traffic flow at the intersection. Usually, an intersection has multiple traffic directions. Indicates that in the time slot Node Towards Traffic flow in the direction; let the node In the time slot The road capacity is ; When the node In the time slot Traffic flow ,Right now When ; make Target slave node Exercise to During the time slot The rate within the time slot The total time cost of target tracking prediction requirements By formula calculate.

6. The multi-camera multi-target trajectory prediction method based on traffic flow and graph convolution as claimed in claim 1, characterized in that: The MCMT fast trajectory prediction algorithm for minimizing loss rate described in step 3 is specifically as follows: First, input the historical observation data of the node target tracking demand, predict it through the target tracking demand prediction model, and obtain the time slot Node forecast demand ; Then, the MCMT minimization loss rate fast trajectory prediction algorithm strategy and the set of nodes that do not satisfy the target tracking prediction request Set to an empty set; track demand based on predicted targets Compute nodes of ;node In the time slot Traffic flow , then ; Otherwise, mark the current node and put it into the collection ; For each node , select the node with the maximum probability that the target will go to ;if , it indicates that the node that meets the conditions exists, then the node yes The next node; The final output MCMT minimizes the loss rate fast trajectory prediction algorithm strategy .

Images