A vehicle-person correlation analysis method based on spatio-temporal feature fusion

Abstract

This invention discloses a method for human-vehicle association analysis based on spatiotemporal feature fusion, comprising the following steps: For all checkpoints in the target road network, two different devices are used to collect vehicle license plate numbers and mobile phone IMSI codes respectively; the collected data is preprocessed, and license plate numbers and IMSI codes collected from the same checkpoint within the same time period are grouped into a sample; a graph is constructed for each sample, with license plate numbers and IMSI codes as nodes; a graph attention network is used to capture the spatial association features of each license plate number or IMSI code within each time slice, and a temporal convolutional long short-term memory network is used to capture the temporal association features of the license plate number and IMSI code under different time slices, generating the final vector representations of the license plate number and IMSI code; finally, the similarity between all license plate number and IMSI code vectors is calculated, and the IMSI code or license plate number with the highest similarity is selected as the association result. This invention can effectively capture the spatiotemporal association between license plate numbers and IMSI codes, improving the accuracy of human-vehicle association analysis.

Description

Technical Field

[0001] This invention relates to the field of spatiotemporal data processing technology, specifically to a method for analyzing the relationship between people and vehicles based on spatiotemporal feature fusion. Background Technology

[0002] Current methods for linking people and vehicles mainly fall into two categories. The first is the early approach that relies on predefined rules and simple statistical features to perform association analysis and build a connection between license plate numbers and mobile phone numbers. These methods are usually difficult to handle complex spatiotemporal relationships and have poor accuracy and robustness. The second method is to use video analysis to extract vehicle images and facial images from videos, and then obtain vehicle information and personal identity information based on the vehicle images and facial images respectively to perform human-vehicle association. This type of method requires a lot of computing resources to process video data and is complex, with poor real-time performance and environmental adaptability. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention proposes a human-vehicle association analysis method based on spatiotemporal feature fusion, which effectively integrates spatial and temporal features to improve the accuracy of human-vehicle association.

[0004] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0005] A method for analyzing the relationship between people and vehicles based on spatiotemporal feature fusion includes the following steps:

[0006] S1. For all checkpoints in the target road network, use two different devices to collect vehicle license plate numbers and mobile phone IMSI codes respectively, forming license plate number related data and IMSI code related data; the license plate number related data includes checkpoint number, license plate number, and collection time, and the IMSI code related data includes checkpoint number, IMSI code, and collection time;

[0007] S2. Preprocess the collected license plate number and IMSI code related data to filter out missing data, redundant data, outlier data, and irrelevant data.

[0008] S3. Divide the preprocessed license plate numbers and IMSI codes according to the checkpoint number and collection time. Combine the license plate numbers and IMSI codes collected at the same checkpoint within the same time period into a sample, and form a sample set from all samples.

[0009] S4. Treat each sample as a document and train it using a word embedding model to obtain the initial word embedding for each license plate number and IMSI code.

[0010] S5. Construct a graph for each sample, with each license plate number and IMSI code as a node, and the word embeddings of the license plate number and IMSI code as the initial feature vector of the corresponding node; establish an edge between every two nodes, and calculate the weight of the edge based on the time difference of data collection between the two nodes.

[0011] S6. For each license plate number, obtain the graph containing the node of the license plate number in each time slice; for each obtained graph, use a graph attention network to capture the spatial features of the nodes and obtain the new feature vector of the license plate number node in each time slice.

[0012] S7. The feature vector corresponding to the license plate number in each time slice is used as the time series input and passed to the temporal convolutional long short-term memory network to capture the temporal features of the nodes and obtain the hidden state of the license plate number at each time step.

[0013] S8. Use the average pooling method to aggregate the hidden states at each time step to obtain the final vector representation of each license plate number.

[0014] S9. Repeat steps S6-S8. For each IMSI code, obtain the graph containing the nodes of that IMSI code in each time slice. Use a graph attention network to extract the spatial features of the nodes to obtain the new feature representation of the IMSI code node in each time slice. Then, use the new feature vector of each time slice as a time series input to a temporal convolutional long short-term memory network to extract the temporal features of the nodes to obtain the hidden state of the IMSI code at each time step. Then, use the average pooling method to aggregate the hidden states output at each time step to obtain the final vector representation of each IMSI code.

[0015] S10. Calculate the cosine similarity of the final vector representation of all license plate numbers and IMSI codes; for any license plate number or IMSI code, select the IMSI code or license plate number with the highest cosine similarity as its association result.

[0016] Preferably, the data preprocessing method is as follows:

[0017] S211. If any field value in the checkpoint number, license plate number or IMSI code, or collection time is missing in a certain data entry, then the data entry shall be deleted directly.

[0018] S212. For data collected at the same checkpoint within the same time period that have different collection time field values ​​but the same license plate number or IMSI code field values, retain the data with the earliest collection time and delete the rest of the redundant data.

[0019] S213. If the license plate number or IMSI code in the data does not conform to the rules, filter out the data; similarly, if the collection time field value exceeds the set time range, filter out the data that exceeds the range.

[0020] S214. If a license plate number or IMSI code appears only once in the data collected from all checkpoints, then delete all data related to that license plate number or IMSI code.

[0021] S215. The preprocessed data is used to form a new set of data related to license plate numbers and IMSI codes.

[0022] Preferably, the data segmentation method starts from 00:00:00, with each minute as a time period, and merges the license plate number and IMSI code collected from each checkpoint in each time period into a single sample data.

[0023] Preferably, the method for calculating the edge weights in the graph structure is as follows: for any pair of nodes p and q, the weight of the edge (p, q) can be expressed as... Where t p and t q These represent the collection times for nodes p and q, respectively, in seconds. Finally, the weight matrix W for each graph is obtained.

[0024] As a preferred embodiment, the spatial feature extraction method is as follows:

[0025] S611. Each node i in the graph has an initial feature vector h′. i These eigenvectors are transformed using a learnable weight matrix M:

[0026] h″ i =Mh′ i

[0027] S612. For each node j connected to node i, calculate the attention coefficient e for the weight of the fused edge. ij :

[0028] e ij =LeakyReLU(a T [h″ i ||h″ j ||w ij ])

[0029] Among them, a T is a learnable weight vector, || denotes vector concatenation, LeakyReLU is an activation function with a negative slope, w ij Let (i, j) be the weight of edge (i, j).

[0030] S613. Use the Softmax function to adjust the attention coefficient e. ij Normalization is performed to ensure that the sum of the attention coefficients of all neighbors of node i is 1;

[0031] S614. Use normalized attention coefficients to perform a weighted summation of the features of neighboring nodes to obtain a new feature vector x for node i. i ;

[0032] S615. Repeat S611-S614 to obtain a new vector representation of the license plate number in each time slice.

[0033] Preferably, the temporal convolutional long short-term memory neural network is composed of stacked temporal convolutional layers and long short-term memory layers. The input of the temporal convolutional layer is a time series composed of new feature vectors of the license plate number in each time slice. The layer uses dilated convolutional layers to perform convolution operations on the time series data to extract local features in the time series, and then outputs new feature vectors of the license plate number at each time step and passes them to the next layer. The input of the long short-term memory layer is the output of the temporal convolutional layer. For each time step, the input vector is calculated through the input gate, forget gate, candidate memory cells and output gate to obtain the hidden state of that time step. Finally, the hidden states of each time step are used as the output of the temporal convolutional long short-term memory network.

[0034] This invention has the following characteristics and beneficial effects:

[0035] By employing the above technical solution, a graph is constructed based on the spatial and temporal relationships between vehicles and people. This allows for the consideration of not only the spatial correspondence between people and vehicles but also their temporal correspondence during human-vehicle association analysis, resulting in more accurate association results. This invention is based on a graph attention network, utilizing an attention mechanism to capture the spatial association features between people and vehicles appearing at the same location. Simultaneously, this invention introduces a temporal convolutional long short-term memory network to capture the temporal association features between people and vehicles at different time slices. By fusing the spatial and temporal features of people and vehicles, this invention can more comprehensively capture the relationships between them, effectively improving the accuracy of human-vehicle association analysis. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1This is a flowchart of the human-vehicle association analysis method based on spatiotemporal feature fusion proposed in this invention;

[0038] Figure 2 This is a schematic diagram of the data preprocessing process;

[0039] Figure 3 This is a schematic diagram of the spatial feature extraction process;

[0040] Figure 4 This is a schematic diagram of the structure of a human-vehicle association analysis method based on spatiotemporal feature fusion. Detailed Implementation

[0041] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings.

[0043] Conversely, this invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of the invention as defined in the claims. Furthermore, to provide a better understanding of the invention, certain specific details are described in detail below. However, those skilled in the art will fully understand the invention even without these detailed descriptions.

[0044] In a preferred embodiment of the present invention, a method for human-vehicle correlation analysis based on spatiotemporal feature fusion is provided, such as... Figure 1 and Figure 4 As shown, the specific steps are as described in S1 to S10. In this method, in addition to using a graph attention network to capture the spatial features of license plate numbers and IMSI codes at different checkpoints, a temporal convolutional long short-term memory network is also used to mine the temporal features of license plate numbers and IMSI codes at different time slices. By combining the spatial and temporal features of the data, accurate association between people and vehicles is achieved. See [link to relevant documentation]. Figure 1 As shown below, the specific implementation methods of the above steps S1 to S10 will be described in detail.

[0045] S1. For all checkpoints in the target road network, use two different devices to collect vehicle license plate numbers and mobile phone IMSI codes respectively, forming license plate number related data and IMSI code related data; the license plate number related data includes checkpoint number, license plate number, and collection time, and the IMSI code related data includes checkpoint number, IMSI code, and collection time.

[0046] It should be noted that the license plate number mentioned above is used to identify the vehicle, and the mobile phone IMSI code is used to identify the person. The association between the license plate number and the IMSI code is used to realize the association between the person and the vehicle. The data collection period lasts for one week, that is, data is collected continuously for 7 days from the specified start time. The license plate number data of the vehicle at the checkpoint and the IMSI code data of the mobile phone can be obtained by the corresponding equipment or sensors set up on the road in actual application scenarios. This is existing technology and will not be described in detail.

[0047] S2. Preprocess the collected license plate number and IMSI code related data to filter out missing data, redundant data, outlier data, and irrelevant data.

[0048] Specifically, such as Figure 2 As shown, the process is as follows:

[0049] S211. If any field value in the checkpoint number, license plate number or IMSI code, or collection time is missing in a certain data entry, then the data entry shall be deleted directly.

[0050] S212. For data collected at the same checkpoint within the same time period that have different collection time field values ​​but the same license plate number or IMSI code field value, retain the data with the earliest collection time and delete the rest of the redundant data.

[0051] S213. If the license plate number or IMSI code in the data does not conform to the rules, filter out the data; similarly, if the collection time field value exceeds the set time range, filter out the data that exceeds the range.

[0052] It should be noted that a regular license plate number is a combination of 6 to 8 Chinese characters, uppercase letters, and numbers. The first character is the abbreviation of the province in Chinese, the second character is a letter, and the remaining characters are numbers or letters. The IMSI code is a 15-digit all-digit code. The first 3 digits are the valid Mobile Country Code (MCC), the next 2 to 3 digits are the Mobile Network Code (MNC), and the rest are the Subscriber Identity Serial Number (SIN).

[0053] S214. If a license plate number or IMSI code appears only once in the data collected from all checkpoints, then delete all data related to that license plate number or IMSI code.

[0054] S215. The preprocessed data is used to form new sets of data related to license plate numbers and IMSI codes, denoted as C′=(c1, c2, ..., c n′ ) and P′=(p1,p2,...,p m′ ), where n′ and m′ represent the number of license plate numbers and IMSI codes related data after preprocessing, respectively.

[0055] S3. Divide the preprocessed license plate numbers and IMSI codes according to checkpoint number and collection time. Group the license plate numbers and IMSI codes collected from the same checkpoint within the same time period into one sample. Starting from 00:00:00, each minute is a time period. Combine the license plate numbers and IMSI codes collected from each checkpoint within each time period into one sample data set, forming a total of T×K samples, where T is the total number of time slices and K is the total number of checkpoints. The set of all samples can be represented as B = (b1, b2, ..., b...). T ), where b i =(b i，1 b i，2 , ..., b i，k Let b represent the sample set for the i-th time slice. i，j This represents a sample.

[0056] It should be noted that each sample only contains the license plate number and IMSI code, and does not include the checkpoint number or collection time.

[0057] S4. Treat each sample as a document and train it using the Word2Vec model to obtain the initial word embeddings for each license plate number and IMSI code.

[0058] S5. Construct a graph for each sample, using the license plate number and IMSI code as nodes, and the word embeddings of the license plate number and IMSI code as the initial feature vectors of the corresponding nodes; establish an edge between every two nodes, and calculate the weight of the edge based on the time difference of data collection between the two nodes.

[0059] It should be noted that graph construction is a prior art technique in Graph Neural Networks (GNNs), and can be implemented by referring to existing techniques. In the embodiments of this invention, the method for calculating the edge weights in the graph structure is as follows: for any pair of nodes p and q, the weight of the edge (p, q) can be expressed as... Where t p and t q These represent the collection times for nodes p and q, respectively, in seconds. Finally, the weight matrix W for each graph is obtained.

[0060] S6. For each license plate number, obtain the graph containing the node of the license plate number in each time slice; for each obtained graph, use a graph attention network to capture the spatial features of the nodes and obtain the new feature vector of the license plate number node in each time slice.

[0061] Specifically, such as Figure 3 As shown, the process is as follows:

[0062] S611. Each node i in the graph has an initial feature vector h′.i These eigenvectors are transformed using a learnable weight matrix M:

[0063] h″ i =Mh′ i °

[0064] S612. For each node j connected to node i, calculate the attention coefficient e for the weight of the fused edge. ij :

[0065] e ij =LeakyReLU(a T [h″ i ||h″ j ||w ij ])

[0066] Among them, a T is a learnable weight vector, || denotes vector concatenation, LeakyReLU is an activation function with a negative slope, w ij Let be the weight of edge (i, j).

[0067] S613. Use the Softmax function to adjust the attention coefficient e. ij Normalization is performed to ensure that the sum of the attention coefficients among all neighbors of node i is 1:

[0068]

[0069] Where N(i) represents the set of neighboring nodes of node i.

[0070] S614. Use normalized attention coefficients to perform a weighted summation of the features of neighboring nodes to obtain a new feature vector x for node i. i :

[0071]

[0072] Where σ is the activation function ReLU.

[0073] S615. Calculate the vector representation of the license plate number node in each graph, thereby obtaining the new vector representation x1, x2, ..., x of the license plate number in each time slice. T , where T is the total number of time slices.

[0074] It should be noted that Graph Attention Network (GAT) is also an existing technology. Its function is to use an attention mechanism to calculate the relevance weights between each node and its neighboring nodes. During training, these attention weights are dynamically calculated, and then a new feature representation for each node is generated by aggregating the features of neighboring nodes and weighting them according to their importance. This effectively captures the complex relationship between license plate numbers and IMSI codes. In the embodiments of this invention, the edge weights are integrated into the calculation of the GAT attention coefficients, enabling the model to better capture important information in the graph.

[0075] S7. The feature vector corresponding to the license plate number in each time slice is used as the time series input and passed to the temporal convolutional long short-term memory network to capture the temporal features of the nodes and obtain the hidden state of the license plate number at each time step.

[0076] It should be noted that the Temporal Convolutional Network-Long Short-Term Memory Network (TCN-LSTM) is composed of stacked Temporal Convolutional Layers (TCN) and Long Short-Term Memory Layers (LSTM). The input to the Temporal Convolutional Layer is a time series composed of the feature vectors corresponding to the license plate number in each time slice. Within the layer, a one-dimensional convolutional layer performs convolution operations on the time series data to extract local features from the time series, and then outputs new feature vectors of the license plate number at each time step, which are then passed to the next layer. The input to the Long Short-Term Memory Layer is the output of the Temporal Convolutional Layer. For each time step, the input vector is processed through the input gate, forget gate, candidate memory cells, and output gate to obtain the hidden state for that time step. Finally, the hidden states at each time step are used as the output of the Temporal Convolutional Network-Long Short-Term Memory Network. The specific process is as follows:

[0077] S711, extract the feature vector x of the license plate number at each time step. t Combined in chronological order, they form a time series data X = {x1, x2, ..., x...} T}, where T is the total number of time slices.

[0078] S712. Input the time series data X into the TCN layer, and after causal convolution and dilated convolution, obtain the new feature vector d for each time step. t :

[0079]

[0080] Where W and b are the convolution kernel and bias term, respectively, K is the kernel size, g is the dilation factor, and f is the ReLU activation function.

[0081] S713. The output sequence D = {d1, d2, ..., dn} of the TCN layer is... T As input to the LSTM layer, input gates, forget gates, and output gates are constructed to process the input:

[0082] i t =σ(W i ·[h t-1 d t ]+b i )

[0083] f t =σ(W f ·[h t-1 d t ]+b f )

[0084] o t =σ(W o ·[h t-1 d t ]+b o )

[0085]

[0086]

[0087] h t =o t *tanh(C t )

[0088] Where h t It is the hidden state at the current time step, i t f t o t These are the activation values ​​of the input gate, forget gate, and output gate, respectively. It is a candidate memory cell, C t It is the state of memory cells, W i W f W o W C These are the weight matrices for the input gate, forget gate, output gate, and candidate memory cells, respectively.

[0089] S714. The model's final output corresponds to the hidden states H = {h1, h2, ..., h...} at each time step. T} serves as the feature vector of the license plate number at each time step.

[0090] S8. Use average pooling to aggregate the hidden states at each time step to obtain the final vector representation v of each license plate number. c :

[0091]

[0092] Where h t [c] is the hidden state vector at time step t; finally, the vector representation of all license plate numbers is obtained.

[0093] S9. Repeat steps S6-S8. For each IMSI code, obtain the graph containing the nodes of that IMSI code in each time slice. Use a graph attention network to extract the spatial features of the nodes to obtain the new feature representation of the IMSI code node in each time slice. Then, use the new feature vectors of each time slice as time series input to a temporal convolutional long short-term memory network to extract the temporal features of the nodes to obtain the feature vectors of the IMSI code at each time step. Then, use the average pooling method to aggregate the feature vectors output at each time step to obtain the final vector representation of each IMSI code.

[0094] S10. Calculate the cosine similarity of the final vector representation of all license plate numbers and IMSI codes; for any license plate number or IMSI code, select the IMSI code or license plate number with the highest cosine similarity as its association result.

[0095] It should be noted that the cosine similarity calculation method described above applies to any license plate number c. x and IMSI code p y Corresponding vector representation and calculate and Cosine similarity between

[0096]

[0097] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments, including components, without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.

Claims

1. A method for analyzing the relationship between people and vehicles based on spatiotemporal feature fusion, characterized in that, Includes the following steps: Step 1: Collect vehicle license plate data and mobile phone IMSI code data from all checkpoints in the target road network, and preprocess them. Step 2: Divide the license plate numbers and IMSI codes in the preprocessed data into sample sets according to checkpoint number and collection time; Step 3: Treat each sample in the sample set as a document, and train it using a word embedding model to obtain the initial word embedding for each license plate number and IMSI code. Step 4: For each sample, construct a graph based on the license plate number, IMSI code, and the initial word embeddings of the license plate number and IMSI code, and use a graph attention network to obtain the new feature vector of the license plate number in each time slice of the node in the graph for each license plate number. The graph construction method is as follows: For each sample, each license plate number and IMSI code in the sample is taken as a node, and the word embeddings of the license plate number and IMSI code are taken as the initial feature vector of the corresponding node; an edge is established between every two nodes, and the weight of the edge is calculated based on the time difference of data collection between the two nodes. For each license plate number, obtain the graph containing the node of the license plate number in each time slice; for each obtained graph, use a graph attention network to capture the spatial features of the nodes and obtain the new feature vector of the license plate number node in each time slice. Methods for obtaining new feature vectors for license plate number nodes in each time slice: Each node i in the graph has an initial feature vector h′. i These eigenvectors are transformed using a learnable weight matrix M; "h" i =Mh′ i For each node j connected to node i, the attention coefficient e is calculated based on the weight of the fused edge. ij : e ij =LeakyReLU(a T [h″ i ||h″ j ||in ij ]) Among them, a T is a learnable weight vector, || denotes vector concatenation, LeakyReLU is an activation function with a negative slope, w ij Let (i, j) be the weight of edge (i, j). Use the Softmax function to evaluate the attention coefficient e. ij Normalization is performed to ensure that the sum of the attention coefficients of all neighbors of node i is 1; We use normalized attention coefficients to perform a weighted summation of the features of neighboring nodes to obtain a new feature vector x for node i. i ; Repeat the above steps to obtain new feature vectors for the license plate number in each time slice; Step 5: Use the feature vectors of each time slice of the license plate number as time series input and pass them to the temporal convolutional long short-term memory network to capture the temporal features of the nodes and obtain the hidden state of the license plate number at each time step. The Temporal Convolutional Long Short-Term Memory Network is composed of stacked temporal convolutional layers and long short-term memory layers. The input of the temporal convolutional layer is a time series composed of new feature vectors of the license plate number in each time slice. The layer uses dilated convolutional layers to perform convolution operations on the time series data to extract local features in the time series, and then outputs the vector representation of the license plate number at each time step and passes it to the next layer. The input of the long short-term memory layer is the output of the temporal convolutional layer. For each time step, the input vector is processed by the input gate, forget gate, candidate memory cells and output gate to obtain the hidden state of that time step. Finally, the hidden states of each time step are used as the output of the Temporal Convolutional Long Short-Term Memory Network. Step 6: Use the average pooling method to aggregate the hidden states at each time step to obtain the final vector representation of each license plate number; Step 7: Obtain the final vector representation of each IMSI code according to steps 4-6; Step 8: Calculate the cosine similarity of the final vector representations of all license plate numbers and IMSI codes; for any license plate number or IMSI code, select the IMSI code or license plate number with the highest cosine similarity as its association result.

2. The method for analyzing the relationship between people and vehicles based on spatiotemporal feature fusion according to claim 1, characterized in that, The license plate number-related data includes the checkpoint number, license plate number, and collection time; the IMSI code-related data includes the checkpoint number, IMSI code, and collection time.

3. The method for analyzing the relationship between people and vehicles based on spatiotemporal feature fusion according to claim 1, characterized in that, The preprocessing method includes filtering out missing data, redundant data, outlier data, and unrelated data.

4. The method for analyzing the relationship between people and vehicles based on spatiotemporal feature fusion according to claim 1, characterized in that, In step 2, the sample set is divided as follows: license plate numbers and IMSI codes collected from the same checkpoint within the same time period are grouped into a sample, and all samples form a sample set.

5. The method for analyzing the relationship between people and vehicles based on spatiotemporal feature fusion according to claim 4, characterized in that, In step 2, when dividing the samples, the calculation starts from 00:00:00, and each minute is a time period.

6. The method for analyzing the relationship between people and vehicles based on spatiotemporal feature fusion according to claim 1, characterized in that, The weight calculation method is as follows: For any pair of nodes p and q, the weight of the edge (p, q) is expressed as: Where t p and t q These represent the collection times for nodes p and q, respectively, in seconds. Finally, the weight matrix W for each graph is obtained.

7. The method for analyzing the relationship between people and vehicles based on spatiotemporal feature fusion according to claim 1, characterized in that, In step 7, when obtaining the new feature representation of the IMSI code node in each time slice, for each IMSI code, a graph containing the IMSI code node in each time slice is obtained.

Images