A multi-source and multi-element fusion method based on traffic calculation network

Abstract

The application discloses a kind of multi-source multi-element fusion methods based on traffic algorithm network, including the following steps: S1, collect traffic scene data, including unstructured data and structured data;S2, for unstructured data, using non-structured data fusion perception module carries out feature extraction;While, for structured data using the encoding and decoding mode based on Peep-couple LSTM carries out feature extraction;By adding attention mechanism at the decoder end, fusion structured data and unstructured data, generate the required specific traffic scene fusion features;S3, traffic scene fusion features are sent into traffic algorithm network platform, and strategy planning and system scheduling are carried out.The application can realize that traffic algorithm network generates appropriate decision, optimizes traffic flow, improves road network utilization, reduces traffic congestion and idling, and improves traffic safety.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation, and in particular to a multi-source, multi-element fusion method based on a traffic computing network. Background Technology

[0002] With the rapid development of modern transportation systems and the advancement of digital technology, Transportation Cyber-Physical Systems (TPS) have become an important development direction in the field of intelligent transportation. TPS is a comprehensive application model that utilizes advanced technologies such as computer science, communication technology, and intelligent control to comprehensively and intelligently transform transportation systems. The continuous expansion of population and urban scale has led to an exponential increase in transportation demand, thus creating an urgent need for more efficient, sustainable, and intelligent transportation solutions to meet travel needs. Intelligent transportation employs various scientifically effective dynamic traffic management methods, such as dynamic traffic signal control and congestion-hour pricing, by acquiring intelligent transportation data in real time and flexibly adjusting traffic management measures. Meanwhile, the decision-making process of traffic travel behavior (including travel destination, mode of transport, travel route, and departure time) changes in real time, resulting in a dynamic evolution of road traffic flow. Existing technologies mostly use a single data source and low-dimensional information when building transportation operation systems. This approach introduces noise and errors into the data, reducing its integrity and accuracy. A single data source is susceptible to equipment failures, communication interruptions, and other problems, and the system's fault tolerance and robustness are challenged in extreme physical environments. Summary of the Invention

[0003] Purpose of the invention: The purpose of this invention is to provide a multi-source, multi-element fusion method based on traffic computing networks. By fusing modal data from different sources, this method reduces data noise and errors, improves data reliability, integrates information from different dimensions, provides more comprehensive data support for traffic management and decision-making, optimizes traffic flow, improves road network utilization, reduces traffic congestion and emissions, and improves traffic safety.

[0004] Technical solution: The present invention is a multi-source, multi-element fusion method based on a traffic computing network, comprising the following steps:

[0005] S1 collects traffic scene data, including unstructured and structured data;

[0006] S2. For unstructured data, an unstructured data fusion perception module is used for feature extraction; at the same time, for structured data, a Peep-coupled LSTM-based encoding and decoding method is used for feature extraction; by adding an attention mechanism at the decoder end, structured and unstructured data are fused to generate the required specific traffic scene fusion features.

[0007] S3 integrates traffic scene features into the traffic computing network platform for strategy planning and system scheduling.

[0008] Furthermore, in step S1, the unstructured data includes road network video surveillance, vehicle-mounted camera images, and vehicle-mounted radar point cloud maps, while the structured data includes traffic gates, consumption systems, and historical traffic flow data.

[0009] Furthermore, the unstructured data is integrated to obtain a dataset, and the dataset is labeled and the timestamps of the collected data are aligned.

[0010] For complex urban traffic scenarios, gantries, ramps, toll plazas, and congested lanes are used as the main nodes, and urban traffic is divided into different areas based on these main nodes.

[0011] Furthermore, nodes for different tasks are obtained by combining different data sets within the dataset, as detailed below:

[0012] In the license plate recognition task, a test dataset is constructed by combining images from vehicle-mounted cameras and road network surveillance videos; a ResNet deep residual network is used for feature extraction, ResNet18 is used as the license plate detection network, and Transform is used as the long license plate sequence decoding network; through the license plate recognition task, the location information of vehicles present at the current node is recorded;

[0013] In the 2D / 3D object detection task, a test dataset is constructed by combining vehicle radar point cloud maps, vehicle camera images, and road network monitoring videos; the 2D task uses the Yolov7 single-stage detection model, while the 3D task uses the PV-RCNN point cloud object detection model; the element composition of the current node is obtained by fusing the results of the detection task and the license plate recognition task.

[0014] In the instance segmentation and vehicle behavior prediction tasks, a test dataset is constructed by combining vehicle radar point cloud maps, vehicle camera images, and road network monitoring videos. In the segmentation task, the PointNet model is used. In the vehicle behavior prediction task, the LSTM time series prediction model is used to record the trajectory prediction of each driving vehicle within a node. By fusing the segmentation task, vehicle trajectory prediction, and the element composition of the node, the element interaction information of the current node is analyzed.

[0015] In the vehicle speed estimation task, a test dataset is constructed by combining road network surveillance video and vehicle camera images; optical flow is used to model video frames and EfficientNet is used to establish a feature extraction network. By detecting pixel motion, vehicle speed is estimated to obtain the real-time vehicle speed of traffic users passing through the node.

[0016] Furthermore, in step S2, a Peep-couple LSTM encoding and decoding network is built for each type of structured data, and attention fusion is performed at the decoder end to finally obtain the required specific traffic feature output;

[0017] The calculation of a single Peep-couple LSTM_Cell is as follows:

[0018]

[0019]

[0020]

[0021]

[0022] h t =o t ·tanh(c t )

[0023] In the formula, σ is the sigmoid activation function; "·" represents the dot product operation; For the input structured data; W * U * b * These are the learnable parameters; f t The output of the forget gate is used to control the cell state C at the previous time step (t-1). t-1 How much of the information in W should be retained in the current time step (t)? f The weight matrix representing the forget gate is used to combine the input vector with the hidden state vector h from the previous time step. t-1 Cellular state C t-1 and the input of the current time step Perform a weighted summation; C t-1 Represents the cell state at the previous time step (t-1); h t-1 This represents the hidden state at the previous time step (t-1); b f This is the bias term for the forget gate, used to adjust the output offset of the forget gate; t The output of the gate represents the cell state C at the current time step (t). t How much information should be output from W? o It is the weight matrix of the output gate, used to combine the input vector with the cell state at the current time step, the hidden state at the previous time step, and the input at the current time step. Perform a weighted summation; b o This is the output gate's bias term, used to adjust the output offset of the output gate; W c and U cIt is a weight matrix, used to weight the input at the current time step. The hidden state h of the previous time step t-1 Perform a weighted summation; b c It is a bias term used to adjust the output offset of the cell state;

[0024] The attention mechanism takes a key K, a value V, and a query Q as inputs. For each k... m ∈K and q m Perform 1D convolution on ∈Q, and then obtain S after convolution. m The data is fed into the softmax function to obtain each set of k. m and q m Attention score α m Each V m ∈V multiplied by its corresponding value α m Finally, the information output after fusion through the attention mechanism is obtained, and the calculation formula is as follows:

[0025] f′(K,Q)=Conv1D[Conv1D(K)+Conv1D(Q)]

[0026]

[0027]

[0028] Where Conv1D is a one-dimensional convolution with a kernel size of 1*1 and a stride of 1; f′(·) represents the similarity function extracted between sequences K and Q; k m q m and v m These represent the characteristics of the current time series; M indicates that the time series has M time points; S m Represents the current time k m q m Similarity score; α m This refers to the attention score at the current moment, representing the attention score for each v. m Its importance in predicting specific traffic temporal characteristics;

[0029] The formula for dimensional fusion of the attention mechanism at the encoder is as follows:

[0030]

[0031]

[0032]

[0033]

[0034]

[0035]

[0036]

[0037] in, and These represent the input features at the decoder, and the time series features of three different modalities, respectively; f′(·) represents the extraction of a similarity function between a pair of sequences; m represents the current time; M represents the total number of time points in the time series; S1 m S2 m S3 m These represent the similarity scores of any pair at the current time; α m β m γ m These refer to the attention score at the current moment, and also represent the importance values ​​of different dimensions; λ1, λ2, and λ3 are the defined feature fusion operators.

[0038] Furthermore, both the encoder and decoder consist of multiple stacked LSTM units, each of which accepts a single element of the input and collects information. Simultaneously, at each encoder input, an attention mechanism is used to fuse the features from the previous time step into the current time step's data features as the current input. The specific implementation steps are as follows:

[0039] In S021, during the encoder stage, the attention mechanism calculates the scores between the dimensions of the original input data. At a historical time step of t, the scores are... Each data dimension in the process is considered as K and V in the attention mechanism, and the hidden state computed by the previous LSTM unit is used. Treat it as Q, then output the weighted value. As input to the Cell cell at the next time step;

[0040] S2022 integrates three different source features in the decoder stage:

[0041] c1) The feature map obtained by the unstructured data feature extraction module contains local and global spatial features as well as future traffic conditions;

[0042] c2) Cell state of the last LSTM_Cell in the encoder stage n and hidden state h n It is a time memory resulting from the fusion of structured features from multiple sources in a specific traffic scenario;

[0043] c3) Structured data extracted from unstructured features Traffic element information features are considered as another type of time-resolution feature;

[0044] Attention mechanisms calculate the hidden state h′ at the next t′ time steps. t′ Each hidden state h with the encoder stage history time step t The correlation between them; all historical hidden states are considered as K and V,h′ t′ Q represents the attention mechanism; the input of LSTM_Cell at time t is the fused features after the attention mechanism. as well as The output of the concatenation, where The prediction is generated at the previous time step t-1; finally, the multi-source, multi-element spatiotemporal fusion feature is output by the decoder.

[0045] Compared with the prior art, the significant advantages of this invention are as follows:

[0046] 1. This invention integrates and analyzes unstructured traffic data, and introduces multiple existing methods for feature-level information transmission and fusion. Compared with a single pattern recognition model, it takes into account specific spatial interaction information and semantic information in traffic application scenarios, enabling the traffic computing network to generate more appropriate decisions, thereby optimizing traffic flow, improving road network utilization, reducing traffic congestion and emissions, and improving traffic safety.

[0047] 2. In the structured traffic data prediction model, the standard LSTM model is improved and a Peep-couple LSTM module is proposed. Compared with the ordinary LSTM_cell, it has the advantages of better time feature preservation and lower parameter number. The traffic computing network can more accurately predict traffic conditions, including changes in traffic flow, trends in road use, and possible congestion points. It also helps to identify potential traffic accidents and dangerous areas.

[0048] 3. In the prediction model of structured traffic data, this invention designs a multi-layer attention mechanism; different fusion schemes are designed in the encoding and decoding parts of the model respectively; through the attention mechanism, the model can capture key information from multiple continuous data, while enhancing the spatial consistency and information correlation of structured and unstructured data. The traffic computing network can monitor traffic status in real time, detect traffic accidents and abnormal situations in a timely manner, and respond quickly, thereby improving the safety and stability of the traffic system.

[0049] 4. In the structured traffic data prediction model, feature maps and auxiliary NS (none-structure) data proposed from unstructured traffic data are introduced and simultaneously fed into the decoder. This design scheme aligns the features between different modalities, learns the target spatial topological relationship in the image data, and enables the data prediction model to better represent future traffic time series information, thereby improving the prediction accuracy and stability of the model. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of the sensing system architecture of the present invention;

[0051] Figure 2 A schematic diagram of the architecture of the unstructured data fusion and sensing module for traffic computing network nodes;

[0052] Figure 3 Architecture diagram of the module for generating multi-source feature fusion perception and time-series feature prediction for traffic computing network nodes;

[0053] Figure 4 This is a schematic diagram of the internal implementation of a Peep-coupled LSTM_cell.

[0054] Figure 5 This is a schematic diagram of the attention mechanism in an encoder.

[0055] Figure 6 This is a schematic diagram of the attention mechanism in the decoder. Detailed Implementation

[0056] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0057] In this embodiment, the traffic computing network integrates information from multiple data sources, such as sensor data, GPS positioning data, traffic camera data, and social media data, to form comprehensive and multi-faceted traffic data. These data sources are extensive, covering all aspects of the traffic system, including traffic flow, vehicle location, road conditions, weather conditions, and event information, thereby enabling real-time monitoring and analysis of the entire traffic system. By integrating and analyzing information from different data sources, and employing technologies such as data mining, machine learning, and artificial intelligence, valuable traffic information and patterns are extracted. Through comprehensive analysis of multi-source data, the traffic computing network can accurately predict traffic congestion, identify traffic accidents, and analyze traffic travel patterns, providing a scientific basis for traffic management and decision-making. The data fusion method of this invention enables the traffic computing network to generate more appropriate decisions to optimize traffic flow, improve road network utilization, reduce congestion and emissions, improve traffic safety, and provide more intelligent traffic services, thereby achieving sustainable development of the urban transportation system and enhancing the travel experience.

[0058] This embodiment takes the rapid construction of intelligent command and dispatch for smart transportation as an application scenario and designs a fusion method suitable for multi-source and multi-element data. The process framework of this invention is as follows: Figure 1As shown, a large amount of traffic scenario data is collected, including unstructured data such as road network video surveillance, vehicle-mounted camera images / videos, and vehicle-mounted radar point cloud maps, as well as structured data such as traffic gates, consumption systems, and historical traffic flow data. For unstructured data, feature extraction modules suitable for different types of unstructured data are designed. Simultaneously, for structured data, a Peep-coupled LSTM-based encoding and decoding method is used for feature extraction. An attention mechanism is added at the decoder to fuse structured and unstructured data to generate the required traffic scenario-specific fused features. Finally, the fused features are sent to the traffic computing network platform for further strategy planning and system scheduling. Detailed implementation steps are as follows:

[0059] Step 1, Figure 2 This invention demonstrates the architecture of the unstructured data fusion perception module proposed in this paper. It integrates existing camera and highway equipment, as well as multi-element perception data provided by facilities, including road network monitoring video, vehicle-mounted equipment reading information, CPC card information, toll system transaction records, and traffic flow. Based on numerous intelligent detection algorithms with different functions, it performs structured identification and analysis of vehicles passing through gantries, ramps, toll plazas, and lanes, obtaining detailed vehicle information, node element composition information, and node element interaction information in complex traffic scenarios. Finally, it performs unified feature extraction on the unstructured data.

[0060] Step 101, specifically, firstly, urban traffic data is collected. Video and image data are acquired through a road network monitoring system built using installed cameras and existing surveillance cameras. Simultaneously, real-time data from vehicle-mounted radar and cameras is collected. This invention integrates the collected video data, image data, and specific vehicle information. The collected video dataset is annotated frame by frame, particularly highlighting key information such as license plate numbers, vehicle speeds, and lanes in vehicle images. When collecting vehicle-mounted camera image datasets, the current traffic location and time need to be additionally annotated. The timestamps of all datasets are aligned for subsequent analysis and modeling.

[0061] Step 102: For complex urban traffic scenarios, gantries, ramps, toll plazas, and congested lanes are used as the main nodes for analysis. Urban traffic is divided into different areas based on these main nodes. Specifically, the datasets collected, cleaned, and organized in step 101 include: vehicle-mounted radar point cloud datasets, vehicle-mounted camera image datasets, road network monitoring video datasets, etc. This invention uses combinations of different datasets from these sources to obtain node features for different tasks.

[0062] Step 1021: In the license plate recognition task, a test dataset is constructed by combining the vehicle-mounted camera image dataset and the road network monitoring video dataset. A ResNet deep residual network is used for feature extraction, with ResNet18 as the license plate detection network and Transform as the long license plate sequence decoding network. The vehicle location information of the current node is recorded through the license plate recognition task.

[0063] Step 1022: In the 2D / 3D object detection task, a test dataset is constructed by combining the vehicle-mounted radar point cloud map dataset, the vehicle-mounted camera image dataset, and the road network monitoring video dataset. The 2D task uses the Yolov7 single-stage detection model, while the 3D task uses the PV-RCNN point cloud object detection model. By fusing the results of the detection task and the license plate recognition task, the element composition of the current node is obtained.

[0064] Step 1023: To obtain the element interaction information of the current node, a test dataset is constructed by combining the vehicle radar point cloud dataset, the vehicle camera image dataset, and the road network monitoring video dataset for instance segmentation and vehicle behavior prediction tasks. In the segmentation task, the PointNet model from 3D segmentation is used to segment not only the target but also the background lanes. In the vehicle behavior prediction task, an LSTM time-series prediction model is used to record the trajectory prediction of each vehicle within the node. By fusing the segmentation task, vehicle trajectory prediction, and the element composition of the node, the element interaction information of the current node is analyzed.

[0065] Step 1024: In the vehicle speed estimation task, a test dataset is constructed by combining the road network monitoring video dataset and the vehicle camera image dataset. The FlowNet optical flow model is used to model video frames, and the highly adaptive EfficientNet is used to build a feature extraction network. This method processes consecutive frames in the video stream sequence and estimates vehicle speed by detecting pixel motion. Through this task, the real-time vehicle speed of traffic users passing through the node is obtained.

[0066] Step 2, in such Figure 3 The multi-source feature fusion perception and time-series feature prediction generation module of the computing network node shown has recorded structured historical data such as the number of vehicles passing through the current node, passage time, queue length of vehicle type, regional traffic flow, cross-sectional traffic flow, and traffic flow by type. For each type of structured data, an LSTM encoding / decoding network is built. At the decoder end, attention fusion is performed based on the unstructured data fusion features obtained in step 1, including real-time vehicle location, node element composition information, node element interaction information, and real-time vehicle speed, to finally obtain the required specific traffic feature output. The specific implementation is shown below:

[0067] Step 201 first briefly introduces the internal principles and advantages of the improved Peep-couple LSTM framework and the attention mechanism under different design conditions, and then explains the details of time series modeling using the encoder-decoder architecture.

[0068] a) Peep-coupled LSTM: LSTM, as a variant of RNN, is commonly used for time series prediction. Using forget gates, input gates, and output gates in LSTM helps overcome the gradient vanishing problem, enabling LSTM to handle longer sequence data that traditional RNNs cannot handle. The internal structure of a single Peep-coupled LSTM_Cell designed in this invention is as follows... Figure 4 As shown, compared to the original LSTM model, it only contains two gates. Furthermore, it borrows a peephole connection to introduce cell state information for both the forget gate and the output gate. First, c t-1 Upon entering the forget gate, a portion of the memory state is lost. New memory states are then added using a coupled forgetting mechanism. Unlike the original LSTM model, which separately determines the forget gate and input gate information, this model makes the decision together. A hidden state h is obtained through the output gate. t Finally, the output c of Cell. t and hidden h t The state is sent to the next Peep-couple LSTM_Cell. The computation of a single Cell is as follows:

[0069]

[0070]

[0071]

[0072]

[0073] h t =o t ·tanh(c t (5)

[0074] In the formula, σ is the sigmoid activation function; "·" represents the dot product operation; For the input structured data; f t The output of the forget gate is a value between 0 and 1, used to control the cell state C at the previous time step (t-1). t-1 How much of the information in W should be retained in the current time step (t)? fThe weight matrix representing the forget gate is used to combine the input vector with the hidden state vector h from the previous time step. t-1 Cellular state C t-1 and the input of the current time step Perform a weighted summation. This weight matrix determines the degree of influence of each input feature on the forget gate; C t-1 This represents the cell state at the previous time step (t-1). The cell state is part of the LSTM network and is used to store previous information; h t-1 This represents the hidden state at the previous time step (t-1). The hidden state is part of the LSTM architecture, used to capture previous information and pass it to the next time step; b f This is the bias term of the forget gate, used to adjust the output offset of the forget gate to adapt the model to the data fit. t The output of the output gate is also a value between 0 and 1, used to control the cell state C at the current time step (t). t How much information should be output from W? o It is the weight matrix of the output gate, used to combine the input vector with the cell state at the current time step, the hidden state at the previous time step, and the input at the current time step. A weighted summation is performed; this weight matrix determines the degree of influence of each input feature on the output gate. o This is the output gate's bias term, used to adjust the output gate's offset to adapt the model's fit to the data. c and U c It is a weight matrix, used to weight the input at the current time step. The hidden state h of the previous time step t-1 Perform a weighted summation. c This is a bias term used to adjust the output offset of the cell state. W * U * b * (* represents general terms) are all learnable parameters. A significant difference compared to the standard LSTM model used for general time series sequencing is that this invention simplifies the traditional input gate and reduces computation by merging forget gates. Secondly, a Peep-hole structure is employed, introducing c during the iteration process. t-1 Information: LSTM contains two types of hidden layers: hiddenstate(h) t ) and cell state(c t ), h t The tendency is to retain more recent information, c tThis records a lot of feature information from earlier times. For the actual traffic application scenario selected in this embodiment, introducing information with a larger time span helps the model convergence and optimization.

[0075] To better map the complex relationships between input and output, multiple LSTM cells are typically stacked and concatenated to form a complex network. This example uses stacked LSTMs to obtain historical temporal features of the input features, improving the robustness of multi-source, multi-factor fusion features.

[0076] b) Attention Mechanism: The core goal of the attention mechanism is to automatically adjust and focus on the correlations between data elements. This mechanism is inspired by how humans focus their visual attention. For example, when observing an object, we don't always examine the entire scene in detail, but rather concentrate on a specific area as needed. Compared to CNNs and RNNs, the attention mechanism has a relatively simple model structure, fewer parameters, and supports parallel processing. More importantly, the attention mechanism can capture key information from multiple consecutive data sets while maintaining long-distance information correlations. In a traffic context, these data elements can be structured data, such as traffic checkpoints, payment systems, and historical traffic data, or unstructured data, such as road surveillance videos, vehicle camera videos, and point cloud maps from vehicle radar. To simulate these time-series characteristics, incorporating an attention mechanism during the encoding-decoding stage helps enhance prediction performance.

[0077] Details of attention mechanisms, such as Figure 6 As shown. The input to the attention mechanism is the key K, the value V, and the query Q. First, for each k... m ∈K and q m Perform 1D convolution (Conv1D) on ∈Q, and then obtain S after convolution. m The data is fed into the softmax function to obtain each set of k. m and q m Attention score α m α m This indicates the importance of this element's information for predicting specific traffic temporal characteristics. Each v m ∈V multiplied by its corresponding value α m By summing these final values, the resulting information output after the attention mechanism is applied is obtained. It's important to note that the attention mechanism used here is applied at the decoder and can be expressed by the following formula:

[0078] f′(K,Q)=Conv1D[Conv1D(K)+Conv1D(Q)] (6)

[0079]

[0080]

[0081] Where Conv1D is a one-dimensional convolution with a kernel size of 1*1 and a stride of 1. K, Q, and V represent the input features at the decoder, and the time-series features of three different modalities, respectively; f′(·) represents the extraction of the similarity function between K and Q sequences; k m q m and v m These represent the characteristics of the current time series. M indicates that the time series has M time points; S m Represents the current time k m q m Similarity score; α m This refers to the attention score at the current moment, representing the attention score for each v. m Its importance in predicting specific traffic time-series characteristics.

[0082] Attention mechanism at encoder, dimension fusion method, such as Figure 5 As shown, it can be expressed by the following formula:

[0083]

[0084]

[0085]

[0086]

[0087]

[0088]

[0089]

[0090] Wherein, Conv1D is a one-dimensional convolution with a kernel size of 1*1 and a stride of 1; and These represent the input features at the decoder, and the time series features of three different modalities; f′(·) represents the extraction of a similarity function between a pair of sequences; m represents the current time; M represents the total number of time points in the time series; S1 m S2 m S3 m α represents the similarity score of any pair at the current moment. m β m γ mThis refers to the attention score at the current moment, and also represents the importance values ​​of different dimensions; λ1, λ2, and λ3 are the set feature fusion operators, which indicate that in a specific traffic prediction task, the main feature will be given a larger weight to make the result more reliable. Generally, the values ​​here are 0.6, 0.2, and 0.2.

[0091] Step 202, combining Figure 3 This section describes the temporal sequence modeling based on an encoder-decoder network and the fusion steps for structured and unstructured data features. Both the encoder and decoder stages consist of stacked multi-layer LSTM units, where each unit accepts a single element of the input and collects information. Simultaneously, at each encoder input, an attention mechanism is used to fuse the features from the previous time step into the current time step's data features as the current input. The fusion mechanism designed in the encoder stage facilitates the extraction of temporal correlations between structured historical traffic data. For example, the cross-sectional flow at the current traffic node and the consumption flow at the node's gate are highly correlated, making this fusion mechanism reasonable and further improving the accuracy of traffic-specific prediction tasks. Here The time-series features represent the highest importance for a specific prediction task, such as vehicle speed features in a road congestion assessment task; typen1,…,typen represent the types of multi-source data in the currently selected traffic scenario. The decoder stage performs feature fusion from different sources and future time modeling. Decoder input includes feature maps and fused features output from the unstructured data feature extraction module, encoder-level output, and the aforementioned f time-step NS (None-Structure) numerical data of different types from the current prediction task. Figure 3 shown This is the output of the decoder for a specific traffic fusion feature.

[0092] Step 2021, in the encoder stage, the attention mechanism calculates scores between dimensions of the original input data (e.g., different types of traffic scenario data such as current node vehicle travel time, regional traffic flow, and classified cross-sectional traffic flow). Figure 5 As shown, when the historical time step is t, Each data dimension in the process is considered as K and V in the attention mechanism, and the hidden state computed by the previous LSTM unit is used. Treat it as Q, then output the weighted value. This serves as the input to the Cell unit at the next time step. Since the parameters of the convolutional kernel in the attention mechanism are learned iteratively, Compare It is more convincing.

[0093] Step 2022, in the decoder stage, such as Figure 3 As shown, the source features are integrated at three different levels. First, there's the feature map obtained from the unstructured data feature extraction module, which includes local and global spatial features as well as future traffic conditions. Second, there's the cell state c of the last LSTM_Cell in the encoder stage. n and hidden state h n These are temporal memories resulting from the fusion of structured features from multiple sources within a specific traffic scenario; finally, there is structured data extracted from the fusion of unstructured features. This feature can be viewed as another type of traffic element information feature with a different temporal resolution. In the decoder stage, the attention mechanism is the same as that in the encoder stage. Here, the attention mechanism calculates the hidden state h′ at the next t′ time steps. t′ Each hidden state h with the encoder stage history time step t The correlation between them. All historical hidden states are considered as K and V,h′. t′ Let Q represent the attention mechanism. The attention score, calculated through the attention mechanism, allows for selective feature extraction from datasets across different time periods. The input to the LSTM_Cell at time t is the fused features resulting from the attention mechanism. as well as The output of the concatenation, where It is generated based on the prediction at the previous time step t-1. Finally, the multi-source, multi-element spatiotemporal fusion feature is output by the decoder.

[0094] In summary, this invention does not rely on a specific single data source, which can improve the fault tolerance and stability of the system. Even if a certain data source has a problem, it can still provide effective information, which can further help decision-makers in traffic management and planning, and provide more accurate real-time traffic information and future traffic trend predictions.

Claims

1. A multi-source, multi-element fusion method based on a traffic computing network, characterized in that, The steps include the following: S1 collects traffic scene data, including unstructured and structured data; S2. For unstructured data, an unstructured data fusion perception module is used for feature extraction; at the same time, for structured data, a Peep-coupled LSTM-based encoding and decoding method is used for feature extraction; by adding an attention mechanism at the decoder end, structured and unstructured data are fused to generate the required specific traffic scene fusion features. S3 inputs the integrated features of traffic scenarios into the traffic computing network platform for strategy planning and system scheduling; In step S2, a Peep-couple LSTM encoding and decoding network is built for each type of structured data, and attention fusion is performed at the decoder to finally obtain the required specific traffic feature output. The calculation of a single Peep-couple LSTM_Cell is as follows: ， ， ， ， ， In the formula, Use the sigmoid activation function; " indicates the dot product operation; The input is structured data; , , These are the learnable parameters; The output of the forget gate is used to control the previous time step. ) cell state The information in should be at the current time step How much should be retained in the original text? The weight matrix representing the forget gate is used to combine the input vector with the hidden state vector from the previous time step. Cell state and the input of the current time step Perform a weighted summation; Represents the previous time step The cellular state; Indicates the previous time step The hidden state of ); It is the bias term of the forget gate, used to adjust the output offset of the forget gate; The output of the gate represents the current time step. ) cell state How much of the information in the code should be output? It is the weight matrix of the output gate, used to combine the input vector with the cell state at the current time step, the hidden state at the previous time step, and the input at the current time step. Perform a weighted summation; It is the output gate's bias term, used to adjust the output offset of the output gate; and It is a weight matrix, used to weight the input at the current time step. The hidden state of the previous time step Perform a weighted summation; It is a bias term used to adjust the output offset of the cell state; The input to the attention mechanism is a key. ,value and query For each and Perform 1D convolution processing, and then obtain the result after convolution. The data is fed into the softmax function to obtain each group. and Attention score each Multiply by its value accordingly Finally, the information output after fusion through the attention mechanism is obtained, and the calculation formula is as follows: ， ， ， in, This is a one-dimensional convolution with a kernel size of 1*1 and a stride of 1. Indicate extraction and Sequence similarity function; , and These represent the features of the current time series; Indicates the total number of time series At that moment; Represents the current moment , Similarity score; This refers to the attention score at the current moment, representing the attention score for each individual. Its importance in predicting specific traffic temporal characteristics; The formula for dimensional fusion of the attention mechanism at the encoder is as follows: ， ， ， ， ， ， ， in, , and These represent the input features at the decoder end, and the time series features of three different modalities, respectively. Indicate extraction Sequence similarity function; Represents the current moment; Indicates the total number of time series At that moment; These represent the similarity scores of any pair at the current time. , These refer to the attention score at the current moment, and also represent the importance values ​​of different dimensions; , , These are the defined feature fusion operators.

2. The multi-source, multi-element fusion method based on traffic computing network according to claim 1, characterized in that, In step S1, the unstructured data includes road network video surveillance, vehicle camera images, and vehicle radar point cloud maps, while the structured data includes traffic gates, consumption systems, and historical traffic flow data.

3. The multi-source, multi-element fusion method based on traffic computing network according to claim 2, characterized in that, Unstructured data is integrated to obtain a dataset, which is then labeled and the timestamps of the collected data are aligned. For complex urban traffic scenarios, gantries, ramps, toll plazas, and congested lanes are used as the main nodes, and urban traffic is divided into different areas based on these main nodes.

4. The multi-source, multi-element fusion method based on traffic computing network according to claim 3, characterized in that, Nodes for different tasks are obtained by combining different data from the dataset, as detailed below: In the license plate recognition task, a test dataset is constructed by combining images from vehicle-mounted cameras and road network surveillance videos; a ResNet deep residual network is used for feature extraction, ResNet18 is used as the license plate detection network, and transform is used as the decoding network for long license plate sequences. The license plate recognition task records the location information of vehicles present at the current node; In the 2D / 3D target detection task, a test dataset is constructed by combining vehicle-mounted radar point cloud maps, vehicle-mounted camera images, and road network monitoring videos. The 2D task uses a Yolov7 single-stage detection model, while the 3D task uses a PV-RCNN point cloud object detection model. The element composition of the current node is obtained by fusing the results of the detection task and the license plate recognition task. In the instance segmentation and vehicle behavior prediction tasks, a test dataset is constructed by combining vehicle radar point cloud maps, vehicle camera images, and road network monitoring videos. In the segmentation task, the PointNet model is used. In the vehicle behavior prediction task, the LSTM time series prediction model is used to record the trajectory prediction of each driving vehicle within a node. By integrating segmentation tasks, vehicle trajectory prediction, and node element composition, the interaction information of the current node's elements is analyzed. In the vehicle speed estimation task, a test dataset is constructed by combining road network surveillance video and vehicle camera images; optical flow is used to model video frames and EfficientNet is used to establish a feature extraction network. By detecting pixel motion, vehicle speed is estimated to obtain the real-time vehicle speed of traffic users passing through the node.

5. The multi-source, multi-element fusion method based on traffic computing network according to claim 1, characterized in that, Both the encoder and decoder consist of multiple stacked LSTM units, each of which accepts a single element of the input and collects information. Simultaneously, at each encoder input, an attention mechanism is used to fuse the features from the previous time step into the current time step's data features as the current input. The specific implementation steps are as follows: S021, in the encoder stage, the attention mechanism calculates the scores between the dimensions of the original input data, at a historical time step of [missing information]. At that time, Each data dimension in the attention mechanism is considered as a component of the attention mechanism. and The hidden state calculated by the previous LSTM unit Considered Then the weighted output As input to the Cell cell at the next time step; S2022 integrates three different source features in the decoder stage: c1) The feature map obtained by the unstructured data feature extraction module contains local and global spatial features as well as future traffic conditions; c2) Cell state of the last LSTM_Cell in the encoder stage and hidden state It is a time memory resulting from the fusion of structured features from multiple sources in a specific traffic scenario; c3) Structured data extracted from unstructured features Traffic element information features are considered as another type of time-resolution feature; Attention mechanism computing the future Hidden state at each time step Each hidden state with the encoder stage history time step The correlation between them; All historical hidden states are considered as and , Indicating attention mechanism ; LSTM_Cell in time The input is the fused features after the attention mechanism and as well as The output of the concatenation, where In the previous time step The prediction is generated; finally, the multi-source, multi-element spatiotemporal fusion features are output through the decoder. .

Images