Spatial-temporal asynchronous multi-source data fusion prediction method based on long short-term memory network

Abstract

The invention relates to the technical field of multi-source data fusion, and particularly provides a time-space asynchronous multi-source data fusion prediction method based on a long short-term memory network, so as to improve the monitoring and management capability of an expressway. According to the technical scheme, the method comprises the following steps: step 1, obtaining highway radar data and video data; 2, time attributes of the radar data and the video data are fused; 3, fusing the spatial attributes of the radar data and the video data; 4, constructing a traffic flow characteristic parameter prediction model based on multi-source data fusion; and step 5, obtaining traffic flow characteristic parameters of multi-source data fusion prediction.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-source data fusion, and in particular to a spatiotemporal asynchronous multi-source data fusion prediction method based on a long short-term memory network. Background Art

[0002] During the operation of highways, a large amount of data is generated, including traffic flow, speed, accident information, etc. These data are crucial for the monitoring and management of highways. With the acceleration of highway informatization construction, various traffic data sources have been widely used on highways, such as radar data and video data. Due to the different working principles and working methods of detectors, each data source has its limitations. It is difficult to fully reflect the real-time traffic operation status on highways based on a single data source.

[0003] How to fuse multi-source data at different time and space granularities and effectively utilize the complementarity between data to make it more effective is an important task for highway traffic control and management. Domestic and foreign scholars have begun to study and discuss this since the late 1990s. With the continuous development of data fusion technology, fusing multi-source data can improve the quality of traffic data, obtain richer and more comprehensive traffic operation status information, and monitor the traffic operation status of highways from different angles, which helps to improve the monitoring and management capabilities of highways. Summary of the invention

[0004] The purpose of the present invention is to overcome the deficiencies in the above-mentioned background technology and provide a spatiotemporal asynchronous multi-source data fusion prediction method based on long short-term memory network to improve the monitoring and management capabilities of highways.

[0005] The technical solution of the present invention is:

[0006] A spatiotemporal asynchronous multi-source data fusion prediction method based on long short-term memory network comprises the following steps:

[0007] Step 1: Obtain highway radar data and video data

[0008] Radar data and video data include basic information of radar equipment and video equipment and traffic flow characteristic parameters collected by radar equipment and video equipment;

[0009] Step 2: Fusion of temporal attributes of radar data and video data

[0010] Align the timestamps of radar data and video data, unify the reporting frequency of radar data and video data through time interpolation method, and realize the time attribute fusion of radar data and video data;

[0011] Step 3: Fusion of spatial attributes of radar data and video data

[0012] The physical topological relationship between the radar device and the video device is determined based on the detection position. The video device adjacent to the radar device participates in the prediction of the traffic flow characteristic parameters of the radar device, realizing the spatial attribute fusion of the radar data and the video data.

[0013] Step 4: Construct a traffic flow characteristic parameter prediction model based on multi-source data fusion

[0014] Using spatiotemporal fusion of radar data and video data, a traffic flow characteristic parameter prediction model is constructed based on long short-term memory network;

[0015] Step 5: Obtain traffic flow characteristic parameters predicted by multi-source data fusion

[0016] The spatiotemporally fused radar data and video data are input into the traffic flow characteristic parameter prediction model to obtain the predicted traffic flow characteristic parameters.

[0017] In step 1, the basic information includes the equipment number, installation location, detection location, lane number, statistical period, reporting frequency and reporting time, and the traffic flow characteristic parameters include flow rate and speed.

[0018] In step 1:

[0019] The statistical cycle is the length of the time period during which the radar equipment and the video equipment collect traffic flow characteristic parameters. The statistical cycle of the radar equipment is 1 minute; the statistical cycle of the video equipment is 2 minutes;

[0020] Traffic volume is the number of vehicles passing through the detection location during the statistical period, and the calculation formula is as follows:

[0021]

[0022] Where Q is the number of vehicles passing the detection location during the statistical period, q i is the number of vehicles passing the detection position in lane i during the statistical period, and n is the number of lanes;

[0023] The speed is the weighted average of the location speeds of all vehicles passing through the detection location during the statistical period. The calculation formula is as follows:

[0024]

[0025] Where V is the weighted average of the location speeds of all vehicles passing through the detection location during the statistical period, v i is the average speed of all vehicles passing through the detection position in lane i during the statistical period, q i is the number of vehicles passing the detection position in lane i during the statistical period, and n is the number of lanes.

[0026] In step 2, the timestamps of the radar data and the video data are aligned, and the video data is supplemented by a time interpolation method so that radar data and video data exist at each reporting time; the time interpolation method includes linear interpolation and nearest neighbor interpolation.

[0027] When supplementing video data of a certain reporting time, if video data exists at both the previous reporting time and the next reporting time, linear interpolation is used to obtain the video data of the reporting time; otherwise, nearest neighbor interpolation is used to obtain the video data of the reporting time.

[0028] In step 3, the video device adjacent to the radar device is a video device located between the upstream radar device and the downstream radar device and closest to the radar device.

[0029] In step 4, the time-space fused radar data and video data are radar data at the same reporting time, upstream radar data, downstream radar data, and video data adjacent to the radar device.

[0030] The traffic flow characteristic parameter prediction model includes an input layer, an LSTM layer, an LSTM layer, a random inactivation layer, and a fully connected layer connected in sequence; the traffic flow characteristic parameter prediction model includes a flow prediction model and a speed prediction model.

[0031] In step 5, the radar data and video data collected in real time are fused with time attributes and space attributes, and then input into the traffic flow characteristic parameter prediction model to obtain the predicted traffic flow characteristic parameters.

[0032] The beneficial effects of the present invention are:

[0033] Compared with the highway traffic flow characteristic parameter prediction method based on a single data source, the method provided by the present invention realizes the temporal attribute fusion of radar data and video data through a time interpolation method, and realizes the spatial attribute fusion of radar data and video data based on the physical topological relationship between devices. It can effectively fuse multi-source data at different time and space granularities, avoid the limitations of small detection range and large data noise that may exist based on a single data source, and at the same time, construct a traffic flow characteristic parameter prediction model based on a long short-term memory network, which can effectively improve the accuracy of traffic flow characteristic parameter prediction and obtain more accurate and comprehensive traffic information. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a schematic diagram of the present invention.

[0035] Figure 2 It is a schematic diagram of the traffic flow characteristic parameter prediction model of the present invention.

[0036] Figure 3 It is a schematic diagram of the layout of the radar equipment and the video equipment according to an embodiment of the present invention.

[0037] Figure 4 It is the flow prediction diagram of the present invention.

[0038] Figure 5 It is the speed prediction diagram of the present invention. DETAILED DESCRIPTION

[0039] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0040] like Figure 1 As shown, a spatiotemporal asynchronous multi-source data fusion prediction method based on long short-term memory network includes the following steps:

[0041] Step 1: Obtain highway radar data and video data

[0042] Radar data and video data include basic information of radar equipment and video equipment and traffic flow characteristic parameters collected by radar equipment and video equipment.

[0043] Traffic flow characteristic parameters include flow rate and speed.

[0044] Basic information includes equipment number, installation location, detection location, lane number, statistical period, reporting frequency and reporting time.

[0045] The statistical period is the length of the time period during which radar equipment and video equipment collect traffic flow characteristic parameters.

[0046] The statistical cycle of radar equipment is 1 minute; the statistical cycle of video equipment is 2 minutes.

[0047] Traffic volume is the number of vehicles passing through the detection location during the statistical period, and the calculation formula is as follows:

[0048]

[0049] Where Q is the number of vehicles passing the detection location during the statistical period, q i is the number of vehicles passing the detection position in lane i during the statistical period, and n is the number of lanes.

[0050] The speed is the weighted average of the location speeds of all vehicles passing through the detection location during the statistical period. The calculation formula is as follows:

[0051]

[0052] Where V is the weighted average of the location speeds of all vehicles passing through the detection location during the statistical period, v iis the average speed of all vehicles passing through the detection position in lane i during the statistical period, q i is the number of vehicles passing the detection position in lane i during the statistical period, and n is the number of lanes.

[0053] Step 2: Fusion of temporal attributes of radar data and video data

[0054] Align the timestamps of radar data and video data, unify the reporting frequency of radar data and video data through time interpolation method, and realize the time attribute fusion of radar data and video data. The reported radar data and video data are both traffic flow characteristic parameters.

[0055] Since the statistical cycle of radar equipment is 1 minute and the statistical cycle of video equipment is 2 minutes, it is necessary to use a time interpolation method to supplement the video data so that radar data and video data exist at each reporting time.

[0056] When supplementing video data of a certain reporting time, if video data exists at both the previous reporting time and the next reporting time, linear interpolation is used to obtain the video data of the reporting time; otherwise, nearest neighbor interpolation is used to obtain the video data of the reporting time.

[0057] If the reporting time of the video data to be supplemented is the earliest reporting time, the video data of the previous reporting time is missing; if the reporting time of the video data to be supplemented is the latest reporting time, the video data of the latter reporting time is missing, so the nearest neighbor interpolation is used for supplementation.

[0058] For example, if the reporting time starts at 0:00:00 and ends at 0:05:00, the radar device reports 6 traffic flow characteristic parameters, and the video device reports 3 traffic flow characteristic parameters. The reporting time of the 6 traffic flow characteristic parameters of the radar device is: 0:00:00, 0:01:00, 0:02:00, 0:03:00, 0:04:00, 0:05:00. The reporting time of the 3 traffic flow characteristic parameters of the video device is: 0:00:00, 0:02:00, 0:04:00.

[0059] To this end, three video data are added so that the frequency of video data becomes one per minute: linear interpolation is used to obtain the video data at 0:01:00 and 0:03:00; since 0:05:00 is the last video data and cannot be obtained through linear interpolation, the video data at 0:05:00 is obtained through nearest neighbor interpolation.

[0060] Step 3: Fusion of spatial attributes of radar data and video data

[0061] The physical topological relationship between the radar device and the video device is determined based on the detection position. The video device adjacent to the radar device participates in the prediction of the traffic flow characteristic parameters of the radar device to realize the spatial attribute fusion of radar data and video data.

[0062] The video device adjacent to the radar device is a video device located between the upstream radar device and the downstream radar device and closest to the radar device.

[0063] If there is a video device between the upstream radar device and the downstream radar device of the radar device, the video device closest to the radar device is marked as the video device adjacent to the radar device; if there is no video device between the upstream radar device and the downstream radar device of the radar device, the video device adjacent to the radar is marked as empty.

[0064] Step 4: Construct a traffic flow characteristic parameter prediction model based on multi-source data fusion

[0065] Using spatiotemporal fused radar data and video data, a traffic flow characteristic parameter prediction model is constructed based on long short-term memory network.

[0066] The time-space fused radar data and video data are a set of radar data, upstream radar data, downstream radar data, and video data adjacent to the radar device at the same reporting time.

[0067] The spatiotemporally fused radar data and video data include training set, validation set, and test set. The ratio of training set to validation set is 8:2. For example, the data from June 5 to 8, 2023 is used to train the traffic flow characteristic parameter prediction model, of which 80% of the data is used for model training and 20% of the data is used for model verification. The data from June 22 to 23, 2023 is used to test the trained traffic flow characteristic parameter prediction model.

[0068] The time-space fused radar data and video data are divided into two types: flow and speed. The traffic flow characteristic parameter prediction model is divided into two types: flow prediction model and speed prediction model.

[0069] The constructed traffic flow characteristic parameter prediction model includes an input layer (InputLayer), two LSTM layers (Long Short-Term Memory Network), a random deactivation layer (Dropout), and a fully connected layer (Dense).

[0070] The input layer needs to input a 10*4 data matrix (None, 10, 4). Among them: "None" means that the batch size is unknown, which can be set to 256 when training the model; "10" means 10 minutes of continuous spatiotemporal fused radar data and video data (10 groups); "4" means that the spatiotemporal fused radar data and video data include 4 parameters, namely radar data, upstream radar data, downstream radar data, and video data adjacent to the radar device.

[0071] The hidden_size (number of features in the hidden layer) of the two LSTM layers is set to 64.

[0072] The Dropout (random dropout rate) of the random dropout layer is selected as 0.2.

[0073] When compiling the model, select the following parameters: select adam for optimizer, mse for loss, mape for metrics, and the default settings for the rest of the parameters. When training the model, select the following parameters: select 256 for batch_size, 500 for epochs, True for shuffle (whether to shuffle the training data before each epoch), 0.2 for validation_split (the proportion of the training set used for the validation set), and the default settings for the rest of the parameters.

[0074] The spatial-temporal fusion radar data and video data of traffic volume are input into the traffic volume prediction model, and the output is the predicted traffic volume traffic flow characteristic parameters. The spatial-temporal fusion radar data and video data of speed are input into the speed traffic flow characteristic parameter prediction model, and the output is the predicted speed traffic flow characteristic parameters.

[0075] Step 5: Obtain traffic flow characteristic parameters predicted by multi-source data fusion

[0076] The real-time radar data and video data are fused with time and space attributes, and then 10 sets of continuous time-space fused radar data and video data are input into the traffic flow characteristic parameter prediction model to obtain the traffic flow characteristic parameters after 10 minutes.

[0077] The following is a specific description of this method using a domestic expressway as an example.

[0078] The research section is about 9 kilometers long, and the radar equipment and video equipment are deployed in the upward direction (the driving direction is the direction of increasing pile numbers). The layout diagram of the radar equipment and video equipment is shown in the figure below. Figure 3As shown in the figure, the blue dots represent radar devices and the red dots represent video devices. Radar devices and video devices are numbered along the driving direction, with a total of 41 radar devices and 13 video devices.

[0079] The radar data and video data from June 5 to 8, 2023 are collected as training data, and the radar data and video data from June 22 to 23, 2023 are collected as test data.

[0080] The statistical cycle of radar equipment is 1 minute; the statistical cycle of video equipment is 2 minutes; the number of lanes is 4;

[0081] The combination of radar equipment and video equipment is shown in Table 1.

[0082] Table 1

[0083] Group No. Intermediate radar number Upstream radar number Downstream radar number Adjacent video number 1 2 1 3 0 2 3 2 4 3 3 4 3 5 0 4 5 4 6 4 5 6 5 7 0 … … … … … 39 40 39 41 0

[0084] The input and output formats of the traffic flow characteristic parameter prediction model are shown in Table 2.

[0085] Table 2

[0086]

[0087] From 00:00:00 on June 5, 2023 to 23:59:00 on June 8, 2023, each group of equipment obtains 5760 sets of spatiotemporal fusion radar data and video data per minute. Because the input step length of the traffic flow characteristic parameter prediction model is 10 minutes and the prediction step length is 10 minutes, the traffic flow characteristic parameter prediction model can predict the traffic flow characteristic parameters from 00:18:01 to 00:19:00 on June 5, 2023 at the earliest, so each group of equipment obtains 5741 sets of training samples as shown in Table 2 per minute. There are 39 sets of equipment in the study section, and a total of 223,899 sets of training samples are obtained, of which 80% (179,119 sets) of training samples are used for traffic flow characteristic parameter model training, and 20% (44,780 sets) of training samples are used for traffic flow characteristic parameter model verification.

[0088] From 0:00:00 on June 22, 2023 to 23:59:00 on June 23, 2023, each set of equipment obtains 2880 sets of spatiotemporal fused radar data and video data per minute. Because the input step length of the traffic flow characteristic parameter prediction model is 10 minutes and the prediction step length is 10 minutes, the traffic flow characteristic parameter prediction model can predict the traffic flow characteristic parameters from 0:18:01 to 0:19:00 on June 22, 2023 at the earliest, so each set of equipment obtains 2861 sets of test samples as shown in Table 2 per minute. There are 39 sets of equipment in the research section, and a total of 111,579 sets of test samples were obtained, all of which were used for traffic flow characteristic parameter model testing.

[0089] The traffic volume predicted by multi-source data fusion is as follows: Figure 4 As shown in Figure 2, the speed of multi-source data fusion prediction is as follows: Figure 5 As shown. Figure 4 and Figure 5 It can be seen that the traffic flow characteristic parameters obtained by the spatiotemporal asynchronous multi-source data fusion prediction based on the long short-term memory network retain the real trend, while having a smoothing effect and reducing the noise of the radar equipment.

[0090] The above embodiments are only for illustrating the technical idea of ​​the present invention, and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the present invention.

Claims

1. A spatiotemporal asynchronous multi-source data fusion prediction method based on long short-term memory network, comprising the following steps: Step 1: Obtain highway radar data and video data Radar data and video data include basic information of radar equipment and video equipment and traffic flow characteristic parameters collected by radar equipment and video equipment; Step 2: Fusion of temporal attributes of radar data and video data Align the timestamps of radar data and video data, unify the reporting frequency of radar data and video data through time interpolation method, and realize the time attribute fusion of radar data and video data; Step 3: Fusion of spatial attributes of radar data and video data The physical topological relationship between the radar device and the video device is determined based on the detection position. The video device adjacent to the radar device participates in the prediction of the traffic flow characteristic parameters of the radar device, realizing the spatial attribute fusion of the radar data and the video data. Step 4: Construct a traffic flow characteristic parameter prediction model based on multi-source data fusion Using spatiotemporal fusion of radar data and video data, a traffic flow characteristic parameter prediction model is constructed based on long short-term memory network; Step 5: Obtain traffic flow characteristic parameters predicted by multi-source data fusion The spatiotemporally fused radar data and video data are input into the traffic flow characteristic parameter prediction model to obtain the predicted traffic flow characteristic parameters.

2. The spatiotemporal asynchronous multi-source data fusion prediction method based on long short-term memory network according to claim 1 is characterized by: In step 1, the basic information includes the equipment number, installation location, detection location, lane number, statistical period, reporting frequency and reporting time, and the traffic flow characteristic parameters include flow rate and speed.

3. The spatiotemporal asynchronous multi-source data fusion prediction method based on long short-term memory network according to claim 2 is characterized by: In step 1: The statistical cycle is the length of the time period during which the radar equipment and the video equipment collect traffic flow characteristic parameters. The statistical cycle of the radar equipment is 1 minute; the statistical cycle of the video equipment is 2 minutes; Traffic volume is the number of vehicles passing through the detection location during the statistical period, and the calculation formula is as follows: Where Q is the number of vehicles passing the detection location during the statistical period, q i is the number of vehicles passing the detection position in lane i during the statistical period, and n is the number of lanes; The speed is the weighted average of the location speeds of all vehicles passing through the detection location during the statistical period. The calculation formula is as follows: Where V is the weighted average of the location speeds of all vehicles passing through the detection location during the statistical period, v i is the average speed of all vehicles passing through the detection position in lane i during the statistical period, q i is the number of vehicles passing the detection position in lane i during the statistical period, and n is the number of lanes.

4. The spatiotemporal asynchronous multi-source data fusion prediction method based on long short-term memory network according to claim 1 is characterized by: In step 2, the timestamps of the radar data and the video data are aligned, and the video data is supplemented by a time interpolation method so that radar data and video data exist at each reporting time; the time interpolation method includes linear interpolation and nearest neighbor interpolation.

5. The spatiotemporal asynchronous multi-source data fusion prediction method based on long short-term memory network according to claim 4 is characterized by: When supplementing video data of a certain reporting time, if video data exists at both the previous reporting time and the next reporting time, linear interpolation is used to obtain the video data of the reporting time; otherwise, nearest neighbor interpolation is used to obtain the video data of the reporting time.

6. The spatiotemporal asynchronous multi-source data fusion prediction method based on long short-term memory network according to claim 1 is characterized by: In step 3, the video device adjacent to the radar device is a video device located between the upstream radar device and the downstream radar device and closest to the radar device.

7. The spatiotemporal asynchronous multi-source data fusion prediction method based on long short-term memory network according to claim 1 is characterized by: In step 4, the time-space fused radar data and video data are radar data at the same reporting time, upstream radar data, downstream radar data, and video data adjacent to the radar device.

8. The spatiotemporal asynchronous multi-source data fusion prediction method based on long short-term memory network according to claim 1 is characterized by: The traffic flow characteristic parameter prediction model includes an input layer, an LSTM layer, an LSTM layer, a random inactivation layer, and a fully connected layer connected in sequence; the traffic flow characteristic parameter prediction model includes a flow prediction model and a speed prediction model.

9. The spatiotemporal asynchronous multi-source data fusion prediction method based on long short-term memory network according to claim 1 is characterized by: In step 5, the radar data and video data collected in real time are fused with time attributes and space attributes, and then input into the traffic flow characteristic parameter prediction model to obtain the predicted traffic flow characteristic parameters.

Images