Rail transit passenger flow real-time prediction method based on streaming computation engine

Abstract

The invention relates to a rail transit passenger flow real-time prediction method based on a flow-oriented computing engine, and belongs to the field of real-time passenger flow prediction. The method comprises the following steps: firstly, acquiring historical ticket card records of a rail transit system, dividing according to time windows, and calculating short-time in-out passenger flow of each station, secondly, in a batch processing environment, constructing a time sequence prediction sample by using multiple rolling windows based on sliding aggregation passenger flow data, training a prediction model and storing the prediction model, and then, constructing an assembly line type real-time calculation environment by using Kafka and Spark-streaming, and finally, carrying out real-time calculation on the assembly line type real-time calculation environment. And a network-level short-time passenger flow calculation algorithm, a feature management module and a prediction control module are deployed, and finally, real-time ticket data flow is continuously processed, and future short-time passenger flow is predicted in a sliding manner. The method is used for solving the problem of real-time passenger flow prediction in a large-scale rail transit scene, meets the high-frequency sliding prediction demand in the flow scene, effectively deals with the influence of late ticket data, and maintains the time delay within 4 seconds when 70,000 ticket records are processed per minute.

Description

Technical Field

[0001] The invention relates to a real-time passenger flow prediction method for a rail transit network based on a distributed stream processing engine, and belongs to the field of real-time passenger flow prediction. Background Art

[0002] Accurate and reliable short-term passenger flow forecasting is the key to realizing intelligent application of rail transit system. Rail transit system runs at high speed in a closed environment and serves millions of passengers every day. Especially during rush hour, some subway stations face serious congestion, and some lines or sections are under extremely high passenger flow pressure, which brings huge challenges to operation management. In this case, accurate short-term passenger flow forecasting can provide important decision-making support for operation management, such as passenger flow guidance, train scheduling optimization and emergency management, thereby improving operational efficiency and resource allocation efficiency. At present, many researchers have proposed short-term passenger flow forecasting solutions based on big data technology. These methods can be roughly divided into two categories: one is offline forecasting based on batch processing, and the other is real-time forecasting based on stream processing. Compared with batch processing methods, stream processing can better meet the real-time needs of passenger flow forecasting and intelligent decision-making. Although the existing solutions have solved the technical problems of short-term passenger flow forecasting to a certain extent, there are still the following shortcomings:

[0003] (1) It cannot meet the needs of high-frequency sliding prediction. In real application scenarios, we expect to obtain passenger flow dynamics at a higher frequency. Compared with rolling prediction, sliding prediction is more suitable for real-time passenger flow prediction due to its high frequency characteristics. However, existing solutions are based on the task scenario of rolling prediction, and construct sample training and evaluation models based on passenger flow aggregated by rolling windows, resulting in insufficient generalization ability of the prediction model and low sliding prediction accuracy, which cannot meet the needs of high-frequency sliding prediction. Therefore, there is still a lot of room for improvement in the existing solutions to improve the model prediction accuracy in sliding prediction scenarios;

[0004] (2) Inefficient real-time passenger flow calculation. Existing solutions often only calculate passenger flow at a few stations, or only for a single entry and exit mode. It is not possible to efficiently calculate the entry and exit passenger flow of all stations in the entire rail transit network at the same time. Therefore, improving the efficiency of network-level entry and exit passenger flow calculation is still a problem that needs to be solved;

[0005] (3) The impact of late data on real-time predictions is not fully considered. Due to the instability of the data collection system and the network, there is usually a delay in the transmission of the card swiping records generated by the subway gate to the data center. These delayed data will cause errors between the real-time passenger flow calculated by the stream processing system and the actual passenger flow, and affect the subsequent online feature engineering and prediction links, thereby reducing the accuracy of real-time predictions. Therefore, studying how to deal with the impact of late data on prediction accuracy can fill the gap in current research.

[0006] For example, the patent "CN114118555A, a distributed subway passenger flow prediction method based on improved KNN" designs a KNN and LightGBM algorithm for short-term passenger flow prediction in subway scenarios, and predicts subway passenger flow in the next hour based on historical card swiping data. This method uses big data components HDFS and Spark to expand the computing power of the solution, but is limited to offline sliding prediction scenarios and cannot meet the needs of high-frequency real-time prediction. In addition, due to its large time granularity, it cannot provide more fine-grained passenger flow information.

[0007] The patent "CN110276474A, a method for predicting short-term passenger flow in urban rail transit stations" uses the parallel computing engine Spark to train the Elman neural network to predict short-term passenger flow for subway scenarios. The solution uses a time granularity of 15 minutes, which can provide more fine-grained passenger flow information than one hour. However, since this solution is a batch processing solution, it cannot meet the timeliness requirements of real-time prediction.

[0008] The patent "CN110782060A, Short-term prediction method and system for rail transit cross-section passenger flow based on big data technology" designs a systematic solution for the short-term prediction scenario of rail transit cross-section passenger flow. The solution uses HDFS to store historical ticket data, historical passenger flow data and real-time passenger flow data; uses the message middleware Kafka to access real-time card swiping data; uses Spark to calculate cross-section passenger flow in real time, and predicts the cross-section passenger flow in the next 5 minutes based on real-time cross-section passenger flow, historical cross-section passenger flow and historical inbound passenger flow. Although this solution uses stream processing technology to calculate and predict real-time cross-section passenger flow. However, it does not support high-frequency sliding prediction needs, cannot predict the inbound and outbound passenger flow of each station at a large spatiotemporal scale (rail transit network), and does not propose effective measures to deal with the impact of late data on prediction accuracy. Summary of the invention

[0009] The technical problem to be solved by the present invention is to perform real-time online short-term passenger flow prediction in a scenario where the rail transit system continuously generates massive amounts of passenger card swiping data. It can support some data-driven intelligent applications in the rail transit scenario, such as transfer recommendations, train scheduling, and emergency management.

[0010] The present invention analyzes the existing short-term passenger flow prediction scheme and its shortcomings. Specifically, the existing scheme still has three shortcomings:

[0011] (1) It does not support efficient large-scale network-level real-time passenger flow calculation. Most existing short-term passenger flow prediction schemes assume that real-time short-term passenger flow is aggregated or provided by a third party. A few real-time short-term passenger flow calculation schemes propose methods for real-time calculation of short-term passenger flow, but the short-term passenger flow calculated is station-level short-term passenger flow or short-term section passenger flow, rather than network-level inbound and outbound passenger flow.

[0012] (2) Only the rolling prediction scenario is considered. The existing short-term passenger flow prediction solution only constructs samples and trains and evaluates models for the rolling prediction scenario. Although the model trained by this method can be used in high-frequency sliding prediction scenarios, the prediction accuracy will decrease, and the generalization ability of the model still has a lot of room for improvement.

[0013] (3) Ignoring the impact of late data on real-time prediction. Existing solutions do not consider the impact of late data on real-time prediction when predicting short-term passenger flow in real time. Specifically, late data will cause an error between the short-term passenger flow calculated in real time and the true value. This error will further propagate along the data processing pipeline to the input features of the prediction model, thereby affecting the accuracy of real-time prediction, resulting in lower online prediction accuracy than offline prediction.

[0014] The technical solution of the present invention is: Based on the above analysis, a real-time passenger flow prediction method for rail transit network based on distributed stream processing engine is proposed. The method uses Spark as a distributed stream processing engine, and designs and implements a parallel computing algorithm for efficiently calculating network-level short-term in-and-out passenger flow, which improves the efficiency and timeliness of real-time network-level short-term passenger flow calculation. A multi-rolling window sample construction method is designed. This method constructs samples based on sliding aggregated passenger flow, generates more samples, and contains more detailed and rich passenger flow patterns, which helps to improve the generalization ability of the model. A real-time multi-mode feature construction method is designed. Through batch stream fusion, real-time and historical short-term passenger flow data are combined to construct multi-mode passenger flow features including real-time mode, daily mode and weekly mode. With the help of accurate daily mode and weekly mode features, the error of late data propagating into multi-mode passenger flow features is diluted, thereby alleviating the impact of late data on online prediction accuracy and improving the accuracy of online prediction. A trigger-controlled online feature engineering and prediction method is designed. This method specifies the sliding interval , trigger interval and watermark size As a parameter for process control, and , . Thus, stream processing applications are Internal trigger Calculation, before The calculation will process the late arrival data belonging to the previous sliding time window and update its calculation status to improve the accuracy of real-time short-term passenger flow calculation and prediction. Perform online feature engineering and prediction in each The first prediction within 1000ms obtains the best timeliness, and the The prediction can achieve the same accuracy as the offline prediction, and the intermediate predictions can find the balance between timeliness and accuracy.

[0015] The specific steps of a method for real-time passenger flow prediction of rail transit based on a streaming computing engine are as follows:

[0016] Step 1: Collect all historical smart card swiping records generated by the rail transit ticket collection system to obtain a static offline card swiping data set;

[0017] Step 2: Preprocess the offline card swiping data set and use the sliding window to calculate the offline network-level short-term in-and-out passenger flow according to the set time granularity and sliding interval;

[0018] Step 3: Use the multi-modal feature sample construction method with multiple rolling windows to process the offline network-level short-term in-and-out passenger flow and construct offline multi-modal feature samples;

[0019] Step 4: Use offline multimodal feature samples to train and evaluate the pre-trained model;

[0020] Step 5: Save the pre-trained model in a serialized manner;

[0021] Step 6: Deploy message middleware, persistent key-value storage service, stream computing engine, feature engineering controller, prediction control module and real-time data simulation module to provide an available stream processing environment;

[0022] Step 7: Use the real-time data simulation module to simulate the rail transit system according to the parameters set during deployment to generate massive real-time card swiping data streams, and connect them to the message middleware;

[0023] Step 8: Use the streaming computing engine to obtain the real-time card swiping data stream from the message middleware, perform data preprocessing, parallelly calculate the real-time network-level short-term inbound and outbound passenger flow, and write it into the persistent key-value storage service;

[0024] Step 9: Input the real-time network-level short-term inbound and outbound passenger flow, and use the trigger control mechanism composed of the feature engineering controller and the prediction control module to control the process of online feature engineering and prediction, and output the prediction results;

[0025] Step 10: Repeat Step 7-Step 9 to form a continuous stream processing pipeline including real-time ticket data stream access, data preprocessing, short-term passenger flow calculation, multi-mode feature construction and passenger flow prediction.

[0026] Specifically, in Step 1, the offline card swiping data set is composed of historical card swiping records of the rail transit system in the past period of time and is static. Each card swiping record includes fields such as card number, transaction time, station name, entry and exit identification, and transaction amount. Among them, only transaction time, station name, and entry and exit identification are the core fields required for network-level short-term passenger flow calculation and prediction.

[0027] Specifically, in Step 2, the data preprocessing of the offline card swiping data set includes extracting the core fields of each card swiping record, standardizing the site name, and mapping the standard site to the site number. In order to implement these preprocessing logics, we use Spark SQL for development and encapsulate it as Function. To standardize the processing flow and improve the performance of the program; when using the sliding window to calculate the offline network-level short-term in-and-out passenger flow at the specified time granularity and sliding interval, first group the pre-processed offline card swiping data set according to the sliding window, and use the parallel algorithm designed by the present invention for each group Calculate short-term inbound and outbound passenger flow at offline network level. Parallel algorithm This is implemented with Spark UDAF (User-Defined Aggregation Function), whose input is the data in the rail transit system within the time window. The card swiping dataset generated by subway stations , have partitions distributed on different nodes. represents the partition number, then the parallel algorithm The specific implementation process is as follows:

[0028] First, initialize a length of , Long integer array with initial value 0 , used to store intermediate calculation results, elements to store the incoming passenger flow. Elements store outbound passenger flow;

[0029] Then, all card swiping data records of each partition are traversed in parallel, and the long integer array is calculated and updated according to the entry and exit identification and site number of the card swiping record. ;

[0030] After that, wait for all partitions to traverse the local partition data set, and then collect each partition's data across nodes through the network. To a list storing long integer arrays middle;

[0031] Finally, the list of long integer arrays is merged two by two by adding the elements in place. An array of long integers for each partition in , and finally the network-level short-term inbound and outbound passenger flow within the time window is obtained.

[0032] The logic of the above offline data preprocessing and calculation of offline network-level short-term in-and-out passenger flow is similar to the logic of online data preprocessing and calculation of real-time network-level short-term in-and-out passenger flow in Step 8, so the function can be reused in Step 8 and parallel algorithms The difference is that the processing objects of offline preprocessing and short-term passenger flow calculation in Step 2 are static offline data sets, while the processing objects of real-time preprocessing and short-term passenger flow calculation in Step 8 are dynamic unbounded data streams. Unbounded data streams are more difficult to process than static offline data sets due to their dynamic and unbounded characteristics, and the data streams may arrive out of order due to network delays. Therefore, it is necessary to introduce a watermark mechanism and consider setting the output mode and trigger mechanism of stream processing.

[0033] Specifically, the multi-modal feature sample construction method of the multi-rolling window in Step 3 is expressed as follows:

[0034] make Represents any time during the operation period of the rail transit system. Indicates the time granularity of the short-term inbound and outbound passenger flow at the network level, Represents a time window The network-level short-term in-and-out passenger flow of the internal rail transit system, Indicates the number of historical short-term passenger flow records used to construct the features of a single mode, Indicates the time interval between adjacent historical short-term passenger flow records when constructing input features, Indicates 24 hours. Indicates the pattern of passenger flow, and Represents real-time mode, daily mode and weekly mode respectively, then , , Represents the network-level short-term inbound and outbound passenger flows in real-time mode, daily mode, and weekly mode, respectively. , , Represent the passenger flow characteristics of real-time mode, daily mode and weekly mode respectively. Network-level short-term inbound and outbound passenger flow and The relationship is as follows:

[0035]

[0036] Specifying a Mode Corresponding passenger flow characteristics It is expressed as follows:

[0037]

[0038] make Represents a sample of multi-modal features for predicting the short-term inbound and outbound passenger flow at the network level at time t:

[0039]

[0040] Let the number of stations in the rail transit system be , Indicates sites, Indicates the entry and exit identification. Indicates entry. Indicates exit, then the network-level short-term in-and-out passenger flow is a length of 2 is a long integer array consisting of the short-term inbound passenger flow and short-term outbound passenger flow of each station. Specifically, let Indicates the site In the time window The entry and exit signs are The station-level short-term passenger flow is a long integer variable, and the network-level short-term passenger flow is and network-level short-term outbound passenger flow is defined as follows:

[0041]

[0042] Contains and Network-level inbound and outbound passenger flow (unless otherwise specified, the short-term passenger flow mentioned below refers to the network-level inbound and outbound passenger flow , so it will be referred to as short-term passenger flow below), which can be expressed as follows:

[0043]

[0044] Let the total operation time of the rail transit system on a certain day be , the sliding time interval for calculating short-term passenger flow is , then , , .Right now can be sized The rolling window is divided into Time slices of non-overlapping parts of the segments; Can also be sized , the interval is The sliding time window is divided into There are time slices with overlapping segments; yes of times. At this time, let , Indicates The multi-mode feature samples of time slices. The sample set obtained based on the traditional feature engineering method is expressed as follows:

[0045]

[0046] The sample set obtained using the multi-modal feature sample construction method based on multi-rolling windows is expressed as follows:

[0047]

[0048] The multi-modal feature sample construction method of multiple rolling windows is equivalent to dividing the short-term passenger flow of sliding aggregation into The time interval of passenger flow records in each group is Then, we construct samples for each group using traditional feature engineering methods to obtain the sample set of each group, and finally merge them to obtain the complete sample set. The number of samples constructed by this method is about 2.5 times that of the traditional method. times, which can provide more and finer-grained useful information for model training, improve the generalization ability of the pre-trained model, and improve its performance in sliding prediction scenarios.

[0049] Let the time when the rail transit system starts operating on a certain day be The end time of the run is , the sliding time interval of sliding prediction is , then the sliding prediction of short-term passenger flow at the network level can be defined as:

[0050]

[0051] To intuitively compare the difference between sliding prediction and rolling prediction, the definition of rolling prediction of network-level short-term passenger flow is as follows:

[0052]

[0053] Specifically, the difference between sliding prediction and rolling prediction is that the , while the sliding prediction Can be specified more flexibly, when sliding prediction When the time is short, you can use the rolling prediction model to perform sliding prediction. Compared with rolling prediction, sliding prediction can specify the sliding time interval and provide more frequent prediction information. At the same time granularity, the number of sliding predictions is twice that of rolling predictions. times. If fixed Unchanged, time granularity The larger the value is, the richer the forecast information provided by the sliding forecast is compared with the rolling forecast.

[0054] Specifically, since the rail transit system usually stops operating at night, Therefore, we use the time period before the rail transit system was shut down on the previous day ( ) to complete the multi-modal features. is the total daily downtime of the rail transit system, express The time that the rail transit system has been running at the moment, then the multi-mode characteristics The construction formula can be redefined as follows:

[0055]

[0056] Specifically, in Step 4, the offline multi-modal feature sample set constructed in Step 3 is used for training and evaluation of the model, and the sample set is divided into a training set, a validation set, and a test set based on the specified date and time. Then, multiple prediction models are trained on the training set, the generalization ability of the model is tested on the validation set, and the prediction performance of the model is evaluated on the test set.

[0057] Specifically, in Step 5, the prediction model with the best performance is selected and saved as a binary file in a serialized manner and saved in a storage system.

[0058] Specifically, the steps for initializing the stream processing module in Step 6 are as follows:

[0059] Step 6.1: Cluster planning and construction. Allocate different numbers of nodes and hardware resources for the message middleware Kafka and the streaming computing engine Spark; install the corresponding software dependencies on each node, set the IP address and port used for component communication, and build the Kafka cluster and Spark cluster.

[0060] Step 6.2: Deploy the message middleware. Create a Topic that connects to the real-time card swiping data stream and is consumed by Spark-Streaming according to the specified Topic name and partition quantity.

[0061] Step 6.3: Deploy a persistent key-value storage service. Enable the persistent key-value storage service to pull and persist the short-term passenger flow data of the past week from the distributed storage system HDFS.

[0062] Step 6.4: Deploy the streaming computing engine. Specify the parameters for the real-time short-term passenger flow prediction task and submit the Spark Structured Streaming application to the Spark cluster. At this point, the stream processing application will create the application daemon Driver on the Driver node of the Spark cluster, load the application code in the Driver, and distribute it to the worker node for execution.

[0063] Step 6.5: Deploy the feature engineering controller. When the daemon Driver loads and runs the application code, it creates the feature engineering controller. Examples , used for subsequent feature engineering.

[0064] Step 6.6: Deploy the prediction control module. When the daemon Driver loads and runs the application code, it deserializes and loads the pre-trained model serialized and saved in Step 5 into the Driver's memory for caching, which is used for subsequent real-time prediction.

[0065] Step 6.7: Deploy the real-time data simulation module. Start the real-time data simulation module by specifying the date and start and end time of the simulated historical card swiping data, the load expansion multiple, the Kafka communication URL, and the delay simulation function.

[0066] After completing the above steps, the stream processing module is initialized, and the components will provide an available stream processing environment for the stream processing pipeline of continuous real-time short-term passenger flow prediction.

[0067] Specifically, the function of the feature engineering controller in Step 6 is to cache historical and real-time network-level short-term inbound and outbound passenger flow data in the application's daemon Driver as the data source for online feature engineering, avoiding the large amount of cross-node network transmission (shuffle) caused by the second time window-based streaming aggregation. In addition, the feature engineering controller abstracts the logic of online feature engineering into four functions in order to standardize the process of online feature engineering and facilitate improvement and expansion. We designed a custom type (Class) To implement the feature engineering controller, when deploying the feature engineering controller, the application daemon Driver will create an instance of the feature engineering controller , used for subsequent control processes. Specifically:

[0068] Feature Engineering Controller The encapsulated member variables are shown in Table 1. Among them, the real-time passenger flow cache and historical passenger flow cache They are used to cache historical and real-time network-level short-term inbound and outbound passenger flow data respectively. TreeMap[Long,Array[Long]] represents a key-value pair set with a key of long integer Long and a value of long integer array Array[Long] sorted by key. The remaining member variables are used to assist in the process control of online feature engineering, such as judging Whether to initialize the current date , stores the current event time of the latest short-term passenger flow event time wait.

[0069] Table 1 FEController important member variables

[0070] Table 2 FEController core function table

[0071] The four encapsulated core member functions are shown in Table 2. The functions of each function are as follows:

[0072] Initialization function : The effect is based on the incoming timestamp (Latest network-level short-term inbound and outbound passenger flow event time), update current date Auxiliary variables such as , calculate and initialize the real-time passenger flow cache and historical passenger flow cache The time range of the required network-level short-term inbound and outbound passenger flow is obtained, and the corresponding data records are queried and loaded from the persistent key-value storage service to initialize the two passenger flow caches.

[0073] Real-time passenger flow data cache update function : Its function is to transmit real-time network-level short-term inbound and outbound passenger flow and update the real-time passenger flow cache If a new short-term passenger flow record is added, the real-time passenger flow cache is removed. The oldest record in the.

[0074] Real-time feature time series calculation function : The effect is based on the current event time passed in Time-indexed sequence of network-level short-term inbound and outbound passenger flow records required to calculate real-time pattern characteristics (make for ,but correspond ).

[0075] Multi-mode feature constructor :The function is to pass in (Right now ),calculate and . and use From real-time customer flow cache Index data records to construct real-time mode passenger flow characteristics ; From using and From the historical passenger flow cache Index data records to construct daily and weekly passenger flow characteristics and ;

[0076] Specifically, the role of the prediction control module in Step 6 is to deserialize the pre-trained model serialized and saved in Step 5 in the application's daemon Driver, and load it from the specified storage path into the Driver's memory for caching. At the same time, the Onnx Runtime operating environment for model inference is initialized in the Driver to ensure that the pre-trained model trained in the Python environment can be used for model inference in the Java programming environment.

[0077] Specifically, the real-time data simulation module in Step 6 simulates the rail transit system to generate ticket data streams and write them into the message middleware. It is implemented by a multi-process program and does not rely on stream processing applications. When starting, you need to specify various parameters such as the date and start and end time of the simulated historical card swipe data, the load expansion multiple, the communication URL of Kafka, and the delay simulation function. The delay simulation function is used to simulate the total transmission delay of the card swipe data from the gate to the site proxy server and then forwarded to the data center. To simulate , we designed a delay simulation function as shown below:

[0078]

[0079] In the above formula, U(a,b) represents a uniform distribution function with a lower bound and b upper bound. Indicates The probability of generating A random number uniformly distributed between .

[0080] After starting the real-time data simulation module according to the specified parameters in Step 7, the workflow is as follows:

[0081] Step 7.1: Data loading. Read the card swipe data within the specified date and time range according to the submitted parameters.

[0082] Step 7.2: Data preprocessing. Expand the total amount of data in the data set according to the load expansion multiple, take the "card swiping time" column to traverse each record, and add the randomly generated delay function to simulate the delay. , generating a new column "Write Time".

[0083] Step 7.3: Data writing. Based on the "Write Time" column, index the sub-datasets of the card swiping dataset loaded in Step 7.2 second by second and write them to the message middleware Kafka.

[0084] Step 7.3.1: Initialize loop variables. Let the start time of the simulated passenger flow writing specified by the parameter be , the end time of simulated passenger flow writing is ;

[0085] Step 7.3.2: Get the system time. Get the system time when entering a new cycle ;

[0086] Step 7.3.3: Loop condition judgment. Is it less than If yes, jump to Step 7.3.4; otherwise, jump to Step 7.6;

[0087] Step 7.3.4: Customer flow interception. Filter the value of the "Write Time" column in sub-datasets of;

[0088] Step 7.3.5: Passenger flow writing. Enable a new auxiliary process to write the intercepted passenger flow data to the specified topic in Kafka. When writing, the data will be assigned a partition number based on the partition strategy of the specified parameters, "partition by site" or "random partition", and then written to the corresponding partition of the Topic in Kafka;

[0089] Step 7.3.6: Processing time calculation. Get the current system time , calculate the processing time used by this loop ;

[0090] Step 7.3.7: Put the main process to sleep. time;

[0091] Step 7.3.8: Update loop variables. Add one second, , and jump to Step7.3.2;

[0092] Step 7.4: Terminate the program. Close the site simulator and terminate the program.

[0093] Specifically, the logic of real-time preprocessing and short-term passenger flow calculation in Step 8 is similar to that in Step 2, so the Spark SQL implementation in Step 2 is reused. Function and parallel customer flow algorithm implemented using Spark UDAF The difference between Step 8 and Step 2 is that Step 8 processes dynamic unbounded data streams, and the data access method is different, and the method of processing and writing calculation results is also different. The specific construction process is as follows:

[0094] Step 8.1: The stream computing engine obtains the real-time card swiping data stream from the message middleware at a certain time interval;

[0095] Step 8.2: Use The function preprocesses the real-time card swiping data stream and groups it based on the sliding time window;

[0096] Step 8.3: Call the parallel algorithm Process the card swiping data in each group (sliding time window) and calculate the real-time network-level short-term in-and-out passenger flow;

[0097] Step 8.4: Write the calculated real-time network-level short-term inbound and outbound passenger flow into the persistent key-value storage service.

[0098] Specifically, the persistent key-value storage service is deployed to store the short-term passenger flow data of the past week and the network-level short-term inbound and outbound passenger flow calculated in real time. implement The data source for the operator to initialize or Checkpoint for process failure recovery.

[0099] Specifically, the role of the trigger-controlled online feature engineering and prediction in Step 9 is to determine whether to perform online feature engineering and prediction based on the trigger control mechanism and perform process control. In order to implement the trigger control mechanism, we specify the watermark size according to the distribution of late data. and stream processing trigger interval and by setting , , To assist process control, Indicates each sliding time interval The number of times the internal trigger flow processing, Indicates each sliding time interval The number of times online feature engineering and prediction are triggered. Each sliding time interval Corresponding to a window size of The sliding time interval is Sliding time window.

[0100] Specifically, at each sliding interval In the stream processing application, Calculation, corresponding to micro-batch, but only the first The calculation will process the late data in the previous sliding window and update the calculation status, so this mechanism only applies to each Inside front This method takes into account the impact of late data on real-time prediction accuracy by performing online feature engineering and prediction in each Trigger multiple calculations internally, process late data and update the calculation status. The first prediction within a period has the best timeliness, but is most affected by late data, and the real-time prediction accuracy decreases the most; each subsequent prediction will introduce delay, while improving the accuracy of real-time predictions until The first The specific steps of the above control process are as follows:

[0101] Step 9.1: Get the trigger time of the current micro-batch ;

[0102] Step 9.2: Example of combining feature engineering controller and real-time network-level short-term inbound and outbound passenger flow , determine whether initialization is needed , and whether an update is required Real-time passenger flow data cache in ;

[0103] Step 9.2.1: Determine the instance of the feature engineering controller Whether to initialize and Is it empty?

[0104] Step 9.2.1.a: If Uninitialized and If it is empty, jump to Step7;

[0105] Step 9.2.1.b: If Uninitialized and If not empty, get The event time of the latest short-term passenger flow record in , and execute the initialization function initialization , and then execute the real-time passenger flow cache update function Update real-time passenger flow cache ;

[0106] Step 9.2.1.c: If Initialize and If not empty, get The event time of the latest short-term passenger flow record in And with the current date Compare and determine whether the date is updated. If the date is updated, the initialization function needs to be executed initialization ; After that, regardless of whether the date is updated, the real-time passenger flow cache update function is executed Update real-time passenger flow cache ;

[0107] Step 9.2.1.d: If Initialize and If it is empty, skip to the next step directly;

[0108] Step 9.3: Example based on feature engineering controller The trigger time of the current micro-batch Calculate the number of calculations within the current sliding window , if we calculate the number of , then with the help of The encapsulated method performs online feature engineering, constructs real-time multi-modal features, and passes them to the pre-trained model cached in the predictive control module. Make real-time predictions and output prediction results;

[0109] Step 9.3.1: Example based on feature engineering controller The trigger time of the current micro-batch Calculate the number of calculations within the current sliding window ;

[0110] Step 9.3.2: Determine the number of calculations in the current window Is it less than the number of predictions within the window? ;

[0111] Step 9.3.2.a: If If not, jump to Step 7.

[0112] Step 9.3.2.b: If If it is established, online feature engineering and prediction will be performed. The specific processing flow is as follows:

[0113] First, an example based on a feature engineering controller The number of calculations in the current window Determine whether the current sliding window is updated. If updated, update the current event time , used for subsequent process control.

[0114] Secondly, the time index sequence of short-term passenger flow records required for multi-modal features is calculated. Compute the time index sequence required to construct the real-time model . Assume that the current event time is The corresponding timestamp is , then the time index sequence for .

[0115] Then, multimodal features are constructed in real time. Using time-indexed sequences From real-time customer flow cache Obtain the short-term passenger flow records required to construct the real-time mode passenger flow characteristics and construct ;based on calculate and and use and From the historical passenger flow cache The short-term passenger flow records required to obtain the daily and weekly passenger flow characteristics are constructed and ; Finally, the multi-modal features are obtained by splicing .

[0116] Finally, call the pre-trained model cached by the Driver daemon , and pass in the multi-modal features constructed in real time ( , ) performs online passenger flow prediction and outputs real-time prediction results.

[0117] Step 9.4: Perform loop control and jump to Step 7 to start a new round of real-time short-term passenger flow prediction.

[0118] In any case, the processing flow will eventually return to Step 7 to start a new round of processing. This is because the processing object of stream processing is the unbounded data stream that continuously reaches the stream processing module. Therefore, the stream processing module needs to form a continuous stream processing pipeline including real-time ticket data stream access, data preprocessing, short-term passenger flow calculation, multi-mode feature construction and passenger flow prediction.

[0119] The beneficial effects of the present invention are:

[0120] (1) The parallel computing algorithm proposed in this invention for efficiently computing network-level short-term inbound and outbound passenger flows can efficiently and scalably aggregate massive card swiping data streams in real time and simultaneously complete the calculation of short-term inbound and outbound passenger flows;

[0121] (2) The multi-rolling window sample construction method proposed in the present invention constructs samples based on the short-term passenger flow of sliding aggregation. The sample data set can provide more detailed and rich passenger flow patterns for model training, which can improve the generalization ability and prediction effect of the prediction model in high-frequency sliding prediction scenarios;

[0122] (3) Compared with the single-mode feature construction method, the real-time multi-mode feature construction method proposed in the present invention can further dilute the error caused by the late arrival data propagating into the multi-mode passenger flow features, thereby alleviating the impact of the late arrival data on the online prediction accuracy and improving the online prediction accuracy.

[0123] (4) The trigger-controlled online feature engineering and prediction method proposed in the present invention fully considers the impact of late data and performs multiple predictions for the same sliding window, making it easier for users to balance the real-time nature and accuracy of the prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0124] Figure 1 is a flow chart of the steps of the present invention;

[0125] Figure 2 A schematic diagram of a real-time short-term passenger flow prediction pipeline constructed by the present invention;

[0126] Figure 3 A schematic diagram of the present invention for constructing multi-mode passenger flow characteristics and making predictions in real time;

[0127] Figure 4 This is a graph of the first online prediction accuracy loss in the sliding window corresponding to different time granularities;

[0128] Figure 5 Graph showing passenger flow accuracy and online prediction accuracy loss corresponding to multiple prediction cycles within the sliding window. DETAILED DESCRIPTION

[0129] The present invention will be further described below in conjunction with the accompanying drawings and specific implementation methods based on the description.

[0130] like Figure 1 As shown, a method for real-time passenger flow prediction of a rail transit network based on a distributed stream processing engine comprises the following steps:

[0131] Step 1: Collect all historical smart card swiping records generated by the rail transit ticket collection system to obtain a static offline card swiping data set;

[0132] Step 2: Preprocess the offline card swiping data set and use the sliding window to calculate the offline network-level short-term in-and-out passenger flow according to the set time granularity and sliding interval;

[0133] Step 3: Use the multi-modal feature sample construction method with multiple rolling windows to process the offline network-level short-term in-and-out passenger flow and construct offline multi-modal feature samples;

[0134] Step 4: Use offline multimodal feature samples to train and evaluate the pre-trained model;

[0135] Step 5: Save the pre-trained model in a serialized manner;

[0136] Step 6: Deploy message middleware, persistent key-value storage service, stream computing engine, feature engineering controller, prediction control module and real-time data simulation module to provide an available stream processing environment;

[0137] Step 7: Use the real-time data simulation module to simulate the rail transit system according to the parameters set during deployment to generate massive real-time card swiping data streams, and connect them to the message middleware;

[0138] Step 8: Use the streaming computing engine to obtain the real-time card swiping data stream from the message middleware, perform data preprocessing, parallelly calculate the real-time network-level short-term inbound and outbound passenger flow, and write it into the persistent key-value storage service;

[0139] Step 9: Input the real-time network-level short-term inbound and outbound passenger flow, and use the trigger control mechanism composed of the feature engineering controller and the prediction control module to control the process of online feature engineering and prediction, and output the prediction results;

[0140] Step 10: Repeat Step 7-Step 9 to form a continuous stream processing pipeline including real-time ticket data stream access, data preprocessing, short-term passenger flow calculation, multi-mode feature construction and passenger flow prediction.

[0141] Figure 1 The processing flow at the bottom corresponds to Step 1 to Step 5 in the above steps. The part corresponding to offline batch processing will only be executed once. Its function is to train the serialized pre-trained model required when deploying the predictive control module in Step 6. Figure 1 The processing flow at the top of the flow corresponds to Step 6 to Step 10 in the above steps, corresponding to the real-time stream processing part. Step 6 is used to initialize the stream processing module to provide an available stream processing environment for the subsequent real-time short-term passenger flow prediction pipeline. It will only be executed once; Step 7 to Step 10 will be executed repeatedly to form a continuous stream processing pipeline for real-time short-term passenger flow prediction. The pipeline is as follows: Figure 2 shown.

[0142] like Figure 2 As shown, Figure 2 From left to right and from bottom to top, the continuous stream processing pipeline for real-time short-term passenger flow prediction includes the following components and modules: real-time data simulation module, message middleware Kafka, distributed stream processing engine Spark-Streaming, and persistent key-value storage service. The interaction between them is as follows:

[0143] First, the real-time data simulation module simulates the card swiping data within the specified historical time range, and writes the real-time card swiping data stream into the message middleware Kafka second by second according to the specified data load multiple and simulation delay;

[0144] Secondly, after receiving the new real-time card swiping data stream, the message middleware Kafka notifies the stream processing engine Spark-Streaming to consume it. After Spark-Streaming sends a consumption request to Kafka, Kafka forwards the real-time card swiping data stream corresponding to the request information and converts it into a streaming structured data set (DataFrame) corresponding to a micro-batch.

[0145] After that, Spark-Streaming uses the streaming DataFrame composed of the current micro-batch real-time card swiping data stream The function performs data preprocessing and groups the preprocessed streaming DataFrame using a sliding time window, calling a parallel algorithm for each group. Calculate real-time network-level short-term inbound and outbound passenger flow , and write to the persistent key-value storage service.

[0146] When the stream processing module is initialized in Step 6, the stream processing application is submitted to the Spark cluster. The Spark cluster starts the daemon process Driver on the Driver node and creates The instantiation object When the continuous stream processing pipeline for real-time short-term passenger flow prediction starts working, an instance of the feature engineering controller Not initialized. At this time, if the real-time network-level short-term inbound and outbound passenger flow Not empty, Will obtain The event time of the latest network-level short-term inbound and outbound passenger flow records , execute the initialization function ( ) Calculate real-time passenger flow cache and historical passenger flow cache The time range of the required historical short-term passenger flow records, and initiate a data query request to the persistent key-value service, pull the query results to the Driver process for initialization In addition, when the Driver node of the stream processing application fails, restarting the stream processing program will still re-run the process. Initialization, at this time the persistent key-value storage service is equivalent to an instance of the feature engineering controller Checkpoint for failure recovery.

[0147] Finally, once the instance of the feature engineering controller Initialized, after Spark-Streaming calculates the real-time network-level short-term inbound and outbound passenger flow , and write it to the persistent key-value storage service. Will collect across nodes Go to the Driver node and execute the real-time passenger flow cache update function Update real-time passenger flow cache , and based on real-time passenger flow cache and historical passenger flow cache Constructing multimodal features , and finally call the pre-trained model cached by the prediction control module to predict the next time interval The real-time network-level short-term inbound and outbound passenger flow is Figure 3 shown.

[0148] The present invention is described in detail below in conjunction with specific experimental data.

[0149] The implementation process of the present invention is divided into 7 parts, namely: development environment construction, offline preprocessing and short-term passenger flow calculation, offline multi-mode feature sample construction, model training, evaluation and preservation, stream processing module initialization, continuous stream processing, and real-time prediction performance evaluation.

[0150] The development environment construction phase is mainly used to install the development environment that offline batch processing and real-time stream processing rely on. The process is as follows:

[0151] Plan an eight-node cluster to install Kafka cluster, Spark cluster, persistent key-value storage service and real-time data simulation module. The Spark cluster consists of 1 master node and 3 slave nodes. In the experiment, we plan to start the daemon process Driver of the Spark application on the master node, which is used as the core controller of the Spark application to manage task scheduling, cluster communication and result collection. The Spark cluster is used to complete offline data preprocessing and sample construction, real-time data preprocessing, real-time network-level short-term inbound and outbound passenger flow calculation, online feature engineering and real-time passenger flow prediction. The underlying Spark uses HDFS as a distributed file system by default. In addition, in order to train the pre-trained model in the Python environment and use the pre-trained model for online inference in the Java environment, we additionally installed the AI ​​engine Pytorch and the inference engine Onnx Runtime environment on the Driver node of the Spark cluster; to ensure the fault tolerance of the feature engineering control module in the Driver process, we also deployed the lightweight database SQLite as a persistent key-value storage service on the Driver node. The Kafka cluster consists of three nodes, which are used to receive and cache real-time card swipe data streams and notify the SparkStructured Streaming application deployed on the Spark cluster to consume the real-time card swipe data streams. The remaining node is used to deploy the real-time data simulation module, which simulates the real-time card swipe data streams generated by multiple subway stations in a multi-process manner and writes them into Kafka.

[0152] In the offline preprocessing and short-term passenger flow calculation stage, the offline card swiping data set is mainly preprocessed, and the offline network-level short-term in-and-out passenger flow is calculated. The process is as follows:

[0153] Collect four months of smart card swiping records from a city's rail transit system, obtain an offline card swiping dataset, and upload it to the distributed file system HDFS cluster;

[0154] Use Spark offline batch processing program to read all card swipe records in HDFS and use Spark SQL to implement preprocessing functions , extract the card number, transaction time, site name, and entry and exit identification of each card swiping record, standardize the site name, and map the site name to the site number to complete the offline preprocessing; then, , the sliding frequency is The sliding time window is used to group the card swiping data set after offline preprocessing and call the parallel algorithm Aggregate the card swiping data of each group to obtain the network-level short-term in-and-out passenger flow and write it to HDFS.

[0155] Parallel Algorithms The input is the rail transit system in the time window Card swiping datasets generated by sites , which has partitions distributed on different nodes. Represents the partition number, then the specific implementation steps of the parallel algorithm are as follows:

[0156] First, initialize a length of , Long integer array with initial value 0 , used to store intermediate calculation results, elements to store the incoming passenger flow. Elements store outbound passenger flow;

[0157] Then, all card swiping data records of each partition are traversed in parallel, and the long integer array is calculated and updated according to the entry and exit identification and site number of the card swiping record. ;

[0158] After that, wait for all partitions to traverse the local partition data set, and then collect each partition's data across nodes through the network. To a list storing long integer arrays middle;

[0159] Finally, the list of long integer arrays is merged two by two by adding the elements in place. An array of long integers for each partition in , and finally the network-level short-term inbound and outbound passenger flow within the time window is obtained.

[0160] In this embodiment, the number of stations in the rail transit system 166, the number of partitions is 192, sliding time interval In order to compare the impact of late data on real-time prediction accuracy at different time granularities during the continuous stream processing and performance testing phase, multiple groups of time granularities are used here. (i.e. 1, 3, 5, 10, 15 minutes) Calculate the short-term inbound and outbound passenger flow at the network level.

[0161] In the offline multi-mode feature sample construction phase, the multi-mode feature sample construction based on multiple rolling windows is mainly carried out, and the process is as follows:

[0162] Pull the network-level short-term inbound and outbound passenger flows calculated in the offline preprocessing and short-term passenger flow calculation construction phase from the distributed file system HDFS cluster to the Python environment of the Driver node;

[0163] In the Python environment of the Driver node, perform multi-modal feature sample construction based on multi-scrolling windows to obtain a sample data set .

[0164] The sample data set Divide into training set , validation set and test set When dividing, the division method cannot be simply and roughly used in the proportional division method, because the sample sequence is the splicing of time slices divided by multiple rolling windows, so it is necessary to specify the date and time used for division to filter the samples.

[0165] In this example, we use the last week's samples as , the penultimate week sample is The remaining data is That is, the date and time of the specified split sample set is Monday to 00:00 on the last week and the second to last week.

[0166] The training, evaluation and saving phase of the model mainly involves training, evaluating and saving the pre-trained model. The process is as follows:

[0167] Select a prediction model, select multiple open source and mature short-term passenger flow prediction models, and initialize the network structure and model parameters in the AI ​​engine Pytorch. This example selects five classic short-term passenger flow prediction models for experiments. These five models are: GCN, GRU, TGCN, ResLSTM and Conv-GCN;

[0168] AI engine Pytorch constructs a sample training set in the offline multi-modal feature sample construction phase by traversing the batch loading method , and train the prediction model and update the model parameters. The training is divided into multiple rounds (epochs), each round uses Evaluate the generalization ability of the model and update the direction of model learning.

[0169] After the training is completed, multiple short-term passenger flow prediction models can be obtained. Evaluate and compare multiple prediction models. Four indicators are used here to evaluate the real network-level short-term passenger flow and predicted network-level short-term passenger flow The difference between is the number of samples, then the four indicators that can be selected can be formulated as:

[0170]

[0171] Among them, Accuracy is the accuracy;

[0172]

[0173] Among them, RMSE is the root mean square error;

[0174]

[0175] Among them, MAE is the mean absolute error;

[0176]

[0177] Where WMAPE is the weighted mean absolute percentage error;

[0178] We used the above four evaluation indicators on the above five classic short-term passenger flow prediction models to compare the prediction performance of the models trained by the traditional feature engineering method and the multi-modal feature sample construction method based on multi-rolling windows. The experimental data obtained are as follows:

[0179] The prediction performance of all models trained by traditional feature engineering methods decreased in sliding scenarios compared to rolling. The accuracy decreased by at least 0.4%, the RMSE increased by at least 1.55, and the MAE increased by at least 0.77. Among them, the ResLSTM model had the most significant decrease, with its accuracy decreased by 1.07%, RMSE increased by 3.74, and MAE increased by 1.79. This shows that the models trained by traditional feature engineering methods lack more fine-grained learning, have weak model generalization ability, and suffer performance loss when facing higher-frequency sliding prediction tasks;

[0180] All models trained using the multi-modal feature sample construction method based on multiple rolling windows have improved their prediction accuracy in rolling prediction and sliding prediction scenarios compared to traditional feature engineering methods. The accuracy of rolling prediction has increased by at least 0.82%, and the accuracy of sliding prediction has increased by at least 1.16%. The most obvious improvement is in the TGCN model, with a rolling prediction accuracy improvement of 1.74% and a sliding prediction accuracy improvement of 2.27%. This shows that the multi-modal feature sample construction method based on multiple rolling windows can effectively improve the generalization ability of the model and improve the prediction effect of the model in high-frequency prediction scenarios;

[0181] Among all the prediction models, ConvGCN achieved the best prediction performance. Specifically, ConvGCN achieved a rolling prediction accuracy of 88.95%, a MAE of 21.58, a RMSE of 38.42, and a WMAPE of 9.90% on the sample test set of the last week; a sliding prediction accuracy of 89.00%, a MAE of 21.55, a RMSE of 38.29, and a WMAPE of 9.88%;

[0182] After evaluating multiple short-term passenger flow prediction models, the ConvGCN model with the best offline sliding prediction performance was selected as the pre-training model. , and convert it into a binary file in a serialized manner and save it to the specified file path of the distributed file system HDFS.

[0183] The stream processing module initialization phase mainly initializes the stream processing module to provide an available stream processing environment for the real-time short-term passenger flow prediction pipeline in the subsequent phase.

[0184] Deploy the message middleware. Create a topic for real-time access to the card swiping data stream and consumption by Spark-Streaming according to the specified topic and partition quantity.

[0185] Deploy a persistent key-value storage service. Enable the persistent key-value storage service to pull and persist the short-term passenger flow data of the past week from the distributed storage system HDFS.

[0186] Deploy the streaming computing engine. Specify the parameters for the real-time short-term passenger flow prediction task and submit the Spark Structured Streaming application to the Spark cluster. At this point, the stream processing application will create the application daemon Driver on the Driver node of the Spark cluster, load the application code in the Driver, and distribute it to the worker node for execution.

[0187] Deploy the feature engineering control module. When the daemon Driver loads and runs the application code, it creates a feature engineering controller. Examples , used for subsequent feature engineering.

[0188] Deploy the prediction control module. When the daemon Driver loads and runs the application code, it deserializes and loads the serialized pre-trained model into the Driver's memory for caching, which is used for subsequent real-time prediction.

[0189] Deploy the real-time data simulation module. Start the real-time data simulation module by specifying the date and start and end time of the simulated historical card swiping data, the load expansion multiple, the Kafka communication URL, and the delay simulation function.

[0190] After completing the above deployment, the stream processing module is initialized and can provide an available stream processing environment for the real-time short-term passenger flow prediction pipeline in the subsequent stage.

[0191] The continuous flow processing stage is mainly used to construct a real-time short-term passenger flow prediction pipeline, continuously process the continuous incoming real-time card swiping data stream, and perform short-term passenger flow prediction. The process is as follows:

[0192] Construct a real-time short-term passenger flow prediction pipeline:

[0193] Real-time card swiping data access: The real-time data simulation module simulates the rail transit system according to the parameter settings specified during deployment to generate massive real-time card swiping data streams, and accesses them to the message middleware;

[0194] When specifying parameters for the real-time data simulation module, this embodiment specifies two data rates: 1x data rate and 5x data rate. The former simulates the normal card swipe data flow load, and the latter simulates the high-voltage card swipe data flow load. In addition, when simulating the parameters of the delay function, this embodiment specifies , , , , , to simulate the maximum lateness within 30 seconds of the delay distribution .Should Under the distribution, there is a 50% probability that the network transmission is normal, and the delay is evenly distributed between 0.5 seconds and 3 seconds; there is a 50% probability that the network is abnormal, and a random delay evenly distributed between 3 seconds and 27 seconds is added on the basis of the normal transmission delay. Compared with the transmission delay in the real application scenario, which is more lagging and unstable, if our invention can work normally in this scenario, then in the scenario with a more stable network environment, our invention can achieve better performance;

[0195] Real-time preprocessing and short-term passenger flow calculation: The stream computing engine obtains the real-time card swiping data stream from the message middleware for data preprocessing, and then calculates the real-time network-level short-term inbound and outbound passenger flow in parallel and writes it to the persistent key-value storage service;

[0196] This process uses offline preprocessing and short-term passenger flow calculation to preprocess data. Parallel algorithm for computing short-term inbound and outbound passenger flow at network level The difference between offline preprocessing and short-term passenger flow calculation is that the processing objects of real-time preprocessing and short-term passenger flow calculation are unbounded data streams, which require watermarks. To control the tolerance for the late arrival time of the real-time card swiping data flow, set the output mode of the real-time network-level short-term in and out passenger flow and the trigger time interval of the trigger calculation .

[0197] In this embodiment, the watermark Refer to the total transmission delay The distribution is designed to be 30 seconds to ensure that all late data can be processed by the stream processing engine. Assume that the event time of the latest network-level short-term inbound and outbound passenger flow when the calculation was last triggered is , then each time the stream processing engine triggers a calculation, it will Calculate water level If the event time of the late data is less than the watermark , then late data will not be processed; if the event time of late data is higher than the watermark , then the late data will be processed and the corresponding state (i.e. calculation result) will be updated at the same time. , stream processing can be triggered more frequently, caching smaller processing delays at the cost of higher resource overhead. To balance resource overhead and processing delay, this embodiment sets 7.5 seconds.

[0198] Trigger-controlled online feature engineering and prediction: input the real-time network-level short-term inbound and outbound passenger flow corresponding to the current micro-batch , and based on the example of feature engineering controller The trigger control mechanism composed of the prediction control module controls the process of online feature engineering and prediction and outputs the prediction results;

[0199] In this embodiment, because the sliding time interval The trigger interval is 1 minute. is 7.5 seconds, so 8 calculations are triggered per minute, each calculation corresponds to a micro-batch. After being initialized, each micro-batch calculates a non-empty real-time network-level short-term inbound and outbound passenger flow When , but only the first 5 micro-batches per minute will be used for online feature engineering and prediction;

[0200] The process of controlling online feature engineering and prediction for a single micro-batch is as follows:

[0201] Get the trigger time of the current micro-batch ;

[0202] Example of combining feature engineering controller and real-time network-level short-term inbound and outbound passenger flow , determine whether initialization is needed , and whether an update is required Real-time passenger flow data cache in ;

[0203] Example of feature engineering controller The trigger time of the current micro-batch Calculate the number of calculations within the current sliding window , if we calculate the number of , then with the help of The encapsulated method performs online feature engineering, constructs real-time multi-modal features, and passes them into the pre-trained model cached in the prediction control module for real-time prediction and outputs the prediction results;

[0204] Jump to the step of real-time passenger flow access and repeat the above processing flow to form a stream processing pipeline including real-time ticket data stream access, data preprocessing, short-term passenger flow calculation, multi-mode feature construction and passenger flow prediction.

[0205] The real-time prediction performance evaluation phase is mainly used to test the real-time prediction performance of the present invention in the presence of late arrival data, as well as the data processing capability of the real-time short-term passenger flow prediction pipeline constructed by the present invention. The process is as follows:

[0206] Test the real-time prediction performance of this solution in the scenario with late data:

[0207] In order to compare the differences between the single-mode feature model and the multi-mode notification model in the real-time prediction scenario, this embodiment selects the short-term passenger flow prediction model ConvGCN, which performs best in the offline sliding prediction scenario, as the pre-training model, and implements the single-mode feature model (SPF-Model) and multi-mode feature model (MPF-Model) of ConvGCN respectively. Let the difference between the real-time passenger flow prediction accuracy and the offline passenger flow prediction accuracy be the real-time prediction accuracy loss. When the first real-time short-term passenger flow prediction is performed within one minute, the late arrival data has the greatest impact on the real-time short-term passenger flow accuracy prediction. Therefore, we selected the real-time network-level short-term inbound and outbound passenger flow accuracy, the accuracy of the real-time single-mode features, the accuracy of the real-time multi-mode features, the real-time prediction accuracy loss of the SPF-Model, and the real-time prediction accuracy loss of the MPF-Model for the first real-time short-term passenger flow prediction within one minute at different time granularities for data visualization, as shown below. Figure 4 shown.

[0208] like Figure 4 As shown in the figure, there are 5 broken lines in total. They correspond to the accuracy of real-time network-level short-term passenger flow in and out of the station, the accuracy of real-time single-mode features, the accuracy of real-time multi-mode features, the real-time prediction accuracy loss corresponding to the single-mode feature model, and the real-time prediction accuracy loss corresponding to the multi-mode feature model. It can be found that under the same late data distribution, the smaller the time granularity, the lower the accuracy of the real-time network-level short-term passenger flow, the lower the accuracy of the real-time input features of the prediction model, and the lower the accuracy of the real-time prediction. When the time granularity is 1 minute, the accuracy of the real-time network-level short-term passenger flow is the lowest at 85.612%, and the real-time prediction accuracy losses of SPF-Model and MPF-Model are also the largest, which are 1.977% and 1.504% respectively. In addition, it is not difficult to find that multi-mode features can further reduce the error of late data propagating to the prediction model compared to single-mode features, thereby alleviating the loss of real-time prediction accuracy.

[0209] In order to evaluate the effectiveness of the trigger control mechanism in balancing the accuracy and timeliness of real-time prediction, this example visualizes the performance of the MPF-Model of ConvGCN at different time granularities, with 5 real-time predictions per minute. Figure 5 shown. Figure 5 There are 10 broken lines in the figure. The 5 solid-marked broken lines represent the real-time prediction accuracy loss of the first 5 online predictions per minute at different time granularities. The 5 hollow-marked broken lines represent the change in real-time passenger flow accuracy of the first 5 online predictions per minute at different time granularities. It can be found that under different time granularities, the real-time prediction accuracy loss of the first prediction per minute is the largest, and the real-time prediction accuracy loss of the fifth prediction is 0, that is, the real-time prediction accuracy is equal to the offline prediction accuracy. Assuming that we believe that when the real-time prediction accuracy loss is less than 0.1%, the accuracy and timeliness of short-term passenger flow prediction can be balanced. When the time granularity is 1 minute, the balance point of the accuracy and timeliness of short-term passenger flow prediction can be achieved in the third prediction per minute; when the time granularity is 3 minutes, the balance point of the accuracy and timeliness of short-term passenger flow prediction can be achieved in the second prediction per minute.

[0210] Evaluate the data processing capabilities of the real-time short-term passenger flow forecasting pipeline:

[0211] We use latency and throughput as two indicators to evaluate the data processing capability of the real-time short-term passenger flow prediction pipeline. Latency refers to the time interval from receiving an event to observing the effect of event processing in the output. Since latency is generally small and sensitive to system jitter, it is usually described in percentiles. Assume that the processing latency of the stream processing application is ,but Includes the latency of the stream processing engine reading data from Kafka , Delay in real-time passenger flow calculation , the delay in writing real-time passenger flow to the database , Latency of Online Feature Engineering and online prediction latency The corresponding calculation formula is as follows:

[0212]

[0213] It is worth noting that we need to distinguish Delays from real-time passenger flow forecasts Specifically, assuming that The prediction is If the predicted service is completed at the time .

[0214] Throughput is an indicator to measure the data processing capability per unit time, indicating the number of records that can be processed per unit time in each link of continuous stream processing. Generally speaking, the lower the latency, the better, and the higher the throughput, the better, and these two indicators are not independent of each other. In extreme cases, when the load is very small, the latency of stream processing is very small, but the throughput is very low; when the load is very large, the throughput of stream processing is extremely high, but newly arrived events will be backlogged in the message queue, causing a sudden increase in the delay of stream processing. In order to better test the data processing capabilities of the real-time short-term passenger flow prediction pipeline, this embodiment selects the historical short-term passenger flow from 08:04 to 08:54 on a certain day in the test set for playback. The frequency of card swiping data generated during this time period can reach up to 14,000 records / minute. In the test scenario of pressure load, the frequency of card swiping data generated during this time period can reach up to 70,000 records / minute.

[0215] When the program is running stably, 95% of the In less than 3.426 seconds; at a load of 70,000 records / minute, 95% of Within 3.487; it can be found that even if the load is magnified by 5 times, the stream processing delay There is no sudden growth, which shows that our invention has good scalability and efficiency, and has the potential to handle larger real-time card swipe data loads.

[0216] The above data show that the real-time passenger flow prediction method for rail transit based on a streaming computing engine is far superior to existing methods in the field of real-time passenger flow prediction. It can efficiently and scalably aggregate massive card swiping data streams in real time to calculate the short-term passenger flow in and out of the station; it can provide the prediction performance of the prediction model in the sliding prediction scenario; it can efficiently implement online feature engineering, construct multi-mode features in real time and avoid a large amount of cross-node network transmission in the second streaming aggregation; it can use real-time multi-mode features to alleviate the loss of real-time prediction accuracy caused by late data to real-time prediction; and it can balance the timeliness and accuracy of real-time prediction through a trigger control mechanism. It can efficiently support real-time network-level short-term passenger flow prediction in real rail transit scenarios with complex network scenarios and late data.

[0217] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A real-time prediction method for rail transit passenger flow based on a streaming computing engine, characterized in that: Step 1: Collect all historical smart card swiping records generated by the rail transit ticket collection system to obtain a static offline card swiping data set; Step 2: Preprocess the offline card swiping data set and use the sliding window to calculate the offline network-level short-term in-and-out passenger flow according to the set time granularity and sliding interval; Step 3: Use the multi-modal feature sample construction method with multiple rolling windows to process the offline network-level short-term in-and-out passenger flow and construct offline multi-modal feature samples; Step 4: Use offline multimodal feature samples to train and evaluate the pre-trained model; Step 5: Save the pre-trained model in a serialized manner; Step 6: Deploy message middleware, persistent key-value storage service, stream computing engine, feature engineering controller, prediction control module and real-time data simulation module to provide an available stream processing environment; Step 7: Use the real-time data simulation module to simulate the rail transit system according to the parameters set during deployment to generate massive real-time card swiping data streams, and connect them to the message middleware; Step 8: Use the streaming computing engine to obtain the real-time card swiping data stream from the message middleware, perform data preprocessing, parallelly calculate the real-time network-level short-term inbound and outbound passenger flow, and write it into the persistent key-value storage service; Step 9: Input the real-time network-level short-term inbound and outbound passenger flow, and use the trigger control mechanism composed of the feature engineering controller and the prediction control module to control the process of online feature engineering and prediction, and output the prediction results; Step 10: Repeat Step 7-Step 9 to form a continuous stream processing pipeline including real-time ticket data stream access, data preprocessing, short-term passenger flow calculation, multi-mode feature construction and passenger flow prediction.

2. According to claim 1, a method for real-time prediction of rail transit passenger flow based on a streaming computing engine is characterized in that: The Step 3 is specifically as follows: make Represents any time during the operation period of the rail transit system. Indicates the time granularity of the short-term inbound and outbound passenger flow at the network level, Represents a time window The network-level short-term in-and-out passenger flow of the internal rail transit system, Indicates the number of historical short-term passenger flow records used to construct the features of a single mode, Indicates the time interval between adjacent historical short-term passenger flow records when constructing input features, Indicates 24 hours, Indicates the pattern of passenger flow, and They represent real-time mode, daily mode and weekly mode respectively, and the network-level short-term in-and-out passenger flow and The relationship is as follows: ； in, , , The network-level short-term inbound and outbound passenger flows represent real-time mode, daily mode, and weekly mode respectively; Specifying a Mode Corresponding passenger flow characteristics It is expressed as follows: ； make Indicates prediction Sample of multi-modal characteristics of short-term inbound and outbound passenger flows at the network level: ； in, , , They represent the passenger flow characteristics of real-time mode, daily mode and weekly mode respectively; Let the total operating time of the rail transit system be , the sliding window size for calculating short-term passenger flow is , the sliding interval is ,have , , the size is , the interval is The sliding time window will Divide into There are time slices with overlapping segments; Indicates Multimodal feature samples of time slices, ,The sample set obtained by the multi-mode feature sample construction method based on multi-rolling windows is expressed as follows: ； Since the rail transit system is suspended at night, It is impossible to construct a complete multi-mode feature within the time range of the previous day, so the previous day's rail transit system shutdown period The historical passenger flow records are used to complete the multi-modal features. The construction formula of is defined as follows: ； in, is the total daily downtime of the rail transit system, express Time The time the rail transit system has been in operation.

3. The method for real-time prediction of rail transit passenger flow based on a streaming computing engine according to claim 1, characterized in that: The Step 7 is specifically as follows: Step 7.1: Set the date and start and end time of the simulated historical card swiping data, load expansion multiple, communication URL of the message middleware and multiple parameters of the delay simulation function, and start the real-time data simulation module; Step 7.2: Read the card swiping data within the specified date and time range; Step 7.3: Expand the total amount of data in the data set according to the load expansion multiple, take the "card swiping time" column to traverse each record, and add the randomly generated delay function of the simulation , generate a new column "write time"; Step 7.4: Based on the new column "write time", index the sub-dataset of the card swiping data loaded in Step 7.2 second by second, and write it to the message middleware Kafka.

4. The method for real-time prediction of rail transit passenger flow based on a streaming computing engine according to claim 3 is characterized in that: The simulation delay function is specifically: Use the delay simulation function to simulate the total transmission delay of the card swiping data from the gate to the site proxy server and then forwarded to the data center , the delay simulation function formula is as follows: ； Among them, U(a,b) represents a uniform distribution function with a lower bound and b upper bound. Indicates The probability of generating A random number uniformly distributed between .

5. The method for real-time prediction of rail transit passenger flow based on a streaming computing engine according to claim 1, characterized in that: The Step 8 is specifically as follows: Step 8.1: The stream computing engine obtains the real-time card swiping data stream from the message middleware at a certain time interval; Step 8.2: Preprocess the real-time card swiping data stream and group it based on the sliding time window; Step 8.3: Calculate the real-time network-level short-term inbound and outbound passenger flow for the data in each group; Step 8.4: Write the calculated real-time network-level short-term inbound and outbound passenger flow into the persistent key-value storage service.

6. A method for real-time prediction of rail transit passenger flow based on a streaming computing engine according to claim 5, characterized in that: The Step 8.3 is specifically as follows: Using parallel algorithms Calculate the short-term inbound and outbound passenger flow at the network level, the parallel algorithm The input is the rail transit system in the time window Card swiping datasets generated by sites , the card swiping dataset have partitions distributed on different nodes, using Represents the partition number, the parallel algorithm The specific implementation steps are: Step8.3.1: Initialize a length of , Long integer array with initial value 0 , used to store intermediate calculation results, elements to store the incoming passenger flow. Elements store outbound passenger flow; Step 8.3.2: Parallel traverse all card swiping data records in each partition, calculate and update the long integer array according to the entry and exit identification and site number of the card swiping record ; Step 8.3.3: Wait for all partitions to traverse the local partition data set, and then collect each partition's data across nodes through the network. To a list storing long integer arrays middle; Step 8.3.4: Merge the list of long integer arrays in pairs by adding elements in place An array of long integers for each partition in , and finally the network-level short-term inbound and outbound passenger flow within the time window is obtained.

7. The method for real-time prediction of rail transit passenger flow based on a streaming computing engine according to claim 1, characterized in that: The Step 9 is specifically as follows: The trigger control mechanism specifies the watermark size based on the late data distribution and stream processing trigger interval , by setting , , , to achieve auxiliary process control, at each sliding interval Internally, stream processing applications trigger Calculation, corresponding to micro-batch, but only the first The calculation status of the previous window will be updated by the next calculation, so only the previous window The calculation is performed online feature engineering and prediction. The specific implementation steps are: Step 9.1: Get the trigger time of the current micro-batch ; Step 9.2: Example of combining feature engineering controller and real-time network-level short-term inbound and outbound passenger flow , determine whether initialization is needed , and whether an update is required Real-time passenger flow data cache in ; Step 9.3: Example based on feature engineering controller The trigger time of the current micro-batch Calculate the number of calculations within the current sliding window , if we calculate the number of , then with the help of The encapsulated method performs online feature engineering, constructs real-time multi-modal features, and passes them into the pre-trained model cached in the prediction control module for real-time prediction and outputs the prediction results; Step 9.4: Perform loop control and jump to Step 7 to start a new round of real-time short-term passenger flow prediction.

8. A method for real-time prediction of rail transit passenger flow based on a streaming computing engine according to claim 7, characterized in that: The feature engineering controller is specifically implemented as follows: Design a custom type As a feature engineering controller, when deploying the feature engineering control module, create a feature engineering controller on the application daemon Driver Examples Used for subsequent control processes, specifically: Feature Engineering Controller Encapsulates the real-time passenger flow data cache of TreeMap[Long,Array[Long]] type and historical passenger flow data cache It is used to cache historical and real-time network-level short-term inbound and outbound passenger flow data, where the TreeMap is a key-value pair set sorted by key, where the key is a long integer Long short-term passenger flow timestamp, and the value is a long integer array Array[Long] storing network-level short-term inbound and outbound passenger flow; in addition, the feature engineering controller It also encapsulates other member variables that assist in the online feature engineering process control, such as Whether to initialize the current date , and the current event time storing the latest short-term passenger flow event time ; Feature Engineering Controller Four core functions are encapsulated to abstract and standardize the process of online feature engineering, specifically: Initialization function : Based on the timestamp passed in , that is, the latest network-level short-term inbound and outbound passenger flow event time, update the current date Auxiliary variables such as , calculate and initialize the real-time passenger flow cache and historical passenger flow cache The time range of the required network-level short-term inbound and outbound passenger flow is retrieved and loaded from the persistent key-value storage service to initialize two passenger flow caches; Real-time passenger flow data cache update function : Incoming real-time network-level short-term inbound and outbound passenger flow, updating the real-time passenger flow cache If a new short-term passenger flow record is added, the real-time passenger flow cache is removed. The oldest record in Real-time feature time series calculation function : Pass in the current event time , the time index sequence of network-level short-term inbound and outbound passenger flow records required to calculate real-time pattern characteristics ,make for ,but correspond ; Multi-mode feature constructor : Incoming ,Right now ,calculate and and use From real-time customer flow cache Index data records to construct real-time mode passenger flow characteristics ,use and From the historical passenger flow cache Index data records to construct daily and weekly passenger flow characteristics and .

Images