A visual system for passenger flow analysis and prediction

Abstract

The application belongs to the technical field of rail transit data processing, and discloses a passenger flow analysis and prediction visualization system, which comprises a data access module, an offline data access module including corresponding stations and lines of rail transit; a data simulation module, which simulates real-time data based on offline data; a storage management module, which pre-processes offline data, stores the real-time data into Hbase after flow processing, and stores the data in Hbase into Redis cache and MySQL database according to historical access records; an operation processing module, which performs dynamic OD matrix estimation, passenger flow distribution calculation and passenger flow trend prediction based on the data in Redis cache and MySQL database, and obtains passenger flow calculation results; and a Web application visualization module, which displays the passenger flow calculation results in a visual chart. The application can support real-time processing requirements of massive rail transit data.

Description

Technical Field

[0001] This application relates to the technical field of rail transit data processing, and in particular to a visualization system for passenger flow analysis and prediction. Background Technology

[0002] Most current rail transit systems typically use traditional methods for data storage and processing, storing data in Oracle databases. The system can only display basic data such as passenger entry and exit information. Current rail transit systems handle frequent daily data writes, and their simplistic data storage design cannot support the real-time processing needs of massive amounts of data. Furthermore, they lack the ability to perform passenger flow analysis / simulation and prediction at the line and station levels to obtain information on each passenger's travel journey. This results in the industry's inability to comprehensively grasp the distribution and dynamics of passenger flow within the network, leading to delayed perception of congestion at stations, lines, and carriages. High-risk passenger flows under large passenger volumes cannot be accurately managed and controlled. The early assessment and prediction of sudden passenger flow events, and the implementation of various contingency plans, are reactive and unable to timely and proactively mitigate potential sudden passenger flow events, ensuring that normal passenger transport is not affected. Summary of the Invention

[0003] This application provides a visualization system for passenger flow analysis and prediction, which supports the real-time processing needs of massive amounts of data in rail transit and analyzes and predicts passenger flow at both the line and station levels.

[0004] This application provides a visualization system for passenger flow analysis and prediction, the system comprising:

[0005] The data access module is used to access source data, which includes offline data of corresponding rail transit stations and lines.

[0006] The data simulation module is used to simulate and obtain real-time data based on the offline data;

[0007] The storage management module is used to preprocess offline data using Spark and stream process real-time data using Flink before storing it in HBase. Based on historical access records, the data in HBase is stored in Redis cache and MySQL database respectively.

[0008] The computation and processing module is used to perform dynamic OD matrix estimation, passenger flow allocation calculation, and passenger flow trend prediction based on data from Redis cache and MySQL database, and obtain passenger flow calculation results.

[0009] The web application visualization module is used to display the passenger flow calculation results in visual charts.

[0010] Furthermore, the storage management module includes an offline data storage submodule, specifically used for:

[0011] Store offline data in HDFS;

[0012] Preprocessing offline data using Spark;

[0013] The preprocessed offline data is stored in HBase;

[0014] Based on business requirements, preprocessed offline data in HBase is processed in batches and then stored in a MySQL database.

[0015] Furthermore, the storage management module includes a real-time data storage submodule, specifically used for:

[0016] The simulated real-time data is sent to the Kafka message queue;

[0017] Real-time data stream processing computation is performed based on Flink;

[0018] Store the computation results of stream processing in HBase;

[0019] Accessed computation results are stored in a Redis cache, while unaccessed computation results are stored in a MySQL database.

[0020] Furthermore, the storage management module employs master-slave database server synchronization, where the master database is responsible for write operations and the slave database is responsible for read operations.

[0021] Furthermore, the storage management module uses the MyISAM storage engine.

[0022] Furthermore, the calculation and processing module includes a passenger flow trend prediction submodule, used for:

[0023] Import the Spark machine learning ML library and use the decision tree algorithm to build a passenger flow prediction model;

[0024] The passenger flow prediction model is used to predict passenger flow at stations.

[0025] Furthermore, the data simulation module is specifically used for:

[0026] Offline data is acquired in the first preset period, and user data for each site and each line in the offline data is analyzed;

[0027] Based on the user age data in offline data, a normal distribution algorithm is used to simulate the data and obtain the user data list corresponding to the real-time data.

[0028] The simulated user data list is filtered based on the LRU algorithm.

[0029] Furthermore, the application backend corresponding to the Web application visualization module is based on the Spring Boot framework, and the application frontend is based on the Vue.js framework.

[0030] Furthermore, the web application visualization module includes Vue components for:

[0031] The system requests data from the backend once every second preset period and determines whether the data has been updated.

[0032] If the data is updated, the corresponding visualization chart will be re-rendered.

[0033] Furthermore, the Web application visualization module is also specifically used for:

[0034] After a user successfully logs in to the front-end of the web application visualization module, the token returned by the back-end is stored in sessionStorage.

[0035] When a user requests a data URL on the visualization page, the token in session storage is retrieved, and the request data URL containing the token header is returned to the backend.

[0036] Display the data returned by the backend on the visualization page.

[0037] In summary, compared with the prior art, the beneficial effects of the technical solution provided in this application include at least the following:

[0038] This application provides a visualization system for passenger flow analysis and prediction. This application can classify, store, and process massive amounts of rail transit data, thereby supporting the real-time processing needs of massive amounts of rail transit data. It analyzes and predicts passenger flow at the line level and station level, and displays the data in a visual manner on the system application page, clearly and effectively conveying information to improve the efficiency and capability of rail transit decision-making in network planning and design, construction management, and network operation management. Attached Figure Description

[0039] Figure 1 A schematic diagram of the functional modules of a visualization system for passenger flow analysis and prediction provided in one embodiment of this application;

[0040] Figure 2 This is a schematic diagram of the web application login process corresponding to a web application visualization module provided in one embodiment of this application.

[0041] Figure 3This is a schematic diagram of the visualization page corresponding to the Web application visualization module provided in one embodiment of this application. Detailed Implementation

[0042] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0043] Please see Figure 1 This application provides a visualization system for passenger flow analysis and prediction, which specifically includes:

[0044] The data access module 10 is used to access source data, which includes offline data of corresponding rail transit stations and lines.

[0045] Data simulation module 20 is used to simulate real-time data based on the offline data;

[0046] The storage management module 30 is used to preprocess offline data using Spark and stream process real-time data using Flink before storing it in HBase. Based on historical access records, the data in HBase is stored in Redis cache and MySQL database respectively.

[0047] The computation and processing module 40 is used to perform dynamic OD matrix estimation, passenger flow allocation calculation, and passenger flow trend prediction based on data from Redis cache and MySQL database, and obtain passenger flow calculation results.

[0048] Web application visualization module 50 is used to visualize and display the passenger flow calculation results in charts.

[0049] In this embodiment, a computer cluster based on Hadoop, Spark, Redis, HBase, and MySQL was built. This computer cluster architecture enables efficient processing and analysis of massive amounts of rail transit data, thereby fulfilling the visualization requirements for rail transit passenger flow analysis and prediction. The passenger flow analysis and prediction visualization system includes a data access module, a data simulation module, a storage management module, a computation processing module, and a web application visualization module. The data access module contains the computer system to which the data source belongs, used to access the source data. The source data includes offline data for corresponding rail transit stations and lines, such as station.csv, trips.csv, Users.csv, and Workdays.csv data files. This source data represents the raw, massive amount of rail transit data. The data simulation module simulates real-time data by analyzing the patterns in the above source data, such as using Spark SQL to perform raw data statistics, like the male-to-female ratio of user data and the proportion of user data at each station, to simulate rail transit user data.

[0050] Because existing rail transit systems rely on a single data storage method, they cannot support real-time computation of massive datasets. Therefore, the storage management module in this embodiment employs different storage management methods for massive data. For offline data, after preprocessing using Spark, it is stored in HBase. For real-time data, after stream processing using Flink, it is stored in HBase. Furthermore, to address the issue of slowed data access due to high data volume, historical access records are used to cache accessed data in Redis, and unaccessed data in the MySQL database, thereby alleviating the pressure on the MySQL database.

[0051] In the computational processing module, a preliminary study of a specific city's rail transit system is conducted. Based on fundamental static and dynamic data such as its network, stations, and operational plans, statistical characteristics at different time granularities are analyzed, resulting in a data processing algorithm model for dynamic OD matrix estimation, passenger flow analysis, and passenger flow prediction. When performing passenger flow prediction analysis on the rail transit system, data from the Redis cache and MySQL database can be retrieved from the storage management module. Based on the data processing algorithm model for dynamic OD matrix estimation, passenger flow analysis, and passenger flow prediction, dynamic OD matrix estimation, passenger flow allocation calculation, and passenger flow trend prediction are performed to obtain the passenger flow calculation results. After obtaining the passenger flow calculation results, these results are visualized using a web application visualization module.

[0052] The passenger flow analysis and prediction visualization system in this embodiment can classify, store, and process massive amounts of rail transit data, thereby supporting the real-time processing needs of massive amounts of rail transit data. It can analyze and predict passenger flow at the line and station levels and display the data in a visual manner on the system application page, clearly and effectively conveying information to improve the efficiency and capability of rail transit decision-making in network planning and design, construction management, and network operation management.

[0053] Furthermore, in one embodiment, the storage management module 30 includes an offline data storage submodule, specifically used for:

[0054] Store offline data in HDFS;

[0055] Preprocessing offline data using Spark;

[0056] The preprocessed offline data is stored in HBase;

[0057] Based on business requirements, preprocessed offline data in HBase is processed in batches and then stored in a MySQL database.

[0058] In this embodiment, the storage management module stores and processes offline data and simulated real-time data separately. This allows for the simultaneous storage of massive amounts of data and real-time computational processing to perform passenger flow analysis and prediction. Specifically, the storage management module includes an offline data storage submodule. This submodule manages the storage of offline data, first storing it in HDFS, then preprocessing it using Spark, storing the processed data in HBase, performing batch processing on the HBase data using Spark SQL, and finally saving the processed data to a MySQL database.

[0059] Furthermore, in one embodiment, the storage management module 30 includes a real-time data storage submodule, specifically used for:

[0060] The simulated real-time data is sent to the Kafka message queue;

[0061] Perform stream processing computations on real-time data in the Kafka message queue based on Flink;

[0062] Store the computation results of stream processing in HBase;

[0063] Accessed computation results are stored in a Redis cache, while unaccessed computation results are stored in a MySQL database.

[0064] In this embodiment, the storage management module also includes a real-time data storage submodule. In this submodule, simulated real-time data is sent to a Kafka message queue. On the other end, Flink is used to retrieve topics from the Kafka message queue in real time, and Flink SQL is used for stream processing. Finally, the processed data is saved to HBase, while accessed data is stored in a Redis cache with an expiration time set to ensure Redis's data capacity. Unaccessed computation results are stored in a MySQL database. Compared to traditional messaging systems, Kafka is a highly stable message queue with high throughput, low latency, high concurrency, support for thousands of clients reading and writing simultaneously, high fault tolerance, high reliability, and scalability. Kafka can provide good ordering and load balancing even with multiple consumer groups operating concurrently. Each partition is distributed to only one consumer group, meaning each partition is consumed by only one consumer in that group, allowing for sequential consumption of messages from that partition. Because there are multiple partitions, load balancing and persistent storage can still be achieved across multiple consumer groups, providing massive data storage capabilities. Flink has a clear advantage as a stream processing engine, supporting both batch and stream processing, as well as EventTime and out-of-order events. Flink supports stateful computation on bounded and unbounded data streams, on an event-by-event basis, and supports SQL, State, and Watermark. It supports "exactly once," guaranteeing that events are delivered only once, thus improving data accuracy. Compared to Storm, it offers higher throughput and lower latency; compared to Spark Streaming, Flink provides true real-time computation with relatively fewer computational resources.

[0065] Furthermore, in one embodiment, the storage management module 30 employs master-slave database server synchronization, wherein the master database is responsible for write operations and the slave database is responsible for read operations.

[0066] In this embodiment, the storage management module selects one server as the master database and other servers as slave databases to synchronize the master and slave databases. This approach results in significant data redundancy and wastes a large amount of storage space. Therefore, this embodiment separates the read and write operations of the MySQL database, moving them away from the same database server. The master database handles only write operations, while the slave databases handle only read operations, acting as backup nodes. Amoeba for MySQL middleware replaces MySQL-Proxy to achieve read / write separation and load balancing, thereby improving the stability of the database corresponding to the storage management module.

[0067] Furthermore, in one embodiment, the storage management module 30 employs the MyISAM storage engine.

[0068] In this embodiment, considering the need to achieve real-time predictive analysis of massive data while storing it, high requirements are placed on the insertion, querying, and synchronization of data in the storage management module. The storage management module uses the MyISAM storage engine, responsible for data storage and retrieval in MySQL, enabling high-speed data querying and insertion. In the MyISAM storage engine, data types are selected according to the principle of small and simple; smaller data types are generally faster, occupy less disk space and memory, and require fewer CPU cycles during processing. High-performance indexes are created, as the time to add an index is much longer than the initial time required to add an index. Table partitioning reduces the database load and improves database efficiency, i.e., improves the efficiency of table CRUD operations. The results of SELECT statements are stored in a cache area for reuse. For concurrent SQL queries, multi-table operations (subqueries, joins) are minimized (avoided), and complex SQL queries are split into multiple executions. If the query is atomic (small), the utilization of the query cache is increased. Conditional filtering methods are used to pinpoint the retrieved data to the required data as precisely as possible.

[0069] Furthermore, in one embodiment, the calculation and processing module 40 includes a passenger flow trend prediction submodule, used for:

[0070] Import the Spark machine learning ML library and use the decision tree algorithm to build a passenger flow prediction model;

[0071] The passenger flow prediction model is used to predict passenger flow at stations.

[0072] In this embodiment, by comparing and analyzing the comprehensive scores of models built using random forest, linear regression, and decision tree algorithms in terms of data training and prediction accuracy, the machine learning model with the highest prediction accuracy is selected as the prediction model. The prediction accuracy is determined based on the root mean square error (RMSE) of the three algorithm models predicting test data; a smaller RMSE indicates higher prediction accuracy. By comparing the RMSE, the decision tree algorithm is chosen to build the passenger flow prediction model. The corresponding processing module includes a passenger flow trend prediction submodule, which imports the Spark machine learning ml library. Spark.ml provides machine learning APIs built on DataFrames. Calling the decision tree algorithm API to build the passenger flow prediction model allows for the prediction of passenger flow at stations. Specifically, building the passenger flow prediction model by calling the decision tree algorithm API includes: initializing Spark, recording data, vectorizing and standardizing the data, building a dataset and a test set, building a decision tree, putting the built decision tree into the pipeline for model training, saving the trained model, using test data for prediction, and evaluating it using the RMSE to obtain a passenger flow prediction model that meets the requirements.

[0073] Furthermore, in one embodiment, the data simulation module 20 is specifically used for:

[0074] Offline data is acquired in the first preset period, and user data for each site and each line in the offline data is analyzed;

[0075] Based on the user age data in offline data, a normal distribution algorithm is used to simulate the data and obtain the user data list corresponding to the real-time data.

[0076] The simulated user data list is filtered based on the LRU algorithm.

[0077] In this embodiment, when simulating real-time data based on offline data, user data from each station and line in the offline data is analyzed periodically. For example, if the offline data shows that the user data is mostly distributed normally between the ages of 16 and 30, then a normal distribution algorithm can be used to simulate the data based on the user age data in the offline data to obtain the user data list corresponding to the real-time data, ensuring that the simulated real-time data conforms to a normal distribution. Simultaneously, as the simulated real-time data grows, the user list corresponding to the real-time data will increase. During simulation, the LRU algorithm is used to quickly retrieve frequently accessed data, while infrequently accessed data is discarded when the capacity exceeds the limit, thus filtering the simulated user data list and preventing memory shortages. Furthermore, in cases with a particularly large number of users, there is a possibility that a user may have multiple traffic records at the same time. To ensure that a user does not have multiple traffic records at the same time, a user is randomly selected from the memory of the simulated real-time data, and the departure time of the traffic record is set to the current time. Subsequent preprocessing of the simulated data can remove dirty data from multiple traffic records of the same user at the same time.

[0078] Furthermore, in one embodiment, the application backend corresponding to the Web application visualization module 50 is based on the Spring Boot framework, and the application frontend is based on the Vue.js framework.

[0079] In this embodiment, the web application layer provides real-time passenger flow prediction and monitoring. The backend is based on the Spring Boot framework, and the frontend is based on the Vue.js framework, achieving front-end and back-end separation. Visualization tools and appropriate graphs are selected for data visualization. Finally, the application is deployed on a Docker container for online access, providing decision support for the operation and management of rail transit. Vue.js allows for component-based development, significantly reducing the amount of code and making it easier to understand. Its most prominent advantage is its ability to perform two-way data binding. The interface built using Vue.js is responsive, ensuring the webpage displays a very attractive appearance on various devices.

[0080] Furthermore, in one embodiment, the Web application visualization module 50 includes a Vue component, used for:

[0081] The system requests data from the backend once every second preset period and determines whether the data has been updated.

[0082] If the data is updated, the corresponding visualization chart will be re-rendered.

[0083] In this embodiment, the web application visualization module includes a Vue component. Vue component listeners can monitor data changes in real time, and the chart will be re-rendered as soon as the data changes. The front-end sets data to the chart every second, and child components only receive data passed from the parent component and modify it accordingly, without performing complex operations internally.

[0084] Furthermore, in one embodiment, the Web application visualization module 50 is also specifically used for:

[0085] After a user successfully logs in to the front-end of the web application visualization module, the token returned by the back-end is stored in sessionStorage.

[0086] When a user requests a data URL on the visualization page, the token in session storage is retrieved, and the request data URL containing the token header is returned to the backend.

[0087] Display the data returned by the backend on the visualization page.

[0088] In this embodiment, see Figure 2 The web application visualization module's front-end receives a token from the backend after a user successfully logs in. This token needs to be stored in session storage. When data needs to be retrieved, this token must be included in the request header to access the backend interface, ensuring data access security. Specifically, when a user requests a data URL on the visualization page, the token is retrieved from session storage, and a request data URL containing the token header is returned to the backend. The data returned by the backend is then displayed on the visualization page. For example, after a user successfully logs in and enters the time, site name, and time attribute, the traffic prediction results are displayed on the visualization page. See [link to relevant documentation]. Figure 3 The data includes overall monthly passenger flow fluctuation analysis data, weekday and weekend passenger flow analysis data, single station point-to-point / entry passenger flow analysis data, user age structure analysis data, morning and evening peak passenger flow station distribution analysis data, station OD passenger flow analysis data, line cross-section (by station) flow analysis data, user gender ratio analysis task description, and the visualization graphics used, as shown in Table 1.

[0089] Table 1

[0090]

[0091]

[0092] Visual page elements are configured through a configuration platform. The page is composed of various components. The overall elements of the page template are obtained. Currently, the main technology used is to query configuration information based on the request URL, assemble and output the page template, and then render and display the page elements from the client page. Data information is obtained by querying the interface according to the request frequency.

[0093] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0094] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A visualization system for passenger flow analysis and prediction, characterized by, The system includes: The data access module is used to access source data, which includes offline data of corresponding rail transit stations and lines. The data simulation module is used to simulate real-time data based on the offline data. Specifically, it is used to acquire offline data at a first preset period, analyze the user data of each station and each line in the offline data, simulate data based on the user age data in the offline data using a normal distribution algorithm, obtain a user data list corresponding to the real-time data, and filter the simulated user data list based on the LRU algorithm. The storage management module is used to preprocess offline data using Spark and stream process real-time data using Flink before storing it in HBase. Based on historical access records, it stores data from HBase in Redis cache and MySQL database respectively. This includes an offline data storage submodule, specifically used to store offline data in HDFS; preprocess offline data using Spark and store the preprocessed offline data in HBase; and batch process the preprocessed offline data in HBase according to business needs before storing it in a MySQL database. The storage management module includes a real-time data storage submodule, specifically used for: sending simulated real-time data to a Kafka message queue; performing stream processing computation on the real-time data based on Flink; storing the computation results of the stream processing in HBase; storing accessed computation result data in a Redis cache and storing unaccessed computation result data in a MySQL database; The computation and processing module is used to perform dynamic OD matrix estimation, passenger flow allocation calculation, and passenger flow trend prediction based on data from Redis cache and MySQL database, and obtain passenger flow calculation results. It includes a passenger flow trend prediction submodule, which is used to import Spark machine learning ML library, use decision tree algorithm to build passenger flow prediction model, and predict passenger flow at passenger flow stations based on the passenger flow prediction model. The web application visualization module is used to visualize and display the passenger flow calculation results in charts. When the user of the application front-end corresponding to the web application visualization module logs in successfully, the token returned by the back-end is stored in sessionStorage. When a user requests a data URL on the visualization page, the token in sessionStorage is retrieved, and a request data URL containing the token header is returned to the back-end. The data returned by the back-end is then displayed on the visualization page.

2. The visualization system for passenger flow analysis and prediction of claim 1, wherein, The storage management module uses a master-slave database server synchronization, where the master database is responsible for write operations and the slave database is responsible for read operations.

3. The visualization system for passenger flow analysis and prediction of claim 1, wherein, The storage management module uses the MyISAM storage engine.

4. The visualization system for passenger flow analysis and prediction according to claim 1, characterized in that, The web application visualization module corresponds to an application backend based on the Spring Boot framework and an application frontend based on the Vue.js framework.

5. The visualization system for passenger flow analysis and prediction according to claim 4, characterized in that, The web application visualization module includes Vue components for: The system requests data from the backend once every second preset period and determines whether the data has been updated. If the data is updated, the corresponding visualization chart will be re-rendered.

Images