Real-time warehousing processing method and device for rail transit data and electronic equipment

Abstract

The invention provides a rail transit data real-time storage processing method and device and electronic equipment, and the method comprises the steps: collecting line side data generated by a rail vehicle in real time, and storing the line side data to a line side distributed event flow platform, the line side data comprises operation data information and operation alarm information of the railway vehicle; based on a one-way transmission device, transmitting the line side data stored in the line side distributed event flow platform to a line network side distributed event flow platform in a one-way manner; performing streaming processing on line side data in the line network side distributed event stream platform to obtain processed line side data; and storing the processed line side data in a database. Therefore, the defects of insufficient real-time performance and low safety in rail transit data processing are overcome, and the timeliness and the safety of rail transit data storage processing are improved.

Description

Technical Field

[0001] This invention relates to the field of rail transit technology, and in particular to a method, apparatus, and electronic equipment for real-time data entry and processing of rail transit data. Background Technology

[0002] In the rail transit operation and management system, efficient data processing and accurate data aggregation are the core support for ensuring line safety and improving operational efficiency.

[0003] According to relevant technologies, the current industry often uses batch processing and simple message queue transmission modes to process and summarize data from various lines. However, batch processing, which performs data operations on a timed or batch basis, leads to data processing delays. Simple message queue transmission modes often lack a one-way isolation mechanism for data transmission; if security issues occur on the line-side network, it may negatively impact the stability of the line-side data acquisition system. Summary of the Invention

[0004] This invention provides a method, apparatus, and electronic device for real-time data entry and processing of rail transit data, which addresses the shortcomings of insufficient real-time performance and low security in existing rail transit data processing technologies, thereby improving the timeliness and security of rail transit data entry and processing.

[0005] This invention provides a real-time data storage and processing method for rail transit data. The method includes: real-time acquisition of track-side data generated by rail vehicles, and storing the track-side data in a track-side distributed event stream platform, wherein the track-side data includes the operation data information and operation alarm information of the rail vehicles; unidirectionally transmitting the track-side data stored in the track-side distributed event stream platform to a network-side distributed event stream platform based on a one-way transmission device; performing streaming processing on the track-side data in the network-side distributed event stream platform to obtain processed track-side data; and storing the processed track-side data in a database.

[0006] According to a real-time data entry processing method for rail transit provided by the present invention, before storing the processed line-side data in the database, the method further includes: classifying the processed line-side data according to the data type of the processed line-side data to obtain classified line-side data, wherein the classified line-side data includes at least a first classified line-side data with a structured data type and a second classified line-side data with a time-series data type; storing the processed line-side data in the database includes: storing the first classified line-side data with a structured data type in a structured database; and storing the second classified line-side data with a time-series data type in a time-series database.

[0007] According to a real-time data entry and processing method for rail transit provided by the present invention, before performing streaming processing on the line-side data in the network-side distributed event stream platform, the method further includes: performing data verification on the line-side data in the network-side distributed event stream platform to obtain line-side data that has passed verification, wherein the data verification includes any one or more of data validity verification and data integrity verification; the streaming processing on the line-side data in the network-side distributed event stream platform includes: performing streaming processing on the line-side data that has passed verification in the network-side distributed event stream platform.

[0008] According to a real-time data storage processing method for rail transit provided by the present invention, before storing the line-side data into a line-side distributed event stream platform, the method further includes: obtaining current time information; storing the line-side data into the line-side distributed event stream platform includes: storing the line-side data into the line-side distributed event stream platform when the current time information is detected to meet a preset time requirement.

[0009] According to a real-time data storage processing method for rail transit provided by the present invention, before storing the line-side data into a line-side distributed event stream platform, the method further includes: real-time monitoring of the operating status of preset components of the rail vehicle; storing the line-side data into the line-side distributed event stream platform includes: storing the line-side data into the line-side distributed event stream platform when an operating alarm information is detected from a preset component of the rail vehicle.

[0010] According to a real-time data entry and processing method for rail transit provided by the present invention, the step of performing streaming processing on the line-side data in the network-side distributed event stream platform to obtain processed line-side data includes: performing any one or more of the following processing on the line-side data in the network-side distributed event stream platform based on a distributed streaming processing engine: data format standardization, data deduplication, and data filtering, to obtain processed line-side data.

[0011] According to a real-time data entry processing method for rail transit provided by the present invention, before performing data format standardization, data deduplication, and data filtering on the line-side data in the network-side distributed event stream platform based on a distributed streaming processing engine to obtain processed line-side data, the method further includes: obtaining the data volume of the line-side data in the network-side distributed event stream platform; determining the processing rate of the distributed streaming processing engine based on the data volume; the step of performing data format standardization, data deduplication, and data filtering on the line-side data in the network-side distributed event stream platform based on a distributed streaming processing engine to obtain processed line-side data includes: performing data format standardization, data deduplication, and data filtering on the line-side data in the network-side distributed event stream platform according to the processing rate based on a distributed streaming processing engine to obtain processed line-side data.

[0012] This invention also provides a real-time data storage and processing device for rail transit. The device includes: a data acquisition module for real-time acquisition of track-side data generated by rail vehicles and storing the track-side data in a track-side distributed event stream platform, wherein the track-side data includes the operation data information and operation alarm information of the rail vehicles; a transmission module for unidirectionally transmitting the track-side data stored in the track-side distributed event stream platform to a network-side distributed event stream platform based on a unidirectional transmission device; a processing module for streaming the track-side data in the network-side distributed event stream platform to obtain processed track-side data; and a storage module for storing the processed track-side data in a database.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the real-time data entry processing method for rail transit data as described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the real-time data entry processing method for rail transit data as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the real-time data entry processing method for rail transit data as described above.

[0016] This invention provides a real-time data entry and processing method, apparatus, and electronic device for rail transit. The method includes: real-time acquisition of track-side data generated by rail vehicles and storing the track-side data in a track-side distributed event stream platform, wherein the track-side data includes the operation data and alarm information of the rail vehicles; unidirectional transmission of the track-side data stored in the track-side distributed event stream platform to a network-side distributed event stream platform based on a one-way transmission device; streaming processing of the track-side data in the network-side distributed event stream platform to obtain processed track-side data; and storing the processed track-side data in a database. This solves the deficiencies of insufficient real-time performance and low security in rail transit data processing, and improves the timeliness and security of rail transit data entry and processing. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the real-time data entry and processing method for rail transit provided by the present invention.

[0019] Figure 2 This is a schematic diagram of the process for storing the processed line-side data in a database, provided by the present invention.

[0020] Figure 3 This invention provides a flowchart illustrating the process of processing line-side data by performing one or more of the following steps on the line-side data in the distributed event stream platform: data format standardization, data deduplication, and data filtering, based on a distributed streaming engine, to obtain the processed line-side data.

[0021] Figure 4 This is a schematic diagram of the structure of the real-time data entry and processing device for rail transit provided by the present invention.

[0022] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0024] The real-time data entry and processing method for rail transit provided by this invention is applied to a rail transit data processing system that includes both the line-side and network-side aspects. The line-side can refer to the local data environment of a rail transit line. The network-side can refer to the central data aggregation and processing center of the entire rail transit network.

[0025] Figure 1 This is a flowchart illustrating the real-time data entry and processing method for rail transit provided by the present invention.

[0026] The following will combine Figure 1 The process of real-time data entry and processing method for rail transit provided by the present invention is described.

[0027] In an exemplary embodiment of the present invention, combined with Figure 1 As can be seen, the real-time data entry and processing method for rail transit can include steps 110 to 140, which will be described in detail below.

[0028] In step 110, track-side data generated by the rail vehicles is collected in real time and stored in the track-side distributed event stream platform. The track-side data includes the operation data and alarm information of the rail vehicles.

[0029] In one embodiment, a data acquisition server can be deployed on the track side to receive onboard data periodically transmitted by the running rail vehicles in real time. This data constitutes track-side data. Track-side data can include operational data and operational alarm information. Operational data may refer to vehicle real-time speed, current position, traction / braking status, door status, and operating parameters of various subsystems (such as air conditioning and lighting); operational alarm information may refer to alarm signals such as equipment fault alarms, overspeed warnings, and door malfunctions.

[0030] In another embodiment, the distributed event streaming platform can be a Kafka cluster. For ease of explanation, a Kafka cluster will be used as an example of a distributed event streaming platform below. The acquisition server can format and encapsulate the received data from the line side according to topics, generate a data stream, and push it in real time to the line-side Kafka cluster deployed on the line side. This Kafka cluster, as a high-throughput, distributed message queue on the line side, is responsible for buffering and temporarily storing massive amounts of real-time data streams.

[0031] In step 120, based on the unidirectional transmission device, the line-side data stored in the line-side distributed event stream platform is unidirectionally transmitted to the network-side distributed event stream platform.

[0032] In another embodiment, to meet network security isolation requirements, a unidirectional transmission device can be deployed between the line side and the network side, such as a physical unidirectional optical gate or a protocol-based unidirectional network gate. This device only allows data to flow unidirectionally from the higher-security line side to the lower-security network side, prohibiting any reverse communication. During application, the line side can periodically or continuously consume data from the line-side Kafka cluster and send it through the sender of the unidirectional transmission device. After passing through the unidirectional transmission device, the data is pushed to the corresponding topic in the network-side Kafka cluster. This achieves secure and irreversible convergence of data from the line side to the network side.

[0033] It should be noted that the Kafka cluster on the wired network side receives data transmitted from the unidirectional optical gate, which serves as the data source for the Flink streaming engine and also acts as a data buffer to prevent excessive processing pressure on Flink due to a surge in data volume.

[0034] The one-way optical shutter achieves physical isolation and unidirectional transmission of data between the line side and the network side, completely blocking reverse data flow and preventing network security issues from affecting the line side system. The built-in data verification function verifies data integrity and legitimacy in real time, automatically discarding abnormal data and logging it to ensure the quality and security of transmitted data. Furthermore, the one-way optical shutter supports high-bandwidth transmission, meeting the needs of simultaneous data transmission across multiple lines and avoiding data backlog caused by insufficient bandwidth. In step 130, the line-side data in the distributed event stream platform on the network side is stream-processed to obtain the processed line-side data.

[0035] In step 140, the processed line-side data is stored in the database.

[0036] In another embodiment, a streaming computing engine, such as Apache Flink or Spark Streaming, can be deployed on the network side. This engine acts as a consumer, continuously subscribing to line-side data streams from various lines within the network-side Kafka cluster. The streaming computing engine can process the data streams in real time, including parsing and formatting, data cleaning and validation, data association and enrichment, etc., to obtain processed line-side data.

[0037] Furthermore, the processed line-side data stream output by the streaming computing engine is then stored in batches or micro-batch in the persistent storage database on the network side via a data writing service.

[0038] In this embodiment, a pipeline architecture of acquisition-Kafka-stream processing-database storage is adopted to transform the traditional batch processing mode into a real-time stream processing mode, greatly shortening the latency from data generation to availability and meeting the stringent timeliness requirements of rail transit dispatching and real-time safety monitoring. Furthermore, by introducing a unidirectional transmission device, unidirectional data flow from the line side to the network side is enforced at the physical or protocol level, effectively preventing reverse penetration of network attacks from the management network to the production control network, and solving the core security problem of data transmission across security domains. Simultaneously, this method enables data dispersed across various lines to be securely aggregated to the network side, providing a data foundation for network-level collaborative analysis and emergency command.

[0039] This invention provides a real-time data storage and processing method for rail transit data, comprising: real-time acquisition of track-side data generated by rail vehicles, and storing the track-side data in a track-side distributed event stream platform, wherein the track-side data includes the operation data and alarm information of the rail vehicles; unidirectionally transmitting the track-side data stored in the track-side distributed event stream platform to a network-side distributed event stream platform based on a one-way transmission device; performing streaming processing on the track-side data in the network-side distributed event stream platform to obtain processed track-side data; and storing the processed track-side data in a database. This method solves the deficiencies of insufficient real-time performance and low security in rail transit data processing, and improves the timeliness and security of rail transit data storage and processing.

[0040] Figure 2 This is a schematic diagram of the process for storing the processed line-side data in a database, provided by the present invention.

[0041] The following will combine Figure 2 The process of storing the processed line-side data in a database, as provided by this invention, will be described.

[0042] In an exemplary embodiment of the present invention, combined with Figure 2As can be seen, storing the processed line-side data in the database may include steps 210 to 230, and each step will be described below.

[0043] In step 210, the processed line-side data is classified according to the data type of the processed line-side data to obtain classified line-side data. The classified line-side data includes at least a first classified line-side data with a structured data type and a second classified line-side data with a time-series data type.

[0044] In one embodiment, a data routing and classification module can also be deployed on the online network side. This module can receive processed online data streams from the streaming computing engine. The classification module can analyze and classify each data record according to predefined data type identification rules. Based on data attribute characteristics, business applications, and subsequent query and analysis patterns, the processed online data can be classified to obtain a first-classified online data with structured data types and a second-classified online data with time-series data types.

[0045] Among them, the first category of line-side data with structured data types typically describes a discrete, complete event or entity, with clear relationships between fields, making it suitable for relational queries and transaction operations. Examples include: alarm information (containing fields such as alarm ID, vehicle number, alarm code, alarm description, occurrence time, and confirmation status), a complete fault event record, or a complete vehicle entry / exit status change event.

[0046] The second category of line-side data, characterized by time-series data types, can be a continuously generated sequence of states or measurements with strict timestamps. Data points are typically arranged closely in chronological order, and query patterns primarily involve time-range-based retrieval and aggregation. Examples include vehicle operation data such as speed values ​​collected every second, real-time location (GPS coordinates), current, voltage, and motor temperature.

[0047] In step 220, the first-classified line-side data with structured data type is stored in the structured database.

[0048] In step 230, the second-classified line-side data with time-series data type is stored in the time-series database.

[0049] In another embodiment, the first-classified line-side data with structured data types can be stored in a structured database, such as a MySQL database or a PostgreSQL database.

[0050] In another embodiment, the second-classified line-side data with time-series data types can be stored in a time-series database, such as IoTDB, InfluxDB, or TimescaleDB.

[0051] It should be noted that, based on the data type, the processed standardized data is written to MySQL and IoTDB databases respectively. The structured data is stored in MySQL, leveraging its efficient transaction processing capabilities and structured query advantages to meet the real-time query and transaction processing needs of the network-side operation monitoring system. The time-series data is stored in IoTDB, utilizing its advantages in time-series data compression storage and efficient time range query to meet the needs of subsequent scenarios such as equipment operation trend prediction and historical data backtracking analysis. In this embodiment, differentiated storage enables upper-layer applications to access data in the most natural and efficient way. When operators need to analyze historical vehicle operation curves, they can directly perform millisecond-level time range retrieval and aggregation from the time-series database; when safety inspectors need to investigate fault event chains, they can utilize complex SQL queries in relational databases to quickly link multiple tables such as alarms, vehicles, and maintenance records. This targeted optimization directly accelerates the processes of data analysis, report generation, and decision support, thereby providing data support for the network-side operation monitoring system, fault analysis system, and trend prediction system, meeting the application needs of different business scenarios.

[0052] In an exemplary embodiment of the present invention, continuing from the preceding text... Figure 1 The above embodiment is used as an example for illustration. Before performing streaming processing on the line-side data in the network-side distributed event stream platform (corresponding to step 130), the real-time data entry processing method for rail transit may also include the following steps: Data validation is performed on the line-side data in the distributed event stream platform on the network side to obtain the line-side data after successful validation. The data validation includes any one or more of the following: data validity validation and data integrity validation. The following methods can be used to perform streaming processing on the line-side data in the distributed event stream platform on the network side: After the verification in the distributed event stream platform on the line network side is passed, the line-side data is processed in a streaming manner.

[0053] In one embodiment, a data verification service can also be deployed or used as the first processing unit in a stream processing job. This module can continuously consume raw line-side data from the Kafka cluster on the line side. The verification module can perform strict data verification on each (or batch) of data, including but not limited to data validity verification and data integrity verification. For example, alarm information must include the fields "alarm_code", "alarm_time", and "location"; if these fields are missing, the data is determined to be invalid.

[0054] Furthermore, for data that passes verification on the line side, a success tag is added, and the data is passed to the downstream streaming engine. For data that fails verification, it can be handled according to a strategy, such as importing it into a separate dead letter queue or an exception topic for subsequent manual or system analysis and troubleshooting; or recording detailed error information and issuing an alert in the log.

[0055] In another embodiment, the data on the line side that has passed verification in the Kafka cluster on the line side can be streamed, that is, the streaming engine only processes high-quality, reliable data.

[0056] In this embodiment, illegal, incomplete, or unreasonable data can be proactively filtered out before entering the streaming computing and storage stages. This ensures that all subsequent status monitoring, safety warnings, performance analysis, and decision support based on this data are accurate and reliable, which is crucial for the safety-first operation of rail transit.

[0057] In an exemplary embodiment of the present invention, continuing with the above-described embodiments as an example, the real-time data storage processing method for rail transit data before storing the line-side data in the line-side distributed event stream platform may further include the following steps: Get the current time information; The storage of line-side data in the line-side distributed event stream platform can be achieved in the following ways: If the current time information is found to meet the preset time requirements, the line-side data will be stored in the line-side distributed event stream platform.

[0058] In one embodiment, the current time information can also be acquired in real time. Furthermore, when the current time information is detected to meet a preset time requirement, the line-side data can be stored in the line-side Kafka cluster. The preset time requirement defines the time window or time conditions that allow data to enter subsequent processing flows. For example, line-side data can be stored in the line-side Kafka cluster within a periodic time window.

[0059] In this embodiment, by setting preset time requirements, data collection is transformed from receiving all data around the clock to precise collection on demand and by time period. This ensures that the data entering the core data processing system is highly relevant to the current business scenario, guaranteeing the business value density of the data from the source and avoiding interference from data from irrelevant time periods on the business analysis model.

[0060] In an exemplary embodiment of the present invention, continuing with the above-described embodiments as an example, the real-time data storage processing method for rail transit data before storing the line-side data in the line-side distributed event stream platform may further include the following steps: Real-time monitoring of the operating status of preset components of rail vehicles; The storage of line-side data in the line-side distributed event stream platform can be achieved in the following ways: When a pre-set component of the rail vehicle issues an operational alarm, the track-side data is stored in the track-side distributed event stream platform.

[0061] In one embodiment, the operating status signals of one or more preset components on the rail vehicle can be monitored in real time. These preset components can refer to core systems or subsystems crucial to the safe and stable operation of the vehicle, such as: traction system, braking system, bogie, door system, air conditioning system, battery, pantograph, etc. In this embodiment, no specific limitation is made to the preset components.

[0062] In another embodiment, when an operational alarm message from a preset component is detected, a complete storage process for the current batch of line-side data can be triggered.

[0063] In this embodiment, by strongly linking the triggering conditions for data entry to the safety status of key vehicle components, the system shifts from passively receiving all data to proactively focusing on anomalies and potential risk events. This significantly improves the intelligence of the data pipeline, ensuring that the data flowing into the core system has extremely high event relevance and safety value density.

[0064] In this embodiment, two data push modes are supported: timed triggering and event triggering. Timed triggering allows for flexible setting of time intervals to meet the collection frequency requirements of different data types; event triggering can respond to specific events in real time, ensuring that critical data is pushed without delay. Simultaneously, it supports multi-type data collection and is compatible with different types of acquisition devices such as track sensors, train position sensors, and equipment temperature sensors. The collected data covers both structured and time-series data, meeting the comprehensive data collection needs of the rail transit line. In yet another exemplary embodiment of the present invention, the following description continues using the previously described embodiments as examples. Streaming processing of line-side data in the distributed event stream platform on the network side to obtain processed line-side data can be achieved in the following manner: Based on the distributed streaming engine, the line-side data in the distributed event stream platform on the network side is subjected to one or more of the following processing methods: data format standardization, data deduplication, and data filtering, to obtain the processed line-side data.

[0065] In one embodiment, the distributed streaming engine can be the Flink streaming engine. For ease of explanation, the Flink streaming engine will be used as an example of a distributed streaming engine. As a core processing component, the Flink streaming engine can perform real-time data computation and cleaning, including data format standardization, data deduplication, and data filtering, or one or more of these functions.

[0066] In this embodiment, based on the Flink streaming engine, real-time data cleaning, standardization, and deduplication are achieved, with data processing latency controlled within 1 second, ensuring that the network side can quickly obtain high-quality standardized data. Flink jobs support dynamic scaling, automatically adjusting the number of parallel processing tasks according to data volume fluctuations to avoid data backlog. Simultaneously, Flink's checkpoint mechanism periodically saves job states, allowing processing to resume from the most recent checkpoint state after a job failure and restart, preventing data duplication or loss.

[0067] Figure 3 This invention provides a flowchart illustrating the process of processing line-side data by performing one or more of the following steps on the line-side data in the distributed event stream platform: data format standardization, data deduplication, and data filtering, based on a distributed streaming engine, to obtain the processed line-side data.

[0068] The following will combine Figure 3 The present invention describes the process by which the distributed streaming engine provides a method for processing line-side data in the network-side distributed event stream platform by performing one or more of the following processes: data format standardization, data deduplication, and data filtering, to obtain processed line-side data.

[0069] In an exemplary embodiment of the present invention, combined with Figure 3 As can be seen, based on the distributed streaming processing engine, performing data format standardization, data deduplication, and data filtering on the line-side data in the network-side distributed event stream platform to obtain the processed line-side data can include steps 310 to 330, and each step will be described below.

[0070] In step 310, the amount of data on the line side in the distributed event stream platform on the line network side is obtained.

[0071] In step 320, the processing rate of the distributed streaming engine is determined based on the amount of data.

[0072] In one embodiment, the data volume information of a specific Topic in the Kafka cluster on the network side, which serves as the data source for Flink jobs, can be obtained. The data volume can be the message backlog, the data inflow rate, or a result determined based on both. The message backlog refers to the total number of pending messages in the Topic that have not yet been consumed by Flink jobs. The data inflow rate refers to the number of messages or data volume (MB / s) newly written to the Topic per unit time (e.g., per second).

[0073] Furthermore, the processing rate of the Flink streaming engine can be determined based on the amount of data. For example, if the message backlog is continuously increasing or the inflow rate is significantly higher than the current processing rate, it can be determined that the system is at risk of backpressure, and the processing rate should be increased. If the message backlog is consistently low or zero, and the inflow rate is stable at a low level, it can be determined that the current computing resources are idle, and the processing rate can be appropriately reduced to save resources.

[0074] In step 330, based on the distributed streaming engine, the line-side data in the distributed event stream platform on the network side is subjected to one or more of the following processing methods according to the processing rate: data format standardization, data deduplication, and data filtering, to obtain the processed line-side data.

[0075] In another embodiment, the Flink streaming engine dynamically adjusts its internal task scheduling and resource allocation according to the processing rate. Subsequently, the execution logic of the processing job changes to: based on the Flink streaming engine, it performs one or more of the following processing on the line-side data in the Kafka cluster on the network side: data format standardization, data deduplication, and data filtering, according to the processing rate, to obtain the processed line-side data. This means that the processing capacity of a Flink job is no longer fixed, but becomes a computing service that elastically scales according to upstream data pressure, always striving to consume and process data at a rate best suited to the current load.

[0076] In this embodiment, a real-time feedback control loop enables the Flink job's processing rate to automatically track and adapt to fluctuations in the amount of data input in the Kafka cluster. This effectively prevents the risks of system backpressure, data backlog, or even processing pipeline blockage or crashes caused by sudden data surges, ensuring high availability and robustness of the entire real-time processing chain under variable loads.

[0077] As described above, the real-time data entry and processing method for rail transit provided by this invention, by introducing a streaming processing framework and an efficient data transmission mechanism, controls the overall latency of data acquisition, transmission, and entry into the database within seconds, meeting the real-time monitoring and processing needs of the network control center. By setting up a one-way optical gate to achieve one-way isolated transmission of data between the line side and the network side, combined with a data verification mechanism, the security and integrity of data transmission are ensured, preventing data tampering, unauthorized access, and reverse impact. Through a categorized storage strategy, different types of rail transit data are written into appropriate databases, improving data storage efficiency and adaptability to subsequent operational calculations and data analysis. By designing fault-tolerance mechanisms, breakpoint resume functionality, and system monitoring and alarm mechanisms, the stability and fault tolerance of the entire data processing system are improved, reducing manual maintenance costs and ensuring the continuous reliability of the data processing flow.

[0078] The real-time data entry and processing device for rail transit provided by the present invention is described below. The real-time data entry and processing device for rail transit described below can be referred to in correspondence with the real-time data entry and processing method for rail transit described above.

[0079] Figure 4 This is a schematic diagram of the structure of the real-time data entry and processing device for rail transit provided by the present invention.

[0080] The following will combine Figure 4 The structure of the real-time data entry and processing device for rail transit provided by the present invention will be described.

[0081] In an exemplary embodiment of the present invention, combined with Figure 4 As can be seen, the real-time data entry and processing device for rail transit can include a data acquisition module 410, a transmission module 420, a processing module 430, and a data entry module 440. Each module will be described in detail below.

[0082] The acquisition module 410 can be configured to acquire track-side data generated by the rail vehicle in real time and store the track-side data in the track-side distributed event stream platform, wherein the track-side data includes the operation data information and operation alarm information of the rail vehicle; Transmission module 420 can be configured to unidirectionally transmit the line-side data stored in the line-side distributed event stream platform to the network-side distributed event stream platform based on a unidirectional transmission device. Processing module 430 can be configured to perform streaming processing on line-side data in the network-side distributed event stream platform to obtain processed line-side data; The data entry module 440 can be configured to store the processed line-side data in a database.

[0083] In an exemplary embodiment of the present invention, the data entry module 440 may further be configured to: According to the data type of the processed line-side data, the processed line-side data is classified to obtain classified line-side data, wherein the classified line-side data includes at least a first classified line-side data with a structured data type and a second classified line-side data with a time-series data type. The data entry module 440 can store the processed line-side data in the database using the following methods: After the first classification, the line-side data with structured data types is stored in a structured database; The second-classified line-side data with time-series data type is stored in the time-series database.

[0084] In an exemplary embodiment of the present invention, the processing module 430 may further be configured to: Data verification is performed on the line-side data in the network-side distributed event stream platform to obtain line-side data that has passed verification. The data verification includes any one or more of the following: data validity verification and data integrity verification. Processing module 430 performs streaming processing on the line-side data in the network-side distributed event stream platform: After the verification is passed, the line-side data in the distributed event stream platform on the network side is processed in a streaming manner.

[0085] In an exemplary embodiment of the present invention, the acquisition module 410 may further be configured to: Get the current time information; The acquisition module 410 can store the line-side data to the line-side distributed event stream platform in the following ways: If the current time information is detected to meet the preset time requirement, the line-side data is stored in the line-side distributed event stream platform.

[0086] In an exemplary embodiment of the present invention, the acquisition module 410 may further be configured to: Real-time monitoring of the operating status of preset components of the rail vehicle; The acquisition module 410 can store the line-side data to the line-side distributed event stream platform in the following ways: When a preset component of the rail vehicle issues an operation alarm, the track-side data is stored in the track-side distributed event stream platform.

[0087] In an exemplary embodiment of the present invention, the processing module 430 can perform streaming processing on the line-side data in the network-side distributed event stream platform to obtain processed line-side data in the following manner: Based on the distributed streaming processing engine, the line-side data in the network-side distributed event stream platform is subjected to one or more of the following processing methods: data format standardization, data deduplication, and data filtering, to obtain the processed line-side data.

[0088] In an exemplary embodiment of the present invention, the processing module 430 may further be configured to: Obtain the amount of data on the line side in the network-side distributed event stream platform; Based on the amount of data, the processing rate of the distributed streaming engine is determined; Processing module 430 can use the following methods to perform data format standardization, data deduplication, and data filtering on the line-side data in the distributed event stream platform on the network side, based on a distributed streaming processing engine, to obtain processed line-side data: Based on the distributed streaming processing engine, the line-side data in the network-side distributed event stream platform is subjected to one or more of the following processing methods according to the processing rate: data format standardization, data deduplication, and data filtering, to obtain the processed line-side data.

[0089] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a real-time data storage processing method for rail transit data. This method includes: real-time acquisition of track-side data generated by rail vehicles and storing the track-side data in a track-side distributed event stream platform, wherein the track-side data includes the operation data information and operation alarm information of the rail vehicles; unidirectionally transmitting the track-side data stored in the track-side distributed event stream platform to a network-side distributed event stream platform based on a one-way transmission device; performing streaming processing on the track-side data in the network-side distributed event stream platform to obtain processed track-side data; and storing the processed track-side data in a database.

[0090] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0091] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the real-time data storage processing method for rail transit provided by the above methods. The method includes: real-time acquisition of track-side data generated by rail vehicles and storing the track-side data in a track-side distributed event stream platform, wherein the track-side data includes the operation data information and operation alarm information of the rail vehicles; unidirectionally transmitting the track-side data stored in the track-side distributed event stream platform to a network-side distributed event stream platform based on a unidirectional transmission device; performing streaming processing on the track-side data in the network-side distributed event stream platform to obtain processed track-side data; and storing the processed track-side data in a database.

[0092] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a real-time data storage processing method for rail transit data provided by the methods described above. The method includes: real-time acquisition of track-side data generated by rail vehicles and storing the track-side data in a track-side distributed event stream platform, wherein the track-side data includes the operation data information and operation alarm information of the rail vehicles; unidirectionally transmitting the track-side data stored in the track-side distributed event stream platform to a network-side distributed event stream platform based on a unidirectional transmission device; performing streaming processing on the track-side data in the network-side distributed event stream platform to obtain processed track-side data; and storing the processed track-side data in a database.

[0093] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0094] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for real-time data entry and processing of rail transit data, characterized in that, The method includes: Real-time acquisition of track-side data generated by rail vehicles, and storage of the track-side data in a track-side distributed event stream platform, wherein the track-side data includes the operation data information and operation alarm information of the rail vehicles; Based on the unidirectional transmission device, the line-side data stored in the line-side distributed event stream platform is unidirectionally transmitted to the network-side distributed event stream platform; The line-side data in the network-side distributed event stream platform is stream-processed to obtain processed line-side data. The processed line-side data is then stored in a database.

2. The real-time data entry and processing method for rail transit according to claim 1, characterized in that, Before storing the processed line-side data in the database, the method further includes: According to the data type of the processed line-side data, the processed line-side data is classified to obtain classified line-side data, wherein the classified line-side data includes at least a first classified line-side data with a structured data type and a second classified line-side data with a time-series data type. The step of storing the processed line-side data in a database includes: After the first classification, the line-side data with structured data types is stored in a structured database; The second-classified line-side data with time-series data type is stored in the time-series database.

3. The real-time data entry and processing method for rail transit data according to claim 1 or 2, characterized in that, Before performing streaming processing on the line-side data in the network-side distributed event stream platform, the method further includes: Data verification is performed on the line-side data in the network-side distributed event stream platform to obtain line-side data that has passed verification. The data verification includes any one or more of the following: data validity verification and data integrity verification. The streaming processing of line-side data in the network-side distributed event stream platform includes: After the verification is passed, the line-side data in the distributed event stream platform on the network side is processed in a streaming manner.

4. The real-time data entry and processing method for rail transit data according to claim 1 or 2, characterized in that, Before storing the line-side data to the line-side distributed event stream platform, the method further includes: Get the current time information; The step of storing the line-side data to the line-side distributed event stream platform includes: If the current time information is detected to meet the preset time requirement, the line-side data is stored in the line-side distributed event stream platform.

5. The real-time data entry and processing method for rail transit data according to claim 1 or 2, characterized in that, Before storing the line-side data to the line-side distributed event stream platform, the method further includes: Real-time monitoring of the operating status of preset components of the rail vehicle; The step of storing the line-side data to the line-side distributed event stream platform includes: When a preset component of the rail vehicle issues an operation alarm, the track-side data is stored in the track-side distributed event stream platform.

6. The real-time data entry and processing method for rail transit according to claim 1, characterized in that, The process of streaming the line-side data in the network-side distributed event stream platform to obtain processed line-side data includes: Based on the distributed streaming processing engine, the line-side data in the network-side distributed event stream platform is subjected to one or more of the following processing methods: data format standardization, data deduplication, and data filtering, to obtain the processed line-side data.

7. The real-time data entry and processing method for rail transit according to claim 6, characterized in that, Before performing data format standardization, data deduplication, and data filtering on the line-side data in the distributed event stream platform based on the distributed streaming processing engine to obtain the processed line-side data, the method further includes: Obtain the amount of data on the line side in the network-side distributed event stream platform; Based on the amount of data, the processing rate of the distributed streaming engine is determined; The distributed streaming engine-based process performs one or more of the following on the line-side data in the distributed event stream platform on the network side: data format standardization, data deduplication, and data filtering, to obtain processed line-side data, including: Based on the distributed streaming processing engine, the line-side data in the network-side distributed event stream platform is subjected to one or more of the following processing methods according to the processing rate: data format standardization, data deduplication, and data filtering, to obtain the processed line-side data.

8. A real-time data entry and processing device for rail transit, characterized in that, The device includes: The acquisition module is used to acquire track-side data generated by the rail vehicle in real time and store the track-side data in the track-side distributed event stream platform. The track-side data includes the operation data information and operation alarm information of the rail vehicle. The transmission module is used to transmit the line-side data stored in the line-side distributed event stream platform to the network-side distributed event stream platform in a one-way manner based on the one-way transmission device. The processing module is used to perform streaming processing on the line-side data in the network-side distributed event stream platform to obtain processed line-side data. The data entry module is used to store the processed line-side data in a database.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the real-time data entry processing method for rail transit data as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the real-time data entry processing method for rail transit data as described in any one of claims 1 to 7.

Images