Rail transit comprehensive monitoring system, big data management method and storage medium
By using a big data platform and stream processing module to sort and store JSON monitoring data from the rail transit monitoring system, the problems of insufficient data timeliness and integrity are solved, and rapid diversion and orderly management are achieved.
Patent Information
- Application Number
- CN202511303364.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-12
- Publication Date
- 2025-11-14
AI Technical Summary
The massive amounts of data generated by the rail transit integrated monitoring system lack effective governance solutions, resulting in insufficient timeliness and completeness of the data.
The system uses a big data platform to receive JSON monitoring data pushed by the data collection server. The data is then sorted based on the sorting data configuration table through the stream processing module, real-time indicator data is calculated, and the data is stored in the real-time storage module. At the same time, the system uses an offline storage module for rolling and fixed storage to ensure the integrity and timeliness of the data.
It enables rapid distribution and orderly management of data from JSON monitoring data to real-time metrics data, ensuring the timeliness and integrity of the data.
Smart Images
Figure CN120942400A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of rail transit technology, and in particular to a rail transit integrated monitoring system, big data governance method, and storage medium. Background Technology
[0002] The Integrated Supervisory Control System (ISCS) is a core automation platform in urban rail transit. It achieves efficient and intelligent operation management through centralized monitoring and coordinated operation of electromechanical equipment. However, due to the massive amounts of data generated by the ISCS every moment, there is currently no solution in terms of relevant technology. Summary of the Invention
[0003] In view of the above-mentioned defects or deficiencies in related technologies, it is desirable to provide a rail transit integrated monitoring system and big data governance method and storage medium, which can effectively govern the big data of the rail transit integrated monitoring system and ensure the timeliness and integrity of the data.
[0004] In a first aspect, this application provides a comprehensive monitoring system for rail transit, which includes a data source layer, a data integration layer, and a data storage and governance layer.
[0005] The data source layer includes a comprehensive monitoring system for each station on different subway lines; the data integration layer includes data line aggregation servers corresponding to the different subway lines, and the data line aggregation servers are used to collect JSON monitoring data pushed by the comprehensive monitoring system.
[0006] The data storage and governance layer includes an interconnected big data platform, a stream processing module, and a real-time storage module. The big data platform is used to receive the JSON monitoring data pushed by the data line aggregation server, and after the stream processing module sorts the JSON monitoring data according to the sorting data configuration table to obtain the data required for indicator processing, it calculates the real-time indicator data corresponding to the data required for indicator processing, and sends the real-time indicator data to the real-time storage module for storage.
[0007] Optionally, in some embodiments of this application, the JSON monitoring data includes first full monitoring data and first incremental monitoring data. The data line aggregation server is specifically used to send the first full monitoring data when establishing a connection with the big data platform for the first time, and to send the first incremental monitoring data at preset intervals after the connection is successful.
[0008] Optionally, in some embodiments of this application, the data line aggregation server is further configured to send a second full monitoring data to the big data platform according to a preset maintenance time. The second full monitoring data includes the first full monitoring data and the first incremental monitoring data.
[0009] Optionally, in some embodiments of this application, the data storage and governance layer further includes an offline storage module, which is used to receive the JSON monitoring data and the data required for indicator processing pushed by the stream processing module, and calculate the offline indicator data corresponding to the data required for indicator processing based on the data required for indicator processing.
[0010] Optionally, in some embodiments of this application, the offline storage module is also used for rolling storage of the JSON monitoring data and fixed storage of the offline indicator data.
[0011] Secondly, this application provides a big data governance method for the rail transit integrated monitoring system described in any one of the first aspects, the big data governance method comprising:
[0012] Collect JSON monitoring data pushed by the integrated monitoring system of each station on different subway lines;
[0013] The JSON monitoring data is sorted according to the sorting data configuration table to obtain the data required for indicator processing, and the real-time indicator data corresponding to the data required for indicator processing is calculated and stored in real time.
[0014] Optionally, in some embodiments of this application, the JSON monitoring data includes first full monitoring data and first incremental monitoring data. The first full monitoring data is used to be sent to the big data platform through the data line aggregation server when the data line aggregation server and the big data platform establish a connection for the first time. The first incremental monitoring data is used to be sent to the big data platform through the data line aggregation server after the data line aggregation server and the big data platform have successfully connected.
[0015] Optionally, in some embodiments of this application, the JSON monitoring data further includes second full monitoring data, which includes the first full monitoring data and the first incremental monitoring data. The second full monitoring data is used to be sent to the big data platform through the data line aggregation server at a preset maintenance time.
[0016] Optionally, in some embodiments of this application, the data structure of the JSON monitoring data includes data source, packet sending timestamp, tag identifier, point description, tag value, tag value description, and device data update time.
[0017] Thirdly, this application provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps of the big data governance method described in any one of the second aspects.
[0018] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:
[0019] This application provides a comprehensive monitoring system for rail transit, a big data governance method, and a storage medium. The system receives JSON monitoring data from the comprehensive monitoring systems of various stations on different subway lines pushed by a data line aggregation server through a big data platform. It then uses a stream processing module to sort the JSON monitoring data based on a sorting data configuration table to obtain the data required for indicator processing. Finally, it calculates the real-time indicator data corresponding to the data required for indicator processing, thereby achieving rapid diversion and orderly management from JSON monitoring data to real-time indicator data, ensuring data timeliness and integrity. Attached Figure Description
[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0021] Figure 1 A schematic diagram of the architecture of a comprehensive rail transit monitoring system provided in this application embodiment;
[0022] Figure 2 An embodiment of this application provides a method for... Figure 1 The diagram shows a flowchart illustrating the big data governance method for the integrated monitoring system of rail transit.
[0023] Figure label:
[0024] 10-Integrated monitoring system for rail transit; 101-Data source layer; 1011-First integrated monitoring system; 1012-Second integrated monitoring system; 1013-Third integrated monitoring system; 1014-Fourth integrated monitoring system; 1015-Fifth integrated monitoring system; 1016-Sixth integrated monitoring system; 102-Data integration layer; 1021-First data line aggregation server; 1022-Second data line aggregation server; 1023-Third data line aggregation server; 103-Data storage and governance layer; 1031-Big data platform; 1032-Processing module; 1033-Real-time storage module; 1034-Offline storage module. Detailed Implementation
[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0026] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0027] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The following examples illustrate this. Figures 1 to 2 This application provides a detailed description of the integrated rail transit monitoring system, big data governance method, and storage medium provided in the embodiments of this application.
[0028] Please refer to Figure 1 This is a schematic diagram of the architecture of a comprehensive rail transit monitoring system 10 provided in this application embodiment. The comprehensive rail transit monitoring system 10 includes a data source layer 101, a data integration layer 102, and a data storage and governance layer 103. The data source layer 101 includes comprehensive monitoring systems for each station on different subway lines. For example, in subway line A, station 1 corresponds to the first comprehensive monitoring system 1011, station 2 corresponds to the second comprehensive monitoring system 1012, station 1 in subway line B corresponds to the third comprehensive monitoring system 1013, station 2 corresponds to the fourth comprehensive monitoring system 1014, station 1 in subway line C corresponds to the fifth comprehensive monitoring system 1015, station 2 corresponds to the sixth comprehensive monitoring system 1016, and so on. The data integration layer 102 includes data aggregation servers corresponding to different subway lines. For example, the data aggregation server can be a Kafka Client. Subway line A corresponds to the first data aggregation server 1021, subway line B corresponds to the second data aggregation server 1022, subway line C corresponds to the third data aggregation server 1023, etc. This data aggregation server can collect JSON monitoring data pushed by the comprehensive monitoring system.
[0029] The data storage and governance layer 103 includes an interconnected big data platform 1031, a stream processing module 1032, and a real-time storage module 1033. The big data platform 1031 receives JSON monitoring data pushed by the data line aggregation server. After the stream processing module 1032 sorts the JSON monitoring data according to the sorting data configuration table to obtain the data required for indicator processing, it calculates the real-time indicator data corresponding to the required data and sends the real-time indicator data to the real-time storage module 1033 for storage. For example, the big data platform 1031 can be a Kafka Server, the stream processing module 1032 can provide Flink data sorting services, the sorting data configuration table can be flexibly selected and set in the system's operation interface according to the user object and purpose of the parameters, the data required for indicator processing can be location data separated according to business needs, and the real-time storage module 1033 can be a Redis data warehouse, an open-source in-memory database that supports multiple data structures such as strings, hashes, and lists, and has high performance and rich functionality, thus providing real-time data query services to external systems.
[0030] In some embodiments of this application, the JSON monitoring data may include first full monitoring data and first incremental monitoring data. The first incremental monitoring data refers to monitoring data added within a certain time period. The data line aggregation server can send the first full monitoring data when it first establishes a connection with the big data platform 1031, and send the first incremental monitoring data at preset intervals after the connection is successful. For example, the preset interval can be 2 seconds. The server pushes the changing data on a station-by-station basis, which facilitates rapid processing and improves efficiency. Furthermore, the data line aggregation server can also send second full monitoring data to the big data platform 1031 at preset maintenance times. The second full monitoring data includes the first full monitoring data and the first incremental monitoring data. For example, the preset maintenance time can be 3:00 AM or 4:00 AM every day. The advantage of this setting is that the first full monitoring data and the first incremental monitoring data can be cross-referenced with the second full monitoring data, ensuring that the data at each station location can be updated on a regular basis.
[0031] In some embodiments of this application, it is still referred to as Figure 1As shown in the example, the data storage and governance layer 103 may also include an offline storage module 1034. This offline storage module 1034 can receive JSON monitoring data and data required for indicator processing pushed by the stream processing module 1032, and calculate the offline indicator data corresponding to the data required for indicator processing based on the data required for indicator processing. For example, the offline storage module 1034 can be a Hive data warehouse, which calculates offline indicator data daily and stores JSON monitoring data and offline indicator data in separate tables. The offline indicator data can be the same as the real-time indicator data. The Hive data warehouse is a data warehouse tool built on Hadoop, providing a SQL-like query language, HiveQL, which can convert queries into MapReduce tasks for execution. Furthermore, the offline storage module 1034 can not only store JSON monitoring data in a rolling manner, that is, data from later periods overwrites data from earlier periods, ensuring the availability of data warehouse space, but also store offline indicator data permanently, that is, adopt a permanent storage method, thereby ensuring data traceability.
[0032] In addition, the data storage and governance layer 103 may also include modules such as HDFS, Click House, HBase, and Hetu Engine. The HDFS module can store large-scale datasets and provide high-throughput data access. The Click House module is a columnar database management module for online analytical processing, featuring linear scalability and fault tolerance. It can store and process petabytes of data, supports SQL, and employs compression algorithms such as LZ4 and ZSTD, and has vectorized query execution capabilities. The HBase module is a distributed columnar database built on Hadoop, capable of storing massive amounts of structured data and providing real-time read / write access. The Hetu Engine module is a high-performance interactive SQL analysis and data virtualization engine that seamlessly integrates with the big data ecosystem, enabling second-level interactive queries on massive datasets, supporting unified data access across sources and domains, and providing one-stop SQL convergence analysis for multiple data sources such as Hive, HBase, GaussDB, and Click House.
[0033] The rail transit integrated monitoring system provided in this application embodiment receives JSON monitoring data from the integrated monitoring systems of various stations on different subway lines pushed by the data line aggregation server through a big data platform. It then uses a stream processing module to sort the JSON monitoring data based on a sorting data configuration table to obtain the data required for indicator processing. Finally, it calculates the real-time indicator data corresponding to the data required for indicator processing, thereby realizing rapid diversion and orderly management from JSON monitoring data to real-time indicator data, ensuring the timeliness and integrity of the data.
[0034] Based on the foregoing embodiments, this application provides a big data governance method, which can be used for... Figure 1 The corresponding embodiment is the integrated rail transit monitoring system 10. Please refer to... Figure 2 This is a flowchart illustrating a big data governance method provided in an embodiment of this application. The big data governance method specifically includes the following steps:
[0035] S101 collects JSON monitoring data pushed by the integrated monitoring system of each station on different subway lines.
[0036] For example, in some embodiments of this application, the JSON monitoring data may include first full monitoring data and first incremental monitoring data. The first full monitoring data can be sent to the big data platform 1031 through the data line aggregation server when the data line aggregation server and the big data platform 1031 establish a connection for the first time. The first incremental monitoring data can be sent to the big data platform 1031 through the data line aggregation server after the connection between the data line aggregation server and the big data platform 1031 is successfully established. Furthermore, in some embodiments of this application, the JSON monitoring data may also include second full monitoring data, which includes the first full monitoring data and the first incremental monitoring data. The second full monitoring data can be sent to the big data platform 1031 through the data line aggregation server at a preset maintenance time.
[0037] For example, in other embodiments of this application, the data structure of JSON monitoring data includes, but is not limited to, data source, packet sending timestamp, tag number identifier, point description, tag number value, tag number value description, and data update time, as shown in Table 1. The data is standardized and facilitates rapid processing.
[0038] Table 1 shows the data structure of the JSON monitoring data.
[0039]
[0040] S102: Based on the sorting data configuration table, sort the JSON monitoring data to obtain the data required for indicator processing, calculate the real-time indicator data corresponding to the data required for indicator processing, and store the real-time indicator data in real time.
[0041] For example, some embodiments of this application can also calculate offline indicator data corresponding to the data required for indicator processing based on the data required for indicator processing. The offline indicator data can be the same as the real-time indicator data, and the offline indicator data is stored in a fixed manner to ensure the traceability of the data.
[0042] It should be noted that the descriptions of the same steps and contents as in other embodiments in this embodiment can be found in the descriptions in other embodiments, and will not be repeated here.
[0043] The big data governance method for rail transit integrated monitoring system provided in this application collects JSON monitoring data pushed by the integrated monitoring system of each station in different subway lines, sorts the JSON monitoring data based on the sorting data configuration table to obtain the data required for indicator processing, and then calculates the real-time indicator data corresponding to the data required for indicator processing. This achieves rapid diversion and orderly management from JSON monitoring data to real-time indicator data, ensuring data timeliness and integrity.
[0044] In another aspect, embodiments of this application provide a computer-readable storage medium for storing program code for executing the aforementioned... Figure 2 Any implementation method of the big data governance method in the corresponding embodiment.
[0045] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0046] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed between each other can be through some interfaces, indirect coupling or communication connection between devices or modules, and can be electrical, mechanical, or other forms. Modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical units, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0047] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more units can be integrated into one module. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.
[0048] Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the big data governance method of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0049] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0050] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A comprehensive monitoring system for rail transit, characterized in that, The rail transit integrated monitoring system includes a data source layer, a data integration layer, and a data storage and governance layer. The data source layer includes a comprehensive monitoring system for each station on different subway lines; the data integration layer includes data line aggregation servers corresponding to the different subway lines, and the data line aggregation servers are used to collect JSON monitoring data pushed by the comprehensive monitoring system. The data storage and governance layer includes an interconnected big data platform, a stream processing module, and a real-time storage module. The big data platform is used to receive the JSON monitoring data pushed by the data line aggregation server, and after the stream processing module sorts the JSON monitoring data according to the sorting data configuration table to obtain the data required for indicator processing, it calculates the real-time indicator data corresponding to the data required for indicator processing, and sends the real-time indicator data to the real-time storage module for storage.
2. The integrated monitoring system for rail transit according to claim 1, characterized in that, The JSON monitoring data includes first full monitoring data and first incremental monitoring data. The data line aggregation server is specifically used to send the first full monitoring data when establishing a connection with the big data platform for the first time, and to send the first incremental monitoring data at preset intervals after the connection is successfully established.
3. The integrated monitoring system for rail transit according to claim 2, characterized in that, The data line aggregation server is also specifically used to send second full monitoring data to the big data platform according to a preset maintenance time. The second full monitoring data includes the first full monitoring data and the first incremental monitoring data.
4. The integrated monitoring system for rail transit according to claim 1, characterized in that, The data storage and governance layer also includes an offline storage module, which is used to receive the JSON monitoring data and the data required for indicator processing pushed by the stream processing module, and calculate the offline indicator data corresponding to the data required for indicator processing based on the data required for indicator processing.
5. The integrated monitoring system for rail transit according to claim 4, characterized in that, The offline storage module is also used for rolling storage of the JSON monitoring data and fixed storage of the offline indicator data.
6. A big data governance method for the rail transit integrated monitoring system according to any one of claims 1 to 5, characterized in that, The big data governance methods include: Collect JSON monitoring data pushed by the integrated monitoring system of each station on different subway lines; The JSON monitoring data is sorted according to the sorting data configuration table to obtain the data required for indicator processing, and the real-time indicator data corresponding to the data required for indicator processing is calculated and stored in real time.
7. The big data governance method according to claim 6, characterized in that, The JSON monitoring data includes first full monitoring data and first incremental monitoring data. The first full monitoring data is used to send to the big data platform through the data line aggregation server when the data line aggregation server establishes a connection with the big data platform for the first time. The first incremental monitoring data is used to send to the big data platform through the data line aggregation server after the data line aggregation server successfully connects with the big data platform.
8. The big data governance method according to claim 7, characterized in that, The JSON monitoring data also includes second full monitoring data, which includes the first full monitoring data and the first incremental monitoring data. The second full monitoring data is used to send to the big data platform through the data line aggregation server at a preset maintenance time.
9. The big data governance method according to any one of claims 6 to 8, characterized in that, The data structure of the JSON monitoring data includes data source, packet sending timestamp, tag identifier, point description, tag value, tag value description, and device data update time.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the big data governance method according to any one of claims 6 to 9.