Rail transit operation data management system and method based on big data

Abstract

The invention discloses a rail transit operation data management system and method based on big data, and the system comprises a data integration module which is used for accessing multi-source heterogeneous data of rail transit operation, and carrying out the data integration of the multi-source heterogeneous data; the data stream batch integrated processing module is used for processing the multi-source heterogeneous data after data integration; the data storage module is used for storing data and performing fusion analysis on the processed data; the data model definition module is used for establishing an equipment model and defining a relationship between the current equipment and upstream and downstream equipment; and the data service module is used for realizing service sharing among multiple users and tenants. The rail transit operation data management platform is a complete rail transit operation data management platform from multi-source heterogeneous data acquisition, multi-source heterogeneous data storage, data processing, asset management, rail transit data model, data sharing, authority management and engineering.

Description

Technical Field

[0001] The present invention relates to the field of rail transit technology, and in particular to a rail transit operation data management system and method based on big data. Background Art

[0002] At present, in the field of rail transit, based on the development trend of smart transportation, the amount of rail transit data is growing geometrically, and the data structure is diversified. With the development of smart rail transit, based on the needs of new businesses, the need for multi-professional fusion analysis of traditional rail transit operation systems is also more urgent. In view of the new situation in the field of rail transit, it is very urgent to introduce a data governance system based on big data technology suitable for the field of rail transit, which can meet the storage of multi-source heterogeneous data, realize data sharing, multi-professional data fusion analysis, and have general data governance capabilities. The commercial big data platforms on the market focus more on general big data processing, the overall framework structure is relatively cumbersome, and the data model is also more general. The rail transit field often needs to rebuild a big data governance platform adapted to rail transit based on commercial use. Summary of the invention

[0003] The purpose of the present invention is to provide a rail transit operation data management system and method based on big data. It aims to solve the problem that the big data platform in the prior art focuses on general big data processing, the overall framework structure is relatively complex, and the data model also focuses on general use, which is not suitable for data management in the rail transit field.

[0004] To achieve the above object, the present invention is implemented through the following technical solutions:

[0005] On the one hand, the present invention provides a rail transit operation data management system based on big data, comprising:

[0006] A data integration module, which is used to access multi-source heterogeneous data and perform data integration on the multi-source heterogeneous data;

[0007] A data stream and batch integrated processing module, which is connected to the data integration module and is used to process the multi-source heterogeneous data after data integration to obtain processed data. The data stream and batch integrated processing module includes a data processing framework for supporting real-time stream data processing and batch data processing;

[0008] A data storage module, which is connected to the data stream batch integrated processing module, and the data storage module includes different types of big data storage components for data storage and fusion analysis of the processed data;

[0009] A data model definition module is connected to the data storage module, and establishes a device model according to the data in the data storage module, and defines the relationship between the current device and its upstream and downstream devices, and realizes data linkage through the device model to facilitate data integration;

[0010] The data service module is connected to the data storage module and is used to realize service sharing among multiple users and tenants.

[0011] Preferably, it also includes: a data intermittent retransmission module, which is connected to the data integration module, and is used to manage the data integration task interface of the data integration module, monitor the integration task status of the data integration module, and start the execution of tasks within the time period by manually filling in the time period requiring data intermittent retransmission.

[0012] Preferably, the data integration module includes:

[0013] A configurable open source data integration plug-in, which is used for data integration of multi-source heterogeneous data;

[0014] A variety of data integrators are used to select a corresponding data integrator for data integration through the data application protocol and the data integration method of the data integration module.

[0015] Preferably, it also includes: a task scheduling module, which is connected to the data integration module, the data stream batch integrated processing module and the data service module respectively, and is used to schedule all tasks in the management system.

[0016] Preferably, the task scheduling module adopts a task scheduling framework of a distributed scheduling platform to manage the start and stop, failures and notifications of all tasks in the system.

[0017] Preferably, it also includes: a data asset management module, which is connected to the data storage module and is used to view the multi-source heterogeneous data information stored in the current big data platform.

[0018] Preferably, it also includes: a permission management module, which is connected to the data storage module, the permission management module includes a user permission management unit and a data permission management unit, the user permission management unit is used to support multi-tenant management and user permission authentication, and the data permission management unit is used to support the management of data permission access to a specific tenant, a certain type of database, a certain data table or a certain row.

[0019] Preferably, it also includes: a development task engineering configuration module, which is used for the migration of tasks and data of different projects in the system.

[0020] On the other hand, the present invention also provides a rail transit operation data management method based on big data, which is implemented by the rail transit operation data management system based on big data, and the method includes:

[0021] The data stream batch integrated processing module processes the data using the real-time stream processing method in the real-time stream processing mode according to the real-time stream data obtained from the sensor or the data integration module;

[0022] The data stream batch integrated processing module also processes data based on offline resources in the database or file system through the processing method of the integrated data source in the batch processing mode.

[0023] Preferably, the equipment model is established by abstractly extracting data elements in the entire life cycle of similar equipment through the working principle of rail transit equipment, sampling principle and application experience.

[0024] Preferably, the data linkage achieved through the device model specifically includes: based on the established device model, obtaining the relationship between each of the device models, and based on the relationship between each of the device models, establishing a fogged view of the master data model of the device model, the data of the fogged view changes with the data of the master data model, so as to realize data linkage changes in the entire system when a certain parameter in the system changes.

[0025] Preferably, the data sharing mode of the data service module includes: a general data access interface of a big data platform, a data service market, and real-time data stream sharing, or any combination thereof.

[0026] An electronic device includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the above-mentioned rail transit operation data management method based on big data is implemented.

[0027] A readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned rail transit operation data governance method based on big data is implemented.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. The rail transit operation data management system based on big data provided by the present invention is a complete platform system for multi-professional heterogeneous data storage and multi-professional data management suitable for rail transit projects. It is a complete rail transit operation data management platform from multi-source heterogeneous data collection, multi-source heterogeneous data storage, data processing, asset management, rail transit data model, data sharing, authority management and engineering;

[0030] 2. The data integration in the rail transit operation data management system and method of the present invention supports two modes, including the currently popular data integration components Kettle (data migration tool), Seatunnel (distributed data integration platform) and FLINK (stream computing framework), and also supports specific data application protocols such as Modbus (serial communication protocol) commonly used in the rail transit field. The data integrator mode implemented and encapsulated by the data integration component is adopted to achieve plug-and-play of the data integration plug-in, and supports user-defined registration of the data integrator;

[0031] 3. The present invention sets up a data stream and batch integrated processing module and adopts a stream and batch integrated rail transit data management framework to create a data platform suitable for rail transit data management, meeting the needs of various data management methods such as operation and maintenance, operation control, etc., rather than just supporting batch processing of big data;

[0032] 4. The present invention adopts a device-oriented rail transit data model to abstract rail transit equipment into corresponding rail transit equipment models, and realizes equipment model stratification and data linkage of equipment models, which facilitates the implementation of data integration, extraction, analysis, linkage and other services, creates a unique and professional rail transit data model, covers basically all rail transit operation and maintenance equipment, supports user-defined registration of rail transit data models, and supports the need to add rail transit equipment models for special business scenarios;

[0033] 5. The rail transit operation data governance system of the present invention takes into account the idea of ​​project engineering and creates engineering management, making the reusable data integrator, rail transit data model, data integration and data development tasks reusable for different projects, rather than just a general data processing platform;

[0034] 6. The rail transit operation data management system of the present invention focuses on data consistency, and considers data consistency issues from the aspects of source data management and data intermittent retransmission to ensure data integrity and consistency. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for the description. Obviously, the drawings described below are an embodiment of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work:

[0036] Figure 1 A logical schematic diagram of a rail transit operation data management system based on big data provided by an embodiment of the present invention;

[0037] Figure 2 A data management logic diagram provided for an embodiment of the present invention;

[0038] Figure 3A schematic diagram of a data integrator list provided by an embodiment of the present invention;

[0039] Figure 4 A schematic diagram of vehicle log data management provided by an embodiment of the present invention;

[0040] Figure 5 A schematic diagram of a rail transit data model provided by an embodiment of the present invention;

[0041] Figure 6 A schematic diagram of real-time data integration and development provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following is combined with Figure 1-6 The rail transit operation data governance system and method based on big data proposed in the present invention are further described in detail with specific implementation methods. According to the following description, the advantages and features of the present invention will be clearer. It should be noted that the drawings are in a very simplified form and use non-precise proportions, which are only used to conveniently and clearly assist in explaining the purpose of the implementation methods of the present invention. In order to make the purposes, features and advantages of the present invention more obvious and easy to understand, please refer to the drawings. It should be noted that the structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the limiting conditions for the implementation of the present invention, so they have no technical substantive significance. Any structural modification, change in proportional relationship or adjustment of size, without affecting the effects that can be produced by the present invention and the purposes that can be achieved, should still fall within the scope of the technical content disclosed by the present invention.

[0043] On the one hand, reference Figure 1As shown, this embodiment provides a rail transit operation data governance system based on big data, which is applied to the fields of rail transit multi-professional operation and maintenance, operation control, etc. The system includes: a data integration module, which is used to access multi-source heterogeneous data of rail transit operation and integrate the multi-source heterogeneous data; a data stream batch integrated processing module, which is connected to the data integration module, and is used to process the multi-source heterogeneous data after data integration to obtain processed data, and the data stream batch integrated processing module includes a data processing framework for supporting real-time stream data processing and batch data processing; a data storage module, which is connected to the data stream batch integrated processing module, and the data storage module includes different types of big data storage components for data storage and fusion analysis of the processed data; a data model definition module, which is connected to the data storage module, establishes a device model according to the data in the data storage module, and defines the relationship between the current device and its upstream and downstream devices, realizes data linkage through the device model, and facilitates data integration; a data service module, which is connected to the data storage module, and is used to realize service sharing between multiple users and tenants. The data intermittent retransmission module is connected to the data integration module and is used to manage the data integration task interface of the data integration module, monitor the integration task status of the data integration module, and start the execution of the tasks within the time period by manually filling in the time period in which the data needs to be intermittently retransmitted; the task scheduling module is connected to the data integration module, the data stream batch processing module and the data service module respectively, and is used to schedule all tasks in the management system. The task scheduling module adopts the task scheduling framework of the distributed scheduling platform to manage the start, stop, failure and notification of all tasks in the system. The data asset management module is connected to the data storage module and is used to view the multi-source heterogeneous data information stored in the current big data platform; the authority control module is connected to the data storage module, and the authority control module includes a user authority control unit and a data authority control unit. The user authority control unit is used to support multi-tenant control and user authority authentication, and the data authority control unit is used to support the control of data authority access to a specific tenant, a certain type of database, a certain data table or a certain row; the development task engineering configuration module is used for the migration of tasks and data of different projects in the system.

[0044] The following is a detailed introduction to the above modules:

[0045] 1. Data Integration Module

[0046] In this embodiment, the data integration module supports two types of data integration modes, one is to support common configurable open source big data data integration plug-ins; the other is to support the data integrator encapsulated by the platform based on specific application protocols and open source integration plug-in interfaces. The corresponding data integrator is selected for data integration according to the data application protocol and the data integration method of the data integration module. For example, according to the data processing method, this embodiment supports three data integration methods, namely real-time data integration, batch data integration, and CDC (Change Data Capture) data integration.

[0047] In the data integration module of this embodiment, facing multi-source heterogeneous data, the data integration module supports a variety of open source data integration mechanisms and supports common configurable open source data integration plug-ins. The data integration mechanisms in this embodiment include: Seatunnel data integration mechanism, FLINK data integration mechanism, Kettle data integration mechanism, CDC data integration mechanism or any combination thereof.

[0048] In the data integration module of this embodiment, the data integrator method mentioned above for specific application protocol collection is also involved. For common rail transit data collection scenarios, because it involves specific data collection application protocols, this data governance system encapsulates the specific data application protocols used for data collection in combination with the big data general integration interface (Seatunel, Flink, Kettle, CDC or custom interface, etc.) to form a data integrator that adapts to the data application protocol and the specific data application protocol of the data integration mechanism.

[0049] The data integrator in this embodiment is plug-in-based. The system provides a variety of data integrators that are adapted to the rail transit industry Modbus, 104 protocols, etc. The data integrator encapsulates almost all data application protocols for rail transit data collection. When the project is expanded, the corresponding data integrator is selected according to the data application protocol and data integration method, and the IP and other parameters corresponding to the specific environment of the project are configured to successfully load the data integrator.

[0050] This system supports users to customize data integration plug-ins according to special business needs, adapt the data integration functions to the required scenarios, and supports registering customized data integration plug-ins to the platform as reusable data integrators.

[0051] The data integrator can be instantiated in specific use. In a specific project, by selecting and loading the adapted application protocol, data integration method, and filling in the relevant configuration, the data integrator corresponding to the application protocol of the project can be instantiated. The data integrator in this system can be managed and upgraded according to the version.

[0052] In this embodiment, data integration includes two integration modes: offline data integration and real-time stream data integration, wherein signal data is collected in real-time stream data mode, and vehicle log data is collected in offline data mode.

[0053] 1) Real-time data integration example: This embodiment takes the real-time acquisition of signal data as an example. Figure 3 As shown in the figure, the real-time signal data stream, as the original data, needs to be parsed according to the agreed data application protocol. The data is parsed using the UDF (User-Defined Function) function. The parsed data is sent to Kafka (an open source distributed event stream platform) for further processing and to the Doris real-time data warehouse for data consistency and integrity comparison during the data collection process.

[0054] For real-time signal data collection, you need to use the real-time signal data collector that is packaged in this system for this scenario. The data integration method adopts the streaming data integration method, and the real-time signal collection integrator that is adapted to the real-time streaming data of Seatunnel is selected. Seatunnel supports high-performance data integration of multiple data sources.

[0055] This embodiment targets the real-time signal data stream and selects the Seatunnel-based data integration plug-in that the platform encapsulates for the signal acquisition application protocol. The source end is the Kafka data stream, which is processed and parsed by the UDF function in the middle. The parsed data is sent to the Kafka bus for further data processing; in addition, the parsed data is sent to the Doris real-time data warehouse target end as the source layer data. Other field information can be removed when the business scenario is expanded, and it can also be used to verify data consistency issues in data collection.

[0056] The integrated data is usually cleaned first and then sent to the real-time data development task for corresponding data processing. During the data ETL (Extract-Transform-Load) process, several necessary data rules will be established for data cleaning, including removing non-standard records, normalizing and mapping fields such as station names, and then forming normalized fact master data. For details, please refer to Figure 6 shown.

[0057] In this embodiment, two quality cleaning nodes are added to the signal real-time flow task cleaning, the associated dictionary table is mapped, and the IOM (In-Out-Memory) extension field is added, which mainly realizes the naming of the station name according to the standard definition specification of the data warehouse data. In this process, the subsequent IOM extension field is added to support subsequent analysis. Among them, the associated dictionary table mapping will access the system table in the system configuration MySQL for dynamic replacement.

[0058] like Figure 6 As shown in the figure, real-time streaming data is integrated through the data collector, and then analyzed, cleaned, and mapped in the associated dictionary table, and then the data is sent to the real-time data development. Real-time data development is basically designed according to business needs, and can be implemented using FLINK nodes.

[0059] In the process of developing real-time signal data stream data in this embodiment, the FLINK computing engine is used, which can efficiently complete real-time data calculation. The FLINK engine supports real-time data processing methods such as Java, Scala, SQL, and PYTHON. Message queues are used to transfer parameters between real-time stream processing nodes.

[0060] The task scheduling of real-time data in this application is also managed by the task scheduling management module, including real-time data integration tasks and real-time data processing tasks.

[0061] 2) Offline data integration example: In this embodiment, the vehicle log data is collected offline and the data is processed in batch mode. For specific vehicle log data management methods, refer to Figure 4 As shown, it includes: step S101: collecting source data; step S102: assembling corresponding data integrator; step S103: file storage; step S104: data development task - parsing data; step S105: data development task - cleaning data; step S106: data development task - rail transit data model; step S107: rail transit data service API (application programming interface).

[0062] In the offline data integration, a corresponding offline data integrator is selected. In this embodiment, a vehicle log integrator developed in the FTP (File Transfer Protocol) mode is used to integrate the vehicle log data into the HDFS file system of the big data platform.

[0063] In this loading method, the vehicle log data FTP integrator is abstracted into a visually configured one, which can be reused among multiple projects through configuration. The data integrators of the same data and the same development method in different projects may also have different versions.

[0064] In the specific vehicle log collection, the collected files are stored in order according to the project number, line number, vehicle number, log generation time, etc. HDFS (Hadoop Distributed File System, a highly fault-tolerant system) file system storage is a type of data storage in this application. For different data types and data formats, this application also integrates different types of big data warehouse components such as Doris data warehouse, ES (ElasticSearch) log retrieval, IOTDB (Internet of Things Database) time series database, etc.

[0065] Data processing is performed on the stored log data. Data processing is divided into two stages: data cleaning and data development.

[0066] For phase 1: data cleaning, a batch task is mounted on the HDFS system to parse and clean the log data and store it in the main data table of the data model. In this process, batch tasks, data parsing, and data cleaning are involved. The batch task periodically scans the specified directory of HDFS to scan the log data files and uses the newly added log data files as the data input for this batch processing. Batch file parsing: according to the template of the specific vehicle log, the vehicle log files are parsed into a recognizable data structure.

[0067] In the batch processing process, data cleaning is required according to business needs. Some data rules are set to clean some data that does not meet the data standards. Data cleaning is also performed for repeated data within the data interval. Finally, the parsed data is cleaned into business master data that meets the data standards. This log data parsing mainly involves invalid data deletion, field filtering, and field name normalization mapping. The cleaned data is injected into the device model corresponding to the master data layer of the device model in real time.

[0068] The master data in the data model serves as the basis for the next step of analysis. Further light summarization of the master data is also called data aggregation. Usually, based on this summary data, a multi-disciplinary joint analysis of the rail transit problems found is conducted. Light summarization of the master data usually involves dimensional summarization, and further summarization of some data indicators or statistical indicators, such as passenger flow statistics, early and late parking statistics, EB (Empirical Bayes Statistics) distribution statistics, etc.

[0069] 2. Data intermittent retransmission module

[0070] This embodiment is aimed at data integration in the rail transit industry, and a discontinuous retransmission module is specially set up. In the data set, ordinary big data platforms only consider the data integration method, but in terms of data consistency, they completely rely on the business itself. This embodiment integrates the data intermittent retransmission function into the system. The intermittent retransmission module is used to manage the data integration task interface, monitor the abnormal situation of the integration task, manually fill in the time period that requires data intermittent retransmission, and start the execution of the task within the set time period of the task.

[0071] 3. Data Storage Module

[0072] This embodiment targets the heterogeneous data in the rail transit industry and integrates different types of big data storage components such as Doris, EleasticSearch, IotDB, HDFS, etc. HDFS is used as data lake storage, and the rest are used as data warehouses for main data storage and multi-professional data fusion analysis in rail transit.

[0073] 4. Data stream batch processing module

[0074] The data processing framework of this embodiment supports both real-time stream data processing and batch data processing.

[0075] In the data governance method of this embodiment, real-time stream data processing is supported. For real-time stream data obtained from channels such as sensors and queues, this platform uses real-time stream processing in stream processing mode to process data. This application mainly uses FLINK stream processing, Seatunnel stream processing, and stream processing methods of custom data integrators.

[0076] In the data governance method of this embodiment, batch processing is supported for data processing. Batch processing tasks are mainly aimed at integrating data sources from offline resources such as databases and file systems through batch processing mode. The batch processing methods used in this application mainly include HTTP batch processing, FLINK batch processing, Seatunnel batch processing, Spark batch processing, and batch processing mode of custom data integrators.

[0077] 5. Task Scheduling Module

[0078] refer to Figure 2 As shown, in the data governance method of this embodiment, all data integration, data development and other tasks of this embodiment have a dedicated task scheduling module, and the task scheduling of this embodiment adopts the DolphinScheduler (distributed scheduling platform) scheduling framework. The task scheduling module realizes real-time monitoring, scheduling, etc., and manages the start and stop, failure, notification, etc. of all tasks.

[0079] In this embodiment, real-time stream processing tasks do not need to be timed, and batch processing tasks support setting batch cycles. This embodiment uses the Cron (scheduled task) mechanism to execute tasks on a regular basis. For example, this embodiment involves a 5-minute passenger flow statistics task for large passenger flows. The task scheduling module mainly manages the tasks by starting tasks, pausing tasks, stopping tasks, monitoring task status, task failure alarms, task failure recovery, etc., and supports the logging function of task operation.

[0080] 6. Data Model

[0081] refer to Figure 5 As shown, in the data governance method of this embodiment, the key data elements of the entire life cycle of similar equipment are abstractly extracted through the working principle of rail transit equipment, sampling principle and application experience to define the equipment model. The establishment of the equipment model specifically includes: abstractly extracting data elements of the entire life cycle of similar equipment through the working principle of equipment, sampling principle and application experience.

[0082] Establish a unique equipment relationship map for rail transit, and automatically associate the interlocking / power supply / transmission / location / vehicle-ground data relationship of the equipment. Provide support for cross-equipment anomaly and fault analysis. Focus on useful data, establish equipment models, and define the relationship between the equipment and upstream and downstream related equipment, and the relationship between related equipment in fusion analysis.

[0083] The rail transit data model of the present application obtains the relationship between each of the equipment models based on the established equipment model, and establishes a fogged view of the main data model of the equipment model based on the relationship between each of the equipment models. The data of the fogged view changes with the data of the main data model, thereby achieving the effect of changing a certain parameter in the system and causing the data of the entire system to change in conjunction.

[0084] The rail transit data model of this embodiment takes advantage of rail transit business experience, abstracts and extracts key data elements of the entire life cycle of similar equipment through equipment working principles, sampling principles, and application experience to define the equipment model. Focus on useful data, establish equipment models, and define the relationship between the equipment and upstream and downstream related equipment, and the relationship between related equipment for fusion analysis. Establish a unique equipment relationship map for rail transit, and automatically associate the interlocking / power supply / transmission / location / vehicle-ground data relationship of the equipment. Provide support for cross-equipment anomaly and fault analysis.

[0085] There are two main types of device data changes at each level in the data model: 1) triggered data changes; 2) periodic summary data.

[0086] For example, for the normalized master data device object data, the source data after cleansing is received, and the master data fact table is updated according to the rules. In this way, when new data is input after data cleansing, the data in the master data fact table will be updated, and the data in the corresponding related views will also be updated accordingly.

[0087] For periodic summary data, the data model is generally driven by periodic data development tasks. This type of model is bound to specific data development tasks, which are generally the process of layer-by-layer aggregation of master data fact tables.

[0088] 7. Data assets

[0089] This embodiment stores data in a big data platform and provides a data asset function, so that users can view the multi-source heterogeneous data stored in the current big data platform through data assets.

[0090] Data assets list in detail the usage of various heterogeneous databases, including detailed information on the data, data entry time, data owner, data version, and data storage structure, as well as data Schema (used to describe the structure and organization of data), data sampling information, etc.

[0091] This application supports users to view the details of the data within the user's authority scope on the current big data platform.

[0092] To support data warehouse construction, this embodiment also considers the idea of ​​dividing data assets by project and data warehouse level. This application supports the idea of ​​hierarchical management by folder under database type, and supports data warehouse stratification by folder under specific database.

[0093] This embodiment stores data in a big data platform and provides a data asset function, so that users can view the multi-source heterogeneous data stored in the current big data platform through data assets.

[0094] This embodiment currently supports the viewing of data assets of storage types such as Redis, MySQL, Doris, IotDB, HDFS, etc. When a user uses data assets to view, the data assets that can be viewed are the user assets that the user is authorized to view, subject to user permissions.

[0095] In this embodiment, real-time data is stored in the data warehouse. The schema information of the relevant tables stored in the warehouse, the user information of the uploaded data, the data creation time information, the data update time information, the data sampling examples, etc. can be viewed through the data assets.

[0096] In order to support the construction of data warehouses, this embodiment also takes into account the idea of ​​dividing data assets by project and data warehouse level. For this embodiment, the data warehouse is divided into four layers, ODS layer, DWD / DWM layer, DWS layer, and ADS layer. The ODS layer stores the source layer data and stores all required integrated source data. It can save the source data for subsequent analysis to avoid storing filtered data, which is not convenient for data requirements for new business expansion. It can also be used to compare data consistency with the data on the collection end. The DWD / DWM layer is a detailed wide table layer used to store complete and detailed historical data. The DWS layer is the core data layer. The ADS layer belongs to the business application layer and is specifically used to connect to upper-level businesses.

[0097] 8. User Rights Management Module

[0098] The platform method of this application supports permission control, including user permission control and data permission control. The user permission control of this application supports multi-tenant control, user permission authentication, and specifically supports user login, resource access authentication, etc. The data permission control of this application supports strict data permission access methods for specific tenants to a certain type of database, a certain table, and a certain row. Data permission control supports that users of different tenants can only see and use data within their own permission range.

[0099] 9. Data service sharing.

[0100] This application supports data sharing, which is used to realize service sharing between multiple data users and tenants. This application supports three types of data sharing: general data access interface of big data platform, data service market, and real-time data stream sharing.

[0101] The big data platform general data access API interface specifies the data access interface specifications and access methods. Through this interface, the big data platform provides the required data to the authenticated users. The big data platform general data access API adopts the RESTful API interface and the Post request method. The request body carries the specific access request content. The interface specifies batch access requests to meet the requirements of batch pull of large data requests.

[0102] The data service market can view a list of all service APIs provided by this platform, and users can find the corresponding business API through this list.

[0103] The data governance method of the present application supports online development and testing of data service APIs, which is used to support users to quickly develop data services that need to support new businesses online, and to test service APIs online.

[0104] Real-time data stream sharing supports sharing of real-time stream data between different users. Currently, the real-time stream data sharing methods supported by this application include Redis data and real-time stream data sharing in message queues.

[0105] 10. Engineering configuration of development tasks.

[0106] Considering that data and tasks in the platform can be reused between multiple projects, this application supports data and task sharing between such platforms of different projects. The configuration of IP and ports involved in data integration and data development tasks may be different for each project or environment. Such parameters will be represented by global engineering parameters through a global engineering mapping table, and specific IP or port class parameters will be represented by global engineering parameters, thus realizing task migration between different projects.

[0107] Task parameters are mapped through a global engineering table. For example, Doris has different IP and port numbers in different projects on the platform, but each task uses engineering parameters such as DorisIP, DorisPort, and DorisDB1. Therefore, when tasks are migrated between different platforms, it is very convenient to migrate tasks across projects on the same platform.

[0108] The overall project of this embodiment is deployed and operated based on K8S (kubernetes, container orchestration engine), supports rapid recovery from service-level failures, and supports the deployment of microservice mechanisms.

[0109] On the other hand, the present embodiment also provides a rail transit operation data governance method based on big data, which is implemented by the rail transit operation data governance system based on big data, and the method includes: the data stream batch integrated processing module uses the real-time stream processing method in the real-time stream processing mode to process data according to the real-time stream data obtained from the sensor or data integration module; the data stream batch integrated processing module also uses the processing method of the integrated data source in the batch processing mode to process data according to the offline resources in the database or file system. Through the working principle of rail transit equipment, sampling principle and application experience, the data elements in the whole life cycle of similar equipment are abstractly extracted to establish the equipment model. The data linkage realized by the equipment model specifically includes: according to the established equipment model, the relationship between each of the equipment models is obtained, and according to the relationship between each of the equipment models, the atomized view of the master data model of the equipment model is established, and the data of the atomized view changes with the data of the master data model, so as to realize that when a certain parameter in the system changes, the data linkage in the whole system changes. The data sharing method of the data service module includes: one of the general data access interface of the big data platform, the data service market, and the real-time data stream sharing, or any combination thereof.

[0110] An electronic device includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the above-mentioned rail transit operation data management method based on big data is implemented.

[0111] A readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned rail transit operation data governance method based on big data is implemented.

[0112] In summary, this embodiment proposes a rail transit operation data governance system based on big data, which is a complete platform system for multi-professional heterogeneous data storage and multi-professional data governance suitable for rail transit projects. It is a complete rail transit operation data governance platform from multi-source heterogeneous data collection, multi-source heterogeneous data storage, data processing, asset management, rail transit data model, data sharing, authority management and engineering. The rail transit operation data governance system and method of this embodiment support two methods of data integration, supporting the currently popular data integration components Kettle, Seatunnel and FLINK, and also supporting specific data application protocols such as Modbus commonly used in the rail transit field. The data integrator method implemented and encapsulated by the data integration component is used to achieve plug-and-play of the data integration plug-in, and supports user-defined registration of the data integrator. This embodiment sets a data stream and batch integrated processing module and adopts a stream-batch integrated rail transit data governance framework to create a data platform suitable for rail transit data governance, which meets the needs of various data governance methods such as operation and maintenance, operation control, etc., rather than just supporting big data batch processing. This embodiment adopts a device-oriented rail transit data model, abstracts the rail transit equipment into the corresponding rail transit equipment model, and realizes the layering of equipment models and the data linkage of equipment models, which facilitates the implementation of data integration, extraction, analysis, linkage and other services, and creates a unique and professional rail transit data model, covering basically all rail transit operation and maintenance equipment, and supports user-defined registration of rail transit data models, and supports the need to add new rail transit equipment models for special business scenarios. The rail transit operation data governance system of this embodiment takes into account the idea of ​​project engineering and creates engineering management, making reusable data integrators, rail transit data models, data integration and data development tasks reusable in different projects, rather than just a general data processing platform. The rail transit operation data governance system of this embodiment focuses on data consistency, and considers data consistency issues from the aspects of source data management and intermittent data retransmission to ensure data integrity and consistency.

[0113] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0114] It should be noted that the devices and methods disclosed in the embodiments of this article can also be implemented in other ways. The device implementation described above is only schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of this article. In this regard, each box in the flowchart or block diagram can represent a part of a module, program or code, and the module, program segment or part of the code contains one or more executable instructions for implementing the specified logical function, and the module, program segment or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented by a dedicated hardware-based system for performing a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0115] In addition, the functional modules in the various embodiments of this document may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0116] Although the content of the present invention has been described in detail through the above preferred embodiments, it should be appreciated that the above description should not be considered as a limitation of the present invention. After reading the above content, it will be apparent to those skilled in the art that various modifications and substitutions of the present invention will occur. Therefore, the protection scope of the present invention should be limited by the appended claims.

Claims

1. A rail transit operation data management system based on big data, characterized in that: include: A data integration module, which is used to access multi-source heterogeneous data of rail transit operations and perform data integration on the multi-source heterogeneous data; A data stream and batch integrated processing module, which is connected to the data integration module and is used to process the multi-source heterogeneous data after data integration to obtain processed data. The data stream and batch integrated processing module includes a data processing framework for supporting real-time stream data processing and batch data processing; A data storage module, which is connected to the data stream batch integrated processing module, and the data storage module includes different types of big data storage components for data storage and fusion analysis of the processed data; A data model definition module is connected to the data storage module, and establishes a device model according to the data in the data storage module, and defines the relationship between the current device and its upstream and downstream devices, and realizes data linkage through the device model to facilitate data integration; The data service module is connected to the data storage module and is used to realize service sharing among multiple users and tenants.

2. The rail transit operation data management system based on big data according to claim 1, characterized in that: Also includes: The data intermittent retransmission module is connected to the data integration module and is used to manage the data integration task interface of the data integration module, monitor the integration task status of the data integration module, and start the execution of tasks within the time period by manually filling in the time period that requires data intermittent retransmission.

3. The rail transit operation data management system based on big data according to claim 1, characterized in that: The data integration module includes: A configurable open source data integration plug-in, which is used for data integration of multi-source heterogeneous data; A variety of data integrators are used to select a corresponding data integrator for data integration through the data application protocol and the data integration method of the data integration module.

4. The rail transit operation data management system based on big data according to claim 1, characterized in that: Also includes: The task scheduling module is connected to the data integration module, the data stream batch integrated processing module and the data service module respectively, and is used for scheduling all tasks in the management system.

5. The rail transit operation data management system based on big data according to claim 4, characterized in that: The task scheduling module adopts the task scheduling framework of the distributed scheduling platform to manage the start and stop, failure and notification of all tasks in the system.

6. The rail transit operation data management system based on big data according to claim 1, characterized in that: Also includes: The data asset management module is connected to the data storage module and is used to view the multi-source heterogeneous data information stored in the current big data platform.

7. The rail transit operation data management system based on big data according to claim 1, characterized in that: It also includes: a permission management module, which is connected to the data storage module, the permission management module includes a user permission management unit and a data permission management unit, the user permission management unit is used for multi-tenant management and user permission authentication, and the data permission management unit is used to manage the data permission access of a specific tenant, a certain type of database, a certain data table and a certain row.

8. The rail transit operation data management system based on big data according to claim 1, characterized in that: Also includes: A task engineering configuration module is developed, which is used for the migration of tasks and data of different projects in the system.

9. A rail transit operation data management method based on big data, characterized in that: It is implemented by using the rail transit operation data management system based on big data as described in any one of claims 1 to 8, and the method includes: the data stream batch integrated processing module processes the data using the real-time stream processing method in the real-time stream processing mode according to the real-time stream data obtained from the sensor or the data integration module; The data stream batch integrated processing module also processes data based on offline resources in the database or file system through the processing method of the integrated data source in the batch processing mode.

10. The rail transit operation data management method based on big data as claimed in claim 9, characterized in that: Through the working principle of rail transit equipment, sampling principle and application experience, data elements of similar equipment in the entire life cycle are abstractly extracted to establish the equipment model.

11. The rail transit operation data management method based on big data as claimed in claim 10, characterized in that: The data linkage achieved through the device model specifically includes: obtaining the relationship between each of the device models based on the established device model, and establishing a fogged view of the master data model of the device model based on the relationship between each of the device models, the data of the fogged view changes with the data of the master data model, so as to realize the data linkage change in the entire system when a certain parameter in the system changes.

12. The rail transit operation data management method based on big data as claimed in claim 9, characterized in that: The data sharing mode of the data service module includes: a general data access interface of a big data platform, a data service market, and real-time data stream sharing, or any combination thereof.

13. An electronic device, characterized in that: include: A processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the rail transit operation data management method based on big data as described in any one of claims 9 to 12 is implemented.

14. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the rail transit operation data management method based on big data as described in any one of claims 9 to 12.

Images