Hierarchical data processing method and system for rail transit

Abstract

The invention discloses a rail transit-oriented hierarchical data processing method and system, and the method comprises the steps: constructing data processing logic through the combination of SQL statements according to business demands; a data processing flow is designed based on the layering theory of the data warehouse, the data processing flow is automatically executed according to the data processing logic, and the data processing flow comprises the steps that original data are obtained from a data source and subjected to format processing; performing data quality inspection, data cleaning and data conversion on the original data after format processing; performing data summarization and data statistics on the cleaned and converted original data to generate an intermediate data table; and performing data extraction and / or data analysis according to the intermediate data table. According to the method, the construction of the data processing logic is simplified into the combination of the SQL statements, so that the technical threshold is reduced, and the development and maintenance efficiency is remarkably improved. The data processing flow is designed according to the layering theory of the data warehouse, and modular and hierarchical management is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of rail transit data processing, and in particular to a layered data processing method and system for rail transit. Background Art

[0002] Rail transit usually refers to a transportation system where operating vehicles need to travel on specific tracks. It is a complex system that integrates multiple professions and types of work. It is mainly composed of track routes, stations, vehicles, maintenance and repair bases, power supply and transformation, communication signals, command and control centers, etc. Common rail transit includes traditional railways (ordinary railways, intercity railways and urban railways), subways, light rails and trams.

[0003] Due to the diversified development of train and railway technology, rail transit presents more and more types, and the complexity of rail transit transportation system itself. Rail transit system generates a wide variety of data with huge amount, which also puts forward higher requirements for efficient management and processing of data.

[0004] However, traditional data processing solutions usually rely on professional programming languages ​​to implement data logic processing. The process of code writing and debugging is relatively complicated, with high technical barriers, which is not conducive to rapid iteration and function expansion, and the cost of development and maintenance is high. In addition, the data processing process in the rail transit industry is relatively complicated. When processing massive amounts of data, there are often problems such as lengthy processing processes and unclear data flow, which makes it difficult to meet the rail transit industry's requirements for real-time and efficiency. Summary of the invention

[0005] The purpose of the present invention is to provide a hierarchical data processing method and system for rail transit, which is used to address the problem that the existing data processing solutions are too dependent on professional programming languages ​​and the data processing process is complicated and lengthy. By simplifying the construction of data processing logic into a combination of SQL (Structured Query Language) statements, the technical threshold is lowered, and the complex code writing process is avoided, which significantly improves the development and maintenance efficiency and helps to reduce costs. By designing the data processing process according to the hierarchical theory of the data warehouse, the complexity is reduced, modular and hierarchical management is achieved, and the efficiency of data processing is improved.

[0006] In order to achieve the above objectives, the present invention is implemented by the following technical solutions:

[0007] A first aspect of the present invention provides a hierarchical data processing method for rail transit, comprising: constructing data processing logic through a combination of SQL statements;

[0008] The data processing flow is designed based on the hierarchical theory of the data warehouse, and the data processing flow is automatically executed according to the data processing logic. The data processing flow includes:

[0009] Get the original data from the data source and process it in a format;

[0010] Performing data quality check, data cleaning and data conversion on the raw data after format processing;

[0011] Summarizing and counting the cleaned and converted raw data to generate an intermediate data table;

[0012] Data extraction and / or data analysis are performed according to the intermediate data table.

[0013] Optionally, constructing the data processing logic by combining the SQL statements includes:

[0014] According to business needs, a preset SQL syntax template is selected, and the SQL statement is generated based on the SQL syntax template to construct the data processing logic.

[0015] Optionally, the SQL syntax template includes at least the following data processing operations: data cleaning, data aggregation, data statistics and data analysis.

[0016] Optionally, generating the SQL statement based on the SQL grammar template includes:

[0017] Directly call the SQL syntax template to generate the SQL statement; or

[0018] Adjustments are made based on the SQL grammar template to generate the SQL statement.

[0019] Optionally, constructing the data processing logic by combining the SQL statements includes:

[0020] During the data processing process, the SQL statements are dynamically generated according to business requirements to construct and adjust the data processing logic.

[0021] Optionally, the configuration content of the data processing flow is adjusted according to business needs;

[0022] The configuration content at least includes: data source configuration, data processing logic configuration, task scheduling configuration and multi-environment configuration.

[0023] Optionally, adjusting the data source configuration includes: adding, modifying or deleting the data source; and setting connection information, data format and data refresh frequency of each data source.

[0024] Optionally, adjusting the task scheduling configuration includes: performing timed scheduling, event-triggered scheduling and batch scheduling on the data processing tasks; and setting the execution frequency and priority of each of the data processing tasks according to business needs.

[0025] Optionally, adjusting the multi-environment configuration includes:

[0026] Adjust the configuration information that needs to be modified according to different deployment environments;

[0027] The configuration information includes at least: a database connection address, a Kafka connection address, and a file transfer size limit.

[0028] Optionally, the data source includes at least: track equipment data, train operation data and passenger flow data.

[0029] A second aspect of the present invention provides a hierarchical data processing system for rail transit, comprising:

[0030] A logic generation module, the logic generation module is used to construct data processing logic through a combination of SQL statements; and

[0031] A data processing module, which is used to design a data processing flow based on the hierarchical theory of the data warehouse and automatically execute the data processing flow according to the data processing logic. The data processing module includes:

[0032] The original data layer is used to obtain and store original data from a data source and perform format processing on the original data;

[0033] A detailed data layer, the detailed data layer is used to perform data quality inspection, data cleaning and data conversion on the original data after format processing;

[0034] A summary data layer, the summary data layer is used to summarize and count the raw data after cleaning and conversion, and generate an intermediate data table; and

[0035] The analytical data layer is used to perform data extraction and / or data analysis based on the intermediate data table.

[0036] Optionally, a SQL syntax template is preset in the logic generation module;

[0037] The logic generation module is used to generate the SQL statement based on the SQL syntax template to construct the data processing logic.

[0038] Optionally, the logic generation module is used to dynamically generate the SQL statement during data processing to construct and adjust the data processing logic.

[0039] Optionally, the hierarchical data processing system further comprises a configuration management module, and the configuration management module is used to adjust the configuration content of the data processing flow;

[0040] The configuration content at least includes: data source configuration, data processing logic configuration, task scheduling configuration and multi-environment configuration.

[0041] Optionally, the data source includes at least: track equipment data, train operation data and passenger flow data.

[0042] A third aspect of the present invention provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the method described in any one of the first aspects is implemented.

[0043] A fourth aspect of the present invention provides a readable storage medium, wherein the readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of the above-mentioned first aspects is implemented.

[0044] The present invention has at least the following technical effects:

[0045] By introducing SQL to build data processing logic and using SQL to replace code writing, the construction of data processing logic is simplified to a combination of SQL statements. This not only lowers the technical threshold for data processing, but also avoids the complex code writing process, significantly improving development and maintenance efficiency.

[0046] By applying the hierarchical theory of data warehouse to the data processing scenarios of the rail transit industry, the data processing process of the rail transit industry is hierarchically structured. The complex data processing process of the rail transit industry is optimized, and the modular and hierarchical management of data flows is realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A schematic flow chart of a hierarchical data processing method for rail transit provided by an embodiment of the present invention;

[0048] Figure 2 A schematic diagram of creating a data processing pipeline through SQL provided by an embodiment of the present invention;

[0049] Figure 3 A schematic diagram of creating data processing logic through SQL provided by an embodiment of the present invention;

[0050] Figure 4 A schematic diagram of layering according to the layering theory of digital warehouse provided by one embodiment of the present invention;

[0051] Figure 5 An engineering configuration business information diagram provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following is a further detailed description of a hierarchical data processing method and system for rail transit proposed by the present invention in combination with the accompanying drawings and specific implementation methods. According to the following description, the advantages and features of the present invention will be clearer. It should be noted that the accompanying drawings are in a very simplified form and use non-precise proportions, which are only used to conveniently and clearly assist in explaining the purpose of the implementation method of the present invention. In order to make the purposes, features and advantages of the present invention more obvious and easy to understand, please refer to the accompanying drawings. It should be noted that the structure, proportion, size, etc. illustrated in the accompanying drawings of this specification are only used to match the contents disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the limiting conditions for the implementation of the present invention, so they have no technical substantive significance. Any modification of the structure, change in the proportional relationship or adjustment of the size, without affecting the effects that the present invention can produce and the purposes that can be achieved, should still fall within the scope of the technical content disclosed by the present invention.

[0053] In order to solve the problems existing in the existing rail transit data processing solutions, the first aspect of this embodiment provides a hierarchical data processing method for rail transit, such as Figure 1 As shown, the method includes: constructing data processing logic through a combination of SQL statements; designing a data processing flow based on the hierarchical theory of data warehouse, and automatically executing the data processing flow according to the data processing logic. SQL, i.e., structured query language, is a database query and programming language used to access data and query, update and manage relational database systems. As a relatively simple and easy-to-understand query language, constructing data processing logic through a combination of SQL statements allows business personnel to complete data processing through SQL without programming skills, which is conducive to simplifying the data processing flow.

[0054] The above-mentioned data warehouse generally refers to a system that centrally stores a large amount of data to support the decision-making process of an organization. The hierarchical theory of data warehouse is one of the foundations of data warehouse construction and design. Its purpose is to store and manage data of different types and processing stages in a hierarchical manner. Figure 1As shown, based on the hierarchical theory of data warehouse and combined with the characteristics of the rail transit industry, the data processing process may include: obtaining raw data from the data source and performing format processing; performing data quality inspection, data cleaning and data conversion on the raw data after format processing; performing data aggregation and data statistics on the raw data after cleaning and conversion to generate an intermediate data table; performing data extraction and / or data analysis based on the intermediate data table. Among them, the data source includes at least: track equipment data, train operation data and passenger flow data.

[0055] To create the data processing logic created by SQL, such as Figure 2 As shown in , we can first create a data processing pipeline using SQL. Figure 3 As shown, the data processing logic can be created through SQL. Specifically, in order to construct the data processing logic through the combination of the SQL statements, the user can select a preset SQL syntax template according to business needs, and generate the SQL statement based on the SQL syntax template to construct the data processing logic. The SQL syntax template can include some common data processing operations in the data processing flow. Specifically, the SQL syntax template includes at least the following data processing operations: data cleaning, data aggregation, data statistics and data analysis.

[0056] In order to generate the SQL statement based on the SQL syntax template, the SQL syntax template can be directly called to generate the SQL statement, or the SQL syntax template can be adjusted to generate the SQL statement to meet specific business needs. By adjusting the SQL syntax template, the details of operations such as data cleaning, data aggregation, data statistics and data analysis in the data processing process can be directly adjusted, and the adaptability is better.

[0057] Since there is a large amount of real-time data in the rail transit system, in order to enhance the flexibility of SQL in the data processing process, this embodiment supports the dynamic generation of SQL statements. That is, in the process of data processing, the SQL statements can be dynamically generated according to business needs to build and adjust the data processing logic, and further automate the execution of the data processing logic. According to the user's requirements, the data processing flow can be automatically performed by node, and there is no need to write the SQL statements separately for each processing node. In this way, the SQL logic can be automatically adjusted according to the changes in the actual data, so that real-time processing of dynamic data can be achieved, greatly improving the efficiency and response speed of data processing. While improving the flexibility of the system, the processing capability of real-time rail transit data is also significantly enhanced.

[0058] The following code is an example of dynamically generating the SQL statement:

[0059] Data is stored in the data warehouse interface regularly:

[0060] public void executeSelect(){

[0061] List <datasourceentity>dataSourceEntities=dymDataSource.getDataSourcelist();

[0062] for(DataSourceEntity dataSourceEntity:dataSourceEntities){

[0063] if("I0TDB".equals(dataSourceEntity.getDatasource())){

[0064] List <deviceentity>devices=

[0065] metaMangerService.selectDevice(dataSourceEntity.getPoolName());

[0066] for(DeviceEntity device:devices

[0067] ){

[0068] metaMangerService.updateDevice(dataSourceEntity.getPoolNa me(),device);

[0069] }

[0070] List <timeseriesentity>timeseriesEntities=

[0071] metaMangerservice.selectTimeseries(dataSourceEntity.getPoolName());

[0072] for(TimeseriesEntity timeseries:timeseriesEntities){

[0073] metaMangerService.updateTimeseries(dataSourceEntity.getPoolName(),timeserie s);

[0074] }

[0075] continue;

[0076] }

[0077] List <infoentity>infos=

[0078] metaMangerService.selectInformation(dataSourceEntity.getPoolName());

[0079] List <colentity>colEntities=

[0080] metaMangerService.selectColumn(dataSourceEntity,getPoolName());

[0081] for(InfoEntity info:infos){

[0082] metaMangerService.updateInformation(dataSourceEntity.getPoolName(),info);

[0083] }

[0084] for(ColEntity colEnty:colEntities){

[0085] metaMangerService.updateColumn(dataSourceEntity.getPoolName(),colEnty);

[0086] }

[0087] }

[0088] Through the above code, you can use the executeSelect method to traverse the data source list and perform corresponding query and update operations based on the data source type.

[0089] Among them, users can dynamically access platform data through SQL via API:

[0090] public DataselectEntity selectByMySql(string sql,string dbType,LongpageNo,stringtable){

[0091] DataSelectEntity dataSelectEntity=new DataSelectEntity();

[0092] Long offSet=(pageNo-1)*dataSelectEntity.getLimit();

[0093] List <Map<String,Object> >data=publishMapper.selectByMysql(sql,offset,dataselectEntity.getLimit());

[0094] dataSelectEntity.setData(data);

[0095] dataSelectEntity.setTotal(publishMapper.getTotalCount(table));

[0096] dataSelectEntity.setPage(pageNo);

[0097] return dataSelectEntity;

[0098] }

[0099] Through the above code, this embodiment can dynamically parse the SQL statement passed in by the user, extract keywords, and return the content of the user's query.

[0100] At the same time, this embodiment also supports the addition and deletion of dynamic multi-data sources:

[0101] public void autoLoad(){

[0102] List <datasourceentity>dbList = dymDataSource.getDataSourceList();

[0103] / / Traverse the data sources and load them

[0104] for (DataSourceEntity dataSourceEntity : dbList) {

[0105] / / Decrypt

[0106] dataSourceEntity.setPassword(EncodeUtil.Base64Decoder(dataSourceEntity.getPassword()));

[0107] DataSourceProperty dataSourceProperty = new DataSourceProperty();

[0108] BeanUtils.copyProperties(dataSourceEntity, dataSourceProperty);

[0109] DynamicRoutingDataSource ds = (DynamicRoutingDataSource)

[0110] dataSource;

[0111] DataSource dataSource =

[0112] defaultDataSourceCreator.createDataSource(dataSourceProperty),

[0113] ds.addDataSource(dataSourceEntity.getPoolName(), dataSource);

[0114] LOGGER.info("name:{},type:{}", dataSourceEntity.getPoolName(), dataSourceEntity.getDatasource());

[0115] }

[0116] Through the above code, the autoLoad method can be used to traverse the data source list, decrypt each of the data sources, and add the data source to the dynamic routing data source.

[0117] In view of the complex and changeable business needs of the rail transit industry, traditional data processing systems usually lack effective engineering configuration capabilities. Operations such as data source configuration and processing logic adjustment often need to be implemented by modifying the code, and there is a lack of unified configuration management tools. Therefore, this embodiment further introduces the design of engineering configuration to provide more flexible configuration management capabilities. Through engineering configuration, the configuration content of the data processing flow can be dynamically adjusted according to business needs to meet the needs of different scenarios. The configuration content mainly includes data source configuration, data processing logic configuration, task scheduling configuration and multi-environment configuration system.

[0118] Specifically, since the rail transit system often involves multiple data sources during the data processing process. Therefore, adjusting the data source configuration may include adding, modifying or deleting the data source. Through engineering configuration, users can quickly add, modify or delete the data source according to actual needs, which is conducive to flexible configuration of multiple data sources. Adjusting the data source configuration may also include setting parameters such as connection information, data format and data refresh frequency of each data source, so as to achieve dynamic management of the data source. Business personnel can dynamically add and delete the required data sources according to needs. The added data source supports modification of connection information, and multiple data source types can be selected, such as MySQL, Doris, IoTDB and other types.

[0119] In the data processing logic configuration, this embodiment allows users to directly adjust the data processing logic generated by SQL through the configuration file or interface. Users can define specific information data processing processes for different business scenarios and flexibly control the details of data cleaning, aggregation (summary and statistics) and analysis. Therefore, different processing logics can be implemented through configuration without modifying the code, thereby enhancing the adaptability of the system.

[0120] In order to adjust the task scheduling configuration, the data processing tasks can be scheduled regularly, triggered by events, and scheduled in batches. The execution frequency and priority of each of the data processing tasks can also be set according to business needs. Since rail transit data processing often requires scheduled or real-time task scheduling to meet business needs. This embodiment provides a flexible scheduling strategy for data processing tasks through engineering configuration, including scheduled scheduling, event-triggered scheduling, batch scheduling, etc. Users can set the execution frequency, priority and other parameters of each of the data processing tasks according to actual needs to ensure that the system can process data efficiently.

[0121] In order to adapt to the needs of different deployment environments (such as development, testing, and production environments), this embodiment supports configuration management of multiple environments, and adjusting the multi-environment configuration includes adjusting the configuration information that needs to be modified for different deployment environments. Users can flexibly switch configurations in different deployment environments to ensure that the system can quickly adapt to different business needs during the deployment process. Specifically, for example, the configurations that need to be modified in different environments can be exposed in configmap. When deployed in different environments, the configuration information of the server can be quickly modified by modifying configmap to avoid frequent modification of the image. The configuration information includes, for example, database connection address, Kafka connection address, file transfer size limit, etc. The main functions of the above-mentioned configmap include storing configuration data, injecting configuration data, and dynamically updating configuration. By using configmap, the configuration information can be decoupled from the container image, so that the configuration can be managed and updated independently of the application, thereby updating the configuration without rebuilding the image.

[0122] This embodiment introduces the concept of engineering configuration to meet the actual needs of rail transit data processing, and supports flexible configuration management functions such as multi-data source configuration, processing logic configuration, and task scheduling configuration. Through this configuration method, the system can adapt to the changing business needs of the rail transit industry, which not only improves the flexibility and scalability of the system, but also enhances the reliability and ease of use of the system when dealing with complex data processing scenarios. This configurable design enables the system to be quickly adjusted in different environments and with different needs, improving the adaptability and scalability of the system.

[0123] In order to implement the above-mentioned hierarchical data processing method for rail transit, the second aspect of this embodiment provides a hierarchical data processing system for rail transit, for example, including a logic generation module and a data processing module, the logic generation module is used to construct data processing logic through a combination of SQL statements, and the data processing module is used to design a data processing flow based on the hierarchical theory of the data warehouse, and automatically execute the data processing flow according to the data processing logic.

[0124] Based on the hierarchical theory of data warehouse, such as Figure 4 As shown, the data processing module may include, for example, an original data layer, a detailed data layer, a summary data layer and an analysis data layer.

[0125] The raw data layer is used to obtain and store raw data from the data source and perform format processing on the raw data. The raw data layer can be connected to various subsystems of rail transit, such as the equipment management system, train operation monitoring system, and passenger flow management system. Thus, rail equipment data, train operation data, and passenger flow data are obtained from the equipment management system, train operation monitoring system, and passenger flow management system as the data source. The data in the raw data layer undergoes simple format processing to ensure the integrity and consistency of the data, providing a basis for subsequent data processing.

[0126] The detailed data layer is used to perform data quality inspection, data cleaning and data conversion on the raw data after format processing. Specifically, the data cleaning may include steps such as data deduplication, missing value filling, and outlier processing. Through the processing of the detailed data layer, the reliability and accuracy of the data can be guaranteed, providing high-quality data input for the subsequent analysis data layer.

[0127] The summary data layer is used to summarize and count the raw data after cleaning and conversion, and generate intermediate data tables. Specifically, the summary data layer can use star modeling to summarize and count the raw data after cleaning in multiple dimensions, and generate intermediate data tables that can be used by business personnel and systems. The main purpose of designing the summary data layer is to improve the efficiency of data query and processing, reduce direct access to the unprocessed raw data, and avoid repeated calculations. By pre-processing and storing high-frequency data in the summary data layer, the response speed of the system can be effectively improved.

[0128] The analysis data layer is used to extract data and / or perform data analysis based on the intermediate data table. As the final level of the data processing architecture, the analysis data layer can generate analysis results or provide decision support for specific business needs. Users can directly extract the required data from the analysis data layer through SQL, or perform deeper analysis operations on the analysis data layer to provide support for operational decisions of the rail transit system. The analysis data layer provides some data analysis functions, such as historical data growth statistics, project distribution statistics, etc. Business personnel can select different components in the data integration and data development modules to perform business data analysis according to business needs, such as alarm data statistics, data quality analysis, etc.

[0129] The logic generation module is preset with an SQL syntax template, so that the SQL statement can be generated based on the SQL syntax template to construct the data processing logic. The logic generation module can also dynamically generate the SQL statement during data processing to construct and adjust the data processing logic.

[0130] The hierarchical data processing system further comprises a configuration management module, which is used to adjust the configuration content of the data processing flow, wherein the configuration content at least comprises data source configuration, data processing logic configuration, task scheduling configuration and multi-environment configuration.

[0131] The hierarchical data processing system may be, for example, a big data platform that provides a data processing interface. The front end of the big data platform may include, for example, a display module that can perform human-computer interaction, and the back end of the big data platform may include, for example, the logic generation module and the data processing module. The display module may be used to display data processing logic, select and adjust the SQL syntax template, and input business requirements. The data processing logic may be displayed in an interface through the display module, which is more intuitive and easier to modify and maintain.

[0132] The user can select the corresponding function in the display module according to the business needs, and the backend automatically generates SQL statements and processes data according to the needs, without the need to manually write SQL statements. This avoids cumbersome business processes and inconvenient modifications. By further presetting the SQL syntax template, the threshold for data processing can be significantly lowered.

[0133] like Figure 5 As shown, the display module may also include a configuration management interface, which provides a workflow definition function for realizing the visualization of the configuration management module. The above-mentioned workflow may include a series of interconnected, automatic data processing tasks.

[0134] Specifically, the user can quickly add, modify or delete the data source according to actual needs in the configuration management interface, and can also set the connection information, data format and data refresh frequency of each data source.

[0135] The user can select configurations related to the data processing flow through different options on the configuration management interface, such as SQL type, data source, task priority, etc., so as to adjust the data processing logic. The user can also directly adjust the data processing logic composed of the SQL statements by inputting configuration files on the configuration management interface, avoiding direct code modification.

[0136] Users can create the required workflow components in the configuration management interface and set the workflow timing. They can control the time of workflow online and offline to achieve timed scheduling. They can also set the trigger conditions of the workflow to achieve event-triggered scheduling. They can also modify the workflows with user-configured names in batches through engineering configuration in the configuration management interface to achieve batch scheduling. Users can set parameters such as execution frequency and priority according to the degree of urgency to determine the sequential logic of different tasks.

[0137] The user can adjust the configuration information that needs to be modified for different environments in the configuration management interface. Specifically, the big data platform can expose the configuration information that needs to be modified in different deployment environments in configmap. Therefore, when deployed in different environments, the configuration information of the server can be quickly modified by modifying the configmap to avoid frequent modification of the image. The configuration information includes, for example, the database connection address, Kafka connection address, file transfer size limit, etc.

[0138] In other aspects, this embodiment further provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the above-mentioned layered data processing method for rail transit is implemented.

[0139] In other aspects, this embodiment further provides a readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the above-mentioned layered data processing method for rail transit is implemented.

[0140] The present invention introduces SQL to construct data processing logic, and uses SQL to replace the code writing method, simplifying the construction of data processing logic into a combination of SQL statements. It not only lowers the technical threshold of data processing, but also avoids the complicated code writing process, significantly improving the development and maintenance efficiency. By applying the hierarchical theory of data warehouse to the data processing scenario of the rail transit industry, the data processing flow of the rail transit industry is hierarchically structured. The complex data processing flow of the rail transit industry is optimized, and the modular and hierarchical management of data flow is realized, which is a significant improvement compared with the existing technology.

[0141] It should be noted that, in this article, the term "comprises", "includes" or any other variation thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0142] It should be noted that the devices and methods disclosed in the embodiments of this article can also be implemented in other ways. The device implementation described above is only schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of this article. In this regard, each box in the flowchart or block diagram can represent a part of a module, program or code, and the module, program segment or part of the code contains one or more executable instructions for implementing the specified logical function, and the module, program segment or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented by a dedicated hardware-based system for performing a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0143] In addition, the functional modules in the various embodiments of this document may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0144] Although the content of the present invention has been described in detail through the above preferred embodiments, it should be appreciated that the above description should not be considered as a limitation of the present invention. After reading the above content, it will be apparent to those skilled in the art that various modifications and substitutions of the present invention will occur. Therefore, the protection scope of the present invention should be limited by the appended claims.< / datasourceentity> < / colentity> < / infoentity> < / timeseriesentity> < / deviceentity> < / datasourceentity>

Claims

1. A hierarchical data processing method for rail transit, characterized in that: include: Build data processing logic through a combination of SQL statements; The data processing flow is designed based on the hierarchical theory of the data warehouse, and the data processing flow is automatically executed according to the data processing logic. The data processing flow includes: Get the original data from the data source and process it in a format; Performing data quality check, data cleaning and data conversion on the raw data after format processing; Summarizing and counting the cleaned and converted raw data to generate an intermediate data table; Data extraction and / or data analysis are performed according to the intermediate data table.

2. The hierarchical data processing method for rail transit according to claim 1, characterized in that: The constructing the data processing logic by combining the SQL statements includes: According to business needs, a preset SQL syntax template is selected, and the SQL statement is generated based on the SQL syntax template to construct the data processing logic.

3. The hierarchical data processing method for rail transit according to claim 2, characterized in that: The SQL syntax template includes at least the following data processing operations: data cleaning, data aggregation, data statistics and data analysis.

4. The hierarchical data processing method for rail transit according to claim 2, characterized in that: Generating the SQL statement based on the SQL grammar template includes: Directly call the SQL syntax template to generate the SQL statement; or Adjustments are made based on the SQL grammar template to generate the SQL statement.

5. The hierarchical data processing method for rail transit according to claim 1, characterized in that: The constructing the data processing logic by combining the SQL statements includes: During the data processing process, the SQL statements are dynamically generated according to business requirements to construct and adjust the data processing logic.

6. The hierarchical data processing method for rail transit according to claim 1, characterized in that: Adjust the configuration content of the data processing flow according to business needs; The configuration content at least includes: data source configuration, data processing logic configuration, task scheduling configuration and multi-environment configuration.

7. The hierarchical data processing method for rail transit according to claim 6, characterized in that: Adjusting the data source configuration includes: Add, modify or delete the data source; The connection information, data format and data refresh frequency of each of the data sources are set.

8. The hierarchical data processing method for rail transit according to claim 6, characterized in that: Adjusting the task scheduling configuration includes: Perform timed scheduling, event-triggered scheduling, and batch scheduling for data processing tasks; According to business requirements, the execution frequency and priority of each data processing task are set.

9. The hierarchical data processing method for rail transit according to claim 6, characterized in that: Adjusting the multi-environment configuration includes: Adjust the configuration information that needs to be modified according to different deployment environments; The configuration information includes at least: a database connection address, a Kafka connection address, and a file transfer size limit.

10. The hierarchical data processing method for rail transit according to claim 1, characterized in that: The data source includes at least: track equipment data, train operation data and passenger flow data.

11. A hierarchical data processing system for rail transit, characterized in that: include: A logic generation module, wherein the logic generation module is used to construct data processing logic through a combination of SQL statements; and A data processing module, which is used to design a data processing flow based on the hierarchical theory of the data warehouse and automatically execute the data processing flow according to the data processing logic. The data processing module includes: The original data layer is used to obtain and store original data from a data source and perform format processing on the original data; A detailed data layer, the detailed data layer is used to perform data quality inspection, data cleaning and data conversion on the original data after format processing; A summary data layer, the summary data layer is used to summarize and count the raw data after cleaning and conversion, and generate an intermediate data table; and The analytical data layer is used to perform data extraction and / or data analysis based on the intermediate data table.

12. The hierarchical data processing system for rail transit according to claim 11, characterized in that: The logic generation module is preset with an SQL syntax template; The logic generation module is used to generate the SQL statement based on the SQL syntax template to construct the data processing logic.

13. The hierarchical data processing system for rail transit according to claim 11, characterized in that: The logic generation module is used to dynamically generate the SQL statement during the data processing process to construct and adjust the data processing logic.

14. The hierarchical data processing system for rail transit according to claim 11, characterized in that: The hierarchical data processing system further includes a configuration management module, and the configuration management module is used to adjust the configuration content of the data processing flow; The configuration content at least includes: data source configuration, data processing logic configuration, task scheduling configuration and multi-environment configuration.

15. The hierarchical data processing system for rail transit according to claim 11, characterized in that: The data source includes at least: track equipment data, train operation data and passenger flow data.

16. An electronic device, characterized in that: The method comprises a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the method according to any one of claims 1 to 10 is implemented.

17. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images