A data warehouse monitoring system for urban traffic big data platform

Abstract

The invention discloses a data warehouse monitoring system for an urban traffic big data platform, comprising a data acquisition system, a data analysis system, a traffic data master control platform, an expected quality import system, a data classification evaluation system, a data quality verification system and a quality problem analysis system. The invention has the following beneficial effects: by adding the data analysis system, the basic attributes of traffic data can be analyzed, so as to obtain an understanding of the data and the environment, and describe the status of the data; by adding the data quality verification system, the quality issues of integrity, consistency, completeness, effectiveness and timeliness of the data can be verified, and problems existing in the traffic data can be discovered; by adding the quality problem analysis system, the root cause of the problem can be discovered based on the quality problem of the data, so as to facilitate subsequent data processing; all-round monitoring of traffic data can be achieved, and the source of the data problem can be discovered, so as to facilitate corresponding processing measures.

Description

Technical Field

[0001] The present invention relates to the technical field of data warehouse monitoring, and in particular to a data warehouse monitoring system for an urban traffic big data platform. Background Art

[0002] Urban transportation refers to public travel and passenger and freight transportation in the urban road system. Before humans used vehicles as a means of transportation, urban public travel was mainly on foot, or by riding livestock or sedan chairs. Goods were mostly transferred on shoulders or using simple transportation tools. In order to facilitate real-time monitoring of urban traffic conditions, a terminal platform based on urban traffic big data will be built, and these data will be stored in the data warehouse within the platform. In order to ensure the quality of these data and ensure that the data will not be wrong, we propose a data warehouse monitoring system for the urban traffic big data platform. Summary of the invention

[0003] The purpose of the present invention is to provide a data warehouse monitoring system for an urban traffic big data platform to solve the problems raised in the above-mentioned background technology.

[0004] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: including a data acquisition system, a data analysis system, a traffic data master control platform, an expected quality import system, a data classification evaluation system, a data quality verification system and a quality problem analysis system, wherein the output end of the data acquisition system is electrically connected to the data analysis system, the output end of the data analysis system is electrically connected to the traffic data master control platform, the output end of the expected quality import system is electrically connected to the traffic data master control platform, the output end of the traffic data master control platform is electrically connected to the data classification evaluation system, the output end of the data classification evaluation system is electrically connected to the data quality verification system, and the output end of the data quality verification system is electrically connected to the quality problem analysis system;

[0005] The data acquisition system is used to collect big data of urban traffic from multiple sources;

[0006] The data analysis system is used to perform attribute-based analysis on the data obtained by the data acquisition system;

[0007] The traffic data master control platform is used to control and transfer the entire system;

[0008] The expected quality import system is used to import the expected data quality into the traffic data master control platform;

[0009] The data classification and evaluation system is used to analyze and preliminarily evaluate the data and classify the data into important data and non-important data;

[0010] The data quality verification system is used to verify various quality classifications of the classified data;

[0011] The quality problem analysis system is used to analyze the problems existing in data with quality problems.

[0012] As a preferred solution of the present invention: the output end of the traffic data master control platform is electrically connected to a monitoring result processing system, the output end of the quality problem analysis system is electrically connected to a monitoring result feedback system, the output end of the monitoring result feedback system is electrically connected to the monitoring result processing system, and the output end of the monitoring result processing system is electrically connected to a terminal display system;

[0013] The monitoring result feedback system is used to provide feedback on data quality issues and problem analysis results;

[0014] The monitoring result processing system is used to process the monitoring results of the data and generate corresponding lists and charts for easy observation;

[0015] The terminal display system is used to display the lists and charts generated by the monitoring result processing system.

[0016] As a preferred solution of the present invention: the quality problem analysis system includes a source system problem analysis module, an ETL problem analysis module, a scheduling system problem analysis module and a business demand problem analysis module.

[0017] As a preferred solution of the present invention: the source system problem analysis module is used to analyze the problems existing in the structure change, business process change, data entry, data evolution and data shock in the source data system;

[0018] The ETL problem analysis module is used to analyze the problems existing in data requirements, mapping rules, program errors and data timeliness in the ETL system;

[0019] The scheduling system problem analysis module is used to analyze the scheduling system problems caused by data delay and its own faults in the timing scheduling system;

[0020] The business demand problem analysis module is used to analyze the changes in data demand and the conflicts between the system and the business.

[0021] As a preferred solution of the present invention: the data quality verification system includes an integrity verification module, a consistency verification module, a completeness verification module, a validity verification module and a timeliness verification module.

[0022] As a preferred solution of the present invention: the integrity verification module is used to verify the degree of compliance with the data relationship rules;

[0023] The consistency verification module is used to verify the degree of data change and variation between data sets;

[0024] The completeness verification module is used to verify whether the data exists, attributes and quantity are complete;

[0025] The validity verification module is used to verify the degree of compliance of data with a set of business rules;

[0026] The timeliness verification module is used to verify the timeliness of data processing.

[0027] As a preferred solution of the present invention: the data analysis system includes a column attribute analysis module and a structure attribute analysis module.

[0028] As a preferred solution of the present invention: the column attribute analysis module is used to analyze the column type, column format, column length, high-frequency value column, low-frequency value column, date column, column validity, column value range, column default value, column mode, column null value and duplicate records of the data;

[0029] The structural attribute analysis module is used to analyze the referential integrity of data, dependencies between columns, data granularity, data accuracy, data hierarchy and business rules.

[0030] As a preferred solution of the present invention: the data acquisition system includes a metadata acquisition module and a warehouse data acquisition module.

[0031] As a preferred solution of the present invention: the metadata collection module is used to collect and obtain basic metadata in traffic data;

[0032] The warehouse data collection module is used to collect and acquire the storage data in the data warehouse.

[0033] Compared with the prior art, the beneficial effects of the present invention are as follows: by adding a data acquisition system, the metadata flowing through the traffic big data platform and the stored data in the data warehouse can be acquired, and the traffic data can be monitored in an all-round manner; by adding a data analysis system, the basic attributes of the traffic data can be analyzed, so as to obtain an understanding of the data and the environment, and describe the status of the data; by adding a data quality verification system, the quality issues of the integrity, consistency, completeness, validity and timeliness of the data can be verified, and problems with the traffic data can be discovered; by adding a quality problem analysis system, the root cause of the problem can be discovered based on the quality problem of the data, which is convenient for subsequent data processing; all-round monitoring of traffic data can be achieved, and the source of the data problem can be discovered, so as to facilitate corresponding processing measures. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a system block diagram of the present invention;

[0035] Figure 2 It is a system block diagram of the data acquisition system of the present invention;

[0036] Figure 3 It is a system block diagram of the data analysis system of the present invention;

[0037] Figure 4 It is a system block diagram of the data quality verification system of the present invention;

[0038] Figure 5 It is a system block diagram of the quality problem analysis system of the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] See also Figure 1 , Figure 2 , Figure 3 , Figure 4 and Figure 5 The present invention provides a technical solution: including a data acquisition system, a data analysis system, a traffic data master control platform, an expected quality import system, a data classification evaluation system, a data quality verification system and a quality problem analysis system, wherein the output end of the data acquisition system is electrically connected to the data analysis system, the output end of the data analysis system is electrically connected to the traffic data master control platform, the output end of the expected quality import system is electrically connected to the traffic data master control platform, the output end of the traffic data master control platform is electrically connected to the data classification evaluation system, the output end of the data classification evaluation system is electrically connected to the data quality verification system, and the output end of the data quality verification system is electrically connected to the quality problem analysis system;

[0041] The data acquisition system is used to collect big data of urban traffic from multiple sources;

[0042] The data analysis system is used to perform attribute-based analysis on the data obtained by the data acquisition system;

[0043] The traffic data master control platform is used to control and transfer the entire system;

[0044] The expected quality import system is used to import the expected data quality into the traffic data master control platform;

[0045] The data classification and evaluation system is used to analyze and preliminarily evaluate the data and classify the data into important data and non-important data;

[0046] The data quality verification system is used to verify various quality classifications of the classified data;

[0047] The quality problem analysis system is used to analyze the problems existing in data with quality problems.

[0048] The output end of the traffic data master control platform is electrically connected to a monitoring result processing system, the output end of the quality problem analysis system is electrically connected to a monitoring result feedback system, the output end of the monitoring result feedback system is electrically connected to the monitoring result processing system, and the output end of the monitoring result processing system is electrically connected to a terminal display system;

[0049] The monitoring result feedback system is used to provide feedback on data quality issues and problem analysis results;

[0050] The monitoring result processing system is used to process the monitoring results of the data and generate corresponding lists and charts for easy observation;

[0051] The terminal display system is used to display the lists and charts generated by the monitoring result processing system.

[0052] Wherein, the quality problem analysis system includes a source system problem analysis module, an ETL problem analysis module, a scheduling system problem analysis module and a business demand problem analysis module.

[0053] The source system problem analysis module is used to analyze the problems existing in the structure change, business process change, data entry, data evolution and data shock in the source data system;

[0054] The ETL problem analysis module is used to analyze the problems existing in data requirements, mapping rules, program errors and data timeliness in the ETL system;

[0055] The scheduling system problem analysis module is used to analyze the scheduling system problems caused by data delay and its own faults in the timing scheduling system;

[0056] The business demand problem analysis module is used to analyze the changes in data demand and the conflicts between the system and the business.

[0057] The data quality verification system includes an integrity verification module, a consistency verification module, a completeness verification module, a validity verification module and a timeliness verification module.

[0058] Wherein, the integrity verification module is used to verify the degree of compliance with the data relationship rules;

[0059] The consistency verification module is used to verify the degree of data change and variation between data sets;

[0060] The completeness verification module is used to verify whether the data exists, attributes and quantity are complete;

[0061] The validity verification module is used to verify the degree of compliance of data with a set of business rules;

[0062] The timeliness verification module is used to verify the timeliness of data processing.

[0063] Wherein, the data analysis system includes a column attribute analysis module and a structure attribute analysis module.

[0064] The column attribute analysis module is used to analyze the column type, column format, column length, high-frequency value column, low-frequency value column, date column, column validity, column value range, column default value, column mode, column null value and duplicate records of the data;

[0065] The structural attribute analysis module is used to analyze the referential integrity of data, dependencies between columns, data granularity, data accuracy, data hierarchy and business rules.

[0066] Wherein, the data collection system includes a metadata collection module and a warehouse data collection module.

[0067] Wherein, the metadata collection module is used to collect and obtain basic metadata in traffic data;

[0068] The warehouse data collection module is used to collect and acquire the storage data in the data warehouse.

[0069] Specifically, when in use, all-round traffic data is first collected through the data collection system, and then the data analysis system analyzes the attributes of the collected traffic data, so as to obtain an understanding of the data and the environment, and describe the status of the data. After that, the data is divided into important data and unimportant data through the data classification and evaluation system, and the data quality verification system is used to verify the integrity, consistency, completeness, validity and timeliness of the data according to the status of the data. When quality problems are found, the quality problems and the problematic data are transmitted to the quality problem analysis system, and the quality problem analysis system analyzes the source of the data quality problem, and then feeds back the processing structure to the monitoring structure processing system, and finally displays the problematic data and the source problem of the data on the terminal display system, so that the staff can make targeted processing.

[0070] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A data warehouse monitoring system for an urban traffic big data platform, comprising a data acquisition system, a data analysis system, a traffic data master control platform, an expected quality import system, a data classification evaluation system, a data quality verification system and a quality problem analysis system, characterized in that: The output end of the data acquisition system is electrically connected to the data analysis system, the output end of the data analysis system is electrically connected to the traffic data master control platform, the output end of the expected quality import system is electrically connected to the traffic data master control platform, the output end of the traffic data master control platform is electrically connected to the data classification and evaluation system, the output end of the data classification and evaluation system is electrically connected to the data quality verification system, and the output end of the data quality verification system is electrically connected to the quality problem analysis system; The data acquisition system is used to collect big data of urban traffic from multiple sources; The data analysis system is used to perform attribute-based analysis on the data obtained by the data acquisition system; The traffic data master control platform is used to control and transfer the entire system; The expected quality import system is used to import the expected data quality into the traffic data master control platform; The data classification and evaluation system is used to analyze and preliminarily evaluate the data and classify the data into important data and non-important data; The data quality verification system is used to verify various quality classifications of the classified data; The quality problem analysis system is used to analyze the problems existing in the data with quality problems; The output end of the traffic data master control platform is electrically connected to a monitoring result processing system, the output end of the quality problem analysis system is electrically connected to a monitoring result feedback system, the output end of the monitoring result feedback system is electrically connected to the monitoring result processing system, and the output end of the monitoring result processing system is electrically connected to a terminal display system; The monitoring result feedback system is used to provide feedback on data quality issues and problem analysis results; The monitoring result processing system is used to process the monitoring results of the data and generate corresponding lists and charts for easy observation; The terminal display system is used to display the lists and charts generated by the monitoring result processing system; The quality problem analysis system includes a source system problem analysis module, an ETL problem analysis module, a scheduling system problem analysis module and a business demand problem analysis module; The source system problem analysis module is used to analyze problems existing in structure changes, business process changes, data entry, data evolution and data shock in the source data system; The ETL problem analysis module is used to analyze the problems existing in data requirements, mapping rules, program errors and data timeliness in the ETL system; The scheduling system problem analysis module is used to analyze the scheduling system problems caused by data delay and its own faults in the timing scheduling system; The business demand problem analysis module is used to analyze the changes in data demand and the conflicts between the system and the business.

2. According to claim 1, a data warehouse monitoring system for an urban traffic big data platform is characterized in that: The data quality verification system includes an integrity verification module, a consistency verification module, a completeness verification module, a validity verification module and a timeliness verification module.

3. According to claim 2, a data warehouse monitoring system for an urban traffic big data platform is characterized in that: The integrity verification module is used to verify the degree of compliance with the data relationship rules; The consistency verification module is used to verify the degree of data change and variation between data sets; The completeness verification module is used to verify whether the data exists, attributes and quantity are complete; The validity verification module is used to verify the degree of compliance of data with a set of business rules; The timeliness verification module is used to verify the timeliness of data processing.

4. The data warehouse monitoring system of the urban traffic big data platform according to claim 3 is characterized by: The data analysis system includes a column attribute analysis module and a structure attribute analysis module.

5. According to claim 4, a data warehouse monitoring system for an urban traffic big data platform is characterized in that: The column attribute analysis module is used to analyze the column type, column format, column length, high-frequency value column, low-frequency value column, date column, column validity, column value range, column default value, column mode, column null value and duplicate records of the data; The structural attribute analysis module is used to analyze the referential integrity of data, dependencies between columns, data granularity, data accuracy, data hierarchy and business rules.

6. The data warehouse monitoring system of the urban traffic big data platform according to claim 5 is characterized in that: The data collection system includes a metadata collection module and a warehouse data collection module.

7. The data warehouse monitoring system of the urban traffic big data platform according to claim 6 is characterized by: The metadata collection module is used to collect and obtain basic metadata in traffic data; The warehouse data collection module is used to collect and acquire the storage data in the data warehouse.

Images