Civil aviation data analysis system and method based on big data

Abstract

The invention discloses a civil aviation data analysis system and method based on big data, belongs to the technical field of big data processing, and aims to solve the technical problem of how to collect and analyze various data in the civil aviation field to provide auxiliary support for data use. Comprising a data connection module used for connecting various data sources, collecting civil aviation data from the data sources and transferring the civil aviation data to a message queue; the data acquisition module is used for consuming civil aviation data from the message queue and storing the consumed civil aviation data to a data table; the data processing module is used for storing the standardized and structured civil aviation data into a data table; the data modeling analysis module is used for extracting detail data from the pre-processed data as application data, storing the associated application data into a data table, performing aggregation calculation on the application data to obtain index data, and storing the index data into the data table; and the data display module is used for displaying the index data in the forms of charts and lists.

Description

Technical Field

[0001] The present invention relates to the technical field of big data processing, and in particular to a civil aviation data analysis system and method based on big data. Background Art

[0002] The current civil aviation industry lacks a method and system for effectively managing, using and mining the value of massive data in the industry. The industry data accumulated over the years only stays in various business systems, making it difficult to explore its potential value. In addition, the means that business personnel, marketing personnel and scientific and technological personnel in the civil aviation industry can rely on for data statistics, data analysis, and decision analysis are relatively scarce. Data statistics and analysis often require the support of technical personnel in the R&D system. Since technical personnel are often not proficient in business and business personnel do not understand technology, work efficiency is reduced and the results of data analysis and mining are not ideal.

[0003] How to collect and analyze various types of data in the civil aviation field to provide auxiliary support for data use is a technical problem that needs to be solved. Summary of the invention

[0004] The technical task of the present invention is to address the above shortcomings and provide a civil aviation data analysis system and method based on big data to solve the technical problem of how to collect and analyze various types of data in the civil aviation field to provide auxiliary support for data use.

[0005] In a first aspect, the present invention provides a civil aviation data analysis system based on big data, comprising a data connection module, a data acquisition module, a data processing module, a data modeling and analysis module, and a data display module;

[0006] The data connection module is used to connect to various data sources and collect civil aviation data from the data sources and transfer the civil aviation data to the message queue, wherein the data sources include various civil aviation equipment and information systems;

[0007] The data collection module is used to consume civil aviation data from the message queue and store the consumed civil aviation data in a data table as an original table, and the original table is constructed in a distributed file storage system;

[0008] The data processing module is used to perform data preprocessing on the civil aviation data stored in the distributed file storage system to obtain standardized and structured civil aviation data, and store the standardized and structured civil aviation data as preprocessed data in a data table as a standard structured table, wherein the standard structured table is constructed in the distributed file storage system;

[0009] The data modeling and analysis module is used to extract detailed data from the pre-processed data as application data, and to associate the application data from different standard structured tables, and store the associated application data in a data table as an application table, the application table is constructed in a distributed file storage system, and is used to aggregate and calculate the application data to obtain index data, and store the index data in a data table as an index table, the index table is constructed in a distributed file storage system, and is used to classify the application data and the index data;

[0010] The data display module is used to read application data and indicator data from the distributed file storage system, store the application data and indicator data permanently, and display the indicator data in the form of charts and lists.

[0011] Preferably, the data connection module is configured with a monitoring program developed based on Java language, and the civil aviation data is collected from the data source through the monitoring program and written into the message queue.

[0012] Preferably, the original table, standard structured table, application table and index table in the distributed file storage system are all Hive tables.

[0013] Preferably, the data collection module is used to continuously consume civil aviation data in the message queue based on Flink, and write the consumed civil aviation data into a distributed file storage system.

[0014] Preferably, the data processing module is used to extract civil aviation data from the original table in a regular and incremental manner through Spark, and perform data preprocessing on the extracted civil aviation data.

[0015] Preferably, the data collection module is configured with a distributed stream computing engine, which is used to collect civil aviation data from the data access module through the distributed stream computing engine in a distributed stream computing manner, and store the civil aviation data in a data table as an original table;

[0016] The data processing module is configured with a distributed stream computing engine, which is used to pre-process the civil aviation data in a distributed stream computing manner through the distributed stream computing engine;

[0017] For real-time application data and indicator time, the data modeling and analysis module is used to perform data association, aggregation calculation and classification on the application data in a distributed streaming computing manner, and to classify the indicator data in a distributed streaming computing manner. Correspondingly, the data display module is used to perform persistent storage of application data and indicator data in a Redis storage manner deployed in cluster mode;

[0018] For non-real-time application data and indicator data, the data modeling and analysis module is used to perform data association, aggregation calculation and classification on the application data in a distributed batch computing manner, and to classify the indicator data in a distributed batch computing manner. Correspondingly, the data display module is used to perform persistent storage of application data and indicator data in the StarRocks database deployed in cluster mode.

[0019] Preferably, the civil aviation data collected from the data source includes structured data and unstructured data, wherein for a data source of equipment type, the collected civil aviation data is unstructured telegram message data, and for a data source of information system type, the collected civil aviation data is structured data;

[0020] Correspondingly, for structured civil aviation data, the data preprocessing module is used to clean the civil aviation data, remove outliers, and supplement missing values ​​to obtain processed civil aviation data, and set a unique primary key for the processed civil aviation data;

[0021] For unstructured civil aviation data, the data processing module is used to parse and standardize the civil aviation data based on various telegram format documents disclosed by the Civil Aviation Administration to obtain structured civil aviation data, wherein the various telegram format documents disclosed by the Civil Aviation Administration include civil aviation meteorological messages and navigation notice message format standard documents. The civil aviation data is parsed to form multiple field information, including the flight number, scheduled take-off time, longitude, latitude, altitude and speed of the flight. The standardized processing of the civil aviation data includes standardization based on the time field and standardization based on the altitude field.

[0022] Preferably, the data modeling and analysis module is used to associate the application data through association fields based on dimensions including air routes and control sectors;

[0023] The application data and indicator data are counted according to different time granularities to form application data and indicator data of different time granularities, and are used to classify the application data and indicator data respectively based on predefined topics.

[0024] In a second aspect, the present invention provides a civil aviation data analysis method based on big data, which collects civil aviation data from various data sources through a civil aviation data analysis system based on big data as described in any one of the first aspects, and analyzes and processes the civil aviation data.

[0025] The civil aviation data analysis system and method based on big data of the present invention have the following advantages: a complete set of civil aviation data life cycle system is established through big data technology, massive data of various civil aviation equipment and information systems are collected together, and a standardized data warehouse is established. Finally, at the business level, the data is provided to users securely and directly in the form of data catalogs and data visualization applications, providing civil aviation practitioners with a direct and secure means of data acquisition and providing data support for various decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0027] The present invention is further described below in conjunction with the accompanying drawings.

[0028] Figure 1 This is a workflow diagram of a civil aviation data analysis system based on big data in Example 1. DETAILED DESCRIPTION

[0029] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments may be combined with each other.

[0030] The embodiments of the present invention provide a civil aviation data analysis system and method based on big data, which are used to solve the technical problem of how to collect and analyze various types of data in the civil aviation field to provide auxiliary support for data use.

[0031] Embodiment 1:

[0032] The present invention discloses a civil aviation data analysis system based on big data, comprising a data connection module, a data acquisition module, a data processing module, a data modeling and analysis module and a data display module.

[0033] The data connection module is used to connect to various data sources, collect civil aviation data from the data sources, and transfer the civil aviation data to the message queue. The data sources include various civil aviation equipment and information systems.

[0034] In this embodiment, a monitoring program developed based on Java language is configured in the data connection module, and civil aviation data is collected from the data source through the monitoring program and written into the message queue.

[0035] The civil aviation data collected from the data source includes structured data and unstructured data. For equipment-type data sources, the collected civil aviation data is unstructured telegram message data, and for information system-type data sources, the collected civil aviation data is structured data.

[0036] The data collection module is used to consume civil aviation data from the message queue and store the consumed civil aviation data in a data table as an original table, and the original table is constructed in a distributed file storage system.

[0037] In this embodiment, the data collection module is used to continuously consume civil aviation data in the message queue based on Flink, and write the consumed civil aviation data into the distributed file storage system. The data collection module is configured with a distributed streaming computing engine, which is used to collect civil aviation data from the data connection module through the distributed streaming computing engine and in a distributed streaming computing manner, and store the civil aviation data in a data table as the original table, wherein the original table is a Hive table.

[0038] The data processing module is used to perform data preprocessing on the civil aviation data stored in the distributed file storage system to obtain standardized and structured civil aviation data, and store the standardized and structured civil aviation data as preprocessed data in a data table as a standard structured table, which is constructed in the distributed file storage system.

[0039] In this embodiment, the data processing module is used to extract civil aviation data from the original table in a timed and incremental manner through Spark, and perform data preprocessing on the extracted civil aviation data. The data processing module is configured with a distributed stream computing engine, which is used to preprocess the civil aviation data through the distributed stream computing engine in a distributed stream computing manner, and the data table is a Hive table.

[0040] The data modeling and analysis module is used to extract detailed data from the pre-processed data as application data, and to associate the application data from different standard structured tables, and to store the associated application data in a data table serving as an application table. The application table is constructed in a distributed file storage system, and is used to perform aggregation calculations on the application data to obtain indicator data, and store the indicator data in a data table serving as an indicator table. The indicator table is constructed in a distributed file storage system, and is used to classify application data and indicator data.

[0041] In this embodiment, the data modeling and analysis module is used to associate application data through associated fields based on dimensions including routes and control sectors; statistics are performed on application data and indicator data according to different time granularities to form application data and indicator data of different time granularities, and the module is used to classify application data and indicator data based on predefined topics.

[0042] As a specific implementation of the data modeling and analysis module, for real-time application data and indicator time, the data modeling and analysis module is used to perform data association, aggregation calculation and classification on the application data in a distributed streaming computing manner, and to classify the indicator data in a distributed streaming computing manner. For non-real-time application data and indicator data, the data modeling and analysis module is used to perform data association, aggregation calculation and classification on the application data in a distributed batch computing manner, and to classify the indicator data in a distributed batch computing manner.

[0043] The data display module is used to read application data and indicator data from the distributed file storage system, store the application data and indicator data permanently, and display the indicator data in the form of charts and lists.

[0044] In this embodiment, the data display module is used to perform persistent storage of real-time application data and indicator data in a Redis storage method deployed in cluster mode, and is used to perform persistent storage of non-real-time application data and indicator data in a StarRocks database deployed in cluster mode.

[0045] The workflow of the civil aviation data analysis system in this embodiment is as follows:

[0046] (1) Data connection: Use a monitoring program developed in Java to monitor civil aviation data from data sources such as civil aviation-related equipment and message systems, and write the civil aviation data into a message queue. The message queue Kafka can be used.

[0047] (2) Data collection: Use Flink to continuously consume civil aviation data in the message queue, write the consumed civil aviation data to the distributed file system HDFS, and use Hive to organize the data table structure of the civil aviation data;

[0048] (3) Data processing: Use Spark to extract civil aviation data collected from Hive tables in a regular and incremental manner. For unstructured civil aviation data, parse and standardize the unstructured civil aviation data according to various telegram format documents (CAT062 format message standard documents) published by the Civil Aviation Administration to form structured data. After parsing the unstructured civil aviation data, fields such as flight number, scheduled take-off time, longitude, latitude, altitude, speed, etc. are formed. The standardization of civil aviation data includes: standardization of time fields, such as standardizing UTC time into Beijing time; standardization of altitude fields, such as standardizing altitude units from feet to meters. Persist the parsed standardized structured data in the HDFS file system, and use Hive to organize the data table structure;

[0049] (4) Data modeling and analysis: Use Spark to extract standard structured civil aviation data from Hive tables in a regular and incremental manner as application data, and associate the application data with dimensions such as routes and control sectors. Perform statistics at different time granularities to form data with different time granularities such as monthly, daily, and hourly granularities, and classify them according to predetermined topics. Perform aggregation calculations on the application data to obtain indicator data. For the indicator data, perform statistics at different time granularities to form data with different time granularities such as monthly, daily, and hourly granularities, and classify them according to predetermined topics. Persist the application data and indicator data in the distributed StarRocks database.

[0050] (5) Data display: Collect application and indicator data, and display the application data and indicator data in the form of charts and lists through a visualization application developed in Java.

[0051] Embodiment 2:

[0052] The present invention provides a civil aviation data analysis method based on big data. The system disclosed in Example 1 collects civil aviation data from various data sources and analyzes and processes the civil aviation data.

[0053] The above is a detailed introduction to the civil aviation data analysis system method based on big data provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A civil aviation data analysis system based on big data, characterized in that: It includes data connection module, data acquisition module, data processing module, data modeling and analysis module and data display module; The data connection module is used to connect to various data sources and collect civil aviation data from the data sources and transfer the civil aviation data to the message queue, wherein the data sources include various civil aviation equipment and information systems; The data collection module is used to consume civil aviation data from the message queue and store the consumed civil aviation data in a data table as an original table, and the original table is constructed in a distributed file storage system; The data processing module is used to perform data preprocessing on the civil aviation data stored in the distributed file storage system to obtain standardized and structured civil aviation data, and store the standardized and structured civil aviation data as preprocessed data in a data table as a standard structured table, wherein the standard structured table is constructed in the distributed file storage system; The data modeling and analysis module is used to extract detailed data from the pre-processed data as application data, and to associate the application data from different standard structured tables, and store the associated application data in a data table as an application table, the application table is constructed in a distributed file storage system, and is used to aggregate and calculate the application data to obtain index data, and store the index data in a data table as an index table, the index table is constructed in a distributed file storage system, and is used to classify the application data and the index data; The data display module is used to read application data and indicator data from the distributed file storage system, store the application data and indicator data permanently, and display the indicator data in the form of charts and lists.

2. The civil aviation data analysis system based on big data according to claim 1, characterized in that: The data connection module is configured with a monitoring program developed based on the Java language, which collects civil aviation data from a data source and writes the civil aviation data into a message queue.

3. The civil aviation data analysis system based on big data according to claim 1, characterized in that: The original table, standard structured table, application table and index table in the distributed file storage system are all Hive tables.

4. The civil aviation data analysis system based on big data according to claim 1, characterized in that: The data collection module is used to continuously consume civil aviation data in the message queue based on Flink, and write the consumed civil aviation data into the distributed file storage system.

5. The civil aviation data analysis system based on big data according to claim 1, characterized in that: The data processing module is used to extract civil aviation data from the original table in a regular and incremental manner through Spark, and perform data preprocessing on the extracted civil aviation data.

6. The civil aviation data analysis system based on big data according to claim 1, characterized in that: The data collection module is configured with a distributed stream computing engine, which is used to collect civil aviation data from the data access module through the distributed stream computing engine in a distributed stream computing manner, and store the civil aviation data in a data table as an original table; The data processing module is configured with a distributed stream computing engine, which is used to pre-process the civil aviation data in a distributed stream computing manner through the distributed stream computing engine; For real-time application data and indicator time, the data modeling and analysis module is used to perform data association, aggregation calculation and classification on the application data in a distributed streaming computing manner, and to classify the indicator data in a distributed streaming computing manner. Correspondingly, the data display module is used to perform persistent storage of application data and indicator data in a Redis storage manner deployed in cluster mode; For non-real-time application data and indicator data, the data modeling and analysis module is used to perform data association, aggregation calculation and classification on the application data in a distributed batch computing manner, and to classify the indicator data in a distributed batch computing manner. Correspondingly, the data display module is used to perform persistent storage of application data and indicator data in the StarRocks database deployed in cluster mode.

7. The civil aviation data analysis system based on big data according to any one of claims 1 to 6, characterized in that: The civil aviation data collected from the data source includes structured data and unstructured data. For the data source of equipment type, the collected civil aviation data is unstructured telegram message data, and for the data source of information system type, the collected civil aviation data is structured data; Correspondingly, for structured civil aviation data, the data preprocessing module is used to clean the civil aviation data, remove outliers, and supplement missing values ​​to obtain processed civil aviation data, and set a unique primary key for the processed civil aviation data; For unstructured civil aviation data, the data processing module is used to parse and standardize the civil aviation data based on various telegram format documents disclosed by the Civil Aviation Administration to obtain structured civil aviation data, wherein the various telegram format documents disclosed by the Civil Aviation Administration include civil aviation meteorological messages and navigation notice message format standard documents. The civil aviation data is parsed to form multiple field information, including the flight number, scheduled take-off time, longitude, latitude, altitude and speed of the flight. The standardized processing of the civil aviation data includes standardization based on the time field and standardization based on the altitude field.

8. The civil aviation data analysis system based on big data according to any one of claims 1 to 6, characterized in that: The data modeling and analysis module is used to associate application data through associated fields based on dimensions including airways and control sectors; The application data and indicator data are counted according to different time granularities to form application data and indicator data of different time granularities, and are used to classify the application data and indicator data respectively based on predefined topics.

9. A civil aviation data analysis method based on big data, characterized in that: Civil aviation data is collected from various data sources through a civil aviation data analysis system based on big data as described in any one of claims 1 to 8, and the civil aviation data is analyzed and processed.

Images