A data integration method and system

Abstract

A data integration method and system, comprising: the gateway adapter converts the received AFTN protocol or ATN protocol data flow into JMS protocol message data, and forwards the message data to the message analysis module through the gateway module; the message analysis module converts the JMS The message data of the protocol is matched with the pre-set message parsing template, the parser is constructed by loading the matched message parsing template, the text of the message data of the JMS protocol is parsed by the parser, and the business message is constructed according to the parsed text Object; first send the business message object to the hot cache module, and then send it to the data integration module; the data integration module uses the expression engine to calculate the expression, and uses the expression to find the data in the hot cache module according to the business message object, and continues to execute A step of searching for data in the hot cache module by using an expression according to the found data until no data is found; performing integrated processing on a data set including all the found data to obtain an integrated data source.

Description

technical field [0001] The invention relates to civil aviation control and data processing technology, especially a data integration method and system. Background technique [0002] Currently commonly used data integration methods are mainly the traditional Extract-Transform-Load (ETL, Extract-Transform-Load) method, the big data ETL method, and the Big Data Extract-Loading-Transform (ELT, Extract-Loading-Transform) method. [0003] Among them, the traditional ETL method refers to extracting the data of the source database, converting the data format of the extracted data into the data format of the target database or target data warehouse, and then loading the data after the converted data format into the target database or target data warehouse. It itself is an independent transit tool for connecting the source database and the target database or the target data warehouse. [0004] The big data ETL method means that big data is used as both the target database and the ETL...

AI Technical Summary

Problems solved by technology

[0008] 1) Cannot convert and process in real time

[0009] In the traditional ETL method, due to manual triggering or timer triggering, only second-level data can be processed, while real-time processing needs to process less than 50 milliseconds or even nanosecond-level data, so real-time processing cannot be performed

[0010] In the big data ELT method and the big data ETL method, the Hadoop software uses a manual trigger task method; the Spark software uses a timer trigger mechanism, which can only trigger second-level processing, so real-time processing is not possible

[0011] 2) Does not support dynamic data sources

[0012] In the traditional ETL method, only static data in the database can be processed, and data flow processing cannot be supported;

[0013] In the big data ELT method and the big data ETL method, Hadoop software can only support block file processing, but cannot support real-time streaming data processing; Spark software can only support streaming data processing under the second timing, and cannot support real-time streaming data sources

[0014] 3) Unable to support multiple data sources

[0015] The traditional ETL method can only operate the database in a driving way, and cannot support data sources that provide services or dynamic data streams;

[0016] In the big data ETL method and the big data ELT method, the processing data source of the Hadoop software is a file, and the data source is single; while the Spark software can only support Socket, Flume and Karaf dynamic data source methods. For the standard protocol Java message service (JMS, Java Message Service), Transmission Control Protocol (TCP, Transmission Control Protocol), Internet Protocol (IP, Internet Protocol) and serial port protocol cannot support

[0017] 4) The conversion method is complicated

[0018] For complex air traffic control information, the format of the information is very different. Each message includes a complete logic, but the traditional ETL method cannot complete the analysis and processing of complex messages; Hadoop software and Spark software use MapReduce The method can define parsing tasks for each type of message, and the encoding complexity is not suitable for personalized message parsing.

Images