An enhanced real-time computing method for dynamic real-time synchronization of multi-source large table data

Abstract

The invention discloses an enhanced real-time calculation method for dynamic real-time synchronization of multi-source large table data. The method includes the following steps: building a distributed dynamic table management component; creating a distributed dynamic table, and standardizing the multi-source through the distributed dynamic table The data structure of the library table; manage the synchronous update of the metadata change information of the distributed dynamic table through real-time synchronization technology; initialize the batch import of distributed dynamic table data; check the integrity of the imported data; update the data in the distributed dynamic table in real time ;Synchronous monitoring of real-time data during the data synchronization update process; convert the data in the distributed dynamic table into a virtual table of real-time streaming data through structured query language; combine the data in the virtual table with the pre-configured streaming data Compute; output stream computation results. Beneficial effects: solve the real-time calculation requirements of complex business logic under real-time calculation combined with external data sources over 100 million tables.

Description

technical field [0001] The invention relates to the technical field of big data computing, in particular to an enhanced real-time computing method for dynamically synchronizing multi-source big table data in real time. Background technique [0002] In the process of real-time computing, many real-time businesses have complex processing requirements such as data completion, data filtering, and data conversion. Due to the data required by complex logic such as completion, filtering conditions, and conversion, it is usually stored outside the real-time computing process. It is necessary to connect to different external data sources to pull data during real-time calculations. Large-scale data usually stores large table data exceeding 100 million. The performance of the process of pulling data depends entirely on the performance of the source data source. During real-time calculations, there is a significant performance impact. [0003] With the popularization of real-time compu...

AI Technical Summary

Problems solved by technology

[0010] 1) It will cause access pressure to the offline database and affect the business of the offline database itself. Since each record of real-time data needs to be connected to the offline database and read the data, if 100 or more real-time data records are processed concurrently , it is necessary to connect to the offline database more than 100 times or more, which will cause a lot of access pressure on the offline database pair;

[0011] 2) It is difficult to meet the second-level performance requirements of real-time computing. Due to the differences in different databases, it is difficult to guarantee the ability to read data records in large tables with a second-level response capability, which will cause real-time computing delays and cause a large amount of real-time data accumulation. ;

[0012] 3) When reading memory data, it is difficult to guarantee the correctness of the data. At the same time, the data of over 100 million large tables may not be completely loaded into the memory. Since the data is preloaded into the memory, the changed data of the offline database table cannot be synchronized in time. To the memory, it will lead to data calculation errors;

[0013] 4) The format of multi-source data is not uniform, which invisibly increases the difficulty of calculation docking. Since different data sources such as hbase and oracle have different data storage formats, when combining calculations with real-time data, each operator needs to target Different data sources are developed separately, which invisibly increases R&D costs and implementation complexity

Images