An incremental data convergence updating method and system based on CarbonData

Abstract

The invention discloses an incremental data convergence updating method and system based on CarbonData, and the method comprises the following steps: 1, carrying out the recognition processing of incremental data, enabling the incremental data to land to a to-be-made directory, and enabling a record to be newly added in a metadata table; Step 2, triggering a trigger preset in the metadata table after newly adding the record, and newly adding a scheduling task on a data increment updating convergence module by the trigger; Step 3, the data increment updating convergence module obtains a scheduling task and determines the current task type according to a trigger type task priority principle; Step 4, performing task logic processing according to the current task type, and moving the incremental file to an ad directory; 5, executing the updated and converged sql statement to converge the increment file increment to a full scale; and 6, after convergence updating is completed, the corresponding record is recovered to be in an unoccupied state, and the field content information corresponding to the record is modified. According to the method and the device, the problem that a CarbonDatastreaming processing scheme does not support incremental updating is solved, and the workload of service transformation is reduced.

Description

technical field [0001] The present invention relates, in particular, to a CarbonData-based incremental data convergence update method and system thereof. Background technique [0002] In order to solve the problem of incremental data update for various external data sources (message stream or file format), quickly converge the changes of external data sources to the full scale table on the Hadoop platform, and be able to provide aggregation query capabilities at any time. In recent years, many companies have gradually begun to use CarbonData, an open-source big data high-performance data storage format, which has obvious advantages over other technologies. It can not only provide rich indexing and coding support, but also be based on the Hadoop ecology and the height of Spark. integration, but there are still some problems that limit its application. [0003] CarbonData's current stream processing only supports data insertion, not data deletion and update. Scenarios that n...

AI Technical Summary

Problems solved by technology

It cannot be used directly for scenarios where data needs to be deleted and updated, and a full update workaround is required, which is complex and inefficient

There is also poor compatibility with old external data source systems (non-message flow file formats, Kafka of lower versions 0.8.x, RabbitMQ, RocketMQ)

CarbonData lacks effective management of incremental data, and cannot trace the sequence and accuracy of the incremental data convergence process

In addition, data deletion and update operations are prone to fragmentation. Although the defragmentation function of merging and clearing fragments through SQL is provided, the time of merging and cleaning is too close to make the data unavailable for a long time. Too frequent merging or cleaning will also affect It provides aggregation query capabilities at any time

For scenarios that need to delete and update data, CarbonData stream processing cannot be directly used to correspond

When the external data source system does not meet the compatibility requirements of CarbonData, the data source will be unavailable

When an exception occurs in incremental data processing, the sequence and accuracy of the incremental data convergence process cannot be traced

Defragmentation affects its ability to serve aggregate queries at any time

Images