Incremental data lake entering method, device and equipment based on Flink and storage medium

Abstract

The invention relates to the technical field of data processing and the technical field of financial science and technology, and discloses an incremental data lake entering method based on Flink, incremental data are obtained from data sources through an Flink task, and the data sources comprise a real-time streaming data source and a static batch processing data source; performing stream processing or batch processing on the incremental data, and writing the incremental data into an incremental log file of the MOR table; in the stream processing process, regularly carrying out state storage operation and storing a data processing state; storing a final processing state when the batch processing task is completed; fragmenting the incremental log file, and generating an optimized incremental log file through distributed node or multi-thread parallel processing and merging operation; and asynchronously executing a merging operation, and merging the incremental log file with the base table file to generate a new base table file. According to the method, the incremental log file and the base table file are combined, and the resource-intensive combination operation is set in the background in an asynchronous execution mode, so that the blockage and the performance influence of the combination process on the system are avoided.

Description

Technical Field

[0001] The present invention relates to the fields of data processing technology and financial technology, and in particular to a Flink-based incremental data lake entry method, device, equipment and storage medium. Background Art

[0002] In the financial field, offline data table merging (MOR, Merge-on-read) strategy is widely used, especially when dealing with large-scale financial data entry operations. Financial institutions need to process a large amount of transaction data, customer records, risk assessment results, etc., which are often generated in incremental form and need to be merged into existing basic tables. The advantage of the MOR strategy is that it allows these incremental data to be written to the log file first, and then integrated with the basic data table through asynchronous merge operations, thereby reducing the pressure on the real-time system.

[0003] The challenge facing the financial system is the need to handle high-concurrency transactions and real-time data updates, which causes the existing MOR solution to expose some deficiencies in performance and system pressure when processing large-scale financial data.

[0004] In financial systems, transaction frequency is very high, especially in the field of securities trading, foreign exchange market or payment settlement. The incremental data generated by each transaction needs to be recorded and merged into the central database in a timely manner, which will lead to a large number of disk I / O operations. Especially when merging data, the repeated reading and writing of incremental logs and basic data will significantly affect the overall performance of the system.

[0005] As transaction volume and data volume continue to increase, financial systems need to consume more computing and storage resources to complete merging operations. Frequent merging tasks will lead to system resource shortages, further reducing system throughput, especially during peak periods of financial markets (such as opening and closing hours), which puts higher demands on the system.

[0006] In a large-scale financial data environment, the maintenance and management complexity of the MOR system is high. The financial system must ensure that the priority, scheduling, and resource allocation of compaction tasks are correct. Especially when the amount of data is huge, this management becomes very complicated, which may affect the stability and processing capacity of the system, and even cause system delays or data loss risks.

[0007] Therefore, in the financial field, the existing MOR technology is difficult to cope with the performance bottleneck and resource burden of such large-scale data processing. In order to improve the efficiency of the system in processing large-scale financial data, it is necessary to optimize the MOR merging operation, reduce system resource consumption, and improve the performance and stability of the system in a high-frequency trading environment. These optimizations include smarter compaction strategies, more efficient parallel processing mechanisms, and further enhancements to the distributed storage system to ensure that the system can operate stably in a high-load financial environment. Summary of the invention

[0008] The main purpose of the present invention is to provide a Flink-based incremental data entry method, device, equipment and storage medium, aiming to solve the technical problem that in the prior art, when performing large-scale data synchronization, the data merging process will cause significant pressure on system performance, easily leading to system congestion and performance degradation.

[0009] To achieve the above object, the present invention provides a Flink-based incremental data entry method, comprising:

[0010] Acquire incremental data from a data source through a Flink task, where the data source includes a real-time streaming data source or a static batch data source;

[0011] According to the type of the data source, the Flink task is used to perform stream processing or batch processing on the incremental data, and the processed data is written into the incremental log file of the MOR table;

[0012] During stream processing, state saving operations are performed periodically and the processing status is saved; when batch processing is completed, the final processing status is maintained;

[0013] The incremental log file is sliced ​​and processed, and the data of each slice is independently processed in parallel and merged to generate an optimized incremental log file;

[0014] An asynchronous merge operation is performed to merge the optimized incremental log file with the base table file in the MOR table to generate a new base table file in the MOR table.

[0015] Furthermore, to achieve the above-mentioned purpose, the present invention provides a Flink-based incremental data lake entry device, comprising:

[0016] A data source connection module is used to obtain incremental data from a data source through a Flink task. The data source includes a real-time streaming data source or a static batch data source.

[0017] A data processing and writing module, used to perform stream processing or batch processing on the incremental data using the Flink task according to the type of the data source, and write the processed data into an incremental log file of the MOR table;

[0018] The state management module is used to periodically perform state saving operations and save the processing status during the stream processing process; when the batch processing is completed, the final processing status is maintained;

[0019] The data sharding and parallel processing module is used to perform sharding processing on the incremental log file, and perform independent parallel processing and merging operations on the data of each shard to generate an optimized incremental log file;

[0020] The data merging module is used to perform an asynchronous merging operation to merge the optimized incremental log file with the base table file in the MOR table to generate a new base table file in the MOR table.

[0021] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, comprising a memory, a processor, and a Flink-based incremental data entry into the lake program stored in the memory and executable on the processor, wherein the Flink-based incremental data entry into the lake program, when executed by the processor, implements the steps of the Flink-based incremental data entry into the lake method as described above.

[0022] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a Flink-based incremental data entry into the lake program is stored. When the Flink-based incremental data entry into the lake program is executed by a processor, the steps of the Flink-based incremental data entry into the lake method as described above are implemented.

[0023] Beneficial effects: The present invention relates to the fields of data processing technology and financial technology, and discloses a Flink-based incremental data entry method, which obtains incremental data from a data source through a Flink task, and the data source includes a real-time streaming data source and a static batch data source. After the incremental data is stream processed or batch processed, it is written into the incremental log file of the MOR table. During the stream processing process, the state saving operation is performed regularly and the data processing state is saved, and the final processing state is saved when the batch task is completed. The incremental log file is sharded, and an optimized incremental log file is generated through distributed node or multi-threaded parallel processing and merging operations. Subsequently, the merge operation is performed asynchronously to merge the optimized incremental log file with the base table file in the MOR table to generate a new base table file. The present invention divides the incremental log file into multiple small data blocks and processes them independently through sharding and parallel processing mechanisms, fully utilizing the computing power of distributed nodes, and realizing efficient processing of large-scale data. At the same time, an asynchronous merge operation is used to merge the incremental log file with the base table file, and the resource-intensive merge operation is placed in the background through asynchronous execution, thereby avoiding the blocking and performance impact of the merge process on the system. In the process of stream processing, the state preservation operation enables the data processing state to be saved regularly to ensure rapid recovery when the task is interrupted, while in batch mode, the processing state of each batch is saved to ensure data consistency. Through these mechanisms, effective management and optimization of system resources are achieved, the burden on the system is reduced, data processing efficiency is improved, and the stability and consistency of data synchronization are ensured, thereby achieving the effect of optimizing system performance and efficiently processing large-scale data during the process of data entering the lake. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0025] Figure 1 A schematic diagram of an application environment of an incremental data lake entry method based on Flink in one embodiment of the present invention;

[0026] Figure 2 This is a flow chart of an embodiment of a method for incremental data entry into a lake based on Flink of the present invention;

[0027] Figure 3 This is a functional module diagram of a preferred embodiment of an incremental data entry device based on Flink of the present invention;

[0028] Figure 4 A schematic diagram of the structure of a computer device in one embodiment of the present invention;

[0029] Figure 5 FIG. 4 is another schematic diagram of the structure of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0030] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0031] The incremental data entry method based on Flink provided in the embodiment of the present invention can be applied in the following aspects: Figure 1 In an application environment, the client communicates with the server through a network. The server can obtain incremental data from a data source through a Flink task of the client, and the data source includes a real-time streaming data source and a static batch data source. After the incremental data is stream processed or batch processed, it is written into the incremental log file of the MOR table. During the stream processing process, the state saving operation is performed regularly and the data processing state is saved, and the final processing state is saved when the batch task is completed. The incremental log file is fragmented, and an optimized incremental log file is generated through distributed node or multi-threaded parallel processing and merging operations. Subsequently, the merge operation is performed asynchronously to merge the optimized incremental log file with the base table file in the MOR table to generate a new base table file. The present invention divides the incremental log file into multiple small data blocks and processes them independently through the fragmentation processing and parallel processing mechanism, fully utilizes the computing power of the distributed nodes, and realizes efficient processing of large-scale data. At the same time, an asynchronous merge operation is used to merge the incremental log file with the base table file, and the resource-intensive merge operation is placed in the background through asynchronous execution, avoiding the blocking and performance impact of the merge process on the system. During the stream processing, the state saving operation enables the data processing state to be saved regularly to ensure rapid recovery when the task is interrupted, while the processing state of each batch is saved in batch mode to ensure data consistency. Through these mechanisms, effective management and optimization of system resources are achieved, the burden on the system is reduced, the data processing efficiency is improved, and the stability and consistency of data synchronization are ensured, thereby achieving the effect of optimizing system performance and efficiently processing large-scale data during the data entry process. Among them, the client can be but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented with an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0032] See also Figure 2 , Figure 2 A flowchart of an embodiment of an incremental data entry method based on Flink provided by the present invention. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0033] like Figure 2 As shown in the figure, the Flink-based incremental data entry method proposed in the present invention includes the following steps:

[0034] S10, acquiring incremental data from a data source through a Flink task, where the data source includes a real-time streaming data source or a static batch data source;

[0035] In this embodiment, the Flink task is a specific operation unit scheduled by the Flink engine, which is responsible for executing data stream or batch processing tasks. The Flink task can realize real-time processing of data streams and batch processing of static data.

[0036] Incremental data refers to data that is newly added or changed after the last processing. Incremental data is the basis for lake entry operations, reducing the need for full data processing.

[0037] The data source can be a real-time data source (such as Kafka, Kinesis) or a static data source (such as a database, file system). Real-time data sources provide streaming data, while static data sources provide data through batch reading.

[0038] When a Flink task is started, it obtains data from the data source through the specified data source connector. For example, use the FlinkKafka Connector to subscribe to real-time streaming data, or read static data sources through a JDBC connection. Flink tasks support parallelism and can process multiple data partitions at the same time, ensuring that data can be efficiently transmitted and processed.

[0039] Real-time streaming data sources continuously generate real-time data (such as log streams and sensor data). Common implementations include Kafka, Kinesis, etc. The flow of data sources is continuous, so a real-time processing framework (such as Flink's DataStream API) is required to process this data.

[0040] Static batch data sources are data sets that are loaded at one time, usually from static storage systems such as file systems and databases, and are processed in batches. Flink's DataSet API is used for batch processing of static data.

[0041] Through Flink's DataStream API, tasks can subscribe to real-time data streams and process data immediately when it arrives. The system can configure parallelism and dynamically scale to cope with high-volume data.

[0042] Through Flink's DataSet API, tasks read data in batches, usually using scheduled tasks or trigger-based mechanisms to read data blocks.

[0043] In one specific implementation, the data stream is obtained from the Kafka topic through the FlinkKafka Connector, and the incremental data is processed in real time using the Flink DataStream API. The Flink task can consume multiple partitions in Kafka in parallel, process each piece of data and save the Checkpoint state, thus realizing continuous processing of streaming data.

[0044] Through the above steps, efficient data acquisition and processing can be performed for different types of data sources (real-time streaming or static batch processing), achieving flexibility and efficiency in data entry into the lake. It reduces repeated processing of the full amount of data, saves system resources, and improves processing speed through sharding and parallel processing mechanisms, ensuring the stability and consistency of data synchronization.

[0045] S20, using the Flink task to perform stream processing or batch processing on the incremental data according to the type of the data source, and writing the processed data into an incremental log file of the MOR table;

[0046] In this embodiment, real-time streaming data sources continuously generate data, such as sensors, logs, Kafka streams, etc.; static batch data sources are one-time data sets, such as data stored in a file system or database.

[0047] Depending on the type of data source, Flink tasks use different APIs to process data. For real-time data sources, use the DataStream API for stream processing; for static data sources, use the DataSet API for batch processing. Streaming data sources continuously receive data, so real-time processing is required. Flink uses the DataStream API to pipeline data processing and processes each data item in real time. For static data sources, Flink uses the DataSet API to process data in batches, operating the entire data set or its fragments each time.

[0048] In the Flink task, configure the data source connector (such as Kafka or Kinesis), and the task will continue to receive data. Whenever new incremental data arrives, the task performs processing operations on the data through the DataStream API, such as cleaning and conversion, and then stores the results in the incremental log file. Read the static batch data source through the DataSet API, load the entire batch of data into the Flink processing engine, and after data preprocessing, conversion or aggregation, the data will be synchronously written to the incremental log file.

[0049] The characteristic of the MOR (Merge-on-Read) table is that the incremental data is first written to the incremental log file instead of being immediately merged with the base table file. This structure helps to reduce the latency of write operations and improve query efficiency. The incremental data is first stored in the log file and then merged with the base table file through the compaction operation. The incremental log file is a temporary storage unit in the MOR table, which records the incremental data generated by each processing.

[0050] After the real-time stream processing is completed, the incremental data will be directly written into the incremental log file of the MOR table. The log file can be divided into multiple shards, and the data is written sequentially to ensure the efficiency and smoothness of the writing.

[0051] After batch processing is completed, the data is written to the incremental log file in batches. Flink ensures the integrity and consistency of batch data through an asynchronous write mechanism. After each batch processing is completed, the task calls the incremental log writing module to append the processed data to the log file.

[0052] In one specific implementation, when processing real-time streaming data sources or static batch data sources, the extension supports heterogeneous data sources, such as processing multiple streaming data sources (such as Kafka and Kinesis) or multiple static data sources (such as databases and file systems) at the same time, combining these data sources through Flink's connector, and uniformly processing incremental data.

[0053] Depending on the data source, real-time and static data are further divided into different levels or priorities. For example, for high-priority data, a higher degree of parallelism is used for processing, and a delayed processing mechanism is adopted for low-priority data to optimize resource usage.

[0054] For heterogeneous data sources, you can use Flink's MultiSource function to extract data from different types of data sources and uniformly process streaming data through the DataStream API, or batch process static data through the DataSet API.

[0055] For multi-level data processing, you can use Flink's priority queue scheduling mechanism to give priority to high-priority data sources, while delaying or processing low-priority data in steps.

[0056] By selecting stream processing or batch processing mode according to different types of data sources, real-time and static data can be effectively processed to ensure the flexibility and efficiency of data processing. Combined with the incremental log file structure of the MOR table, both real-time data and batch data can be written quickly, and the delayed merge operation improves the system throughput and reduces the performance bottleneck caused by the merge operation, thereby achieving high efficiency of data entry into the lake, optimizing the use of system resources, and ensuring the stability of large-scale data processing.

[0057] S30, during the stream processing, periodically performing a state saving operation and saving the processing state; when the batch processing is completed, maintaining the final processing state;

[0058] In this embodiment, the stream processing in Flink is performed through the DataStreamAPI, the data flows in real time, and the system needs to process continuously. The characteristic of stream processing is that the data is unbounded, that is, the data stream is continuously generated. Checkpoint is a fault-tolerant mechanism in Flink, which is used to periodically save the state of data processing during stream processing. The system will automatically execute Checkpoint at a specified time interval and save the current processing state (such as data offset, processing progress) to external storage (such as HDFS, S3, etc.) so that the task can be restored from the last successful Checkpoint when it is interrupted. The processing status includes the processing progress of the data, the intermediate state of the task execution, etc. The saved state can ensure that when the system restarts or fails, the processing flow can continue from the last state, and there will be no duplicate processing or loss of data.

[0059] Through Flink's Checkpoint mechanism, the system periodically saves the current state of the data stream processing task. When each Checkpoint is executed, the intermediate state of the task, data processing offset, etc. are asynchronously saved to the specified external storage system. The system supports multi-level state preservation, including operator state and keyed state, to ensure the fault tolerance of stream processing tasks. When a stream processing task is interrupted, Flink will restore the task state from the most recent Checkpoint, reload the intermediate state of the task, and continue to process the data stream.

[0060] Batch tasks process a complete data set at one time and save the processing results after the data set is processed. Batch tasks use Flink's DataSetAPI, which can process bounded data sets (such as files and database data) and save the result status after processing. The state of batch processing is not saved periodically like stream processing, but the state is saved after the entire batch task is completed. This state may include information such as processing results, processing logs, and task execution time. The state saving of batch tasks is helpful for subsequent inspection, analysis, or task restart. The saved state also includes an indicator of successful completion of processing, which helps the system avoid repeated processing of successfully processed data when the task fails.

[0061] When a batch task is completed, the system saves the final state of the processing, including the execution results of all tasks, the processing time, and the execution records of the tasks. Flink's batch tasks can save these states through storage systems (such as databases and file systems) for reference when subsequent analysis or task restart.

[0062] By periodically performing state saving operations, stream processing tasks can be quickly restored when failures occur, effectively reducing data loss or duplicate processing problems caused by task interruptions. At the same time, the final state is saved when batch processing is completed to ensure that batch processing tasks can be correctly executed and provide a basis for subsequent analysis and fault recovery. Through these two mechanisms, efficient management and fault-tolerant recovery capabilities of data processing tasks are achieved, greatly improving the stability and security of data processing.

[0063] S40, slicing the incremental log file, and performing independent parallel processing and merging operations on the data of each slicing to generate an optimized incremental log file;

[0064] In this embodiment, the incremental log file in the MOR table records the newly added or updated data relative to the base table data. When the amount of data is large, further sharding is required to improve the processing efficiency. An incremental log file is divided into multiple smaller logical units (shards) so that these shards can be processed in parallel. The sharding standard can be divided according to the timestamp, primary key range or number of record rows in the log file to ensure that each shard can be processed independently.

[0065] Flink tasks divide incremental log files according to the set sharding strategy. For example, if divided by the number of record lines, the task will generate a shard for every N records; if divided by time, a shard may be generated for every hour of data; when divided by primary key range, each shard corresponds to a primary key range. After the division, the data of each shard can be stored separately in the distributed file system, and the system will allocate computing resources to each shard to ensure that the data of each shard can be processed independently.

[0066] Each shard data is assigned to a different computing node for independent processing. This parallel processing can improve the overall throughput and processing speed of the system. After the parallel processing is completed, the shard data is merged to integrate the processed shard data and generate a new incremental log file. The merge operation ensures the data consistency of each shard and prepares for the subsequent compaction operation.

[0067] Through Flink's distributed processing engine, tasks can automatically distribute the data of each shard to different computing nodes for parallel processing. Each shard can perform data processing tasks such as cleaning, conversion, and deduplication. After each shard is processed, the system merges all shard data into a complete incremental log file to ensure data order and consistency. The merge operation can be implemented through Flink's operator to merge multiple processed shard streams into one.

[0068] After the sharding and merging operations, the generated incremental log files are optimized, including data compression, sorting, deduplication, etc. The optimized log file data is smaller in size and more neatly structured, which is conducive to subsequent compaction operations.

[0069] When merging sharded data, data compression operations are performed to reduce storage space, and sorting operations are performed to improve the efficiency of subsequent queries. Through data deduplication, only the latest records are retained for each primary key, further optimizing the data quality. The final optimized incremental log file will be stored in the specified file system (such as HDFS, S3, etc.) and prepared for subsequent asynchronous compaction operations to ensure efficient processing and merging of data during compaction.

[0070] For example, the incremental log file is processed by sharding according to the number of record lines. When processing log files, the Flink task divides 1,000 records into one shard, and processes and merges each shard independently. Each shard data is assigned to different computing nodes, and the processed shard data is merged into a complete incremental log file, which is sorted and compressed and stored in HDFS.

[0071] By sharding and processing incremental log files in parallel, the efficiency of large-scale data processing can be effectively improved and the load pressure on a single node can be reduced. The merge operation optimizes the storage structure of the data, and compression and deduplication improve the data storage efficiency, ensuring that the data can be stored efficiently and maintain consistency during the process of entering the lake. Generating optimized incremental log files not only reduces storage space, but also improves the performance of subsequent queries and processing, thereby improving the processing capacity and response speed of the overall system.

[0072] S50, performing an asynchronous merge operation to merge the optimized incremental log file with the base table file in the MOR table to generate a new base table file in the MOR table.

[0073] In this embodiment, compaction is the process of merging incremental log files and base table files. In traditional synchronous compaction, the task may suspend processing to complete the merge, while asynchronous compaction executes the merge operation in the background without blocking other real-time processing tasks. Through the asynchronous method, the system can gradually complete the data merge in the background, reducing the impact on the foreground data processing.

[0074] Flink tasks execute compaction operations as a background task, which does not affect the data flow processing of the main task. Compaction tasks will be triggered periodically or according to preset conditions (such as log file size or time interval). The system uses asynchronous threads to handle merge operations to ensure that the main processing flow of the system is not blocked. Asynchronous compaction can be triggered by a preset scheduling strategy, such as checking whether the incremental log file meets the merge condition at regular intervals. When the conditions are met, the compaction operation starts and the merged data is saved as a new base table file.

[0075] The optimized incremental log file generated after sharding and parallel processing has completed the data compression, deduplication and sorting. The base table file is the basic data storage unit in the MOR table, containing the original full data. The data in the incremental log file represents the new or modified data since the last merge. The purpose of the merge operation is to integrate this data with the base table file to maintain data consistency.

[0076] The merge operation requires comparing the data in the incremental log file and the base table file. Usually, the data is matched by the primary key, and the new data in the incremental log file replaces the old data in the base table to ensure that the latest data is merged into the base table file. When merging, the Flink task reads the incremental log file and the base table file, compares and updates the data of each shard, and finally generates a new base table file. This process involves a large number of disk reads, writes, and sorting operations, so a good resource scheduling mechanism is required.

[0077] The new base table file after merging is the final storage file in the MOR table, containing the latest complete data. After the incremental data is merged, the new base table file replaces the old base table file to ensure that the data can maintain consistency and the latest status when querying.

[0078] After the new base table file is generated, it will overwrite the old base table file. At the same time, the old file will be marked as expired or invalid and cleaned up. The data merge will be compressed, sorted, and deduplicated to ensure that the new base table file data is stored more efficiently and reduce storage space. The new base table file is sorted and compressed, which improves the efficiency of data query. When querying, you can directly access the merged complete data set, reducing multiple read I / O operations, thereby improving query performance.

[0079] For example, when merging, the data in the incremental log file and the base table file are matched by the primary key. Only the data that matches the primary key will be updated, and the data that does not match will remain unchanged. The Flink task reads the data in the incremental log file, sorts it by the primary key, compares it with the base table file, writes the incremental data to the new base table file, and marks and cleans up the expired old base table file.

[0080] Through asynchronous compaction operations, data merging no longer blocks real-time processing tasks, improving the system's concurrent processing capabilities. By merging incremental log files with base table files, data consistency and integrity are ensured. The generated new base table files are optimized, significantly improving storage efficiency and reducing the I / O burden during queries. This reduces system resource usage, improves system throughput and data processing stability, thereby ensuring the efficiency and performance optimization of the data entry process.

[0081] The present invention relates to the fields of data processing technology and financial technology, and discloses a Flink-based incremental data entry method, which obtains incremental data from a data source through a Flink task, and the data source includes a real-time streaming data source and a static batch processing data source. After the incremental data is stream processed or batch processed, it is written into an incremental log file of a MOR table. During the stream processing process, a state saving operation is performed regularly and the data processing state is saved, and the final processing state is saved when the batch processing task is completed. The incremental log file is fragmented, and an optimized incremental log file is generated through distributed node or multi-threaded parallel processing and merging operations. Subsequently, the merge operation is performed asynchronously to merge the optimized incremental log file with the base table file in the MOR table to generate a new base table file. The present invention divides the incremental log file into multiple small data blocks and processes them independently through fragmentation processing and parallel processing mechanisms, fully utilizes the computing power of distributed nodes, and realizes efficient processing of large-scale data. At the same time, an asynchronous merge operation is used to merge the incremental log file with the base table file, and the resource-intensive merge operation is placed in the background through asynchronous execution, thereby avoiding the blocking and performance impact of the merge process on the system. In the process of stream processing, the state preservation operation enables the data processing state to be saved regularly to ensure rapid recovery when the task is interrupted, while in batch mode, the processing state of each batch is saved to ensure data consistency. Through these mechanisms, effective management and optimization of system resources are achieved, the burden on the system is reduced, data processing efficiency is improved, and the stability and consistency of data synchronization are ensured, thereby achieving the effect of optimizing system performance and efficiently processing large-scale data during the process of data entering the lake.

[0082] In one embodiment, the above S50 includes:

[0083] S501, the merging operation is periodically triggered and executed by an offline task, the offline task is independent of the Flink task, and the offline task is triggered at a preset time interval;

[0084] S502, selecting multiple incremental log files from the MOR table, and merging the data in the selected incremental log files with the base table file through the offline task;

[0085] S503, during the merging process, the offline task is used to perform compression and sorting operations on the merged data, and duplicate data is deduplicated to retain the latest data;

[0086] S504, when the merge is completed, a base table file in a new SSTable format in the MOR table is generated;

[0087] S505: Mark the incremental log files and base table files used for merging as invalid through the offline task for cleaning and recycling.

[0088] In this embodiment, the asynchronous compaction operation refers to the process of merging the data in the incremental log file with the base table file of the MOR table without affecting the real-time data processing. Asynchronous means that the compaction task is performed in the background and will not block other foreground Flink task processing, ensuring the efficiency and stability of the system. The base table file stores the main data set in the MOR table. By merging with the incremental log file, the base table file is updated and the latest data records are retained.

[0089] Asynchronous compaction is handled by background tasks. The system starts compaction operations through a scheduling mechanism without affecting foreground tasks. The merging process involves reading incremental log files and base table files, and comparing and replacing the updated data in the incremental log with the existing data in the base table file. After the merge is completed, a new base table file is generated and the old file is marked as invalid to ensure that the next query can read the latest merged data.

[0090] Offline tasks are tasks that are not executed in the main data processing flow of Flink. They are usually used to perform time-consuming background operations, such as data merging and storage optimization. Offline tasks are triggered according to preset time intervals to ensure that compaction operations do not frequently interfere with real-time data processing. The selection of time intervals is usually based on system load, data update frequency, and storage requirements.

[0091] The system starts offline tasks through the scheduler to perform compaction operations within the set time interval. Flink tasks continue to process in real time, while compaction tasks are performed independently without interfering with the real-time performance of the system. The interval of regular triggering can be dynamically adjusted according to the accumulation of log files to ensure timely merging after the amount of data reaches a certain level.

[0092] According to the size and number of incremental log files, the system selects multiple log files in batches through offline tasks for merging to improve the efficiency of merging. During the merging process, the data in the incremental log files will update or replace the old data in the base table file to ensure that the latest data can be merged correctly.

[0093] The offline task selects multiple incremental log files from the MOR table, compares them with the base table files one by one, and merges them. This process uses an efficient batch processing algorithm to ensure that the merge operation can be completed in a reasonable time and reduce system resource consumption.

[0094] When merging, the data compression algorithm is used to reduce storage space usage, ensure that the data is more compact, and reduce the consumption of storage resources. The data is sorted to facilitate quick location and retrieval of data in subsequent queries. When the incremental log file is merged with the base table file, there may be duplicate records. The deduplication mechanism ensures that each primary key corresponds to only the latest version of data.

[0095] When merging data, offline tasks first sort the data to ensure the order of the data, and then use compression algorithms to reduce storage usage. When merging, the system checks each primary key and removes old and duplicate records to ensure that the latest updated data is retained.

[0096] SSTable is a file format for storing sorted, immutable key-value pairs, commonly used for efficient data storage and retrieval. The merged data is converted into SSTable format to ensure efficient storage and query performance. After the merge is completed, the offline task writes the merged data to a new SSTable file and adds it to the MOR table. The SSTable file format provides an efficient storage structure that helps improve query performance and data management.

[0097] The merged incremental log files and old base table files will be marked as invalid, so that the system can clean up the old files and free up storage space. Through the garbage collection mechanism, the system regularly cleans up invalid log files and base table files to ensure that storage space is not occupied by invalid files and improve the overall system performance.

[0098] After the offline task completes the data merge, the old log files and base table files are marked as invalid. Through regular cleanup tasks, the system deletes these invalid files, frees up storage space, and ensures that the system can run efficiently in the long term.

[0099] This embodiment effectively improves the concurrent processing capability of the system and reduces the pressure of real-time processing by using asynchronous compaction operations and offline tasks to merge data. By regularly merging incremental log files, it can ensure that the base table files remain up to date. Compression, sorting, and deduplication operations optimize storage space, reduce duplicate data, and improve query efficiency. The base table file in the SSTable format finally generated ensures efficient storage and fast retrieval, and at the same time releases storage space through an automatic recycling mechanism to ensure efficient and stable operation of the system.

[0100] In one embodiment, the above S502 includes:

[0101] S5021, performing access frequency analysis on the data in the MOR table through the offline task, and dividing the data into hot data and cold data based on the access frequency of the data;

[0102] S5022, based on data classification, storing the hot data in a high-speed partition and storing the cold data in a low-speed partition through the offline task;

[0103] S5023, selecting multiple incremental log files to be processed from the incremental log files of the hotspot data through an offline task, and merging the selected incremental log files with the base table file;

[0104] S5024, delaying the merging operation of the incremental log files of the cold data, and performing the merging process only in a preset time window or when processing resources are idle.

[0105] In this embodiment, multiple incremental log files accumulated in the MOR table are periodically selected for merging. Incremental log files contain data that has been added or modified since the last merge, and the system selects these files in batches through offline tasks. Offline tasks are independent of real-time tasks and are scheduled by periodic triggering or based on system load to ensure that the merge operation does not interfere with real-time processing.

[0106] The offline task selects incremental log files from the MOR table in batches at preset time intervals and merges these log files with the base table files through an asynchronous mechanism. Through the distributed processing architecture, multiple log files can be processed in parallel to ensure efficient merging of data.

[0107] By analyzing the data access patterns in the MOR table, the system can identify frequently accessed data (hot data) and infrequently accessed data (cold data). Hot data is frequently queried and modified, while cold data is less frequently accessed or modified. Based on the classification of access frequency, the system can optimize data storage and processing methods and manage hot data and cold data separately.

[0108] The offline task regularly analyzes the access frequency of the data in the MOR table and uses the monitoring system or statistics module to record the access of the data. The classification criteria for hot data and cold data can be based on a certain time window, such as the access frequency in the past week or dynamically adjusted according to the query history.

[0109] Data is divided into two categories according to access frequency: hot data is stored in high-speed partitions for fast access, and cold data is stored in low-speed partitions to reduce resource consumption. High-speed partitions usually use high-performance storage media, such as SSDs, while low-speed partitions can use cheap storage, such as HDDs. During the offline task merging process, hot data is stored in high-speed partitions first to ensure that the data can be read and processed quickly. Cold data is stored in low-speed partitions to reduce the occupation of high-performance storage and improve system resource utilization efficiency.

[0110] The system prioritizes processing the incremental log files of hot data to ensure that these frequently accessed data can be merged with the base table files as soon as possible so that the system can quickly obtain the latest data when querying. Offline tasks will first select the incremental log files of hot data for processing to ensure that the updates of hot data can be merged in a shorter time. The system will dynamically adjust the merging frequency based on the access frequency of hot data to ensure that the merging of high-frequency data will not delay system performance.

[0111] For cold data, due to the low access frequency, the system can choose to postpone its merge operation to a preset time window or perform it when system resources are idle to reduce the impact on system performance. Offline tasks perform delayed merge operations on incremental log files of cold data, which are usually triggered when the system load is low, such as at night or during non-peak hours. The preset time window ensures that low-priority data does not compete with high-priority data for system resources.

[0112] By analyzing the frequency of data access, the system in this embodiment can intelligently identify hot and cold data, realize data classification processing, and improve the overall processing efficiency of the system. Storing hot data in high-speed partitions can improve the response speed of queries and processing, while delaying the merging of cold data reduces the resource usage and processing pressure of the system. Through this intelligent hierarchical processing strategy, while ensuring data consistency, it maximizes the optimization of system resource utilization and improves the system throughput and response performance.

[0113] In one embodiment, the above S30 includes:

[0114] S301, during the stream processing, periodically performing a state saving operation according to a predetermined time interval or data processing event;

[0115] S302, when executing the state saving operation, asynchronously saving the current data processing state, wherein the current data processing state includes the current processing progress of the stream data and the internal intermediate state of the task;

[0116] S303, when the data processing task is interrupted, resuming the data processing based on the data processing state saved by the most recent state saving operation;

[0117] S304, during the batch processing, the data is divided into a plurality of batches according to predetermined conditions, and a unique batch ID is generated for each batch;

[0118] S305, performing data cleaning and formatting operations on each batch of data, and recording the processing status of each batch when each batch is processed;

[0119] S306, when the batch processing fails, restart the processing task of the failed batch based on the batch ID.

[0120] In this embodiment, Checkpoint in Flink is an important mechanism to ensure task fault tolerance in stream processing, and regularly saves processing status to cope with task interruption. Checkpoint saves the processing progress and internal status of the task, helping the task to continue from the last saved position when a failure occurs.

[0121] Flink tasks automatically perform Checkpoint operations at set time intervals or based on specific data processing events. Checkpoints save the offset of stream data and the intermediate state within the task, which are asynchronously saved to external storage (such as HDFS or S3) to ensure that they do not hinder the normal progress of real-time data processing.

[0122] In order to avoid hindering the processing of real-time data streams, the Checkpoint operation is performed asynchronously. This means that when executing a Checkpoint, the system will asynchronously save the current processing progress and internal state to external storage, without affecting the execution efficiency of real-time tasks. The current processing state includes the progress of the processed data (such as the offset of the Kafka stream) and the intermediate state of the task (such as the intermediate results of the window calculation, etc.).

[0123] When a Flink task performs a Checkpoint operation, it writes the processing progress and internal state to a persistent storage system through an asynchronous save mechanism. When a task fails, the task state can be restored from the most recent successful Checkpoint to avoid data loss or duplicate processing.

[0124] Flink's recovery mechanism is based on the state saved by Checkpoint. When a task is interrupted or fails, the system can automatically recover the task from the most recent successful Checkpoint. The recovered data includes processing offsets and intermediate calculation states. When a task is interrupted due to an exception, Flink automatically loads the state saved by the most recent Checkpoint, recovers the current processing progress and intermediate state of the task, and continues to execute data processing to ensure the consistency and accuracy of the task.

[0125] In batch processing tasks, the system divides the input data set into multiple batches based on the size of the data, time window, or business needs to ensure that data processing can be performed in segments. A unique batch ID is generated for each batch to facilitate tracking and managing the status of each batch during subsequent processing, especially when processing fails, it can be reprocessed based on the batch ID. When Flink's batch processing task is started, the data is divided according to custom conditions, and each batch of data processing is associated with a unique batch ID. This mechanism ensures that data can be processed in batch order and that the processing progress can be accurately restored in the event of an exception.

[0126] Data cleaning includes removing invalid data and fixing errors in the data, while formatting converts the data into a unified standard format. After processing, the system records the processing status of each batch, including information on success or failure. When processing data, batch tasks first clean and format the data to ensure data consistency. After processing, the system saves the batch status for subsequent viewing or debugging.

[0127] In a batch task, if a batch fails to be processed, the system will restart the processing task of the batch through the batch ID to avoid reprocessing other batches that have been successfully completed. When a batch of a batch task fails to be processed, the system will record the ID of the failed batch. When the task is restored, the system will restart the failed batch processing through the batch ID, and will not repeat the processing of the completed batch data.

[0128] This embodiment ensures high fault tolerance and task recovery capability of stream processing tasks by periodically performing state saving operations. When a task is interrupted, the system can recover from the most recent checkpoint to reduce data loss or repeated processing. For batch processing tasks, by dividing the data into multiple batches and generating batch IDs, data segmentation management and recovery can be achieved, ensuring that the task can be restarted based on the batch ID when a batch fails, avoiding repeated processing of completed data. Overall, the fault tolerance, processing efficiency and system stability of data processing tasks are improved.

[0129] In one embodiment, the above S40 includes:

[0130] S401, dividing the data records of each incremental log file into multiple fragments according to the timestamp, primary key range or number of record rows of the data records in each incremental log file;

[0131] S402, performing parallel processing on the data records of each shard through distributed nodes or multi-threading, wherein the processing includes data cleaning, formatting, and sorting;

[0132] S403, performing deduplication, compression and merging operations on the processed data records in each shard to generate an optimized incremental log file;

[0133] S404, saving the optimized incremental log file to the distributed storage.

[0134] In this embodiment, in order to improve processing efficiency and reduce the load of a single node, the data records in the incremental log file are divided into multiple fragments according to different conditions. The basis for division includes timestamp (recorded time), primary key range (unique identification range of records), and number of record rows (the number of records contained in each fragment). After fragmentation, the data of each fragment can be processed in parallel to avoid affecting the processing speed due to excessive file size.

[0135] The data of each shard is processed independently, usually using distributed nodes or multi-threaded operations in parallel to make full use of system resources. The processing steps include cleaning, formatting and sorting the data to ensure data consistency and optimize storage.

[0136] The processed data shards will be deduplicated, compressed, and merged to generate optimized incremental log files. Deduplication can remove redundant data, compression can save storage space, and the merge operation will integrate the shards into a complete incremental log file.

[0137] The final incremental log files are stored in a distributed system to ensure high availability and fault tolerance, usually using HDFS or cloud storage systems such as S3.

[0138] The incremental log file is split according to the specified conditions (timestamp, primary key range or number of record rows). The amount of data in each shard is small, which is convenient for subsequent parallel processing, ensuring that each node in the multi-node system can efficiently handle its own shard tasks. The system sends the data of each shard to different nodes or threads for processing. The data operations performed include data cleaning (such as filtering invalid or erroneous data), formatting (ensuring data structure consistency), and sorting (improving the efficiency of subsequent queries or processing). The deduplication operation ensures the uniqueness of each primary key and avoids data duplication, and the compression operation reduces the storage space occupied. The final merge operation integrates the processed multiple shards into a complete incremental log file, ensuring a compact file structure and efficient storage. The optimized incremental log files are stored in a distributed storage system, and the system can access and process these data simultaneously through multiple nodes to improve data availability and fault tolerance.

[0139] By sharding data using the timestamp of each piece of data, you can divide the data into multiple parts by hour, day, or other time intervals. This method is suitable for scenarios where data has temporal continuity or segmented processing.

[0140] In addition, the data can be divided according to the range of the primary key, which is suitable for scenarios where the primary key values ​​are ordered. For example, the data can be divided into multiple shards according to the interval range of the primary key, and each shard processes a part of the data in the primary key value range.

[0141] You can also divide the data into multiple shards according to the specified number of rows, which is suitable for data sources without clear primary keys or timestamps, ensuring that the amount of data in each shard is roughly the same.

[0142] After the incremental log file is generated, the system will shard the data according to the set conditions (such as time, primary key or number of rows). The system can dynamically select the sharding basis and choose the optimal sharding method according to different data characteristics to achieve load balancing and efficient processing.

[0143] Through distributed architecture or multi-threading technology, the data of each shard can be processed in parallel. Distributed nodes are usually distributed on multiple servers, and each server processes different shard data. The cleaning step includes filtering invalid or erroneous data to ensure data quality. For example, delete duplicate records or correct outliers in the data. Format the data uniformly to ensure data consistency. For example, date fields can be unified into the same format and strings can be standardized. Sort the data according to the primary key or other fields to facilitate subsequent merge operations and query processing.

[0144] Each shard data is assigned to different nodes or threads and processed through predefined operation steps. First, data cleaning is performed to remove erroneous or invalid data. Then formatting is performed to ensure the consistency of data structure. Finally, the data is sorted to ensure consistency when the subsequent data is merged.

[0145] During parallel processing, duplicate data may appear. Deduplication ensures that only unique records are retained in each shard, especially duplicate data with the same primary key. Compress the data to reduce storage space. Common compression algorithms such as Snappy and LZ4 can efficiently compress large-scale data. After processing, multiple shards need to be merged into a complete log file to ensure data integrity and consistency.

[0146] During the deduplication process, the system scans the data of each shard, checks whether the primary key is repeated, and deletes redundant data. Subsequently, the system compresses the deduplicated data to reduce storage space. Finally, all shard data is merged into an optimized incremental log file for subsequent query and storage.

[0147] The incremental log files are stored in a distributed file system. Common storage systems include HDFS, S3, etc. These systems have high availability and fault tolerance, ensuring that data is still accessible even if a node fails. The optimized incremental log files are written to a distributed storage system such as HDFS or a cloud storage system through a storage interface. The storage system automatically stores the file shards on multiple nodes and provides replicas to ensure data persistence and reliability.

[0148] This embodiment improves the efficiency of the system when processing large-scale data by sharding and parallel processing of incremental log files. Deduplication, compression, and merging operations optimize the storage and processing process, reduce the storage space occupied by duplicate data, and improve the performance of data reading. Finally, the optimized incremental log files are stored in a distributed system to ensure high availability and scalability of data, significantly improving the overall performance and processing efficiency of the system.

[0149] In one embodiment, in the above S10, obtaining incremental data from a data source through a Flink task includes:

[0150] S101, connect the real-time streaming data source through the stream connector of the Flink task, or start the static batch data source through the batch trigger mechanism, and establish connections for the real-time streaming data source and the static batch data source respectively;

[0151] S102, using the DataStream API through the Flink task to subscribe to incremental data from the real-time streaming data source, and perform data cleaning, format conversion, and time window division;

[0152] S103, periodically pull batch incremental data from the static batch data source through Flink tasks, and use DataSetAPI to perform data cleaning and formatting;

[0153] S104, through the parallel processing mechanism of Flink tasks, the incremental data in the real-time streaming data source and the static batch data source are processed simultaneously;

[0154] S105, when the connection is interrupted, the current data processing state is stored by the Flink task using the state saving mechanism, and the processing is continued from the last data processing state after the data source connection is restored.

[0155] In this embodiment, Flink is a distributed stream processing framework for processing real-time and batch data. In the Flink task, the system connects to the data source through different connectors and selects the appropriate API for processing according to the type of source data (streaming or batch). Incremental data refers to the new or modified data relative to the last processing. The Flink task specifically processes these changed data to improve the efficiency of data processing. Flink tasks can dynamically connect to multiple data sources (such as Kafka, HDFS, etc.) and extract incremental data from these sources. The system will decide whether to use stream processing (DataStreamAPI) or batch processing (DataSetAPI) to obtain and process data based on the type of data source.

[0156] Flink connects to real-time streaming data sources through stream connectors. Common streaming data sources include Kafka, Kinesis, etc. Through these connectors, Flink can receive and process data streams in real time. For static data sources (such as HDFS, databases, etc.), Flink regularly pulls batch data from these data sources through a batch trigger mechanism, which is usually used to process bounded data sets.

[0157] Flink tasks use corresponding connectors according to the type of data source. For streaming data sources such as Kafka, tasks subscribe to data streams through Kafka Connectors; for static data sources, the system periodically starts batch tasks through a scheduling trigger mechanism to obtain data.

[0158] The DataStream API provided by Flink is used to process unbounded data streams and supports subscribing to incremental data from real-time streaming data sources (such as Kafka). Real-time data may contain noise or erroneous information. The system filters out invalid or abnormal data through data cleaning steps. Formatting conversion is to standardize data into a unified structure to ensure the consistency of subsequent processing. Time windows are a way to process unbounded data streams in segments. By setting time windows, Flink can process real-time data within a specified time (such as incremental data within every 10 seconds).

[0159] Flink tasks subscribe to incremental data streams through stream connectors such as Kafka, use DataStreamAPI to receive data and perform cleaning and formatting operations. The system divides the data into multiple time periods according to the set window time, and each time period processes incremental data independently.

[0160] Flink's DataSet API is an API for bounded data sets (static data sources) used in batch processing scenarios. Unlike unbounded data streams, batch processing tasks acquire the entire data set for processing at one time. Similar to stream processing, batch processing tasks also need to clean and format the acquired data to ensure that the data is consistent and error-free when written to the target storage.

[0161] Flink tasks regularly pull incremental data from static batch data sources (such as HDFS, databases) through a scheduling mechanism, and use the DataSet API to process the data and perform cleaning and formatting operations. After processing, the data is written to incremental log files or other storage systems.

[0162] The power of Flink lies in its ability to improve task execution efficiency through parallel processing. Flink tasks can simultaneously process incremental data from different sources, such as streaming data sources and batch data sources. Flink's parallel mechanism enables the system to process real-time streaming data and static batch data at the same time, ensuring that the system can achieve efficient incremental data processing between different data sources.

[0163] Flink tasks are configured with multiple parallel subtasks to process incremental data from streaming and batch data sources respectively. Through parallel processing of data, the system can efficiently process multiple data sources at the same time, improving overall processing performance.

[0164] Flink's Checkpoint mechanism is used to save the current processing status of the task, especially when the data source connection is interrupted or the task fails, the system can restore the task from the most recent Checkpoint to avoid data loss or repeated processing. When the data source is restored, the Flink task will continue processing from the last saved Checkpoint to ensure the consistency of the task. Flink tasks use the Checkpoint mechanism to periodically save the intermediate state of data processing, including the offset of stream data, the internal state of the task, etc. When the data source connection is interrupted, the task state will be saved to external storage (such as HDFS, S3). After the connection is restored, the task will continue to execute from the last saved state.

[0165] This embodiment uses Flink tasks to obtain incremental data from multiple data sources, and combines stream processing and batch processing technologies to efficiently process large-scale data. At the same time, Flink tasks support parallel processing and can process real-time streaming data and static batch data at the same time, greatly improving processing speed and resource utilization efficiency. When a task is interrupted, the Checkpoint mechanism ensures the fault tolerance and continuity of data processing, ensuring the high availability of the system in complex scenarios.

[0166] In one embodiment, in the above S20, writing the processed data into the incremental log file of the MOR table includes:

[0167] S201, writing the processed data into a memtable buffer in the memory;

[0168] S202, when the data in the memtable buffer reaches a set threshold, converting the memtable into an immutable table;

[0169] S203, refreshing the data of the immutable table into the incremental log file of the MOR table.

[0170] In this embodiment, under the MOR (Merge-on-Read) table structure, the incremental log file is used to temporarily store the processed data. Before the data is written to the base table file, the processed incremental data will first enter the incremental log file to ensure that the gradual merging of the data can proceed smoothly. After the data is processed by the Flink task, it will not be directly written to the base table file of the MOR table, but will be written to the incremental log file as the data source for subsequent asynchronous merge operations.

[0171] Memtable is a data structure in memory that is used to temporarily store write operations. Whenever new data processing is completed, the data is first written to the memtable instead of being written directly to the disk or file. Memtable provides efficient write speed and supports batch processing of data. When the Flink task completes data processing, the data is written to the memtable buffer. The memtable in memory can quickly respond to data write operations and temporarily store data until a certain threshold is reached. This operation avoids the I / O bottleneck caused by frequent disk writes.

[0172] When the data in the memtable reaches the set threshold (usually a memory limit or a data volume limit), the memtable will be converted to an immutable table, also known as an Immutable Table. An immutable table is a data structure that can no longer be modified, which can ensure the integrity and consistency of the data and is ready to be written to persistent storage. When the amount of data in the memtable reaches the predetermined threshold, the system will "freeze" the data in the memtable and convert it to an immutable table. At this point, the memtable no longer receives new write operations, but is ready to refresh the data from memory to the incremental log file or disk. At the same time, a new memtable will be created to receive new incremental data.

[0173] The process of writing the data of the immutable table to the incremental log file is called "data refresh". This process persists the immutable table data in memory to the incremental log file of the MOR table, ensuring the persistence and recoverability of the data. After the contents of the immutable table are converted, the system triggers the writing of this data to the incremental log file of the MOR table. This process involves writing the memory data to disk to ensure the safe and persistent storage of the data. These incremental log files can then be merged with the base table file of the MOR table through asynchronous compaction operations.

[0174] By using the Memtable buffer, this embodiment can significantly improve the data writing performance and avoid frequent disk I / O operations. When the data in the memtable reaches the set threshold, it is converted into an immutable table and persisted to the incremental log file of the MOR table to ensure data integrity and consistency. This mechanism not only speeds up the data writing speed, but also provides a basis for subsequent merge operations, improving the overall efficiency of data processing.

[0175] In one embodiment, a Flink-based incremental data entry lake device is provided, and the Flink-based incremental data entry lake device corresponds one-to-one to the Flink-based incremental data entry lake method in the above embodiment. Figure 3 , Figure 3 This is a functional module diagram of a preferred embodiment of the incremental data entry device based on Flink of the present invention. It includes a data source connection module 10, a data processing and writing module 20, a state management module 30, a data sharding and parallel processing module 40 and a data merging module 50. The functional modules are described in detail as follows:

[0176] A data source connection module 10 is used to obtain incremental data from a data source through a Flink task, wherein the data source includes a real-time streaming data source or a static batch data source;

[0177] A data processing and writing module 20 is used to perform stream processing or batch processing on the incremental data using the Flink task according to the type of the data source, and write the processed data into an incremental log file of the MOR table;

[0178] The state management module 30 is used to periodically perform state saving operations and save the processing state during the stream processing process; when the batch processing is completed, the final processing state is maintained;

[0179] The data sharding and parallel processing module 40 is used to perform sharding processing on the incremental log file, and perform independent parallel processing and merging operations on the data of each shard to generate an optimized incremental log file;

[0180] The data merging module 50 is used to perform an asynchronous merging operation to merge the optimized incremental log file with the base table file in the MOR table to generate a new base table file in the MOR table.

[0181] In one embodiment, the data merging module 50 is specifically used for:

[0182] The merging operation is periodically triggered and executed by an offline task, which is independent of the Flink task and is triggered at a preset time interval;

[0183] Select multiple incremental log files from the MOR table, and merge the data in the selected incremental log files with the base table file through the offline task;

[0184] During the merging process, the offline task is used to compress and sort the merged data, deduplicate the duplicate data, and retain the latest data;

[0185] When the merge is completed, a base table file in a new SSTable format in the MOR table is generated;

[0186] The incremental log files and base table files used for merging are marked as invalid through the offline task for cleaning and recycling.

[0187] In one embodiment, the data merging module 50 is specifically used for:

[0188] Performing access frequency analysis on the data in the MOR table through the offline task, and dividing the data into hot data and cold data based on the access frequency of the data;

[0189] Based on data classification, the hot data is stored in a high-speed partition and the cold data is stored in a low-speed partition through the offline task;

[0190] Select multiple incremental log files to be processed from the incremental log files of hotspot data through offline tasks, and merge the selected incremental log files with the base table files;

[0191] The merging operation of the incremental log files of the cold data is delayed, and the merging process is performed only in a preset time window or when processing resources are idle.

[0192] In one embodiment, the state management module 30 is specifically used to:

[0193] During stream processing, state preservation operations are performed periodically at predetermined time intervals or data processing events;

[0194] When executing the state saving operation, asynchronously saving the current data processing state, the current data processing state includes the current processing progress of the stream data and the internal intermediate state of the task;

[0195] When a data processing task is interrupted, data processing is resumed based on the data processing state saved by the most recent state saving operation;

[0196] In batch processing, data is divided into multiple batches according to predetermined conditions, and a unique batch ID is generated for each batch;

[0197] Perform data cleaning and formatting operations on each batch of data, and record the processing status of each batch when each batch is processed;

[0198] When batch processing fails, restart the processing task of the failed batch based on the batch ID.

[0199] In one embodiment, the data sharding and parallel processing module 40 is specifically used to:

[0200] Divide the data records of each incremental log file into multiple shards according to the timestamp, primary key range or number of record rows of the data records in each incremental log file;

[0201] Processing the data records of each shard in parallel through distributed nodes or multi-threading, including data cleaning, formatting and sorting;

[0202] Perform deduplication, compression, and merging operations on the data records in each processed shard to generate optimized incremental log files;

[0203] Save the optimized incremental log files to distributed storage.

[0204] In one embodiment, the data source connection module 10 is specifically used for:

[0205] Connect the real-time streaming data source through the stream connector of the Flink task, or start the static batch data source through the batch trigger mechanism to establish connections for the real-time streaming data source and the static batch data source respectively;

[0206] Use the DataStream API to subscribe to incremental data from real-time streaming data sources through Flink tasks, and perform data cleaning, format conversion, and time window division;

[0207] Use Flink tasks to periodically pull batch incremental data from static batch data sources, and use the DataSet API to perform data cleansing and formatting.

[0208] Through the parallel processing mechanism of Flink tasks, the incremental data in real-time streaming data sources and static batch data sources can be processed simultaneously;

[0209] When the connection is interrupted, the Flink task uses the state preservation mechanism to store the current data processing status, and continues processing from the last data processing status after the data source connection is restored.

[0210] In one embodiment, the data processing and writing module 20 is specifically used for:

[0211] Writing the processed data into a memtable buffer in memory;

[0212] When the data in the memtable buffer reaches a set threshold, the memtable is converted into an immutable table;

[0213] The data of the immutable table is refreshed into the incremental log file of the MOR table.

[0214] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client through a network connection. When the computer program is executed by the processor, it implements a function or step on the server side of a Flink-based incremental data entry method.

[0215] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the client side of a Flink-based incremental data entry method

[0216] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0217] Acquire incremental data from a data source through a Flink task, where the data source includes a real-time streaming data source or a static batch data source;

[0218] According to the type of the data source, the Flink task is used to perform stream processing or batch processing on the incremental data, and the processed data is written into the incremental log file of the MOR table;

[0219] During stream processing, state saving operations are performed periodically and the processing status is saved; when batch processing is completed, the final processing status is maintained;

[0220] The incremental log file is sliced ​​and processed, and the data of each slice is independently processed in parallel and merged to generate an optimized incremental log file;

[0221] An asynchronous merge operation is performed to merge the optimized incremental log file with the base table file in the MOR table to generate a new base table file in the MOR table.

[0222] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0223] Acquire incremental data from a data source through a Flink task, where the data source includes a real-time streaming data source or a static batch data source;

[0224] According to the type of the data source, the Flink task is used to perform stream processing or batch processing on the incremental data, and the processed data is written into the incremental log file of the MOR table;

[0225] During stream processing, state saving operations are performed periodically and the processing status is saved; when batch processing is completed, the final processing status is maintained;

[0226] The incremental log file is sliced ​​and processed, and the data of each slice is independently processed in parallel and merged to generate an optimized incremental log file;

[0227] An asynchronous merge operation is performed to merge the optimized incremental log file with the base table file in the MOR table to generate a new base table file in the MOR table.

[0228] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0229] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0230] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0231] It should be noted that if software tools or components other than those of the Company appear in the embodiments of the present application, they are only used for illustration and do not represent actual use. The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above-mentioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for incremental data entry into a lake based on Flink, characterized in that: The following steps are involved: Acquire incremental data from a data source through a Flink task, where the data source includes a real-time streaming data source or a static batch data source; According to the type of the data source, the Flink task is used to perform stream processing or batch processing on the incremental data, and the processed data is written into the incremental log file of the MOR table; During stream processing, state saving operations are performed periodically and the processing status is saved; when batch processing is completed, the final processing status is maintained; The incremental log file is sliced ​​and processed, and the data of each slice is independently processed in parallel and merged to generate an optimized incremental log file; An asynchronous merge operation is performed to merge the optimized incremental log file with the base table file in the MOR table to generate a new base table file in the MOR table.

2. The Flink-based incremental data entry method according to claim 1, characterized in that: Performing an asynchronous merge operation to merge the optimized incremental log file with the base table file in the MOR table to generate a new base table file in the MOR table, including: The merging operation is periodically triggered and executed by an offline task, which is independent of the Flink task and is triggered at a preset time interval; Select multiple incremental log files from the MOR table, and merge the data in the selected incremental log files with the base table file through the offline task; During the merging process, the offline task is used to compress and sort the merged data, deduplicate the duplicate data, and retain the latest data; When the merge is completed, a base table file in a new SSTable format in the MOR table is generated; The incremental log files and base table files used for merging are marked as invalid through the offline task for cleaning and recycling.

3. The Flink-based incremental data entry method according to claim 2, characterized in that: Selecting multiple incremental log files from the MOR table, and merging the data in the selected incremental log files with the base table file through the offline task, includes: Performing access frequency analysis on the data in the MOR table through the offline task, and dividing the data into hot data and cold data based on the access frequency of the data; Based on data classification, the hot data is stored in a high-speed partition and the cold data is stored in a low-speed partition through the offline task; Select multiple incremental log files to be processed from the incremental log files of hotspot data through offline tasks, and merge the selected incremental log files with the base table files; The merging operation of the incremental log files of the cold data is delayed, and the merging process is performed only in a preset time window or when processing resources are idle.

4. The Flink-based incremental data entry method according to claim 1, characterized in that: During stream processing, state preservation operations are performed periodically to save the processing status; when batch processing is completed, the final processing status is maintained, including: During stream processing, state preservation operations are performed periodically at predetermined time intervals or data processing events; When executing the state saving operation, asynchronously saving the current data processing state, the current data processing state includes the current processing progress of the stream data and the internal intermediate state of the task; When a data processing task is interrupted, data processing is resumed based on the data processing state saved by the most recent state saving operation; In batch processing, data is divided into multiple batches according to predetermined conditions, and a unique batch ID is generated for each batch; Perform data cleaning and formatting operations on each batch of data, and record the processing status of each batch when each batch is processed; When batch processing fails, restart the processing task of the failed batch based on the batch ID.

5. The Flink-based incremental data entry method as claimed in claim 1, characterized in that: The incremental log file is sliced ​​and processed, and the data of each slice is independently processed in parallel and merged to generate an optimized incremental log file, including: Divide the data records of each incremental log file into multiple shards according to the timestamp, primary key range or number of record rows of the data records in each incremental log file; Processing the data records of each shard in parallel through distributed nodes or multi-threading, including data cleaning, formatting and sorting; Perform deduplication, compression, and merging operations on the data records in each processed shard to generate optimized incremental log files; Save the optimized incremental log files to distributed storage.

6. The Flink-based incremental data entry method according to claim 1, characterized in that: Get incremental data from the data source through Flink tasks, including: Connect the real-time streaming data source through the stream connector of the Flink task, or start the static batch data source through the batch trigger mechanism to establish connections for the real-time streaming data source and the static batch data source respectively; Use the DataStream API to subscribe to incremental data from real-time streaming data sources through Flink tasks, and perform data cleaning, format conversion, and time window division; Use Flink tasks to periodically pull batch incremental data from static batch data sources, and use the DataSet API to perform data cleansing and formatting. Through the parallel processing mechanism of Flink tasks, the incremental data in real-time streaming data sources and static batch data sources can be processed simultaneously; When the connection is interrupted, the Flink task uses the state preservation mechanism to store the current data processing status, and continues processing from the last data processing status after the data source connection is restored.

7. The incremental data entry method based on Flink as claimed in claim 1, characterized in that: Write the processed data to the incremental log file of the MOR table, including: Writing the processed data into a memtable buffer in memory; When the data in the memtable buffer reaches a set threshold, the memtable is converted into an immutable table; The data of the immutable table is refreshed into the incremental log file of the MOR table.

8. A Flink-based incremental data entry device, characterized in that: The Flink-based incremental data entry device includes: A data source connection module is used to obtain incremental data from a data source through a Flink task. The data source includes a real-time streaming data source or a static batch data source. A data processing and writing module, used to perform stream processing or batch processing on the incremental data using the Flink task according to the type of the data source, and write the processed data into an incremental log file of the MOR table; The state management module is used to periodically perform state saving operations and save the processing status during the stream processing process; when the batch processing is completed, the final processing status is maintained; The data sharding and parallel processing module is used to perform sharding processing on the incremental log file, and perform independent parallel processing and merging operations on the data of each shard to generate an optimized incremental log file; The data merging module is used to perform an asynchronous merging operation to merge the optimized incremental log file with the base table file in the MOR table to generate a new base table file in the MOR table.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a Flink-based incremental data entry into the lake program stored in the memory and executable on the processor. When the Flink-based incremental data entry into the lake program is executed by the processor, the steps of the Flink-based incremental data entry into the lake method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The storage medium stores a Flink-based incremental data entry into the lake program, and when the Flink-based incremental data entry into the lake program is executed by the processor, the steps of the Flink-based incremental data entry into the lake method as described in any one of claims 1 to 7 are implemented.

Images