Real-time update insertion merge processing method based on distributed file system streaming data

Abstract

The application discloses a kind of based on distributed file system streaming data real-time update insertion merging processing method, the method includes: step 1, streaming data is accessed on distributed file system with single table as unit and is partitioned according to time interval operation;Step 2, the value of the primary key field in stream data is used as the key value of the data;Step 3, the key value of data and the partition number where the current data is located are established unique mapping relationship, and the mapping relationship is stored in the third-party database, and the construction of index is completed using the third-party database;Step 4, when each streaming data starts to access, update or insertion operation is executed;Step 5, according to trigger mechanism to merge the file in partition folder;Solve the merging operation of the real-time update insertion of streaming data in the prior art distributed file system.

Description

Technical Field

[0001] This invention belongs to the field of big data streaming data processing technology, and in particular relates to a method for real-time update, insertion and merging of streaming data based on a distributed file system. Background Technology

[0002] With the increasing volume of data and the growing complexity of business processes across industries, the demand for timeliness in processing large volumes of data using distributed file systems (DSPs) is also rising. In previous scenarios where data was accessed through DSPs, processing was limited to batch processing, with no real-time updates or merging of inserted data, resulting in significant latency in data analysis. Simultaneously, as data access to DSPs shifts towards streaming environments, the challenge of effectively updating and merging streaming data within DSPs has become a critical technical issue that needs to be addressed. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a method for real-time update, insertion and merging of streaming data based on a distributed file system, so as to solve the problem that the existing technology cannot perform real-time update and insertion merging operations on streaming data in a distributed file system.

[0004] Technical solution of the present invention:

[0005] A method for real-time update, insertion, and merging of streaming data based on a distributed file system, the method comprising:

[0006] Step 1: On the distributed file system, partition the incoming streaming data into individual tables at time intervals.

[0007] Step 2: Use the value of the primary key field in the streaming data as the key value for this data entry;

[0008] Step 3: Establish a unique mapping relationship between the key value of the data and the partition number where the current data is located, store this mapping relationship in a third-party database, and use the third-party database to complete the index construction;

[0009] Step 4: When each stream of data begins to be received, perform an update or insert operation;

[0010] Step 5: Merge the files in the partition folder according to the trigger mechanism.

[0011] Each partition is a separate folder, and the folder name is the same as the partition name. The partition name is the creation time of the current folder, in the format 'year month day hour minute second millisecond'.

[0012] The key value is a globally unique identifier for a piece of data, used to determine whether the data belongs to an update operation or an insert operation.

[0013] The method for determining whether to perform an update or insert operation is as follows: First, search for the key value of the current data in the index table of the third-party database. If the key exists, the data is an update operation; if the key does not exist, the data is an insert operation.

[0014] The update and insert operations are performed as follows: if the index can locate the partition where the current data is located, the data is directly appended to the log file of the current partition; the insert operation directly appends the data to the log file of the latest partition, and at the same time adds an index to the data in the index library.

[0015] The method for merging files is as follows: There are two different file types in each partition folder, and the files in the partition folder are merged according to the trigger mechanism; a log file is used to store the newly added data generated before a single merge, and a Parquet file is used to store the merged data. The log file will be deleted after each merge.

[0016] Files can be merged either actively or passively. Passive merging refers to merging all partitions when accessing a table, and then accessing the latest Parquet file after merging as the latest data.

[0017] Active merging refers to merging all previous partitions when a new partition is created.

[0018] The beneficial effects of this invention:

[0019] The present invention provides a method for real-time update, insertion, and merging of streaming data based on a distributed file system, which can solve the update and insertion problem of streaming data based on a distributed file system, enabling real-time access to the latest data in the distributed file system.

[0020] This solves the problem of merging operations that cannot perform real-time updates and insertions of streaming data in existing distributed file systems. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the process of the present invention. Detailed Implementation

[0022] The present invention discloses a method for real-time update, insertion, and merging of streaming data based on a distributed file system, comprising the following steps:

[0023] Step A: Partitioning

[0024] We first partition the incoming streaming data on the distributed file system, using individual tables and time intervals. Each partition is a separate folder, named after the partition. The partition name is the creation time of the current folder, in the format 'year month day hour minute second millisecond', accurate to the millisecond, for example: 20221109010101001.

[0025] Within each partition, we define two different file extensions based on their intended use: *.parquet and *.log. The log file stores newly added data for that partition, while the parquet file stores data after merging.

[0026] Step B: Construct Key

[0027] We use the value of the primary key field in the streaming data as the key value of this data. The key value is a globally unique identifier for a data record, which is used to determine whether the data record belongs to an update operation or an insert operation.

[0028] Step C: Build the index

[0029] As data accumulates and time goes by, we will generate a large number of partitions. Therefore, we built an indexing mechanism to help us determine whether the current data belongs to an update operation or an insert operation, and if it belongs to an update operation, quickly obtain the partition number where the data belongs.

[0030] We establish a unique mapping between the data's key value and the partition number where the data resides. This mapping remains unchanged after the data is first written. We store this mapping in a third-party database and use the database to build the index.

[0031] Step D: Update Insertion

[0032] When each piece of streaming data is first received, the system first searches for the key value of that data in the index table of the third-party database. If the key exists, the data is updated. If the key does not exist, the data is inserted.

[0033] Depending on the type of data operation, different data processing methods are used. For update operations, we can quickly locate the partition where the current data is located based on the index and directly append the data to the log file of the current partition. For insert operations, we directly append the data to the log file of the latest partition and simultaneously add an index to the data in the index database.

[0034] Step E: Merge

[0035] Each partition folder contains two different file types, and we merge files within the partition folder based on a trigger mechanism. We use a log file to store newly added data generated before each merge, and a Parquet file to store the merged data. The log file is deleted after each merge. During initialization until the first merge, there are no Parquet files in the folder; only log files are possible. Parquet files only appear after the first merge. If the folder only contains Parquet files and no log files, it means that no new data has been added to that partition since the last merge, so that partition does not need to be merged in the current merge.

[0036] We categorize the merging trigger mechanism into two scenarios: active merging and passive merging. A passive merging occurs when all partitions are merged upon table access, and the latest merged Parquet file is accessed as the newest data. An active merging occurs when all previous partitions are merged each time a new partition is created. Both active and passive merging work together to merge streaming data.

Claims

1. A method for real-time update, insertion, and merging of streaming data based on a distributed file system, characterized in that: The method includes: Step 1: On the distributed file system, partition the incoming streaming data into individual tables at time intervals. Step 2: Use the value of the primary key field in the streaming data as the key value for this data entry; Step 3: Establish a unique mapping relationship between the key value of the data and the partition number where the current data is located, store this mapping relationship in a third-party database, and use the third-party database to complete the index construction; Step 4: When each stream of data begins to be received, perform an update or insert operation; Step 5: Merge files in the partition folder according to the trigger mechanism. The method for merging files is as follows: There are two different file types in each partition folder. Merge the files in the partition folder according to the trigger mechanism. Use a log file to store the newly added data generated before each merge, and use a Parquet file to store the merged data. The log file will be deleted after each merge.

2. The method for real-time update, insertion, and merging of streaming data based on a distributed file system according to claim 1, characterized in that: Each partition is a separate folder, and the folder name is the same as the partition name; The partition name is the creation time of the current folder, in the format of 'year month day hour minute second millisecond'.

3. The method for real-time update, insertion, and merging of streaming data based on a distributed file system according to claim 1, characterized in that: The key value is a globally unique identifier for a piece of data, used to determine whether the data belongs to an update operation or an insert operation.

4. The method for real-time update, insertion, and merging of streaming data based on a distributed file system according to claim 1, characterized in that: The method for determining whether to perform an update or insert operation is as follows: First, search for the key value of the current data in the index table of the third-party database. If the key exists, the data is an update operation; if the key does not exist, the data is an insert operation.

5. The method for real-time update, insertion, and merging of streaming data based on a distributed file system according to claim 4, characterized in that: The update and insert operations are performed as follows: if the index can locate the partition where the current data is located, the data is directly appended to the log file of the current partition; the insert operation directly appends the data to the log file of the latest partition, and at the same time adds an index to the data in the index library.

6. The method for real-time update, insertion, and merging of streaming data based on a distributed file system according to claim 1, characterized in that: Files can be merged either actively or passively. Passive merging refers to merging all partitions when accessing a table, and then accessing the latest Parquet file after merging as the latest data.

7. The method for real-time update, insertion, and merging of streaming data based on a distributed file system according to claim 6, characterized in that: Active merging refers to merging all previous partitions whenever a new partition is created.

Images