A method, apparatus, device, and medium for avoiding data duplication

Abstract

This application relates to the field of big data computing technology, specifically to a method, apparatus, device, and medium for avoiding data duplication, used to solve the problem of duplicate data being read when multiple data search engines write the same data sequentially. The method includes: performing data searches using multiple data search engines to obtain data source files corresponding to each data search engine; responding to a directory index determination instruction, calling a directory index determination unit pre-set for each data search engine to parse the data source files and generate a directory index corresponding to each data source file, wherein the parsing rules of the directory index determination units set for each data search engine are the same; and calling each data search engine to write the data source file into the data storage unit corresponding to the determined directory index of the data source file.

Description

Technical Field

[0001] This application relates to the field of big data computing technology, specifically to a method, apparatus, device, and medium for avoiding data duplication. Background Technology

[0002] Typically, a single data engine is used to update, insert, and delete data on the Hadoop Distributed File System (HDFS). Hudi, as a data lake storage format, provides the ability to update and delete data, as well as consume changed data, on top of HDFS.

[0003] However, in a special business scenario where multiple engines are used to write data, the real-time stream processing data engine processes the streaming data obtained in a specific order (such as the order in which events occur) and writes it into the Hudi data lake. Then, the batch processing data engine processes some data that was missed by the real-time stream processing data engine or data that needs to be updated and writes it into the Hudi data lake.

[0004] For special business scenarios involving multi-engine data writing, taking data D1 as an example, several situations may arise:

[0005] 1) If the real-time stream processing data engine successfully writes D1, and the batch processing data engine does not include writing D1 when performing batch processing, then data duplication will not occur when performing read queries in this scenario.

[0006] 2) The real-time stream processing data engine successfully writes D1, and the batch processing data engine performs batch processing, including writing D1. In this scenario, if the D1 data processed by the batch processing data engine is completely consistent with the D1 data that already exists in the data lake, D1 data will be written to the data lake again, resulting in duplicate data when performing read queries.

[0007] 3) The real-time stream processing data engine failed to write D1, and there is no data duplication.

[0008] For the second scenario mentioned above, a solution needs to be proposed to avoid retrieving duplicate data from the data lake. Summary of the Invention

[0009] This application provides a method, apparatus, device, and medium for avoiding data duplication, which solves the problem of reading duplicate data when multiple data search engines write the same data sequentially.

[0010] In a first aspect, this application provides a method for avoiding data duplication, the method comprising:

[0011] Data is searched using multiple data search engines to obtain the data source files corresponding to each data search engine.

[0012] In response to the directory index determination instruction, the directory index determination unit pre-set for each data search engine is invoked to parse the data source file and generate a directory index corresponding to each data source file. The parsing rules of the directory index determination unit set for each data search engine are the same.

[0013] Each data search engine is invoked to write the data source file into the data storage unit corresponding to the directory index, based on the determined data source file's directory index.

[0014] In one possible embodiment, the directory index includes a bucket partition identifier (Bucket id) and a file group identifier (File id), and the method further includes:

[0015] In response to a data read command, determine the Bucket and File Group where the data source file to be read is located. Based on the data write type of each data source file, call each data search engine to read the data source file indexed by the Bucket ID and File ID.

[0016] In one possible embodiment, data searching is performed using multiple data search engines, including:

[0017] In response to real-time data write commands, a real-time stream processing data search engine is used to retrieve the data source files corresponding to the real-time data from the front-end server in a specific order.

[0018] In response to a delayed data write command, a batch data search engine is used to retrieve the data source file corresponding to the delayed data from the background storage space.

[0019] In one possible embodiment,

[0020] Pre-define a directory index determination unit independent of each data search engine, as a shared directory index determination unit for all data search engines; or

[0021] Pre-set corresponding directory index determination units in each data search engine.

[0022] In one possible embodiment, a directory index determination unit pre-set for each data search engine is invoked to parse the data source file and determine the directory index corresponding to each data source file, including:

[0023] The system invokes a directory index determination unit pre-set for each data search engine, parses the attribute information of the data source files according to a preset hash algorithm, and generates a directory index corresponding to each data source file.

[0024] In one possible embodiment, the Bucket id is used to identify the bucket partition where the data storage unit corresponding to the data source file is located, and the File id is used to identify the file group corresponding to the data source file in its bucket partition. A bucket partition includes multiple file groups.

[0025] In one possible embodiment, after generating the directory index corresponding to each data source file, the process further includes:

[0026] Store the directory index corresponding to each data source file in a memory variable;

[0027] Call various data search engines to determine the directory index corresponding to the data source file, including:

[0028] The system calls various data search engines to access memory variables and determines the directory index corresponding to each data source file.

[0029] In one possible embodiment, based on the data writing type of each data source file, each data search engine is invoked to write the data source file into the data storage unit corresponding to the directory index, according to the determined directory index of the data source file, including:

[0030] When the data writing type of the data source file is determined to be Copy-on-Write (COW), the corresponding data search engine is invoked to write the data source file as a File Slice into the File Group indexed by Bucketid and Fileid in the format of a Base File.

[0031] In one possible embodiment, based on the data writing type of each data source file, each data search engine is invoked to write the data source file into the data storage unit corresponding to the directory index, according to the determined directory index of the data source file, including:

[0032] When the data writing type of the data source file is determined to be Read-on-Demand Merge (MOR), the corresponding data search engine is invoked to write the data source file into the File Group indexed by the Bucket id and File id in the format of a Log File.

[0033] In one possible embodiment, based on the data write type of each data source file, each data search engine is invoked to read the data source file indexed by the Bucket id and File id, including:

[0034] When the data write type of the data source file is determined to be Copy-on-Write (COW), the corresponding data search engine is called to read the latest Base File in the File Group indexed by the Bucket ID and File ID.

[0035] In one possible embodiment, the method further includes:

[0036] When the data merging cycle is determined, the latest Log File and Base File in the File Group corresponding to the read-to-merge MOR type are merged to obtain a new Base File;

[0037] Based on the data write type of each data source file, the various data search engines are invoked to read the data source files indexed by Bucket ID and File ID, including:

[0038] When the data write type of the data source file is determined to be Read-on-Demand Merge (MOR), the corresponding data search engine is called to read the latest Base File in the File Group indexed by the Bucket ID and File ID.

[0039] Secondly, this application provides an apparatus for avoiding data duplication, the apparatus comprising:

[0040] The data search module is used to search for data using multiple data search engines and obtain the data source files corresponding to each data search engine.

[0041] The directory index determination module is used to respond to the directory index determination instruction by calling the directory index determination unit pre-set for each data search engine to parse the data source file and determine the directory index corresponding to each data source file. The parsing rules of the directory index determination unit set for each data search engine are the same.

[0042] The data writing module is used to call various data search engines to write the data source file into the data storage unit corresponding to the directory index, based on the determined directory index of the data source file.

[0043] In one possible embodiment, the directory index includes a bucket partition identifier (Bucket id) and a file group identifier (File id), and the device further includes:

[0044] The data reading module is used to respond to data reading commands, determine the Bucket and File Group where the data source file to be read is located, and, based on the data writing type of each data source file, call each data search engine to read the data source file indexed by the Bucket ID and File ID.

[0045] In one possible embodiment, the data search module is further configured to:

[0046] In response to real-time data write commands, a real-time stream processing data search engine is used to retrieve the data source files corresponding to the real-time data from the front-end server in a specific order.

[0047] In response to a delayed data write command, a batch data search engine is used to retrieve the data source file corresponding to the delayed data from the background storage space.

[0048] In one possible embodiment, the device pre-configures a directory index determination unit independent of each data search engine, serving as a shared directory index determination unit for all data search engines; or

[0049] Pre-set corresponding directory index determination units in each data search engine.

[0050] In one possible embodiment, the directory index determination module is further configured to call the directory index determination unit pre-set for each data search engine, parse the attribute information of the data source file according to a preset hash algorithm, and generate a directory index corresponding to each data source file.

[0051] In one possible embodiment, the Bucket id is used to identify the bucket partition where the data storage unit corresponding to the data source file is located, and the File id is used to identify the file group corresponding to the data source file in its bucket partition. A bucket partition includes multiple file groups.

[0052] In one possible embodiment, after generating the directory index corresponding to each data source file, the directory index determination module is further configured to:

[0053] Store the directory index corresponding to each data source file in a memory variable;

[0054] The system calls various data search engines to access memory variables and determines the directory index corresponding to each data source file.

[0055] In one possible embodiment, the data writing module is further configured to:

[0056] When the data writing type of the data source file is determined to be Copy-on-Write (COW), the corresponding data search engine is invoked to write the data source file as a File Slice into the File Group indexed by Bucketid and Fileid in the format of a Base File.

[0057] In one possible embodiment, the data writing module is further configured to:

[0058] When the data writing type of the data source file is determined to be Read-on-Demand Merge (MOR), the corresponding data search engine is invoked to write the data source file into the File Group indexed by the Bucket id and File id in the format of a Log File.

[0059] In one possible embodiment, the data reading module is further configured to:

[0060] When the data write type of the data source file is determined to be Copy-on-Write (COW), the corresponding data search engine is called to read the latest Base File in the File Group indexed by the Bucket ID and File ID.

[0061] In one possible embodiment, the data reading module is further configured to:

[0062] When the data merging cycle is determined, the latest Log File and Base File in the File Group corresponding to the read-to-merge MOR type are merged to obtain a new Base File;

[0063] When the data write type of the data source file is determined to be Read-on-Demand Merge (MOR), the corresponding data search engine is called to read the latest Base File in the File Group indexed by the Bucket ID and File ID.

[0064] Thirdly, this application provides an electronic device, including:

[0065] Memory, used to store program instructions;

[0066] A processor is configured to invoke program instructions stored in the memory and execute the steps of the method described in any one of the first aspects according to the obtained program instructions.

[0067] Fourthly, this application provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the method described in any one of the first aspects.

[0068] Fifthly, this application provides a computer program product comprising: computer program code, which, when run on a computer, causes the computer to perform the method described in any one of the first aspects.

[0069] The technical solutions provided by the embodiments of this application bring at least the following beneficial effects:

[0070] The purpose of this application is to provide a method, apparatus, device, and medium for avoiding data duplication. In scenarios where multiple data search engines write data sequentially, duplicate data generates the same directory index, and duplicate data written by multiple data search engines is written to the same data storage unit based on the same directory index. Hudi's duplicate read mechanism is used to ensure that duplicate data is not read. Attached Figure Description

[0071] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0072] Figure 1 A flowchart illustrating a method for avoiding data duplication provided in an embodiment of this application;

[0073] Figure 2 This is a schematic diagram of the Hudi file layout provided in an embodiment of this application;

[0074] Figure 3 A structural diagram of a device for avoiding data duplication provided in an embodiment of this application;

[0075] Figure 4 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0076] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0077] The terms "first" and "second" in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the term "comprising" and any variations thereof are intended to cover non-exclusive protection. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. The term "multiple" in this application can mean at least two, for example, two, three, or more, and the embodiments of this application do not impose limitations.

[0078] The data collection, dissemination, and use in this application all comply with relevant national laws and regulations.

[0079] Before introducing the method for avoiding data duplication provided in the embodiments of this application, the technical background of the embodiments of this application will be described in detail below for ease of understanding.

[0080] To illustrate this embodiment, some concepts will be explained below.

[0081] (1) Data Lake

[0082] When all the data generated by an enterprise is kept on a single platform, this platform is called a "data lake." A data lake is a data management solution that supports storing multiple raw data formats and multiple computing engines.

[0083] (2)Hudi(Hadoop Upserts and D Incrementals)

[0084] Hudi is a data lake storage format used to manage the storage of large analytical datasets on the Hadoop Distributed File System (HDFS). It provides the ability to update and delete data, as well as consume changed data on HDFS.

[0085] In the Hudi data storage format, data files are stored using multiple file groups, each identified by a unique file ID. Each file group may have different file versions, therefore each file group has multiple file slices. Each file slice consists of a base file stored in column-oriented Parquet format and a log file stored in row-oriented format. The log file contains insertions and updates to the base file since its creation. Hudi periodically merges the base file and the log file to generate a new base file.

[0086] When reading data, Hudi merges data within the same File Group to ensure that the latest Base File data is read.

[0087] (3) Stream processing and batch processing

[0088] Stream processing: The processing of unbounded data streams is often referred to as stream processing, which typically involves real-time processing as the data is generated. Because the input to an unbounded data stream is infinite, processing must be continuous. Data needs to be processed immediately upon acquisition; it's impossible to wait until all data arrives. Processing unbounded data streams often requires acquiring events in a specific order (such as the order in which events occur) to ensure the integrity of the inference results.

[0089] Batch processing: The processing of bounded data streams is often referred to as batch processing. Batch processing does not require the data to be retrieved in an ordered manner. In batch processing mode, the data stream is first persisted to a storage system (file system or object storage), then the data in the entire dataset is read, sorted, statistically analyzed, or summarized, and finally the results are output.

[0090] (4) Data Engine

[0091] Flink: A stream processing framework engine used in distributed, high-performance, and highly available data stream applications. It can handle both finite and infinite data streams, that is, it can handle bounded and unbounded data streams, and is generally used for real-time stream computing.

[0092] Spark is a memory-based parallel computing framework for big data that can be used to build large-scale, low-latency data analysis applications, typically for batch processing.

[0093] This uses Spark / Flink to update, insert, and delete data on HDFS.

[0094] Typically, data updates, insertions, and deletions on HDFS are performed using a single data engine. Hudi, as a data lake storage format, provides the ability to update and delete data, as well as consume changed data, on top of HDFS.

[0095] However, in a special business scenario where multiple engines are used to write data, a real-time streaming data engine, such as Flink, is used to process the streaming data obtained in a specific order (such as the order in which events occur) and write it into the Hudi data lake. Then, a batch processing data engine, such as Spark, is used to batch process some data that was missed by Flink or that needs to be updated and write it into the Hudi data lake.

[0096] For special business scenarios involving multi-engine data writing, taking data D1 as an example, several situations may arise:

[0097] 1) If Flink successfully writes D1, but Spark's batch processing does not include writing D1, then in this scenario, data duplication will not occur when performing read queries.

[0098] 2) Flink successfully writes D1, and Spark performs batch processing including writing D1. In this scenario, if the D1 data processed by Spark is completely consistent with the existing D1 data in the data lake, D1 data will be written again in the data lake to a different file group than the existing D1 data. This causes duplicate data to be unable to be merged, resulting in duplicate data when performing read queries.

[0099] 3) Flink failed to write D1, so there is no data duplication.

[0100] To address the issue of duplicate data being read from the data lake in the second scenario mentioned above, this application provides a method to avoid data duplication. When multiple data search engines write data files to the data lake, each data search engine uses the same directory index for the same data file, writing duplicate data files into the same File Group. This allows each data search engine to read the latest data in the File Group, thereby avoiding the reading of duplicate data.

[0101] like Figure 1 The diagram shown is a flowchart of a method for avoiding data duplication provided in an embodiment of this application. The method includes steps 11-13 as follows.

[0102] Step 11: Use multiple data search engines to search for data and obtain the data source files corresponding to each data search engine;

[0103] In the embodiments of this application, different data search engines are used to implement different data search functions in order to obtain the data source files corresponding to each data search engine.

[0104] As a feasible implementation method, data searches are conducted using multiple data search engines, including the following two scenarios:

[0105] Scenario 1: In response to the real-time data write command, the real-time stream processing data search engine is used to obtain the data source file corresponding to the real-time data from the front-end server in a specific order;

[0106] For example, in a business scenario that records bank transaction records and reconciles them within a specific period, it is necessary to utilize a real-time stream processing data search engine, such as Flink. Flink will write the bank transaction record data obtained from the front-end server in real time into a data lake.

[0107] Scenario 2: In response to the delayed data write command, use the batch data search engine to retrieve the data source file corresponding to the delayed data from the background storage space.

[0108] In the aforementioned business scenarios, network outages or other reasons may cause Flink to miss some real-time data, resulting in discrepancies in bank transaction records. Therefore, it is also necessary to utilize a batch data search engine, such as Spark. Spark retrieves the data source files corresponding to the delayed data from the backend storage space for batch processing. Spark also writes delayed data that was missed by Flink or delayed data that needs to be updated into the Hudi data lake after batch processing.

[0109] Step 12: In response to the directory index determination instruction, the directory index determination unit pre-set for each data search engine is invoked to parse the data source file and generate a directory index corresponding to each data source file. The parsing rules of the directory index determination unit set for each data search engine are the same.

[0110] Hudi uses a directory indexing mechanism to consistently map a given Hudi record to a File Group / File ID, thereby providing efficient data writing and updating. This mapping ensures that all versions of a set of records are necessarily in the same File Group. In this embodiment, after each data search engine finds the corresponding data source file, it calls a directory index determination unit pre-set for each data search engine to parse the data source file and determine the directory index corresponding to each data source file, so that the data source file is written to the data storage unit corresponding to the directory index.

[0111] As a feasible implementation method, the directory index determination unit provided in this application can be a pre-set directory index determination unit independent of each data search engine. In this case, the directory index determination unit can be regarded as an independent pre-module that needs to be executed before each data search engine writes the data source file. The pre-module will generate the same directory index for the same data source file.

[0112] As a feasible implementation method, the directory index determination unit provided in this application can also be a corresponding directory index determination unit pre-set in each data search engine. In this case, the directory index determination unit with the same parsing rules is configured for each data search engine, which can also make the same data source file generate the same directory index.

[0113] In this embodiment, the parsing rules of the directory index determination unit set for each data search engine are the same, so that when multiple data search engines write data source files to Hudi, the directory indexes generated corresponding to the data source files repeatedly written by multiple data search engines are the same.

[0114] As mentioned earlier, Hudi merges data within the same File Group during data reading to ensure that the latest Base File data is read. Therefore, based on the same directory index, multiple data search engines will write duplicate data source files into the same File Group to ensure that duplicate Base File data is not read.

[0115] In this embodiment, a directory index corresponding to each data source file is generated based on the Bucket Index pattern. The Bucket Index is a hash-based index that divides Hudi into N buckets, each bucket containing multiple file groups. The directory index calculated using a hash function determines which bucket and file group a Hudi record belongs to; this Hudi record is the data source file to be written to Hudi.

[0116] In this embodiment, the attribute information of the data source files is parsed according to a preset hash algorithm to generate a directory index corresponding to each data source file. For example, in the aforementioned business scenario of recording bank transaction records, a data source file can be understood as a transaction record, and the attribute information of the data source file can be understood as the transaction type, transaction serial number, and user ID of a transaction record. Figure 2The diagram shown is a schematic of the Hudi file layout provided in an embodiment of this application. For example, two buckets are defined, each bucket partition includes two file groups, and each file group includes file slices (File Slice1, Slice2…SliceN). Each file slice includes a base file and at least one log file. In this embodiment, a directory index corresponding to each data source file is generated based on the Bucket Index pattern. The directory index includes a bucket partition identifier (Bucket id) and a file group identifier (Fileid). The Bucket id identifies the bucket partition where the data storage unit corresponding to the data source file is located, and the Fileid identifies the file group (File Group) corresponding to the data source file within its bucket partition.

[0117] In one or more embodiments, after generating the directory index corresponding to each data source file, the method further includes:

[0118] The directory indexes corresponding to each data source file are stored in memory variables; therefore, each data search engine is called to determine the directory indexes corresponding to the data source files, including: calling each data search engine to access the memory variables and determine the directory indexes corresponding to each data source file.

[0119] For example, in the aforementioned business scenario of using Flink and Spark to record bank transaction records, if Flink and Spark successively write to the same data source file D1, the process of generating the directory index corresponding to the data source file D1 is as follows:

[0120] ① After receiving the data source file D1, Flink executes the directory index determination unit, calculates the Bucket id and File id based on a hash algorithm, generates the directory index corresponding to the data source file and stores it in a memory variable, and then writes it to Hudi. The format of the generated directory index corresponding to the data source file D1 is as follows:

[0121] 00000002-c1b6-4b39-b8b3-5aa9c10fbdf1_20221209180912789.log.1_0-1-0; where "00000002" is the Bucket ID, and the remaining part is the File ID.

[0122] ②After receiving the data source file D1, Spark will also execute the directory index determination unit, calculate the Bucket id and File id based on the hash algorithm, generate the directory index corresponding to the data source file and store it in the memory variable. It should be noted that because the attribute information of the data source file is the same as in ①, the generated Bucket id and File id are the same as in ①.

[0123] Based on the directory index corresponding to the generated data source file D1, Flink and Spark can access the corresponding memory variables and write the data source file D1 into the same file group.

[0124] In this embodiment, based on the Bucket Index mode, by setting a directory index determination unit, the same directory index determination rules are set for each data search engine, and a directory index corresponding to each data source file is generated. Duplicate data source files can be written to the same file group in the same bucket partition, so as to avoid writing duplicate data source files to different file groups, which would result in duplicate data being read when reading Hudi data, thus affecting the user experience.

[0125] Step 13: Call each data search engine to write the data source file into the data storage unit corresponding to the directory index of the determined data source file.

[0126] As one possible implementation, the method further includes:

[0127] In response to a data read command, determine the Bucket and File Group where the data source file to be read is located. Based on the data write type of each data source file, call each data search engine to read the data source file indexed by the Bucket ID and File ID.

[0128] Hudi supports two data write types: Copy On Write (COW) and Merge On Read (MOR). The following explanation of Hudi's data writing and reading is based on these two data write types.

[0129] (1) Copy-on-Write (COW):

[0130] As a feasible implementation method, based on the data writing type of each data source file, each data search engine is invoked to write the data source file into the data storage unit corresponding to the directory index according to the determined directory index of the data source file, including:

[0131] When the data writing type of the data source file is determined to be Copy-on-Write (COW), the corresponding data search engine is invoked to write the data source file as a File Slice into the File Group indexed by Bucketid and Fileid in the format of a Base File.

[0132] ① Data writing: Copy-on-write (COW) refers to copying the old base file and merging it with the newly written data source file to generate a new base file when writing data. The new base file is then written as a new file slice to the file group indexed by the bucket id and the file id.

[0133] Copy-on-Write (COW) creates a new version of the data file, i.e., a new Base File, for each new batch of data written to the data source file. In other words, when the data write type is COW, each file group contains only the Base File; a new Base File is created each time a data file is written.

[0134] It should be noted that the new Base File is obtained by merging the old version Base File and the newly written data source file. If it is the first time writing, there is no old version Base File. The corresponding data search engine is directly called to treat the data source file as a file slice and write it into the file group indexed by Bucketid and Fileid in the format of the base file.

[0135] ② Data reading: When the data writing type of the data source file is Copy-on-Write, the set data search engine is directly called to read the latest Base File in the corresponding File Group.

[0136] In one or more embodiments, based on the data write type of each data source file, each data search engine is invoked to read the data source file indexed by Bucket id and File id, including:

[0137] When the data write type of the data source file is determined to be Copy-on-Write (COW), the corresponding data search engine is called to read the latest Base File in the File Group indexed by the Bucket ID and File ID.

[0138] (2) Merge MOR during read:

[0139] As a feasible implementation method, based on the data writing type of each data source file, each data search engine is invoked to write the data source file into the data storage unit corresponding to the directory index according to the determined directory index of the data source file, including:

[0140] When the data writing type of the data source file is determined to be Read-on-Demand Merge (MOR), the corresponding data search engine is invoked to write the data source file into the File Group indexed by the Bucket id and File id in the format of a Log File.

[0141] ① Data writing: MOR during reading means that the data source file is first written in the format of a log file to the file group indexed by the bucket id and file id.

[0142] When Hudi determines that the data merging cycle has arrived, it merges the latest Log File and Base File in the FileGroup corresponding to the MOR data write type during reading, and obtains a new Base File. The latest Log File is obtained by merging all Log Files in the File Group.

[0143] In other words, when the data write type is Read-on-Merge (MOR), each file slice consists of a base file stored in column Parquet format and a log file stored in row format. The log file contains the insertions and updates to the base file since its creation. Hudi periodically merges the base file and the log file to generate a new base file.

[0144] For an existing base file to be updated, Hudi stores the data source file representing the updated data in the format of a log file. During the write process, no new base files are merged or created, which is less expensive than copy-on-write (COW) and is suitable for scenarios with many writes and few reads.

[0145] ② Data reading: Each time Hudi reads MOR type data, Hudi merges the base file and the latest log file, and outputs the data search engine, which then reads the real-time base file.

[0146] As a feasible implementation method, based on the data write type of each data source file, various data search engines are invoked to read the data source files indexed by Bucket ID and File ID, including:

[0147] When the data write type of the data source file is determined to be Read-on-Demand Merge (MOR), the corresponding data search engine is called to read the latest Base File in the File Group indexed by the Bucket ID and File ID.

[0148] The method provided in this application for avoiding data duplication involves configuring a directory index determination unit with the same parsing rules for each data search engine, so that duplicate data source files written by each data search engine have the same directory index, and duplicate data source files are written to the same file group. This allows each data search engine to read the latest data in the file group, thereby avoiding the reading of duplicate data.

[0149] Based on the same inventive concept, embodiments of this application also provide a device for avoiding data duplication, please refer to... Figure 3 The device includes:

[0150] The data search module 301 is used to perform data searches using multiple data search engines and obtain the data source files corresponding to each data search engine.

[0151] The directory index determination module 302 is used to respond to the directory index determination instruction by calling the directory index determination unit pre-set for each data search engine to parse the data source file and determine the directory index corresponding to each data source file. The parsing rules of the directory index determination unit set for each data search engine are the same.

[0152] The data writing module 303 is used to call various data search engines to write the data source file into the data storage unit corresponding to the directory index, based on the determined directory index of the data source file. In one possible embodiment, the directory index includes a bucket partition identifier (Bucket id) and a file group identifier (File id), and the device further includes:

[0153] The data reading module is used to respond to data reading commands, determine the Bucket and File Group where the data source file to be read is located, and, based on the data writing type of each data source file, call each data search engine to read the data source file indexed by the Bucket ID and File ID.

[0154] In one possible embodiment, the data search module is further configured to:

[0155] In response to real-time data write commands, a real-time stream processing data search engine is used to retrieve the data source files corresponding to the real-time data from the front-end server in a specific order.

[0156] In response to a delayed data write command, a batch data search engine is used to retrieve the data source file corresponding to the delayed data from the background storage space.

[0157] In one possible embodiment, the device pre-configures a directory index determination unit independent of each data search engine, serving as a shared directory index determination unit for all data search engines; or

[0158] Pre-set corresponding directory index determination units in each data search engine.

[0159] In one possible embodiment, the directory index determination module is further configured to call the directory index determination unit pre-set for each data search engine, parse the attribute information of the data source file according to a preset hash algorithm, and generate a directory index corresponding to each data source file.

[0160] In one possible embodiment, the Bucket id is used to identify the bucket partition where the data storage unit corresponding to the data source file is located, and the File id is used to identify the file group corresponding to the data source file in its bucket partition. A bucket partition includes multiple file groups.

[0161] In one possible embodiment, after generating the directory index corresponding to each data source file, the directory index determination module is further configured to:

[0162] Store the directory index corresponding to each data source file in a memory variable;

[0163] The system calls various data search engines to access memory variables and determines the directory index corresponding to each data source file.

[0164] In one possible embodiment, the data writing module is further configured to:

[0165] When the data writing type of the data source file is determined to be Copy-on-Write (COW), the corresponding data search engine is invoked to write the data source file as a File Slice into the File Group indexed by Bucketid and Fileid in the format of a Base File.

[0166] In one possible embodiment, the data writing module is further configured to:

[0167] When the data writing type of the data source file is determined to be Read-on-Demand Merge (MOR), the corresponding data search engine is invoked to write the data source file into the File Group indexed by the Bucket id and File id in the format of a Log File.

[0168] In one possible embodiment, the data reading module is further configured to:

[0169] When the data write type of the data source file is determined to be Copy-on-Write (COW), the corresponding data search engine is called to read the latest Base File in the File Group indexed by the Bucket ID and File ID.

[0170] In one possible embodiment, the data reading module is further configured to:

[0171] When the data merging cycle is determined, the latest Log File and Base File in the File Group corresponding to the read-to-merge MOR type are merged to obtain a new Base File;

[0172] When the data write type of the data source file is determined to be Read-on-Demand Merge (MOR), the corresponding data search engine is called to read the latest Base File in the File Group indexed by the Bucket ID and File ID.

[0173] Based on the same inventive concept, this application provides an electronic device that can achieve the data duplication avoidance function described above. Please refer to... Figure 4 The device includes a processor 401 and a memory 402. The memory 402 stores program instructions. The processor 401 calls the program instructions stored in the memory 402 and executes the aforementioned method for avoiding data duplication according to the obtained program instructions.

[0174] Memory 402 is used to store programs. Specifically, the program may include program code, which includes computer operation instructions. Memory 402 may be volatile memory, such as random-access memory (RAM); it may also be non-volatile memory, such as flash memory, hard disk drive (HDD), or solid-state drive (SSD); or it may be any one or a combination of the above-mentioned volatile and non-volatile memory types.

[0175] Processor 401 can be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and an NP. It can also be a hardware chip. The aforementioned hardware chip can be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0176] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium. The computer program product includes computer program code, which, when executed on a computer, causes the computer to perform any of the data duplication avoidance methods discussed above. Since the principle by which the above-described computer-readable storage medium solves the problem is similar to that of the page switching method, the implementation of the above-described computer-readable storage medium can be found in the implementation of the method; repeated details will not be elaborated further.

[0177] Based on the same inventive concept, this application also provides a computer program product, which includes computer program code. When the computer program code is run on a computer, it causes the computer to execute any of the methods for avoiding data duplication discussed above. Since the principle of the above-described computer program product in solving the problem is similar to that of the page switching method, the implementation of the above-described computer program product can be referred to the implementation of the method, and the repeated parts will not be described again.

[0178] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0179] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0180] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0181] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of user-operated steps to be executed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0182] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method of avoiding data duplication, characterized by, The method comprises: searching data by using multiple data search engines to obtain data source files corresponding to each data search engine; in response to a directory index determination instruction, calling a directory index determination unit set in advance for each data search engine to parse the data source files to generate a directory index corresponding to each data source file, wherein the parsing rules of the directory index determination unit set in advance for each data search engine are the same; calling each data search engine to write the data source files into a data storage unit corresponding to the determined directory index of the data source files according to the determined directory index of the data source files.

2. The method of claim 1, wherein, The directory index comprises a bucket partition identifier Bucket id and a file group identifier File id, and the method further comprises: in response to a data reading instruction, determining the Bucket and File Group where the read data source file is located, and calling each data search engine to read the data source file indexed by the Bucket id and File id based on the data writing type of each data source file.

3. The method of claim 1, wherein, searching data by using multiple data search engines, comprising: in response to a real-time data writing instruction, using a real-time stream processing data search engine to obtain a data source file corresponding to real-time data from a front-end server in real time in a specific order; in response to a delay data writing instruction, using a batch processing data search engine to obtain a data source file corresponding to delay data from a background storage space.

4. The method of claim 1, wherein: an independent directory index determination unit is set in advance for each data search engine as a directory index determination unit shared by each data search engine; or a corresponding directory index determination unit is set in advance in each data search engine.

5. The method according to claim 1 or 2, characterized in that, calling the directory index determination unit set in advance for each data search engine to parse the data source files to determine the directory index corresponding to each data source file comprises: calling the directory index determination unit set in advance for each data search engine to parse attribute information of the data source files according to a preset hash algorithm to generate the directory index corresponding to each data source file.

6. The method of claim 2, wherein, The Bucket id is used to identify a bucket partition where a data storage unit corresponding to the data source file is located, and the File id is used to identify a file group File Group corresponding to the data source file in the bucket partition, and one bucket partition comprises multiple file groups File Group.

7. The method of claim 1, wherein, After generating the directory index corresponding to each data source file, the method further comprises: saving the directory index corresponding to each data source file in a memory variable; calling each data search engine to determine the directory index corresponding to the data source file comprises: calling each data search engine to access the memory variable to determine the directory index corresponding to each data source file.

8. The method of claim 6, wherein, based on the data writing type of each data source file, calling each data search engine to write the data source files into a data storage unit corresponding to the determined directory index of the data source files according to the determined directory index of the data source files, comprises: When it is determined that the data write type of the data source file is copy-on-write COW, a corresponding data search engine is called to write the data source file as a file slice File Slice in the format of a basic file Base File into a File Group indexed by the Bucket id and the File id.

9. The method of claim 6, wherein, Based on the data write type of each data source file, each data search engine is called to write the data source file into a data storage unit corresponding to a determined directory index of the data source file, including: When it is determined that the data write type of the data source file is read-time merge MOR, a corresponding data search engine is called to write the data source file in the format of a log file Log File into a File Group indexed by the Bucket id and the File id.

10. The method according to claim 8 or 9, characterized in that, Based on the data write type of each data source file, each data search engine is called to read a data source file indexed by the Bucket id and the File id, including: When it is determined that the data write type of the data source file is copy-on-write COW, a corresponding data search engine is called to read the latest Base File in a File Group indexed by the Bucket id and the File id.

11. The method according to claim 8 or 9, characterized in that, Further comprising: When a data merge period is reached, the latest Log File and the Base File in a File Group corresponding to the data write type of read-time merge MOR are merged to obtain a new Base File; Based on the data write type of each data source file, each data search engine is called to read a data source file indexed by the Bucket id and the File id, including: When it is determined that the data write type of the data source file is read-time merge MOR, a corresponding data search engine is called to read the latest Base File in a File Group indexed by the Bucket id and the File id.

12. An apparatus for avoiding data duplication, the apparatus comprising: Comprising: A data search module configured to perform data search by using a plurality of data search engines to obtain data source files corresponding to each data search engine; A directory index determination module configured to, in response to a directory index determination instruction, call a directory index determination unit set in advance for each data search engine to parse the data source files to determine a directory index corresponding to each data source file, wherein the parsing rules of the directory index determination units set in advance for each data search engine are the same; A data write module configured to call each data search engine to write the data source files into a data storage unit corresponding to a determined directory index of the data source file.

13. An electronic device, comprising: Comprising: A memory configured to store program instructions; A processor configured to call the program instructions stored in the memory to perform the steps included in the method according to any one of claims 1-11 according to the obtained program instructions.

14. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, the computer program comprising program instructions which, when executed by a computer, cause the computer to perform the method of any one of claims 1-11.

15. A computer program product, characterised in that, The computer program product comprises computer program code which, when run on a computer, causes the computer to perform the method of any one of claims 1-11.

Images