Hudi-based merged partition determination method and apparatus, and product

Abstract

The invention provides a combined partition determination method and device based on hudi and a product, and belongs to the technical field of data processing. The method comprises the steps that a time axis object corresponding to a mor table of hudi is obtained, an increment time interval corresponding to the merging process is determined, and operation information of all operations corresponding to the mor table is recorded in the time axis object according to the time sequence; determining all completed increment operations in the increment time interval in the time axis object; and determining a target merging partition corresponding to the merging process according to the metadata files corresponding to all the completed incremental operations. The method and the device aim at more accurately determining the partition range during each time of merging.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and more specifically, to a method, apparatus, and product for determining merged partitions based on Hudi. Background Technology

[0002] Apache Hudi is an open data lake platform built on the high-performance Open Table Format. It is designed to provide database functionality for data lakes. Hudi reconstructs the traditional inefficient batch data processing mode through a powerful new incremental processing framework, enabling minute-level low-latency analysis capabilities. The MR (merge on read) tables provided in Hudi can not only avoid write amplification, but also improve data update efficiency.

[0003] The mor table is divided into multiple partitions. In each partition, the data is organized according to file groups. Each file group consists of a base file and several log files. Each time data is inserted or updated in the mor table, it is written incrementally, which will generate an independent log file in the corresponding partition. Then, the multiple log files in each partition are periodically compacted with the corresponding base file, i.e., merged.

[0004] Currently, when determining the partitions to be merged in the mor table each time, all partitions are used as the partition range for the merge, or the partition range for the merge is filtered by selecting a date. However, the partition range determined by the current method for each merge is not accurate. For example, there may be partitions within the determined partition range that have not undergone incremental operations. Furthermore, the method of filtering the partition range for the merge by selecting a date may also result in the omission of partitions that have undergone incremental operations. Summary of the Invention

[0005] This application provides a method, apparatus, and product for determining merged partitions based on Hudi, aiming to more accurately determine the partition range for each merge.

[0006] In a first aspect, embodiments of this application provide a method for determining merged partitions based on Hudi, the method comprising: Obtain the timeline object corresponding to the mor table in Hudi, and determine the incremental time interval corresponding to this merge process. The timeline object records the operation information of all operations corresponding to the mor table in chronological order. Identify all completed incremental operations within the incremental time interval in the time axis object; Based on the metadata files corresponding to all the completed incremental operations, determine the target merge partition for this merge process.

[0007] Optionally, the incremental time interval corresponding to this merger process is determined, including: Iterate through the timeline objects corresponding to the mor table and filter all completed merge operations; Sort all the completed merge operations in ascending order of time to construct a set of completed merge operations; Based on the set of completed merge operations, determine the incremental time interval corresponding to this merge process.

[0008] Optionally, based on the set of completed merge operations, the incremental time interval corresponding to the current merge process is determined, including: When the set of completed merge operations is not empty, the operation time corresponding to the last merge operation in the set of completed merge operations is taken as the start time of the increment. Based on the incremental start time and the current time, the incremental time interval corresponding to this merging process is determined.

[0009] Optionally, based on the set of completed merge operations, the incremental time interval corresponding to the current merge process is determined, including: When the set of completed merge operations is empty, any time is randomly determined as the increment start time, and the increment start time is less than the operation time corresponding to the first operation in the time axis object; Based on the incremental start time and the current time, the incremental time interval corresponding to this merging process is determined.

[0010] Optionally, determining all completed incremental operations within the incremental time interval in the timeline object includes: Iterate through the timeline objects corresponding to the mor table and filter all completed incremental operations; Determine whether the operation time of any completed incremental operation falls within the incremental time interval; Construct an incremental operation set based on all completed incremental operations whose operation times fall within the incremental time interval.

[0011] Optionally, based on the metadata files corresponding to all the completed incremental operations, the target merge partition for this merge process is determined, including: Traverse the set of incremental operations; For any incremental operation, obtain the metadata file of the incremental operation, deserialize the metadata file of the incremental operation to obtain the metadata object corresponding to the incremental operation, determine the incremental partition corresponding to the incremental operation in the metadata object corresponding to the incremental operation, and add the incremental partition to the incremental partition set. The incremental partition set is deduplicated, and all incremental partitions in the deduplicated incremental partition set are used as the target merge partitions for this merge process.

[0012] Secondly, embodiments of this application provide a Hudi-based merge partition determination apparatus, the apparatus comprising: The incremental time interval determination module is used to obtain the time axis object corresponding to the mor table of Hudi and determine the incremental time interval corresponding to this merging process. The time axis object records the operation information of all operations corresponding to the mor table in chronological order. The incremental operation determination module is used to determine all completed incremental operations within the incremental time interval in the time axis object. The target merge partition determination module is used to determine the target merge partition corresponding to this merge process based on the metadata files corresponding to all the completed incremental operations.

[0013] Thirdly, embodiments of this application provide an electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the Hudi-based merged partition determination method as described in the first aspect of the embodiments.

[0014] Fourthly, embodiments of this application provide a readable storage medium storing a program or instructions that, when executed by a processor, implement the Hudi-based merge partition determination method as described in the first aspect of the embodiments.

[0015] Fifthly, embodiments of this application provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the Hudi-based merged partition determination method described in the first aspect of the embodiments.

[0016] Beneficial effects: This method obtains the timeline object corresponding to the mor table in Hudi, determines the incremental time interval corresponding to this merge process, then identifies all completed incremental operations within the incremental time interval in the timeline object, and determines the target merge partition corresponding to this merge process based on the metadata files corresponding to each of the completed incremental operations. In other words, by determining the partitions corresponding to each of the completed incremental operations within the incremental time interval, the range of partitions that need to be merged can be determined more accurately. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of the steps of a Hudi-based method for determining merged partitions according to an embodiment of this application; Figure 2 This is a flowchart illustrating the determination of the increment start time according to an embodiment of this application; Figure 3 This is a flowchart illustrating the construction of an incremental operation set according to an embodiment of this application; Figure 4 This is a flowchart illustrating the determination of a target merging partition according to an embodiment of this application; Figure 5 This is a functional block diagram of a Hudi-based merging partition determination device proposed in an embodiment of this application; Figure 6 This is a schematic diagram of an electronic device according to an embodiment of this application; Figure 7 This is a schematic diagram of a readable storage medium proposed in an embodiment of this application; Figure 8 This is a schematic diagram of a computer program product proposed in an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0021] mor table: merge on read, the data file in the mor table consists of column-oriented base file (BaseFile) and row-oriented log file (Log File). All writes are incremental writes. Specifically, the log file is generated first, and then periodically merged into the base file. DeltaCommit: Represents an incremental operation on the mor table. The incrementally written data will be saved as a log file in the corresponding partition. Incremental operations include update, insert, or delete operations. Compaction: A merge operation that combines incrementally written log files in the mor table into the base file. Timeline: Records all operations performed on the hudi table at different points in time in chronological order, such as Compaction and DeltaCommit. Each operation corresponds to a metadata file that records the specific operation information. Timeline Server: A web service component that provides Hudi file system view functionality, such as partition filegroup information and timeline information.

[0022] Apache Hudi is an open data lake platform built on the high-performance Open Table Format. It is designed to provide database functionality for data lakes. Hudi reconstructs the traditional inefficient batch data processing model through a powerful new incremental processing framework, enabling minute-level low-latency analysis capabilities.

[0023] Hudi provides two table types: COW tables (copy on write) and MR tables. MR tables not only avoid write amplification but also improve data update efficiency. MR tables are divided into multiple partitions, and data in each partition is organized according to filegroups. Each filegroup has a corresponding file ID as a unique identifier; each filegroup consists of a base file and several log files.

[0024] For example, the mor table is divided into partition 1, partition 2, etc. Taking partition 1 as an example, partition 1 includes multiple file groups, such as file group 1, file group 2, etc. Each file group has a corresponding file ID as a unique identifier. File group 1 includes basic files and several log files.

[0025] During the writing process, each insertion or update to the mor table is an incremental write, which generates an independent log file in the corresponding partition. Then, the multiple log files in each partition are periodically merged with the corresponding base file. This process is called compaction.

[0026] For partition table compaction, Hudi can use all partitions as the partition range for each compaction, or use pluggable compaction strategies to filter partitions to reduce the partition range for compaction. However, currently it only supports partitioning by date and determining a continuous partition range based on the date calculation of the partition value. But this method of determining the partition range has at least the following shortcomings: 1. Currently, partitioning is only possible by date. Filtering the partition range by date is inaccurate because each data write to the MOR partitioned table may involve multiple partitions within the full partition range. If the contiguous partition range for compaction is limited too narrowly, some partitions that need to be merged will be missed. For example, during data writing, incremental operations may be performed on some partitions older than one month and some partitions within the current month. If the compaction range is set to select partitions within the current month, some partitions older than one month that have incremental operations will be missed. On the other hand, if the date limit is too large, it will include too many non-target partitions. For example, within an excessively large partition range, there may be partitions that have not undergone incremental operations. 2. Since each partition within the compaction partition range needs to obtain group information from the Timeline Server to determine whether the partition is the target partition, if the determined compaction partition range is too large, such as directly using all partitions as the compaction partition range, it will cause a large number of HTTP requests to be sent to the Timeline Server in a short period of time, resulting in great pressure, service instability, and even memory overflow problems.

[0027] Therefore, in order to overcome the inaccurate partitioning and performance issues caused by the mor table during compaction, this application provides a Hudi-based method for determining merged partitions. This method does not require date as the partitioning standard; it can use any field for partitioning. By utilizing the metadata information in the Hudi timeline object, all target merged partitions that need to be compacted can be accurately obtained. The target merged partitions are all partitions that have undergone incremental operations, eliminating a large number of non-target partitions that have not undergone incremental operations. This also reduces the request pressure on the Timeline Server and improves service stability.

[0028] The following description, in conjunction with the accompanying drawings, details a Hudi-based method for determining merged partitions provided in this application, through specific embodiments and application scenarios.

[0029] Reference Figure 1 This document illustrates a flowchart of a Hudi-based method for determining merged partitions, as provided in an embodiment of this application. The method specifically includes the following steps: S101: Obtain the timeline object corresponding to the mor table in Hudi, and determine the incremental time interval corresponding to this merge process. The timeline object records the operation information of all operations corresponding to the mor table in chronological order.

[0030] In Hudi, each table has a corresponding timeline object, which records all operations performed on the Hudi table at different points in time in chronological order. These operations include Compaction and DeltaCommit, and each operation corresponds to a metadata file that records the specific operation information.

[0031] Therefore, when performing a merging operation (Compaction) on the mor table, the timeline corresponding to the mor table in Hudi is first obtained. The timeline corresponding to the mor table records the operation information of all operations corresponding to the mor table in chronological order. The operation information of any operation includes the operation time, operation type, status, and detailed metadata file.

[0032] In one feasible implementation, the process of determining the incremental time interval corresponding to this merging process may include the following steps: A1: Iterate through the timeline objects corresponding to the mor table and filter all completed merge operations.

[0033] Specifically, after obtaining the TimeLine of the mor table, iterate through the TimeLine of the mor table, determine whether the operation type of any operation is a compaction operation, and whether the status of any operation is completed, and filter out all completed compaction operations.

[0034] A2: Sort all the completed merge operations in ascending order of time to construct a set of completed merge operations.

[0035] Based on all the completed merge operations that have been filtered out, sort all the completed merge operations in ascending order of their operation time to construct a set of completed merge operations.

[0036] A3: Based on the set of completed merge operations, determine the incremental time interval corresponding to this merge process.

[0037] Specifically, when the set of completed merge operations is not empty, it indicates that the mor table has undergone at least one compaction operation. Log files and base files that have already been merged do not need to be merged again. Therefore, the operation time corresponding to the last merge operation in the set of completed merge operations is taken as the increment start time. The increment start time indicates that the log files written incrementally after this time have not yet been merged. Therefore, the partitions that are incrementally written after this time should be the partition objects that undergo another compaction operation.

[0038] When the set of completed merge operations is empty, it indicates that the mor table has never undergone a compaction operation. Therefore, all partitions that have been incrementally written to, i.e., have undergone incremental operations, should be partition objects that have undergone compaction operations again. For example, any time can be randomly determined as the start time of the increment, and the randomly determined start time of the increment should be less than the operation time corresponding to the first operation in the timeline object.

[0039] Reference Figure 2 The flowchart illustrating the determination of the increment start time provided in an embodiment of this application is shown. In one feasible implementation, the process of determining the increment start time may include the following steps: First, we can obtain the timeline object corresponding to the mor table, i.e., TimeLine. We then iterate through all the operations in the Timeline object. For any operation, we determine whether the operation type is a compaction operation and whether the status is completed. If the operation is a compaction operation and has been completed, we add the operation to the set of completed compaction operations. If the operation is not a compaction operation or is not in a completed status, we continue to iterate through the next operation in the TimeLine.

[0040] Once the Timeline object (TimeLine) has been traversed, it is determined whether the set of completed merge operations is empty. If the set of completed merge operations is not empty, the operation time corresponding to the last merge operation in the set of completed merge operations is taken as the increment start time. If the set of completed merge operations is empty, any time can be randomly determined as the increment start time, and the randomly determined increment start time should be less than the operation time corresponding to the first operation in the Timeline object, thus obtaining the increment start time.

[0041] Finally, based on the incremental start time and the current time, the incremental time interval corresponding to this merging process is determined, that is, the interval from the incremental start time to the current time is taken as the incremental time interval.

[0042] In actual implementation, the triggering conditions for each merging process can be set according to the actual application requirements. For example, a fixed time interval can be set, or the number of incremental operations can be counted. When the number of incremental operations reaches the preset number of operations, the merging process can be triggered. This application embodiment does not impose any restrictions.

[0043] S102: Determine all completed incremental operations within the incremental time interval in the time axis object.

[0044] After determining the incremental time interval, it is necessary to further determine the incremental operations that have been completed within the incremental time interval.

[0045] In one feasible implementation, determining all completed incremental operations within the incremental time interval in the timeline object may include the following steps: B1: Iterate through the timeline objects corresponding to the mor table and filter all completed incremental operations.

[0046] Specifically, iterate through the TimeLine of the mor table, determine whether the operation type of any operation is an incremental operation (i.e., DeltaCommit operation), and whether the status of any operation is completed, and filter out all completed incremental operations.

[0047] B2: Determine whether the operation time of any completed incremental operation falls within the incremental time interval.

[0048] Specifically, for all the completed incremental operations that have been selected, it is necessary to further determine whether the incremental operation occurred within the incremental time interval. For example, by determining whether the operation time of any completed incremental operation belongs to the incremental time interval, all the incremental operations completed within the incremental time interval can be determined.

[0049] B3: Construct an incremental operation set based on all completed incremental operations whose operation times fall within the incremental time interval.

[0050] Finally, add all incremental operations completed within the incremental time interval to the incremental operation set.

[0051] Reference Figure 3 The diagram illustrates a flowchart of constructing an incremental operation set according to an embodiment of this application. In one feasible implementation, constructing an incremental operation set may include the following process: First, iterate through all operations in the TimeLine corresponding to the mor table. For any operation, determine whether the operation type is an incremental operation (i.e., DeltaCommit operation) and whether the status is completed. If the operation is not a DeltaCommit operation or the status is not completed, continue iterating through the next operation in the TimeLine.

[0052] If the operation is a DeltaCommit operation and its status is completed, then it is further determined whether the operation time is within the incremental time interval. If the operation time is within the incremental time interval, the operation is added to the incremental operation set; if the operation time is not within the incremental time interval, the next operation in the TimeLine is traversed.

[0053] The final set of incremental operations is obtained only after the Timeline object has been traversed.

[0054] S103: Based on the metadata files corresponding to all the completed incremental operations, determine the target merge partition for this merge process.

[0055] After determining the incremental operations that have been completed within the incremental time interval, it is necessary to further determine the partitions corresponding to each completed incremental operation, thereby determining the target merge partitions for this merge process.

[0056] In one feasible implementation, the process of determining the target merge partition for this merge process based on the metadata files corresponding to all the completed incremental operations may include the following steps: C1: Traverse the set of incremental operations. For any incremental operation, obtain the metadata file of the incremental operation, deserialize the metadata file of the incremental operation to obtain the metadata object corresponding to the incremental operation, determine the incremental partition corresponding to the incremental operation in the metadata object corresponding to the incremental operation, and add the incremental partition to the incremental partition set.

[0057] Specifically, for any incremental operation in the set of incremental operations, the metadata file corresponding to the incremental operation is obtained. In the mor table, each operation has its own corresponding metadata file, which records the detailed information of the operation, including the partition corresponding to each operation.

[0058] The metadata file for any operation is in string form. Therefore, the contents of the metadata file can be deserialized using the HoodieCommitMetadata#fromBytes method to obtain the metadata object corresponding to any operation, that is, to recover the metadata object from the byte data.

[0059] Once the metadata object for any incremental operation is determined, the incremental partition corresponding to that incremental operation can be determined from the metadata object.

[0060] In practice, the getPartitionToWriteStats method can be used to obtain all the partitions actually involved in the incremental operation from the metadata object, which are then used as the incremental partitions corresponding to the incremental operation. Each incremental operation's corresponding incremental partition is then added to the incremental partition collection.

[0061] C3: Deduplicate the incremental partition set, and use all incremental partitions in the deduplicated incremental partition set as the target merge partitions for this merge process.

[0062] Specifically, the incremental partitions corresponding to any two incremental operations may overlap. Therefore, it is necessary to overlap the incremental partitions in the incremental partition set so that each incremental partition in the deduplicated incremental partition set has only one instance.

[0063] The set of incremental partitions after deduplication contains all the incremental partitions that are the target partitions for this merge process. Further merge operations can be performed on all the incremental partitions in the set of deduplicated incremental partitions.

[0064] Reference Figure 4 The flowchart illustrating the determination of the target merge partition provided in an embodiment of this application is shown. In one feasible implementation, determining the target merge partition may include the following process: First, iterate through the set of incremental operations. For any incremental operation, obtain the metadata file corresponding to the incremental operation. Then, deserialize the metadata file to determine the metadata object corresponding to the incremental operation. Obtain the incremental partition corresponding to the incremental operation from the metadata object corresponding to the incremental operation and add the incremental partition to the incremental partition set. Then, continue to iterate through the set of incremental operations for the next incremental operation.

[0065] After the incremental operation set has been traversed, the incremental partition set is deduplicated, and all incremental partitions in the deduplicated incremental partition set are used as the target merge partitions for this merge process.

[0066] The method provided in this application, based on the sequential characteristics of the timeline of the mor table in Hudi, divides the compaction target partitions in the entire time domain of the mor table according to the time increment, and makes full use of Hudi's metadata to accurately obtain the target merge partitions for each compaction operation. The target merge partitions do not contain invalid partitions, i.e., partitions without incremental operations, nor do they miss partitions with incremental operations.

[0067] Since the target merged partitions for each compaction operation determined by the method provided in this application embodiment are more accurate and are all partitions with incremental operations, the number of invalid partitions within the compaction operation range is significantly reduced. This can reduce a large number of requests to the Timeline Server due to invalid partitions, reduce resource usage, and improve service stability.

[0068] Furthermore, when applying the method provided in the embodiments of this application, the partitions of the mor table are not required to be divided by date, which allows the mor table to support more use cases and improve the application scope of the mor table.

[0069] Reference Figure 5 The diagram illustrates a functional block diagram of a Hudi-based partition merging determination device according to an embodiment of this application. The device includes: The incremental time interval determination module 100 is used to obtain the time axis object corresponding to the mor table of Hudi and determine the incremental time interval corresponding to this merging process. The time axis object records the operation information of all operations corresponding to the mor table in chronological order. Incremental operation determination module 200 is used to determine all completed incremental operations within the incremental time interval in the time axis object; The target merge partition determination module 300 is used to determine the target merge partition corresponding to this merge process based on the metadata files corresponding to all the completed incremental operations.

[0070] Optionally, the incremental time interval determination module includes: The first traversal unit is used to traverse the timeline object corresponding to the mor table and filter all completed merge operations; The completed merge operation set construction unit is used to sort all the completed merge operations in ascending order of time to construct the completed merge operation set. The incremental time interval determination unit is used to determine the incremental time interval corresponding to the current merging process based on the set of completed merging operations.

[0071] Optionally, the incremental time interval determination unit is further configured to: When the set of completed merge operations is not empty, the operation time corresponding to the last merge operation in the set of completed merge operations is taken as the start time of the increment. Based on the incremental start time and the current time, the incremental time interval corresponding to this merging process is determined.

[0072] Optionally, the incremental time interval determination unit is further configured to: When the set of completed merge operations is empty, any time is randomly determined as the increment start time, and the increment start time is less than the operation time corresponding to the first operation in the time axis object; Based on the incremental start time and the current time, the incremental time interval corresponding to this merging process is determined.

[0073] Optionally, the incremental operation determination module includes: The second traversal unit is used to traverse the timeline object corresponding to the mor table and filter all completed incremental operations. The judgment unit is used to determine whether the operation time of any completed incremental operation belongs to the incremental time interval; The incremental operation set determination unit is used to construct an incremental operation set based on all completed incremental operations whose operation times belong to the incremental time interval.

[0074] Optionally, the target merging partition determination module includes: The third traversal unit is used to traverse the set of incremental operations. For any incremental operation, the metadata file of the incremental operation is obtained, the metadata file of the incremental operation is deserialized to obtain the metadata object corresponding to the incremental operation, the incremental partition corresponding to the incremental operation is determined in the metadata object corresponding to the incremental operation, and the incremental partition is added to the incremental partition set. The target merge partition determination unit is used to deduplicate the incremental partition set and use all incremental partitions in the deduplicated incremental partition set as the target merge partitions for this merge process.

[0075] The Hudi-based partition merging and determination device in this application embodiment can be a device, or it can be a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device; this application embodiment does not specifically limit its functionality.

[0076] The Hudi-based partition merging determination device in this application embodiment can be a device with an operating system, and this application embodiment does not specifically limit it.

[0077] Reference Figure 6 The diagram illustrates an electronic device provided in an embodiment of this application. The electronic device includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the various processes of the above-described Hudi-based merged partition determination method embodiment and achieve the same technical effect.

[0078] For example, when the program or instructions stored in the memory are executed by the processor, the specific steps include: obtaining the timeline object corresponding to the mor table of Hudi, and determining the incremental time interval corresponding to the current merge process, wherein the timeline object records the operation information of all operations corresponding to the mor table in chronological order; determining all completed incremental operations within the incremental time interval in the timeline object; and determining the target merge partition corresponding to the current merge process based on the metadata files corresponding to each of the completed incremental operations.

[0079] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0080] Reference Figure 7 The diagram illustrates a readable storage medium provided in an embodiment of this application. The readable storage medium stores a program or instructions, which, when executed by a processor, implement the various processes of the above-described Hudi-based merged partition determination method embodiment and achieve the same technical effect.

[0081] For example, when a program or instruction stored on a readable storage medium is executed by a processor, the specific steps include: obtaining the timeline object corresponding to the mor table of Hudi, and determining the incremental time interval corresponding to this merge process, wherein the timeline object records the operation information of all operations corresponding to the mor table in chronological order; determining all completed incremental operations within the incremental time interval in the timeline object; and determining the target merge partition corresponding to this merge process based on the metadata files corresponding to each of the completed incremental operations.

[0082] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0083] Reference Figure 8 The diagram illustrates a computer program product provided in an embodiment of this application, including a computer program / instruction. When executed by a processor, the computer program / instruction implements the various processes of the above-described Hudi-based merged partition determination method embodiment and achieves the same technical effect.

[0084] For example, when a computer program / instruction is executed by a processor, it specifically retrieves the timeline object corresponding to the mor table in Hudi and determines the incremental time interval corresponding to this merge process. The timeline object records the operation information of all operations corresponding to the mor table in chronological order. It then determines all completed incremental operations within the incremental time interval from the timeline object and, based on the metadata files corresponding to each of the completed incremental operations, determines the target merge partition corresponding to this merge process.

[0085] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0086] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0087] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. The description of the embodiments above is only for the purpose of helping to understand the method and core idea of ​​this application. Those skilled in the art can make many forms under the guidance of this application without departing from the spirit and scope of protection of the claims, and all of these are within the protection scope of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation and application scope based on the idea of ​​this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for determining merged partitions based on Hudi, characterized in that, The method includes: Obtain the timeline object corresponding to the mor table in Hudi, and determine the incremental time interval corresponding to this merge process. The timeline object records the operation information of all operations corresponding to the mor table in chronological order. Identify all completed incremental operations within the incremental time interval in the time axis object; Based on the metadata files corresponding to all the completed incremental operations, determine the target merge partition for this merge process.

2. The method according to claim 1, characterized in that, Determine the incremental time interval corresponding to this merger process, including: Iterate through the timeline objects corresponding to the mor table and filter all completed merge operations; Sort all the completed merge operations in ascending order of time to construct a set of completed merge operations; Based on the set of completed merge operations, determine the incremental time interval corresponding to this merge process.

3. The method according to claim 2, characterized in that, Based on the set of completed merge operations, determine the incremental time interval corresponding to this merge process, including: When the set of completed merge operations is not empty, the operation time corresponding to the last merge operation in the set of completed merge operations is taken as the start time of the increment. Based on the incremental start time and the current time, the incremental time interval corresponding to this merging process is determined.

4. The method according to claim 2, characterized in that, Based on the set of completed merge operations, determine the incremental time interval corresponding to this merge process, including: When the set of completed merge operations is empty, any time is randomly determined as the increment start time, and the increment start time is less than the operation time corresponding to the first operation in the time axis object; Based on the incremental start time and the current time, the incremental time interval corresponding to this merging process is determined.

5. The method according to claim 1, characterized in that, Determine all completed incremental operations within the incremental time interval in the timeline object, including: Iterate through the timeline objects corresponding to the mor table and filter all completed incremental operations; Determine whether the operation time of any completed incremental operation falls within the incremental time interval; Construct an incremental operation set based on all completed incremental operations whose operation times fall within the incremental time interval.

6. The method according to claim 5, characterized in that, Based on the metadata files corresponding to all completed incremental operations, determine the target merge partition for this merge process, including: Iterate through the set of incremental operations; For any incremental operation, obtain the metadata file of the incremental operation, deserialize the metadata file of the incremental operation to obtain the metadata object corresponding to the incremental operation, determine the incremental partition corresponding to the incremental operation in the metadata object corresponding to the incremental operation, and add the incremental partition to the incremental partition set. The incremental partition set is deduplicated, and all incremental partitions in the deduplicated incremental partition set are used as the target merge partitions for this merge process.

7. A Hudi-based merge partition determination device, characterized in that, The device includes: The incremental time interval determination module is used to obtain the time axis object corresponding to the mor table of Hudi and determine the incremental time interval corresponding to this merging process. The time axis object records the operation information of all operations corresponding to the mor table in chronological order. The incremental operation determination module is used to determine all completed incremental operations within the incremental time interval in the time axis object. The target merge partition determination module is used to determine the target merge partition corresponding to this merge process based on the metadata files corresponding to all the completed incremental operations.

8. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the Hudi-based merge partition determination method as described in any one of claims 1-6.

9. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the Hudi-based merge partition determination method as described in any one of claims 1-6.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the Hudi-based merge partition determination method as described in any one of claims 1-6.

Images