A data processing method, device, storage medium and equipment in Iceberg

Abstract

The present application discloses a method, device, storage medium and equipment for data processing in Iceberg. The method comprises: first, when creating an Iceberg data table, adding index information to the metadata file of the data table, and then establishing a corresponding relationship between the data file and the index in Iceberg; then, when executing a processing operation of the target data in Iceberg, according to the corresponding relationship between the data file and the index, performing index matching at the row group level on the target data, and determining the target data file; and then performing corresponding data processing operations on the target row group data in the target data file, and obtaining a data processing result. Since the present application first adds index information to the metadata when creating an Iceberg table, after establishing a corresponding relationship between the data file and the index, when performing operations such as data update and query, the location of the target data file can be quickly located through the index matching at the row group level and the corresponding processing operation can be performed, thereby improving the processing efficiency of data in Iceberg.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, device, storage medium and equipment in Iceberg. Background Art

[0002] Iceberg is one of the more popular data lake technology frameworks. It provides an independent data organization method that can realize data addition, deletion, modification, query and other processing operations in big data scenarios.

[0003] Currently, Iceberg implements various data processing functions for tables through metadata files (manifest). The Iceberg table structure diagram is as follows: Figure 1 As shown. Currently, when performing data modification, deletion, query and other processing operations in Iceberg, the processing flow is usually: metadata file -> metadata file list -> manifest file -> data file. Only through these four layers of processing can the list information of the data file be obtained. Then, all data files are traversed and read to see if there is data that matches the data to be queried, resulting in a slow overall query rate and poor data processing performance. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to provide a data processing method, device, storage medium and equipment in Iceberg, which can quickly implement data processing (such as adding, deleting, modifying, querying, etc. of data) by adding an index in Iceberg, thereby improving the data processing efficiency in Iceberg.

[0005] The present application embodiment provides a method for processing data in Iceberg, including:

[0006] When creating an Iceberg data table, adding index information to a metadata file of the data table;

[0007] Establishing a corresponding relationship between the data file and the index in the Iceberg;

[0008] When executing a processing operation of the target data in the Iceberg, index matching is performed on the target data at a row group level according to a correspondence between the data file and the index to determine a target data file;

[0009] A corresponding data processing operation is performed on the target row group data in the target data file to obtain a data processing result.

[0010] In a possible implementation, establishing a correspondence between the data file and the index in Iceberg includes:

[0011] In the Iceberg, the data file is segmented into row groups to generate a corresponding task list;

[0012] A correspondence between the column name of the row group in the data file and the index is constructed.

[0013] In a possible implementation, after building a corresponding relationship between the column name of the row group in the data file and the index, the method further includes:

[0014] The name of the data file where the row group corresponding to the index is located, the interval range of the row group, and the length of the row group are stored in the corresponding data storage object in the statistical file;

[0015] When the target data processing operation is executed in the Iceberg, index matching is performed on the target data at the row group level according to the corresponding relationship between the data file and the index to determine the target data file, including:

[0016] When executing a processing operation of the target data in the Iceberg, determining a corresponding target data storage object in the statistical file according to a corresponding relationship between the data file and the index;

[0017] According to the index information included in the target data storage object, a target data file and an interval range and a length of target row group data in the target data file are determined.

[0018] In a possible implementation, performing corresponding data processing operations on the target row group data in the target data file to obtain a data processing result includes:

[0019] The task is segmented according to the interval range and length of the target row group data in the target data file, and the data processing result is obtained after the segmented task is executed.

[0020] In a possible implementation, the method further includes:

[0021] Determine a first data file newly added in the Iceberg, and generate index information and metadata information corresponding to the first data file;

[0022] Comparing snapshot information before and after the first data file is added in Iceberg, and deleting the second data file according to the comparison result;

[0023] The index information corresponding to the newly added first data file is merged with the index information before the newly added first data file in Iceberg, and the index information corresponding to the second data file is deleted to obtain updated index information.

[0024] In a possible implementation, the method further includes:

[0025] Incrementally construct index information in the Iceberg according to a preset period, and obtain an updated correspondence between the data file in the Iceberg and the index.

[0026] In a possible implementation, the method further includes:

[0027] After the configuration information of the index is modified, the index information is fully constructed in the Iceberg to obtain an updated correspondence between the data file in the Iceberg and the index.

[0028] The present application also provides a data processing device in Iceberg, the device comprising:

[0029] An adding unit, used for adding index information in a metadata file of an Iceberg data table when creating the data table;

[0030] An establishing unit, used for establishing a corresponding relationship between a data file and the index in the Iceberg;

[0031] A matching unit, configured to perform index matching at a row group level on the target data according to a correspondence between the data file and the index to determine a target data file when executing a processing operation of the target data in the Iceberg;

[0032] The processing unit is used to perform corresponding data processing operations on the target row group data in the target data file to obtain a data processing result.

[0033] In a possible implementation, the establishing unit includes:

[0034] A split subunit, used to split the data file into row groups in Iceberg and generate a corresponding task list;

[0035] The construction subunit is used to construct a corresponding relationship with the index according to the column name of the row group in the data file.

[0036] In a possible implementation manner, the establishing unit further includes:

[0037] A storage subunit is used to store the name of the data file where the row group corresponding to the index is located, the interval range of the row group, and the length of the row group into a corresponding data storage object in the statistical file;

[0038] The matching unit comprises:

[0039] a first determining subunit, configured to determine a corresponding target data storage object in a statistical file according to a correspondence between the data file and the index when executing a target data processing operation in the Iceberg;

[0040] The second determining subunit is used to determine the target data file and the interval range and length of the target row group data in the target data file according to the index information included in the target data storage object.

[0041] In a possible implementation, the processing unit is specifically configured to:

[0042] The task is segmented according to the interval range and length of the target row group data in the target data file, and the data processing result is obtained after the segmented task is executed.

[0043] In a possible implementation, the device further includes:

[0044] A generating unit, configured to determine a first data file newly added in the Iceberg, and generate index information and metadata information corresponding to the first data file;

[0045] A comparison unit, used to compare the snapshot information before and after the first data file is added in Iceberg, and delete the second data file according to the comparison result;

[0046] The merging unit is used to merge the index information corresponding to the newly added first data file with the index information before the newly added first data file in Iceberg, and delete the index information corresponding to the second data file to obtain updated index information.

[0047] In a possible implementation, the device further includes:

[0048] The first construction unit is used to incrementally construct index information in the Iceberg according to a preset period to obtain an updated corresponding relationship between the data file in the Iceberg and the index.

[0049] In a possible implementation, the device further includes:

[0050] The second construction unit is used to fully construct the index information in the Iceberg after modifying the configuration information of the index, and obtain an updated corresponding relationship between the data file in the Iceberg and the index.

[0051] The embodiment of the present application also provides a data processing device in Iceberg, including: a processor, a memory, and a system bus;

[0052] The processor and the memory are connected via the system bus;

[0053] The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the processor executes any one of the implementations of the data processing method in Iceberg.

[0054] An embodiment of the present application further provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes any one of the implementations of the data processing method in Iceberg.

[0055] The embodiment of the present application provides a method, device, storage medium and equipment for processing data in Iceberg. First, when creating an Iceberg data table, index information is added to the metadata file of the data table. Then, a corresponding relationship between the data file and the index is established in Iceberg. Then, when executing the processing operation of the target data in Iceberg, the target data is matched at the row group level according to the corresponding relationship between the data file and the index to determine the target data file. Then, the target row group data in the target data file can be subjected to corresponding data processing operations to obtain data processing results. It can be seen that since the present application first adds index information to the metadata when creating the Iceberg table, after the corresponding relationship between the data file and the index is established, when performing operations such as data update, deletion, and query, the location of the target data file can be quickly located and the corresponding processing operation can be performed by performing index matching at the row group level, thereby improving the processing efficiency of data in Iceberg. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0057] Figure 1A schematic diagram of the Iceberg table structure provided in the embodiments of the present application;

[0058] Figure 2 An example diagram of implementing data query in the existing Iceberg table structure provided in an embodiment of the present application;

[0059] Figure 3 A schematic diagram of a data processing method in Iceberg provided in an embodiment of the present application;

[0060] Figure 4 An example diagram of adding index information to a metadata file of a data table when creating an Iceberg data table provided in an embodiment of the present application;

[0061] Figure 5 An example diagram of the implementation of the BitMap index provided in the embodiment of the present application;

[0062] Figure 6 An example diagram of the correspondence between data files and indexes provided in an embodiment of the present application;

[0063] Figure 7 An example diagram of a COW table provided in an embodiment of the present application;

[0064] Figure 8 A flowchart for implementing data query in Iceberg provided in an embodiment of the present application;

[0065] Fig. 9 An example diagram of implementing data query in Iceberg provided in an embodiment of the present application;

[0066] Fig.10 A schematic diagram of the composition of a data processing device in Iceberg provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] Iceberg is a table format that has similar functions to the Structured Query Language (SQL) table in traditional databases, but it is completely open and accessible, and implements many functions for the table through metadata files (manifest), such as querying historical data through time travel, verifying changes between data versions, overall planning and filtering functions, etc. Figure 1As shown, these metadata files are tracked through point-in-time snapshots, maintaining all deltas as the table is updated over time. Each snapshot provides a complete description of the table's schema, partitions, and file information, with full isolation and consistency. In addition, Iceberg intelligently organizes snapshot metadata in a hierarchical structure. This allows tables to be changed quickly and efficiently without redefining all dataset files, ensuring optimal performance when working at data lake scale.

[0068] When using Iceberg, the complete history is maintained in the Iceberg metadata file, with no storage system dependencies. This supports an open architecture and the flexibility to change the system over time without disrupting users or existing workloads. Since the historical state is immutable and the historical provenance is clear, users can query previous states at any Iceberg snapshot or any historical point in time to obtain consistent results, compare, or roll back data to correct problems.

[0069] Currently common index types include, but are not limited to, minimum and maximum value (MinMax) index, Bloom Filter, Ribbon Filter, Bitmap, Range-Encoded (for example, encoded in binary Base-2), and BTreeIndex.

[0070] However, current research on the index level is limited to the partition level or file level and requires the number of index files to be as small as possible. When the number of index files is large, various problems may arise.

[0071] Furthermore, since the Iceberg table structure is based on Figure 1 The structure shown in the figure means that when performing data modification, deletion, query and other processing operations in Iceberg, it needs to go through four layers of processing, specifically: metadata file (metadatafile) -> metadata file list (mamifest list) -> metafile (manifest file) -> data file (datafile). Only after these four layers of processing can the list information of the data file be obtained. Then, all data files are traversed and read to see if there is data that matches the data to be queried, resulting in a slow overall query rate and poor data processing performance.

[0072] For example, for the data processing statement "update table set column_1=1where column_2='A'" in Iceberg, it is necessary to traverse all the data file lists in Iceberg and check whether the content contains data with column_2='A' after reading all the data files. Figure 2 As shown, the first and third data files marked with "√" may be found to contain data with column_2 = 'A', and other data files marked with "X" do not contain data with column_2 = 'A', and the overall query rate is slow.

[0073] In order to solve the above defects, the present application provides a data processing method in Iceberg. First, when creating an Iceberg data table, index information is added to the metadata file of the data table. Then, a corresponding relationship between the data file and the index is established in Iceberg. Then, when executing the processing operation of the target data in Iceberg, the target data is matched at the row group level according to the corresponding relationship between the data file and the index to determine the target data file. Then, the target row group data in the target data file can be subjected to corresponding data processing operations to obtain data processing results. It can be seen that since the present application first adds index information to the metadata when creating the Iceberg table, after the corresponding relationship between the data file and the index is established, when performing operations such as data update, deletion, and query, the location of the target data file can be quickly located and the corresponding processing operation can be performed by performing index matching at the row group level, thereby improving the processing efficiency of data in Iceberg.

[0074] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0075] First embodiment

[0076] See also Figure 3 , is a flow chart of a data processing method in Iceberg provided in this embodiment, the method comprising the following steps:

[0077] S301: When creating an Iceberg data table, add index information to a metadata file of the data table.

[0078] It should be noted that Iceberg is a data table format. When creating an iceberg data table, you can specify index information. For example, the code to specify index information can be "create index idx_year using{index method}oncustomer(year)with{properties}". In addition, Iceberg can store this index information in the metadata file of the Iceberg data table, such as Figure 4 The "statistical information" shown is used to perform the subsequent step S302. The metadata information of the index can be stored in the data storage object (blob-metadata) of the "statistical information", and Figure 4 The path corresponding to the index information is defined in the statistics file.

[0079] For example, when creating the Iceberg data table, the code for storing the index metadata information in blob-metadata is as follows:

[0080]

[0081] For the sync-build parameter, when it is set to true, the index is built synchronously; when it is set to false, it is built asynchronously. When building synchronously, it means that the index information will be built synchronously when inserting data. For the auto-build parameter, when using asynchronous building, whether to enable the regular building function.

[0082] exist Figure 4 In the statistics file, the definition example may be "MagicBlob1Blob2...Blob n "Footer", where Blob stores index information and Footer stores metadata information corresponding to the blob. It does not distinguish between file level and partition level. The statistics file is at the snapshot level, so when creating an index, the index of the current snapshot is created. The following takes the BitMap index as an example to introduce the metadata information:

[0083] Taking Range-Encoded (Base-2) and BitMap indexes as examples, the supported data types are integer types such as int, or fields with lexicographical order, and the cardinality value is specified, such as max-min+1.

[0084] like Figure 5As shown in the figure, for each animal name (such as Sea Horse, etc.), the corresponding raw data of the number are 3, 392, 047, 956, and 219 respectively. After reading the raw data of the index, it can be converted into binary data with a length of the base. For example, 3, 392, 047, 956, and 219 can be converted into binary data with a length of 10. Figure 5 The horizontal coordinates are "0000000011", "0110001000", "0000101111", "1110111100", "0011011011". Then the binary data is encoded according to range-encoded. For example, a matrix with a length of 10 can be defined, and the stored data is: 2^10-real data, for example, Figure 5 The amount of data in Manatee is 3, and the stored data means that the stored value has reached the actual value, that is, the actual value does not exceed the stored value. In this way, for the scenario of finding information about the number of animals exceeding 3, the matrix can be read. The actual value exceeds 3, which is converted to: construct a range-encoded value exceeding 3 (that is, 4): [11011111], then the data in the bitmap matrix is ​​less than this value. For the scenario of finding information about the number of animals less than 3, it is 2^10-stored value <3.

[0085] It should be noted that the blob element in the statistics file can be a bitmap matrix corresponding to a file or a bitmap matrix of a rowgroup. The definition code of blob is as follows:

[0086] public Blob(

[0087] String type,

[0088] List <integer>inputFields,

[0089] long snapshotId,

[0090] long sequenceNumber,

[0091] ByteBuffer blobData,

[0092] @Nullable PuffinCompressionCodec requestedCompression,

[0093] Map<String,String> properties)

[0094] Among them, blobData stores the bitmap matrix information: [0011111111 11101110011...], and its length is: 10*number of file lines. properties includes the data file name (filename), bitmap parameters, and the interval range (offset) information corresponding to the data file name.

[0095] In addition, the definition code format for Footer can be "Magic FooterPayloadFooterPayloadSize Flags Magic". Among them, FooterPayloadda stores the meta of the blob: index parameter configuration, seq-no, column field corresponding to the index, whether to compress, blob length (compressed), etc.

[0096] S302: Establishing a correspondence between data files and indexes in Iceberg.

[0097] In this embodiment, after adding index information to the metadata file of the Iceberg data table through step S301, the data file can be further divided into row groups in Iceberg to generate a corresponding task list, and then a corresponding relationship with the index is constructed according to the column name of the row group in the data file to execute the subsequent step S303.

[0098] Specifically, if Figure 6 As shown in the figure, after adding index information to the metadata file of the Iceberg data table, you can read the data file in Iceberg, split it into row groups, and generate the corresponding task list. <task>), and each task processes a split, then reads each row of data in the split, generates index information according to the index algorithm, that is, builds a corresponding relationship with the index according to the column name of the row group in the data file, and records the interval range (offset) and length (length) of the row group data.

[0099] Taking the Spark engine to build an index as an example, the specific implementation code is as follows:

[0100] val df=spark.read("iceberg_table")

[0101] df.map(x->index_encode(x)).repartition(1).writePuffin(" / path")

[0102] Just implement the index_encode and writePuffin interfaces.

[0103] It should be noted that the above implementation is a process of fully building index information. In order to improve resource utilization, an optional implementation method is that the present application provides an implementation method for incrementally building indexes: first, it is necessary to determine the newly added data file in Iceberg (defined here as the first data file), and generate index information and metadata information corresponding to the first data file, and then compare the snapshot information before and after the addition of the first data file in Iceberg, and delete the data file to be deleted according to the comparison result (defined here as the second data file), then, the index information corresponding to the newly added first data file can be merged with the index information before the addition of the first data file in Iceberg, and the index information corresponding to the second data file is deleted to obtain the updated index information.

[0104] Specifically, in this implementation, the statistical files and snapshots correspond one to one. The example diagram of the COW table is as follows: Figure 7 As shown, after the index file statFile_s1 is created at snapshot s1, the build index operation is triggered at snapshot s3. At this time, we can first compare the differences between s3 and s1, delete data file1, and add data file11 and data file4. The sample code can be "TableScan appendsBetween(long fromSnapshotId, long toSnapshotId)", where the newly added DataFile list can be obtained in TableScan. Then, index information and metadata information are generated for datafile11 and datafile4. Next, snapshot s1 and snapshot s3 are compared, and the datafile list with status DELETED (that is, data file1) is parsed. Then, the index information of datafile11 and datafile4 can be merged with statFile_s1, and the index information of datafile1 can be deleted to obtain the updated index information.

[0105] S303: When executing the processing operation of the target data in Iceberg, according to the correspondence between the data file and the index, the target data is matched at the row group level to determine the target data file.

[0106] In this embodiment, after the correspondence between the data file and the index is established in Iceberg through step S302, when further processing operations on the target data are executed in Iceberg, the target data can be index matched at the row group level according to the correspondence to determine the target data file for executing the subsequent step S304.

[0107] Specifically, after building a corresponding relationship with the index based on the column name of the row group in the data file, the name of the data file where the row group corresponding to the index is located, the interval range of the row group, and the length of the row group can be stored in the corresponding data storage object (blob) in the statistical file, such as Figure 6 In this way, when performing processing operations such as querying the target data, the corresponding target data storage object (blob) in the statistical file can be determined based on the correspondence between the data file and the index, and then the target data file and the interval range and length of the target row group data in the target data file can be determined based on the index information contained in the target data storage object to execute the subsequent step S304.

[0108] S304: Perform corresponding data processing operations on the target row group data in the target data file to obtain a data processing result.

[0109] In this embodiment, after the target data file is determined in step S303, a corresponding data processing operation may be further performed on the target row group data in the target data file to obtain a data processing result.

[0110] Specifically, an optional implementation method is, after determining the target data file and the interval range and length of the target row group data in the target data file based on the index information contained in the target data storage object, the task can be split according to the interval range and length of the target row group data in the target data file, thereby greatly reducing the number of tasks, and after executing the split tasks, the data processing results are obtained, which also improves data processing efficiency.

[0111] Example: Based on the above Figure 7 In the example of s1, s2, and s3, statFile_s1 is generated at s1, and SQL statement query is performed at s3. Since the current snapshot has statFile_s1, that is, the index is not invalid, the flowchart for implementing data query in Iceberg is as follows: Figure 8 As shown. It should be noted that for the or scenario and when there is no index-column, index can be omitted. For example, col1 is an index column, and col2 is not an index column. The implementation code example can be "select * from A where A.col1 = 'a' or A.col2 = 'b'".

[0112] It should also be noted that historical Index information can also be used. If there are many snapshot lists between the current snapshot and the snapshot used to build the index, the index can be omitted. This is because the snapshots used for the current snapshot and the snapshots used to build the index will be quite different. Otherwise, first get the List corresponding to the Current snapshot. <datafile>Then, according to the index file, get each offset and length of the row group data in the data file, and record the list that is not in the index file <datafile>In the end, we get some data file fragments with offset and length, and data files without offset. Next, we switch tasks according to splitSize. The specific implementation code is as follows:

[0113]

[0114] It should be noted that other data processing operations such as deletion, modification, and insertion in Iceberg can be performed by referring to the implementation method of the above data query, which will not be described in detail here.

[0115] In this way, by adding index information to the metadata when creating the Iceberg table, when executing a data query statement, matching data can be filtered out from all files according to the query filtering conditions, and the data can be reallocated according to the task size, greatly reducing the number of tasks and improving the execution time of the query statement.

[0116] In addition, in order to ensure the accuracy of the index information, an optional implementation method is to periodically build the index information incrementally in Iceberg according to a preset period to obtain an updated correspondence between the data file and the index in Iceberg.

[0117] Among them, the specific value of the preset period can be set according to actual conditions and experience values, and the embodiment of the present application does not limit this. For example, the preset period can be set to 1 week or one month, etc.

[0118] Another optional implementation is that this embodiment also supports changing the index configuration. Generally, after modifying the index configuration information, it is necessary to fully build the index information in Iceberg, rebuild the index for all data files, and obtain the updated correspondence between the data files and the index in Iceberg. The historical index files (i.e., statistical files) will also continue to be retained, so that it is convenient to accelerate the use when accessing historical snapshots.

[0119] In addition, it should be noted that in addition to manually adding your index, you can also specify index information when creating the Iceberg table. When executing a query, first use the index to filter some unnecessary data to achieve less data reading. For different query scenarios, the required index is different. For example: bloomfilter is only suitable for equal conditions in queries (such as select*from table where col="value"); bitmap is only suitable for scenarios with a fixed cardinality, that is, the number of different values ​​in the table. If the user needs to specify it manually every time, it is more cumbersome. This application may also consider providing an automatic index building function so that users do not perceive it and improve the user experience. Specifically, first, you can count the queries executed by users in history, analyze the frequently accessed columns and query filtering conditions in the query, and then analyze the cardinality of the corresponding column in the table (that is, the number of different values), as well as the tree value type of the column (such as int, string). If it is not an integer type, nor an encodable string type (such as a city), and the cardinality is large, you can build a bloomfilter; if it is an integer type and the cardinality is fixed, you can build a bitmap index.

[0120] To facilitate understanding of the data processing method in Iceberg provided in this application, Figure 2 The example diagram of implementing data query in the Iceberg table structure shown in FIG. Fig. 9 The following is an example diagram of implementing data query in Iceberg. Figure 2 For example, when executing the data processing statement "select * from where column_2 = 'A'" in Iceberg, some data files can be filtered out by adding an index, and the data position can be quickly located to achieve less data reading. For example, a table with 10T of data has 1,000 data files, and each data file is 10G. For a 10G data file, generally 128M (that is, a rowGroup) is a task, so 1 datafile has 80 tasks, and a total of 80,000 tasks. In the absence of an index, 80,000 tasks need to be executed to scan all data files. Using the method provided in this embodiment, index information is added to Iceberg, and the index level is the rowGroup level (it can be assumed that a rowGroup is 128M). After reading the index file, it can be quickly determined that it is a rowGroup that meets the conditions, which greatly shortens the range of datafiles. Fig. 9 shown.

[0121] In summary, the present embodiment provides a method for processing data in Iceberg. First, when creating an Iceberg data table, index information is added to the metadata file of the data table. Then, a corresponding relationship between the data file and the index is established in Iceberg. Then, when executing the processing operation of the target data in Iceberg, the target data is matched at the row group level according to the corresponding relationship between the data file and the index to determine the target data file. Then, the target row group data in the target data file can be subjected to corresponding data processing operations to obtain data processing results. It can be seen that since the present application first adds index information to the metadata when creating the Iceberg table, after the corresponding relationship between the data file and the index is established, when performing operations such as data update, deletion, and query, the location of the target data file can be quickly located and the corresponding processing operation can be performed by performing index matching at the row group level, thereby improving the processing efficiency of data in Iceberg.

[0122] Second embodiment

[0123] This embodiment will introduce a data processing device in Iceberg. For related content, please refer to the above method embodiment.

[0124] See also Fig.10 , is a schematic diagram of the composition of a data processing device in Iceberg provided in this embodiment, the device 1000 includes:

[0125] An adding unit 1001 is used to add index information to a metadata file of an Iceberg data table when creating the data table;

[0126] An establishing unit 1002 is used to establish a corresponding relationship between a data file and the index in the Iceberg;

[0127] A matching unit 1003 is used to perform index matching at a row group level on the target data according to the corresponding relationship between the data file and the index to determine the target data file when executing the processing operation of the target data in the Iceberg;

[0128] The processing unit 1004 is used to perform corresponding data processing operations on the target row group data in the target data file to obtain a data processing result.

[0129] In an implementation of this embodiment, the establishing unit 1002 includes:

[0130] A split subunit, used to split the data file into row groups in Iceberg and generate a corresponding task list;

[0131] The construction subunit is used to construct a corresponding relationship with the index according to the column name of the row group in the data file.

[0132] In an implementation of this embodiment, the establishing unit 1002 further includes:

[0133] A storage subunit is used to store the name of the data file where the row group corresponding to the index is located, the interval range of the row group, and the length of the row group into a corresponding data storage object in the statistical file;

[0134] The matching unit 1003 includes:

[0135] a first determining subunit, configured to determine a corresponding target data storage object in a statistical file according to a correspondence between the data file and the index when executing a target data processing operation in the Iceberg;

[0136] The second determining subunit is used to determine the target data file and the interval range and length of the target row group data in the target data file according to the index information included in the target data storage object.

[0137] In an implementation of this embodiment, the processing unit 1004 is specifically configured to:

[0138] The task is segmented according to the interval range and length of the target row group data in the target data file, and the data processing result is obtained after the segmented task is executed.

[0139] In an implementation of this embodiment, the device further includes:

[0140] A generating unit, configured to determine a first data file newly added in the Iceberg, and generate index information and metadata information corresponding to the first data file;

[0141] A comparison unit, used to compare the snapshot information before and after the first data file is added in Iceberg, and delete the second data file according to the comparison result;

[0142] The merging unit is used to merge the index information corresponding to the newly added first data file with the index information before the newly added first data file in Iceberg, and delete the index information corresponding to the second data file to obtain updated index information.

[0143] In an implementation of this embodiment, the device further includes:

[0144] The first construction unit is used to incrementally construct index information in the Iceberg according to a preset period to obtain an updated corresponding relationship between the data file in the Iceberg and the index.

[0145] In an implementation of this embodiment, the device further includes:

[0146] The second construction unit is used to fully construct the index information in the Iceberg after modifying the configuration information of the index, and obtain an updated corresponding relationship between the data file in the Iceberg and the index.

[0147] Furthermore, an embodiment of the present application also provides a data processing device in Iceberg, including: a processor, a memory, and a system bus;

[0148] The processor and the memory are connected via the system bus;

[0149] The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the processor executes any one of the implementation methods of the data processing method in Iceberg.

[0150] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes any implementation method of the data processing method in the above-mentioned Iceberg.

[0151] It can be known from the description of the above implementation mode that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling a computer device (which can be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0152] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0153] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0154] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.< / datafile> < / datafile> < / task> < / integer>

Claims

1. A data processing method in Iceberg, characterized in that: The method comprises: When creating an Iceberg data table, adding index information to a metadata file of the data table; Establishing a corresponding relationship between the data file and the index in the Iceberg; When executing a processing operation of the target data in the Iceberg, index matching is performed on the target data at a row group level according to a correspondence between the data file and the index to determine a target data file; Perform corresponding data processing operations on the target row group data in the target data file to obtain a data processing result; The establishing of the correspondence between the data file and the index in Iceberg includes: In the Iceberg, the data file is segmented into row groups to generate a corresponding task list; According to the column name of the row group in the data file, a corresponding relationship with the index is constructed, and the corresponding relationship between the column name and the index is generated by processing each row of data included in each segmentation using an index algorithm.

2. The method according to claim 1, characterized in that After constructing a corresponding relationship between the column name of the row group in the data file and the index, the method further includes: The name of the data file where the row group corresponding to the index is located, the interval range of the row group, and the length of the row group are stored in the corresponding data storage object in the statistical file; When the target data processing operation is performed in the Iceberg, index matching is performed on the target data at the row group level according to the corresponding relationship between the data file and the index to determine the target data file, including: When executing a processing operation of the target data in the Iceberg, determining a corresponding target data storage object in the statistical file according to a corresponding relationship between the data file and the index; According to the index information included in the target data storage object, a target data file and an interval range and a length of target row group data in the target data file are determined.

3. The method according to claim 2, characterized in that The performing corresponding data processing operations on the target row group data in the target data file to obtain the data processing results includes: The task is segmented according to the interval range and length of the target row group data in the target data file, and the data processing result is obtained after the segmented task is executed.

4. The method according to claim 1, characterized in that The method further comprises: Determine a first data file newly added in the Iceberg, and generate index information and metadata information corresponding to the first data file; Comparing snapshot information before and after the first data file is added in Iceberg, and deleting the second data file according to the comparison result; The index information corresponding to the newly added first data file is merged with the index information before the newly added first data file in Iceberg, and the index information corresponding to the second data file is deleted to obtain updated index information.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Incrementally construct index information in the Iceberg according to a preset period, and obtain an updated correspondence between the data file in the Iceberg and the index.

6. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: After the configuration information of the index is modified, the index information is fully constructed in the Iceberg to obtain an updated correspondence between the data file in the Iceberg and the index.

7. A data processing device in Iceberg, characterized in that: The device comprises: An adding unit, used for adding index information in a metadata file of an Iceberg data table when creating the data table; An establishing unit, used for establishing a corresponding relationship between a data file and the index in the Iceberg; A matching unit, configured to perform index matching at a row group level on the target data according to a correspondence between the data file and the index to determine a target data file when executing a processing operation of the target data in the Iceberg; A processing unit, used to perform corresponding data processing operations on the target row group data in the target data file to obtain a data processing result; The building blocks include: A split subunit, used to split the data file into row groups in Iceberg and generate a corresponding task list; The construction subunit is used to construct a correspondence between the column name and the index according to the column name of the row group in the data file, and the correspondence between the column name and the index is generated by processing each row of data included in each segmentation using an index algorithm.

8. A data processing device in Iceberg, characterized in that: include: Processor, memory, system bus; The processor and the memory are connected via the system bus; The memory is used to store one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by the processor, the processor executes the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes the method according to any one of claims 1 to 6.

Images