A method, device and equipment for realizing a table service scheme of a stream-batch integrated data lake

Abstract

The application provides a method, device and equipment for realizing a table service scheme of a flow batch integrated data lake, the method comprising: creating a data table, determining a primary key field of the table, and specifying an attribute of a partition quantity threshold; performing Hash calculation according to the primary key to generate an integer type Hash value, then taking the Hash value modulo the partition quantity threshold, and the remainder value is a partition number corresponding to the primary key record, thereby completing partition; reading data, performing Hash calculation according to the primary key field to obtain a partition number field, when data is written, partitioning according to the partition number field, generating a partition directory, and each partition directory corresponding to multiple data files; and performing operation on the data according to requirements, including data writing, data querying and data merging. In this way, the same set of data processing architecture is used, offline and real-time data processing is simultaneously supported, data can be quickly updated and deleted according to the primary key, and data can be queried in real time.

Description

Technical Field

[0001] The embodiments of the present invention generally relate to the field of data processing technology, and in particular to a table service solution, apparatus and equipment for realizing a data lake integrating stream and batch processing. Background Technology

[0002] With the development of technology, many application scenarios involve infinite amounts of data, and batch processing can no longer meet the needs of these scenarios. As market demand increases, stream processing engines are becoming increasingly popular. Stream processing engines are used for real-time computing, processing data as it arrives, unlike batch processing which requires waiting for a batch of data to arrive before processing. The advantage of real-time computing in stream processing engines makes them widely used in applications such as network monitoring and e-commerce.

[0003] Hive is a data warehouse analysis and processing tool based on Hadoop, and it can be queried using SQL statements. Because Hive's underlying layer is a wrapper around streaming Hadoop, Hive's data is stored in Hadoop files, giving it excellent batch processing capabilities. However, Hive does not support stream processing. For data in different partitions, each partition is a folder, and the files in the folder are the stored data. In Hive, when new data is added to an old partition, new files are automatically created in HDFS. Data cannot be added to existing files. Therefore, the stream processing engine cannot read newly added files in old partitions of Hive tables, meaning it does not support dynamically reading new files in old partitions of Hive.

[0004] When a large number of new partition files are added at once, the streaming engine cannot read all the new partition files at once. Due to the time difference, the partitions in the streaming engine and the partitions in Hive will not match, resulting in some new partitions being missed.

[0005] For example, the patent "A method, apparatus and device for dynamically reading Hive table data using a stream processing approach (application number: CN202111194393.0)" states that the method includes: setting partitionMap and partitionValueList, setting the timestamp in the first round of reading to a preset initial value; if the partition name to be read in partitionValueList can be found in partitionMap, then the corresponding partition to be read is determined to be an old partition; if the file modification time of the file to be read in the old partition is greater than the largest file modification time already read in the corresponding partition in partitionMap, and the file modification time of the file to be read in the old partition is greater than the timestamp of the current round, then the file to be read in the old partition is determined to be a new file in the old partition, and the new file in the old partition is read according to the path of the corresponding file to be read in partitionValueList. The streaming processing method provided by this invention supports dynamic reading of Hive table data. It can execute Hive data sources on the streaming mechanism, enabling the streaming engine to dynamically read new files from old partitions in Hive. However, it cannot process offline and real-time data simultaneously in Hive. Furthermore, the logic of writing data to Hive tables is complex. Migrating to a new storage system that supports updates, such as Iceberg, would involve a large amount of work and high risk. Summary of the Invention

[0006] To address the above issues, this invention employs a single data processing architecture that supports both offline and real-time data processing. Real-time data processing supports data updates and deletions, allowing for rapid updates and deletions based on primary keys, and enabling real-time data queries. It also enables the merging of data in three different ways and is compatible with Hive, supporting the upgrading of Hive's offline architecture to a real-time architecture, and supporting update and deletion operations on top of Hive.

[0007] According to embodiments of the present invention, a method, apparatus, and device are provided for implementing a table service scheme for a batch processing integrated data lake.

[0008] In a first aspect of the present invention, a method is provided for implementing a table service scheme for a stream-batch integrated data lake. The method includes:

[0009] S01: Create a data table, determine the primary key field of the table, and specify the attribute for the threshold number of partitions;

[0010] S02: Perform a hash calculation based on the primary key to generate an integer hash value. Then, take the remainder of the hash value with respect to the partition number threshold. The remainder is the partition number corresponding to the primary key record, thus completing the partitioning.

[0011] S03: Read data, perform hash calculation based on the primary key field to obtain the partition number field, and partition the data according to the partition number field when writing data, generate partition directories, and each partition directory corresponds to multiple data files;

[0012] S04: Perform data operations as needed, including: data writing, data querying, and data merging.

[0013] Furthermore, the number of primary key fields described in S01 is at least one, and multiple primary key fields are separated by commas.

[0014] Furthermore, the hash calculation method described in S02 is not unique.

[0015] Furthermore, the data described in S03 is stored in the real-time data directory after being written. The real-time data is merged periodically, merging newly added data files and deleted data files. The merged files are stored in the Hive data directory. Offline data is written directly to Hive. When reading data, both the data in the Hive table and the real-time data are read simultaneously, thereby achieving compatibility with the Hive table.

[0016] Furthermore, the data writing operations described in S04 include: adding data, deleting data, and updating data;

[0017] The steps for adding data are as follows: add data, create a new data file, and partition the data file;

[0018] The data deletion operation steps are as follows: specify a primary key, identify the data with this primary key as to be deleted, and store the primary key in a file of type data deletion.

[0019] The data update operation steps are as follows: add a file of deletion type and a file of addition type respectively, store the primary key of the data to be updated in the file of deletion type, and store the updated data in the file of addition type.

[0020] Furthermore, the data to be updated must contain data from all fields.

[0021] Furthermore, during the data writing operation, a submission sequence number plus a record offset is used to ensure the order of data writing;

[0022] The commit sequence number is generated each time a write operation is performed. The commit sequence number is an incrementing integer type, and all files in the current write batch share the same commit sequence number.

[0023] The record offset is generated as an incrementing sequence number for each record. The record offset is recorded inside each data file, and each record corresponds to a record offset.

[0024] Furthermore, the data query steps described in S04 are as follows:

[0025] CX01: Reads the metadata file for each partition;

[0026] CX02: Within each partition, read data files according to the order of data files defined in the metadata file;

[0027] CX03: Within each partition, perform logical merging after filtering based on the data file type;

[0028] CX04: Return the merged data to the user.

[0029] Furthermore, the logical merging steps described in CX03 are as follows:

[0030] CX031: Iterate through each file sequentially;

[0031] CX02: If the file is a newly added file, it will be read directly, and the read data will be used as input data for subsequent processes;

[0032] CX033: If the file is a deleteable file, then if the data read before this point contains the primary key, then delete the data containing the primary key;

[0033] CX034: The processed data is used as input data for subsequent file traversal and logical merging.

[0034] Furthermore, the data merging described in S04 includes three types: merging small files of the deleted type, merging files of the newly added type, and merging files of both the deleted and newly added types simultaneously;

[0035] The steps for merging and deleting small files are as follows: find and remove the files of the deletion type in each partition, and merge and store the removed files in the parent node;

[0036] The steps for merging newly added type files are as follows: find newly added type small files in each partition, and merge multiple newly added type small files in the same partition into one large newly added type file;

[0037] The steps for merging deleted and newly added files simultaneously are as follows: Locate deleted files and newly added small files in each partition, remove the deleted files, and merge the newly added small files at the same time.

[0038] In a second aspect of the invention, an apparatus is provided for implementing a table service scheme for a stream-batch integrated data lake. The apparatus includes:

[0039] Data table creation module: Used to create data tables, determine the primary key field of the table, and specify the attribute of the number of partitions threshold;

[0040] Partitioning module: Used to perform a hash calculation based on the primary key, generate an integer hash value, and then take the remainder of the hash value with respect to the number of partitions threshold. The remainder is the partition number corresponding to the primary key record, thus completing the partitioning.

[0041] Data writing module: Reads data, performs hash calculation based on the primary key field to obtain the partition number field, and partitions the data according to the partition number field during data writing, generating partition directories, with each partition directory corresponding to multiple data files;

[0042] Operation module: Used to perform operations on data according to requirements, including: data writing, data querying, and data merging.

[0043] In a third aspect of the invention, an electronic device is provided. The electronic device includes a memory and a processor, the memory storing a computer program, the processor executing the program to implement the method according to a first aspect of the invention.

[0044] In a fourth aspect of the invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method according to a first aspect of the invention.

[0045] The above-mentioned English abbreviations have the following meanings:

[0046] Hash: Hash function

[0047] ID: Identity document, ID card number

[0048] Spark: A general-purpose big data analytics engine

[0049] Hive: A data warehouse tool based on Hadoop

[0050] Hadoop: A distributed system infrastructure developed by the Apache Software Foundation

[0051] SQL: Structured Query Language

[0052] HDFS: Hadoop Distributed File System

[0053] This invention uses the same data processing architecture, supporting both offline and real-time data processing. Real-time data processing supports data updates and deletions, and data can be quickly updated and deleted based on the primary key. Data can be queried in real time. It can perform three different types of data merging. It is also compatible with Hive, supporting the upgrading of Hive's offline architecture to a real-time architecture, and supporting update and deletion operations on top of Hive.

[0054] It should be understood that the description in the Summary of the Invention is not intended to limit the key or essential features of the embodiments of the present invention, nor is it intended to restrict the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0055] The above and other features, advantages, and aspects of the various embodiments of the present invention will become more apparent from the accompanying drawings and the following detailed description. Wherein:

[0056] Figure 1 A flowchart illustrating a method for implementing a table service scheme for a stream-batch integrated data lake according to an embodiment of the present invention is shown.

[0057] Figure 2 A block diagram of an apparatus for implementing a table service scheme for a stream-batch integrated data lake according to an embodiment of the present invention is shown;

[0058] Figure 3 A schematic diagram of partitioning according to an embodiment of the present invention is shown;

[0059] Figure 4 A schematic diagram illustrating partitioning using a primary key for hash calculation according to an embodiment of the present invention is shown.

[0060] Figure 5 A schematic diagram of a merge-delete type of small file is shown according to an embodiment of the present invention;

[0061] Figure 6 A schematic diagram illustrating the merging of newly added file types according to an embodiment of the present invention is shown;

[0062] Figure 7 A schematic diagram illustrating the simultaneous merging of deleted and newly added files according to an embodiment of the present invention is shown;

[0063] Figure 8 A schematic diagram illustrating the simultaneous merging of deleted and newly added files according to an embodiment of the present invention is shown;

[0064] Figure 9 A schematic diagram of an apparatus for implementing a table service scheme for a stream-batch integrated data lake according to an embodiment of the present invention is shown. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] According to embodiments of the present invention, a method, apparatus, and device are proposed for implementing a table service scheme for a data lake integrating stream and batch processing. These solutions use the same data processing architecture and simultaneously support offline and real-time data processing. Real-time data processing supports data updates and deletions, allowing for rapid updates and deletions based on primary keys, and enabling real-time data queries. The solutions can perform three different types of data merging and are compatible with Hive, supporting the upgrading of Hive's offline architecture to a real-time architecture, and supporting update and deletion operations on top of Hive.

[0067] The principles and spirit of the present invention will be explained in detail below with reference to several representative embodiments.

[0068] Figure 1 This is a flowchart illustrating a method for implementing a table service solution for a unified stream and batch data lake according to an embodiment of the present invention. The method includes:

[0069] S01: Create a data table, determine the primary key field of the table, and specify the attribute for the threshold number of partitions;

[0070] S02: Perform a hash calculation based on the primary key to generate an integer hash value. Then, take the remainder of the hash value with respect to the partition number threshold. The remainder is the partition number corresponding to the primary key record, thus completing the partitioning.

[0071] S03: Read data, perform hash calculation based on the primary key field to obtain the partition number field, and partition the data according to the partition number field when writing data, generate partition directories, and each partition directory corresponds to multiple data files;

[0072] S04: Perform data operations as needed, including: data writing, data querying, and data merging.

[0073] It should be noted that although the operation of the method of the present invention has been described in a specific order in the above embodiments and figures, this does not require or imply that the operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0074] To provide a clearer explanation of the above-described method for implementing a unified batch and stream data lake table service, a specific embodiment will be used for illustration below. However, it is worth noting that this embodiment is only for better illustrating the present invention and does not constitute an improper limitation of the present invention.

[0075] The following concrete example will further illustrate the method for implementing a table service solution for a unified stream and batch data lake:

[0076] Create a data table and determine the primary key field, as shown in the example below:

[0077]

[0078] Using `primary key` specifies `id` as the primary key field for the table. When creating the table, multiple fields can be specified as primary keys, separated by commas. Alternatively, `partitioned by` can be used to specify the partition key field for the table.

[0079] The using statement specifies the format of the data table, which is the format of our custom data source. Here, the name is specified as "lake".

[0080] When creating a table, you can specify its relevant attributes and partition thresholds, as shown in the following example:

[0081]

[0082]

[0083] The table partition threshold (max_partition_threshold) is specified as 4 using the properties property.

[0084] A unique integer value is generated based on the primary key. For example, a hash calculation is performed on a single primary key or multiple primary keys to generate an integer hash value. Then, the remainder of the hash value is taken by dividing it by the partition number threshold. The remainder is the partition number corresponding to the primary key record.

[0085] The partition number threshold is a fixed value and is generally not modified after it is determined. Partitions are used to store data files, and a partition can store multiple data files. The hash value generated from the primary key of the data corresponding to these data files, when modulo the partition number threshold, will always have the same remainder.

[0086] The remainder is used as the partition number and stored in the leaf node of the binary tree. The partition number corresponds to the node number of the leaf node of the binary tree.

[0087] Let's illustrate this with a specific example, assuming we have data as shown in Table 1:

[0088] Table 1

[0089] id value create_time 1 aaa 2022-09-0712:11:21 2 aab 2022-09-0713:12:25 3 aac 2022-09-0711:15:30 4 bbb 2022-09-0714:22:45 5 BBC 2022-09-0715:35:52 6 fef 2022-09-0715:35:55 7 qee 2022-09-0715:37:37 8 eee 2022-09-0716:35:56

[0090] Using Spark's data source interface, define an interface for writing / querying data sources. Create a new data source class that inherits from the abstract classes `reatableRelationProvider` and `DataSourceRegister`, and implement the relevant methods as follows:

[0091]

[0092]

[0093] In the code of the class above, two methods are overridden: createRelation and shortName. createRelation is used to implement the logic of writing data, and shortName is the name of the data source.

[0094] Based on the previous steps, we know that the partition threshold is 4, which means that a maximum of 4 partitions will be generated to store data, but each partition can contain multiple data files.

[0095] The hash value is calculated based on the primary key. There are many types of hash algorithms. Here, we assume that a certain hash algorithm is selected, a value is generated, and then the remainder is taken with respect to the number of partitions, as shown in Table 2:

[0096] Table 2

[0097] id Hash value calculated based on a certain hash algorithm Hash modulo the number of partitions 1 100 100％4＝0 2 101 101％4＝1 3 102 102％4＝2 4 103 103％4＝3 5 104 104％4＝0 6 105 105％4＝1 7 106 106％4＝2 8 107 107％4＝3

[0098] In each batch, under each partition, one or more files may be generated. Multiple batches of data will be written to the generated data files within each partition.

[0099] The number of files generated in each batch depends on the parallelism of the data writing. If the parallelism is 2, then each partition will generate a maximum of 2 files. A partition diagram is shown below. Figure 3 As shown.

[0100] To add data, follow these steps:

[0101] First, insert two data entries and generate a new file, file 1. The contents of file 1 are shown in Table 3.

[0102] Table 3

[0103] id value 11 aa 12 bb

[0104] Next, after inserting two more data entries, a new file, file 2, will be generated. The contents of file 2 are shown in Table 4:

[0105] Table 4

[0106] id value 13 cc 14 dd

[0107] Both of the above files are of the new file type.

[0108] To delete data, follow these steps:

[0109] To delete the record with id=2, create a new file: File 3. The file type is deletion, and this file records the primary key to be deleted. The contents of File 3 are shown in Table 5.

[0110] Table 5

[0111] id 12

[0112] During data querying, files 1, 2, and 3 are read sequentially. File 3 records the primary keys of the deleted data. The primary key lists read from files 1 and 2 are: 11, 12, 13, and 14. File 3 reads primary key 12. Since file 3 is a deletion type, the record with primary key 12 is deleted from the primary key lists [11, 12, 13, 14] generated from files 1 and 2. Ultimately, only data with id = 11, 13, and 14 will be returned to the user, as shown in Table 6.

[0113] Table 6

[0114] id value 11 aa 13 cc 14 dd

[0115] The steps to update the data are as follows:

[0116] Suppose that the name of the record with id=13 is updated to ccc1.

[0117] First, create a new file for deletion: File 4, which records the primary key of the data to be updated, as shown in Table 7:

[0118] Table 7

[0119] id 13

[0120] Then, generate a new file type: File 5. Record the updated data, the contents of which are shown in Table 8:

[0121] Table 8

[0122] id value 13 cc1

[0123] When performing data queries, the results are merged according to the order and type of the generated files. Data with id=3 in the original files is removed, and data with id=3 in the last newly added file is added. The results are then returned to the user. The final data returned to the user is shown in Table 9.

[0124] Table 9

[0125] id value 11 aa 13 cc1 14 dd

[0126] Data updates are split into two steps: deletion and addition. Therefore, multiple consecutive data deletion and update operations will generate multiple files of both addition and deletion types. To ensure sequentiality, a commit sequence number plus a record offset is maintained during data writing. The commit sequence number is generated for each write operation and is an incrementing integer. All files in the current write batch share this commit sequence number. The record offset is an incrementing sequence number generated for each record, ensuring it is incrementing and unique within the current batch.

[0127] For example, a batch might execute the following five consecutive steps:

[0128] INSERT(100,101,104,105,108)

[0129] DELETE 101, 104

[0130] INSERT(112,113,116,120,124)

[0131] DELETE 96

[0132] INSERT(128,129)

[0133] The value is the primary key of the inserted record. In this current batch, 5 operations were performed. Assuming the partition threshold is 4 and the batch commit sequence number is 10, according to the previous rules, a hash calculation is performed using the primary key. The above data will generate data for two partitions, partition 0 and partition 1, as follows: Figure 4 As shown.

[0134] The steps to perform a data novelty search are as follows:

[0135] Taking partition 0 above as an example, when querying the record with primary key 104, this record is included in both the newly added type file 1 and the deleted type file 1. However, the offset of the deleted type file 1 (equal to 7) is greater than the offset of the newly added type file 1 (equal to 3). Therefore, the operation is to add first and then delete. In the end, this record is in the deleted state, so the query result is empty.

[0136] Create a new data source class that inherits from the abstract classes RelationProvider and DataSourceRegister, and implement the relevant methods as follows:

[0137]

[0138] To merge the data, follow these steps:

[0139] 1. Merge and delete small files: such as Figure 5 As shown, deleting file 1 in partition 0 and partition 1 will remove file 1 from each partition and merge the two files into the parent node.

[0140] 2. Merge newly added file types: such as Figure 6 As shown, partition 0 contains three files: newly added file 1, newly added file 2, and deleted file 1; partition 1 contains three files: newly added file 1, newly added file 2, and deleted file 1; newly added file 1 and newly added file 2 in partition 0 are merged to form a larger newly added file 1, and newly added file 1 and newly added file 2 in partition 1 are merged to form a larger newly added file 1.

[0141] 3. Simultaneously merge files that are deleted or newly added: such as Figure 7 and Figure 8 As shown, partition 0 contains three files: newly added type file 1, newly added type file 2, and deleted type file 1; partition 1 contains three files: newly added type file 1, newly added type file 2, and deleted type file 1; while removing deleted type file 1 from each of the two partitions, newly added type file 1 and newly added type file 2 in each partition are merged to form a larger newly added type file 1.

[0142] Small files of the deletion type are very small because they only store information such as the primary key column and offset. The cost of merging them is also the lowest among the three types, so they can be merged frequently. For files of the addition type, a threshold can be set for the number of small files. When the number of small files reaches the threshold, the small files are merged. Merging files of the deletion and addition types at the same time is more expensive, but the efficiency of data querying after merging is the highest. Merging can be performed periodically, such as once a day or once an hour.

[0143] However, many enterprises still have existing data processing workflows that use Hive for data storage and retrieval. While migrating directly to this architecture would support real-time data updates and deletions, a direct migration would require rebuilding the underlying Hive files according to the new architecture. This is a massive undertaking with high risks, especially when Hive tables contain very large amounts of data, making it almost impossible to implement. Therefore, it's advisable to optimize the existing architecture to achieve compatibility with Hive.

[0144] Data is stored in two locations: real-time data and offline data in Hive. Real-time data is written using the architecture described above and stored in the real-time data directory. Real-time data is periodically merged, including newly added and deleted data files; the merged files are then stored in the Hive data directory. Offline data can still be directly written to Hive. When reading data, both Hive table data and real-time data are read simultaneously, ensuring compatibility with Hive tables.

[0145] Based on the same inventive concept, this invention also proposes an apparatus for implementing a table service scheme for a unified stream and batch data lake. The implementation of this apparatus can be found in the implementation of the method described above; repeated details will not be repeated. Figure 2 As shown, the device 100 includes:

[0146] Data table creation module 101: Used to create data tables, determine the primary key field of the table, and specify the attribute of the number of partitions threshold;

[0147] Partitioning module 102: Used to perform hash calculation based on the primary key, generate an integer hash value, and then take the remainder of the hash value with respect to the number of partitions threshold. The remainder is the partition number corresponding to the primary key record, thus completing the partitioning.

[0148] Data writing module 103: Reads data, performs hash calculation based on the primary key field to obtain the partition number field, and partitions the data according to the partition number field during data writing, generating partition directories, with each partition directory corresponding to multiple data files;

[0149] Operation module 104: Used to perform operations on data according to requirements, including: data writing, data querying, and data merging.

[0150] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the described module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0151] like Figure 9As shown, the device includes a central processing unit (CPU), which can perform various appropriate actions and processes based on computer program instructions stored in read-only memory (ROM) or loaded from storage units into random access memory (RAM). The RAM can also store various programs and data required for device operation. The CPU, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0152] Multiple components in the device are connected to the I / O interface, including: input units such as keyboards and mice; output units such as various types of displays and speakers; storage units such as disks and optical discs; and communication units such as network interface cards (NICs), modems, and wireless transceivers. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0153] The processing unit executes the various methods and processes described above, such as methods S01 to S04. For example, in some embodiments, methods S01 to S04 may be implemented as computer software programs tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps of methods S01 to S04 described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute methods S01 to S04 by any other suitable means (e.g., by means of firmware).

[0154] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload programmable logic devices (CPLDs), and so on.

[0155] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0156] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0157] Furthermore, although the operations are described in a specific order, this should be understood as requiring that such operations be performed in the specific order shown or in sequential order, or requiring that all illustrated operations be performed to achieve the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the invention. Certain features described in the context of individual embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations.

[0158] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for implementing a table service scheme for a stream-batch integrated data lake, characterized in that, The method includes: S01: Create a data table, determine the primary key field of the table, the number of primary key fields is at least one, multiple primary key fields are separated by commas, and specify the total number threshold of partitions; S02: Perform a hash calculation based on the primary key to generate an integer hash value. Then, take the remainder of the hash value with respect to the total number of partitions threshold. The remainder is the partition number corresponding to the primary key record, thus completing the partitioning. S03: Read data, perform hash calculation based on the primary key field to obtain the partition number field, partition the data according to the partition number field, generate partition directories, each partition directory corresponds to multiple data files, and store the data in the real-time data directory after writing. The real-time data is merged periodically, merging newly added data files and deleted data files. The merged files are stored in the Hive data directory. Offline data is directly written to Hive. When reading data, both the data in the Hive table and the real-time data are read simultaneously, thereby achieving compatibility with the Hive table. S04: Perform data operations according to requirements, including: data writing, data querying, and data merging; The data writing operations include: adding data, deleting data, and updating data; The steps for adding data are as follows: add data, create a new data file, and partition the data file; The data deletion operation steps are as follows: specify a primary key, identify the data with this primary key as to be deleted, and store the primary key in a file of type data deletion. The data update operation steps are as follows: add a file of deletion type and a file of addition type respectively, store the primary key of the data to be updated in the file of deletion type, and store the updated data in the file of addition type; The steps for data querying are as follows: CX01: Reads the metadata file for each partition; CX02: Within each partition, read data files according to the order of data files defined in the metadata file; CX03: Within each partition, perform logical merging after filtering based on the data file type; CX04: Return the merged data to the user; The data merging includes three types: merging deleted small files, merging newly added files, and merging both deleted and newly added files simultaneously. The steps for merging and deleting small files are as follows: find and remove the files of the deletion type in each partition, and merge and store the removed files in the parent node; The steps for merging newly added type files are as follows: find newly added type small files in each partition, and merge multiple newly added type small files in the same partition into one large newly added type file; The steps for merging deleted and newly added files simultaneously are as follows: Locate deleted files and newly added small files in each partition, remove the deleted files, and merge the newly added small files at the same time.

2. The method for implementing a table service scheme for a stream-batch integrated data lake according to claim 1, characterized in that, The hash calculation method described in S02 is not unique.

3. The method for implementing a table service scheme for a stream-batch integrated data lake according to claim 1, characterized in that, The data to be updated must contain data from all fields.

4. The method for implementing a table service scheme for a stream-batch integrated data lake according to claim 1, characterized in that, During the data writing operation, a submission sequence number plus a record offset is used to ensure the order of data writing. The commit sequence number is generated each time a write operation is performed. The commit sequence number is an incrementing integer type, and all files in the current write batch share the same commit sequence number. The record offset is generated as an incrementing sequence number for each record. The record offset is recorded inside each data file, and each record corresponds to a record offset.

5. The method for implementing a table service scheme for a stream-batch integrated data lake according to claim 1, characterized in that, The logical merging steps described in CX03 are as follows: CX031: Iterate through each file sequentially; CX032: If the file is a newly added file, it will be read directly, and the read data will be used as input data for subsequent processes; CX033: If the file is a deleteable file, then if the data read before this point contains the primary key, then delete the data containing the primary key; CX034: The processed data is used as input data for subsequent file traversal and logical merging.

6. An apparatus for implementing a table service solution for a stream-batch integrated data lake, characterized in that, The device includes: Data table creation module: Used to create data tables, determine the primary key field of the table, and specify the total number of partitions threshold; Partitioning module: Used to perform a hash calculation based on the primary key, generate an integer hash value, and then take the remainder of the hash value with respect to the total number of partitions threshold. The remainder is the partition number corresponding to the primary key record, thus completing the partitioning. Data writing module: Reads data, performs hash calculation based on the primary key field to obtain the partition number field, and partitions the data according to the partition number field during data writing, generating partition directories, with each partition directory corresponding to multiple data files; Operation module: Used to perform operations on data according to requirements, including: data writing, data querying, and data merging; The data writing operations include: adding data, deleting data, and updating data; The steps for adding data are as follows: add data, create a new data file, and partition the data file; The data deletion operation steps are as follows: specify a primary key, identify the data with this primary key as to be deleted, and store the primary key in a file of type data deletion. The data update operation steps are as follows: add a file of deletion type and a file of addition type respectively, store the primary key of the data to be updated in the file of deletion type, and store the updated data in the file of addition type; The steps for data querying are as follows: CX01: Reads the metadata file for each partition; CX02: Within each partition, read data files according to the order of data files defined in the metadata file; CX03: Within each partition, perform logical merging after filtering based on the data file type; CX04: Return the merged data to the user; The data merging includes three types: merging deleted small files, merging newly added files, and merging both deleted and newly added files simultaneously. The steps for merging and deleting small files are as follows: find and remove the files of the deletion type in each partition, and merge and store the removed files in the parent node; The steps for merging newly added type files are as follows: find newly added type small files in each partition, and merge multiple newly added type small files in the same partition into one large newly added type file; The steps for merging deleted and newly added files simultaneously are as follows: Locate deleted files and newly added small files in each partition, remove the deleted files, and merge the newly added small files at the same time.

7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 5.

Images