Data lake lock-free writing method, device, equipment, medium and product
By splitting the write operations into insertion and modification operations, and sorting them according to timestamps, the performance and delay problems caused by optimistic locks are solved, and the lock-free writing of data lakes is realized, and the performance and consistency of the data storage system is improved. .
Patent Information
- Application Number
- CN202411117352.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-15
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2044-08-15
AI Technical Summary
In multi-end write scenarios, existing data storage systems cannot effectively ensure data consistency due to the optimistic lock configuration.
Using the data lake lock-free writing method, each write operation is split into an insertion operation and a modification operation, and put it into the data insertion queue and a data modification queue respectively, and sort it according to the timestamp, using the submission queue to playback and modify operations to generate Copy the table on write or merge the format data of the table on read to complete the write task.
Improve the performance of the write scenario, reduce the write delay, improve the processing speed of the write task, and realize the lock-free writing process.
Smart Images

Figure CN118642665B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data writing technology, and in particular to a data lake lock-free writing method, device, equipment, medium and product. Background Art
[0002] Writing to the data storage system is a common usage scenario. For the same data table, there are roughly three scenarios: multi-end writing without asynchronous table service; multi-end writing with asynchronous table service; single-end writing with asynchronous table service. This writing scenario requires the data storage system to have targeted processing, otherwise the data consistency cannot be guaranteed.
[0003] In the relevant technical scheme, the data storage system is usually configured with an optimistic lock, which is a file-level lock. If multiple write operations involve different files, the write is successful; if they involve the same file, only one of these write operations can succeed. However, the configuration of the optimistic lock will have a certain impact on the performance and write latency of the write scenario. Summary of the invention
[0004] The purpose of the present invention is to provide a data lake lock-free writing method, device, equipment, medium and product, which can support lock-free writing of data, improve the performance of writing scenarios, and increase the processing speed of writing tasks.
[0005] In order to solve the above technical problems, the present invention provides a data lake lock-free writing method, the method comprising:
[0006] Receive write tasks;
[0007] Split each write operation of each write end into an insert operation and a modify operation, and put the insert operation into a data insert queue, and put the modify operation into a data modify queue;
[0008] Using the data insertion queue, all insertion operations are sorted according to timestamps, and the results of newly added data are saved; using the data modification queue, all modification operations are sorted according to timestamps, and the process of data modification is saved;
[0009] The data insertion queue and the data modification queue are sent to the submission queue, and the submission queue is used to replay the modification operation based on each insertion operation in timestamp order to generate and submit the format data of the write-time copy table or the read-time merge table to complete the write task.
[0010] In a first aspect, in the data lake lock-free writing method provided by the present invention, the result of using the data insertion queue to save the newly added data includes:
[0011] Using the data insertion queue to write the data files corresponding to each insertion operation, and sorting the written data files according to the partition path and the primary key value;
[0012] Save the sorted data file.
[0013] On the other hand, in the data lake lock-free writing method provided by the present invention, it also includes:
[0014] When there are two adjacent insert operations in the data insert queue and data writing is completed, determining whether there is a modification operation of the data modification queue between the adjacent insert operations;
[0015] If there is a modification operation, it will be directly marked as processed;
[0016] If there is no modification operation, determine whether there is duplicate data in adjacent insertion operations;
[0017] If there are duplicate data in adjacent insertion operations, the duplicate data will be overwritten or extracted into a modification operation accordingly. After the processing is completed, the adjacent insertion operation will be marked as completed.
[0018] On the other hand, in the data lake lock-free writing method provided by the present invention, determining whether adjacent insert operations have duplicate data includes:
[0019] View the partition path and primary key value field range information of the data files written by adjacent insert operations;
[0020] If the partition path of the data file being viewed overlaps with the range information of the primary key value field, it is determined that duplicate data exists in adjacent insert operations.
[0021] On the other hand, in the data lake lock-free writing method provided by the present invention, checking the partition path and range information of the primary key value field of the data file written by the adjacent insert operation includes:
[0022] Traverse the data files written by adjacent insert operations;
[0023] Use binary search to view the partition path of the data file and the range information of the primary key value field.
[0024] On the other hand, in the data lake lock-free writing method provided by the present invention, if there is duplicate data in adjacent insertion operations, the duplicate data is overwritten or extracted into a modification operation, including:
[0025] If there is duplicate data in adjacent insertion operations, determine the table type corresponding to the duplicate data; the adjacent insertion operations include an insertion operation of a first timestamp and an insertion operation of a second timestamp; the second timestamp is later than the first timestamp;
[0026] If the table type is a copy-on-write table, taking a union of the data files corresponding to the first timestamp and the second timestamp, and overwriting duplicate data in the data file corresponding to the first timestamp;
[0027] If the table type is a read-time merge table, the modification of the duplicate data is extracted as a modification operation and put into the data modification queue; wherein the timestamp of the modification of the duplicate data extracted as the modification operation is the same as the second timestamp.
[0028] On the other hand, in the data lake lock-free writing method provided by the present invention, after the modification of duplicate data is extracted as a modification operation and put into the data modification queue, it also includes:
[0029] The data file corresponding to the second timestamp is taken as a difference set between the data file corresponding to the second timestamp and the data file corresponding to the first timestamp, so that the insertion operation of the second timestamp generates a new data file.
[0030] On the other hand, in the data lake lock-free writing method provided by the present invention, before inserting the data into the queue and sending it to the submission queue, it also includes:
[0031] Periodically checking the head of the data insertion queue, if there is one or more consecutive insertion operations that have been marked as processed, and there is no modification operation in the data modification queue between the consecutive insertion operations, then sending the data insertion queue to the submission queue;
[0032] If there is a modification operation of the data modification queue corresponding to two consecutive insertion operations, a merge operation is performed in the data insertion queue and the data modification queue.
[0033] On the other hand, in the data lake lock-free writing method provided by the present invention, it also includes:
[0034] When the same write operation fails, the data insertion queue is used to remove the insert operations split from the failed write operation, and the data modification queue is used to remove the modification operations split from the failed single write operation;
[0035] Correspondingly, when the same write operation is successful, the process of saving the data modification by using the data modification queue includes:
[0036] The data modification queue is used to record the execution conditions of the modification corresponding to the same write operation, the modified target field and the modified value and save them.
[0037] On the other hand, in the data lake lock-free writing method provided by the present invention, it also includes:
[0038] Determining in the data modification queue whether a single modification operation involves multiple modifications to the same field data;
[0039] If multiple modifications are made to the same field data, the multiple modification operations for the same field data are merged so that the operation at the latest timestamp covers the operation before the latest timestamp;
[0040] If multiple modifications to the same field data are not involved, the multiple modifications to the same field data will not be merged.
[0041] On the other hand, in the data lake lock-free writing method provided by the present invention, determining whether a single modification operation involves multiple modifications to the same field data includes:
[0042] In a single modification operation, determine whether there is a data modification execution with the same structured query language conditions;
[0043] If there are data modifications with the same structured query language conditions, it is determined that multiple modifications to the same field data are involved;
[0044] If there is no data modification execution with the same structured query language condition, then determine whether there is a data modification execution with different structured query language conditions but with a covering relationship;
[0045] If there are data modification executions with different structured query language conditions but with covering relations, it is determined that it involves multiple modifications to the same field data;
[0046] If there is no data modification execution with different structured query language conditions but with covering relations, it is determined that multiple modifications to the same field data are not involved.
[0047] On the other hand, in the data lake lock-free writing method provided by the present invention, sending the data insertion queue and the data modification queue to the submission queue includes:
[0048] Periodically scan the heads of the data insertion queue and the data modification queue;
[0049] According to the scanning result, the insertion operations and modification operations marked as processed in the data insertion queue and the data modification queue are continuously taken out from the head of the queue in the order of timestamps;
[0050] The retrieved insert operations and modify operations are placed in the commit queue.
[0051] On the other hand, in the data lake lock-free writing method provided by the present invention, the submission queue is used to replay the modification operation based on each insertion operation in the order of timestamps, generate and submit the format data of the copy-on-write table or the merge-on-read table, including:
[0052] sorting the insertion operations in the data insertion queue and the modification operations in the data modification queue according to timestamps in the submission queue, and replaying the modification operations based on each insertion operation during the sorting process;
[0053] Taking out each data operation from the head of the sorted submission queue in turn for processing, and generating format data of a copy-on-write table or a merge-on-read table;
[0054] After processing is completed, take out the next data operation;
[0055] If the timestamp corresponding to the next data operation retrieved is later than the timestamp corresponding to the previous data operation, it is determined that the previous write operation has been processed and marked as submitted.
[0056] On the other hand, in the data lake lock-free writing method provided by the present invention, each data operation is sequentially taken out from the head of the sorted submission queue for processing, and format data of a copy-on-write table or a merge-on-read table is generated, including:
[0057] Determine whether each data operation taken from the head of the sorted submission queue is an insert operation;
[0058] If it is an insert operation, it is determined that there is no need to process the data file in the table corresponding to the insert operation;
[0059] If it is a modification operation, the corresponding file conversion is completed according to the type of table to generate format data of a copy-on-write table or a merge-on-read table.
[0060] On the other hand, in the data lake lock-free writing method provided by the present invention, the corresponding file conversion is completed according to the type of the table to generate the format data of the copy-on-write table or the merge-on-read table, including:
[0061] If the table type is a copy-on-write table, the modification operation is merged with the submitted data file storing the target data to generate a new data file;
[0062] If the table type is a read-time merge table, the modification operation is converted into a log file corresponding to the submitted data file storing the target data.
[0063] On the other hand, in the data lake lock-free writing method provided by the present invention, it also includes:
[0064] Compare the target data file list with the modification operations in the data modification queue and the submission queue respectively;
[0065] If the comparison results involve the same file data, the execution of the table service is postponed;
[0066] If the comparison result involves different file data, the target data file list and write operation are performed.
[0067] On the other hand, in the data lake lock-free writing method provided by the present invention, it also includes:
[0068] In the submission queue, if there is a pending modification operation and there is a running table service at the same time, then check the data files involved in the execution plan of the table service;
[0069] If the pending modification operations in the submission queue conflict with the data files involved in the execution plan of the table service, the pending modification operations in the submission queue are processed after the table service is finished running.
[0070] On the other hand, in the data lake lock-free writing method provided by the present invention, before splitting each writing operation of each writing end into an insert operation and a modify operation, it also includes:
[0071] Determine whether the write operation specifies a timestamp;
[0072] If a timestamp is specified during the write operation, the specified timestamp shall prevail;
[0073] If no timestamp is specified during a write operation, the server time when the write operation starts shall prevail;
[0074] The insert operation and modify operation split from the same write operation have the same timestamp.
[0075] In order to solve the above technical problems, the present invention also provides a data lake lock-free writing device, the device comprising:
[0076] A task receiving module, used for receiving a writing task;
[0077] An operation splitting module is used to split each write operation of each write end into an insert operation and a modify operation, and put the insert operation into a data insert queue and the modify operation into a data modify queue;
[0078] A data insertion module, used to use the data insertion queue to sort all insertion operations according to timestamps and save the results of the newly added data;
[0079] A data modification module, used to use the data modification queue to sort all modification operations according to timestamps and save the data modification process;
[0080] A data submission module is used to send the data insertion queue and the data modification queue to a submission queue, and use the submission queue to replay the modification operation based on each insertion operation in timestamp order, generate the format data of the copy-on-write table or the merge-on-read table and submit it to complete the write task.
[0081] In order to solve the above technical problems, the present invention further provides a data lake lock-free writing device, the device comprising:
[0082] Memory for storing computer programs;
[0083] A processor is used to implement the steps of the above-mentioned data lake lock-free writing method when executing the computer program.
[0084] In order to solve the above technical problem, the present invention also provides a non-volatile storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above data lake lock-free writing method are implemented.
[0085] In order to solve the above technical problems, the present invention also provides a computer program product, including a computer program / instruction, which implements the steps of the above data lake lock-free writing method when executed by a processor.
[0086] It can be seen from the above technical scheme that the present invention provides a data lake lock-free writing method, which includes: receiving a write task; splitting each write operation of each write end into an insert operation and a modification operation, and putting the insert operation into a data insertion queue, and putting the modification operation into a data modification queue; using the data insertion queue to sort all insert operations according to timestamps, and save the results of the newly added data; using the data modification queue to sort all modification operations according to timestamps, and save the data modification process; sending the data insertion queue and the data modification queue to the submission queue, using the submission queue to replay the modification operation based on each insertion operation in timestamp order, generate the format data of the write-time copy table or the read-time merge table and submit it to complete the write task.
[0087] The beneficial effect of the present invention lies in that the data lake lock-free writing method provided by the present invention designs three queues, namely, a data insertion queue, a data modification queue and a submission queue; each write operation of each write end is split into an insert operation and a modification operation, and the data insertion queue and the data modification queue are used to separate the insert operation and the modification operation. The data insertion queue and the data modification queue both sort the corresponding operations put in according to the timestamp and save the corresponding data, so that the current operation does not need to wait for the previous operation to end and be successfully submitted; then the submission queue can replay the modification operation based on the data inserted each time in the order of the timestamp, and finally generate and submit the format data of the copy-on-write table or the merge-on-read table to complete the writing task, so that the entire writing process is in a lock-free state, that is, the data lake lock-free writing method of the present invention supports lock-free writing of data, and uses the data insertion queue, the data modification queue and the submission queue to insert, modify and submit data accordingly according to the results of the timestamp sorting, thereby improving the data insertion capability and the execution efficiency of the modification operation, thereby improving the performance of the writing scenario, reducing the writing delay, and improving the processing speed of the writing task.
[0088] In addition, the present invention also provides a corresponding data lake lock-free writing device, a data lake lock-free writing equipment, a non-volatile storage medium and a product for the data lake lock-free writing method, which have the same or corresponding technical features as the above-mentioned data lake lock-free writing method and have the same effect as above. BRIEF DESCRIPTION OF THE DRAWINGS
[0089] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0090] Figure 1 A flowchart of a data lake lock-free writing method provided by an embodiment of the present invention;
[0091] Figure 2 An interactive relationship diagram corresponding to the data lake lock-free writing method provided by an embodiment of the present invention;
[0092] Figure 3 A schematic diagram of queue merging provided in an embodiment of the present invention;
[0093] Figure 4 A schematic diagram of the structure of a data lake lock-free writing device provided by an embodiment of the present invention;
[0094] Figure 5 A schematic diagram of the structure of a data lake lock-free writing device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0095] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0096] The core of the present invention is to provide a data lake lock-free writing method, device, equipment, medium and product to solve the technical problem that the performance and write delay of the writing scenario are affected by the configuration of optimistic locks.
[0097] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Figure 1 A flowchart of a data lake lock-free writing method provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method includes:
[0098] S101, receiving a write task.
[0099] It should be noted that the writing tasks received by the present invention can be of various types, such as serial writing tasks, concurrent writing tasks, etc., wherein serial writing means that only one process or thread performs a writing operation on a file within a time period; parallel writing means that multiple processes or threads perform a writing operation on a file at the same time within the same time period, that is, multiple writing operations can be performed simultaneously, and multiple processes or threads share the same data storage device. The tasks to be performed by the present invention can be serial writing tasks, concurrent writing tasks, and of course other types of tasks, which are not limited here.
[0100] A data lake is a centralized repository that allows for storage of all structured and unstructured data from multiple sources at any scale. The data lake can store and manage data in the original format of the data source without any structural processing of the data. When the downstream extracts data from the data lake, it can be structured according to demand and run different types of analysis to process the data.
[0101] The present invention can be applied to a data lake platform that can manage large analytical data sets. The main purpose is to effectively reduce data latency during the ingestion process and improve the efficiency of lock-free writing of data lakes on large-scale data sets, especially in scenarios where frequent data insertion (Insert), modification (Update) and deletion (Delete) operations are required. The data lake platform aims to enable the modification and streaming reading and consumption of data on distributed storage.
[0102] S102, splitting each write operation of each write end into an insert operation and a modify operation, and putting the insert operation into a data insert queue, and putting the modify operation into a data modify queue.
[0103] In implementation, after receiving the write task, step S102 is executed. In order to achieve conflict-free concurrent write support, the present invention further subdivides each write operation of each write end into data insertion (insert) operation and modification operation; wherein the modification operation may include update (update) operation and delete operation (delete).
[0104] Figure 2 The interactive relationship diagram corresponding to the data lake lock-free writing method provided by the embodiment of the present invention. Figure 2 As shown, the present invention mainly improves the relationship between the write end and the storage layer, which is equivalent to adding a processing module between the write end and the storage layer to take over the operation of the write end. In addition, after the present invention splits each write operation of the write end into an insert operation and a modify operation, these insert operations and modify operations can be placed in two different queues, which are respectively called data insert queues and data modify queues, that is, the data insert queue and the data modify queue are used to separate the insert operation and the modify operation, and the insert operation is placed in the data insert queue, and the modify operation is placed in the data modify queue.
[0105] In the present invention, the current operation does not need to wait for the previous operation to end and be successfully submitted, but can directly perform its own operation. The present invention can allow the insertion operation and data modification operation of batch writing data to be performed simultaneously, that is, the insertion operation and modification operation split from the same writing operation can have the same timestamp, which further ensures the operation efficiency and accuracy of writing.
[0106] S103, using the data insertion queue to sort all the insertion operations according to the timestamps, and save the results of the newly added data; using the data modification queue to sort all the modification operations according to the timestamps, and save the data modification process.
[0107] It should be noted that the corresponding operations in the data insertion queue and the data modification queue of the present invention are sorted according to the timestamp of the write operation. The purpose of sorting according to the timestamp is to keep the order of the corresponding write operations of these data. If there is duplicate data, the data written later can overwrite the data written earlier. The data insertion queue is to sort all the insertion operations according to the timestamp, without considering whether the insertion operation is completed, and the insertion operation in the data insertion queue corresponds to the writing of new data files. Each operation corresponds to one or more new data files (the data lake limits the size of a single file, and a single large batch write file will be split), and the new data file can be saved in the data insertion queue. The data modification queue also sorts all the modification operations according to the timestamp, and the modification operations in the data modification queue can store the process of data modification, such as the modification conditions, the modified fields and the modified values. In summary, the data insertion queue stores the results of the newly added data and supports lock-free insertion operations; the data modification queue stores the process of data modification and supports lock-free modification operations.
[0108] S104: Send the data insertion queue and the data modification queue to the submission queue, and use the submission queue to replay the modification operation based on each insertion operation in the order of timestamps, generate the format data of the copy-on-write table or the merge-on-read table, and submit it to complete the write task.
[0109] It is understandable that a Copy On Writ (COW) table creates a new copy of the file for each write operation and overwrites the original table instead of modifying the original file directly. When the data needs to be modified, it writes the new record to a new file instead of overwriting the old record. This ensures the immutability of the data and is very beneficial for snapshot queries because the data at each point in time is static. A Copy On Write table does not require any modifications to existing files when writing. Since a new file needs to be created for each modification, disk space usage may be high and inefficient for frequent modification and deletion operations. The main advantages of this type of table include reduced memory usage, improved write performance, and scalability. The new Copy On Write table contains records from the old version of the file as well as records from the newly incoming batch, thereby achieving full up-to-date data.
[0110] Merge On Read (MOR) tables append changed data to existing files in the form of log files when writing, rather than creating new files. When reading, the data lake merges these incremental data to provide a consistent view. Merge On Read tables are more efficient when handling large numbers of modifications and deletions because they reduce file creation and rewriting. Compared to copy-on-write tables, merge-on-read tables may be slightly slower when reading because additional merge operations are required when reading. Both copy-on-write and merge-on-read tables use primary keys to distinguish duplicate records to enable modification and deletion operations.
[0111] In implementation, after receiving the relevant data from the data insertion queue and the data modification queue, the submission queue of the present invention can replay the modification operation according to the data inserted each time in the order of timestamps, and finally generate the copy-on-write table format data or the merge-on-read table format data, and submit the modification to complete the entire writing process.
[0112] In the data lake lock-free writing method provided by the embodiment of the present invention, three queues are designed, namely, a data insertion queue, a data modification queue and a submission queue; each write operation of each write end is split into an insert operation and a modification operation, and the data insertion queue and the data modification queue are used to separate the insert operation and the modification operation. The data insertion queue and the data modification queue both sort the corresponding operations put in according to the timestamp and save the corresponding data, so that the current operation does not need to wait for the previous operation to end and be successfully submitted; then the submission queue can replay the modification operation based on the data inserted each time in the order of the timestamp, and finally generate and submit the format data of the copy-on-write table or the merge-on-read table to complete the writing task, so that the entire writing process is in a lock-free state, that is, the data lake lock-free writing method of the present invention supports lock-free writing of data, and uses the data insertion queue, the data modification queue and the submission queue to insert, modify and submit data accordingly according to the results of the timestamp sorting, thereby improving the data insertion capability and the execution efficiency of the modification operation, thereby improving the performance of the writing scenario, reducing the writing delay, and improving the processing speed of the writing task.
[0113] Furthermore, in a specific implementation, in the above-mentioned data lake lock-free writing method provided in an embodiment of the present invention, before executing step S102 to split each write operation of each write end into an insert operation and a modification operation, it can also include: determining whether a timestamp is specified for the write operation; if a timestamp is specified during the write operation, the specified timestamp shall prevail; if no timestamp is specified during the write operation, the server time when the write operation starts shall prevail.
[0114] In implementation, in order to distinguish the order of write operations, the present invention maintains a set of monotonically increasing timelines. Any number of write operations at any time must be marked with a timestamp, following the first-come-first-served principle, and the start time of the write operation shall be used as the timestamp of the write operation. Write operations with later timestamps must be based on the results after the earlier write operations are completed. The logic for determining the timestamp of a write operation is as follows: If a timestamp is specified during a write operation, the timestamp specified by the operation shall prevail; if a timestamp is not specified during a write operation, the server time when the write operation starts shall prevail.
[0115] Furthermore, in a specific implementation, in the data lake lock-free writing method provided in an embodiment of the present invention, step S103 uses a data insertion queue to save the result of the newly added data, which may specifically include: using the data insertion queue to write the data file corresponding to each insertion operation, and sorting the written data files according to the partition path and primary key value; and saving the sorted data files.
[0116] In practice, the data insertion queue corresponds to the insertion operation. In the process of executing step S103, the data insertion queue can be used to directly write the data file corresponding to each insertion operation, and the written data files are sorted according to the partition path and the primary key value (record key), and the sorted data files are saved. This ensures that the insertion operation will not be blocked due to concurrency, and the submission success cannot be marked after the writing is completed, that is, so far, the concurrent query operation is not visible to these newly written data.
[0117] The present invention mainly considers two reasons for sorting the written data files according to the partition path and the primary key value: the first point is that a concurrent writing end with a later timestamp may complete the writing before a concurrent writing end with an earlier timestamp. Although concurrent writing with a later timestamp is allowed to succeed, it does not mean that the data is immediately visible, because the insert operation with a later time is based on the result of the insert data operation with an earlier time. The second point is that the data involved in different insert data operations may have potential conflicts, even if the probability is small (there are multiple writing ends writing the same data at the same time, that is, the partition path and primary key value are the same but some field values are different).
[0118] In practical applications, the partition path and primary key value can be used to determine whether two pieces of data are the same data, so as to maintain the uniqueness of the partition path and the primary key value. The purpose of the present invention to sort the operations of inserting data into the queue according to the timestamp is to preserve the order of the insertion operations corresponding to these data, and ensure that if there is duplicate data (that is, the partition path and the primary key value are the same), the data written later can overwrite the data written earlier.
[0119] Furthermore, in a specific implementation, the data lake lock-free writing method provided in the embodiment of the present invention may also include: when there are two adjacent insert operations in the data insertion queue and the writing of data is completed, determining whether there is a modification operation in the data modification queue between the adjacent insert operations; if there is a modification operation, directly marking it as processed; if there is no modification operation, determining whether the adjacent insert operations have duplicate data; if the adjacent insert operations have duplicate data, performing corresponding overwriting processing on the duplicate data or extracting it into a modification operation, and marking the adjacent insert operations as processed after processing.
[0120] In implementation, the data insertion queue of the present invention can perform corresponding overwriting processing or extract into modification operation to duplicate data to avoid the occurrence of data writing duplication, and further enhance the data insertion capability. During execution, the data insertion queue can be periodically scanned first to determine whether there are two adjacent insertion operations in the data insertion queue to complete the writing of data. Once two consecutive adjacent insertion operations are completed, and there is no data modification between the two adjacent insertion operations, it is necessary to check whether the written data is repeated, which can also be understood as checking whether the written data conflicts. If there is duplicate data, it is necessary to perform corresponding overwriting processing or extract into modification operation for the duplicate data row. If there is a modification operation between two adjacent data insertion operations, the above processing is not performed, and it is directly marked as completed. If there is no duplicate data, no operation is required.
[0121] After the inspection or processing is completed, the two adjacent inserted data can be marked as processed, regarded as data written at the same time, and put back into the data insertion queue. Continue to participate in the judgment of the above situation and repeat the above steps. It should be pointed out that the steps of checking the duplication of adjacent inserted data in the present invention can be executed asynchronously, and multiple groups of adjacent completed inserted data file checks in the queue can perform such duplicate data checks at the same time, thereby speeding up the execution speed.
[0122] Furthermore, in a specific implementation, in the data lake lock-free writing method provided in an embodiment of the present invention, determining whether adjacent insert operations have duplicate data in the above steps may specifically include: checking the partition path of the data file written by the adjacent insert operations and the range information of the primary key value field; if the partition path of the checked data file and the range information of the primary key value field overlap, it is determined that the adjacent insert operations have duplicate data.
[0123] In implementation, the present invention mainly checks the statistical information of file writes when judging whether there are duplicate data in adjacent insert operations. A write operation may involve a large amount of data, and may involve multiple data files (the data lake has file size control, and files exceeding a certain size will be split), and the written files are sorted according to the partition path and primary key value, so using the statistical information of the file can avoid scanning the entire data one by one. The statistical information of the data file contains the range information of the field value. Check the range information of the partition path and primary key value field of the data files involved in the two insertions. If it is found that the partition path and primary key value field range of a file overlap, it means that there may be duplicate data between the two insertion operations.
[0124] Furthermore, in a specific implementation, in the data lake lock-free writing method provided in an embodiment of the present invention, the above steps of checking the partition path and range information of the primary key value field of the data file written by adjacent insert operations may specifically include: traversing the data files written by adjacent insert operations; and using a binary search method to check the partition path and range information of the primary key value field of the data file.
[0125] In implementation, the present invention can traverse the content of the file for checking the range conflict. Since the data has been sorted according to the partition path and the primary key value, a binary search method can be used to accelerate the viewing of the partition path and the range information of the primary key value field of the data file. The viewing method can also be other methods, as long as the partition path and the range information of the primary key value field of the data file can be accurately viewed, which will not be elaborated here.
[0126] Further, in a specific implementation, in the data lake lock-free writing method provided by an embodiment of the present invention, if there is duplicate data in adjacent insertion operations in the above steps, the duplicate data is overwritten or extracted into a modification operation accordingly, which may specifically include: if there is duplicate data in adjacent insertion operations, the table type corresponding to the duplicate data is determined; the adjacent insertion operations include an insertion operation of a first timestamp and an insertion operation of a second timestamp; the second timestamp is later than the first timestamp; if the table type is a copy-on-write table, the data files corresponding to the first timestamp and the second timestamp are taken as a union, and the duplicate data in the data file corresponding to the first timestamp is overwritten; if the table type is a merge-on-read table, the modification of the duplicate data is extracted as a modification operation and placed in a data modification queue; wherein the modification of the duplicate data is extracted as a modification operation with the same timestamp as the second timestamp.
[0127] In implementation, if two adjacent insert operations have duplicate data (that is, conflicting data), they need to be processed by using a copy-on-write table and a merge-on-read table.
[0128] For a write-on-copy table, the data file with a later insertion time stamp (i.e., the data file corresponding to the second timestamp) and the data file with an earlier insertion time stamp (i.e., the data file corresponding to the first timestamp) are combined, and then the duplicate data in the data file corresponding to the earlier timestamp is overwritten. After this processing, a new data file is generated to replace the original data file with a later insertion time stamp.
[0129] For the read-time merge table, the modification of the duplicate data is extracted as a modification operation and put into the data modification queue for subsequent processing. At this time, the timestamp of the modification of the duplicate data extracted as the modification operation can be set to the same timestamp as the second timestamp.
[0130] Next, after extracting the modification of duplicate data as a modification operation and putting it into the data modification queue, it can also include: taking the difference between the data file corresponding to the second timestamp and the data file corresponding to the first timestamp, so that the insertion operation of the second timestamp generates a new data file.
[0131] In the implementation, the difference between the data file with a later inserted timestamp (i.e., the data file corresponding to the second timestamp) and the data file with an earlier inserted operation (i.e., the data file corresponding to the first timestamp) is first obtained, and then the data with a later inserted timestamp is taken as the difference to remove the modified data and generate a new data file. The above operation can be called duplicate data insertion to update.
[0132] Furthermore, in a specific implementation, in the above-mentioned data lake lock-free writing method provided in an embodiment of the present invention, before executing step S104 to send the data insertion queue to the submission queue, it may also include: periodically checking the head of the data insertion queue, if there is one or more consecutive insertion operations that have been marked as processed, and there is no modification operation in the data modification queue between the consecutive insertion operations, then sending the data insertion queue to the submission queue; if there is a modification operation in the data modification queue between two consecutive insertion operations, then performing a merge operation in the data insertion queue and the data modification queue.
[0133] In the implementation, during the periodic check of the head of the data insertion queue, if one or more consecutive insertion operations are found to have been marked as processed, and there is no data modification operation in the data modification queue between the consecutive insertion operations, they are marked as submitted successfully and then removed from the queue, indicating that the result is visible to the query. If there is a modification operation in the data modification queue between two consecutive insertion operations, the next step is to merge the data insertion queue and the data modification queue.
[0134] The above processing process for inserting data into the queue supports multi-end lock-free concurrent writing of data and can automatically handle conflicting / duplicate data.
[0135] Furthermore, in a specific implementation, the data lake lock-free writing method provided in the embodiment of the present invention may also include: when the same write operation fails, using the data insertion queue to remove the insertion operations split from the failed write operation, and using the data modification queue to remove the modification operations split from the failed single write operation.
[0136] In implementation, for scenarios where writes fail, it is necessary to remove the data insertion and modification operations corresponding to the failed writes from both the data insertion and modification queues to ensure that these changes are not committed, thereby ensuring the atomicity of data lake writes (a single write is either completely successful or completely failed).
[0137] Correspondingly, when the same write operation is successful, step S103 uses the data modification queue to save the data modification process, which may specifically include: using the data modification queue to record the corresponding modified execution conditions in the same write operation, the modified target field and the modified value and save them.
[0138] In implementation, the data modification queue can save all data modification operations such as update operations and delete operations, and record the detailed data modification process in the order of operation timestamps, including operation type, execution conditions and modified value of the target field.
[0139] Furthermore, in a specific implementation, the data lake lock-free writing method provided in the embodiment of the present invention may also include: determining in the data modification queue whether a single modification operation involves multiple modifications to the same field data; if multiple modifications to the same field data are involved, merging the multiple modification operations for the same field data so that the operation at the last timestamp covers the operation before the last timestamp; if multiple modifications to the same field data are not involved, the multiple modification operations for the same field data are not merged.
[0140] In implementation, the data modification queue can merge multiple modification operations for the same field data in a single modification operation, reduce the generation of intermediate result data for modification, and improve the execution efficiency of modification. Unlike the previous data insertion queue, the data modification queue needs to determine whether a single modification operation involves multiple modifications to the same field data. If so, the multiple modification operations for the same field data need to be merged, and the latter operation overwrites the former operation. Otherwise, no merging process is performed.
[0141] Further, in a specific implementation, in the above-mentioned data lake lock-free writing method provided by an embodiment of the present invention, determining whether a single modification operation involves multiple modifications to the same field data may specifically include: in a single modification operation, determining whether there are data modification executions with the same structured query language conditions; if there are data modification executions with the same structured query language conditions, determining that multiple modifications to the same field data are involved; if there are no data modification executions with the same structured query language conditions, determining whether there are data modification executions with different structured query language conditions but with a covering relationship; if there are data modification executions with different structured query language conditions but with a covering relationship, determining that multiple modifications to the same field data are involved; if there are no data modification executions with different structured query language conditions but with a covering relationship, determining that multiple modifications to the same field data are involved.
[0142] In practice, the modification process in the data modification queue can be described by using Structured Query Language (SQL) statements. To determine whether a single modification operation involves multiple modifications to the same field data, there are three situations:
[0143] The first is concurrent execution of data modifications with the same conditions. For example:
[0144] update table_a set col_a=2 where col_a=0;
[0145] The above expression is an SQL statement, which means to update the value of the col_a column of the data whose value is 0 in the col_a column of the table_a to 2.
[0146] update table_a set col_a=1 where col_a=0;
[0147] The above expression is an SQL statement, which means to update the value of the col_a column of the data whose value is 0 in the col_a column of the table table_a to 1. The following expression can be used as a reference based on this meaning.
[0148] The result of the first case is easier to handle. The result is that the SQL corresponding to the later operation will overwrite the previous SQL operation. After merging, only the later SQL is retained. The update operation converted from upsert above is equivalent to update table_a set col_a='xx' where record_key='xx', which can naturally be understood in this way.
[0149] In the second case, the conditions are different, but the conditions of the SQL executed later can cover the SQL executed earlier, for example:
[0150] update table_a set col_a=2 where col_a=0and col_b=1;
[0151] update table_a set col_a=1 where col_a=0;
[0152] At this time, you can still use the later operation to overwrite the previous operation, and only the later SQL is retained after the merger.
[0153] The third situation is that the conditions are different and the covering relationship cannot be formed. For example, the following two SQL statements:
[0154] update table_a set col_a=2 where col_a=10;
[0155] update table_a set col_a=1 where col_a=0;
[0156] At this time, if we only look at the operation records in the data modification queue without checking the original data, we cannot know which SQL will overwrite the other one or both SQLs can be executed. In this case, both SQLs need to be retained and cannot be merged.
[0157] It should be noted that the specific operations between different modification operations are not merged, that is, the two modification operations in the data modification queue have different timestamps, and the detailed operations in these two modification operations are not merged. This is to preserve the modification records of the data lake (version information of the data lake table, what changes were made in each time period, if the specific operations between different modification operations are merged, the results of the earlier modification operations will be lost). Therefore, the data modification queue only merges multiple modifications to the same data field in the same modification operation.
[0158] Furthermore, in a specific implementation, in the above-mentioned data lake lock-free writing method provided in an embodiment of the present invention, sending the data insertion queue and the data modification queue to the submission queue can specifically include: periodically scanning the head of the data insertion queue and the data modification queue; according to the scanning result, continuously retrieving the insertion operations and modification operations marked as processed in the data insertion queue and the data modification queue from the head of the queue in timestamp order; and placing the retrieved insertion operations and modification operations in the submission queue.
[0159] In the implementation, the heads of the two queues, the data insertion queue and the data modification queue, are scanned periodically. The insertion and modification operations marked as processed in the two queues are continuously taken out in order from first to last in time, and placed in a new queue, which can be called the submission queue. The operations in the submission queue are still sorted in the order of timestamps.
[0160] Furthermore, in a specific implementation, in the data lake lock-free writing method provided in an embodiment of the present invention, a submission queue is used to replay modification operations based on each insertion operation in timestamp order, and format data of a copy-on-write table or a merge-on-read table is generated and submitted. Specifically, it may include: sorting the insertion operations in the data insertion queue and the modification operations in the data modification queue according to timestamps in the submission queue, and replaying the modification operations based on each insertion operation during the sorting process; taking out each data operation from the head of the sorted submission queue in turn for processing, and generating format data of a copy-on-write table or a merge-on-read table; taking out the next data operation after processing; if the timestamp corresponding to the next data operation taken out is later than the timestamp corresponding to the previous data operation, it is determined that the previous write operation has been processed and marked as submitted.
[0161] In practice, the present invention sequentially takes out each data operation from the head of the queue for processing. After processing, the next operation is taken out. If the timestamp corresponding to the next operation taken out is later than the previous one, it means that the previous write operation has been processed and marked as submitted, and the result is visible to the query. For each data operation taken out, the next step is to generate data that meets the table format requirements based on it.
[0162] Figure 3 A schematic diagram of queue merging provided in an embodiment of the present invention. Figure 3 As shown, the insert operations in the data insert queue are sorted by timestamp as insert A, insert B, insert C, insert D, and insert E, and the modification operations in the data modification queue are sorted by timestamp as modification 1, modification 2, and modification 3. The submission queue sorts the insert operations in the data insert queue and the modification operations in the data modification queue by timestamp, which are insert A, modification 1, insert B, insert C, insert D, modification 2, insert E, and modification 3.
[0163] Furthermore, in a specific implementation, in the data lake lock-free writing method provided in an embodiment of the present invention, each data operation is sequentially taken out from the head of the sorted submission queue for processing to generate format data of a copy-on-write table or a merge-on-read table, which may specifically include: determining whether each data operation taken out from the head of the sorted submission queue is an insert operation; if it is an insert operation, determining that there is no need to process the data file in the table corresponding to the insert operation; if it is a modification operation, completing the corresponding file conversion according to the type of the table to generate format data of the copy-on-write table or the merge-on-read table.
[0164] In implementation, no matter it is a copy-on-write table or a merge-on-read table, the data files in the corresponding table for data insertion operations do not need to be processed. Therefore, when processing each data operation, first determine whether each data operation is an insert operation; if so, no processing is required; if not, the data operation is determined to be a modification operation, and the corresponding file conversion needs to be completed according to the type of table to generate the format data of the copy-on-write table or the merge-on-read table.
[0165] Furthermore, in a specific implementation, in the data lake lock-free writing method provided in an embodiment of the present invention, the above steps complete corresponding file conversion according to the type of the table to generate format data of a copy-on-write table or a merge-on-read table, which may specifically include: if the type of the table is a copy-on-write table, merging the modification operation with the submitted data file storing the target data to generate a new data file; if the type of the table is a merge-on-read table, converting the modification operation into a log file corresponding to the submitted data file storing the target data.
[0166] In practice, for a copy-on-write table, the data modification operation needs to be merged with the previously submitted data file storing the target data, generating a new data file and replacing it. For a merge-on-read table, the data modification operation is to generate a log file corresponding to the submitted data file storing the target data. This operation ensures that after the data is finally submitted, it still meets the characteristics and storage format requirements of the copy-on-write table or the merge-on-read table.
[0167] Furthermore, in a specific implementation, the data lake lock-free writing method provided in the embodiment of the present invention may also include: comparing the target data file list with the modification operations in the data modification queue and the submission queue respectively; if the comparison result involves the same file data, postponing the execution of the table service; if the comparison result involves different file data, executing the target data file list and the write operation.
[0168] In practice, the table service is to maintain the availability and efficiency of the data lake table. It is a periodically run service to maintain table data and storage structure, such as compaction, clustering, and cleaning.
[0169] The table service of the data lake is accompanied by a planning phase, during which the data lake generates an execution plan. Based on the generated execution plan, you can get a list of target data files that need to be operated. Compare the target data file list with the data modification queue and the data modification operations in the submission queue. If the same data files are involved, postpone the execution of the table service and make the above judgment again after a period of time. If the data files involved are different, the target data file list can be executed concurrently with the write operation.
[0170] Furthermore, in a specific implementation, the data lake lock-free writing method provided in the embodiment of the present invention may further include: in the submission queue, if there is a pending modification operation and there is a running table service at the same time, checking the data files involved in the execution plan of the table service; if there is a conflict between the pending modification operation in the submission queue and the data files involved in the execution plan of the table service, waiting for the table service to run to be completed, and then processing the pending modification operation in the submission queue.
[0171] In implementation, if there are data modification operations that need to be processed in the submission queue, and if there is a table service running at the same time, it is necessary to check the data files involved in the execution plan of the table service. If there is a conflict, it is necessary to wait until the table service is finished running before processing the data modification operations in the submission queue.
[0172] In the above embodiments, the data lake lock-free writing method is described in detail, and the present invention also provides a data lake lock-free writing device and a data lake lock-free writing device corresponding to the embodiments. It should be noted that the present invention describes the embodiments of the device part from two perspectives, one is based on the functional module perspective, and the other is based on the hardware perspective.
[0173] Figure 4 This is a structural diagram of a data lake lock-free writing device provided in an embodiment of the present invention. This embodiment is based on the perspective of functional modules, and the device includes:
[0174] A task receiving module 10, used for receiving a writing task;
[0175] An operation splitting module 11 is used to split each write operation of each write end into an insert operation and a modify operation, and put the insert operation into a data insert queue and put the modify operation into a data modify queue;
[0176] The data insertion module 12 is used to sort all the insertion operations according to the timestamps using the data insertion queue and save the results of the newly added data;
[0177] The data modification module 13 is used to sort all modification operations according to timestamps using a data modification queue and save the data modification process;
[0178] The data submission module 14 is used to send the data insertion queue and the data modification queue to the submission queue, and use the submission queue to replay the modification operation based on each insertion operation in the order of timestamps, generate the format data of the write-time copy table or the read-time merge table and submit it to complete the write task.
[0179] In the above-mentioned data lake lock-free writing device provided by the embodiment of the present invention, each writing operation of each writing end can be split into an insertion operation and a modification operation through the interaction of the above-mentioned five modules, and the insertion operation and the modification operation are separated by using the data insertion queue and the data modification queue. The data insertion queue and the data modification queue both sort the corresponding operations put in according to the timestamp and save the corresponding data, so that the current operation does not need to wait for the previous operation to end and be successfully submitted; the submission queue is used to replay the modification operation based on the data inserted each time in the order of timestamps, and finally generate and submit the format data of the copy-on-write table or the merge-on-read table to complete the writing task. In this way, the entire writing process is in a lock-free state, that is, it supports lock-free writing of data, and uses the data insertion queue, the data modification queue and the submission queue to insert, modify and submit data accordingly according to the results of timestamp sorting, thereby improving the data insertion capability and the execution efficiency of the modification operation, thereby improving the performance of the writing scene, reducing the writing delay, and increasing the processing speed of the writing task.
[0180] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, and will not be repeated here. And it has the same beneficial effects as the above-mentioned data lake lock-free writing method.
[0181] Furthermore, in a specific implementation, the data lake lock-free writing device provided in the embodiment of the present invention may further include:
[0182] The timestamp setting module is used to determine whether the write operation specifies a timestamp; if the timestamp is specified during the write operation, the specified timestamp shall prevail; if the timestamp is not specified during the write operation, the server time when the write operation starts shall prevail.
[0183] Furthermore, in a specific implementation, in the above-mentioned data lake lock-free writing device provided in an embodiment of the present invention, the data insertion module 12 can be specifically used to use the data insertion queue to write the data file corresponding to each insertion operation, and sort the written data files according to the partition path and primary key value; and save the sorted data files.
[0184] The data insertion module 12 can also be specifically used to determine whether there is a modification operation in the data modification queue between the adjacent insertion operations when two adjacent insertion operations have completed writing data in the data insertion queue; if there is a modification operation, it is directly marked as processed; if there is no modification operation, it is determined whether the adjacent insertion operations have duplicate data; if the adjacent insertion operations have duplicate data, the duplicate data is overwritten accordingly or extracted into a modification operation, and after processing, the adjacent insertion operations are marked as processed.
[0185] Among them, judging whether there are duplicate data in adjacent insert operations may include: checking the partition path of the data file written by the adjacent insert operations and the range information of the primary key value field; if the partition path of the viewed data file and the range information of the primary key value field overlap, it is determined that there are duplicate data in the adjacent insert operations.
[0186] If there are duplicate data in adjacent insertion operations, the duplicate data is overwritten or extracted into a modification operation accordingly, which may include: if there are duplicate data in adjacent insertion operations, the table type corresponding to the duplicate data is determined; the adjacent insertion operations include an insert operation of a first timestamp and an insert operation of a second timestamp; the second timestamp is later than the first timestamp; if the table type is a copy-on-write table, the data files corresponding to the first timestamp and the second timestamp are taken as a union, and the duplicate data in the data file corresponding to the first timestamp is overwritten; if the table type is a merge-on-read table, the modification of the duplicate data is extracted as a modification operation and put into a data modification queue; wherein the modification of the duplicate data is extracted as a modification operation with the timestamp being the same as the second timestamp; the data file corresponding to the second timestamp is taken as the difference between the data file corresponding to the second timestamp and the data file corresponding to the first timestamp, so that the insertion operation of the second timestamp generates a new data file.
[0187] Furthermore, in a specific implementation, the data lake lock-free writing device provided in the embodiment of the present invention may further include:
[0188] The queue processing module can be used to periodically check the head of the data insertion queue. If there is one or more consecutive insertion operations that have been marked as processed, and there is no modification operation in the data modification queue between the consecutive insertion operations, the data insertion queue will be sent to the submission queue; if there is a modification operation in the data modification queue between two consecutive insertion operations, a merge operation will be performed in the data insertion queue and the data modification queue; it can also be used to periodically scan the heads of the data insertion queue and the data modification queue; according to the scanning results, the insertion operations and modification operations marked as processed in the data insertion queue and the data modification queue are continuously taken out from the heads of the queues in the order of timestamps; and the taken out insertion operations and modification operations are placed in the submission queue.
[0189] Furthermore, in a specific implementation, in the data lake lock-free writing device provided in the embodiment of the present invention, the data insertion module 12 can also be used to remove the insertion operation split from the failed write operation when the same write operation fails. Correspondingly, the data modification module 13 can also be used to remove the modification operation split from the failed single write operation when the same write operation fails.
[0190] Furthermore, in a specific implementation, in the above-mentioned data lake lock-free writing device provided in an embodiment of the present invention, the data modification module 13 can also be used to determine in the data modification queue whether a single modification operation involves multiple modifications to the same field data; if it involves multiple modifications to the same field data, the multiple modification operations for the same field data are merged so that the operation at the last timestamp covers the operation before the last timestamp; if it does not involve multiple modifications to the same field data, the multiple modification operations for the same field data are not merged.
[0191] Among them, judging whether a single modification operation involves multiple modifications to the same field data may include: in a single modification operation, judging whether there are data modification executions with the same structured query language conditions; if there are data modification executions with the same structured query language conditions, judging that multiple modifications to the same field data are involved; if there are no data modification executions with the same structured query language conditions, judging whether there are data modification executions with different structured query language conditions but with a covering relationship; if there are data modification executions with different structured query language conditions but with a covering relationship, judging that multiple modifications to the same field data are involved; if there are no data modification executions with different structured query language conditions but with a covering relationship, judging that multiple modifications to the same field data are involved.
[0192] Furthermore, in a specific implementation, in the above-mentioned data lake lock-free writing device provided in an embodiment of the present invention, the data submission module 14 can be specifically used to sort the insertion operations in the data insertion queue and the modification operations in the data modification queue according to the timestamps in the submission queue, and replay the modification operations based on each insertion operation during the sorting process; take out each data operation from the head of the sorted submission queue in turn for processing, and generate the format data of the copy-on-write table or the merge-on-read table; take out the next data operation after processing; if the timestamp corresponding to the next data operation taken out is later than the timestamp corresponding to the previous data operation, it is determined that the previous write operation has been processed and marked as submitted.
[0193] Furthermore, in a specific implementation, the data lake lock-free writing device provided in the embodiment of the present invention may further include:
[0194] The table service processing module can be used to compare the target data file list with the modification operations in the data modification queue and the submission queue respectively; if the comparison result involves the same file data, the execution of the table service is postponed; if the comparison result involves different file data, the target data file list is executed with the write operation; it can also be used in the submission queue, if there is a pending modification operation and there is a running table service at the same time, then check the data files involved in the execution plan of the table service; if the pending modification operation in the submission queue conflicts with the data files involved in the execution plan of the table service, wait until the table service is finished running, and then process the pending modification operation in the submission queue.
[0195] Figure 5 This is a structural diagram of the data lake lock-free writing device provided by an embodiment of the present invention. This embodiment is based on the hardware perspective, such as Figure 5 As shown, the data lake lock-free writing devices include:
[0196] A memory 20, used for storing computer programs;
[0197] The processor 21 is configured to implement the steps of the data lake lock-free writing method mentioned in the above embodiment when executing a computer program.
[0198] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of a digital signal processor (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU; the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.
[0199] The memory 20 may include one or more non-volatile storage media, which may be non-transitory. The memory 20 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 20 is at least used to store the following computer program 201, wherein, after the computer program is loaded and executed by the processor 21, it can implement the relevant steps of the data lake lock-free writing method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 202 may include Windows, Unix, Linux, etc. Data 203 may include, but is not limited to, the data involved in the above-mentioned data lake lock-free writing method, etc.
[0200] In some embodiments, the data lake lock-free writing device may also include a display screen 22, an input and output interface 23, a communication interface 24, a power supply 25, and a communication bus 26. Those skilled in the art will appreciate that Figure 5 The structure shown in does not constitute a limitation on the data lake lock-free writing device, and may include more or fewer components than shown. The data lake lock-free writing device provided by an embodiment of the present invention includes a memory and a processor. When the processor executes a program stored in the memory, it can implement the following method: a data lake lock-free writing method, the effect is the same as above.
[0201] Finally, the present invention also provides an embodiment corresponding to a non-volatile storage medium. The non-volatile storage medium stores a computer program, and when the computer program is executed by a processor, the steps recorded in the above method embodiment are implemented.
[0202] It is understandable that if the method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and executes all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc. Various media that can store program code. The non-volatile storage medium provided by the present invention can realize the above-mentioned data lake lock-free writing method, and the effect is the same as above.
[0203] Finally, the present invention also provides an embodiment corresponding to a computer program product. The computer program product includes a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps recorded in the above-mentioned data lake lock-free writing method embodiment are implemented. The computer program product provided by the present invention can implement the above-mentioned data lake lock-free writing method, and the effect is the same as above.
[0204] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.
[0205] The above is a detailed introduction to the data lake lock-free writing method, device, equipment, medium and product provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can refer to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can refer to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified in several ways, and these improvements and modifications also fall within the scope of protection of the present invention.
Claims
1. A lock-free writing method for a data lake, characterized in that: The method comprises: Receive write tasks; Split each write operation of each write end into an insert operation and a modify operation, and put the insert operation into a data insert queue, and put the modify operation into a data modify queue; Using the data insertion queue, all insertion operations are sorted according to timestamps, and the results of newly added data are saved; using the data modification queue, all modification operations are sorted according to timestamps, and the process of data modification is saved; The data insertion queue and the data modification queue are sent to a submission queue, and the insertion operations in the data insertion queue and the modification operations in the data modification queue are sorted in the submission queue according to timestamps, and the modification operations are replayed based on each insertion operation during the sorting process; each data operation is taken out from the head of the sorted submission queue in turn for processing, and format data of a copy-on-write table or a merge-on-read table is generated; after the processing is completed, the next data operation is taken out; if the timestamp corresponding to the next data operation taken out is later than the timestamp corresponding to the previous data operation, it is determined that the previous write operation has been processed and marked as submitted to complete the write task.
2. The data lake lock-free writing method according to claim 1 is characterized in that: The result of using the data to insert the queue to save the newly added data includes: Using the data insertion queue to write the data files corresponding to each insertion operation, and sorting the written data files according to the partition path and the primary key value; Save the sorted data file.
3. The data lake lock-free writing method according to claim 1 is characterized in that: Also includes: When there are two adjacent insert operations in the data insert queue and data writing is completed, determining whether there is a modification operation of the data modification queue between the adjacent insert operations; If there is a modification operation, it will be directly marked as processed; If there is no modification operation, determine whether there is duplicate data in adjacent insertion operations; If there are duplicate data in adjacent insertion operations, the duplicate data will be overwritten or extracted into a modification operation accordingly. After the processing is completed, the adjacent insertion operation will be marked as completed.
4. The data lake lock-free writing method according to claim 3 is characterized in that: Determine whether there are duplicate data in adjacent insert operations, including: View the partition path and primary key value field range information of the data files written by adjacent insert operations; If the partition path of the data file being viewed overlaps with the range information of the primary key value field, it is determined that duplicate data exists in adjacent insert operations.
5. The data lake lock-free writing method according to claim 4 is characterized in that: View the partition path and primary key value field range information of the data files written by adjacent insert operations, including: Traverse the data files written by adjacent insert operations; Use binary search to view the partition path of the data file and the range information of the primary key value field.
6. The data lake lock-free writing method according to claim 3 is characterized in that: If there are duplicate data in adjacent insert operations, the duplicate data is overwritten or extracted into a modification operation, including: If there is duplicate data in adjacent insertion operations, determine the table type corresponding to the duplicate data; the adjacent insertion operations include an insertion operation of a first timestamp and an insertion operation of a second timestamp; the second timestamp is later than the first timestamp; If the table type is a copy-on-write table, taking a union of the data files corresponding to the first timestamp and the second timestamp, and overwriting duplicate data in the data file corresponding to the first timestamp; If the table type is a read-time merge table, the modification of the duplicate data is extracted as a modification operation and put into the data modification queue; wherein the timestamp of the modification of the duplicate data extracted as the modification operation is the same as the second timestamp.
7. The data lake lock-free writing method according to claim 6, characterized in that: After extracting the modification of duplicate data as a modification operation and putting it into the data modification queue, the method further includes: The data file corresponding to the second timestamp is taken as a difference set between the data file corresponding to the second timestamp and the data file corresponding to the first timestamp, so that the insertion operation of the second timestamp generates a new data file.
8. The data lake lock-free writing method according to claim 1, characterized in that: Before inserting the data into the queue and sending it to the submission queue, the method further includes: Periodically checking the head of the data insertion queue, if there is one or more consecutive insertion operations that have been marked as processed, and there is no modification operation in the data modification queue between the consecutive insertion operations, then sending the data insertion queue to the submission queue; If there is a modification operation of the data modification queue corresponding to two consecutive insertion operations, a merge operation is performed in the data insertion queue and the data modification queue.
9. The data lake lock-free writing method according to claim 1, characterized in that: Also includes: When the same write operation fails, the data insertion queue is used to remove the insert operations split from the failed write operation, and the data modification queue is used to remove the modification operations split from the failed single write operation; Correspondingly, when the same write operation is successful, the process of saving the data modification by using the data modification queue includes: The data modification queue is used to record the execution conditions of the modification corresponding to the same write operation, the modified target field and the modified value and save them.
10. The data lake lock-free writing method according to claim 1, characterized in that: Also includes: Determining in the data modification queue whether a single modification operation involves multiple modifications to the same field data; If multiple modifications are made to the same field data, the multiple modification operations for the same field data are merged so that the operation at the latest timestamp covers the operation before the latest timestamp; If multiple modifications to the same field data are not involved, the multiple modifications to the same field data will not be merged.
11. The data lake lock-free writing method according to claim 10, characterized in that: Determine whether a single modification operation involves multiple modifications to the same field data, including: In a single modification operation, determine whether there is a data modification execution with the same structured query language conditions; If there are data modifications with the same structured query language conditions, it is determined that multiple modifications to the same field data are involved; If there is no data modification execution with the same structured query language condition, then determine whether there is a data modification execution with different structured query language conditions but with a covering relationship; If there are data modification executions with different structured query language conditions but with covering relations, it is determined that it involves multiple modifications to the same field data; If there is no data modification execution with different structured query language conditions but with covering relations, it is determined that multiple modifications to the same field data are not involved.
12. The data lake lock-free writing method according to claim 1, characterized in that: Sending the data insertion queue and the data modification queue to a submission queue includes: Periodically scan the heads of the data insertion queue and the data modification queue; According to the scanning result, the insertion operations and modification operations marked as processed in the data insertion queue and the data modification queue are continuously taken out from the head of the queue in the order of timestamps; The retrieved insert operations and modify operations are placed in the commit queue.
13. The data lake lock-free writing method according to claim 1, characterized in that: Each data operation is sequentially taken out from the head of the sorted submission queue for processing, and format data of a copy-on-write table or a merge-on-read table is generated, including: Determine whether each data operation taken from the head of the sorted submission queue is an insert operation; If it is an insert operation, it is determined that there is no need to process the data file in the table corresponding to the insert operation; If it is a modification operation, the corresponding file conversion is completed according to the type of table to generate format data of a copy-on-write table or a merge-on-read table.
14. The data lake lock-free writing method according to claim 13, characterized in that: According to the table type, complete the corresponding file conversion to generate the format data of the write-time copy table or the read-time merge table, including: If the table type is a copy-on-write table, the modification operation is merged with the submitted data file storing the target data to generate a new data file; If the table type is a read-time merge table, the modification operation is converted into a log file corresponding to the submitted data file storing the target data.
15. The data lake lock-free writing method according to claim 1, characterized in that: Also includes: Compare the target data file list with the modification operations in the data modification queue and the submission queue respectively; If the comparison results involve the same file data, the execution of the table service is postponed; If the comparison result involves different file data, the target data file list and write operation are performed.
16. The data lake lock-free writing method according to claim 1, characterized in that: Also includes: In the submission queue, if there is a pending modification operation and there is a running table service at the same time, then check the data files involved in the execution plan of the table service; If the pending modification operations in the submission queue conflict with the data files involved in the execution plan of the table service, the pending modification operations in the submission queue are processed after the table service is finished running.
17. The data lake lock-free writing method according to claim 1, characterized in that: Before each write operation of each write end is split into an insert operation and a modify operation, it also includes: Determine whether the write operation specifies a timestamp; If a timestamp is specified during the write operation, the specified timestamp shall prevail; If no timestamp is specified during a write operation, the server time when the write operation starts shall prevail; The insert operation and modify operation split from the same write operation have the same timestamp.
18. A data lake lock-free writing device, characterized in that: The device comprises: A task receiving module, used for receiving a writing task; An operation splitting module is used to split each write operation of each write end into an insert operation and a modify operation, and put the insert operation into a data insert queue and the modify operation into a data modify queue; A data insertion module, used to use the data insertion queue to sort all insertion operations according to timestamps and save the results of the newly added data; A data modification module, used to use the data modification queue to sort all modification operations according to timestamps and save the data modification process; A data submission module is used to send the data insertion queue and the data modification queue to a submission queue, sort the insertion operations in the data insertion queue and the modification operations in the data modification queue according to timestamps in the submission queue, and replay the modification operations based on each insertion operation during the sorting process; take out each data operation from the head of the sorted submission queue in turn for processing, and generate format data of a copy-on-write table or a merge-on-read table; take out the next data operation after processing; if the timestamp corresponding to the next data operation taken out is later than the timestamp corresponding to the previous data operation, it is determined that the previous write operation has been processed and marked as submitted to complete the write task.
19. A data lake lock-free writing device, characterized in that: The device comprises: Memory for storing computer programs; A processor, configured to implement the steps of the data lake lock-free writing method according to any one of claims 1 to 17 when executing the computer program.
20. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the data lake lock-free writing method according to any one of claims 1 to 17 are implemented.
21. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the data lake lock-free writing method as described in any one of claims 1 to 17 are implemented.
