Data processing method, device and equipment based on data lake and storage medium

Abstract

The embodiment of the invention provides a data processing method and device based on a data lake, equipment and a storage medium. The data lake is stored in a disk and comprises a plurality of data tables and a log table; the log table comprises multiple pieces of change information; the method is applied to a memory of the electronic equipment. The method comprises the following steps: in response to a change operation on a data table, obtaining initial information corresponding to the change operation and to-be-updated contents in the data table, and adding the initial information into a log table in a disk; according to multiple pieces of change information of the data table in the log table, determining update content of the data table, and performing replacement operation on the update content; wherein the replacement operation is used for replacing the updated content in the memory to the to-be-updated content in the disk; and according to an operation result of the replacement operation, updating the initial information in the disk to obtain target information. The method is used for improving the accuracy of data processing in the data lake.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a data processing method, apparatus, device and storage medium based on a data lake. Background Technology

[0002] Data lakes are widely used for storing various types of data because they offer advantages such as low-cost storage of multiple data types (structured, semi-structured, and unstructured) and support for multiple data formats.

[0003] However, inconsistencies can occur when data flows between storage and computing engines in a data lake, leading to data processing errors. Therefore, improving the accuracy of data processing in a data lake has become an urgent technical problem to be solved. Summary of the Invention

[0004] This application provides data processing methods, apparatus, devices, and storage media based on data lakes, in order to improve the accuracy of data processing in data lakes.

[0005] In a first aspect, embodiments of this application provide a data processing method based on a data lake, wherein the data lake is stored on a disk and includes multiple data tables and a log table; the data tables represent business data for a preset business; the log table includes multiple change messages; the change messages represent the operation process of performing change operations on the data content in the data tables; the method is applied in the memory of an electronic device; the method includes:

[0006] In response to a change operation on a data table, the system obtains the initial information corresponding to the change operation and the content to be updated in the data table, and adds the initial information to the log table on the disk; wherein the initial information represents the change operation to be performed on the data table.

[0007] Based on multiple change records of the data table in the log table, the updated content of the data table is determined, and a replacement operation is performed on the updated content; wherein, the replacement operation is used to replace the updated content in memory with the content to be updated in the disk;

[0008] Based on the result of the replacement operation, the initial information in the disk is updated to obtain the target information; the target information represents the change information corresponding to the current change operation after the operation is completed.

[0009] In one possible implementation, the change information includes the first status information, second status information, time information, and storage path of the data table;

[0010] The first status information includes an expired status and a used status; the expired status represents data content that is no longer used; the used status represents data content that is allowed to be used.

[0011] The second status information includes an in-operation status, a commit status, and a rollback status; the in-operation status indicates that a change operation is being performed; the commit status indicates that the change operation was successfully executed; and the rollback status indicates that the change operation failed.

[0012] The time information represents the start time of the change operation;

[0013] The storage path represents the storage location of the data table in the data lake.

[0014] In one possible implementation, determining the updated content of the data table based on multiple change records in the log table includes:

[0015] Based on the first status information, second status information, and time information of each change information corresponding to the data table in the log table, the first information is filtered out from multiple change information;

[0016] Based on the storage path of the first information, the updated content of the data table is generated.

[0017] In one possible implementation, the step of filtering out the first information from multiple change information entries based on the first status information, second status information, and time information of each change information entry corresponding to the data table in the log table includes:

[0018] Based on the first status information of each change information corresponding to the data table in the log table, the first candidate information is determined; the first status information in the first candidate information is the in-use status.

[0019] Based on the second status information of each first candidate information, the second candidate information is determined; the second status information of the second candidate information is the submission status.

[0020] Based on the time information of each second candidate, the first information is selected from the second candidate information.

[0021] In one possible implementation, the change information further includes change object information and change type information; the change object information represents the data content in the data table; the change type represents the method of changing the data content in the data table; and the change operation represents the data rollback operation.

[0022] The step of determining the updated content of the data table based on multiple change records in the log table includes:

[0023] Based on the preset backtracking time, and the second status information and time information of each change information of the data table in the log table, the second information is filtered out from each change information of the data table;

[0024] Based on the changed object information and change type information in the second information, the updated content of the data table is generated.

[0025] In one possible implementation, the step of filtering out the second information from the various change information in the data table based on a preset backtracking time and the second status information and time information of each change information in the log table includes:

[0026] Based on the second status information of each change information corresponding to the data table in the log table, a third candidate information is determined; the second status information of the third candidate information is a submission status.

[0027] Based on the time information of each third candidate and the preset backtracking time, the second information is selected from each third candidate.

[0028] In one possible implementation, the first state information in the initial information is the in-use state; the second state information in the initial information is the operating state.

[0029] The step of updating the initial information in the disk based on the result of the replacement operation includes:

[0030] If the replacement operation is successful, the second status information of the initial information in the disk is updated from the "in operation" status to the "committed" status.

[0031] If the replacement operation fails, the first status information of the initial information in the disk is updated from the in-use status to the expired status, and the second status information is updated from the in-operation status to the rollback status.

[0032] In one possible implementation, the data lake further includes a permission table for recording permission information for each user; the permission information represents the user's access permissions to the data table.

[0033] The method further includes:

[0034] In response to a user's data table query operation, determine the data table to be queried and the user's permission information;

[0035] If, based on the user's permission information, it is determined that the user has access permission to the data table to be queried, then the business data corresponding to the query operation is returned from the data table to be queried.

[0036] In one possible implementation, the change operation is used to perform a synchronized change operation on multiple data tables. Each change operation corresponds to a change message upon completion, and the change message includes multiple first status messages, the number of which is consistent with the number of data tables. Returning the business data corresponding to the query operation from the data table to be queried includes:

[0037] The fourth candidate information is selected from multiple change information entries corresponding to the data table to be queried in the log table; the first status information of the data table to be queried in the fourth candidate information is the in-use status.

[0038] The fifth candidate information is selected from each of the fourth candidate information; the second state of the fifth candidate information is the submission state.

[0039] Based on the time information of each fifth candidate information, the third information is selected from each fifth candidate information; the time information of the third information is adjacent to the current time.

[0040] Based on the storage path in the third information, obtain the business data from the data table to be queried.

[0041] In one possible implementation, the method further includes:

[0042] In response to a field query command on a data table, determine the field to be queried in the data table;

[0043] Based on a preset metadata table, the storage area of ​​the field to be queried on the disk is obtained; wherein, the preset metadata table stores the storage area of ​​each field on the disk;

[0044] Based on the storage area of ​​the field to be queried, a computing node corresponding to the field to be queried is determined from a preset computing network; wherein, the preset computing network includes multiple computing nodes, and the computing nodes are used to retrieve the field to be queried from the storage area;

[0045] Send a data retrieval command to the computing node and receive the fields to be queried from the computing node.

[0046] In one possible implementation, the method further includes:

[0047] In response to a cross-engine query command, a first query table and a second query table are determined; wherein the first query table and the second query table are stored in different storage engines.

[0048] Based on the size information of the first lookup table and the size information of the second lookup table, determine the broadcast table and the fusion table;

[0049] The table to be broadcast is broadcast to the storage engine corresponding to the table to be merged; wherein, the storage engine corresponding to the table to be merged is used to merge the table to be broadcast and the table to be merged into a target table;

[0050] Obtain the target table from the storage engine corresponding to the table to be merged.

[0051] Secondly, embodiments of this application provide a data processing apparatus based on a data lake, wherein the data lake is stored on a disk and includes multiple data tables and a log table; the data tables represent business data for a preset business; the log table includes multiple change messages; the change messages represent the operation process of performing change operations on the data content in the data tables; the apparatus is applied in the memory of an electronic device; the apparatus includes:

[0052] The acquisition module is used to, in response to a change operation on a data table, acquire the initial information corresponding to the change operation and the content to be updated in the data table, and add the initial information to the log table in the disk; wherein, the initial information represents the change operation to be performed on the data table;

[0053] The determination module is used to determine the updated content of the data table based on multiple change information entries of the data table in the log table, and to perform a replacement operation on the updated content; wherein, the replacement operation is used to replace the updated content in memory with the content to be updated in the disk;

[0054] The update module is used to update the initial information in the disk according to the operation result of the replacement operation to obtain target information; the target information represents the change information corresponding to the current change operation after the operation is completed.

[0055] Thirdly, embodiments of this application provide a data processing device based on a data lake, including: a memory and a processor;

[0056] The memory stores computer-executed instructions;

[0057] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0058] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0059] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0060] This application provides a data lake-based data processing method, apparatus, device, and storage medium. The data lake is stored on a disk and includes multiple data tables and a log table. The data tables represent business data for a preset service. The log table includes multiple change records, representing the operation process of performing change operations on the data content in the data tables. The method is applied in the memory of an electronic device. By receiving change operations on the data tables, the method obtains the initial information corresponding to the change operation and the information to be updated in the data tables, and adds the initial information to the log table on the disk. This ensures that every change operation on the data tables in the data lake is recorded in the log table, thereby better managing the data in the data lake. Then, based on the multiple change records in the log table, the updated content of the data table is determined, and a replacement operation is performed on the updated content. The replacement operation replaces the updated content in memory with the content to be updated on the disk. Because the method is applied in the memory of the electronic device, the processing of the data in the data tables is performed in memory before the replacement operation is performed, without modifying the data on the disk. This ensures that the processing of the data in the data tables before the replacement operation does not affect the data on the disk. After receiving the updated content, the updated content in memory is replaced in the disk at once, ensuring that the update operation is completed in one go. This prevents inconsistencies that may occur during data transfer between storage and computing engines, i.e., preventing data inconsistencies during disk operations. After the replacement operation is completed, the initial information on the disk is updated based on the result to obtain the target information. The target information represents the changes made after the current change operation. This ensures the accuracy of the change process records in the log table, enabling better management of the data tables in the data lake. Attached Figure Description

[0061] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0062] Figure 1 A flowchart illustrating the data processing method based on a data lake provided in this application;

[0063] Figure 2 A flowchart illustrating the process of determining the updated content of the data table provided in this application. Figure 1 ;

[0064] Figure 3 A flowchart illustrating the process of filtering the first information provided in this application;

[0065] Figure 4 A flowchart illustrating the process of determining the updated content of the data table provided in this application. Figure 2 ;

[0066] Figure 5 A flowchart illustrating the process of updating initial information on a disk, as provided in this application;

[0067] Figure 6 Flowchart of the data processing method for the data lake provided in this application Figure 2 ;

[0068] Figure 7 A flowchart illustrating the business data corresponding to the return query operation provided in this application;

[0069] Figure 8 Flowchart of the data processing method for the data lake provided in this application Figure 3 ;

[0070] Figure 9 Flowchart of the data processing method for the data lake provided in this application Figure 4 ;

[0071] Figure 10 A schematic diagram of the data processing device based on a data lake provided in this application;

[0072] Figure 11 A schematic diagram of the structure of the data processing device based on the data lake provided in this application.

[0073] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0074] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0075] First, the terms used in this application will be explained:

[0076] Data lake: A centralized storage system used to centrally store raw, unprocessed data;

[0077] A data warehouse is a highly structured, integrated data storage system used to store data that has undergone processing, cleaning, modeling, and other data processing operations.

[0078] Storage engine: This is a software component responsible for how data is persistently stored on disk and how it is read and written.

[0079] The computing engine is a software framework or system responsible for executing data computation logic. It retrieves data from the storage engine, performs various calculations, and then returns the results to the user or writes them to the storage area.

[0080] Transactionality refers to a mechanism provided by a database management system that ensures a series of database operations are executed as a single, indivisible logical unit.

[0081] Metadata refers to information used to describe, interpret, locate, or manage data. It is not the content of the data itself, but rather data that describes the data.

[0082] Data lakes offer the advantage of flexible storage of various data types, while data warehouses provide the advantages of rapid analysis and high query efficiency. Therefore, in enterprise data management, a lake-warehouse integrated architecture is constructed to store the enterprise's business data. In this architecture, the underlying data lake provides low-cost, flexible data storage, while the upper layer, based on the data warehouse, provides various data computing and analysis engines, offering efficient data access and query performance for scenarios with large data volumes and diverse data types.

[0083] However, insufficient transaction management support in data lakes makes it difficult to guarantee data consistency when data flows between storage and computing engines. For example, the lack of a transactional commit mechanism when performing modification operations such as inserting, updating, and deleting data in a data lake can lead to data inconsistency. Furthermore, in a lake-warehouse architecture, diverse data formats and heterogeneous storage engines make it difficult to directly apply traditional database query optimization techniques, such as indexing and query planning, resulting in low query efficiency for data in the data lake. Simultaneously, the significant differences in metadata formats and management methods between different storage engines and computing engines necessitate additional data migration and format conversion operations for cross-engine joint queries, increasing computational overhead.

[0084] Therefore, when enterprises manage enterprise data based on a lake-warehouse integrated architecture, they face technical problems such as low accuracy and low efficiency in data processing when performing modification or query operations on enterprise data in the data lake.

[0085] The data processing method, apparatus, equipment, and storage medium based on data lake provided in this application are intended to solve the aforementioned technical problems.

[0086] The technical solution of this application and how it solves the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0087] Figure 1 This is a flowchart illustrating the data processing method based on a data lake provided in this application. The data lake is stored on a disk and includes multiple data tables and a log table. The data tables represent business data for a preset business; the log table includes multiple change records; the change records represent the operation process of performing change operations on the data content in the data tables; the method is applied to the memory of an electronic device, such as... Figure 1 As shown, the method includes:

[0088] S101. In response to a change operation on a data table, obtain the initial information corresponding to the change operation and the content to be updated in the data table, and add the initial information to the log table on the disk; wherein, the initial information represents the change operation to be performed on the data table.

[0089] For example, a data lake is stored on disk and includes multiple data tables and a log table. Data tables represent business data for a predefined business, such as a table storing information about multiple departments within an enterprise, or a table storing employee information. Each data lake corresponds to a log table, which includes multiple change records. These change records represent the process of performing change operations on the data content in the data tables. For example, change records include a unique identifier for the change operation, the version number of the data table after the change, the storage path of the data table, the status information of the data table (data table in use, data table expired, etc.), the time information of the change operation, the type of change operation (insert / update / delete, etc.), and the information of the changed object (data table row number / file offset, etc.). One change record corresponds to one change operation. A change operation can be performed on the data content of one or more data tables in the data lake. Change operations can include adding, deleting, or modifying data content in the data tables.

[0090] In response to a user's modification operation on a data table, such as receiving a user's request to modify the department information of a target department in a data table storing information about multiple departments within an enterprise, the system retrieves the initial information corresponding to the modification operation and the content to be updated in the data table. The initial information includes the data table to be manipulated, as expressed by the modification operation, and the modification operation itself. For example, the data table to be manipulated is a data table storing information about multiple departments within the enterprise, and the modification operation is a data modification operation. The content to be updated in the data table is the data content that needs to be manipulated, as expressed by the modification operation. For example, the content to be updated is the department information of the target department. The initial information is then added to a log table on disk. For example, the initial information is added as a modification record corresponding to this modification operation to the log table on disk.

[0091] S102. Based on multiple change records of the data table in the log table, determine the updated content of the data table and perform a replacement operation on the updated content; wherein, the replacement operation is used to replace the updated content in memory with the content to be updated on the disk.

[0092] For example, after obtaining the initial information and the content to be updated in the data table, based on multiple change records of the data table in the log table (if the data table to be operated on consists of multiple change records of that data table), the content to be updated in the data table is determined. A replacement operation is then performed on the updated content, such as replacing the content to be updated in the disk with the updated content. Specifically, the replacement operation replaces the updated content in memory with the content to be updated in the disk.

[0093] It's important to note that after performing a change operation on a data table, the resulting data table becomes the new version. Since each change operation records a change entry in the log table, multiple change entries in the log table correspond to multiple versions of the data table. You can determine the latest version of the data table based on these log entries, and then, based on the data content of the latest version and the change operation instructions, determine the content to be updated.

[0094] For example, the change information includes version number information, which can be ordered to increase sequentially according to the order of modification. From multiple change information entries in the log table, the change information with the largest version number can be identified, and the data table corresponding to the change operation completed for that change information can be identified as the latest version data table. Then, based on the data content of the latest version data table and the change operation indicated by the change operation, the updated content of the data table can be determined.

[0095] For example, if a change operation instructs the modification of the target department's information in a data table storing information about multiple departments within a company, changing the total number of employees in that department to 50, then the latest version of the target department's information in the data table can be obtained, and the total number of employees in that department can also be changed to 50 to obtain the modified department information. This modified department information is then used as the updated content in the data table.

[0096] S103. Based on the result of the replacement operation, update the initial information in the disk to obtain the target information; the target information represents the change information corresponding to the current change operation after the operation is completed.

[0097] For example, based on the result of the replacement operation, such as success or failure, the initial information in the disk is updated to obtain the target information. The target information represents the changes made after the current change operation is completed.

[0098] Since change information represents the process of performing a change operation on the data content in a data table, it can include fields that record the operation results. Therefore, based on the result of the replacement operation, the record field of the operation result in the initial information on disk can be updated to obtain the target information. For example, the record field of the operation result can be changed from "Operation in progress" to "Operation successful" or "Operation failed".

[0099] The data processing method based on a data lake provided in this application embodiment involves a data lake stored on a disk, comprising multiple data tables and a log table. The data tables represent business data for a preset service. The log table includes multiple change records, each representing an operation performed on the data content in the data tables. The method is applied in the memory of an electronic device. By receiving change operations on the data tables, the method obtains the initial information corresponding to the change operation and the information to be updated in the data tables, and adds the initial information to the log table on the disk. This ensures that every change operation on the data tables in the data lake is recorded in the log table, thereby improving the management of data in the data lake. Then, based on the multiple change records in the log table, the updated content of the data tables is determined, and a replacement operation is performed on the updated content. The replacement operation replaces the updated content in memory with the content to be updated on the disk. Because the method is applied in the memory of the electronic device, the processing of data in the data tables is performed in memory before the replacement operation, without modifying the data on the disk. This ensures that the processing of data in the data tables before the replacement operation does not affect the data on the disk. After receiving the updated content, the updated content in memory is replaced in the disk at once, ensuring that the update operation is completed in one go. This prevents inconsistencies that may occur during data transfer between storage and computing engines, i.e., preventing data inconsistencies during disk operations. After the replacement operation is completed, the initial information on the disk is updated based on the result to obtain the target information. The target information represents the changes made after the current change operation. This ensures the accuracy of the change process records in the log table, enabling better management of the data tables in the data lake.

[0100] In some specific implementations, the change information includes the first status information, second status information, time information, and storage path of the data table.

[0101] The first status information includes expired and in-use statuses. An expired status indicates data content that is no longer used, while an in-use status indicates data content that is permitted for use. If the change information corresponds to a change operation that modifies multiple data tables, then the first status information includes the first status information for each data table. That is, one data table corresponds to one set of first status information. If the first status information for a data table is expired, then the change operation corresponding to the change information has expired and is no longer usable by the user. If the first status information for a data table is in-use, then the change operation corresponding to the change information modifies the content of the data table and is usable by the user. If the change operation corresponding to the change information modifies only one data table, then the first status information is the status information for that single data table.

[0102] The second status information includes the "In Operation" status, the "Committed" status, and the "Rollback" status. The "In Operation" status indicates that the change operation is being executed. The "Committed" status indicates that the change operation was successfully executed. The "Rollback" status indicates that the change operation failed. Each change information corresponds to one second status information. When the change operation corresponding to the change information is being executed, the second status information in the change information is "In Operation". When the change operation corresponding to the change information fails, the second status information in the change information is "Rollback". When the change operation corresponding to the change information is successfully executed, the second status information in the change information is "Committed".

[0103] Time information represents the start time of executing a change operation. For example, time information is a timestamp of the start time of executing a change operation. One change information corresponds to one time information.

[0104] A storage path represents the location where a data table is stored in the data lake. Each change message corresponds to a storage path, and the data table can be found in the data lake using the storage path.

[0105] Figure 2 A flowchart illustrating the process of determining the updated content of the data table provided in this application. Figure 1 ,like Figure 2 As shown, in S102 above, determining the updated content of the data table based on multiple change records in the log table includes the following sub-steps:

[0106] S201. Based on the first status information, second status information, and time information of each change information corresponding to the data table in the log table, filter out the first information from multiple change information.

[0107] For example, based on the first status information, second status information, and time information of each change information corresponding to the data table in the log table, the first piece of information is selected from multiple change information entries. Here, the change information corresponding to the data table refers to the change information corresponding to the change operation performed on the data table, and there can be multiple change information entries for the data table. The first piece of information is the change information selected from the multiple change information entries.

[0108] For example, each change operation requires working with the latest version of the data table. Therefore, the first piece of information to be filtered can be the change information corresponding to the latest version of the data table, provided that the latest version of the data table is in use and in a committed state. That is, the first piece of information to be filtered is the change information corresponding to the most recent valid change operation performed on the data table. For example, based on the first status information of each change information corresponding to the data table, we can find the change information whose first status is "in use". Based on the second status information, we can find the change information whose second status is "committed". Based on the time information, we can find the change information whose change operation time is closest to the current time. Thus, the change information whose first status is "in use", second status is "committed", and time information is closest to the current time is identified as the first piece of information.

[0109] S202. Generate the updated content of the data table according to the storage path of the first information.

[0110] For example, after obtaining the first information, the updated content of the data table is generated based on the storage path of the first information. For instance, the latest version of the data table is located using the storage path of the first information. Then, based on the data content of the latest version of the data table, combined with the change operation indicated by the change operation, the updated content of the data table is determined.

[0111] In this embodiment, based on the first status information, second status information, and time information of each change information corresponding to the data table in the log table, the first information can be filtered from multiple change information entries. The filtered first information corresponds to the change information corresponding to the most recent valid change operation performed on the data table. Therefore, the data table corresponding to the storage path in the first information is the latest version of the data table. Subsequently, based on the storage path of the first information, the updated content of the generated data table is the updated content of the data table generated on the basis of the latest version of the data table. This ensures the validity of the change operation, thereby improving the accuracy of the change operation.

[0112] Figure 3 The flowchart for filtering the first information provided in this application is as follows: Figure 3 As shown, in step S201 above, based on the first status information, second status information, and time information of each change information corresponding to the data table in the log table, the first information is filtered out from multiple change information entries, including the following sub-steps:

[0113] S301. Based on the first status information of each change information corresponding to the data table in the log table, determine the first candidate information; the first status information in the first candidate information is the in-use status.

[0114] For example, based on the first status information of each change information corresponding to the data table in the log table, a first candidate information is determined. There is at least one first candidate information. The first status information of the first candidate information is in use. For instance, by traversing each change information corresponding to the data table in the log table, it is determined whether the first status information of the data table in the traversed change information is in use. If it is in use, the change information is determined to be a first candidate information. If it is not in use, the change information is determined not to be a first candidate information. After the traversal is completed, at least one first candidate information is obtained.

[0115] S302. Determine the second candidate information based on the second status information of each first candidate information; the second status information of the second candidate information is the submission status.

[0116] For example, second candidate information is determined based on the second status information of each first candidate information. There is at least one second candidate information. The second status information of the second candidate information is a submitted status. For example, all first candidate information is traversed, and it is determined whether the second status information of the data table among the traversed first candidate information is a submitted status. If it is a submitted status, the first candidate information is determined to be a second candidate information. If it is not a submitted status, the first candidate information is determined not to be a second candidate information. After the traversal is completed, at least one second candidate information is obtained.

[0117] S303. Based on the time information of each second candidate information, select the first information from each second candidate information.

[0118] For example, based on the time information of each second candidate piece of information, the first piece of information is selected from the second candidate pieces of information. There is only one first piece of information. For example, the time information of each second candidate piece of information is sorted, and the time information closest to the current time is determined as the target time information, and the second candidate information corresponding to the target time information is determined as the first information.

[0119] In this embodiment, first candidate information is determined based on the first status information of each change information corresponding to the data table in the log table. The first status information of the first candidate information is "in use". Change information in the "in use" status can be filtered first. Then, second candidate information is determined based on the second status information of each first candidate information. The second status information of the second candidate information is "submitted". Based on the first round of filtering, change information with the second status "submitted" can be further filtered. Finally, first information is selected from the second candidate information based on the time information. Based on the first and second rounds of filtering, change information with the most recent change operation can be further filtered. Thus, the first information with the first status information being "in use", the second status information being "submitted", and the most recent change operation is obtained; that is, the data table corresponding to the first information is the latest available version of the data table.

[0120] Figure 4 A flowchart illustrating the process of determining the updated content of the data table provided in this application. Figure 2 The change information also includes change object information and change type information; change object information represents the data content in the data table; change type represents the method of changing the data content in the data table; change operation represents the data rollback operation; such as Figure 4 As shown, in S102 above, determining the updated content of the data table based on multiple change records in the log table includes the following sub-steps:

[0121] S401. Based on the preset backtracking time and the second status information and time information of each change information in the log table, filter out the second information from each change information in the data table.

[0122] For example, change information also includes object information and change type information. Change object information represents the data content in the data table. This information can be the data content that a change operation requires to be modified in the data table. For instance, in a data table insert operation, the data inserted into the data table is the change object information. Change type represents the method of changing the data content in the data table. For example, in a data table insert operation, the change type is "insert operation." Change operation represents the data rollback operation. For example, a change operation might be to roll back the data table from its current version to a historical version at a specific point in time.

[0123] Based on the preset rollback time and the second status and time information of each change to the data table in the log table, the second information is filtered out from each change to the data table in the log table. There is at least one second piece of information. The preset rollback time is the historical point to which the data table is to be rolled back, and this preset rollback time can be obtained from the change operation. For example, if the change operation is to roll back the data table from the current version to a version before a certain historical point, the preset rollback time is that historical point.

[0124] Based on the second status information of each change information in the log table, all change information with a second status of "committed" can be filtered out. Based on the time information of each change information in the log table, at least one time information whose time is later than or equal to the rollback time can be filtered out. Therefore, based on the preset rollback time, the second status information and time information of each change information in the log table, all change information whose time information is later than or equal to the rollback time and whose change operation was successfully executed can be found, which constitutes at least one second piece of information.

[0125] S402. Based on the changed object information and change type information in the second information, generate the updated content of the data table.

[0126] For example, after obtaining all the second information, it is sorted in ascending order of time to obtain at least one sorted piece of second information. Then, in ascending order of time, a target piece of second information is sequentially retrieved from the sorted at least one piece of second information. For each retrieved target piece of second information, the updated content of the data table is generated based on the changed object information and changed type information within that target piece of second information. For example, if the changed type information in the target piece of second information is an insertion operation, the data inserted is the data corresponding to the changed object information in the target piece of second information. Therefore, the backtracking operation needs to perform the reverse operation of the insertion operation, that is, delete the inserted data from the data table. At this time, the deleted inserted data is the updated content. The data table with the deleted inserted data is used as the new data table. The corresponding backtracking operation is then performed on the new data table based on the next retrieved target piece of second information until the corresponding backtracking operation is completed on the new data table based on the last target piece of second information. The new data table obtained after the last backtracking operation is used as the final backtracked data table.

[0127] In this embodiment, based on a preset backtracking time and the second status information and time of each change information of the data table in the log table, all change information before or at the preset backtracking time can be filtered from the change information of the data table in the log table. Then, based on the change object information and change type information in the second information, the updated content of the data table is generated. That is, based on each piece of second information, the data table is gradually backtracked to the version before the change operation corresponding to that second information was executed. Thus, version backtracking of data tables in the data lake is achieved based on the log table.

[0128] In some specific implementations, step S401 above, based on a preset backtracking time and the second status and time information of each change information in the data table in the log table, filters out the second information from each change information in the data table, including the following sub-steps:

[0129] First, based on the second status information of each change information corresponding to the data table in the log table, determine the third candidate information; the second status information of the third candidate information is the submission status.

[0130] For example, based on the second status information of each change information corresponding to the data table in the log table, a third candidate information is determined. There is at least one third candidate information. The second status information of the third candidate information is in the committed state. For instance, by traversing each change information corresponding to the data table in the log table, it is determined whether the second status information of the data table in the traversed change information is in the committed state. If it is in the committed state, the change information is determined to be a third candidate information. Otherwise, the change information is determined not to be a third candidate information. After the traversal is completed, at least one third candidate information is obtained.

[0131] Second, based on the time information of each third candidate information and the preset backtracking time, the second information is selected from each third candidate information.

[0132] For example, after obtaining at least one third candidate information, second information is selected from the third candidate information based on the time information of each third candidate information and a preset backtracking time. There is at least one second piece of information. For instance, all the third candidate information is traversed, and it is determined whether the time information of the traversed third candidate information is later than or equal to the preset backtracking time. If it is later than or equal to the preset backtracking time, then the third candidate information is determined to be a second piece of information. Otherwise, the third candidate information is determined not to be a second piece of information.

[0133] In these implementations, based on the second status information of each change information corresponding to the data table in the log table, at least one third candidate information can be determined. The second status information of the third candidate information is a committed status, ensuring that subsequent backtracking is only performed on historical change operations that were successfully committed. Based on this, according to the time information of each third candidate information and a preset backtracking time, second information is selected from the third candidate information to obtain at least one second piece of information. Each of these second pieces of information corresponds to a historical change operation, thus yielding multiple historical change operations. This allows for subsequent reverse operations on the data table based on these historical change operations, enabling the data table to be gradually backtracked to a specific historical version.

[0134] Figure 5 This application provides a flowchart illustrating the process of updating initial information on a disk. The first state information in the initial information indicates an "in use" state; the second state information indicates an "operation in progress" state. Figure 5 As shown, in S103 above, the initial information in the disk is updated according to the result of the replacement operation, including the following sub-steps:

[0135] S501. If the replacement operation is successful, the second status information of the initial information on the disk is updated from the "in operation" status to the "committed" status.

[0136] For example, the first status information in the initial information is "in use". That is, for each change operation, by default, after the change operation is performed on the data table, the changed data table is the latest data table and is available for user use. The second status information in the initial information is "in operation", that is, the change operation on the data table is being executed, and the data table is not available for user use during the change process.

[0137] If the replacement operation is successful, the second status information of the initial information on the disk will be updated from the "in operation" status to the "committed" status. At this point, the modified data table will be available to users.

[0138] S502. If the replacement operation fails, the first status information of the initial information on the disk is updated from the in-use status to the expired status, and the second status information is updated from the in-operation status to the rollback status.

[0139] For example, if the replacement operation fails, the first status information of the initial information on the disk is updated from the reused state to the expired state, and the second status information is updated from the in-operation state to the rollback state. That is, the change operation fails, the change operation performed on the data table in memory is ineffective, and the change operation performed on the data table in memory does not affect the data in the data table on disk. Subsequently, based on the first status information being expired and the second status information being rollback, it can be determined that the corresponding change operation is invalid.

[0140] In this embodiment, for each change operation, the first status information in the initial information of the change operation is in use, and the second status information is in operation. The data table in the operation state is not used by the user. If the replacement operation is successful, the second status information of the initial information on the disk is updated from in use to committed, indicating that the data changes to the data table have taken effect, and the changed data table is promptly made available to the user. If the replacement operation fails, the first status information of the initial information on the disk is updated from in use to expired, and the second status information is updated from in operation to rollback, indicating that the data changes to the data table only take effect in memory and do not affect the data in the data table on the disk. This avoids introducing data from the change process into the data table on the disk, that is, it avoids writing dirty data to the data table on the disk, ensuring the stability and accuracy of the data in the data table.

[0141] Figure 6 Flowchart of the data processing method for the data lake provided in this application Figure 2A data lake also includes a permissions table, which records the permissions information of each user. Permission information represents a user's access rights to the data tables, such as... Figure 6 As shown, the above method also includes the following steps:

[0142] S601. In response to a user's data table query operation, determine the data table to be queried and the user's permission information.

[0143] For example, a data lake also includes a permissions table, and a data lake can have one permissions table. The permissions table is used to record the permission information of each user, which represents the user's access permissions to the data tables. For example, the permissions table contains fields such as: user identifier, data table identifier, and access permission level to the data table (including read-only permission, read-write permission, or full control permission, etc.).

[0144] In response to a user's data table query operation, the system determines the data table to be queried and the user's permission information. The data table to be queried can be determined from the user's data table query operation. For example, if the query operation is to retrieve employee information for employee "Zhang San" from data table A, then data table A is determined to be the data table to be queried. The system can also match the user's permission information from the permission table based on the user's user identifier.

[0145] S602. If, based on the user's permission information, it is determined that the user has access rights to the data table to be queried, then the business data corresponding to the query operation is returned from the data table to be queried.

[0146] For example, if the user's permission information determines that the user has access to the data table to be queried, such as read-only, read-write, or full control permissions, then the user has access to the data table to be queried. If the user's permission information does not record a permission level, then the user is assumed to have no access. After determining that the user has access to the data table to be queried, the business data corresponding to the query operation is returned from the data table to be queried. That is, based on the query operation, the relevant business data is retrieved from the data table to be queried, and the retrieved business data is returned to the user.

[0147] In some implementations, the data lake also includes an access table that records access logs of users accessing data tables in the data lake. The log content includes user information, request time information, information about the accessed data table, the type of operation on the data table (such as add, delete, modify, etc.), and the result of the access to the data table (such as access success or access failure), so as to understand the access status of the data table in a timely manner and discover potential security risks or violations.

[0148] Figure 7This is a flowchart illustrating the business data returned by the query operation provided in this application. The change operation is used to synchronize changes across multiple data tables. Each change operation corresponds to a change message upon completion, which includes multiple first-state information entries. The number of these first-state information entries matches the number of data tables. Figure 7 As shown, step S602 above, which retrieves the business data corresponding to the query operation from the data table to be queried, includes the following steps:

[0149] S701. Filter out the fourth candidate information from the multiple change information corresponding to the data table to be queried in the log table; the first status information of the data table to be queried in the fourth candidate information is the in-use status.

[0150] For example, a change operation applies to multiple data tables for synchronization. For instance, a change operation might retrieve all information about employee "Zhang San" from both the employee and department data tables. Upon completion, the change operation corresponds to one change message, which includes multiple first-state information entries. The number of first-state information entries matches the number of data tables. In other words, one change operation corresponds to one change message, and each change message includes multiple first-state information entries. Each first-state information entry corresponds to the first-state information entry of one data table; for example, the employee data table corresponds to one first-state information entry, and the department data table corresponds to one first-state information entry.

[0151] The fourth candidate information is selected from multiple change records corresponding to the data table to be queried in the log table. The fourth candidate information indicates that the first status information of the data table to be queried is in use. There can be at least one fourth candidate information. For example, the log table is iterated through multiple change records corresponding to the data table to be queried, and it is determined whether the first status information of the data table to be queried is in use. If it is in use, then the change record is determined to be a fourth candidate information. Otherwise, the change record is determined not to be a fourth candidate information. After the iteration is complete, at least one fourth candidate information is obtained.

[0152] S702. Select the fifth candidate information from each of the fourth candidate information; the second state of the fifth candidate information is the submission state.

[0153] For example, a fifth candidate is selected from each of the fourth candidate information. There can be at least one fifth candidate. The second state of the fifth candidate is a submitted state. For instance, at least one fourth candidate is traversed, and it is determined whether the second state of the traversed fourth candidate is a submitted state. If it is a submitted state, then the fourth candidate is determined to be a fifth candidate. Otherwise, the fourth candidate is determined not to be a fifth candidate. After traversal, at least one fifth candidate is obtained.

[0154] S703. Based on the time information of each fifth candidate information, select the third information from each fifth candidate information; the time information of the third information is adjacent to the current time.

[0155] For example, based on the time information of each fifth candidate, a third candidate is selected from the fifth candidate. There is only one third candidate. The time information of the third candidate is adjacent to the current time. For instance, at least one fifth candidate is traversed, and the time length between the time information of the traversed fifth candidate and the current time is determined. After traversal, the fifth candidate with the shortest time length is determined as the third candidate.

[0156] S704. Based on the storage path in the third information, obtain the business data from the data table to be queried.

[0157] For example, based on the storage path in the third information, the business data in the data table to be queried is obtained. For instance, the data table is accessed through the storage path in the third information, and a query operation is performed on the data table, returning the business data from the queried data table.

[0158] In this embodiment, a fourth candidate information is selected from multiple change information entries corresponding to the data table to be queried in the log table. The fourth candidate information indicates that the first state of the data table to be queried is "in use," thus achieving the first round of filtering and selecting all change information entries that match the first state of the data table to be queried as "in use." Next, a second round of filtering is performed, selecting a fifth candidate information entry from each of the fourth candidate information entries. The second state of the fifth candidate information entry is "submitted." Based on the first round of filtering, all change information entries matching the second state of "submitted" are further selected. The data tables corresponding to each change information entry selected in the second round are currently available data tables. Finally, a third round of filtering is performed, selecting a third information entry based on the time information of each fifth candidate information entry. The time information of the third information entry is adjacent to the current time. Based on the storage path in the third information entry, the business data in the data table to be queried is obtained. Based on the storage path in the third information entry, the latest version of the available data table can be obtained, thus ensuring that the data obtained from the query operation is the latest version of the business data.

[0159] Figure 8 Flowchart of the data processing method for the data lake provided in this application Figure 3 ,like Figure 8 As shown, the above method also includes the following steps:

[0160] S801. In response to a field query command for a data table, determine the field to be queried in the data table.

[0161] For example, in response to a field query instruction on a data table, the fields to be queried in the data table are determined. For instance, if the field query instruction on a data table instructs the retrieval of employee "Zhang San's" workstation information and start date from the employee data table, then the fields to be queried in the data table are determined to include the field corresponding to the workstation information and the field corresponding to the start date.

[0162] S802. Based on the preset metadata table, obtain the storage area of ​​the field to be queried on the disk; wherein, the preset metadata table stores the storage area of ​​each field on the disk.

[0163] For example, based on a preset metadata table, the storage area of ​​the field to be queried on the disk is obtained. The preset metadata table stores the storage area of ​​each field on the disk.

[0164] It's important to note that a data lake includes a metadata table, which stores multiple metadata entries for each table within the data lake. If the data volume in a table is extremely large, the data in the table needs to be stored in partitions. For example, in an employee data table, employee workstation information might be in the first partition, and employee start date in the second partition. In this case, the multiple metadata entries for the employee data table would include the first metadata entry for the field corresponding to the employee workstation information in the employee data table, and the second metadata entry for the field corresponding to the employee start date in the employee data table, both stored in the same area on disk.

[0165] Therefore, the storage area of ​​the field to be queried on the disk can be obtained from the preset metadata table.

[0166] S803. Based on the storage area of ​​the field to be queried, determine the computing node corresponding to the field to be queried from the preset computing network; wherein, the preset computing network includes multiple computing nodes, and the computing nodes are used to retrieve the field to be queried from the storage area.

[0167] For example, based on the storage area of ​​the field to be queried, a computing node corresponding to the field to be queried is determined from a preset computing network. This preset computing network includes multiple computing nodes, which are used to retrieve the field to be queried from its respective storage area. That is, a field query instruction for a data table uses these multiple computing nodes to retrieve the field to be queried from its corresponding storage area.

[0168] The compute node with the least network data consumption can be determined based on the storage area of ​​the field to be queried. For example, if a compute node in the preset compute network is the same compute node as the compute node corresponding to the storage area of ​​the field to be queried, then using that compute node to retrieve the field to be queried from the storage area of ​​that field has the highest priority, and no network resources are consumed in this case. If a compute node in the preset compute network is under the same switch as the compute node corresponding to the storage area of ​​the field to be queried, then using that compute node to retrieve the field to be queried from the storage area of ​​that field has the second highest priority, and a small amount of network resources are consumed in this case. If a compute node in the preset compute network is not under the same switch as the compute node corresponding to the storage area of ​​the field to be queried, then using that compute node to retrieve the field to be queried from the storage area of ​​that field has the lowest priority, and a large amount of network resources are consumed in this case.

[0169] S804. Send a data retrieval command to the compute node and receive the fields to be queried from the compute node.

[0170] For example, a data retrieval command is sent to the compute nodes, and the compute nodes return the fields to be queried. The data retrieval command instructs the compute nodes to retrieve the fields to be queried from the corresponding storage areas. That is, each compute node retrieves its own corresponding fields to be queried from its corresponding storage area, and the system receives the fields retrieved by each compute node.

[0171] In some specific implementations, the metadata table also includes third-party metadata such as the file version number of the data table. The update frequency of the data can be determined based on the third-party metadata of the data table in the metadata table (e.g., the update frequency can be determined by the number of third-party metadata of the data table within a preset time period, where the larger the number, the more frequent the update frequency). This allows for the creation of appropriate indexes for frequently updated data tables, facilitating quick retrieval of recently changed data.

[0172] In this embodiment, in response to a field query instruction on a data table, the field to be queried in the data table is determined. Based on a preset metadata table, the storage area of ​​the field to be queried on the disk is obtained. Then, based on the storage area of ​​the field to be queried, a computing node corresponding to the field to be queried is determined from a preset computing network. The optimal computing node can be selected for data reading, thereby optimizing the query plan in conjunction with the metadata table. Data retrieval instructions are sent to each computing node, and the field to be queried is received from the computing nodes, reducing network data transmission overhead.

[0173] Figure 9 Flowchart of the data processing method for the data lake provided in this application Figure 4 ,like Figure 9 As shown, the above method also includes the following steps:

[0174] S901, In response to the cross-engine query instruction, determine the first query table and the second query table; wherein the first query table and the second query table are stored in different storage engines.

[0175] For example, in response to a cross-engine query instruction, such as an instruction to perform a joint query on an employee information table and a department information table, where the employee information table corresponds to a first storage engine and the department information table corresponds to a second storage engine, a first query table and a second query table are determined. For example, the first query table is the employee information table and the second query table is the department information table. The first query table and the second query table are stored in different storage engines, such as the first storage engine and the second storage engine being different.

[0176] S902. Based on the size information of the first lookup table and the second lookup table, determine the broadcast table and the fusion table.

[0177] For example, the tables to be broadcast and to be merged are determined based on the size information of the first and second tables to be looked up. For instance, the metadata table includes metadata containing statistical information about the data tables, where the statistical information represents the number of rows and columns counted and the number of rows counted. The statistical information of the first and second tables to be looked up can be obtained from the metadata table. Based on this statistical information, the size information of the first and second tables to be looked up is calculated. For example, the product of the number of rows and columns counted in the first table to be looked up is determined as the size information of the first table to be looked up. The first or second table to be looked up with the smaller size information is determined as the broadcast table, and the first or second table to be looked up with the larger size information is determined as the merged table.

[0178] S903. Broadcast the table to be broadcast to the storage engine corresponding to the table to be merged; wherein, the storage engine corresponding to the table to be merged is used to merge the table to be broadcast and the table to be merged into the target table.

[0179] For example, the table to be broadcast is broadcast to the storage engine corresponding to the table to be merged. The storage engine corresponding to the table to be merged is used to merge the table to be broadcast and the table to be merged into a target table. The broadcast table and the table to be merged can be merged according to a cross-engine query command to obtain the target table. For example, if a cross-engine query command instructs a joint query on the employee information table and the department information table, the employee information table and the department information table can be merged based on the employee's code in the employee information table being the same as the employee's code in the department information table.

[0180] S904. Obtain the target table from the storage engine corresponding to the table to be merged.

[0181] For example, the target table is retrieved from the storage engine corresponding to the table to be merged. This allows for a join query on the target table. For instance, all employee information and department information for employee "Zhang San" can be retrieved from the target table, which is being merged from the employee information table and the department information table.

[0182] In this embodiment, in response to a cross-engine query command, a first query table and a second query table are determined, wherein the first query table and the second query table are stored in different storage engines. Based on the size information of the first query table and the second query table, a broadcast table and a fusion table are determined. When a cross-engine join query is initiated, a small amount of metadata from the data tables in the metadata table is used to calculate the size information of the first query table and the second query table. The size information reflects the cost of the cross-engine join query, resulting in a broadcast table with lower cost and a fusion table with higher cost. Thus, the broadcast table can be broadcast to the storage engine corresponding to the fusion table, reducing the cost of the cross-engine query. The storage engine corresponding to the fusion table merges the broadcast table and the fusion table into a target table. The resulting target table can then be used for cross-engine join queries. This reduces the overall cost of cross-engine join queries and improves the overall efficiency of cross-engine queries.

[0183] In some specific implementations, frequently accessed log tables, permission information tables, and metadata tables can be cached in memory to avoid reading relevant data from disk or other storage media for each query. This allows for the rapid use of various information from the data tables during the data query process, improving the query response speed.

[0184] The following example, using a company's sales data table, further illustrates the technical solution of this application. The sales data table is stored in a data lake. The sales data table records the sales information of each business unit within the company.

[0185] When a business unit adds a new sales record, a change information entry is first inserted into the log table. This change information entry includes: the unique identifier of the change operation, the version number of the sales data table after the change operation, the storage path of the sales data table, the first status information (in use status), the time information, and the change operation (insertion).

[0186] After inserting the sales record, the modified data in memory is committed to the table on disk. If the insertion fails or times out, the change operation is rolled back to the initial state to ensure data consistency, and the second state corresponding to the modified information is the rollback state.

[0187] If errors are subsequently discovered in newly added sales records, the sales data table needs to be restored to a previous version. This can be done by following the reverse process, based on the time information and change operations recorded in the log table, to gradually restore the sales data table to the correct version.

[0188] The log table and metadata table are cached in memory. When querying sales data in the sales data table, the metadata of the sales data table can be retrieved from memory, avoiding direct reading from the disk.

[0189] When querying the sales data table, changes to expired data are pre-filtered based on the first status information in the log table. The latest valid sales data table is then located from changes to data where the first status information is "in use." This reduces unnecessary data scanning and improves query efficiency.

[0190] When generating a query plan for the sales data table, the storage path of the sales data table recorded in the metadata table is used to select the computing node with the least network resource consumption for data reading, thereby reducing data transmission overhead.

[0191] In the sales data table, data from the past year is stored in the third storage engine, while data from a year ago is stored in the fourth storage engine. When querying cross-year data from the sales data table, a cross-engine join query is initiated. This query simultaneously queries data from both the third and fourth storage engines. Based on statistical information from the past year's data and the data from a year ago in the metadata table, the cross-engine cost is estimated. Based on the cost estimation result, the optimal query path is selected, thereby reducing unnecessary data transfer and computational overhead and shortening query execution time.

[0192] Figure 10 This is a schematic diagram of the data processing device based on a data lake provided in this application. The data lake is stored on a disk and includes multiple data tables and a log table. The data tables represent business data for preset services; the log table includes multiple change records; the change records represent the operation process of performing change operations on the data content in the data tables; the device is applied in the memory of an electronic device; such as Figure 10 As shown, the data processing device 100 based on a data lake provided in this embodiment includes:

[0193] The acquisition module 1001 is used to respond to a change operation on a data table, acquire the initial information corresponding to the change operation and the content to be updated in the data table, and add the initial information to the log table on the disk; wherein, the initial information represents the change operation to be performed on the data table;

[0194] The determination module 1002 is used to determine the updated content of the data table based on multiple change information of the data table in the log table, and to perform a replacement operation on the updated content; wherein, the replacement operation is used to replace the updated content in memory with the content to be updated on the disk;

[0195] The update module 1003 is used to update the initial information in the disk according to the result of the replacement operation to obtain the target information; the target information represents the change information corresponding to the current change operation after the operation is completed.

[0196] In one specific implementation, the change information includes the first status information, second status information, time information, and storage path of the data table;

[0197] The first status information includes expired status and in-use status; expired status indicates data content that is no longer used; in-use status indicates data content that is allowed to be used.

[0198] The second status information includes the in-process status, the committed status, and the rollback status; the in-process status indicates that the change operation is being executed; the committed status indicates that the change operation was executed successfully; and the rollback status indicates that the change operation failed.

[0199] The time information represents the start time of the change operation;

[0200] Storage path represents the storage location of a data table within the data lake.

[0201] In one specific implementation, the determining module 1002 is further configured to:

[0202] Based on the first status information, second status information, and time information of each change information corresponding to the data table in the log table, the first information is filtered out from multiple change information;

[0203] Based on the storage path of the first information, generate the updated content of the data table.

[0204] In one specific implementation, the determining module 1002 is further configured to:

[0205] Based on the first status information of each change information corresponding to the data table in the log table, the first candidate information is determined; the first status information in the first candidate information is the in-use status.

[0206] Based on the second status information of each first candidate information, the second candidate information is determined; the second status information of the second candidate information is the submission status.

[0207] Based on the time information of each second candidate, the first information is selected from the second candidate information.

[0208] In one specific implementation, the change information further includes change object information and change type information; the change object information represents the data content in the data table; the change type represents the method of changing the data content in the data table; the change operation represents the data backtracking operation; the determination module 1002 is also used for:

[0209] Based on the preset backtracking time, and the second status information and time information of each change information in the data table in the log table, the second information is filtered out from each change information in the data table;

[0210] Based on the changed object information and change type information in the second information, generate the updated content of the data table.

[0211] In one specific implementation, the determining module 1002 is further configured to:

[0212] Based on the second status information of each change information corresponding to the data table in the log table, the third candidate information is determined; the second status information of the third candidate information is the submission status.

[0213] Based on the time information of each third candidate and the preset backtracking time, the second information is selected from each third candidate.

[0214] In one specific implementation, the first state information in the initial information is the "in use" state; the second state information in the initial information is the "operating" state; the update module 1003 is further used for:

[0215] If the replacement operation is successful, the second status information of the initial information on the disk will be updated from the "in operation" status to the "committed" status.

[0216] If the replacement operation fails, the first status information of the initial information on the disk will be updated from the in-use status to the expired status, and the second status information will be updated from the in operation status to the rollback status.

[0217] In one specific implementation, the data lake also includes a permission table, which records the permission information of each user; the permission information represents the user's access permissions to the data table; the data processing device 100 based on the data lake also includes a query module, used for:

[0218] In response to a user's data table query operation, determine the data table to be queried and the user's permission information;

[0219] If, based on the user's permission information, it is determined that the user has access rights to the data table to be queried, then the business data corresponding to the query operation is returned from the data table to be queried.

[0220] In one specific implementation, the change operation is used to synchronize changes to multiple data tables. Each change operation corresponds to a change message upon completion, which includes multiple first status messages, the number of which matches the number of data tables. The query module also returns the business data corresponding to the query operation from the data table to be queried.

[0221] The fourth candidate information is selected from multiple change information entries corresponding to the data table to be queried in the log table; the first status information of the data table to be queried in the fourth candidate information is the in-use status.

[0222] The fifth candidate information is selected from each of the fourth candidate information; the second status of the fifth candidate information is the submission status.

[0223] Based on the time information of each fifth candidate, the third information is selected from the fifth candidate; the time information of the third information is adjacent to the current time.

[0224] Based on the storage path in the third information, retrieve the business data from the data table to be queried.

[0225] In one specific implementation, the query module is also used for:

[0226] In response to a field query command on a data table, determine the field to be queried in the data table;

[0227] Based on a preset metadata table, obtain the storage area on the disk for the field to be queried; the preset metadata table stores the storage area on the disk for each field.

[0228] Based on the storage area of ​​the field to be queried, the computing node corresponding to the field to be queried is determined from the preset computing network; wherein, the preset computing network includes multiple computing nodes, and the computing nodes are used to retrieve the field to be queried from the storage area;

[0229] Send data retrieval instructions to the compute nodes and receive the fields to be queried from the compute nodes.

[0230] In one specific implementation, the query module is also used for:

[0231] In response to a cross-engine query command, a first query table and a second query table are determined; wherein the first query table and the second query table are stored in different storage engines.

[0232] Based on the size information of the first lookup table and the second lookup table, determine the broadcast table and the fusion table;

[0233] The table to be broadcast is broadcast to the storage engine corresponding to the table to be merged; the storage engine corresponding to the table to be merged is used to merge the table to be broadcast and the table to be merged into the target table.

[0234] Obtain the target table from the storage engine corresponding to the table to be merged.

[0235] The data processing device based on a data lake provided in this embodiment can execute the methods provided in the above method embodiments. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0236] Figure 11 This is a schematic diagram of the structure of the data processing device based on a data lake provided in this application. Figure 11 As shown, the data processing device 110 based on a data lake provided in this embodiment includes at least one processor 1101 and a memory 1102. Optionally, the data processing device 110 based on a data lake further includes a communication component 1103. The processor 1101, the memory 1102, and the communication component 1103 are connected via a bus.

[0237] In a specific implementation, at least one processor 1101 executes computer execution instructions stored in memory 1102, causing at least one processor 1101 to perform the above-described method.

[0238] The specific implementation process of processor 1101 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0239] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0240] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0241] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0242] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0243] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0244] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0245] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0246] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0247] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0248] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0249] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0250] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0251] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A data lake-based data processing method, characterized in that, The data lake is stored on a disk and includes multiple data tables and a log table; the data tables represent business data for preset services; the log table includes multiple change records. The change information represents the operation process of performing a change operation on the data content in the data table; The method is applied to the memory of an electronic device; the method includes: In response to a change operation on a data table, the system obtains the initial information corresponding to the change operation and the content to be updated in the data table, and adds the initial information to the log table on the disk; wherein the initial information represents the change operation to be performed on the data table. Based on multiple change records of the data table in the log table, the updated content of the data table is determined, and a replacement operation is performed on the updated content; wherein, the replacement operation is used to replace the updated content in memory with the content to be updated in the disk; Based on the result of the replacement operation, the initial information in the disk is updated to obtain the target information; the target information represents the change information corresponding to the current change operation after the operation is completed.

2. The method of claim 1, wherein, The change information includes the first status information, second status information, time information, and storage path of the data table; The first status information includes an expired status and a used status; the expired status represents data content that is no longer used; the used status represents data content that is allowed to be used. The second status information includes an in-operation status, a commit status, and a rollback status; the in-operation status indicates that a change operation is being performed; the commit status indicates that the change operation was successfully executed; and the rollback status indicates that the change operation failed. The time information represents the start time of the change operation; The storage path represents the storage location of the data table in the data lake.

3. The method of claim 2, wherein, The step of determining the updated content of the data table based on multiple change records in the log table includes: Based on the first status information, second status information, and time information of each change information corresponding to the data table in the log table, the first information is filtered out from multiple change information; Based on the storage path of the first information, the updated content of the data table is generated.

4. The method of claim 3, wherein, The first information is selected from multiple change information entries based on the first status information, second status information, and time information of each change information entry corresponding to the data table in the log table, including: Based on the first status information of each change information corresponding to the data table in the log table, the first candidate information is determined; the first status information in the first candidate information is the in-use status. Based on the second status information of each first candidate information, the second candidate information is determined; the second status information of the second candidate information is the submission status. Based on the time information of each second candidate, the first information is selected from the second candidate information.

5. The method of claim 2, wherein, The change information also includes change object information and change type information; the change object information represents the data content in the data table; the change type represents the method of changing the data content in the data table; The change operation represents a data backtracking operation; The step of determining the updated content of the data table based on multiple change records in the log table includes: Based on the preset backtracking time, and the second status information and time information of each change information of the data table in the log table, the second information is filtered out from each change information of the data table; Based on the changed object information and change type information in the second information, the updated content of the data table is generated.

6. The method of claim 5, wherein, The step of filtering out the second information from the various change information in the data table based on a preset backtracking time and the second status information and time information of each change information in the log table includes: Based on the second status information of each change information corresponding to the data table in the log table, a third candidate information is determined; the second status information of the third candidate information is a submission status. Based on the time information of each third candidate and the preset backtracking time, the second information is selected from each third candidate.

7. The method of claim 2, wherein, The first state information in the initial information is the in-use state; the second state information in the initial information is the operating state. The step of updating the initial information in the disk based on the result of the replacement operation includes: If the replacement operation is successful, the second status information of the initial information in the disk is updated from the "in operation" status to the "committed" status. If the replacement operation fails, the first status information of the initial information in the disk is updated from the in-use status to the expired status, and the second status information is updated from the in-operation status to the rollback status.

8. The method according to any one of claims 2-7, characterized in that, The data lake also includes a permission table, which records the permission information of each user; the permission information represents the user's access permissions to the data table. The method further includes: In response to a user's data table query operation, determine the data table to be queried and the user's permission information; If, based on the user's permission information, it is determined that the user has access permission to the data table to be queried, then the business data corresponding to the query operation is returned from the data table to be queried.

9. The method of claim 8, wherein, The change operation is used to perform a synchronized change operation on multiple data tables. After the change operation is completed, there is a change information corresponding to the change operation. The change information includes multiple first status information, and the number of the multiple first status information is the same as the number of the multiple data tables. The step of returning the business data corresponding to the query operation from the data table to be queried includes: The fourth candidate information is selected from multiple change information entries corresponding to the data table to be queried in the log table; the first status information of the data table to be queried in the fourth candidate information is the in-use status. The fifth candidate information is selected from each of the fourth candidate information; the second state of the fifth candidate information is the submission state. Based on the time information of each fifth candidate information, the third information is selected from each fifth candidate information; the time information of the third information is adjacent to the current time. Based on the storage path in the third information, obtain the business data from the data table to be queried.

10. The method of claim 9, wherein, The method further includes: In response to a field query command on a data table, determine the field to be queried in the data table; Based on a preset metadata table, the storage area of ​​the field to be queried on the disk is obtained; wherein, the preset metadata table stores the storage area of ​​each field on the disk; Based on the storage area of ​​the field to be queried, a computing node corresponding to the field to be queried is determined from a preset computing network; wherein, the preset computing network includes multiple computing nodes, and the computing nodes are used to retrieve the field to be queried from the storage area; Send a data retrieval command to the computing node and receive the fields to be queried from the computing node.

11. The method of claim 10, wherein, The method further includes: In response to a cross-engine query command, a first query table and a second query table are determined; wherein the first query table and the second query table are stored in different storage engines. Based on the size information of the first lookup table and the size information of the second lookup table, determine the broadcast table and the fusion table; The table to be broadcast is broadcast to the storage engine corresponding to the table to be merged; wherein, the storage engine corresponding to the table to be merged is used to merge the table to be broadcast and the table to be merged into a target table; Obtain the target table from the storage engine corresponding to the table to be merged.

12. A data lake based data processing apparatus, characterized by, The data lake is stored on a disk and includes multiple data tables and a log table; the data tables represent business data for preset services; the log table includes multiple change records. The change information represents the operation process of performing a change operation on the data content in the data table; The device is used in the memory of an electronic device; The device includes: The acquisition module is used to, in response to a change operation on a data table, acquire the initial information corresponding to the change operation and the content to be updated in the data table, and add the initial information to the log table in the disk; wherein, the initial information represents the change operation to be performed on the data table; The determination module is used to determine the updated content of the data table based on multiple change information entries of the data table in the log table, and to perform a replacement operation on the updated content; wherein, the replacement operation is used to replace the updated content in memory with the content to be updated in the disk; The update module is used to update the initial information in the disk according to the operation result of the replacement operation to obtain target information; the target information represents the change information corresponding to the current change operation after the operation is completed.

13. A data lake based data processing device, characterized by, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-11.

15. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-11.

Images