Cache-based asynchronous data updating method and device for data lake bin

Abstract

The invention discloses an asynchronous data updating method and device based on a cache for a data lake bin, and relates to the technical field of asynchronous data updating. The asynchronous data updating method comprises the following steps: receiving a data updating request from an application layer; determining a target update data file according to the data update request; caching the target update data file based on the cache layer module; based on the cache layer module, correspondingly updating data in the cache layer module according to the cached target update data file to generate first update feedback information; and returning the first update feedback information to the application layer. According to the asynchronous data updating method, the target updating data file can be cached to the cache layer module, so that the rapid data writing and reading capability is realized, and the real-time performance and the response speed of the data updating request can be ensured.

Description

Technical Field

[0001] This application relates to the technical field of asynchronous data update, and more specifically, to a cache-based asynchronous data update method and device for a data lake warehouse. Background Art

[0002] After the integration architecture of data lake and data warehouse emerged, technologies such as Delta Lake and Apache Hudi have become the mainstream solutions to support data update and transaction processing.

[0003] Delta Lake is an open-source storage layer built on Apache Spark, which can support the data processing mode of stream-batch integration. Based on the snapshot and optimistic concurrency control mechanism, it has built-in data version control and rollback functions. Apache Hudi provides unified processing capabilities for streaming and batch data, and supports efficient UPSERT (update or insert) and DELETE operations.

[0004] However, technologies such as Delta Lake and Apache Hudi still mainly rely on synchronous write strategies, requiring that data must be confirmed updated only after the write is completed. Although this write-through strategy ensures data persistence, it also brings obvious performance bottlenecks. For example, technologies such as Delta Lake and Apache Hudi adopt synchronous write strategies, that is, before the data is written to the underlying storage and the persistence is completed, the feedback of successful update cannot be returned. This results in a relatively high latency during the update process and is not suitable for real-time application scenarios that require quick responses.

[0005] The content of the background art part is only the technology known to the applicant and does not necessarily represent the prior art in this field. Summary of the Invention

[0006] This application aims to provide a cache-based asynchronous data update method and device for a data lake warehouse to solve the technical problem that the synchronous write strategy has a relatively high latency and is not suitable for real-time application scenarios that require quick responses.

[0007] According to one aspect of this application, this application provides a cache-based asynchronous data update method for a data lake warehouse. The asynchronous data update method includes: receiving a data update request from the application layer; determining a target update data file according to the data update request; caching the target update data file based on the cache layer module; based on the cache layer module, updating the data in the cache layer module according to the cached target update data file to generate a first update feedback message; and returning the first update feedback message to the application layer.

[0008] According to some embodiments of the present application, after the step of returning the first update feedback information to the application layer, the data update method further includes: storing the cached target update data file based on the persistent storage module to update the data in the persistent storage module according to the cached target update data file.

[0009] According to some embodiments of the present application, after the step of determining the target update data file according to the data update request, the asynchronous data update method further includes: determining the metadata file of the target update data file according to the data update request and the target update data file. After the step of caching the target update data file based on the cache layer module, the asynchronous data update method further includes: storing the metadata file based on the persistent storage module; generating the second update feedback information after storing the metadata file; and returning the second update feedback information to the application layer.

[0010] According to some embodiments of the present application, after the step of returning the second update feedback instruction to the application layer, the asynchronous data update method further includes: restoring the target update data file according to the metadata file in the case of loss of the target update data file.

[0011] According to some embodiments of the present application, after the step of returning the second update feedback information to the application layer, the asynchronous data update method further includes: storing the cached target update data file based on the persistent storage module to update the data in the persistent storage module according to the cached target update data file.

[0012] According to one aspect of the present application, the present application provides a cache-based asynchronous data update device for a data lake warehouse. The asynchronous data update device may include an asynchronous update main process management module and a cache layer module. The asynchronous update main process management module receives a data update request from the application layer, and the asynchronous update main process management module determines the target update data file according to the data update request; the cache layer module caches the target update data file, updates the data in the cache layer module according to the cached target update data file to generate the first update feedback information; and the asynchronous update main process management module returns the first update feedback information to the application layer.

[0013] According to some embodiments of the present application, the asynchronous data update device further includes a persistent storage module. The persistent storage module stores the cached target update data file and updates the data in the persistent storage module according to the cached target update data file.

[0014] According to some embodiments of the present application, the asynchronous data update device further includes a metadata management module and a persistent storage module. The metadata management module determines the metadata file of the target update data file according to the data update request and the target update data file; the persistent storage module stores the metadata file; after storing the metadata file, the metadata management module generates second update feedback information; the asynchronous update main process management module returns the second update feedback information to the application layer.

[0015] According to some embodiments of the present application, the asynchronous data update device further includes a recovery mechanism module. In the case of losing the target update data file, the recovery mechanism module restores the target update data file according to the metadata file.

[0016] According to some embodiments of the present application, the asynchronous data update device further includes a persistent storage module. The persistent storage module stores the cached target update data file and updates the data in the persistent storage module according to the cached target update data file.

[0017] On the other hand, the present application also provides a non-volatile computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it can implement the cache-based asynchronous data update method for a data lake warehouse as described above.

[0018] On the other hand, the present application also provides an electronic device, including: one or more processors; a storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors can implement the cache-based asynchronous data update method for a data lake warehouse as described above.

[0019] On the other hand, the present application also provides a computer program product, including: a computer program stored on a computer-readable storage medium; the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the cache-based asynchronous data update method for a data lake warehouse as described above.

[0020] Beneficial effects The present application can determine the target update data file through a data update request. The present application can cache the target update data file through the cache layer module. The present application can update the data in the cache layer module according to the cached target update data file through the cache layer module, and generate first update feedback information. The present application can return the first update feedback information to the application layer.

[0021] The asynchronous data update method provided by this application can achieve fast data writing and reading capabilities by caching the target update data file in the cache layer module, and can ensure the real-time nature and response speed of data update requests. After the corresponding update data in the cache layer module, this application can generate the first update feedback information and send it to the application layer, thereby determining that the target update data file has been correspondingly updated and cached in the cache layer module, and can avoid users waiting for the completion of the persistent storage of the target update data file. Brief Description of the Drawings

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of this application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0023] Figure 1 Fig. 1 shows a schematic flowchart of an asynchronous data update method 1000 according to an embodiment of this application; Figure 2 Fig. 2 shows another schematic flowchart of an asynchronous data update method 1000 according to an embodiment of this application; Figure 3 Fig. 3 shows a schematic flowchart of an asynchronous data update method 2000 according to an embodiment of this application; Figure 4 Fig. 4 shows another schematic flowchart of an asynchronous data update method 2000 according to an embodiment of this application; Figure 5 Fig. 5 shows a schematic structural diagram of an asynchronous data update device according to an embodiment of this application; Figure 6 Fig. 6 shows another schematic structural diagram of an asynchronous data update device according to an embodiment of this application; Figure 7 Fig. 7 shows a schematic diagram of information in a metadata file according to an embodiment of this application.

[0024] Description of the Reference Numerals: Asynchronous data update device 300.

[0025] Asynchronous update main process management module 310; cache layer module 320; persistent storage module 330; metadata management module 340; recovery mechanism module 350. Detailed Description of the Embodiments

[0026] Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the concept of the example embodiments to those skilled in the art. Like reference numerals in the figures denote like or similar parts, and thus their repetitive description will be omitted.

[0027] The features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of the embodiments of the present disclosure. However, those skilled in the art will realize that the technical solutions of the present disclosure can be practiced without one or more of these specific details, or can be implemented in other ways, components, materials, devices, etc. In these cases, well-known structures, methods, devices, implementations, materials, or operations will not be shown or described in detail.

[0028] Furthermore, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products, or devices.

[0029] The terms "first", "second", etc. in the description and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order.

[0030] The technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative efforts fall within the scope of protection of this application.

[0031] The English involved in this application, its full English name, and the corresponding Chinese interpretations: Alluxio is an open-source caching system.

[0032] JYdata, a performance-centric efficient big data processing system.

[0033] SQL, Structured Query Language, that is, Structured Query Language.

[0034] HDFS, Hadoop Distributed File System, Hadoop Distributed File System.

[0035] Delta Lake, an open-source data lake component.

[0036] Apache Spark, an open-source distributed computing framework.

[0037] Apache Hudi, a data lake component.

[0038] According to one aspect of the present application, the present application provides a cache-based asynchronous data update method 1000 for a data lake warehouse. The asynchronous data update method 1000 can be executed by an asynchronous data update device. Exemplarily, the asynchronous data update device can be a host or a server with data processing capabilities.

[0039] A data lake is a system for storing a large amount of different types of raw data, including structured, semi-structured, and unstructured data. The data lake supports flexible access and subsequent processing, and is suitable for scenarios such as big data analysis and model training. A data warehouse is a system for storing organized and structured data, mainly used for efficient data query and analysis, and is often used in enterprise reports and decision support. The asynchronous data update method 1000 of the present application can be applied to the technical fields of data lakes and data warehouses.

[0040] See Figure 1 , the asynchronous data update method 1000 may include step S110-step S150.

[0041] In step S110, the asynchronous data update device receives a data update request from the application layer.

[0042] According to an example embodiment, the application layer can be an application program layer for interaction between the user and the asynchronous data update device. The data update request can be an operation request for the user to request changes to the existing data. The data update request can include operations such as modification, deletion, and addition of write data. The data update request can also include data information to be updated, such as the data to be updated, data source, data destination, change type, and timestamp. The data to be updated can be relational data, text file data, image file data, etc.

[0043] The user can send a data update request through the application layer, and the asynchronous data update device receives the data update request.

[0044] For example, the SQL statement of the data update request can be "INSERT INTO ta5 SELECT * FROM ts5".

[0045] In step S120, the asynchronous data update device determines a target update data file according to the data update request.

[0046] According to an exemplary embodiment, the target update data file may be a file of data to be updated. The asynchronous data update device may parse the data update request and generate a query plan. The asynchronous data update device may calculate the data to be updated according to the data update request and form a target update data file.

[0047] For example, the asynchronous data update device parses the data update request of "INSERT INTO ta5 SELECT * FROM ts5", inserts data into the target table of the data table "ta5", and the source table is all column data and all row data of "ts5". The target update data file may be a file formed by all column data and all row data of "ts5".

[0048] In step S130, the asynchronous data update device caches the target update data file based on the cache layer module.

[0049] According to an exemplary embodiment, the cache layer module may be a component with cache capabilities. For example, the cache layer module may be Alluxio, JYdata or other cache systems, which have high throughput and low latency characteristics. The asynchronous data update device may cache the target update data file in the cache layer module. The cache layer module may provide fast data writing and reading capabilities, and can ensure the real-time performance and response speed of data update requests.

[0050] In step S140, the asynchronous data update device updates the data in the cache layer module according to the cached target update data file based on the cache layer module to generate a first update feedback message.

[0051] According to an exemplary embodiment, the first update feedback message may be feedback operation information after the cache layer module has cached the target update data file. The asynchronous data update device may generate the first update feedback message based on the cache layer module after the cache layer module updates the data. The first update feedback message may determine that the target update data file has been correspondingly updated and cached in the cache layer module.

[0052] In step S150, the asynchronous data update device returns the first update feedback message to the application layer.

[0053] According to an exemplary embodiment, the asynchronous data update device may send the first update feedback message to the application layer, and the user may receive the first update feedback message through the application layer, so as to determine that the target update data file has been correspondingly updated and cached in the cache layer module. The asynchronous data update device returns the first update feedback message to the application layer, which can avoid the user waiting for the completion of the persistent storage of the target update data file. After receiving the first update feedback message through the application layer, the user can continue with the next operation.

[0054] Through the above embodiments, the present application can determine a target update data file through a data update request. The present application can cache the target update data file through a cache layer module. The present application can, through the cache layer module, update the data in the cache layer module according to the cached target update data file, and generate a first update feedback message. The present application can return the first update feedback message to the application layer.

[0055] The asynchronous data update method provided by the present application can achieve fast data writing and reading capabilities by caching the target update data file in the cache layer module, and can ensure the real-time performance and response speed of data update requests. After the data in the cache layer module is updated correspondingly, the present application can generate a first update feedback message and send it to the application layer, so as to determine that the target update data file has been updated and cached in the cache layer module correspondingly, and can avoid the user waiting for the completion of the persistent storage of the target update data file.

[0056] Optionally, referring to Figure 2 , the asynchronous data update method may further include step S160.

[0057] In step S160, the asynchronous data update device stores the cached target update data file based on the persistent storage module, so as to update the data in the persistent storage module according to the cached target update data file.

[0058] According to an exemplary embodiment, the persistent storage module may be based on HDFS or other big data storage systems. The asynchronous data update device can store the target update data file that has been cached in the cache layer module in the persistent storage module.

[0059] The asynchronous data update device can store the target update data file cached in the cache layer module in the persistent storage module through a background batch task writing mechanism. For example, the asynchronous data update device can also, when the cache layer module is full or a timed writing task is triggered, batch-write the target update data file cached in the cache layer module into the persistent storage module to ensure the ultimate consistency of the target update data file.

[0060] In the process of the asynchronous data update device storing the target update data file cached in the cache layer module in the persistent storage module through the background batch task writing mechanism, it can control the priority of storing the target update data file in the cache layer module in the persistent storage module through a task queue and a priority control mechanism, and ensure the stability of the background batch task writing mechanism.

[0061] Through the above embodiments, the present application can store the cached target update data file through the persistent storage module, so as to update the data in the persistent storage module according to the cached target update data file.

[0062] The asynchronous data update method provided by this application can permanently store the cached target update data file through the persistent storage module, thereby ensuring the eventual consistency of the target update data file. This application can first cache the target update data file in the cache layer module and then batch write it to the persistent storage module in the background, reducing the frequency of write / read operations of the persistent storage module and improving the overall throughput capacity of the asynchronous data update device.

[0063] According to one aspect of this application, this application provides a cache-based asynchronous data update method 2000 for a data lake warehouse. The asynchronous data update method 2000 can be executed by an asynchronous data update device. Exemplarily, the asynchronous data update device can be a host or a server with data processing capabilities.

[0064] See Figure 3 , the asynchronous data update method 2000 can include step S210 - step S290.

[0065] See Figure 1 and Figure 3 , steps S210 - S220 in the asynchronous data update method 2000 are the same as steps S110 - S120 in the asynchronous data update method 1000, so they will not be elaborated here.

[0066] In step S230, the asynchronous data update device determines the metadata file of the target update data file according to the data update request and the target update data file.

[0067] According to the exemplary embodiment, the metadata file can be an update operation file of the target update data file. The metadata file can include key fields such as read / write version numbers (readVersion) and the number of output lines.

[0068] The metadata file can also include cache status (cacheWrite), cache status timestamp, and data lineage information (lineage). The data lineage information can include file information of the target update data file, such as the name of the target update data file. The name of the target update data file can include a list of file names deleted in this update and a list of newly added file names.

[0069] The metadata file can be used to record the source, change history, and dependencies of the target update data file, ensuring that the target update data file can accurately identify and track data versions during data reading and recovery. Each update of the target update data file can correspond to the generation of a metadata file.

[0070] For example, the metadata file can be an update log of the target update data file. For example, the information in the metadata file can be as Figure 7as shown

[0071] See Figure 1 and Figure 3 , steps S240 - S260 in the asynchronous data update method 2000 are the same as steps S130 - S150 in the asynchronous data update method 1000, so they will not be elaborated here.

[0072] Steps S270 - S290 can be executed synchronously with steps S240 - S260.

[0073] In step S270, the asynchronous data update device stores the metadata file based on the persistent storage module.

[0074] According to the exemplary embodiment, the asynchronous data update device can store the metadata file in the persistent storage module. The size of the metadata file is relatively small and can be directly stored in the persistent storage module.

[0075] In step S280, after storing the metadata file, the asynchronous data update device generates second update feedback information.

[0076] According to the exemplary embodiment, the second update feedback information can be feedback operation information after the persistent storage module has stored the metadata file.

[0077] The asynchronous data update device can generate second update feedback information after the persistent storage module correspondingly stores the metadata file. The second update feedback information can determine that the metadata file has been stored in the persistent storage module.

[0078] In step S290, the asynchronous data update device returns the second update feedback information to the application layer.

[0079] The asynchronous data update device can send the second update feedback information to the application layer, and the user can receive the second update feedback information through the application layer, thereby determining that the metadata file has been stored in the persistent storage module.

[0080] Through the above embodiments, the present application can determine the metadata file through a data update request and a target update data file. The present application can store the metadata file through the persistent storage module. The present application can generate second update feedback information after storing the metadata file. The present application can return the second update feedback information to the application layer.

[0081] The asynchronous data update method provided by the present application can record the file information of the target update data file through the metadata file and record the update log.

[0082] Optionally, see Figure 4 , after step S290, the asynchronous data update method 2000 may further include step S200a.

[0083] In step S200a, when the asynchronous data update device loses the target update data file, it restores the target update data file according to the metadata file.

[0084] According to the example embodiment, when the asynchronous data update device is processing a data reading task, if it detects that the cache of the target update data file in the cache layer module fails or the target update data file is missing in the persistent storage module, a recovery mechanism is automatically triggered.

[0085] The asynchronous data update device can reconstruct the lost target update data file based on the lineage information in the metadata file and update the relevant metadata tags to ensure the consistency of the target update data file. The user can also control the execution frequency and scope of the process of restoring the target update data file by configuring recovery parameters to reduce performance overhead.

[0086] For example, when the asynchronous data update device detects that a certain file of "ta5" is missing, the asynchronous data update device can locate the metadata file 1 that records the "ta5" file, and then read the lineage information in the metadata file, and re-execute INSERT INTO ta5 SELECT * FROM ts5 to restore the data of the "ta5" file. The asynchronous data update device can record the restored metadata information, that is, form metadata file 2. The restored metadata file 2 includes the newly added data file, and at the same time records all the data files in metadata file 1 as deleted files. Such a metadata file 2 is directly stored in the persistent storage module, that is, the data recovery is completed.

[0087] Through the above embodiments, the present application can restore the target update data file through the metadata file when the target update data file is lost, and can avoid the problems of data inconsistency or data loss. Moreover, when the cache of the target update data file in the cache layer module fails or the target update data file is missing in the persistent storage module, the target update data file is automatically restored, and the restoration process is simple and efficient without manual intervention.

[0088] Optionally, referring to Figure 4 , after step S290, the asynchronous data update method 2000 may further include step S200b.

[0089] Referring to Figure 2 and Figure 4 , step S200b in the asynchronous data update method 2000 is the same as step S160 in the asynchronous data update method 1000, so it will not be described in detail here.

[0090] Through the above embodiments, the asynchronous data update method provided by the present application can permanently store the cached target update data file through the persistent storage module, thereby ensuring the ultimate consistency of the target update data file. The present application can first cache the target update data file to the cache layer module and then batch write it to the persistent storage module in the background, reducing the frequency of write / read operations of the persistent storage module and improving the overall throughput capacity of the asynchronous data update device.

[0091] According to an example embodiment, the asynchronous data update method provided by the present application can be used in the e-commerce field. For example, it can be applied to scenarios such as a large number of item inventory and order status updates in inventory and order management.

[0092] According to one aspect of the present application, there is provided a cache-based asynchronous data update device 300 for a data lake warehouse. The concepts of data lake and data warehouse have been described in the asynchronous data update method 1000 above, so they will not be elaborated here. The asynchronous data update device 300 can be applied to the technical fields of data lake and data warehouse.

[0093] See Figure 5 , the asynchronous data update device 300 may include an asynchronous update main process management module 310 and a cache layer module 320. ]>

[0094] According to an example embodiment, the asynchronous update main process management module 310 receives a data update request from the application layer.

[0095] According to an example embodiment, the application layer can be an application program layer for users to interact with the asynchronous data update device 300. The data update request can be an operation request for a user to request changes to existing data. The data update request can include operations such as modification, deletion, and addition of write data. The data update request can also include information about the data to be updated, such as the data to be updated, data source, data destination, change type, and timestamp. The data to be updated can be relational data, text file data, image file data, etc.

[0096] The user can send a data update request through the application layer, and the asynchronous update main process management module 310 receives the data update request.

[0097] For example, the SQL statement of the data update request can be "INSERT INTO ta5 SELECT * FROM ts5".

[0098] According to an example embodiment, the asynchronous update main process management module 310 determines a target update data file according to the data update request.

[0099] According to an exemplary embodiment, the target update data file may be the file of the data to be updated. The asynchronous update main process management module 310 may parse the data update request and generate a query plan. The asynchronous update main process management module 310 may calculate the data to be updated according to the data update request to form a target update data file.

[0100] For example, the asynchronous update main process management module 310 parses the data update request of "INSERT INTO ta5 SELECT * FROM ts5", inserts data into the target table of the data table "ta5", and the source table is all column data and all row data of "ts5". The target update data file may be a file formed by all column data and all row data of "ts5".

[0101] According to an exemplary embodiment, the cache layer module 320 caches the target update data file. The cache layer module 320 may be a component with cache capabilities. For example, the cache layer module 320 may be Alluxio, JYdata or other cache systems, with high throughput and low latency characteristics. The cache layer module 320 may cache the target update data file. The cache layer module 320 may provide fast data writing and reading capabilities, and can ensure the real-time performance and response speed of the data update request.

[0102] According to an exemplary embodiment, the cache layer module 320 updates the data in the cache layer module 320 according to the cached target update data file to generate a first update feedback message. The first update feedback message may be the feedback operation message after the cache layer module 320 has cached the target update data file. After the cache layer module 320 updates the data correspondingly, the cache layer module 320 generates a first update feedback message. The first update feedback message may determine that the target update data file has been updated and cached in the cache layer module 320 correspondingly. The cache layer module 320 may send the first update feedback message to the asynchronous update main process management module 310.

[0103] According to an exemplary embodiment, the asynchronous update main process management module 310 returns the first update feedback message to the application layer.

[0104] The asynchronous update main process management module 310 may send the first update feedback message to the application layer, and the user may receive the first update feedback message through the application layer, so as to determine that the target update data file has been updated and cached in the cache layer module 320 correspondingly. The asynchronous update main process management module 310 returns the first update feedback message to the application layer, which can avoid the user waiting for the completion of the persistent storage of the target update data file. After the user receives the first update feedback message through the application layer, the user can continue with the next operation.

[0105] Through the above embodiments, the present application can determine a target update data file through a data update request. The present application can cache the target update data file through a cache layer module. The present application can, through the cache layer module, update the data in the cache layer module according to the cached target update data file, and generate a first update feedback message. The present application can return the first update feedback message to the application layer.

[0106] The asynchronous data update device of the present application can achieve fast data writing and reading capabilities by caching the target update data file in the cache layer module, and can ensure the real-time performance and response speed of data update requests. After the data in the cache layer module is updated correspondingly, the present application can generate a first update feedback message and send it to the application layer, so as to determine that the target update data file has been updated and cached in the cache layer module correspondingly, and can avoid users waiting for the completion of the persistent storage of the target update data file.

[0107] Optionally, referring to Figure 6 , the asynchronous data update device 300 may further include a persistent storage module 330.

[0108] According to an exemplary embodiment, the persistent storage module 330 stores the cached target update data file, and updates the data in the persistent storage module 330 according to the cached target update data file.

[0109] The persistent storage module 330 may be based on HDFS or other big data storage systems. The cache layer module 320 may store the target update data file that has been cached in the cache layer module 320 to the persistent storage module 330.

[0110] The cache layer module 320 may store the target update data file cached in the cache layer module 320 to the persistent storage module 330 through a background batch task writing mechanism. For example, the cache layer module 320 may also, when the cache layer module 320 is full or a timed writing task is triggered, batch-write the target update data file cached in the cache layer module 320 to the persistent storage module 330 to ensure the eventual consistency of the target update data file.

[0111] During the process of the cache layer module 320 storing the target update data file cached in the cache layer module 320 to the persistent storage module 330 through the background batch task writing mechanism, the priority of storing the target update data file of the cache layer module 320 to the persistent storage module 330 can be controlled through a task queue and a priority control mechanism, and the stability of the background batch task writing mechanism can be ensured.

[0112] Through the above embodiments, the present application can store the cached target update data file through the persistent storage module, so as to update the data in the persistent storage module according to the cached target update data file.

[0113] The asynchronous data update device of the present application can permanently store the cached target update data file through the persistent storage module, thereby ensuring the final consistency of the target update data file. The present application can first cache the target update data file to the cache layer module and then batch write it to the persistent storage module in the background, reducing the frequency of write / read operations of the persistent storage module and improving the overall throughput capacity of the asynchronous data update device.

[0114] Optionally, referring to Figure 6 , the asynchronous data update device 300 may further include a metadata management module 340.

[0115] According to the exemplary embodiment, the metadata management module 340 determines the metadata file of the target update data file according to the data update request and the target update data file.

[0116] The metadata management module 340 may be an extended module based on Delta Lake. The metadata file may be an update operation file of the target update data file. The metadata file may include key fields such as readVersion and output row count.

[0117] The metadata file may further include cache status (cacheWrite), cache status timestamp, and data lineage information (lineage). The data lineage information may include file information of the target update data file, such as the name of the target update data file. The name of the target update data file may include the list of file names deleted in this update and the list of newly added file names.

[0118] The metadata file can be used to record the source, change history, and dependency relationship of the target update data file, ensuring that the target update data file can accurately identify and track data versions during data reading and recovery. Each update of the target update data file can correspond to the generation of a metadata file.

[0119] For example, the metadata file may be the update log of the target update data file. For example, the information in the metadata file may be as Figure 7 shown.

[0120] According to the exemplary embodiment, the persistent storage module 330 stores the metadata file. The metadata management module 340 can store the metadata file to the persistent storage module 330. The size of the metadata file is relatively small and can be directly stored in the persistent storage module 330.

[0121] According to an example embodiment, after storing the metadata file, the metadata management module 340 generates second update feedback information. The second update feedback information may be feedback operation information after the persistent storage module 330 has stored the metadata file.

[0122] The metadata management module 340 may generate second update feedback information after the persistent storage module 330 stores the corresponding metadata file. The second update feedback information may determine that the metadata file has been stored in the persistent storage module 330. The metadata management module 340 may send the second update feedback information to the asynchronous update main process management module 310.

[0123] According to an example embodiment, the asynchronous update main process management module 310 returns the second update feedback information to the application layer, and the user may receive the second update feedback information through the application layer, so as to determine that the metadata file has been stored in the persistent storage module 330.

[0124] Through the above embodiments, the present application may determine the metadata file through a data update request and a target update data file. The present application may store the metadata file through the persistent storage module. After storing the metadata file, the present application may generate second update feedback information. The present application may return the second update feedback information to the application layer.

[0125] The asynchronous data update device of the present application may record the file information of the target update data file through the metadata file and record the update log.

[0126] Optionally, the asynchronous update main process management module 310 may parse the data update request through the SparkSQL module and generate a query plan (QueryPlan). In the stage of generating the query plan, the asynchronous main process module may read the table structure information provided by the metadata management module 340 to verify the existence of the target table and data compatibility. For example: data update request: INSERT INTO ta5 SELECT * FROM ts5. The asynchronous update main process management module 310 may parse and generate a query plan through the SparkSQL module, check the structural compatibility of "ta5" and "ts5", and obtain the latest version number of "ta5" from the metadata management module 340.

[0127] Optionally, referring to Figure 6 , the asynchronous data update device 300 may further include a recovery mechanism module 350.

[0128] According to an example embodiment, in the case of losing the target update data file, the recovery mechanism module 350 restores the target update data file according to the metadata file.

[0129] According to an exemplary embodiment, when the asynchronous update main process management module 310 is processing a data reading task and detects that the target update data file cache in the cache layer module 320 fails or the target update data file is missing in the persistent storage module 330, a recovery mechanism is automatically triggered.

[0130] The recovery mechanism module 350 can reconstruct the lost target update data file based on the lineage information in the metadata file and update the relevant metadata tags to ensure the consistency of the target update data file. The user can also control the execution frequency and scope of the process of recovering the target update data file by configuring recovery parameters to reduce performance overhead.

[0131] For example, when the asynchronous update main process management module 310 detects that a certain file of "ta5" is missing, the recovery mechanism module 350 can, through the metadata management module 340, locate the metadata file 1 that records the "ta5" file, and then read the lineage information in the metadata file and re-execute INSERT INTO ta5 SELECT * FROM ts5 to recover the data of the "ta5" file. The metadata management module 340 can record the recovered metadata information, that is, form the metadata file 2. The recovered metadata file 2 includes the newly added data file, and at the same time records all the data files in the metadata file 1 as deleted files. Such a metadata file 2 is directly stored in the persistent storage module 330, that is, the data recovery is completed.

[0132] Through the above embodiments, the present application can recover the target update data file through the metadata file in the case of losing the target update data file, which can avoid problems such as data inconsistency or data loss. Moreover, when the target update data file cache in the cache layer module fails or the target update data file is missing in the persistent storage module, the target update data file is automatically recovered, and the recovery process is simple and efficient without manual intervention.

[0133] Optionally, the persistent storage module 330 stores the cached target update data file and updates the data in the persistent storage module 330 according to the cached target update data file.

[0134] The present application can first cache the target update data file in the cache layer module 320 and then batch write it to the persistent storage module 330 in the background, reducing the frequency of write / read operations of the persistent storage module 330 and improving the overall throughput capacity of the asynchronous data update device.

[0135] The asynchronous data update device of the present application can solve the problems of high latency, low throughput, and insufficient data consistency guarantee in the prior art in high-frequency data update scenarios. Moreover, the asynchronous data update device of the present application can balance real-time performance and reliability, is applicable to high-concurrency scenarios such as log analysis and Internet of Things data collection, and has the advantages of high efficiency, flexibility, and scalability.

[0136] According to an example embodiment, the asynchronous data update device provided by the present application can be used in the e-commerce field. For example, in scenarios such as a large number of item inventory and order status updates in inventory and order management.

[0137] According to another aspect of the present application, the present application also provides a non-volatile computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it can implement the cache-based asynchronous data update method for a data lake warehouse as described above.

[0138] According to another aspect of the present application, the present application also provides an electronic device, including: one or more processors; a storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors can implement the cache-based asynchronous data update method for a data lake warehouse as described above.

[0139] According to another aspect of the present application, the present application also provides a computer program product, including: a computer program stored on a computer-readable storage medium; the computer program includes program instructions. When the program instructions are executed by a computer, the computer is caused to execute the cache-based asynchronous data update method for a data lake warehouse as described above.

[0140] Finally, it should be noted that the above are only the preferred embodiments of the present application and are not used to limit the present application. Although the present application has been described in detail with reference to the foregoing embodiments, for those skilled in the art, they can still modify the technical solutions of the foregoing embodiments, or perform equivalent replacements for some of the technical features. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A cache-based asynchronous data update method for a data lakehouse, characterized in that, The asynchronous data update method includes: Receiving a data update request from the application layer; Determining a target update data file according to the data update request; Caching the target update data file based on the cache layer module; Based on the cache layer module, updating the data in the cache layer module according to the cached target update data file to generate a first update feedback message; Returning the first update feedback message to the application layer.

2. The asynchronous data update method according to claim 1, wherein After returning the first update feedback message to the application layer, the data update method further includes: Storing the cached target update data file based on the persistent storage module to update the data in the persistent storage module according to the cached target update data file.

3. The asynchronous data update method according to claim 1, wherein After determining the target update data file according to the data update request, the asynchronous data update method further includes: Determining a metadata file of the target update data file according to the data update request and the target update data file; After caching the target update data file based on the cache layer module, the asynchronous data update method further includes: Storing the metadata file based on the persistent storage module; Generating a second update feedback message after storing the metadata file; Returning the second update feedback message to the application layer.

4. The asynchronous data update method according to claim 3, wherein After returning the second update feedback instruction to the application layer, the asynchronous data update method further includes: Restoring the target update data file according to the metadata file in the case of losing the target update data file.

5. The asynchronous data update method according to claim 4, wherein After returning the second update feedback message to the application layer, the asynchronous data update method further includes: Storing the cached target update data file based on the persistent storage module to update the data in the persistent storage module according to the cached target update data file.

6. A cache-based asynchronous data update device for a data lake warehouse, characterized in that, The asynchronous data update device includes: An asynchronous update main process management module that receives a data update request from the application layer, and the asynchronous update main process management module determines a target update data file according to the data update request; A cache layer module that caches the target update data file and updates the data in the cache layer module according to the cached target update data file to generate a first update feedback message; The asynchronous update main process management module returns the first update feedback message to the application layer.

7. The asynchronous data update device according to claim 6, characterized in that, The asynchronous data update device further includes: A metadata management module that determines a metadata file of the target update data file according to the data update request and the target update data file; A persistent storage module that stores the metadata file; The metadata management module generates a second update feedback message after storing the metadata file; The asynchronous update main process management module returns the second update feedback message to the application layer.

8. The asynchronous data update device according to claim 7, characterized in that, The asynchronous data update device further includes: A recovery mechanism module that restores the target update data file according to the metadata file in the case of losing the target update data file.

9. The asynchronous data update device according to claim 6 or 8, characterized in that The asynchronous data update device further includes: Persistent storage module, stores the cached target update data file, and updates the data in the persistent storage module according to the cached target update data file.

Images