Reading method and device based on Hudi data, electronic equipment and storage medium

Abstract

The invention relates to the field of data request processing, and discloses a reading method based on Hudi data, which comprises the following steps: receiving a request for accessing a target object to a Hudi storage system, and extracting an identifier of the target object from the reading request; querying a data cache region of the Hudi storage system by using the identifier to determine whether an update record merge file of the target object is stored in a read-time merge layer of the data cache region or not; if the update record merged file corresponding to the target object is stored, retrieving an initial data record of the target object from the MOR table according to the identifier; and sending the retrieved initial data record to a data cache region, and merging the initial data record and the update record merging file to generate a final data file of the target object in response to the reading request. By introducing the data cache region and the merging layer during reading, the situation that a large number of historical records need to be traversed during reading each time is avoided, and therefore the query efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of data request processing, and in particular to a Hudi data reading method, device, electronic device and storage medium. Background Art

[0002] Hudi (Hadoop Upsert Data Incrementally) in the Hadoop ecosystem is an open source data storage system that has gradually been widely used in e-commerce, insurance, finance and other industries.

[0003] The traditional Hudi storage system relies on the MOR (Merge-on-Read) table when reading data. The MOR table usually saves each update record of the initial data. Although this approach can ensure the historical traceability of data, it also brings challenges in query performance. When users read data from the MOR table, they need to traverse each update record of the initial data in the MOR table and merge each update record.

[0004] When data is updated frequently and the number of historical update records is huge, this full-data traversal and merging method will cause query performance to degrade significantly as the amount of data grows.

[0005] Therefore, in the Hudi data storage system, how to improve data reading performance is a technical problem that needs to be solved urgently. Summary of the invention

[0006] In view of the above, it is necessary to provide a method for reading data based on Hudi data, the purpose of which is to improve the performance of reading data in the Hudi data storage system.

[0007] The Hudi data reading method provided by the present invention includes:

[0008] Receive a request to the Hudi storage system to access a target object, and extract an identifier of the target object from the request;

[0009] The data cache area of ​​the Hudi storage system is queried using the identifier to determine whether the read-time merge layer of the data cache area stores a merged file of update records corresponding to the target object, wherein the merged file of update records is obtained by traversing the update records associated with the target object in the MOR table of the Hudi storage system in advance and merging the update records, and the MOR table stores the update records of each iteration of the target object;

[0010] If the update record merge file is stored, the initial data record of the target object is retrieved from the Hudi storage system according to the identifier, the retrieved initial data record is sent to the read-time merge layer, and the initial data record and the record of the update record merge file are merged using the preset merge function of the read-time merge layer to obtain the data file corresponding to the target object and respond to the request;

[0011] If the update record merge file is not stored, the update records associated with the target object in the MOR table of the Hudi storage system are traversed, all update records associated with the target object are merged to generate an update record merge file, the initial data record of the target object is retrieved from the Hudi storage system according to the identifier, the update record merge file and the initial data record are sent to the read-time merge layer for merging, the data file corresponding to the target object is obtained and the request is responded to.

[0012] Optionally, the using the identifier to query the data cache area of ​​the Hudi storage system to determine whether the read-time merge layer of the data cache area stores an update record merge file corresponding to the target object, including:

[0013] Using the identifier as a query key of a preset query interface of the data cache;

[0014] querying in the read-time merge layer whether there is an update record merge file matching the query key stored therein according to the preset query interface;

[0015] If the hash value of the merged update record file is consistent with the hash value of the query key, it is determined that the merged update record file corresponding to the target object is stored.

[0016] Optionally, the MOR table is obtained according to the following method:

[0017] Create a Hive table in a preset storage path of the Hudi storage system, wherein the Hive table is used to organize and query storage units of initial data stored in the Hudi storage system;

[0018] Configure the MOR table type function for the Hive table to obtain the MOR table of the Hudi storage system;

[0019] The initial data of the target object is stored in the Hudi storage system to obtain an initial data record, and the initial data record is stored in the MOR table.

[0020] Optionally, storing the initial data of the target object in the Hudi storage system to obtain the initial data record includes:

[0021] Preprocessing the initial data to obtain preprocessed initial data;

[0022] Extract key fields from the preprocessed initial data, wherein the key fields include a timestamp and a version number;

[0023] The key field is used as compression description information of the preprocessed initial data, and the initial data is compressed according to the compression description information to obtain the initial data record.

[0024] Optionally, it is characterized in that, after storing the initial data record in the MOR table, the method further comprises:

[0025] Create a .Log document in the MOR table, wherein the .Log document is used to record a log file of changes to the initial data operation;

[0026] Recording a data operation of a preset type performed on the initial data to obtain an update record of the initial data record;

[0027] Each update record to the initial data record is stored in the .Log file.

[0028] Optionally, the read-time merging layer of the data cache area is obtained according to the following method:

[0029] Divide a spatial area in the data buffer area as an initial layer;

[0030] After detecting that the initial data record generates an update record, all update records of the initial data record are traversed in the MOR table and merged according to the timestamp and version number of the update record to obtain an update record merge file;

[0031] The update record merge file and the preset merge function are stored in the initial layer to obtain the read-time merge layer.

[0032] Optionally, the merging the initial data record and the update record merge file by using a preset merge function of the read-time merge layer to generate a result record corresponding to the target object includes:

[0033] Parse the update record merge file to obtain the latest update record of the target object, wherein the latest update record includes a record key and an updated new value;

[0034] Decompress the initial data record to obtain the initial data of the target object, and query the corresponding row of the latest updated record in the initial data according to the record key and the preset merge function;

[0035] The updated new value replaces the original value on the corresponding row to obtain the final target object.

[0036] In order to solve the above problems, the present invention also provides a reading device based on Hudi data, the device comprising:

[0037] A receiving module, configured to receive a request to access a target object from the Hudi storage system, and extract an identifier of the target object from the request;

[0038] A determination module is used to query the data cache area of ​​the Hudi storage system using the identifier to determine whether the read-time merge layer of the data cache area stores an update record merge file corresponding to the target object, wherein the update record merge file is obtained by pre-traversing the update records associated with the target object in the MOR table of the Hudi storage system and merging the update records, and the MOR table stores the update records of each iteration of the target object;

[0039] A retrieval module, configured to retrieve the initial data record of the target object from the Hudi storage system according to the identifier if the update record merge file is stored, send the retrieved initial data record to the read-time merge layer, merge the initial data record with the record of the update record merge file using a preset merge function of the read-time merge layer, obtain the data file corresponding to the target object and respond to the request;

[0040] A traversal module is used to traverse the update records associated with the target object in the MOR table of the Hudi storage system if the update record merge file is not stored, merge all update records associated with the target object to generate an update record merge file, retrieve the initial data record of the target object from the Hudi storage system according to the identifier, send the update record merge file and the initial data record to the read-time merge layer for merging, obtain the data file corresponding to the target object and respond to the request.

[0041] In order to solve the above problem, the present invention further provides an electronic device, the electronic device comprising:

[0042] at least one processor; and,

[0043] a memory communicatively connected to the at least one processor; wherein,

[0044] The memory stores a Hudi data-based reading program that can be executed by the at least one processor, and the Hudi data-based reading program is executed by the at least one processor so that the at least one processor can execute the above-mentioned Hudi data-based reading method.

[0045] In order to solve the above problems, the present invention also provides a computer-readable storage medium, on which a reading program based on Hudi data is stored. The reading program based on Hudi data can be executed by one or more processors to implement the above-mentioned reading method based on Hudi data.

[0046] Compared with the prior art, the present invention receives a request to access a target object to a Hudi storage system, and extracts an identifier of the target object from the request; uses the identifier to query a data cache area of ​​the Hudi storage system to determine whether a read-time merge layer of the data cache area stores an update record merge file corresponding to the target object; by pre-generating an update record merge file and storing it in the data cache area, the latest data in the cache can be directly used when an access request arrives, avoiding traversing all update records in the MOR table, thereby improving the data reading speed and reducing query delays.

[0047] If an update record merge file corresponding to the target object is stored, the initial data record of the target object is retrieved from the MOR table according to the identifier; the retrieved initial data record is sent to the data cache area, and the initial data record is merged with the update record merge file to generate the final target object data file in response to the read request. By introducing the data cache area and the read-time merge layer, the performance bottleneck problem of the traditional Hudi storage system when processing frequently updated data is solved. By caching the latest update record merge file, the need to traverse a large number of historical records each time reading is avoided, thereby improving query efficiency, reasonably allocating storage and computing resources, reducing unnecessary resource waste, and improving the overall performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A flowchart of a method for reading Hudi data provided by an embodiment of the present invention;

[0049] Figure 2 A schematic diagram of a module of a Hudi data reading device provided in one embodiment of the present invention;

[0050] Figure 3 A schematic diagram of the structure of an electronic device for implementing a method for reading Hudi data provided by an embodiment of the present invention;

[0051] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0053] It should be noted that the descriptions of "first", "second", etc. in the present invention are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0054] Reference Figure 1 FIG. 1 is a flow chart of a method for reading Hudi data provided by an embodiment of the present invention. The method is executed by an electronic device.

[0055] In this embodiment, the reading method based on Hudi data includes:

[0056] S1. Receive a request to access a target object from the Hudi storage system, and extract an identifier of the target object from the request.

[0057] In this embodiment, a request from a user or application to the Hudi storage system to access a target object is received, and an identifier of the target object is extracted from the request. The Hudi storage system refers to (Hadoop Upsert Data Incrementally) is an open source data storage framework, mainly used for incremental data processing applications built on the Apache Hadoop ecosystem. The Hudi storage system can simplify the support for data updates, deletions, and incremental processing in the big data processing process, and provides an efficient way to manage and process large-scale data sets, especially in application scenarios that need to support real-time or near real-time data updates and queries.

[0058] The target object refers to the specific data set or data record that the user or application wants to access or operate. In the financial application scenario of a bank, the target object is the customer's account information, including but not limited to account balance, account status and other information.

[0059] A request is a request made by a user or application to the Hudi storage system to obtain a specified target object stored in the Hudi storage system. The request contains some necessary information (such as the identifier of the target object) so that the Hudi storage system can identify and locate the data required by the user or application.

[0060] The identifier of a target object refers to one or a set of attributes or values ​​used to uniquely identify and locate the target object record. In the Hudi storage system, the identifier is the key information used to distinguish different data items, which can help the Hudi storage system accurately find the target object that the user or application wants to access.

[0061] For example:

[0062] Suppose there is a banking system that uses the Hudi storage system to manage customer account information. When a customer wants to query their account information, they can submit a request to read the account information through the bank's application or website. The request contains a unique identifier such as the customer's account number or ID number.

[0063] After the customer logs in to the bank's application or website, he enters his account number (Account Number) or ID number (ID Number), and then clicks to query the account information. After the bank's back-end Hudi storage system receives the request, it extracts the identifier from the request and then searches for the corresponding data record based on this identifier.

[0064] S2. Use the identifier to query the data cache area of ​​the Hudi storage system to determine whether the read-time merge layer of the data cache area stores an update record merge file corresponding to the target object, wherein the update record merge file is obtained by pre-traversing the update records associated with the target object in the MOR table of the Hudi storage system and merging the update records, and the MOR table stores the update records of each iteration of the target object.

[0065] In this embodiment, after receiving the identifier of the target object, the query interface provided by the data cache area is called, and the identifier is passed to the query interface as a query key. According to the passed-in identifier, the read-time merge layer is searched to see whether there is an update record merge file that matches the identifier. If an update record merge file that matches the identifier is found and its hash value is consistent with the hash value of the identifier, then it is considered that the update record merge file corresponding to the target object is stored in the data cache area.

[0066] If the hash value matches successfully, it means that the data cache does store the merged update record file corresponding to the target object. This indicates that before this, the Hudi storage system has completed the traversal and merging of all update records of the target object in the MOR table, and stored the results in the read-time merge layer of the cache.

[0067] In this way, if the cache hits, the Hudi storage system can directly use the update record merge file in the cache to generate the final target object data file without traversing the update records related to the target object in the MOR table again, thereby significantly improving data reading performance. This is especially important for banking scenarios, because banks need to frequently process a large number of account information queries, and any method that can speed up the query will greatly improve the user experience and service quality.

[0068] The data cache is a temporary area in the Hudi storage system that is used to store commonly used or recently frequently accessed data to increase data reading speed.

[0069] The read-time merge layer refers to a space area divided in the data cache area, which is specially used to store the merged update record files that have been merged. The merged update record file is generated by pre-traversing the update records associated with the target object in the MOR table of the Hudi storage system and merging the associated update records.

[0070] In the prior art, whenever the target object needs to be accessed, the Hudi storage system must traverse all update records related to the target object in the MOR table and merge these update records to generate the latest update record merge file. Since the update records are randomly stored in the MOR table in log format, the traversal and merging process consumes a lot of computing resources and time. Especially when data updates frequently and the number of historical update records is huge, the full traversal and merging method will cause a significant decrease in query performance, especially when the amount of data continues to grow.

[0071] One of the innovations of the present invention is that after detecting that the latest update record of the target object is stored or updated in the MOR table, all update records in the MOR table are pre-traversed, and these records are merged to generate an update record merge file. The generated update record merge file is stored in the read-time merge layer. In this way, when the user or application requests to access the target object again, the merged update record merge file can be directly obtained from the read-time merge layer without having to traverse and merge all records in the MOR table again.

[0072] Since the data cache area usually resides in memory and has a much faster access speed than the Hudi storage system with disk storage, setting the read-time merge layer in the data cache area can utilize the high-speed reading characteristics of the cache area, reduce dependence on the Hudi storage system, and thus improve reading efficiency.

[0073] The present invention traverses and merges the update records in the MOR table in advance, and stores the latest update record merge file in the read-time merge layer in the data cache area, effectively solving the performance problem caused by full traversal and merging in the prior art. It not only improves the efficiency of data reading, but also reduces the resource dependence on the Hudi storage system, thereby significantly improving the overall performance of the system when data updates frequently and the number of historical update records is huge.

[0074] In the present invention, the data cache area of ​​the Hudi storage system is constructed as follows:

[0075] First, analyze which hot data is most frequently accessed. In the financial application scenario of a bank, hot data (such as the latest transaction records, frequently updated user information, etc.) are the best candidates for caching.

[0076] Determine the size of the data cache area based on hot data, available resources of the Hudi storage system, and expected cache hit rate, and select a suitable cache mechanism for the data cache area. The cache mechanism includes memory cache, distributed cache, etc. In the financial application scenario of the bank, it is suitable for large distributed environments, such as Redis, Memcached, etc. Memory cache is suitable for single-machine environments or small clusters, such as using Java's local cache library (such as Caffeine). Set the configuration parameters of the cache server in the Hudi storage system (for example, set the maximum memory, memory overflow policy, and network connection parameters, etc.) to obtain the data cache area of ​​the Hudi storage system.

[0077] In one embodiment, the querying of the data cache area of ​​the Hudi storage system using the identifier to determine whether the read-time merge layer of the data cache area stores a merged file of update records corresponding to the target object includes:

[0078] Using the identifier as a query key of a preset query interface of the data cache;

[0079] querying in the read-time merge layer whether there is an update record merge file matching the query key stored therein according to the preset query interface;

[0080] If the hash value of the merged update record file is consistent with the hash value of the query key, it is determined that the merged update record file corresponding to the target object is stored.

[0081] Extract the identifier of the target object (such as account number, ID number, etc.) from the request. This identifier is the key information that uniquely identifies the target object. Use the extracted identifier as the query key of the query interface. The query key is the key information used by the Hudi storage system to locate the update record merge file.

[0082] The query interface searches the read-time merge layer for an update record merge file that matches the identifier according to the passed identifier. The read-time merge layer stores the update record merge files that have been merged.

[0083] During the query process, the Hudi storage system will confirm whether the hash value of the update record merge file is consistent with the hash value of the query key. The consistency check of the hash value is to ensure that the data found is indeed the target object requested.

[0084] If the hash value of the update record merge file is consistent with the hash value of the query key, it means that the update record merge file of the target object is indeed stored in the read-time merge layer. If they are inconsistent, it means that the required data is not in the data cache.

[0085] The present invention provides an efficient query mechanism that can improve the performance of the Hudi storage system when reading data. By using a pre-set query interface and using the identifier of the target object as the query key, the system can quickly locate the update record merge file in the cache area, avoiding traversing the entire MOR table, and greatly reducing the query time. Since the query interface is specially designed for the cache area, it can process query requests more efficiently, thereby reducing the consumption of computing resources and improving the overall performance of the system.

[0086] By comparing the hash value of the update record merge file with the hash value of the query key, we can ensure that the queried data is indeed the target object, which helps to ensure the consistency and accuracy of the data. By storing the update record merge file in the cache area, we can greatly reduce direct access to the MOR table, which not only reduces the load on the Hudi storage system, but also improves the overall response speed of the system.

[0087] For example:

[0088] Suppose in a banking system, a customer wants to check his account balance. The customer initiates a read request through the banking application, and the request contains the account number (for example: 1234567890) as an identifier. The account number 1234567890 is extracted from the read request, and the pre-set query interface is called, and the account number 1234567890 is used as the query key. The query interface uses the account number 1234567890 to find out whether there is an updated record merge file matching the account number in the read-time merge layer. If matching data records are found, the system compares the hash values ​​of these records with the hash value of the account number 1234567890. If the hash values ​​are consistent, it confirms that the latest merged state of the account is stored in the read-time merge layer; if they are inconsistent, it indicates that the required data is not available in the data cache.

[0089] S3. If the update record merge file is stored, the initial data record of the target object is retrieved from the Hudi storage system according to the identifier, the retrieved initial data record is sent to the read-time merge layer, and the preset merge function of the read-time merge layer is used to merge the initial data record with the record of the update record merge file to obtain the data file corresponding to the target object and respond to the request.

[0090] In this embodiment, after confirming that the update record merge file of the target object exists in the data cache, the identifier of the target object is used to locate the Parquet document storing the target object in the MOR table. The Parquet document is an efficient columnar storage format that is very suitable for storing and processing large-scale data sets.

[0091] After locating the corresponding Parquet document, the initial data record of the initial data corresponding to the target object is retrieved from the Parquet document. The initial data record refers to the data state after the initial data corresponding to the target object is stored in the Hudi storage system and compressed, which is used to save the storage space of the Hudi storage system.

[0092] By storing the update record merge file (usually a small file) in the data cache, and storing the initial data record (usually a large file) in the MOR table of the Hudi storage system, more efficient reading can be achieved. This is because the data cache is usually located in memory or on high-speed storage media, with faster access speeds; while the MOR table may be located on HDFS (Hadoop Distributed File System), which is suitable for long-term storage of large amounts of data.

[0093] In addition, by placing frequently accessed small files (update record merge files) in the data cache, the frequency of access to HDFS is reduced, thereby reducing I / O overhead and improving overall data reading performance.

[0094] For example:

[0095] Suppose a bank customer wants to check his latest account balance. After receiving the query request, the Hudi storage system confirms that the latest merged data of the account exists in the cache by the account number (identifier). Then, the Hudi storage system uses the same account number to locate the Parquet document in the MOR table and retrieve the initial data of the account from it. In this way, the Hudi storage system can quickly combine the initial data record with the updated record merge file in the cache to generate the latest account balance information.

[0096] Extract the identifier of the target object (such as account number, ID number, etc.) from the read request, locate and retrieve the compressed information of the target object in the MOR table according to the identifier, send the retrieved initial data record to the data cache, decompress the initial data record, restore the state of the initial data, merge the decompressed initial data record with the update record merge file, and generate the final target object.

[0097] For example:

[0098] Assume that a bank customer queries his account information through a banking application. The specific steps are as follows:

[0099] The customer submits a read request through the bank application or website, and the request contains the customer's account number as an identifier. After receiving the read request, the bank's backend system extracts the customer's account number from it, uses the account number as an identifier, and locates and retrieves the compressed information of the account in the MOR table of the Hudi storage system. Assuming that the compressed information is stored in a Parquet document, the system will find the corresponding Parquet document and retrieve the initial data record.

[0100] The retrieved initial data records are sent to the read-time merge layer. At this point, the read-time merge layer already has the merged file of the account's update records (if there has been an update and merge before), and the initial data records are decompressed in the cache to restore the original state of the account information. For example, the initial data may contain basic account information, such as account balance, account opening date, etc.

[0101] The decompressed initial data is merged with the update record merge file in the cache using the preset merge function of the read-time merge layer. The specific steps include: obtaining the latest update records, which contain the most recent account changes, such as deposits, withdrawals, etc., applying these latest update records to the initial data, such as updating the account balance. Generate the final target object (i.e. the latest account information). After the final account information is generated, the system returns it to the client in response to the user's read request.

[0102] In one embodiment, the MOR table is obtained according to the following method:

[0103] Create a Hive table in a preset storage path of the Hudi storage system, wherein the Hive table is used to organize and query storage units of initial data stored in the Hudi storage system;

[0104] Configure the MOR table type function for the Hive table to obtain the MOR table of the Hudi storage system;

[0105] The initial data of the target object is stored in the Hudi storage system to obtain an initial data record, and the initial data record is stored in the MOR table.

[0106] First determine the preset storage path of the Hudi storage system. The preset storage path can be an HDFS (Hadoop Distributed File System) path, which is used to specify the actual physical storage location of the Hive table. To create a Hive table in the preset storage path of the Hudi storage system, you need to define the table structure (column name, data type, etc.) of the Hive table so that the Hive table meets the storage requirements of the Hudi storage system so that it can be converted to a MOR table later.

[0107] Configure the newly created Hive table to support the MOR (Merge-on-Read) function, such as setting Hudi-specific table properties, table type (hoodie.table.type), write type (hoodie.write.operation), etc.

[0108] Also set the table type to MOR (hoodie.table.type = MOR) and configure other necessary Hudi parameters, such as primary key field, partition field, etc. After completing the above configuration, the Hive table is converted into a MOR table in the Hudi storage system.

[0109] The initial data of the target object is stored in the Hudi storage system to obtain an initial data record, and the initial data record is stored in the MOR table.

[0110] In one embodiment, storing the initial data of the target object in the Hudi storage system to obtain the initial data record includes:

[0111] Preprocessing the initial data to obtain preprocessed initial data;

[0112] Extract key fields from the preprocessed initial data, wherein the key fields include a timestamp and a version number;

[0113] The key field is used as compression description information of the preprocessed initial data, and the initial data is compressed according to the compression description information to obtain the initial data record.

[0114] The original data record is a record generated after the original data of the target object is stored in the Hudi storage system for the first time.

[0115] Preprocess the initial data, including data cleaning, format standardization, and removal of redundant information. Data cleaning includes removing blank characters, invalid characters, or data with incorrect formats from the initial data. Format standardization includes converting the initial data into a unified format for subsequent processing. For example, converting a date and time string into a standard timestamp format. Removing redundant information includes removing unnecessary fields or information and retaining only the core data to reduce the amount of data.

[0116] Identify key fields in the preprocessed initial data. Key fields include timestamps and version numbers. Key fields help track the change history of the target object. Extract these key fields from the preprocessed data. For example, extract timestamps and version numbers from each record.

[0117] Combine the extracted key fields into compressed description information. These description information contain timestamps and version numbers to describe the version status of the data. Use a preset compression algorithm (for example, Gzip, Snappy, LZ4) to compress the preprocessed initial data. During the compression process, the compressed description information can be stored together as metadata for subsequent decompression. Store the compressed initial data records in the Parquet document of the MOR table to ensure that the initial data records are organized according to Hudi's requirements for subsequent efficient query and management.

[0118] In one embodiment, after storing the initial data record in the MOR table, the method further comprises:

[0119] Create a .Log document in the MOR table, wherein the .Log document is used to record a log file of changes to the initial data operation;

[0120] Recording a data operation of a preset type performed on the initial data to obtain an update record of the initial data record;

[0121] Each update record to the initial data record is stored in the .Log file.

[0122] A .Log document is created in the preset storage path of the MOR table. The preset storage path of the MOR table can be a Hadoop Distributed File System (HDFS) path, which is used to specify the actual physical storage location of the MOR table. The created .Log document is used to record all operation records of the initial data record.

[0123] .Log documents are log files used to record changes to stored data. Log files usually contain a series of operation records, each of which describes a change in the data, such as the timestamp of an update operation, data records before and after the change, and other information.

[0124] Record each data operation of a preset type on the initial data to obtain each update record of the initial data record. The preset types of operations include insert, update, delete, etc. Whenever these data operations are performed, record the timestamp of the operation, the type of operation, the data identifier involved (such as account number, transaction ID, etc.), version number, and other related information.

[0125] Convert detailed information of each data operation into update records that are easy to understand and store. For example, convert update records into log records, and write the converted log records into .Log documents to ensure that the records of each operation are completely saved.

[0126] By recording each operation on the initial data record and storing these operation records in the .Log file, the data operation history can be tracked and managed. This helps to conduct audits or restore data when needed later.

[0127] In one embodiment, the read-time merge layer of the data cache area is obtained according to the following method:

[0128] Divide a spatial area in the data buffer area as an initial layer;

[0129] After detecting that the initial data record generates an update record, all update records of the initial data record are traversed in the MOR table and merged according to the timestamp and version number of the update record to obtain an update record merge file;

[0130] The update record merge file and the preset merge function are stored in the initial layer to obtain the read-time merge layer.

[0131] A spatial area is divided in the data cache as an initial layer. The initial layer can be an area in the data cache (for example, the preset storage path is / cache / transactions) for storing frequently accessed data. The initial layer is used to store update record merge files, which usually contain identifiers, version merged data, and other necessary metadata (such as timestamps and version numbers).

[0132] Set up a listener to detect any updated records in the MOR table. Whenever new updated records are generated in the MOR table, the listener will capture these updates and record the timestamp and version number of each updated record through the listener.

[0133] According to the timestamp, version number and identifier of the update record, all related update records of the initial data record are traversed in the MOR table, and all the traversed update records are merged. The merging process includes integrating all versions of data into the latest state.

[0134] The update record merge file and the preset merge function are stored in the initial layer to obtain the read-time merge layer. If the initial layer stores the previous update record merge file of the initial data record, the latest update record merge file will overwrite the previous update record merge file. If the update record merge file is stored in the initial layer for the first time, it can be directly stored.

[0135] A preset merge function is a function or algorithm used to merge different versions of data. In the Hudi framework, the preset merge function is used to handle the differences between different versions of the same data record and determine the logic of the final data state.

[0136] For example:

[0137] Assume that a bank needs to maintain customer account information. Every time the account information changes (such as deposits, withdrawals, transfers, etc.), the account status needs to be updated. The following are the specific steps:

[0138] Create an initial layer in the data cache area, the path is / cache / transactions, which is used to store the update record merge file of the customer's account information.

[0139] Set up a listener on the MOR table of the Hudi storage system to listen to all update records about account information. Whenever the account information changes, the listener captures these changes and records the relevant timestamps and version numbers. When the listener detects that the account information has changed, it traverses all relevant update records in the MOR table according to the timestamp and version number of the record. For example, a customer's account has multiple transactions in a day, and all these transaction records need to be traversed. Merge all traversed update records into the latest account status. For example, if a customer first deposits 1,000 yuan and then withdraws 500 yuan, the latest account balance should be an increase of 500 yuan. Store the latest update record merge file in the initial layer in the cache area. If the customer's account information already exists, overwrite the old data with the latest update record merge file; if this is the first storage, add it directly to the initial layer.

[0140] In one embodiment, the merging the initial data record and the update record merge file using the preset merge function of the read-time merge layer to generate a result record corresponding to the target object includes:

[0141] Parse the update record merge file to obtain the latest update record of the target object, wherein the latest update record includes a record key and an updated new value;

[0142] Decompress the initial data record to obtain the initial data of the target object, and query the corresponding row of the latest updated record in the initial data according to the record key and the preset merge function;

[0143] The updated new value replaces the original value on the corresponding row to obtain the final target object.

[0144] Parse the update record merge file and extract the latest update record of the target object. The latest update record includes the record key and the updated new value.

[0145] The initial data records retrieved from the MOR table are decompressed to restore the state of the initial data. According to the record key in the latest update record and the preset merge function, the corresponding row is queried in the initial data, and the new value in the latest update record replaces the original value on the corresponding row in the initial data.

[0146] The preset merge function is used to process data update records to generate the latest update record merge file. Specifically, it is an algorithm or function used to solve the merging logic of the same data record after multiple updates. In the Hudi (Hadoop Universal Data Index) framework, the preset merge function is used to process the differences between different versions of the same data record and determine the final data state.

[0147] The role of the preset merge function: When the same data record is updated multiple times, the preset merge function is responsible for determining which update record's data is valid. For example, in the scenario of a bank account, if an account has multiple deposit and withdrawal operations in a short period of time, the preset merge function needs to determine the latest account balance. The preset merge function merges all related update records into a latest update record merge file. For example, if an account first deposits 1,000 yuan and then withdraws 500 yuan, the preset merge function needs to merge these two operations and calculate the final balance. By applying the preset merge function when reading data, you can avoid re-traversing all update records for each read, thereby improving reading performance. In particular, when there is no update record merge file in the data cache, the preset merge function can quickly generate the latest update record merge file when reading.

[0148] The present invention merges the initial data record and the update record merge file in the data cache area, simplifies the data processing flow by decompressing the initial data record and replacing only the latest update value instead of regenerating the entire data set, and performs the merging in the data cache area. By utilizing the characteristics of the data cache area (located in the memory or on a high-speed storage medium), the data processing speed can be improved, which not only saves processing time but also reduces memory usage.

[0149] S4. If the update record merge file is not stored, traverse the update records associated with the target object in the MOR table of the Hudi storage system, merge all update records associated with the target object to generate an update record merge file, retrieve the initial data record of the target object from the Hudi storage system according to the identifier, send the update record merge file and the initial data record to the read-time merge layer for merging, obtain the data file corresponding to the target object and respond to the request.

[0150] If no update record merge file matching the target object identifier in the request is found in the data cache, it is necessary to traverse the MOR table (Merge-on-Read table) in the Hudi storage system. The purpose of the traversal is to find all update records associated with the target object. These update records contain all historical change information of the target object.

[0151] In the process of traversing the MOR table, all update records related to the target object are collected and merged to generate an update record merge file. The merge here means merging multiple updates into a latest state according to certain rules (such as timestamp, version number, etc.). By merging the update records, a merged update record file containing all the latest changes is generated. This data represents the current state of the target object.

[0152] At the same time, it is also necessary to retrieve the initial data records of the target object from the Hudi storage system, and send the generated update record merge file and the retrieved initial data records to the read-time merge layer of the data cache. The read-time merge layer is a design that optimizes read performance. It can temporarily store data in memory or other high-speed storage to quickly respond to read requests.

[0153] In the read-time merge layer, the initial data record is merged with the update record merge file using the preset merge function. The preset merge function is a logic or algorithm that determines how to merge the initial data with the update record to generate the final data state. Through the merge operation, the final target object is generated. This means that the system has generated a data set that reflects the current state based on the latest update record and the initial data. Finally, the generated final target object is returned to the application or user who originally issued the request, completing the processing of the read request.

[0154] In summary, in steps S1-S4, the present invention pre-generates an update record merge file and stores it in the data cache area. When a read request arrives, the latest data in the cache can be directly used, avoiding traversing all update records in the MOR table, thereby improving the data reading speed, reducing query delays, reasonably allocating storage and computing resources, reducing unnecessary waste of resources, and improving the overall performance of the system.

[0155] like Figure 2 As shown, it is a module schematic diagram of a Hudi data reading device provided by an embodiment of the present invention.

[0156] The Hudi data-based reading device 100 of the present invention can be installed in an electronic device. According to the implemented functions, the Hudi data-based reading device 100 may include a receiving module 110, a determining module 120, a retrieval module 130 and a traversal module 140. The module of the present invention may also be referred to as a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.

[0157] In this embodiment, the functions of each module / unit are as follows:

[0158] A receiving module 110 is used to receive a request to access a target object from the Hudi storage system, and extract an identifier of the target object from the request;

[0159] The determination module 120 is used to query the data cache area of ​​the Hudi storage system using the identifier to determine whether the read-time merge layer of the data cache area stores an update record merge file corresponding to the target object, wherein the update record merge file is obtained by pre-traversing the update records associated with the target object in the MOR table of the Hudi storage system and merging the update records, and the MOR table stores the update records of each iteration of the target object;

[0160] A retrieval module 130 is configured to retrieve the initial data record of the target object from the Hudi storage system according to the identifier if the update record merge file is stored, send the retrieved initial data record to the read-time merge layer, merge the initial data record with the record of the update record merge file using a preset merge function of the read-time merge layer, obtain the data file corresponding to the target object and respond to the request;

[0161] The traversal module 140 is used to traverse the update records associated with the target object in the MOR table of the Hudi storage system if the update record merge file is not stored, merge all update records associated with the target object to generate an update record merge file, retrieve the initial data record of the target object from the Hudi storage system according to the identifier, send the update record merge file and the initial data record to the read-time merge layer for merging, obtain the data file corresponding to the target object and respond to the request.

[0162] like Figure 3 As shown, it is a schematic diagram of the structure of an electronic device for implementing a Hudi data reading method provided by an embodiment of the present invention.

[0163] In this embodiment, the electronic device 1 includes, but is not limited to, a memory 11, a processor 12, and a network interface 13, which can be interconnected and communicated via a system bus. The memory 11 stores a reading program 10 based on Hudi data, and the reading program 10 based on Hudi data can be executed by the processor 12. Figure 3 Only an electronic device 1 having components 11-13 and a reading program 10 based on Hudi data is shown, and it can be understood by those skilled in the art that Figure 3 The structure shown does not constitute a limitation on the electronic device 1 , and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0164] Among them, the memory 11 includes a memory and at least one type of readable storage medium. The memory provides a cache for the operation of the electronic device 1; the readable storage medium may be a non-volatile storage medium such as a flash memory, a hard disk, a multimedia card, a card-type memory (for example, SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the readable storage medium may be an internal storage unit of the electronic device 1; in other embodiments, the non-volatile storage medium may also be an external storage device of the electronic device 1, such as a plug-in hard disk equipped on the electronic device 1, a smart memory card (SmartMediaCard, SMC), a secure digital (SecureDigital, SD) card, a flash card (FlashCard), etc. In this embodiment, the readable storage medium of the memory 11 is generally used to store an operating system and various application software installed on the electronic device 1, such as storing the code of the Hudi data-based reading program 10 in one embodiment of the present invention. In addition, the memory 11 can also be used to temporarily store various types of data that have been output or are to be output.

[0165] The processor 12 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 12 is generally used to control the overall operation of the electronic device 1, such as performing control and processing related to data interaction or communication with other devices. In this embodiment, the processor 12 is used to run the program code stored in the memory 11 or process data, such as running a reading program 10 based on Hudi data.

[0166] The network interface 13 may include a wireless network interface or a wired network interface, and the network interface 13 is used to establish a communication connection between the electronic device 1 and a terminal (not shown in the figure).

[0167] Optionally, the electronic device 1 may further include a user interface, which may include a display (Display), an input unit such as a keyboard (Keyboard), and the optional user interface may also include a standard wired interface and a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device 1 and to display a visual user interface.

[0168] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0169] The Hudi data-based reading program 10 stored in the memory 11 in the electronic device 1 is a combination of multiple instructions. When running in the processor 12, it can achieve:

[0170] Receive a request to the Hudi storage system to access a target object, and extract an identifier of the target object from the request;

[0171] The data cache area of ​​the Hudi storage system is queried using the identifier to determine whether the read-time merge layer of the data cache area stores a merged file of update records corresponding to the target object, wherein the merged file of update records is obtained by traversing the update records associated with the target object in the MOR table of the Hudi storage system in advance and merging the update records, and the MOR table stores the update records of each iteration of the target object;

[0172] If the update record merge file is stored, the initial data record of the target object is retrieved from the Hudi storage system according to the identifier, the retrieved initial data record is sent to the read-time merge layer, and the initial data record and the record of the update record merge file are merged using the preset merge function of the read-time merge layer to obtain the data file corresponding to the target object and respond to the request;

[0173] If the update record merge file is not stored, the update records associated with the target object in the MOR table of the Hudi storage system are traversed, all update records associated with the target object are merged to generate an update record merge file, the initial data record of the target object is retrieved from the Hudi storage system according to the identifier, the update record merge file and the initial data record are sent to the read-time merge layer for merging, the data file corresponding to the target object is obtained and the request is responded to.

[0174] Specifically, the specific implementation method of the processor 12 for the above-mentioned Hudi data-based reading program 10 can be referred to Figure 1 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0175] Furthermore, if the module / unit integrated in the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable medium can be non-volatile or non-volatile. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0176] The computer-readable storage medium stores a reading program 10 based on Hudi data, and the reading program 10 based on Hudi data can be executed by one or more processors. The specific implementation method of the computer-readable storage medium of the present invention is basically the same as the above-mentioned embodiments of the reading method based on Hudi data, which will not be repeated here.

[0177] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0178] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0179] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0180] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0181] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0182] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The second and other words are used to indicate names, but not to indicate any particular order.

[0183] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for reading Hudi data, characterized in that: The method comprises: Receive a request to the Hudi storage system to access a target object, and extract an identifier of the target object from the request; The data cache area of ​​the Hudi storage system is queried using the identifier to determine whether the read-time merge layer of the data cache area stores a merged file of update records corresponding to the target object, wherein the merged file of update records is obtained by traversing the update records associated with the target object in the MOR table of the Hudi storage system in advance and merging the update records, and the MOR table stores the update records of each iteration of the target object; If the update record merge file is stored, the initial data record of the target object is retrieved from the Hudi storage system according to the identifier, the retrieved initial data record is sent to the read-time merge layer, and the initial data record and the record of the update record merge file are merged using the preset merge function of the read-time merge layer to obtain the data file corresponding to the target object and respond to the request; If the update record merge file is not stored, the update records associated with the target object in the MOR table of the Hudi storage system are traversed, all update records associated with the target object are merged to generate an update record merge file, the initial data record of the target object is retrieved from the Hudi storage system according to the identifier, the update record merge file and the initial data record are sent to the read-time merge layer for merging, the data file corresponding to the target object is obtained and the request is responded to.

2. The method for reading Hudi data according to claim 1, characterized in that: The using the identifier to query the data cache area of ​​the Hudi storage system to determine whether the read-time merge layer of the data cache area stores the update record merge file corresponding to the target object, includes: Using the identifier as a query key of a preset query interface of the data cache; querying in the read-time merge layer whether there is an update record merge file matching the query key stored therein according to the preset query interface; If the hash value of the merged update record file is consistent with the hash value of the query key, it is determined that the merged update record file corresponding to the target object is stored.

3. The method for reading Hudi data according to claim 1, characterized in that: The MOR table is obtained according to the following method: Create a Hive table in a preset storage path of the Hudi storage system, wherein the Hive table is used to organize and query storage units of initial data stored in the Hudi storage system; Configure the MOR table type function for the Hive table to obtain the MOR table of the Hudi storage system; The initial data of the target object is stored in the Hudi storage system to obtain an initial data record, and the initial data record is stored in the MOR table.

4. The method for reading Hudi data according to claim 3, characterized in that: The initial data of the target object is stored in the Hudi storage system to obtain the initial data record, including: Preprocessing the initial data to obtain preprocessed initial data; Extract key fields from the preprocessed initial data, wherein the key fields include a timestamp and a version number; The key field is used as compression description information of the preprocessed initial data, and the initial data is compressed according to the compression description information to obtain the initial data record.

5. The method for reading Hudi data according to claim 3 or 4, characterized in that: After storing the initial data record in the MOR table, the method further comprises: Create a .Log document in the MOR table, wherein the .Log document is used to record a log file of changes to the initial data operation; Recording a data operation of a preset type performed on the initial data to obtain an update record of the initial data record; Each update record to the initial data record is stored in the .Log file.

6. The method for reading Hudi data according to claim 1 or 3, characterized in that: The read-time merge layer of the data cache area is obtained according to the following method: Divide a spatial area in the data buffer area as an initial layer; After detecting that the initial data record generates an update record, all update records of the initial data record are traversed in the MOR table and merged according to the timestamp and version number of the update record to obtain an update record merge file; The update record merge file and the preset merge function are stored in the initial layer to obtain the read-time merge layer.

7. The method for reading Hudi data according to claim 1, characterized in that: The step of merging the initial data record and the update record using the preset merging function of the read-time merging layer to generate a result record corresponding to the target object includes: Parse the update record merge file to obtain the latest update record of the target object, wherein the latest update record includes a record key and an updated new value; Decompress the initial data record to obtain the initial data of the target object, and query the corresponding row of the latest updated record in the initial data according to the record key and the preset merge function; The updated new value replaces the original value on the corresponding row to obtain the final target object.

8. A reading device based on Hudi data, characterized in that: The device comprises: A receiving module, configured to receive a request to access a target object from the Hudi storage system, and extract an identifier of the target object from the request; A determination module is used to query the data cache area of ​​the Hudi storage system using the identifier to determine whether the read-time merge layer of the data cache area stores an update record merge file corresponding to the target object, wherein the update record merge file is obtained by pre-traversing the update records associated with the target object in the MOR table of the Hudi storage system and merging the update records, and the MOR table stores the update records of each iteration of the target object; A retrieval module, configured to retrieve the initial data record of the target object from the Hudi storage system according to the identifier if the update record merge file is stored, send the retrieved initial data record to the read-time merge layer, merge the initial data record with the record of the update record merge file using a preset merge function of the read-time merge layer, obtain the data file corresponding to the target object and respond to the request; A traversal module is used to traverse the update records associated with the target object in the MOR table of the Hudi storage system if the update record merge file is not stored, merge all update records associated with the target object to generate an update record merge file, retrieve the initial data record of the target object from the Hudi storage system according to the identifier, send the update record merge file and the initial data record to the read-time merge layer for merging, obtain the data file corresponding to the target object and respond to the request.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a Hudi data-based reading program that can be executed by the at least one processor, and the Hudi data-based reading program is executed by the at least one processor so that the at least one processor can execute the Hudi data-based reading method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a reading program based on Hudi data, and the reading program based on Hudi data can be executed by one or more processors to implement the reading method based on Hudi data as described in any one of claims 1 to 7.

Images