Hudi data-based reading method and device, electronic equipment and storage medium

Abstract

The present application relates to the field of data request processing, and discloses a reading method based on Hudi data, comprising receiving a request for accessing a target object to a Hudi storage system, extracting an identifier of the target object from the reading request; querying a data cache area of the Hudi storage system by using the identifier to determine whether an update record merge file of the target object is stored in a read-time merge layer of the data cache area; if the update record merge file corresponding to the target object is stored, retrieving initial data records of the target object from a MOR table according to the identifier; sending the retrieved initial data records to the data cache area, merging the initial data records with the update record merge file, and generating a final data file of the target object to respond to the reading request. By introducing the data cache area and the read-time merge layer, the present application avoids the need to traverse a large number of historical records each time reading, thereby improving the query efficiency.

Description

Technical Field

[0001] This invention relates to the field of data request processing, and more particularly to a method, apparatus, electronic device, and storage medium for reading Hudi data. Background Technology

[0002] Hudi (Hadoop Upsert Data Incrementally), part of the Hadoop ecosystem, is an open-source data storage system that has gradually gained widespread application in industries such as e-commerce, insurance, and finance.

[0003] Traditional Hudi storage systems rely on MOR (Merge-on-Read) tables when reading data. MOR tables typically store records of each update to the initial data. While this approach ensures historical traceability of data, it also presents challenges for query performance. When a user reads data from the MOR table, it is necessary to traverse the records of each update to the initial data in the MOR table and merge each update record.

[0004] When data is updated frequently and the number of historical update records is huge, this method of full traversal and merging will lead to a significant decrease in query performance as the amount of data increases.

[0005] Therefore, improving data retrieval performance is a critical technical issue that needs to be addressed in the Hudi data storage system. Summary of the Invention

[0006] In view of the above, it is necessary to provide a method for reading Hudi data, the purpose of which is to improve the performance of reading data in the Hudi data storage system.

[0007] The Hudi data reading method provided by this invention includes:

[0008] Receive a request to access a target object in the Hudi storage system, and extract the identifier of the target object from the request;

[0009] The identifier is used to query the data cache of the Hudi storage system to determine whether the read-time merging layer of the data cache stores the update record merge file corresponding to the target object. The update record merge file is obtained by merging the update records associated with the target object in the MOR table of the Hudi storage system in advance. The MOR table stores the update records of the target object for each iteration.

[0010] If the update record merge file is stored, the initial data record of the target object is retrieved from the Hudi storage system according to the identifier, the retrieved initial data record is sent to the read-time merging layer, and the initial data record and the record of the update record merge file are merged using the preset merging function of the read-time merging layer to obtain the data file corresponding to the target object and respond to the request.

[0011] If no update record merge file is stored, the update records associated with the target object in the MOR table of the Hudi storage system are traversed, all update records associated with the target object are merged to generate an update record merge file, the initial data record of the target object is retrieved from the Hudi storage system according to the identifier, the update record merge file and the initial data record are sent to the read-time merging layer for merging, the data file corresponding to the target object is obtained, and the request is responded to.

[0012] Optionally, the step of querying the data cache of the Hudi storage system using the identifier to determine whether the read-time merging layer of the data cache stores an update record merge file corresponding to the target object includes:

[0013] Use the identifier as the query key for the preset query interface of the data cache area;

[0014] According to the preset query interface, query the read-time merging layer to see if there is an update record merge file that matches the query key;

[0015] If the hash value of a merged update record file matches the hash value of the query key, then it is determined that a merged update record file corresponding to the target object is stored.

[0016] Optionally, the MOR table is obtained as follows:

[0017] Create a Hive table in the preset storage path of the Hudi storage system. The Hive table is used to organize and query the storage units of the initial data stored in the Hudi storage system.

[0018] Configure the Hive table with the MOR table type to obtain the MOR table of the Hudi storage system;

[0019] The initial data of the target object is stored in the Hudi storage system to obtain the initial data record, and the initial data record is stored in the MOR table.

[0020] Optionally, storing the initial data of the target object in the Hudi storage system to obtain initial data records includes:

[0021] The initial data is preprocessed to obtain preprocessed initial data;

[0022] Extract key fields from the preprocessed initial data, including timestamps and version numbers;

[0023] The key fields are used as compressed description information of the preprocessed initial data. The initial data is compressed according to the compressed description information to obtain the initial data record.

[0024] Optionally, the method further includes, after storing the initial data record in the MOR table:

[0025] Create a .Log document in the MOR table. The .Log document is used to record log changes to the initial data operations.

[0026] The record performs a preset type of data operation on the initial data to obtain an update record of the initial data record;

[0027] Each update to the initial data record is stored in the .Log document.

[0028] Optionally, the read-time merging layer of the data buffer is obtained in the following manner:

[0029] A spatial region is allocated in the data cache area as an initial layer;

[0030] After detecting that the initial data record has generated an update record, according to the timestamp and version number of the update record, all update records of the initial data record are traversed in the MOR table and merged to obtain the update record merge file;

[0031] The updated record merge file and the preset merge function are stored in the initial layer to obtain the read-time merge layer.

[0032] Optionally, the step of merging the initial data record and the updated record merge file using the preset merge function of the read-time merging layer to generate the result record corresponding to the target object includes:

[0033] Parse the merged update record file to obtain the latest update record of the target object. The latest update record includes a record key and the updated new value.

[0034] Decompress the initial data record to obtain the initial data of the target object, and query the corresponding row of the latest updated record in the initial data according to the record key and the preset merge function;

[0035] Replace the original value in the corresponding row with the updated new value to obtain the final target object.

[0036] To address the above problems, the present invention also provides a Hudi data reading device, the device comprising:

[0037] The receiving module is used to receive a request to access a target object from the Hudi storage system and extract the identifier of the target object from the request;

[0038] The determination module is used to query the data cache of the Hudi storage system using the identifier to determine whether the read-time merging layer of the data cache stores the update record merge file corresponding to the target object. The update record merge file is obtained by merging the update records associated with the target object in the MOR table of the Hudi storage system in advance. The MOR table stores the update records of the target object for each iteration.

[0039] The retrieval module is used to retrieve the initial data record of the target object from the Hudi storage system according to the identifier if the update record merge file is stored, send the retrieved initial data record to the read-time merging layer, and use the preset merging function of the read-time merging layer to merge the initial data record with the record of the update record merge file to obtain the data file corresponding to the target object and respond to the request.

[0040] The traversal module is used to traverse the update records associated with the target object in the MOR table of the Hudi storage system if no update record merge file is stored, merge all update records associated with the target object to generate an update record merge file, retrieve the initial data record of the target object from the Hudi storage system according to the identifier, send the update record merge file and the initial data record to the read-time merging layer for merging, obtain the data file corresponding to the target object, and respond to the request.

[0041] To address the above problems, the present invention also provides an electronic device, the electronic device comprising:

[0042] At least one processor; and,

[0043] A memory communicatively connected to the at least one processor; wherein,

[0044] The memory stores a Hudi-based data reading program that can be executed by the at least one processor, which is then executed by the at least one processor to enable the at least one processor to perform the Hudi-based data reading method described above.

[0045] To address the aforementioned problems, the present invention also provides a computer-readable storage medium storing a Hudi-based data reading program, which can be executed by one or more processors to implement the aforementioned Hudi-based data reading method.

[0046] Compared to existing technologies, this invention receives a request to access a target object in the Hudi storage system, extracts the identifier of the target object from the request, and uses the identifier to query the data cache of the Hudi storage system to determine whether the read-time merging layer of the data cache stores an update record merge file corresponding to the target object. By pre-generating the update record merge file and storing it in the data cache, the latest data in the cache can be used directly when the access request arrives, avoiding traversing all update records in the MOR table, thereby improving the data reading speed and reducing query latency.

[0047] If a merged update record file corresponding to the target object is stored, the initial data record of the target object is retrieved from the MOR table based on the identifier; the retrieved initial data record is sent to the data cache, and the initial data record is merged with the merged update record file to generate the final data file of the target object in response to the read request. By introducing a data cache and a read-time merging layer, the performance bottleneck problem of traditional Hudi storage systems when processing frequently updated data is solved. By caching the latest merged update record file, the need to traverse a large number of historical records for each read is avoided, thereby improving query efficiency, rationally allocating storage and computing resources, reducing unnecessary resource waste, and improving the overall performance of the system. Attached Figure Description

[0048] Figure 1 This is a flowchart illustrating a Hudi-based data reading method according to an embodiment of the present invention.

[0049] Figure 2 This is a schematic diagram of a module of a Hudi-based data reading device provided in an embodiment of the present invention;

[0050] Figure 3 This is a schematic diagram of the structure of an electronic device that implements a Hudi data reading method according to an embodiment of the present invention;

[0051] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0053] It should be noted that the descriptions involving "first," "second," etc., in this invention are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0054] Reference Figure 1 The diagram shown is a flowchart illustrating a Hudi-based data reading method according to an embodiment of the present invention. This method is executed by an electronic device.

[0055] In this embodiment, the method for reading Hudi data includes:

[0056] S1. Receive a request to access the target object from the Hudi storage system, and extract the identifier of the target object from the request.

[0057] In this embodiment, a request from a user or application to access a target object in the Hudi storage system is received, and the identifier of the target object is extracted from the request. Hudi storage system (Hadoop Upsert Data Incrementally) is an open-source data storage framework primarily used for incremental data processing applications built on the Apache Hadoop ecosystem. Hudi storage system simplifies support for data updates, deletions, and incremental processing in big data processing, providing an efficient way to manage and process large-scale datasets, especially in application scenarios requiring real-time or near-real-time data updates and queries.

[0058] A target object refers to a specific dataset or data record that a user or application wants to access or manipulate. In banking and financial applications, the target object is the customer's account information, including but not limited to account balance and account status.

[0059] A request is a statement made by a user or application to the Hudi storage system to retrieve a specified target object stored within it. The request includes essential information (such as the identifier of the target object) so that the Hudi storage system can identify and locate the data needed by the user or application.

[0060] An identifier for a target object refers to one or more attributes or values ​​used to uniquely identify and locate a target object record. In the Hudi storage system, identifiers are crucial information used to distinguish different data items, helping the Hudi storage system accurately locate the target objects that users or applications want to access.

[0061] For example:

[0062] Suppose there is a bank system that uses the Hudi storage system to manage customer account information. When a customer wants to query their account information, they can submit a request to read the account information through the bank's application or website. The request includes a unique identifier such as the customer's account number or ID number.

[0063] After a customer logs into the bank's application or website, they enter their account number or ID number and then click to query account information. The bank's Hudi storage system receives the request, extracts the identifier from the request, and then uses the identifier to find the corresponding data record.

[0064] S2. Use the identifier to query the data cache of the Hudi storage system to determine whether the read-time merging layer of the data cache stores the update record merge file corresponding to the target object. The update record merge file is obtained by merging the update records associated with the target object in the MOR table of the Hudi storage system in advance. The MOR table stores the update records of the target object for each iteration.

[0065] In this embodiment, after receiving the identifier of the target object, the query interface provided by the data cache is called, and the identifier is passed to the query interface as the query key. Based on the input identifier, the read-time merging layer searches for whether there is an update record merge file that matches the identifier. If an update record merge file that matches the identifier is found, and its hash value is consistent with the hash value of the identifier, then it is considered that the data cache stores the update record merge file corresponding to the target object.

[0066] If the hash value matches successfully, it means that the data cache does indeed store the merged update record file corresponding to the target object. This indicates that the Hudi storage system had previously completed the traversal and merging of all update records for the target object in the MOR table and stored the results in the read-time merging layer of the cache.

[0067] In this way, if a cache hit occurs, the Hudi storage system can directly use the cached update record merge file to generate the final target object's data file, without having to traverse the update records related to the target object in the MOR table again, thus significantly improving data read performance. This is particularly important for banking scenarios, as banks need to frequently process a large number of account information queries, and any method that can speed up queries will greatly improve user experience and service quality.

[0068] The data cache is a temporary area in the Hudi storage system used to store frequently accessed or recently accessed data to improve data retrieval speed.

[0069] The read-time merging layer refers to a space within the data cache specifically used to store merged update record files that have already undergone merging processing. These merged update record files are generated by pre-traversing the Hudi storage system's MOR table of update records associated with the target object and merging these related update records.

[0070] In existing technologies, whenever a target object needs to be accessed, the Hudi storage system must traverse all update records related to the target object in the MOR table and merge these update records to generate the latest update record merge file. Since the update records are randomly stored in the MOR table in log format, the traversal and merging process consumes a significant amount of computing resources and time. Especially when data updates are frequent and the number of historical update records is large, the full traversal and merging method can lead to a significant decrease in query performance, particularly as the data volume continues to grow.

[0071] One of the innovations of this invention is that, upon detecting that the latest update record of the target object is stored or updated in the MOR table, all update records in the MOR table are pre-traversed, and these records are merged to generate a merged update record file. This merged update record file is then stored in the read-time merging layer. Thus, when a user or application requests access to the target object again, the merged update record file can be directly retrieved from the read-time merging layer without having to traverse and merge all records in the MOR table again.

[0072] Since the data cache typically resides in memory, its access speed is much faster than that of disk storage in the Hudi storage system. By placing the read-time merging layer in the data cache, the high-speed read characteristics of the cache can be utilized, reducing the dependence on the Hudi storage system and thus improving read efficiency.

[0073] This invention effectively solves the performance problems caused by full traversal and merging in existing technologies by pre-traversing and merging update records in the MOR table and storing the latest merged update record file in a read-time merging layer within the data cache. This not only improves data reading efficiency but also reduces resource dependence on the Hudi storage system, thereby significantly improving the overall system performance when data updates are frequent and the number of historical update records is large.

[0074] In this invention, the data cache area of ​​the Hudi storage system is constructed as follows:

[0075] First, analyze which frequently accessed data is the most frequently accessed. In banking financial applications, frequently accessed data (such as the latest transaction records and frequently updated user information) are the best candidates for caching.

[0076] The size of the data cache is determined based on hot data, available resources in the Hudi storage system, and the expected cache hit rate. A suitable caching mechanism is then selected, including in-memory caching and distributed caching. In banking applications, in-memory caching is suitable for large-scale distributed environments such as Redis and Memcached. In-memory caching is suitable for single-machine environments or small clusters, such as using Java's native caching library (e.g., Caffeine). The data cache of the Hudi storage system is obtained by configuring the cache server's parameters (e.g., setting maximum memory, memory overflow policies, and network connection parameters).

[0077] In one embodiment, querying the data cache of the Hudi storage system using the identifier to determine whether the read-time merging layer of the data cache stores an update record merge file corresponding to the target object includes:

[0078] Use the identifier as the query key for the preset query interface of the data cache area;

[0079] According to the preset query interface, query the read-time merging layer to see if there is an update record merge file that matches the query key;

[0080] If the hash value of a merged update record file matches the hash value of the query key, then it is determined that a merged update record file corresponding to the target object is stored.

[0081] Extract the identifier (e.g., account number, ID number, etc.) of the target object from the request. This identifier is the key information that uniquely identifies the target object. Use the extracted identifier as the query key for the query interface. The query key is the key information used by the Hudi storage system to locate the merged update record.

[0082] The query interface searches the read-time merging layer for a matching update record merge file based on the passed identifier. The read-time merging layer stores the merged update record files that have already been processed.

[0083] During the query process, the Hudi storage system verifies whether the hash value of the merged update record matches the hash value of the query key. This hash consistency check ensures that the found data is indeed the requested target object.

[0084] If the hash value of the update record merge file matches the hash value of the query key, it indicates that the update record merge file of the target object is indeed stored in the read-time merge layer. If they do not match, it indicates that the required data is not in the data cache.

[0085] This invention provides an efficient query mechanism that improves the performance of the Hudi storage system when reading data. By using a pre-configured query interface and using the identifier of the target object as the query key, the system can quickly locate the update record merge file in the cache, avoiding traversing the entire MOR table and greatly reducing query time. Since the query interface is specifically designed for the cache, it can process query requests more efficiently, thereby reducing the consumption of computing resources and improving the overall performance of the system.

[0086] By comparing the hash value of the update record merge file with the hash value of the query key, it can be ensured that the queried data is indeed the target object, which helps to ensure data consistency and accuracy. By storing the update record merge file in the cache, direct access to the MOR table can be greatly reduced, which not only reduces the load on the Hudi storage system, but also improves the overall response speed of the system.

[0087] For example:

[0088] Suppose in a banking system, a customer wants to check their account balance. The customer initiates a read request through the banking application, including the account number (e.g., 1234567890) as an identifier. The system extracts the account number 1234567890 from the read request, calls a pre-configured query interface, and uses 1234567890 as the query key. The query interface uses the account number 1234567890 to search the read-time merging layer for a matching update record merge. If a matching record is found, the system compares the hash value of this record with the hash value of the account number 1234567890. If the hash values ​​match, it confirms that the latest merge status of the account is stored in the read-time merging layer; if they do not match, it indicates that the required data is not in the data cache.

[0089] S3. If the update record merge file is stored, retrieve the initial data record of the target object from the Hudi storage system according to the identifier, send the retrieved initial data record to the read-time merging layer, and use the preset merging function of the read-time merging layer to merge the initial data record with the record of the update record merge file to obtain the data file corresponding to the target object and respond to the request.

[0090] In this embodiment, after confirming that the update record merge file of the target object exists in the data cache, the identifier of the target object is used to locate the Parquet document storing the target object in the MOR table. Parquet document is an efficient columnar storage format that is very suitable for storing and processing large-scale datasets.

[0091] After locating the corresponding Parquet document, the initial data record corresponding to the target object is retrieved from the Parquet document. The initial data record refers to the data state after the initial data corresponding to the target object is stored in the Hudi storage system and compressed to save storage space in the Hudi storage system.

[0092] By storing the updated record merge files (usually small files) in the data cache, and storing the initial data records (usually large files) in the MOR table of the Hudi storage system, more efficient reading can be achieved. This is because the data cache is usually located in memory or on high-speed storage media, resulting in faster access speeds; while the MOR table may reside on HDFS (Hadoop Distributed File System), suitable for long-term storage of large amounts of data.

[0093] In addition, by placing frequently accessed small files (update record merge files) in the data cache, the frequency of access to HDFS is reduced, thereby reducing I / O overhead and improving overall data read performance.

[0094] For example:

[0095] Suppose a bank customer wants to check their latest account balance. Upon receiving the query request, the Hudi storage system verifies that the latest merged data for that account exists in the cache using the account number (identifier). Next, the Hudi storage system uses the same account number to locate the Parquet document in the MOR table and retrieves the initial data for that account. In this way, the Hudi storage system can quickly combine the initial data record with the updated merged record in the cache to generate the latest account balance information.

[0096] Extract the identifier of the target object (such as account number, ID number, etc.) from the read request, locate and retrieve the compressed information of the target object in the MOR table based on the identifier, send the retrieved initial data record to the data buffer, decompress the initial data record to restore the initial data state, merge the decompressed initial data record with the update record merge file to generate the final target object.

[0097] For example:

[0098] Assuming a bank customer checks their account information through a banking application, the specific steps are as follows:

[0099] Customers submit read requests through the bank's application or website, including their account number as an identifier. Upon receiving the request, the bank's backend system extracts the account number and uses it as an identifier to locate and retrieve the compressed information for that account from the Hudi storage system's MOR table. Assuming the compressed information is stored in a Parquet document, the system will find the corresponding Parquet document and retrieve the initial data record.

[0100] The retrieved initial data records are sent to the read-time merging layer. At this point, the read-time merging layer already contains the merged update records for that account (if there have been previous updates and merges). The initial data records are decompressed in the cache to restore the original state of the account information. For example, the initial data might contain basic account information such as account balance and account opening date.

[0101] The system uses a pre-defined merging function in the read-time merging layer to merge the decompressed initial data with the update record merge file in the cache. The specific steps include: obtaining the latest update records, which contain recent account changes such as deposits and withdrawals; applying these latest update records to the initial data, for example, updating the account balance; generating the final target object (i.e., the latest account information); and returning the final account information to the client in response to the user's read request.

[0102] In one embodiment, the MOR table is obtained as follows:

[0103] Create a Hive table in the preset storage path of the Hudi storage system. The Hive table is used to organize and query the storage units of the initial data stored in the Hudi storage system.

[0104] Configure the Hive table with the MOR table type to obtain the MOR table of the Hudi storage system;

[0105] The initial data of the target object is stored in the Hudi storage system to obtain the initial data record, and the initial data record is stored in the MOR table.

[0106] First, determine the default storage path for the Hudi storage system. This default path can be an HDFS (Hadoop Distributed File System) path, which specifies the actual physical storage location of the Hive table. Create a Hive table within this default storage path, defining its structure (column names, data types, etc.) to ensure it meets the Hudi storage system's requirements for conversion to a MOR table later.

[0107] Configure the newly created Hive table to support MOR (Merge-on-Read) functionality. For example, set Hudi-specific table properties, table type (hoodie.table.type), write type (hoodie.write.operation), etc.

[0108] You also need to set the table type to MOR (hoodie.table.type = MOR) and configure other necessary Hudi parameters, such as the primary key field and partition field. After completing the above configuration, the Hive table is converted into a MOR table in the Hudi storage system.

[0109] The initial data of the target object is stored in the Hudi storage system to obtain the initial data record, and the initial data record is stored in the MOR table.

[0110] In one embodiment, storing the initial data of the target object in the Hudi storage system to obtain initial data records includes:

[0111] The initial data is preprocessed to obtain preprocessed initial data;

[0112] Extract key fields from the preprocessed initial data, including timestamps and version numbers;

[0113] The key fields are used as compressed description information of the preprocessed initial data. The initial data is compressed according to the compressed description information to obtain the initial data record.

[0114] The raw data record is the record generated after the raw data of the target object is first stored in the Hudi storage system.

[0115] The initial data undergoes preprocessing, which includes data cleaning, format standardization, and redundancy removal. Data cleaning removes whitespace, invalid characters, and incorrectly formatted data. Format standardization converts the initial data into a uniform format for easier subsequent processing. For example, date and time strings are converted to a standard timestamp format. Redundancy removal removes unnecessary fields or information, retaining only core data to reduce the data volume.

[0116] Identify key fields in the preprocessed initial data, including timestamps and version numbers. These key fields help track the change history of the target object. Extract these key fields from the preprocessed data. For example, extract the timestamp and version number from each record.

[0117] The extracted key fields are combined into compressed descriptive information. This descriptive information includes timestamps and version numbers to describe the data's version status. The preprocessed initial data is compressed using a preset compression algorithm (e.g., Gzip, Snappy, LZ4). During compression, the compressed descriptive information can be stored as metadata for later use during decompression. The compressed initial data records are stored in a Parquet document within the MOR table, ensuring that the initial data records are organized according to Hudi's requirements for efficient subsequent querying and management.

[0118] In one embodiment, after storing the initial data record in the MOR table, the method further includes:

[0119] Create a .Log document in the MOR table. The .Log document is used to record log changes to the initial data operations.

[0120] The record performs a preset type of data operation on the initial data to obtain an update record of the initial data record;

[0121] Each update to the initial data record is stored in the .Log document.

[0122] Create a .Log file in the default storage path of the MOR table. The default storage path of the MOR table can be a Hadoop Distributed File System (HDFS) path, which specifies the actual physical storage location of the MOR table. The created .Log file is used to record all operations performed on the initial data records.

[0123] The .Log file is used to record changes to stored data. The log file usually contains a series of operation records, each record describing a change to the data, such as the timestamp of an update operation, the data records before and after the change, and other information.

[0124] Record each data operation of a preset type performed on the initial data, resulting in an update record for each initial data record. The preset operation types include Insert, Update, and Delete. Each time these data operations are performed, record the timestamp of the operation, the operation type, the data identifiers involved (such as account number, transaction ID, etc.), the version number, and other relevant information.

[0125] Transform the details of each data operation into an easy-to-understand and store update record. For example, convert the update record into a log record and write the converted log record to a .Log file to ensure that the record of each operation is completely saved.

[0126] By recording each operation performed on the initial data record and storing these operations in a .Log file, the history of data operations can be tracked and managed. This facilitates auditing or data recovery when needed later.

[0127] In one embodiment, the read-time merging layer of the data buffer is obtained as follows:

[0128] A spatial region is allocated in the data cache area as an initial layer;

[0129] After detecting that the initial data record has generated an update record, according to the timestamp and version number of the update record, all update records of the initial data record are traversed in the MOR table and merged to obtain the update record merge file;

[0130] The updated record merge file and the preset merge function are stored in the initial layer to obtain the read-time merge layer.

[0131] A space is allocated within the data cache as an initial layer. This initial layer can be a region within the data cache (e.g., with a default storage path of / cache / transactions) to store frequently accessed data. The initial layer stores merged update records, typically containing identifiers, merged version data, and other necessary metadata (such as timestamps and version numbers).

[0132] Set up a listener to detect any updated records in the MOR table. Whenever a new updated record is generated in the MOR table, the listener will capture these updates and record the timestamp and version number of each updated record.

[0133] Based on the timestamp, version number, and identifier of the initial data record, the MOR table is traversed to find all related update records of the initial data record. All the traversed update records are then merged. The merging process involves integrating the data of all versions into a latest state.

[0134] The updated record merge file and the preset merge function are stored in the initial layer to obtain the read-time merge layer. If the initial layer stores previous updated record merge files of the initial data records, the latest updated record merge file overwrites the previous updated record merge files. If the updated record merge file is being stored in the initial layer for the first time, it can be stored directly.

[0135] A default merge function is a function or algorithm used to merge different versions of data. In the Hudi framework, the default merge function is the logic used to handle the differences between different versions of the same data record and determine the final data state.

[0136] For example:

[0137] Suppose a bank needs to maintain customer account information, and each time the account information changes (e.g., deposits, withdrawals, transfers, etc.), the account status needs to be updated. Here are the specific steps:

[0138] Create an initial layer in the data cache area, with the path / cache / transactions, to store the merged files of update records for customer account information.

[0139] Set up a listener on the MOR table in the Hudi storage system to monitor all updates to account information. Whenever account information changes, the listener captures these changes and records the relevant timestamp and version number. When the listener detects a change, it iterates through all relevant update records in the MOR table based on the recorded timestamp and version number. For example, if a customer's account made multiple transactions within a day, all these transaction records need to be iterated through. All the iterated update records are then merged into the latest account state. For example, if a customer deposits 1000 yuan and then withdraws 500 yuan, the latest account balance should have increased by 500 yuan. The latest merged update record is stored in the initial layer of the cache. If the customer's account information already exists, the latest merged update record overwrites the old data; if this is the first time it's being stored, it's directly added to the initial layer.

[0140] In one embodiment, the step of merging the initial data record and the updated record merge file using a preset merge function of the read-time merging layer to generate a result record corresponding to the target object includes:

[0141] Parse the merged update record file to obtain the latest update record of the target object. The latest update record includes a record key and the updated new value.

[0142] Decompress the initial data record to obtain the initial data of the target object, and query the corresponding row of the latest updated record in the initial data according to the record key and the preset merge function;

[0143] Replace the original value in the corresponding row with the updated new value to obtain the final target object.

[0144] Parse the merged update record file and extract the latest update record of the target object. The latest update record includes the record key and the new value after the update.

[0145] The initial data records retrieved from the MOR table are decompressed to restore the initial data state. Based on the record key in the latest updated record and the preset merge function, the corresponding row in the initial data is queried, and the new value in the latest updated record is used to replace the original value in the corresponding row of the initial data.

[0146] Predefined merge functions are used to process updated data records to generate the latest merged archive. Specifically, they are algorithms or functions used to handle the merging logic of the same data record after multiple updates. In the Hudi (Hadoop Universal Data Index) framework, predefined merge functions are used to handle the differences between different versions of the same data record and determine the final data state.

[0147] The purpose of the pre-defined merge function is to determine which update record is valid when the same record is updated multiple times. For example, in a bank account scenario, if an account experiences multiple deposits and withdrawals within a short period, the pre-defined merge function needs to determine the latest account balance. It merges all related update records into a single, updated merge file. For instance, if an account deposits 1000 yuan and then withdraws 500 yuan, the pre-defined merge function needs to merge these two operations and calculate the final balance. By applying the pre-defined merge function when reading data, all update records are avoided from being re-traversed on each read, thus improving read performance. Especially when there is no updated merge file in the data cache, the pre-defined merge function can quickly generate the latest updated merge file during the read process.

[0148] This invention merges the initial data records and the updated records in the data cache. By decompressing the initial data records and replacing only the latest updated values, instead of regenerating the entire dataset, the data processing flow is simplified. Merging in the data cache, by utilizing the characteristics of the data cache (located in memory or on a high-speed storage medium), can improve the data processing speed, saving processing time and reducing memory usage.

[0149] S4. If no update record merge file is stored, then traverse the update records associated with the target object in the MOR table of the Hudi storage system, merge all update records associated with the target object to generate an update record merge file, retrieve the initial data record of the target object from the Hudi storage system according to the identifier, send the update record merge file and the initial data record to the read-time merging layer for merging, obtain the data file corresponding to the target object, and respond to the request.

[0150] If no merged update record matching the target object identifier in the request is found in the data cache, the MOR (Merge-on-Read) table in the Hudi storage system needs to be traversed. The purpose of this traversal is to find all update records associated with the target object. These update records contain all historical change information for the target object.

[0151] During the traversal of the MOR table, all update records related to the target object are collected and merged to generate a merged update record file. Merging here refers to combining multiple updates into a single, latest state according to certain rules (such as timestamps, version numbers, etc.). By merging update records, a merged update record file containing all the latest changes is generated. This data represents the current state of the target object.

[0152] Simultaneously, the initial data records of the target object need to be retrieved from the Hudi storage system. The generated updated record merge file and the retrieved initial data records are then sent to the read-time merging layer in the data cache. The read-time merging layer is a design optimized for read performance; it can temporarily store data in memory or other high-speed storage to quickly respond to read requests.

[0153] In the read-time merging layer, a predefined merging function is used to merge the initial data records with the updated records. This predefined merging function is a logical OR algorithm that determines how to merge the initial data with the updated records to generate the final data state. Through the merging operation, the final target object is generated. This indicates that the system has generated a dataset reflecting the current state based on the latest updated records and the initial data. Finally, the generated final target object is returned to the application or user that initially issued the request, completing the read request processing.

[0154] In summary, by generating a merged file of update records in advance and storing it in the data cache, the present invention can directly use the latest data in the cache when a read request arrives, avoiding traversing all update records in the MOR table. This improves the speed of data reading, reduces query latency, rationally allocates storage and computing resources, reduces unnecessary resource waste, and improves the overall performance of the system.

[0155] like Figure 2 The diagram shown is a schematic of a Hudi-based data reading device according to an embodiment of the present invention.

[0156] The Hudi-based data reading device 100 described in this invention can be installed in an electronic device. Depending on the functions implemented, the Hudi-based data reading device 100 may include a receiving module 110, a determining module 120, a retrieving module 130, and a traversing module 140. The module described in this invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and which are stored in the memory of the electronic device.

[0157] In this embodiment, the functions of each module / unit are as follows:

[0158] The receiving module 110 is configured to receive a request to access a target object from the Hudi storage system and extract the identifier of the target object from the request;

[0159] The determination module 120 is used to query the data cache of the Hudi storage system using the identifier to determine whether the read-time merging layer of the data cache stores an update record merge file corresponding to the target object. The update record merge file is obtained by merging the update records associated with the target object in the MOR table of the Hudi storage system in advance. The MOR table stores the update records of the target object for each iteration.

[0160] The retrieval module 130 is used to retrieve the initial data record of the target object from the Hudi storage system according to the identifier if the update record merge file is stored, send the retrieved initial data record to the read-time merging layer, and use the preset merging function of the read-time merging layer to merge the initial data record with the record of the update record merge file to obtain the data file corresponding to the target object and respond to the request.

[0161] The traversal module 140 is used to traverse the update records associated with the target object in the MOR table of the Hudi storage system if no update record merge file is stored, merge all update records associated with the target object to generate an update record merge file, retrieve the initial data record of the target object from the Hudi storage system according to the identifier, send the update record merge file and the initial data record to the read-time merging layer for merging, obtain the data file corresponding to the target object, and respond to the request.

[0162] like Figure 3 The diagram shown is a structural schematic of an electronic device that implements a Hudi data reading method according to an embodiment of the present invention.

[0163] In this embodiment, the electronic device 1 includes, but is not limited to, a memory 11, a processor 12, and a network interface 13 that can be interconnected via a system bus. The memory 11 stores a Hudi data reading program 10, which can be executed by the processor 12. Figure 3 Only the electronic device 1, which includes components 11-13 and a Hudi data reading program 10, is shown. Those skilled in the art will understand that... Figure 3 The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0164] The memory 11 includes RAM and at least one type of readable storage medium. The RAM provides a cache for the operation of the electronic device 1; the readable storage medium can be a non-volatile storage medium such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the readable storage medium can be an internal storage unit of the electronic device 1; in other embodiments, the non-volatile storage medium can also be an external storage device of the electronic device 1, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device 1. In this embodiment, the readable storage medium of the memory 11 is typically used to store the operating system and various application software installed on the electronic device 1, such as storing the code of the Hudi data reading program 10 in one embodiment of the present invention. Furthermore, the memory 11 can also be used to temporarily store various types of data that have been output or will be output.

[0165] In some embodiments, processor 12 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. Processor 12 is typically used to control the overall operation of the electronic device 1, such as performing control and processing related to data interaction or communication with other devices. In this embodiment, processor 12 is used to run program code stored in memory 11 or process data, such as running a Hudi data reading program 10.

[0166] The network interface 13 may include a wireless network interface or a wired network interface, which is used to establish a communication connection between the electronic device 1 and the terminal (not shown in the figure).

[0167] Optionally, the electronic device 1 may further include a user interface, which may include a display, an input unit such as a keyboard, and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device 1 and to display a visual user interface.

[0168] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0169] The Hudi data reading program 10 stored in the memory 11 of the electronic device 1 is a combination of multiple instructions, which, when run in the processor 12, can achieve the following:

[0170] Receive a request to access a target object in the Hudi storage system, and extract the identifier of the target object from the request;

[0171] The identifier is used to query the data cache of the Hudi storage system to determine whether the read-time merging layer of the data cache stores the update record merge file corresponding to the target object. The update record merge file is obtained by merging the update records associated with the target object in the MOR table of the Hudi storage system in advance. The MOR table stores the update records of the target object for each iteration.

[0172] If the update record merge file is stored, the initial data record of the target object is retrieved from the Hudi storage system according to the identifier, the retrieved initial data record is sent to the read-time merging layer, and the initial data record and the record of the update record merge file are merged using the preset merging function of the read-time merging layer to obtain the data file corresponding to the target object and respond to the request.

[0173] If no update record merge file is stored, the update records associated with the target object in the MOR table of the Hudi storage system are traversed, all update records associated with the target object are merged to generate an update record merge file, the initial data record of the target object is retrieved from the Hudi storage system according to the identifier, the update record merge file and the initial data record are sent to the read-time merging layer for merging, the data file corresponding to the target object is obtained, and the request is responded to.

[0174] Specifically, the processor 12's implementation method for the Hudi data reading program 10 described above can be found in [reference needed]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0175] Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable medium can be non-volatile or non-combustible. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0176] The computer-readable storage medium stores a Hudi-based data reading program 10, which can be executed by one or more processors. The specific implementation of the computer-readable storage medium of the present invention is basically the same as the above-described embodiments of the Hudi-based data reading method, and will not be described in detail here.

[0177] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0178] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0179] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0180] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0181] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0182] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. The term "second class" is used to indicate names and does not indicate any specific order.

[0183] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for reading Hudi data, characterized in that, The method includes: Receive a request to access a target object in the Hudi storage system, and extract the identifier of the target object from the request; The identifier is used to query the data cache of the Hudi storage system to determine whether the read-time merging layer of the data cache stores the update record merge file corresponding to the target object. The update record merge file is obtained by merging the update records associated with the target object in the MOR table of the Hudi storage system in advance. The MOR table stores the update records of the target object for each iteration. If the update record merge file is stored, the initial data record of the target object is retrieved from the Hudi storage system according to the identifier, the retrieved initial data record is sent to the read-time merging layer, and the initial data record and the record of the update record merge file are merged using the preset merging function of the read-time merging layer to obtain the data file corresponding to the target object and respond to the request. If no update record merge file is stored, the update records associated with the target object in the MOR table of the Hudi storage system are traversed, all update records associated with the target object are merged to generate an update record merge file, the initial data record of the target object is retrieved from the Hudi storage system according to the identifier, the update record merge file and the initial data record are sent to the read-time merging layer for merging, the data file corresponding to the target object is obtained, and the request is responded to. The read-time merging layer of the data cache is obtained as follows: a spatial region is allocated in the data cache as an initial layer; after an update record is detected in the initial data record, all update records of the initial data record are traversed and merged in the MOR table according to the timestamp and version number of the update record to obtain an update record merge file; the update record merge file and a preset merging function are stored in the initial layer to obtain the read-time merging layer.

2. The method for reading Hudi data as described in claim 1, characterized in that, The step of querying the data cache of the Hudi storage system using the identifier to determine whether the read-time merging layer of the data cache stores an update record merge file corresponding to the target object includes: Use the identifier as the query key for the preset query interface of the data cache area; According to the preset query interface, query the read-time merging layer to see if there is an update record merge file that matches the query key; If the hash value of a merged update record file matches the hash value of the query key, then it is determined that a merged update record file corresponding to the target object is stored.

3. The method for reading Hudi data as described in claim 1, characterized in that, The MOR table is obtained as follows: Create a Hive table in the preset storage path of the Hudi storage system. The Hive table is used to organize and query the storage units of the initial data stored in the Hudi storage system. Configure the Hive table with the MOR table type to obtain the MOR table of the Hudi storage system; The initial data of the target object is stored in the Hudi storage system to obtain the initial data record, and the initial data record is stored in the MOR table.

4. The method for reading Hudi data as described in claim 3, characterized in that, The step of storing the initial data of the target object into the Hudi storage system to obtain the initial data record includes: The initial data is preprocessed to obtain preprocessed initial data; Extract key fields from the preprocessed initial data, including timestamps and version numbers; The key fields are used as compressed description information of the preprocessed initial data. The initial data is compressed according to the compressed description information to obtain the initial data record.

5. The method for reading Hudi data as described in claim 3 or 4, characterized in that, After storing the initial data record in the MOR table, the method further includes: Create a .Log document in the MOR table. The .Log document is used to record log changes to the initial data operations. The record performs a preset type of data operation on the initial data to obtain an update record of the initial data record; Each update to the initial data record is stored in the .Log document.

6. The method for reading Hudi data as described in claim 1, characterized in that, The step of merging the initial data record and the updated record merge file using the preset merge function of the read-time merging layer to generate the result record corresponding to the target object includes: Parse the merged update record file to obtain the latest update record of the target object. The latest update record includes a record key and the updated new value. Decompress the initial data record to obtain the initial data of the target object, and query the corresponding row of the latest updated record in the initial data according to the record key and the preset merge function; Replace the original value in the corresponding row with the updated new value to obtain the final target object.

7. A Hudi data reading device, characterized in that, The device includes: The receiving module is used to receive a request to access a target object from the Hudi storage system and extract the identifier of the target object from the request; The determination module is used to query the data cache of the Hudi storage system using the identifier to determine whether the read-time merging layer of the data cache stores the update record merge file corresponding to the target object. The update record merge file is obtained by merging the update records associated with the target object in the MOR table of the Hudi storage system in advance. The MOR table stores the update records of the target object for each iteration. The retrieval module is used to retrieve the initial data record of the target object from the Hudi storage system according to the identifier if the update record merge file is stored, send the retrieved initial data record to the read-time merging layer, and use the preset merging function of the read-time merging layer to merge the initial data record with the record of the update record merge file to obtain the data file corresponding to the target object and respond to the request. The traversal module is used to traverse the update records associated with the target object in the MOR table of the Hudi storage system if no update record merge file is stored, merge all update records associated with the target object to generate an update record merge file, retrieve the initial data record of the target object from the Hudi storage system according to the identifier, send the update record merge file and the initial data record to the read-time merging layer for merging, obtain the data file corresponding to the target object, and respond to the request. The read-time merging layer of the data cache is obtained as follows: a spatial region is allocated in the data cache as an initial layer; after an update record is detected in the initial data record, all update records of the initial data record are traversed and merged in the MOR table according to the timestamp and version number of the update record to obtain an update record merge file; the update record merge file and a preset merging function are stored in the initial layer to obtain the read-time merging layer.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a Hudi-based data reading program that can be executed by the at least one processor, the Hudi-based data reading program being executed by the at least one processor to enable the at least one processor to perform the Hudi-based data reading method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a Hudi-based data reading program, which can be executed by one or more processors to implement the Hudi-based data reading method as described in any one of claims 1 to 6.

Images