Data updating method and device

Abstract

The application provides a data updating method and device, wherein the method comprises the following steps: obtaining change data of target data; based on the change data, storing field updating information in an updating information field of the target data and storing data change type information in an operation type field of the target data; based on the updating information field and the operation type field, determining an updating mode of a field to be updated; based on the updating mode and the change data, updating historical data corresponding to the target data stored in a database, and performing column storage on the updated historical data. The data updating method and device provided by the application expand the fields of the target data in the Hudi architecture, are used for assisting the updating process of the change data in the target data, realize the storage of necessary information of field updating, enable the accurate updating of the fields in the Hudi architecture based on the updating information stored in the expanded fields, and improve the accuracy of data updating.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a data update method and apparatus. Background Technology

[0002] Hudi is an open-source architecture for rapidly updating and storing data on data lakes / data warehouses built on the Hadoop ecosystem.

[0003] When updating data with non-full fields, the existing Hudi implementation has limitations in its input / output components. It cannot definitively distinguish whether a field in the data to be updated is NULL or not updated. Therefore, it can only generally select the latest updated field and treat non-updated fields as NULL. This causes Hudi to discard non-updated fields during non-full field updates, resulting in inaccurate data updates. Summary of the Invention

[0004] This invention provides a data update method and apparatus to solve the technical problem in the prior art Hudi that discards fields that are not being updated when processing non-full field updates, resulting in inaccurate updated data.

[0005] This invention provides a data update method, comprising:

[0006] Retrieve change data for the target data in the Hudi data storage architecture;

[0007] Based on the changed data, the field update information of the changed data is stored in the update information field of the target data, and the data change type information of the changed data is stored in the operation type field of the target data;

[0008] Based on the update information field, determine the field to be updated in the target data, and based on the operation type field, determine the update method for the field to be updated;

[0009] Based on the update method and the changed data, the historical data corresponding to the target data stored in the database is updated to obtain the updated historical data, and the updated historical data is stored in columns.

[0010] According to a data update method provided by the present invention, based on the update information field, determining the field to be updated in the target data includes:

[0011] Based on the update information field, determine the empty fields in the target data and the fields in the target data that are not updated;

[0012] Based on the empty field and the field that is not being updated, determine the field in the target data that needs to be updated.

[0013] According to a data update method provided by the present invention, based on the update method and the changed data, the historical data corresponding to the target data stored in the database is updated to obtain the updated historical data, and the updated historical data is column-stored, including:

[0014] The update method and the changed data are stored in rows to obtain row-stored data;

[0015] Based on the update method and the changed data in the row storage data, the historical data corresponding to the target data stored in the database is updated to obtain the updated historical data, and the updated historical data is then stored in columns.

[0016] According to a data update method provided by the present invention, after obtaining the change data of the target data in the Hudi data storage architecture, the method further includes:

[0017] The system reserved fields in Hudi are expanded to obtain the key-value fields of the target data and the exchange data fields of the target data;

[0018] If the changed field corresponding to the changed data is a unique key field in the target data, the unique key change value of the target data is determined based on the changed data, and the unique key change value is sent to the HTTP server at the target address so that the HTTP server stores the unique key change value.

[0019] The target data stores the historical unique key value of the target data in the key value field and the target address in the exchange data field of the target data.

[0020] According to a data update method provided by the present invention, after storing the target address in the exchange data field of the target data, the method further includes:

[0021] Obtain the target address from the exchanged data field, and obtain the unique key change value from the HTTP server;

[0022] Based on the changed unique key value and the historical unique key value in the key value field, update the unique key value of the target data, and write the updated target data into a columnar storage format data file to obtain the columnar storage data of the target data;

[0023] Based on the column storage data, replace the historical data corresponding to the target data stored in the database.

[0024] According to a data update method provided by the present invention, before storing the field update information of the changed data in the update information field of the target data based on the changed data, the method further includes:

[0025] The system reserved fields in Hudi are expanded to obtain the update information field of the target data and the operation type field of the target data.

[0026] The present invention also provides a data updating device, comprising:

[0027] The acquisition module is used to acquire change data of target data in the Hudi data storage architecture;

[0028] The change data processing module is used to store field update information of the change data in the update information field of the target data based on the change data, and to store data change type information of the change data in the operation type field of the target data;

[0029] The update method determination module is used to determine the field to be updated in the target data based on the update information field, and to determine the update method of the field to be updated based on the operation type field.

[0030] The storage update module is used to update the historical data corresponding to the target data stored in the database based on the update method and the changed data, to obtain the updated historical data, and to perform column storage on the updated historical data.

[0031] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the data update methods described above.

[0032] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any of the data update methods described above.

[0033] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the data update method as described above.

[0034] The data update method and apparatus provided by this invention extend the fields of target data in the Hudi architecture, storing change information of the target data in extended update information fields and operation type fields. This assists in the update process of changed data in the target data, achieving the storage of necessary information for field updates. For cases where not all fields in the target data are updated, the update information stored in the extended fields can accurately update the fields in the Hudi architecture, improving the accuracy of data updates. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in this invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly described below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a flowchart illustrating the data update method provided by the present invention;

[0037] Figure 2 This is a schematic diagram of the basic architecture of Hudi provided by the present invention;

[0038] Figure 3 This is a schematic diagram of the replacement component structure provided by the present invention;

[0039] Figure 4 This is a schematic diagram of the Key update architecture provided by the present invention;

[0040] Figure 5 This is a schematic diagram of the internal structure of the component provided by the present invention;

[0041] Figure 6 This is a schematic diagram of the data update device provided by the present invention;

[0042] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0044] In data lake or real-time data warehouse scenarios, it is often necessary to update massive amounts of historical data. Due to the sheer volume of historical data, such updates are often very costly. A common approach is to use database sharding and table partitioning, placing frequently modified datasets (generally known as hot data) into separate databases / tables, and then periodically synchronizing them to the historical database / data warehouse (generally known as cold data). This process results in data in the historical database experiencing delays of up to several days. To reduce this latency and improve the timeliness of data lakes, several new software programs have been developed, such as Apache Hudi / ClickHouse.

[0045] Apache Hudi is an open-source architecture for rapidly updating data on data lakes / data warehouses built on the Hadoop ecosystem. Its source data typically comes from change logs obtained from message queues such as Kafka and Pulsar, or from binlogs in databases like MySQL. Output data is generally written to columnar data files on file systems such as HDFS (Hadoop Distributed File System), which can be read and used by other data analysis components in the Hadoop ecosystem (such as Spark and Hive).

[0046] The basic principle of data updating in the Hudi architecture in the relevant methods is as follows:

[0047] 1. Historical data is stored using Parquet, a columnar file format commonly used in the Hadoop ecosystem, which facilitates data sharing with other components and is referred to as Base data;

[0048] 2. First, write the source change log to a row storage format (AVRO) data file, which is called log data. Since the amount of data is small, this operation generally does not have much overhead.

[0049] 3. At this point, if a data query is performed, there are two query methods: The first is to query real-time data (called RT view), which requires merging historical data and already written log data according to unique keys during the query. Each unique key retains the latest data, so logically it's as if the data has been updated. This process is called merge. The second is to query historical data (called RO view), which only queries historical data. Obviously, RT view requires merging during the query, resulting in significantly lower performance compared to RO view, but it can retrieve the latest data.

[0050] 4. To prevent log data from persisting and requiring the use of RT views to query the latest data, Hudi will integrate the data according to certain triggering conditions. The result of merging the base data and log data will be written as the new base data file and replace the old base data file. This process is called compaction.

[0051] 5. After compaction is completed, the RO view can retrieve the latest data with high performance.

[0052] Apache Hudi has a functional limitation: it cannot support updates to non-full fields. Details are as follows:

[0053] As described above, the key merge process in Apache Hudi to complete data updates is to retain only the latest record for each unique key's corresponding change log, generally referred to as a record.

[0054] Suppose each record in a table has the following fields: {unique key, logical time (the larger the logical time, the newer the data), field1, field2, field3}. The continuous processing of a unique key KeyA is as follows:

[0055] 1. KeyA initially exists in historical data, with a logical time of 1. This record is denoted as {KeyA, 1, field1-1, field2-1, field3-1}.

[0056] 2. KeyA was updated once, and the new data is saved in the log with a logical time of 2. This record is recorded as {KeyA, 2, field1-2, field2-2, field3-2}.

[0057] 3. KeyA is compacted into the new historical data (the merged data), and only the latest record is retained, denoted as {KeyA, 2, field1-2, field2-2, field3-2};

[0058] 4. KeyA was updated again, and the new data is stored in the log. This time, only the value of field1 was updated, with a logical time of 3. This record is recorded as {KeyA, 3, field1-3}.

[0059] 5. Merge KeyA again, and denote the resulting record as {KeyA, 3, field1-3};

[0060] As you can see, in step 3, the result after merging the data is as expected. However, in step 5, after merging the data, only the value of field1 is as expected. The values ​​of field2 and field3 are discarded (at this time, the query will usually show NULL values), instead of the expected field2-2 and field3-2.

[0061] An update operation like step 2 is called a full-field update, characterized by providing updated values ​​for every field of the record. An update operation like step 4 is called a partial-field update, characterized by providing updated values ​​for only a subset of fields. A full-field update is actually a special case of a partial-field update. Hudi's input / output components have limited capabilities and cannot explicitly indicate whether a field is NULL or not updated. Therefore, it can only generally select the latest record and treat fields that are not updated as NULL. Thus, Apache Hudi cannot guarantee the correctness of data results when dealing with ordinary partial-field updates.

[0062] Meanwhile, when Apache Hudi encounters key updates in data updates, Hudi uses Spark for parallel computation. It distributes the dataset across multiple parallel computation units using a hash algorithm based on the unique key. Each unique key is assigned to only one computation unit, and these units have no dependencies on each other and can run independently. However, in key update scenarios, the computation of the new key depends on data from the old key. Since the keys are different, they might be assigned to different computation units, making data handover impossible.

[0063] To address the shortcomings of related methods, this invention provides a data update method. Figure 1 A schematic flowchart illustrating the data update method provided by this invention. (Refer to...) Figure 1 The data update method provided by this invention may include:

[0064] Step 110: Obtain the change data of the target data in the Hudi data storage architecture;

[0065] Step 120: Based on the changed data, store the field update information of the changed data in the update information field of the target data, and store the data change type information of the changed data in the operation type field of the target data;

[0066] Step 130: Based on the update information field, determine the field to be updated in the target data, and based on the operation type field, determine the update method of the field to be updated;

[0067] Step 140: Based on the update method and the changed data, update the historical data corresponding to the target data stored in the database to obtain the updated historical data, and store the updated historical data in columns.

[0068] The data update method provided by this invention can be executed by an electronic device, a component within an electronic device, an integrated circuit, or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device. For example, a mobile electronic device can be a mobile phone, tablet computer, laptop computer, PDA, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc., while a non-mobile electronic device can be a server, network attached storage (NAS), or personal computer (PC), etc. This invention does not impose specific limitations.

[0069] The technical solution of this invention will be described in detail below using the example of a computer executing the data update method provided by this invention.

[0070] In step 110, change data of the target data in the Hudi data storage architecture is obtained.

[0071] When it is necessary to update target data in the Hudi data storage architecture, retrieve the change data for the target data.

[0072] The target data can include data fields (field), unique key fields (key), and timestamp fields (timestap). For example, when the target data is {key, timestamp, field1, field2, field3}, a change to the target data could be updating field1 to field1-1.

[0073] In step 120, the fields of the existing data in Hudi are expanded by adding an update information field and an operation type field. After obtaining the changed data, based on the changed data, the field update information of the changed data is stored in the update information field of the target data, and the data change type information of the changed data is stored in the operation type field of the target data.

[0074] After expanding the fields of the existing data in Hudi, the corresponding changed data is stored in the expanded fields based on the changed data of the target data.

[0075] Understandably, updating target data in Hudi can be divided into full-field updates and partial-field updates. A full-field update refers to updating all data fields in the target data, while a partial-field update refers to updating only some data fields in the target data.

[0076] In step 130, based on the update information field of the target data, the field to be updated in the target data is determined, and based on the operation type field of the target data, the update method of the field to be updated is determined.

[0077] Since the update information field stores the field update information of the changed data, the fields to be updated in the target data can be determined based on the update information field of the target data. Similarly, since the operation type field stores the data change type information of the changed data, the data change type of the field to be updated can be determined based on the operation type field in the target data.

[0078] Optionally, the data change type can be upsert or delete. Upsert means that if the record corresponding to the key exists, it will be updated, and if it does not exist, it will be added. Delete means that the record corresponding to the key will be deleted.

[0079] In step 140, based on the update method and the changed data, the historical data corresponding to the target data stored in the database is updated to obtain the updated historical data, and the updated historical data is stored in columns.

[0080] Data storage updates include row storage (Avro) and column storage (Parquet). Specifically, based on changed data, the update method of the target data and the changed data are stored in row storage, while the historical data of the target data is stored in column storage.

[0081] After determining the update method for the target data, the changed data within the target data is stored, thereby achieving the data update. Data storage updates include row storage (Avro) and column storage (Parquet). The update method and changed data for the target data are stored in row storage, while historical data of the target data is stored in column storage. Whenever certain conditions are met (after n commits or after a specific time), Hudi initiates a compaction operation, merging the data from the row storage and column storage to generate a new column storage file, ultimately achieving the update of the target data.

[0082] For example, suppose the target data has the following fields: {key, timestamp, field1, field2, field3}, and the system-reserved fields for its extension are: {__null_fields__, __op_type__}. The meanings and functions of these fields are as follows:

[0083] __null_fields__: Update information field, stores the field names of fields where the value is NULL in the changed data, used to distinguish between NULL values ​​and values ​​that are not updated in the changed data;

[0084] __op_type__: Operation type field, stores information about the type of data change, categorized as upsert, delete, and update key;

[0085] The meanings of the operation types are as follows: upsert means that if the record corresponding to the key exists, it will be updated, and if it does not exist, it will be added; delete means that the record corresponding to the key will be deleted; update key means that, based on upsert, the field to be updated also includes the unique key.

[0086] The merge strategy in Hudi has also been adjusted accordingly:

[0087] For each unique key, retain the latest value among all the values ​​to be updated in each column (normal values ​​or NULL values ​​represented by __null_fields__).

[0088] A continuous processing procedure for the target data is as follows:

[0089] 1. The unique key KeyA of the target data initially exists in the historical data, with a logical time of 1. This data is denoted as:

[0090] {key: "KeyA", timestamp:1, field1:field1-1, field2:field2-1, field3:field3-1, __null_fields__: "", __op_type__:upsert}

[0091] 2. KeyA was updated once, and the new data is stored in the log. The logical time is 2. All fields except key need to be updated. The updated data is as follows:

[0092] {key: "KeyA", timestamp:2, field1:field1-2, field3:field3-2, __null_fields__: "field2", __op_type__:upsert};

[0093] 3. KeyA is compacted into the new historical data (the merged data), and all field values ​​are based on logical time 2, therefore it is recorded as:

[0094] {key: "KeyA", timestamp:2, field1:field1-2, field3:field3-2, __null_fields__: "field2", __op_type__:upsert};

[0095] 4. KeyA was updated again, and the new data is stored in the log. Only the value of field1 was updated, with a logical time of 3. This updated data is recorded as follows:

[0096] {key: "KeyA", timestamp:3, field1:field1-3, __null_fields__: "", __op_type__:upsert};

[0097] 5. Merge KeyA again, where the value of field1 is based on logical time 3, and the values ​​of other fields are still based on logical time 2. The updated data is recorded as: {key:"KeyA", timestamp:3, field1:field1-3, field3:field3-2, __null_fields__:"field2", __op_type__:upsert};

[0098] As can be seen, by adding some fields and adjusting the merge processing strategy, updates to non-full fields can be handled correctly.

[0099] Optionally, since the fields in the original Hudi have been expanded, the functionality of the component structure in Hudi needs to be adjusted during the process of row storage and subsequent column storage of the field to be updated based on Hudi.

[0100] Hudi's basic architecture Figure 2 The basic architecture diagram of Hudi provided by this invention is shown. The entire operation process of Hudi consists of the following steps:

[0101] 1. After receiving DML (Data Manipulation Language) SQL, the source table modifies the database;

[0102] 2. Changes will be recorded in binlog format, forming a record (change log);

[0103] 3. The Record will first be formatted to make it suitable for storage in the Hoodie Payload;

[0104] 4. All Hoodie Payloads will be merged once in memory. In the merged set of Hoodie Payloads, each unique key corresponds to only one HoodiePayload.

[0105] 5. The Hoodie Payload will then be persisted to the Hoodie Log (AVRO);

[0106] 6. Under certain conditions, Hoodie Log (AVRO) will compact with HoodieBase (Parquet);

[0107] 6.1. During the compaction process, the Hoodie Log (AVRO) will first be read into a Hoodie Payload structure;

[0108] 6.2. Merge it with Hoodie Base (Parquet), and store the merge result in a new Hoodie Base (Parquet).

[0109] The architecture in Hudi was adjusted, and the adjusted component was named HoodieDeltaChangePayload. The specific structure is as follows: Figure 3 The schematic diagram of the replacement component structure provided by this invention is shown. HoodieDeltaChangePayload is used to store multiple field information, and each field stores three elements: field name, logical time, and field value. "Required" indicates that the field cannot be missing, and "optional" indicates that the field can be missing. Missing fields are used to indicate that no update will be performed.

[0110] Three methods with the same specifications as the original component are provided, and their working process is as follows:

[0111] 1. The main steps of preCombine() are:

[0112] a) Enter two HoodieDeltaChangePayloads;

[0113] b) Compare the logical times of each field to be updated and select the latest value for each;

[0114] c) The handling of __null_fields__ is different from other columns. When a field is updated to a NULL value, the corresponding field name is appended. When a field is updated from a NULL value to a non-NULL value, the corresponding field name is deleted.

[0115] d) All selected word values ​​are ultimately combined to form a HoodieDeltaChangePayload.

[0116] Each time this method is used, a comparison is performed for each field to be updated. Let the average number of fields to be updated per record be NC, and the total number of records to be updated be NL. Then the number of comparison operations can be approximated as NC*(NL-1) times.

[0117] 2. The main steps of getInsertValue() are:

[0118] a) When the operation type is delete, an empty object is given;

[0119] b) Otherwise, convert the values ​​of all fields to be updated into a single value.

[0120] IndexedRecord;

[0121] c) An IndexedRecord object can only have one logical time value. Since each field records its own logical time, it is necessary to select a suitable one from multiple logical times. The selection strategy is also very simple: pick the latest logical time and store it in IndexedRecord.

[0122] d) Provide the generated IndexedRecord;

[0123] 3. The main steps of combineAndGetUpdateValue() are:

[0124] a) When the operation type is delete, provide an empty object.

[0125] b) Input an IndexedRecord derived from the base data;

[0126] c) Compare the logical time of each field to be updated with the input logical time, and select the latest value to overwrite the IndexedRecord;

[0127] d) Clear all field names in __null_fields__, because at this point it is no longer necessary to distinguish between NULL values ​​and values ​​that are not updated;

[0128] e) Provide the modified IndexedRecord;

[0129] Note that each time this method is used, a comparison is performed for each field to be updated. Let the average number of fields to be updated in each record be NC, and the total number of records in the Base data be NB. Then the number of comparison operations can be approximated as NC*NB times.

[0130] The data update method provided in this invention extends the fields of target data in the Hudi architecture, storing change information of the target data in extended update information fields and operation type fields. This assists in the update process of changed data in the target data, achieving the storage of necessary information for field updates. For cases where not all fields in the target data are updated, the update information stored in the extended fields can accurately update the fields in the Hudi architecture, improving the accuracy of data updates.

[0131] In one embodiment, based on the update information field, empty fields and fields in the target data that are not being updated are determined; based on the empty fields and the fields that are not being updated, fields in the target data that need to be updated are determined.

[0132] Understandably, the update information field stores the field update information of the changed data. When updating the target data, the field update information of the changed data stored in the target data's update information field can be read. Based on identifying empty fields and fields not being updated in the target data, the fields in the target data that need to be updated can be determined.

[0133] Empty fields and fields that are not updated in the target data are fields that are not involved in the process of updating the target data based on changed data; that is, fields that do not need to be updated. In related methods, due to the limited capabilities of the input / output components used by Hudi, it is impossible to definitively determine whether a field in the target data is NULL or not updated. Therefore, it can only generally select the latest data entry and treat all fields that are not updated as NULL values, resulting in inaccurate updated data.

[0134] By expanding the fields in the target data and recording the empty fields and fields that are not updated, it is possible to determine the fields in the target data that need to be updated. In the subsequent process of updating and storing the target data, accurate updates to the target data can be achieved.

[0135] The data update method provided in this invention reads the field update information of the changed data stored in the update information field of the target data. Based on identifying empty fields and fields not being updated in the target data, the fields to be updated in the target data can be determined, enabling accurate updates of the target data during subsequent update and storage processes.

[0136] In one embodiment, based on the update method and the changed data, the historical data corresponding to the target data stored in the database is updated to obtain updated historical data, and the updated historical data is column-stored, including: storing the update method and the changed data in rows to obtain row-stored data; updating the historical data corresponding to the target data stored in the database based on the update method and the changed data in the row-stored data to obtain updated historical data, and storing the updated historical data in columns.

[0137] Data storage updates include row storage (Avro) and column storage (Parquet). Specifically, based on the changed data, the update method and the changed data of the target data are first stored in row storage, while the corresponding historical data in the target data database is stored in column storage.

[0138] Based on the information stored in the update information field and the operation type field, the fields to be updated in the target data and the corresponding update methods are determined, and the update methods and changed data are stored in rows.

[0139] Based on the update method and changed data in the row-stored data, the field to be updated is updated, and the updated target data is written to a column-stored data file, thus achieving column-store update of the target data. After performing column-store update on the target data based on the changed data, the target data column-stored data is obtained.

[0140] After obtaining the target data column storage data, the corresponding historical data stored in the database is replaced based on the column storage data, thereby updating the target data stored in the database and completing the target data update process.

[0141] The data update method provided in this embodiment of the invention updates the field to be updated based on the changed data and the update method, and writes the updated target data into a column storage format data file, thereby realizing column storage update of the target data.

[0142] In one embodiment, after obtaining the change data of the target data in the Hudi data storage architecture, the method further includes: expanding the system reserved fields in the Hudi to obtain the key-value field and the exchange data field of the target data; if the change field corresponding to the change data is a unique key field in the target data, determining the unique key change value of the target data based on the change data, and sending the unique key change value to the HTTP server at the target address so that the HTTP server stores the unique key change value; storing the historical unique key value of the target data in the key-value field of the target data, and storing the target address in the exchange data field of the target data.

[0143] The system-reserved fields in Hudi are expanded to obtain the key-value field and the exchange data field for the target data. The key-value field stores the historical unique key value of the target data. The unique key of the target data is used to distinguish it from other data, essentially acting as its identity ID. Based on the unique key, the target data can be retrieved from a large dataset.

[0144] If the changed field corresponding to the changed data is a unique key field in the target data, the unique key change value of the target data is determined based on the changed data, and the unique key change value is sent to the HTTP server at the target address for storage.

[0145] The target data's historical unique key values ​​are stored in the expanded key-value field, and the target address is stored in the target data's exchange data field. This allows the target address to be retrieved from the exchange data field when the target data's unique key value is subsequently updated, thus retrieving the unique key change value stored on the HTTP server.

[0146] The data update method provided in this invention stores the historical unique key value of the target data in the expanded key-value field of the target data, and stores the target address in the exchange data field of the target data. This provides a foundation for subsequently obtaining the target address, thereby obtaining the unique key change value stored in the HTTP server, and updating the unique key value of the target data.

[0147] In one embodiment, after storing the target address in the exchange data field of the target data, the method further includes: obtaining the target address from the exchange data field and obtaining the unique key change value from the HTTP server; updating the unique key value of the target data based on the unique key change value and the historical unique key value in the key value field, and writing the updated target data into a columnar storage format data file to obtain columnar storage data of the target data; and replacing the historical data corresponding to the target data stored in the database based on the columnar storage data.

[0148] The process begins by retrieving the target address of the HTTP server storing the changed unique key value from the exchanged data field. Based on this address, the changed unique key value of the target data is retrieved from the corresponding HTTP server. Then, using this changed unique key value and the historical unique key value stored in the key-value field, the unique key value of the target data is updated. The updated target data is then written to a columnar data file, resulting in columnar stored data of the target data. Further, based on this columnar stored data, the corresponding historical data stored in the database is replaced, thus implementing columnar storage of the target data and ultimately completing the target data update process.

[0149] Optionally, in scenarios where the unique key of the target data is updated, there are two key values: the historical key value of the target data and the key value to be updated. Hudi requires that the key and the target data must be one-to-one. For data in columns that are not being updated, it can only be read from the historical data when processing delete operations on the old key, and this data is necessary for upsert operations on the new key. Therefore, Hudi must provide specific support to enable the data exchange between the two.

[0150] For example, if the target data is {key, timestamp, field1, field2, field3, __null_fields__, __op_type__}, based on the fields reserved in the Hudi system, two additional fields are added: {__swap_token_for_uk__, __old_keys__}. The meaning and function of these added fields are as follows:

[0151] __swap_token_for_uk__: The swap data field, used to store the target URL of the HTTP service used to exchange old and new key data;

[0152] __old_keys__: A key-value field used to store old, historical unique keys; (a reserved field for expansion when updating the key value)

[0153] Meanwhile, the merge strategy in Hudi has also changed accordingly:

[0154] For target data with an operation type of update key, it is first split into two: one uses the historical unique key value in __old_keys__ as the key, and the operation type is denoted as update key delete(UKD), which is used to indicate the deletion of old key data; the other uses the original target data, but removes the __old_keys__ field, and denotes the operation type as update key insert(UKI), which is used to indicate the addition of new key data; both target data are assigned the same HTTP path and saved in their respective __swap_token_for_uk__ fields;

[0155] For UKD, the processing method has one more step than a normal delete: the target data corresponding to the deleted old key needs to be sent to the HTTP service according to the target address specified by __swap_token_for_uk__. For UKI, the processing method also has one more step than a normal upsert: the target data corresponding to the old key needs to be received from the target address specified by __swap_token_for_uk__ and merged with its own data.

[0156] The continuous process of updating the unique key KeyA of the target data to KeyB is as follows:

[0157] 1. KeyA initially exists in the historical data of the target data, with a logical time of 1. This record is denoted as: {key:"KeyA", timestamp:1, field1:field1-1, field2:field2-1, field3:field3-1, __null_fields__:"", __op_type__:upsert}

[0158] 2. KeyA was updated once, and the new target data is stored in the log with a logical time of 2. The key field was updated to KeyB, and the value of field1 was also updated. This record is recorded as:

[0159] {key: "KeyB", timestamp:2, field1:field1-2, __null_fields__: "",

[0160] __op_type__:update key,__old_keys__:{key:"KeyA"}}

[0161] 3. KeyA is split into two data entries, denoted as follows:

[0162] {key:"KeyA", timestamp:2, __op_type__:update key delete,

[0163] __swap_token_for_uk__:"http: / / ip:port / swaptoken / timestamp2"}

[0164] and

[0165] {key: "KeyB", timestamp: 2, field1: field1-2, __null_fields__: "", __op_type__: update key insert,

[0166] __swap_token_for_uk__:"http: / / ip:port / swaptoken / timestamp2"}

[0167] 4. First, process KeyA's UKD, merging its own fields into KeyA's old data, clearing the __swap_token_for_uk__ field, and then sending it to the HTTP service. The data is recorded as follows:

[0168] {key:"KeyA", timestamp:2, field1:field1-1, field2:field2-1, field3:field3-1, __null_fields__:"", __op_type__:update key delete, __swap_token_for_uk__:""}

[0169] 5. Post-process KeyB's UKI: Receive the old data from the HTTP service, merge its own fields into the old data, clear the __swap_token_for_uk__ field, and record the final result as:

[0170] {key:"KeyB", timestamp:2, field1:field1-2, field2:field2-1, field3:field3-1, __null_fields__:"", __op_type__:update key insert, __swap_token_for_uk__:""}

[0171] At this point, the unique key of the target data has been updated.

[0172] Among them, the scenario of updating the unique key of the target data, such as Figure 4 The key update architecture diagram provided by this invention is shown. The overall operating architecture is based on the COW mode and consists of the following steps:

[0173] 1. After receiving the DMLSQL, the source table modifies the database;

[0174] 2. Changes will be recorded in binlog format, forming a record (change log);

[0175] 3. The record will first undergo format adaptation to ensure it can be recognized by DivergenceSource.

[0176] 4. DivergenceSource will split each record with an operation type of update key into two records and store them in two HoodieDeltaChangePayloads. Their operation types are represented by UKD and UKI respectively. At the same time, they are assigned the same URL (based on the same logical time) and point to the HTTP service in PayloadCacheServer.

[0177] 5. Records with operation type UKD are processed first;

[0178] 5.1. Directly trigger the merge with Hoodie Base(Parquet), and provide an empty object to indicate that the old key has been deleted;

[0179] 5.2. Additional steps: During the merge process, the data of the old key read from Hoodie Base (Parquet) is sent to PayloadCacheServer using a URL;

[0180] 6. For UKI Record post-processing, a regular pre-merge process is triggered;

[0181] 7. After pre-merge, the Record operation type will definitely be UKI, and it will be treated as newly added data;

[0182] 7.1. Persist its own data to the new Hoodie Base (Parquet);

[0183] 7.2. Additional steps: Before persistence, you need to use a URL to receive the old key data from the PayloadCacheServer and merge it with your own data;

[0184] In addition to HoodieDeltaChangePayload, the key update scenario has replaced another component, tentatively named DivergenceSource, and a new component, PayloadCacheServer, has also been added. Their structure and workflow are detailed below.

[0185] The DivergenceSource component is used to retrieve an external collection of Records and convert it into a data structure used internally by Apache Hudi. Its main functionality is concentrated in the fetchNewDataInAvroFormat method, and the specific steps are as follows:

[0186] 1. fetchNewDataInAvroFormat():

[0187] a) Obtain the collection of source records;

[0188] b) Identify records of type update key, split them into two records, and replace the original records in the set. Note that the split records must have UKD first and UKI second.

[0189] c) The split record set is divided from the position between UKD and UKI, and finally divided into multiple record sets;

[0190] d) The multiple record sets are given in the order of the original dataset. Since each record set will be processed and submitted as a batch of data in the upper-level modules of the Hudi architecture, this ensures that records with operation type UKD will be processed and submitted in the first batch and UKI will be processed and submitted in the second batch, thus guaranteeing the processing order of these two types of records.

[0191] e) Once all sets are given, begin repeating the above process;

[0192] The PayloadCacheServer component is newly added. Internally, it starts a lightweight, all-in-memory HTTP service and maintains a map, providing two HTTP interfaces externally:

[0193] 1. put();

[0194] a) Respond to the HTTP request to obtain the URL and data content;

[0195] b) Extract the last part of the received URL as the key and the received data content as the value, and store it in a map;

[0196] 2.get();

[0197] a) Respond to the HTTP request and obtain the URL;

[0198] b) Extract the last part of the received URL as the key and look up the corresponding data content in the map;

[0199] c) If the data content can be found, then provide the data content via an HTTP request;

[0200] d) If no results are found, return empty data;

[0201] The internal structure of HoodieDeltaChangePayload is as follows: Figure 5 The internal structure diagram of the component modification provided by this invention is shown.

[0202] The workflows for getInsertValue and combineAndGetUpdateValue have changed, as follows:

[0203] 1. `getInsertValue()` retains the original processing logic but adds a new processing branch. The main steps are:

[0204] a) When the operation type is UKI, run the new processing flow;

[0205] b) Using the URL of its own __swap_token_for_uk__ field as the input parameter, use the get method of PayloadCacheServer to receive the old key data and convert it into IndexedRecord;

[0206] c) Compare the logical time of each field to be updated with the input logical time, and select the latest value to overwrite IndexedRecord;

[0207] d) Provide the generated IndexedRecord;

[0208] 2. `combineAndGetUpdateValue()` retains the original processing logic but adds a new processing branch. The main steps are:

[0209] a) Input an IndexedRecord derived from the base data;

[0210] b) When the operation type is UKD, run the new processing flow.

[0211] c) Compare the logical time of each field to be updated with the input logical time, and select the latest value to overwrite IndexedRecord;

[0212] d) Clear the __swap_token_for_uk__ field in IndexedRecord, because it is now being sent to PayloadCacheServer as the old key, and this field is no longer needed;

[0213] e) Use the IndexedRecord and the URL of its own __swap_token_for_uk__ field as input parameters, and use the put method of PayloadCacheServer to send the old key data;

[0214] f) Providing an empty object indicates that the old key data has been deleted;

[0215] The data update method provided in this embodiment of the invention obtains the unique key change value of the target data from the corresponding HTTP server based on the target address, updates the unique key value of the target data based on the obtained unique key change value and the historical unique key value stored in the key value field, and writes the updated target data into a columnar storage data file to obtain the columnar storage data of the target data, thereby finally realizing the target data update process.

[0216] In one embodiment, before storing the field update information of the changed data in the update information field of the target data based on the changed data, the method further includes: expanding the system reserved field in Hudi to obtain the update information field of the target data and the operation type field of the target data.

[0217] The update information field and the operation type field in the target data are fields that did not originally exist in the Hudi architecture. They can be obtained by expanding upon system-reserved fields in Hudi.

[0218] Based on the system reserved fields in Hudi, the original fields in the target data are expanded to obtain the update information field and the operation type field of the target data.

[0219] The data update method provided in this embodiment of the invention expands the original fields in the target data by using the system reserved fields in Hudi, thereby expanding the update information field and the operation type field of the target data.

[0220] Figure 6 A schematic diagram of the data update device provided by the present invention is shown below. Figure 6 As shown, the device includes:

[0221] Module 610 is used to acquire change data of the target data in the Hudi data storage architecture.

[0222] The change data processing module 620 is used to store field update information of the change data in the update information field of the target data based on the change data, and to store data change type information of the change data in the operation type field of the target data.

[0223] The update method determination module 630 is used to determine the field to be updated in the target data based on the update information field, and to determine the update method of the field to be updated based on the operation type field;

[0224] The storage update module 640 is used to update the historical data corresponding to the target data stored in the database based on the update method and the changed data, to obtain the updated historical data, and to perform column storage on the updated historical data.

[0225] The data update apparatus provided in this invention extends the fields of target data in the Hudi architecture, storing change information of the target data in extended update information fields and operation type fields. This assists in the update process of changed data in the target data, achieving the storage of necessary information for field updates. For cases where not all fields in the target data are updated, the update information stored in the extended fields can accurately update the fields in the Hudi architecture, improving the accuracy of data updates.

[0226] In one embodiment, the update method determination module 630 is specifically used for:

[0227] Based on the update information field, determine the fields to be updated in the target data, including:

[0228] Based on the update information field, determine the empty fields in the target data and the fields in the target data that are not updated;

[0229] Based on the empty field and the field that is not being updated, determine the field in the target data that needs to be updated.

[0230] In one embodiment, the storage update module 640 is specifically used for:

[0231] Based on the update method and the changed data, the historical data corresponding to the target data stored in the database is updated to obtain the updated historical data, and the updated historical data is then stored in columns, including:

[0232] The update method and the changed data are stored in rows to obtain row-stored data;

[0233] Based on the update method and the changed data in the row storage data, the historical data corresponding to the target data stored in the database is updated to obtain the updated historical data, and the updated historical data is then stored in columns.

[0234] In one embodiment, the acquisition module 610 is specifically used for:

[0235] After retrieving the change data for the target data in the Hudi data storage architecture, the process also includes:

[0236] The system reserved fields in Hudi are expanded to obtain the key-value fields of the target data and the exchange data fields of the target data;

[0237] If the changed field corresponding to the changed data is a unique key field in the target data, the unique key change value of the target data is determined based on the changed data, and the unique key change value is sent to the HTTP server at the target address so that the HTTP server stores the unique key change value.

[0238] The target data stores the historical unique key value of the target data in the key value field and the target address in the exchange data field of the target data.

[0239] In one embodiment, the acquisition module 610 is further configured to:

[0240] After storing the target address in the exchange data field of the target data, the method further includes:

[0241] Obtain the target address from the exchanged data field, and obtain the unique key change value from the HTTP server;

[0242] Based on the changed unique key value and the historical unique key value in the key value field, update the unique key value of the target data, and write the updated target data into a columnar storage format data file to obtain the columnar storage data of the target data;

[0243] Based on the column storage data, replace the historical data corresponding to the target data stored in the database.

[0244] In one embodiment, the change data processing module 620 is specifically used for:

[0245] Before storing the field update information of the changed data in the update information field of the target data based on the changed data, the method further includes:

[0246] The system reserved fields in Hudi are expanded to obtain the update information field of the target data and the operation type field of the target data.

[0247] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include: a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communications interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a data update method, which includes:

[0248] Retrieve change data for the target data in the Hudi data storage architecture;

[0249] Based on the changed data, the field update information of the changed data is stored in the update information field of the target data, and the data change type information of the changed data is stored in the operation type field of the target data;

[0250] Based on the update information field, determine the field to be updated in the target data, and based on the operation type field, determine the update method for the field to be updated;

[0251] Based on the update method and the changed data, the historical data corresponding to the target data stored in the database is updated to obtain the updated historical data, and the updated historical data is stored in columns.

[0252] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0253] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the data update method provided by the above methods, the method comprising:

[0254] Retrieve change data for the target data in the Hudi data storage architecture;

[0255] Based on the changed data, the field update information of the changed data is stored in the update information field of the target data, and the data change type information of the changed data is stored in the operation type field of the target data;

[0256] Based on the update information field, determine the field to be updated in the target data, and based on the operation type field, determine the update method for the field to be updated;

[0257] Based on the update method and the changed data, the historical data corresponding to the target data stored in the database is updated to obtain the updated historical data, and the updated historical data is stored in columns.

[0258] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the data update methods provided above, the method comprising:

[0259] Retrieve change data for the target data in the Hudi data storage architecture;

[0260] Based on the changed data, the field update information of the changed data is stored in the update information field of the target data, and the data change type information of the changed data is stored in the operation type field of the target data;

[0261] Based on the update information field, determine the field to be updated in the target data, and based on the operation type field, determine the update method for the field to be updated;

[0262] Based on the update method and the changed data, the historical data corresponding to the target data stored in the database is updated to obtain the updated historical data, and the updated historical data is stored in columns.

[0263] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0264] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0265] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data update method, characterized in that, The method includes: Retrieve change data for the target data in the Hudi data storage architecture; Based on the changed data, the field update information of the changed data is stored in the update information field of the target data, and the data change type information of the changed data is stored in the operation type field of the target data; the update information field is used to record empty fields and fields that are not updated in the target data. Based on the update information field, determine the field to be updated in the target data, and based on the operation type field, determine the update method for the field to be updated; Based on the update method and the changed data, the historical data corresponding to the target data stored in the database is updated to obtain the updated historical data, and the updated historical data is stored in columns. After obtaining the change data of the target data in the Hudi data storage architecture, the process also includes: The system reserved fields in Hudi are expanded to obtain the key-value fields of the target data and the exchange data fields of the target data; If the changed field corresponding to the changed data is a unique key field in the target data, the unique key change value of the target data is determined based on the changed data, and the unique key change value is sent to the HTTP server at the target address so that the HTTP server stores the unique key change value. The target data stores the historical unique key value of the target data in the key value field and the target address in the exchange data field of the target data.

2. The data update method according to claim 1, characterized in that, The step of determining the field to be updated in the target data based on the update information field includes: Based on the update information field, determine the empty fields in the target data and the fields in the target data that are not updated; Based on the empty field and the field that is not being updated, determine the field in the target data that needs to be updated.

3. The data update method according to claim 1, characterized in that, The step of updating the historical data corresponding to the target data stored in the database based on the update method and the changed data to obtain updated historical data, and then storing the updated historical data in columns, includes: The update method and the changed data are stored in rows to obtain row-stored data; Based on the update method and the changed data in the row storage data, the historical data corresponding to the target data stored in the database is updated to obtain the updated historical data, and the updated historical data is stored in columns.

4. The data update method according to claim 1, characterized in that, After storing the target address in the exchange data field of the target data, the method further includes: Obtain the target address from the exchanged data field, and obtain the unique key change value from the HTTP server; Based on the changed unique key value and the historical unique key value in the key value field, update the unique key value of the target data, and write the updated target data into a columnar storage format data file to obtain the columnar storage data of the target data; Based on the column storage data, replace the historical data corresponding to the target data stored in the database.

5. The data update method according to claim 1, characterized in that, Before storing the field update information of the changed data in the update information field of the target data based on the changed data, the method further includes: The system reserved fields in Hudi are expanded to obtain the update information field of the target data and the operation type field of the target data.

6. A data update device, characterized in that, include: The acquisition module is used to acquire change data of target data in the Hudi data storage architecture; The change data processing module is used to store field update information of the change data in the update information field of the target data based on the change data, and to store data change type information of the change data in the operation type field of the target data; The update information field is used to record empty fields and fields that are not updated in the target data; The update method determination module is used to determine the field to be updated in the target data based on the update information field, and to determine the update method of the field to be updated based on the operation type field. The storage update module is used to update the historical data corresponding to the target data stored in the database based on the update method and the changed data, to obtain the updated historical data, and to perform column storage on the updated historical data. After obtaining the change data of the target data in the Hudi data storage architecture, the process also includes: The system reserved fields in Hudi are expanded to obtain the key-value fields of the target data and the exchange data fields of the target data; If the changed field corresponding to the changed data is a unique key field in the target data, the unique key change value of the target data is determined based on the changed data, and the unique key change value is sent to the HTTP server at the target address so that the HTTP server stores the unique key change value. The target data stores the historical unique key value of the target data in the key value field and the target address in the exchange data field of the target data.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the data update method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the data update method as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the data update method as described in any one of claims 1 to 5.

Images