Zipper upgrading method and device, equipment, storage medium and computer program product

Abstract

The invention relates to the technical field of big data, and discloses a zipper upgrading method and device, equipment, a storage medium and a computer program product.The method comprises the steps that changed data is obtained, a current time partition is newly built in an effective table, data in a previous time partition of the effective table is copied to the current time partition, and the current time partition is updated; the effective table is a COW type effective table based on Hudi, the changed data is processed, the processed data is inserted into a current time partition, the data in the current time partition of the effective table is updated through target operation, the target operation is specific operation combining PreCombine and Payload technologies, the failure table is asynchronously updated according to the effective table, and the failure table is updated according to the effective table. The failure table is a COW type failure table based on Hudi; therefore, the problems that the updating speed of the zipper table is low, the retransmission / rollback process is complex, and the statistical efficiency is low due to the need of screening are solved.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a zipper upgrading method, device, equipment, storage medium and computer program product. Background Art

[0002] In a dimensionally modeled data warehouse, the attributes of a dimension often change over time. This problem is called "slowly changing dimensions (SCD)". In big data scenarios, this problem is generally solved by building a zipper table. A zipper table is a data model used to track historical data and record current valid data. Whenever new change data (including updated and newly added parts) arrives, the zipper model can ensure that the original value is retained while inserting a new value, and the validity period of each data record can be marked through specific fields (such as effective time and expiration time).

[0003] However, in ultra-large-scale data scenarios, traditional zipper models (such as Hive zipper) and other existing zipper tables have problems such as slow update speed, complex retransmission / rollback process, and low statistical efficiency due to the need for screening. Summary of the invention

[0004] The main purpose of this application is to provide a zipper upgrade method, device, equipment, storage medium and computer program product, aiming to solve the technical problems of slow update speed, complex retransmission / rollback process and low statistical efficiency due to screening in the existing zipper table.

[0005] To achieve the above-mentioned purpose, the present application provides a zipper upgrading method, which comprises:

[0006] Obtain the changed data, create a new current time partition in the effective table, and copy the data in the previous time partition of the effective table to the current time partition. The effective table is a Hudi-based COW type effective table.

[0007] Processing the changed data, inserting the processed data into the current time partition, and updating the data in the current time partition of the effective table through a target operation, wherein the target operation is a specific operation combining PreCombine and Payload technologies;

[0008] The invalidation table is asynchronously updated according to the effective table, and the invalidation table is a COW type invalidation table based on Hudi.

[0009] Optionally, the acquiring of the changed data, creating a new current time partition in the effective table, and copying the data in the previous time partition of the effective table to before the current time partition, further includes:

[0010] Obtaining data to be processed, and determining whether the data to be processed is changed data of an existing model or brand new data that requires a new model;

[0011] Accordingly, the acquisition of the changed data, the creation of a new current time partition in the effective table, and the copying of the data in the previous time partition of the effective table to the current time partition include:

[0012] When the data to be processed is change data of an existing model, the change data is obtained, a new current time partition is created in the effective table, and data in the previous time partition of the effective table is copied to the current time partition.

[0013] Optionally, after obtaining the data to be processed and determining whether the data to be processed is changed data of an existing model or brand new data requiring a new model, the method further includes:

[0014] When the data to be processed is brand new data that requires a new model, creating a validation table and an invalidation table;

[0015] The new data is used as the changed data, and the changed data is obtained, a new current time partition is created in the effective table, and the step of processing the changed data and inserting the processed data into the current time partition is returned.

[0016] Optionally, the effectiveness table creates an index for the changed data based on Bucket Index, and uses RecordKey and PartitionPath to retrieve location information of the record to be updated within the current time partition range.

[0017] Optionally, processing the changed data and inserting the processed data into the current time partition includes:

[0018] Set the first occurrence time of the change data to an invalid value;

[0019] Insert the processed data into the current time partition.

[0020] Optionally, the asynchronously updating the invalidation table according to the validation table includes:

[0021] Acquire updated data in the effective table, where the first appearance time of the updated data in the current account period is different from the effective time;

[0022] The values ​​of other fields of the updated data in the previous account period except the time partition and the effective time are obtained, and the values ​​of the time partition and the expiration time are set in the current account period to obtain the expiration table.

[0023] In addition, to achieve the above-mentioned purpose, the present application also proposes a zipper upgrading device, which comprises:

[0024] The data processing module is used to obtain the changed data, create a new current time partition in the effective table, and copy the data in the previous time partition of the effective table to the current time partition. The effective table is a COW type effective table based on Hudi;

[0025] An effective table update module, used to process the changed data, insert the processed data into the current time partition, and update the data in the current time partition of the effective table through a target operation, wherein the target operation is a specific operation combining PreCombine and Payload technologies;

[0026] The invalidation table update module is used to asynchronously update the invalidation table according to the effective table, and the invalidation table is a COW type invalidation table based on Hudi.

[0027] In addition, to achieve the above-mentioned purpose, the present application also proposes a zipper upgrade device, which includes a memory, a processor, and a zipper upgrade program stored in the memory and executable on the processor, wherein the zipper upgrade program is configured to implement the zipper upgrade method described above.

[0028] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, on which a zipper upgrade program is stored, and when the zipper upgrade program is executed by a processor, the zipper upgrade method described above is implemented.

[0029] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a zipper upgrade program, and when the zipper upgrade program is executed by a processor, it implements the zipper upgrade method described above.

[0030] One or more technical solutions proposed in this application have at least the following technical effects:

[0031] In the present application, the acquisition of change data is disclosed, a new current time partition is created in the effective table, and the data in the previous time partition of the effective table is copied to the current time partition. The effective table is a COW type effective table based on Hudi. The change data is processed and the processed data is inserted into the current time partition. The data in the current time partition of the effective table is updated through a target operation, wherein the target operation is a specific operation combining PreCombine and Payload technologies. The invalidation table is asynchronously updated according to the effective table, and the invalidation table is a COW type invalidation table based on Hudi. Since the present application is based on Hudi's COW type effective table and invalidation table asynchronously updated, combined with time partitioning and indexing mechanism, through PreCombine and Payload features, the data has the latest version while retaining all historical states, and supports fast backtracking and redoing, which solves the problems of slow update speed of zipper tables, complex retransmission / rollback process, and low statistical efficiency due to screening. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0033] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0034] Figure 1 This is a schematic diagram of the process of the first embodiment of the zipper upgrading method of the present application;

[0035] Figure 2 This is a device interaction diagram of an embodiment of the zipper upgrading method of the present application;

[0036] Figure 3 This is a flowchart of an implementation example of a zipper upgrading method of the present application;

[0037] Figure 4 This is a flow chart of a second embodiment of the zipper upgrading method of the present application;

[0038] Figure 5 This is a flow chart of a third embodiment of the zipper upgrading method of the present application;

[0039] Figure 6 This is a schematic diagram of the module structure of the zipper upgrading device according to an embodiment of the present application;

[0040] Figure 7This is a schematic diagram of the device structure of the hardware operating environment involved in the zipper upgrading method in the embodiment of the present application.

[0041] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0042] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0043] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0044] The main solution of the embodiment of the present application is to obtain the changed data, create a new current time partition in the effective table, and copy the data in the previous time partition of the effective table to the current time partition. The effective table is a COW type effective table based on Hudi. The changed data is processed and the processed data is inserted into the current time partition. The data in the current time partition of the effective table is updated through the target operation, wherein the target operation is a specific operation combining PreCombine and Payload technologies. The invalidation table is asynchronously updated according to the effective table, and the invalidation table is a COW type invalidation table based on Hudi.

[0045] In a dimensionally modeled data warehouse, the attributes of a dimension often change over time. This problem is called "slowly changing dimensions (SCD)". In big data scenarios, this problem is generally solved by building a zipper table. A zipper table is a data model used to track historical data and record current valid data. Whenever new change data (including updated and newly added parts) arrives, the zipper model can ensure that the original value is retained while inserting a new value, and the validity period of each data record can be marked through specific fields (such as effective time and expiration time).

[0046] The entire process of the zipper model operation can be abstracted into three steps:

[0047] 1. Get the changed data, and then compare it with the historical data in the zipper table to find out the primary key set KS1 that has appeared in the historical data and the primary key set KS2 that has not appeared in the primary key of the changed data; the records corresponding to the primary key set KS1 in the changed data are the updated part, and the records corresponding to the primary key set KS2 are the newly added part; the records corresponding to the primary key set KS1 in the historical data are the part to be updated;

[0048] 2. Keep the part of the historical data in the zipper table that does not need to be updated unchanged, and set the part to be updated as "invalid". This operation is also called "closing the chain".

[0049] 3. Insert the changed data into the zipper table and set the newly inserted data to "effective". This operation is also called "opening the chain". Repeat from step 1 after the next cycle's changed data arrives.

[0050] Methods to implement zipper tables or other forms that can support the above scenarios are:

[0051] 1. Traditional Hive zipper, using methods such as INNER JOIN to associate the updated part of the changed data, close the link of the historical data corresponding to the primary key of this part of the data (such as setting the expiration time to the current account period), and insert the changed data into the zipper table to open the link (such as setting the expiration time to 2099-12-31). This method is commonly used in the industry;

[0052] 2. The improved version of Hive zipper adds a new "status" partition, including Valid (effective), Invalid (invalid), TMP (temporary) and ValidBackup (effective in the previous version). The data of historical effective A is stored in the Valid partition. First, the changed data B is completely inserted into the TMP partition. The primary key INNER JOIN in the Valid and TMP partitions is used to obtain the updated part B1 in the changed data. The historical data A1 corresponding to the primary key of B1 is copied to the Invalid partition, and the historical data A2 after removing A1 is inserted into the TMP partition. The current ValidBackup partition is deleted, and the data A in Valid is moved to the ValidBackup partition, and the data A2+B in TMP are moved to the Valid partition. This is also the implementation method used by the proposer in a data warehouse;

[0053] 3. Hive is still used to implement the data model, but Hudi is used to maintain a timeline containing data operations based on Hive. The main purpose of this method is to reduce Hive update and deletion delays and improve rollback speed;

[0054] 4. Use Hudi to implement the data model. The principle is similar to the zipper method in the first method. After obtaining the updated part of the changed data through matching, the zipper table is opened and closed;

[0055] 5. Use Hudi to implement the data model, and use the characteristics of Bloom Index log files that update data and new data are stored separately to distinguish the write type ("modify" or "insert") of each record in the current change data, and then distinguish the generated and invalid data based on the write type.

[0056] This application mainly solves the following technical problems existing in traditional zipper models (such as Hive zipper) and other existing methods in ultra-large-scale data scenarios:

[0057] Slow update speed - Due to the need to save the historical status of data and determine whether the data needs to be updated, when the changed data arrives, it is necessary to distinguish the newly added and updated parts of the changed data through association and other means. Association calculation is generally the most time-consuming step in zipper processing. For example, the JOIN operation in Hive includes three stages: Map, Shuffle, and Reduce.

[0058] This application does not require association. Through a special data writing method, the newly added and updated parts can be quickly distinguished after the data update is completed.

[0059] The retransmission / rollback process is complex - Due to the nature of traditional zipper tables that record historical changes, they are heavily dependent on the time sequence of data. If a problem is found in the data at a certain point in the past and needs to be changed, it is necessary to roll back first (such as filtering and updating related values ​​through the generation and failure fields), and then retransmit the data of each account period in chronological order and repeat the processing (association, closed chain, open chain) in sequence, which is very resource-consuming.

[0060] This application can achieve a one-step rollback by deleting the partition. In addition, it abandons the traditional open and close chain operation. When the historical account period data needs to be changed, there is no need to repeat the process in chronological order. (1) To add data to the historical account period: just insert the data into the relevant partition of the effective table. If the data has expired before the latest account period, insert the data into the invalid table at the same time; (2) To remove data from the historical account period: just remove the data from the relevant partition of the effective table and the invalid table; (3) To change the status of historical account period data: just merge all subsequent change data, take out the change node of the change data, and update it to the invalid table.

[0061] The efficiency is low due to the need for screening when doing statistics - Since there are both valid data and invalid data in the zipper table, and most statistics only require valid data, most of the time it is necessary to screen first, which affects the efficiency of statistics.

[0062] This application separates the birth and failure data and adopts asynchronous update, which can be used separately or combined for statistics. The effective table can provide services upward before the invalid table starts to be updated.

[0063] Difficult to maintain - Maintenance of general zipper tables requires a lot of work. Other existing methods also require additional maintenance work (such as maintaining indexes, maintaining query log programs, etc.), which will be an unstable factor when there is an urgent need for data supply.

[0064] The focus of this application maintenance is on the use of logic, which is simpler and more efficient than traditional zipper table maintenance.

[0065] It should be noted that the executor of this embodiment can be a zipper upgrade device with data processing, network communication and program running functions, such as a server, etc., or other electronic devices that can achieve the same or similar functions, and this embodiment does not limit this.

[0066] Based on this, the present application embodiment provides a zipper upgrading method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the zipper upgrading method of the present application.

[0067] In this embodiment, the zipper upgrading method includes:

[0068] Step S10: Obtain the changed data, create a new current time partition in the effective table, and copy the data in the previous time partition of the effective table to the current time partition. The effective table is a COW type effective table based on Hudi.

[0069] It should be understood that the present application provides a method for replacing and upgrading the traditional zipper model in a super-large-scale data scenario, including: creating live and invalid Hudi tables with time partitions and fields such as "first appearance time"; obtaining change data including updates and additions and processing them; assuming that there is existing stock data, when obtaining the change data, the program will be automatically triggered to create a new time partition for the effective table and copy the data of the previous time partition to the current partition; insert the change data into the current partition, and use the technical means of PreCombine and Payload to mark the updated records while generating a snapshot of the current partition's valid data; take out the records in the previous partition corresponding to the primary key of the marked record and insert them into the invalid table. The present application proposes asynchronous updates of COW-type effective tables and invalid tables based on Hudi, combined with time partitions and index mechanisms, and through PreCombine and Payload features, so that the data has the latest version while retaining all historical states, and supports fast backtracking and redoing, solving the problems of slow zipper table updates, complex retransmission / rollback processes, and low statistical efficiency due to screening.

[0070] It should be noted that the changed data may be changed data including a newly added and / or updated portion of a primary key, which is not limited in this embodiment.

[0071] Furthermore, this embodiment uses Bucket Index to greatly improve processing efficiency, and can quickly obtain the effective table while still being able to distinguish between primary keys whose attributes have changed and those that have not changed. The effective table creates an index for the changed data based on Bucket Index, and uses RecordKey and PartitionPath to retrieve the location information of the record to be updated within the current time partition range.

[0072] Step S20: Process the changed data, insert the processed data into the current time partition, and update the data in the current time partition of the effective table through a target operation, wherein the target operation is a specific operation combining PreCombine and Payload technologies.

[0073] Furthermore, the processing of the changed data and inserting the processed data into the current time partition includes: setting the first appearance time of the changed data to an invalid value; and inserting the processed data into the current time partition.

[0074] It should be noted that the invalid value may be preset. In this embodiment and other embodiments, the invalid value may be a Null value.

[0075] Step S30: Asynchronously update the invalidation table according to the effective table, where the invalidation table is a COW type invalidation table based on Hudi.

[0076] It should be understood that asynchronously updating the invalidation table according to the effective table may be to identify updated data according to the fields in the effective table and asynchronously update the invalidation table.

[0077] For ease of understanding, refer to Figure 2 and Figure 3 This invention is provided for illustration, but is not intended to limit the present application. Figure 2 This is a device interaction diagram of an embodiment of the zipper upgrading method of the present application. Figure 3 This is a flowchart of an implementation example of the zipper upgrading method of this application. Figure 2 and Figure 3 In the zipper upgrade method, the following steps are included:

[0078] 1. Get data from the "collection module" and determine whether this part of the data belongs to the change data of the existing model or the new data that needs to be created for a new model, so as to perform different processing.

[0079] If you need to create a new model, Figure 3 Step 1, create Hudi's COW type effective table ods.USER_PACK_HUDI_COW and invalid table ods.USER_PACK_EXP_HUDI_COW, use the account period as the time partition (PARTITION_DATE), USER_ID as the primary key, and the package type (PACKAGE_TYPE) as the attribute that may change. At the same time, set the first appearance time (FIRST_APPEAR_TIME) and the birth / expiration time (EFF_TIME / EXP_TIME). The purpose of using EFF_TIME and EXP_TIME here is to make it easier for people to understand. The actual value stored in it is the same as the value of the current time partition.

[0080] If it is a changed data, it will flow to the "data processing module". Assume that the following is the data of the previous account period, that is, the stock data that has been placed in the effective table. Assume that the invalid table is empty at this time. At this time, the content of the effective table is shown in Table 1:

[0081] Table 1

[0082] PARTITION_DATE USER_ID PACKAGE_TYPE FIRST_APPEAR_TIME EFF_TIME 20240401 A 1 20240331 20240401 20240401 B 1 20240401 20240401

[0083] 2. Use the "Data Processing Module" to perform preparatory operations such as creating new partitions on the effective table. Figure 3 Step 2.

[0084] First, set the partition variables ${current_date} and ${previous_date} to represent the current time partition and the previous time partition respectively.

[0085] After the change data arrives, the "data processing module" will automatically trigger the partition creation function of the "effective table update module" and copy the data of the previous time partition of the effective table to the current partition after processing through the partition variable. Because the PreCombine operation needs to be under one partition, it is prepared for the subsequent incremental processing. At this time, the content of the effective table is shown in Table 2:

[0086] Table 2

[0087] PARTITION_DATE USER_ID PACKAGE_TYPE FIRST_APPEAR_TIME EFF_TIME 20240401 A 1 20240331 20240401 20240401 B 1 20240401 20240401 20240402 A 1 20240331 20240402 20240402 B 1 20240401 20240402

[0088] 3. Use the "effective table update module" to update the effective table and mark the primary key of the record to be inserted into the invalid table. Figure 3 Step 3.

[0089] Assume that the "collection module" obtains two change data (including one update and one addition), USER_ID is the primary key, PACKAGE_TYPE is the attribute that may change, and the (USER_ID, PACKAGE_TYPE) value is (B, 2), (C, 3). The "data processing module" will process it, set its first appearance time (FIRST_APPEAR_TIME) to NULL, and then insert it into the partition of the current account period. The following data is an intermediate process, which is only for the convenience of understanding and will not be used as the output of any node. At this time, the content of the effective table is shown in Table 3:

[0090] Table 3

[0091] PARTITION_DATE USER_ID PACKAGE_TYPE FIRST_APPEAR_TIME EFF_TIME 20240401 A 1 20240331 20240401 20240401 B 1 20240401 20240401 20240402 A 1 20240331 20240402 20240402 B 1 20240401 20240402 20240402 B 2 NULL 20240402 20240402 C 3 NULL 20240402

[0092] The "Effective Table Update Module" will create an index for the changed data based on the Bucket Index, and use the RecordKey and PartitionPath to retrieve the location information of the record to be updated within the current time partition range. However, the index can only be used to distinguish whether the record is to be updated, and cannot provide an operation identifier (update / add) for the upper-level data display.

[0093] This application provides a method for immediately identifying the operation type (update / addition) after an incremental update: using Hudi's extensible data processing mechanism Payload to customize the writing of data. The program in the "Effective Table Update Module" implements the HoodieRecordPayload interface and rewrites the relevant methods therein. By setting PreCombineField, the data in the current partition is merged and deduplicated, so that the first appearance time (FIRST_APPEAR_TIME) field in the record to be updated retains its original value, and all other fields are replaced by the updated part in the changed data, and the first appearance time (FIRST_APPEAR_TIME) of the newly added part will be set to the current account period.

[0094] After finding the file containing the record to be updated according to the index, the "effective table update module" will use Hudi's Upsert and PreCombine to rewrite the file with the updated value. At this time, the content of the effective table is shown in Table 4:

[0095] Table 4

[0096] PARTITION_DATE USER_ID PACKAGE_TYPE FIRST_APPEAR_TIME EFF_TIME 20240401 A 1 20240331 20240401 20240401 B 1 20240401 20240401 20240402 A 1 20240331 20240402 20240402 B 2 20240401 20240402 20240402 C 3 20240402 20240402

[0097] At this point, the effective table has been updated and there is no need to wait for the asynchronous update of the invalid table (i.e. Figure 3 Step 4) can be used and support statistical needs.

[0098] 4. The "invalidation table update module" asynchronously updates the invalidation table according to the effective table, corresponding to Figure 3 Step 4.

[0099] In the records corresponding to the primary key (B, C) of the changed data in the current account period (20240402) in the effective table, the records with FIRST_APPEAR_TIME≠EFF_TIME (records corresponding to USER_ID value B) are the updated data. The "invalidation table update module" takes the values ​​of all fields except PARTITION_DATE and EFF_TIME corresponding to the primary key in the previous account period, sets the value of EXP_TIME to the current account period, and adds it to the invalidation table through the partition variable. This step can still retain the entire historical state of the data without complex closed-chain and open-chain operations. At this time, the content of the invalidation table is shown in Table 5:

[0100] Table 5

[0101] PARTITION_DATE USER_ID PACKAGE_TYPE FIRST_APPEAR_TIME EXP_TIME 20240402 B 1 20240401 20240402

[0102] Compared with traditional zipper processing, this application uses a customized writing method to replace the most time-consuming operation of associating existing data with changed data. At the same time, the use of Bucket Index greatly improves processing efficiency, and can quickly obtain the effective table while still being able to distinguish between primary keys whose attributes have changed and those that have not changed. In addition, each partition of the effective table is valid data for the corresponding account period, which greatly reduces the cost of version maintenance and rollback / retransmission.

[0103] This application implements various usage scenarios of traditional zipper tables as follows:

[0104] Scenario 1: Get the full amount or valid data at a certain point in time or a certain period of time:

[0105] (1) The latest account period data in the effective table is the full valid data;

[0106] (2) Based on the time partition, you can obtain a snapshot of the valid data at a certain point in time of the effective table;

[0107] (3) By taking the data of certain account periods in the effective table according to the time partition, the data that has been effective within a certain time period can be obtained.

[0108] Scenario 2: Get all the invalid data or the invalid data at a certain time point or time period:

[0109] The principle is the same as that of situation 1, except that the operation object is changed from the effective table to the invalid table.

[0110] Scenario 3: The state change process of the primary key corresponding to one or more records:

[0111] According to the primary key, the corresponding records can be directly merged from the effective table and the invalid table and sorted by time.

[0112] Scenario 4: Same-day retransmission, that is, rolling back to the previous version and rewriting the changed data:

[0113] You can delete the current account period partition and rewrite it through the "Generate / Expire Update Module".

[0114] Scenario 5: Roll back to the version N cycles ago (and retransmit):

[0115] Rollback: Delete all partitions from N cycles ago to the present through the "live / fail update module".

[0116] Retransmission: Since the traditional open-close chain operation is abandoned, when the historical account period data needs to be changed, there is no need to repeat the processing in chronological order. The "Generate / Invalidate Update Module" provides relevant components. (1) Adding data to the historical account period: Just insert the data into the relevant partition of the effective table. If the data has expired before the latest account period, insert the data into the invalid table at the same time; (2) Removing data from the historical account period: Just remove the data from the relevant partition of the effective table and the invalid table; (3) Changing the status of historical account period data: Just merge all subsequent change data, take out the change node of the change data, and update it to the Generate / Invalidate table.

[0117] Scenario 6: Full retransmission, that is, all historical data is invalid and the full data is rewritten:

[0118] The data in the effective table is moved to the invalid table through the "effective / invalid update module", and the new data can be directly written into the effective table.

[0119] The configuration method for this application is:

[0120] 1. If you upgrade the Hive zipper to this application, take all the effective data and save it as the initial effective table, and take the invalid data and save it as the initial invalid table.

[0121] In scenarios with large amounts of data, you can use bulk_insert to initialize the table.

[0122] You can use hoodie.cleaner.policy and hoodie.cleaner.commits.retained to control the version.

[0123] In the case of large concurrent writing to multiple / same partitions, errors may occur due to the limitations of Hudi itself. Here is a solution for multiple tasks to write to the same table: turn on optimistic_concurrency_control and combine it with zookeeper lock.

[0124] This application performs well in scenarios with billions or even tens of billions of data volumes. Compared with traditional Hive zippers, it can speed up the warehousing process by an average of about 30%. When used for statistical support needs, it can speed up the process by an average of 12% depending on the complexity of the business (mainly saving time in screening effective data). As shown in Table 6, the following is real test data for the warehousing scenario.

[0125] Table 6

[0126] Resource usage Stock data (items) Change data (items) Average time (minutes) Hudi Solution 200VC / 1T memory 3.2 billion (780 million effective) 2.3 million 9 Hive Zipper 200VC / 1T memory 3.2 billion (780 million effective) 2.3 million 13

[0127] This application has the following advantages:

[0128] 1. Based on Hudi's asynchronous update of COW type effective tables and invalid tables, combined with time partitioning and indexing mechanisms, the PreCombine and Payload features enable data to have the latest version while retaining all historical states and supporting fast backtracking and redoing. This solves the problems of slow zipper table updates, complex retransmission / rollback processes, and low statistical efficiency due to the need for screening.

[0129] 2. Fast data processing speed: Compared with various existing zipper processing methods, the execution efficiency of this application is very advantageous. The most time-consuming step in the zipper is to find the updated and newly added parts in the changed data to realize the opening and closing chain operation, but this application uses Hudi's PreCombine and Payload features supplemented by clever processing methods to greatly speed up this step, and it is more obvious under large data volumes.

[0130] 3. Convenient retransmission / rollback: A major drawback of ordinary zipper tables is that retransmission / rollback is complicated and requires complex processing to gradually roll back and redo according to time, which is time-consuming and labor-intensive. This application only needs to simply delete the corresponding partition.

[0131] 4. Easy to use: As the existing zipper tables carry a large amount of data, when the upper layer uses them for further processing, it is often necessary to select the effective data according to the judgment conditions. However, the application separates the birth and death tables, and the corresponding table can be selected according to the needs or the result of splicing the birth and death tables can be used, which converts the complex judgment into addition and subtraction.

[0132] 5. Extremely easy to maintain: The existing methods are either complex to process or complex to customize. In either case, there is a lot of operation and maintenance overhead. The core of this application is the application of logical design and features. It is the only zipper table implementation method using Hudi that does not require additional development. The operation and maintenance difficulty is comparable to that of a general Hudi table.

[0133] This embodiment is based on Hudi's asynchronous update of COW type effective table and invalid table, combined with time partition and index mechanism, through PreCombine and Payload features, so that the data has the latest version while retaining all historical states, and supports fast backtracking and redo, which solves the problems of slow zipper table update speed, complex retransmission / rollback process, and low statistical efficiency due to screening.

[0134] Reference Figure 4 , Figure 4 This is a flow chart of the second embodiment of the zipper upgrading method of the present application, based on the above Figure 1 The first embodiment shown is a second embodiment of the zipper upgrading method of the present application.

[0135] In the second embodiment, before step S10, the method further includes:

[0136] Step S01: Acquire data to be processed, and determine whether the data to be processed is modified data of an existing model or brand new data that requires a new model.

[0137] Furthermore, after step S01, it also includes: when the data to be processed is change data of an existing model, obtaining the change data, creating a new current time partition in the effective table, and copying the data in the previous time partition of the effective table to the current time partition.

[0138] Step S02: When the data to be processed is brand new data that requires a new model, a validation table and an invalidation table are created.

[0139] Accordingly, the step S10 includes:

[0140] Step S10': taking the new data as the changed data, obtaining the changed data, creating a new current time partition in the effective table, and returning to the step of processing the changed data and inserting the processed data into the current time partition.

[0141] For ease of understanding, refer to Figure 2 and Figure 3 This is for illustration only, but does not limit this application. If a new model is needed, Figure 3 Step 1, create Hudi's COW type effective table ods.USER_PACK_HUDI_COW and invalid table ods.USER_PACK_EXP_HUDI_COW, use the account period as the time partition (PARTITION_DATE), USER_ID as the primary key, and the package type (PACKAGE_TYPE) as the attribute that may change. At the same time, set the first appearance time (FIRST_APPEAR_TIME) and the birth / expiration time (EFF_TIME / EXP_TIME). The purpose of using EFF_TIME and EXP_TIME here is to make it easier for people to understand. The actual value stored in it is the same as the value of the current time partition.

[0142] If it is a changed data, it will flow to the "Data Processing Module". Assume that the following is the data of the previous account period, that is, the stock data that has been placed in the effective table. Assume that the invalid table is empty at this time

[0143] Before performing a zipper upgrade, this embodiment will also determine whether the data to be processed is changed data of an existing model or brand new data that requires a new model. When the data to be processed is brand new data that requires a new model, an effective table and an invalid table are created, thereby improving the reliability of the zipper upgrade and avoiding zipper upgrade failures.

[0144] Reference Figure 5 , Figure 5 This is a flow chart of the third embodiment of the zipper upgrading method of the present application. Based on the above embodiments, the third embodiment of the zipper upgrading method of the present application is proposed.

[0145] In the third embodiment, the step S30 includes:

[0146] Step S301: obtaining updated data in the effectiveness table, wherein the first appearance time of the updated data in the current account period is different from the effectiveness time.

[0147] Step S302: Obtain other field values ​​of the updated data in the previous account period except the time partition and the effective time, and set the values ​​of the time partition and the expiration time in the current account period to obtain an expiration table.

[0148] For ease of understanding, refer to Figure 2 and Figure 3 This is for illustration, but does not limit the present application. In one example, in the records corresponding to the primary keys (B, C) of the changed data in the current account period (20240402) in the effective table, the records with FIRST_APPEAR_TIME≠EFF_TIME (records corresponding to USER_ID value B) are the updated data. The "invalidation table update module" takes the values ​​of all fields except PARTITION_DATE and EFF_TIME corresponding to the primary keys in the previous account period, sets the value of EXP_TIME to the current account period, and adds it to the invalidation table through the partition variable. This step can retain the entire historical state of the data without complex closed-chain and open-chain operations.

[0149] This embodiment obtains the updated data in the effective table first, then the other field values ​​of the updated data in the previous account period except the time partition and the effective time, and sets the values ​​of the time partition and the expiration time in the current account period to obtain the expiration table, thereby improving the accuracy of the expiration table.

[0150] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the zipper upgrading method of the present application. More simple transformations based on this technical concept are all within the protection scope of the present application.

[0151] This application also provides a zipper upgrade device, please refer to Figure 6 , the zipper upgrading device comprises:

[0152] The data processing module 10 is used to obtain the changed data, create a new current time partition in the effective table, and copy the data in the previous time partition of the effective table to the current time partition. The effective table is a COW type effective table based on Hudi;

[0153] The effective table update module 20 is used to process the changed data, insert the processed data into the current time partition, and update the data in the current time partition of the effective table through a target operation, wherein the target operation is a specific operation combining PreCombine and Payload technologies;

[0154] The invalidation table updating module 30 is used to asynchronously update the invalidation table according to the effective table, and the invalidation table is a COW type invalidation table based on Hudi.

[0155] The zipper upgrade device provided by the present application adopts the zipper upgrade method in the above embodiment, which can solve the technical problems of the existing zipper table, such as slow update speed, complex retransmission / rollback process, and low statistical efficiency due to the need for screening. Compared with the prior art, the beneficial effects of the zipper upgrade device provided by the present application are the same as the beneficial effects of the zipper upgrade method provided by the above embodiment, and the other technical features of the zipper upgrade device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0156] The present application provides a zipper upgrading device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the zipper upgrading method in the above-mentioned embodiment 1.

[0157] Reference below Figure 7 , which shows a schematic diagram of the structure of a zipper upgrade device suitable for implementing the embodiment of the present application. The zipper upgrade device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The zipper upgrading device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0158] like Figure 7As shown, the zipper upgrade device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the zipper upgrade device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the zipper upgrade device to communicate with other devices wirelessly or wired to exchange data. Although the zipper upgrade device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided alternatively.

[0159] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0160] The zipper upgrade device provided by the present application adopts the zipper upgrade method in the above embodiment, which can solve the technical problems of the existing zipper table, such as slow update speed, complex retransmission / rollback process, and low statistical efficiency due to the need for screening. Compared with the prior art, the beneficial effects of the zipper upgrade device provided by the present application are the same as the beneficial effects of the zipper upgrade method provided by the above embodiment, and the other technical features of the zipper upgrade device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0161] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0162] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0163] The present application provides a storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the zipper upgrading method in the above-mentioned embodiment.

[0164] The storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0165] The storage medium may be included in the zipper upgrading device; or may exist independently without being assembled into the zipper upgrading device.

[0166] The above-mentioned storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the zipper upgrade device, the zipper upgrade device: obtains the change data, creates a new current time partition in the effective table, and copies the data in the previous time partition of the effective table to the current time partition. The effective table is a COW type effective table based on Hudi. The change data is processed and the processed data is inserted into the current time partition. The data in the current time partition of the effective table is updated through the target operation, wherein the target operation is a specific operation combining PreCombine and Payload technologies. The invalidation table is asynchronously updated according to the effective table, and the invalidation table is a COW type invalidation table based on Hudi.

[0167] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0168] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0169] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0170] The storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned zipper upgrade method, and can solve the technical problems of the existing zipper table, such as slow update speed, complex retransmission / rollback process, and low statistical efficiency due to the need for screening. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the zipper upgrade method provided in the above-mentioned embodiment, and will not be elaborated here.

[0171] The present application also provides a computer program product, including a computer program, which implements the above-mentioned zipper upgrading method when executed by a processor.

[0172] The computer program product provided by the present application can solve the technical problems of the existing zipper table, such as slow update speed, complex retransmission / rollback process, and low statistical efficiency due to the need for screening. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the zipper upgrade method provided by the above embodiment, which will not be elaborated here.

[0173] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A zipper upgrading method, characterized in that: The zipper upgrading method comprises: Obtain the changed data, create a new current time partition in the effective table, and copy the data in the previous time partition of the effective table to the current time partition. The effective table is a Hudi-based COW type effective table. Processing the changed data, inserting the processed data into the current time partition, and updating the data in the current time partition of the effective table through a target operation, wherein the target operation is a specific operation combining PreCombine and Payload technologies; The invalidation table is asynchronously updated according to the effective table, and the invalidation table is a COW type invalidation table based on Hudi.

2. The zipper upgrading method according to claim 1, characterized in that: The acquiring of the changed data, creating a new current time partition in the effective table, and copying the data in the previous time partition of the effective table to before the current time partition, further includes: Obtaining data to be processed, and determining whether the data to be processed is changed data of an existing model or brand new data that requires a new model; Accordingly, the acquisition of the changed data, the creation of a new current time partition in the effective table, and the copying of the data in the previous time partition of the effective table to the current time partition include: When the data to be processed is change data of an existing model, the change data is obtained, a new current time partition is created in the effective table, and data in the previous time partition of the effective table is copied to the current time partition.

3. The zipper upgrading method according to claim 2, characterized in that: After obtaining the data to be processed and determining whether the data to be processed is changed data of an existing model or brand new data requiring a new model, the method further includes: When the data to be processed is brand new data that requires a new model, creating a validation table and an invalidation table; The new data is used as the changed data, and the changed data is obtained, a new current time partition is created in the effective table, and the step of processing the changed data and inserting the processed data into the current time partition is returned.

4. The zipper upgrading method according to any one of claims 1 to 3, characterized in that: The effectiveness table creates an index for the changed data based on the Bucket Index, and uses the RecordKey and PartitionPath to retrieve the location information of the record to be updated within the current time partition range.

5. The zipper upgrading method according to any one of claims 1 to 3, characterized in that: The processing of the changed data and inserting the processed data into the current time partition includes: Set the first occurrence time of the change data to an invalid value; Insert the processed data into the current time partition.

6. The zipper upgrading method according to any one of claims 1 to 3, characterized in that: The asynchronously updating the invalidation table according to the effective table includes: Acquire updated data in the effective table, where the first appearance time of the updated data in the current account period is different from the effective time; The values ​​of other fields of the updated data in the previous account period except the time partition and the effective time are obtained, and the values ​​of the time partition and the expiration time are set in the current account period to obtain the expiration table.

7. A zipper upgrading device, characterized in that: The zipper upgrading device comprises: The data processing module is used to obtain the changed data, create a new current time partition in the effective table, and copy the data in the previous time partition of the effective table to the current time partition. The effective table is a COW type effective table based on Hudi; An effective table update module, used to process the changed data, insert the processed data into the current time partition, and update the data in the current time partition of the effective table through a target operation, wherein the target operation is a specific operation combining PreCombine and Payload technologies; The invalidation table update module is used to asynchronously update the invalidation table according to the effective table, and the invalidation table is a COW type invalidation table based on Hudi.

8. A zipper upgrading device, characterized in that: The zipper upgrading device comprises: a memory, a processor, and a zipper upgrading program stored in the memory and executable on the processor, wherein the zipper upgrading program implements the zipper upgrading method according to any one of claims 1 to 6 when executed by the processor.

9. A storage medium, characterized in that: The storage medium stores a zipper upgrade program, and when the zipper upgrade program is executed by the processor, the zipper upgrade method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The computer program product comprises a zipper upgrading program, and when the zipper upgrading program is executed by a processor, the zipper upgrading method according to any one of claims 1 to 6 is implemented.

Images