Data processing method and device

Abstract

The present application provides a data processing method and device applicable to the field of data access, including: receiving a target processing time input by a user; determining the changed business data at the target processing time based on the existing zipper data, full original business data, and incremental original business data read from Hive; determining the unchanged zipper, changed zipper, and newly added business data at the target processing time based on the existing zipper data and the changed business data; pre-processing the unchanged zipper, changed zipper, and newly added business data to obtain a target zipper, and determining the full zipper data based on the target zipper and the changed zipper; and overwriting the existing zipper data with the full zipper data. The present application can supplement and repair zipper data based on the full original business data and incremental original business data at any processing time, thereby improving the maintenance efficiency of zipper tables under massive data.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data access technology, and in particular to a data processing method and device. Background Art

[0002] In an enterprise-level data warehouse, a large number of zipper tables need to be maintained. These zipper tables record the changes in the entire life cycle of the corresponding business data and can restore the cross-section of the business data at a certain point in time at any time, which is very beneficial for data statistics and data analysis.

[0003] With the advent of the big data era, the demand for massive data storage and processing has led to the large-scale application of related components of the Hadoop ecosystem. Hive is often used as a data warehouse, providing Hadoop-based data storage capabilities. However, since Hive cannot directly update internal table data, it is difficult to maintain zipper tables using Hive. Summary of the Invention

[0004] The embodiments of the present application provide a data processing method and device, which can solve the technical problem in the prior art of high difficulty in maintaining zipper tables using Hive.

[0005] In a first aspect, an embodiment of the present application provides a data processing method, the method comprising:

[0006] Receive the target processing time input by the user, and determine the changed business data at the target processing time based on the existing zipper data, full original business data, and incremental original business data read from Hive;

[0007] Determining unchanged zippers, changed zippers, and newly added business data at the target processing time based on the existing zipper data and the changed business data;

[0008] Preprocessing the unchanged zipper, the changed zipper, and the newly added business data respectively to obtain a target zipper, and determining full zipper data based on the target zipper and the changed zipper;

[0009] The full zipper data is used to overwrite the existing zipper data to obtain updated existing zipper data.

[0010] In a feasible implementation, determining the changed business data at the target processing time based on the existing zipper data, full original business data, and incremental original business data read from Hive includes:

[0011] Restoring the full amount of business data at a first processing moment according to the existing zipper data, where the first moment is before the target processing moment;

[0012] The changed business data at the target processing moment is determined based on the full business data at the first processing moment, the full original business data at the target processing moment, and the incremental original business data at the target processing moment.

[0013] In a feasible implementation manner, determining the changed business data at the target processing moment based on the full business data at the first processing moment, the full original business data at the target processing moment, and the incremental original business data at the target processing moment includes:

[0014] Performing a full join comparison between the full amount of business data at the first processing time and the full amount of original business data at the target processing time;

[0015] The changed business data at the target processing moment is determined based on the comparison result and the incremental original business data at the target processing moment.

[0016] In a feasible implementation manner, pre-processing the unchanged zipper, the changed zipper, and the newly added service data to obtain the target zipper includes:

[0017] maintaining the unchanged zipper;

[0018] Performing preset closing and opening processing on the variable zipper;

[0019] Establishing a new chain for the newly added business data;

[0020] The unchanged zipper, the changed zipper after the closing and opening processes, and the established new chain are merged to obtain the target zipper.

[0021] In a feasible embodiment, performing preset closing and opening processing on the variable zipper includes:

[0022] When the target processing time is equal to the opening time of the variable zipper, the chain of the variable zipper is kept unchanged, and the zipper data of the variable zipper is updated;

[0023] Alternatively, when the target processing time is between the opening time and closing time of the changed zipper, the chain of the changed zipper is split to obtain the closed chain of the original zipper data and the newly opened chain of the new data;

[0024] Alternatively, when the target processing time is outside the opening time and closing time of the variable zipper, the chain of the variable zipper is supplemented to obtain a new chain.

[0025] In a feasible implementation manner, the chain of the variable zipper is supplemented to obtain a new chain, including:

[0026] When the target processing time T is greater than the maximum closed chain time in the variable zipper, a new chain [T, 9999-12-31) is added;

[0027] When the target processing time T is less than the minimum chain opening time MinTs in the variable zipper, a new chain [T, MinTs) is added;

[0028] When the target processing time T is less than or equal to the maximum chain closing time in the variable zipper and greater than or equal to the minimum chain opening time MinTs in the variable zipper, a new chain [T, MinGTs) is added; where MinGTs represents the chain opening time in the variable zipper that is greater than the target processing time T and closest to the target processing time T.

[0029] In a feasible implementation manner, determining full zipper data based on the target zipper and the changed zipper includes:

[0030] Performing a left anti-join operation on the target zipper and the changed zipper to obtain a first target zipper;

[0031] A UNION ALL operation is performed on the target zipper and the first target zipper to obtain the full zipper data.

[0032] In a second aspect, an embodiment of the present application provides a data processing device, the device comprising:

[0033] The first processing module is configured to receive a target processing time input by a user and determine the changed business data at the target processing time based on the existing zipper data, full original business data, and incremental original business data read from the data warehouse tool Hive;

[0034] A second processing module is configured to determine unchanged zippers, changed zippers, and newly added business data at the target processing moment based on the existing zipper data and the changed business data;

[0035] a zipper processing module, configured to pre-process the unchanged zipper, the changed zipper, and the newly added business data respectively to obtain a target zipper, and determine full zipper data based on the target zipper and the changed zipper;

[0036] An updating module is used to overwrite the existing zipper data with the full zipper data to obtain updated existing zipper data.

[0037] In a third aspect, an embodiment of the present application provides an electronic device, comprising: at least one processor and a memory;

[0038] The memory stores computer-executable instructions;

[0039] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the data processing method provided in the first aspect.

[0040] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the data processing method provided in the first aspect is implemented.

[0041] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the data processing method provided in the first aspect.

[0042] The data processing method and device provided by the embodiment of the present application can be applied to the field of data access, including: receiving a target processing time input by a user, determining the changed business data at the target processing time based on the existing zipper data, full original business data, and incremental original business data read from Hive; determining the unchanged zipper, changed zipper, and newly added business data at the target processing time based on the existing zipper data and the changed business data; pre-processing the unchanged zipper, changed zipper, and newly added business data respectively to obtain a target zipper, and determining the full zipper data based on the target zipper and the above-mentioned changed zipper; and overwriting the above-mentioned existing zipper data with the full zipper data. After the user inputs any processing time, the embodiment of the present application can supplement and repair the zipper data based on the full original business data and incremental original business data at the processing time, complete the update of the zipper data, and improve the maintenance efficiency of the zipper table under massive data. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments of the present application or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0044] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application;

[0045] Figure 2 This is another flowchart of the data processing method provided in an embodiment of the present application;

[0046] Figure 3 A schematic diagram of a program module of a data processing device provided in an embodiment of the present application;

[0047] Figure 4 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. In addition, although the disclosure in this application is introduced according to one or more exemplary examples, it should be understood that each aspect of these disclosures can also constitute a complete implementation method separately.

[0049] It should be noted that the brief descriptions of terms in this application are only for the purpose of facilitating the understanding of the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their ordinary and usual meanings.

[0050] In the specification and claims of this application, as well as in the accompanying drawings, the terms "first," "second," and the like are used to distinguish similar or similar objects or entities and are not necessarily intended to limit a particular order or precedence, unless otherwise noted. It should be understood that such terms are interchangeable where appropriate, e.g., embodiments of this application can be implemented in an order other than that shown or described in the drawings or descriptions.

[0051] In addition, the terms "comprises" and "comprising" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to those components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0052] The term "module" as used in this application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0053] The following are some explanations of the terms that appear in the examples of this application:

[0054] Hive: Hive is a data warehouse analysis system built on the Hadoop distributed file system. Hive can map structured data files into a database table and provide complete Structured Query Language (SQL) query functions. SQL statements are eventually converted into MapReduce tasks running on a big data cluster to process massive data.

[0055] Spark: Apache Spark is a computing engine designed for large-scale data processing and is known for its high performance. Under in-memory computing, the processing speed of Spark tasks is much faster than that of MapReduce tasks. Spark is often used for the calculation and processing of big data tasks. The Spark-sql component can convert SQL statements into Spark tasks to run on big data clusters.

[0056] Spark DataFrame (DF): It is a distributed dataset with a schema defined by Spark. It is a logical collection that describes the structured distributed data in the Spark processing process.

[0057] Zipper tables: A zipper table records the state of specific data within a specific time interval, or the data changes at a specific point in time. Zipper tables effectively trace the state of data at each time point and save space for recording data changes. They are a commonly used table type in data warehouses.

[0058] Enterprise data warehouses require the maintenance of numerous zipper tables. These tables record changes throughout the entire lifecycle of business data and can be used to restore data slices at a specific point in time, making them highly useful for data statistics and analysis. With the advent of the big data era, the demand for massive data storage and processing has led to the widespread adoption of components within the Hadoop ecosystem. Hive is often used as a data warehouse, providing Hadoop-based data storage capabilities. However, since Hive cannot directly update internal table data, maintaining zipper tables in Hive is challenging.

[0059] In the existing technical solutions, Hive zipper data is constructed in a forward order for full or incremental data, that is, the data involved in the zipper data construction each time is the latest business data. This type of solution is only applicable to generating zipper data by time under normal conditions, and cannot correct or supplement the zipper data.

[0060] Among them, the existing technical solutions only take into account the normal forward construction situation, and do not consider the problems that may arise in the batch establishment of zipper data and the repair of zipper table damage. Such problems often occur in the big data production environment of various enterprises: the source system only partially provides recent data of a certain business data, and then supplements historical data; there are broken links in the existing zipper data, etc. When such problems occur, the existing technical solutions will find it difficult to accurately complete the construction of the zipper data.

[0061] To address this technical problem, the present application provides a data processing method for flexibly constructing and repairing Hive zipper data, based on the Hive component and using the Spark computing engine. This method allows data from any historical point in time to be used to supplement and repair zipper data, supports multi-batch out-of-order processing of data, and ensures that all data is processed, regardless of the order in which it is executed. The use of the Spark component also significantly improves data processing efficiency under massive data volumes. This is explained in detail below using a detailed embodiment.

[0062] Reference Figure 1 , Figure 1 This is a flow chart of a data processing method provided in an embodiment of the present application. In one feasible implementation, the method can be implemented by establishing a Spark task, specifically including:

[0063] S101. Receive a target processing time input by a user, and determine the changed business data at the target processing time based on the existing zipper data, full original business data, and incremental original business data read from Hive.

[0064] In some embodiments, a Spark task may be pre-established, and when the input target processing time is received, the Spark task is used to read the existing zipper data, the full original business data, and the incremental original business data from Hive.

[0065] In some embodiments, the full amount of business data at a first processing moment can be restored based on the read stock zipper data, where the first moment is before the target processing moment. For example, if the target processing moment is January 30, the first moment can be January 29.

[0066] The changed business data at the target processing moment is determined based on the full business data at the first processing moment, the full original business data at the target processing moment, and the incremental original business data at the target processing moment.

[0067] In a feasible implementation, the full business data at the first processing moment can be compared with the full original business data at the target processing moment through a full join, and then the changed business data at the target processing moment can be determined based on the comparison results and the incremental original business data at the target processing moment.

[0068] Among them, FULL JOIN connects two zipper tables based on the business key. Regardless of the left table or the right table, unmatched data is also retained in the result.

[0069] S102. Determine unchanged zippers, changed zippers, and newly added business data at the target processing time based on the existing zipper data and the changed business data.

[0070] In some embodiments, a zipper that only exists in the existing zipper data can be determined as a changing zipper; a zipper that can match the business key of the changing business data can be determined as a changing zipper; data that only exists in the changing business data at the target processing time can be determined as new business data.

[0071] S103 , pre-processing the unchanged zipper, the changed zipper, and the newly added business data respectively to obtain a target zipper, and determining full zipper data based on the target zipper and the changed zipper.

[0072] In some embodiments, after pre-processing the unchanged zipper, changed zipper, and newly added business data, the full amount of zipper data can be determined based on operations such as LEFT ANTI JOIN and UNION ALL.

[0073] S104: Overwrite the existing zipper data with the full zipper data to obtain updated existing zipper data.

[0074] The data processing method provided in the embodiment of the present application can supplement and repair the zipper data based on the full original business data and incremental original business data at any processing time after the user enters the processing time, complete the update of the zipper data, and effectively improve the maintenance efficiency of the zipper table under massive data.

[0075] Based on the content described in the above embodiments, in a feasible implementation of the present application, the following pre-processing can be performed on the unchanged zipper, the changed zipper, and the newly added service data respectively:

[0076] Keep the zipper unchanged.

[0077] The above variable zipper is subjected to preset closing and opening processing.

[0078] Establish a new chain for the above-mentioned new business data.

[0079] The unchanged zipper, the changed zipper after closing and opening, and the newly established chain are combined to obtain the target zipper.

[0080] In some embodiments, when performing the preset closing and opening processing on the variable zipper, if the target processing time is equal to the opening time of the variable zipper, the chain of the variable zipper remains unchanged and the zipper data of the variable zipper is updated.

[0081] Alternatively, when the target processing time is between the opening time and closing time of the changed zipper, the chain of the changed zipper is split to obtain the closed chain of the original zipper data and the newly opened chain of the new data.

[0082] Alternatively, when the target processing time is outside the opening time and closing time of the variable zipper, the chain of the variable zipper is supplemented to obtain a new chain.

[0083] In some embodiments, when the target processing time T is less than the minimum chain opening time MinTs in the variable zipper, a new chain [T, MinTs) is added; when the target processing time T is less than or equal to the maximum chain closing time in the variable zipper, and greater than or equal to the minimum chain opening time MinTs in the variable zipper, a new chain [T, MinGTs) is added; wherein MinGTs represents the chain opening time in the variable zipper that is greater than the target processing time T and is closest to the target processing time T.

[0084] In some embodiments, a LEFT ANTI JOIN operation may be performed on the target zipper and the changed zipper to obtain a first target zipper; and then a UNION ALL operation may be performed on the target zipper and the first target zipper to obtain the full zipper data.

[0085] The above-mentioned data processing method provided in the embodiment of the present application can flexibly use full / incremental data at different time points to construct, supplement, and repair zipper data when using Hive as a data warehouse.

[0086] In order to better understand the embodiments of the present application, refer to Figure 2 , Figure 2 This is another flow chart of the data processing method provided in the embodiments of the present application. In one feasible implementation, the data processing method includes:

[0087] S1. Read the existing zipper data, full original business data, and incremental original business data from the data warehouse tool Hive based on the Spark task.

[0088] In some embodiments, the above raw data must contain at least the following:

[0089] (1) Business key: a field or set of fields that uniquely locates a business or transaction. Zipper data will describe the changes in the business or transaction within a certain time range.

[0090] (2) Data time: the actual time when the factual data changes. This time will serve as the only basis for constructing the zipper time chain.

[0091] For example, refer to Table 1, which is a schematic table of raw data.

[0092] Table 1: Schematic table of raw data

[0093] Business Key Data time State 1 State n 10001 20211010 A A 10001 20211023 A B 10002 20211010 A A 10002 20211015 B A

[0094] In some embodiments, the zipper data must contain at least the following:

[0095] (1) Business key: a field or set of fields that uniquely locates a business or transaction. Zipper data will describe the changes in the business or transaction within a certain time range.

[0096] (2) Chain information: The time interval [chain opening time, chain closing time) constitutes a complete chain information.

[0097] Among them, the chain opening time: describes the start time of the period to which a single zipper data belongs. For data without a chain opening time, the default chain opening time is 1900-01-01.

[0098] Closed chain time: describes the end time of the cycle to which a single zipper data belongs. For data without a closed chain, the default closed chain date is 9999-12-31.

[0099] In some embodiments, Spark can be used as the processing engine for Hive data.

[0100] In some embodiments, Spark Dataframe (DF) can be used to describe and process massive distributed data sets and perform in-memory computing. Spark-SQL can be used to read and process data.

[0101] S2. Processing time T for receiving input.

[0102] S3. Based on the acquired existing zipper data, restore the full business data at time T-1; based on the acquired full original data, read the full original business data at time T; based on the acquired incremental original business data, read the incremental original business data at time T.

[0103] Here, T-1 is before the processing time T. For example, when the processing time T is January 30, T-1 is January 29.

[0104] S4. Perform a full join (FULL JOIN) on the full business data at time T-1 and the full original business data at time T to compare the changed business data at time T, and obtain the changed business data at time T based on the changed business data at time T and the incremental original business data at time T.

[0105] Among them, FULL JOIN connects two tables based on the business key. Regardless of the left table or the right table, unmatched data is also retained in the result.

[0106] S5. Perform a full connection comparison on the existing zipper data and the changed business data at time T obtained above, and separate the unchanged zipper DF1, the newly added data DF3 and the changed zipper DF2 according to the existence of the data.

[0107] Among them, zippers that only exist in the existing zipper data can be determined as changing zippers; zippers that can match the business keys of changing business data can be determined as changing zippers; data that only exists in the changing business data at the target processing time can be determined as new business data.

[0108] The unchanged zipper DF1 is kept as is; a new chain [T, 9999-12-31] is created for the newly added data DF3. The following processing is performed on the changed zipper DF2:

[0109] For a zipper that meets the time T = chain opening time Ts, the chain remains unchanged and only the data is updated.

[0110] For zippers whose time T is in the interval (opening time Ts, closing time Te), the original chain is split to obtain the original zipper data closed chain [Ts, T] and the new data new open chain [T, Te).

[0111] For zippers whose time T is not in the interval (opening time Ts, closing time Te), when the new data date T> the maximum closing time Te, open a new chain [T, 9999-12-31); record it as DF_2_3_1; when the new data date T< the minimum opening time MinTs, open a new chain [T, MinTs), record it as DF_2_3_2; for data that does not meet DF2_3_1 and DF_2_3_2, perform a chain-filling operation to find the start time MinGTs that is greater than time T and the most recent, and open a new chain [T, MinGTs).

[0112] S6. Merge all the zippers in S5 to obtain a merged zipper.

[0113] S7. Perform a left antijoin (LEFT ANTIJOIN) on the merged zipper in S6 and the changed zipper obtained in S5 to obtain the unchanged zipper.

[0114] LEFT ANTI JOIN connects two tables based on the business key and filters the results where the left table does not contain data from the right table, that is, obtains the difference between the left and right tables.

[0115] S8. Perform a UNION ALL operation on the unchanged zipper obtained in S7 and the merged zipper in S6 to obtain the full zipper data.

[0116] S9. Use the obtained full zipper data to overwrite the original stock zipper data to obtain the current latest stock zipper data.

[0117] The data processing method provided in the embodiment of the present application can use data from any historical point in time to supplement and repair zipper data, and can support multi-batch out-of-order processing of data. No matter what order is used for execution, as long as all data is processed, the zipper table data will remain accurate and consistent. In addition, by establishing Spark tasks, the data processing efficiency under massive data can also be greatly improved.

[0118] Based on the contents described in the above embodiments, a data processing device is also provided in the embodiments of the present application. Figure 3 , Figure 3 This is a schematic diagram of a program module of a data processing device provided in an embodiment of the present application, wherein the data processing device includes:

[0119] The first processing module 301 is used to receive the target processing time input by the user, and determine the changed business data at the target processing time based on the existing zipper data, full original business data and incremental original business data read from the data warehouse tool Hive.

[0120] The second processing module 302 is configured to determine unchanged zippers, changed zippers, and newly added business data at the target processing moment based on the existing zipper data and the changed business data.

[0121] The zipper processing module 303 is configured to pre-process the unchanged zipper, the changed zipper, and the newly added business data respectively to obtain a target zipper, and determine full zipper data based on the target zipper and the changed zipper.

[0122] The updating module 304 is configured to overwrite the existing zipper data with the full zipper data to obtain updated existing zipper data.

[0123] The data processing device provided in the embodiment of the present application can supplement and repair the zipper data based on the full original business data and incremental original business data at any processing time after the user enters the processing time, complete the update of the zipper data, and effectively improve the maintenance efficiency of the zipper table under massive data.

[0124] In a feasible implementation, the first processing module 301 is configured to:

[0125] Restore the full business data at the first processing moment based on the existing zipper data, where the first moment is before the target processing moment; determine the changed business data at the target processing moment based on the full business data at the first processing moment, the full original business data at the target processing moment, and the incremental original business data at the target processing moment.

[0126] In a feasible implementation, the first processing module 301 is configured to:

[0127] Perform a full join comparison between the full business data at the first processing moment and the full original business data at the target processing moment; determine the changed business data at the target processing moment based on the comparison result and the incremental original business data at the target processing moment.

[0128] In a feasible implementation, the zipper processing module 303 is used to:

[0129] maintaining the unchanged zipper;

[0130] Performing preset closing and opening processing on the variable zipper;

[0131] Establishing a new chain for the newly added business data;

[0132] The unchanged zipper, the changed zipper after the closing and opening processes, and the established new chain are merged to obtain the target zipper.

[0133] In a feasible implementation, the zipper processing module 303 is used to:

[0134] When the target processing time is equal to the unzipping time of the variable zipper, the chain of the variable zipper is kept unchanged, and the zipper data of the variable zipper is updated.

[0135] Alternatively, when the target processing time is between the opening time and the closing time of the changed zipper, the chain of the changed zipper is split to obtain the closed chain of the original zipper data and the newly opened chain of the new data.

[0136] Alternatively, when the target processing time is outside the opening time and closing time of the variable zipper, the chain of the variable zipper is supplemented to obtain a new chain.

[0137] In a feasible implementation, the zipper processing module 303 is used to:

[0138] When the target processing time T is greater than the maximum closed chain time in the variable zipper, a new chain [T, 9999-12-31) is added.

[0139] When the target processing time T is less than the minimum chain opening time MinTs in the variable zipper, a new chain [T, MinTs) is added.

[0140] When the target processing time T is less than or equal to the maximum chain closing time in the variable zipper and greater than or equal to the minimum chain opening time MinTs in the variable zipper, a new chain [T, MinGTs) is added; where MinGTs represents the chain opening time in the variable zipper that is greater than the target processing time T and closest to the target processing time T.

[0141] In a feasible implementation, the zipper processing module 303 is used to:

[0142] A LEFT ANTI JOIN operation is performed on the target zipper and the changed zipper to obtain a first target zipper; and a UNION ALL operation is performed on the target zipper and the first target zipper to obtain the full zipper data.

[0143] It should be noted that the specific implementation of the first processing module 301, the first processing module 302, the zipper processing module 303 and the update module 304 in the embodiment of the present application can be referred to. Figures 1 to 2 The relevant contents in the illustrated embodiment will not be described in detail here.

[0144] Furthermore, based on the contents described in the above embodiments, an electronic device is also provided in an embodiment of the present application, which includes at least one processor and a memory; wherein the memory stores computer-executable instructions; the above-mentioned at least one processor executes the computer-executable instructions stored in the memory to implement the various steps of the data processing method described in the above embodiments, which will not be repeated here in this embodiment.

[0145] In order to better understand the embodiments of the present application, refer to Figure 4 , Figure 4 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.

[0146] like Figure 4As shown, the electronic device 40 of this embodiment includes: a processor 401 and a memory 402; wherein:

[0147] Memory 402, for storing computer-executable instructions;

[0148] The processor 401 is used to execute the computer-executable instructions stored in the memory to implement the various steps of the data processing method described in the above embodiment. For details, please refer to the relevant description in the above method embodiment, which will not be repeated here in this embodiment.

[0149] Optionally, the memory 402 may be independent or integrated with the processor 401 .

[0150] When the memory 402 is independently provided, the device further includes a bus 403 for connecting the memory 402 and the processor 401 .

[0151] Furthermore, based on the contents described in the above embodiments, a computer-readable storage medium is also provided in an embodiment of the present application, in which computer execution instructions are stored. When the processor executes the computer execution instructions, the various steps of the data processing method described in the above embodiments are implemented. For details, please refer to the relevant description in the above method embodiments, which will not be repeated here in this embodiment.

[0152] Furthermore, based on the contents described in the above embodiments, a computer program product is also provided in the embodiments of the present application, including a computer program. When the computer program is executed by a processor, the various steps of the data processing method described in the above embodiments can be implemented. For details, please refer to the relevant description in the above method embodiments, which will not be repeated here in this embodiment.

[0153] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or module, which can be electrical, mechanical or other forms.

[0154] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected to achieve the purpose of the solution of this embodiment according to actual needs.

[0155] In addition, the functional modules in the various embodiments of the present application may be integrated into a single processing unit, or each module may exist physically separately, or two or more modules may be integrated into a single unit. The aforementioned module integration unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0156] The above-mentioned integrated module implemented in the form of a software functional module can be stored in a computer-readable storage medium. The above-mentioned software functional module is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform some steps of the method described in various embodiments of the present application.

[0157] It should be understood that the processor described above may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), or application-specific integrated circuits (ASICs). A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the application may be directly executed by a hardware processor or by a combination of hardware and software modules within the processor.

[0158] The memory may include a high-speed RAM memory, and may also include non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk.

[0159] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be classified into address buses, data buses, and control buses. For ease of illustration, the buses in the drawings of this application are not limited to just one bus or just one type of bus.

[0160] The storage medium may be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0161] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and storage medium can be located in an application-specific integrated circuit (ASIC). Of course, the processor and storage medium can also exist as discrete components in an electronic device or a main control device.

[0162] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0163] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Receive the target processing time input by the user, and determine the changed business data at the target processing time based on the existing zipper data, full original business data, and incremental original business data read from the data warehouse tool Hive; Determining unchanged zippers, changed zippers, and newly added business data at the target processing time based on the existing zipper data and the changed business data; Merging the unchanged zipper, the changed zipper after closing and opening, and a new chain established based on the newly added business data to obtain a target zipper, and determining full zipper data based on the target zipper and the changed zipper; Overwriting the existing zipper data with the full zipper data to obtain updated existing zipper data; The closing and opening processing of the variable zipper includes: When the target processing time is equal to the opening time of the variable zipper, the chain of the variable zipper is kept unchanged, and the zipper data of the variable zipper is updated; Alternatively, when the target processing time is between the opening time and closing time of the changed zipper, the chain of the changed zipper is split to obtain the closed chain of the original zipper data and the newly opened chain of the new data; Alternatively, when the target processing time is outside the opening time and closing time of the variable zipper, the chain of the variable zipper is supplemented to obtain a new chain.

2. The method according to claim 1, characterized in that Determining the changed business data at the target processing time based on the existing zipper data, full original business data, and incremental original business data read from Hive includes: Restoring the full amount of business data at a first processing time according to the existing zipper data, where the first processing time is before the target processing time; The changed business data at the target processing moment is determined based on the full business data at the first processing moment, the full original business data at the target processing moment, and the incremental original business data at the target processing moment.

3. The method according to claim 2, characterized in that The determining, based on the full amount of business data at the first processing moment, the full amount of original business data at the target processing moment, and the incremental original business data at the target processing moment, of the changed business data at the target processing moment includes: Performing a full join comparison between the full amount of business data at the first processing time and the full amount of original business data at the target processing time; The changed business data at the target processing moment is determined based on the comparison result and the incremental original business data at the target processing moment.

4. The method according to claim 1, wherein The method of supplementing the chain of the variable zipper to obtain a new chain includes: When the target processing time T is greater than the maximum closed chain time in the variable zipper, a new chain [T, 9999-12-31) is added; When the target processing time T is less than the minimum chain opening time MinTs in the variable zipper, a new chain [T, MinTs] is added; When the target processing time T is less than or equal to the maximum chain closing time in the variable zipper and greater than or equal to the minimum chain opening time MinTs in the variable zipper, a new chain [T, MinGTs) is added; where MinGTs represents the chain opening time in the variable zipper that is greater than the target processing time T and closest to the target processing time T.

5. The method according to claim 1, characterized in that The determining of full zipper data based on the target zipper and the changed zipper includes: Performing a left anti-join operation on the target zipper and the changed zipper to obtain a first target zipper; A UNION ALL operation is performed on the target zipper and the first target zipper to obtain the full zipper data.

6. A data processing device, characterized in that: The device comprises: The first processing module is configured to receive a target processing time input by a user and determine the changed business data at the target processing time based on the existing zipper data, full original business data, and incremental original business data read from the data warehouse tool Hive; A second processing module is configured to determine unchanged zippers, changed zippers, and newly added business data at the target processing moment based on the existing zipper data and the changed business data; a zipper processing module, configured to merge the unchanged zipper, the changed zipper after closing and opening, and the newly added business data to obtain a target zipper, and determine full zipper data based on the target zipper and the changed zipper; An updating module, configured to overwrite the existing zipper data with the full zipper data to obtain updated existing zipper data; The zipper processing module is specifically used to: When the target processing time is equal to the opening time of the variable zipper, the chain of the variable zipper is kept unchanged, and the zipper data of the variable zipper is updated; Alternatively, when the target processing time is between the opening time and closing time of the changed zipper, the chain of the changed zipper is split to obtain the closed chain of the original zipper data and the newly opened chain of the new data; Alternatively, when the target processing time is outside the opening time and closing time of the variable zipper, the chain of the variable zipper is supplemented to obtain a new chain.

7. An electronic device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the data processing method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, and when a processor executes the computer-executable instructions, the data processing method according to any one of claims 1 to 5 is implemented.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the data processing method according to any one of claims 1 to 5 is implemented.

Images