Incremental data processing method and device and storage medium

Abstract

The invention provides an incremental data processing method and device and a storage medium, and the method comprises the steps: firstly receiving incremental data and corresponding service time information from a data source, then writing the incremental data into corresponding partitions according to the service time, and synchronously updating the first modification time information of each partition. Each partition can be used for storing the data of the business time information within the business time range, and it is ensured that each partition only contains the data of which the business time is within the specific range, so that the time sequence and logicality of the data are kept. And determining a first target partition with late data according to the data processing time window in combination with the service time range corresponding to each partition and the first modification time information thereof. According to the embodiment of the invention, by flexibly coping with the limitation of the time window range, the late data occurring in the data increment processing process can be effectively identified and processed, so that the accuracy and timeliness of data processing are remarkably improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to, but are not limited to, the technical field of data processing, and in particular to a processing method, device and storage medium for incremental data. BACKGROUND

[0002] In the field of data processing, especially in the scenario of large-scale business data processing, incremental data processing is a common and effective data processing method. In the traditional incremental data processing method, a time window range is usually set to limit the data set for each processing. This method helps to control the consumption of computing resources and ensure the real-time nature of data processing, but there are still the following problems in actual operation: due to network delay, system processing speed difference and other reasons, part of the data cannot arrive at the data processing node within the time window range, this part of data is late data, if the late data is not identified and processed in time, it will affect the integrity and accuracy of the data processing result. SUMMARY

[0003] Embodiments of the present application provide a processing method, device and storage medium for incremental data, aiming to effectively identify late data in incremental data processing, so as to improve the accuracy and timeliness of data processing.

[0004] In one aspect, the present application provides a processing method for incremental data, comprising: receiving incremental data and business time information of the incremental data from a data source; according to the business time information, writing the incremental data into a corresponding partition, and updating first modification time information of the partition, wherein the partition is used to store data within the business time range of the business time information, and the first modification time information is used to indicate the time of updating data of the partition; according to the data processing time window and the business time range and the first modification time information corresponding to each partition, determining a first target partition in which there is late data.

[0005] In another aspect, the present application further provides an electronic device, comprising: one or more processors; a memory having one or more programs stored thereon, when the one or more programs are executed by the one or more processors, the one or more processors implement the processing method for incremental data as described above.

[0006] In another aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, when the program is executed by a processor, the processing method for incremental data as described above is implemented.

[0007] In another aspect, the present application further provides a computer program product comprising a computer program, when the computer program is executed by a processor, the processing method for incremental data as described above is implemented.

[0008] In the embodiment of the present application, first, the incremental data and its corresponding business time information are received from the data source, then the incremental data is accurately distributed to the corresponding partition according to the business time, and the first modification time information of each partition is updated synchronously, which accurately records the time point when the data in the partition is last updated and can be used to indicate the time when the partition updates the data. Each partition can be used to store the data within the business time range of the business time information, ensuring that each partition only contains data whose business time falls within a specific range, thereby maintaining the time sequence and logicality of the data. Then, according to the data processing time window, in combination with the business time range and the latest modification time information (i.e. the first modification time information) of each partition, the first target partition where the late data exists can be determined. That is, according to the analysis of these information, the first target partition where the late data exists can be accurately identified, which is delayed due to various reasons although the business time has exceeded the current processing time window. The embodiment of the present application can effectively identify and process the late data that appears in the data incremental processing process by flexibly coping with the limitation of the time window range, thereby significantly improving the accuracy and timeliness of data processing. BRIEF DESCRIPTION OF DRAWINGS

[0009] Figure 1 is a flowchart of the processing method of incremental data provided by an embodiment of the present application; Figure 2 is a processing flow diagram of incremental data of a data system provided by an embodiment of the present application; Figure 3 is a processing flow diagram of incremental data of a data system provided by another embodiment of the present application; Figure 4 is a processing flow diagram of incremental data provided by an embodiment of the present application; Figure 5 is a flowchart of late data identification display provided by an embodiment of the present application; Figure 6 is a data supplementing flowchart provided by an embodiment of the present application; Figure 7 is a late data monitoring user interface provided by a specific example of the present application; Figure 8 is a supplementing menu provided by a specific example of the present application. DETAILED DESCRIPTION

[0010] In order to make the purpose, technical method and advantages of the present application more clear and explicit, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0011] It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in an order different from that in the flowchart. In the description of the specification and claims and the above description of the drawings, the meaning of multiple (or multiple) is more than two, greater than, less than, more than, etc. is understood as not including the number, above, below, within, etc. is understood as including the number. If it is described as "first", "second", etc. is only used to distinguish the technical features for the purpose, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of indicated technical features or implicitly indicating the relationship between the indicated technical features.

[0012] Full data processing refers to the processing of all data from the beginning of history in each stage of the data processing flow. For example, after collecting all data once a day, all processing and index calculation. The advantage is to provide full flow data consistency guarantee with the simplest logic. The disadvantage is that as time goes on, the accumulated business data, the resources and time required for full calculation are more and more, and finally exceed the system resource range. Incremental processing processes relatively new data from the last time or data within a certain time window in the data processing flow. Compared with full data processing, it greatly reduces the data set of each processing, reduces the computing resource cost and execution time, and the system's demand for resources remains stable overall.

[0013] In dealing with large-scale business data processing tasks, incremental data processing is an efficient strategy, which is based on distinguishing the time dimension of data to optimize storage and processing procedures. Traditionally, incremental data processing tends to simply partition data according to the data collection time, although this approach is simple, but in the face of highly time-sensitive and complex business logic scenarios, its limitations become particularly prominent. Especially in complex environments where there is a significant deviation between business time and data collection time, simply relying on collection time for partitioning is likely to lead to misunderstanding of the business state, especially in critical applications that pursue real-time or near real-time processing capabilities, small differences in network latency and system processing speed can generate a large amount of late data. If this kind of data continues to use collection time as the basis for partitioning, it will inevitably be misclassified into mismatched partitions, thereby severely weakening the accuracy and timeliness of data processing. In order to effectively solve the problem of late data, time windows are widely used in data processing procedures, which limit the data within a specific time period covered by each calculation. However, this time window is essentially based on an assumption that data may be late within a certain range, and data beyond this range is considered no longer to be reprocessed. But the reality is complex, on the one hand, it is challenging to accurately determine whether the data is late; on the other hand, there is a lack of accurate recalculation capability for late data. The traditional time window mechanism is not effective in dealing with this problem, as it is difficult to accurately define the range of late data and perform effective recalculation. This not only increases processing costs, but also sacrifices data accuracy. The setting of the time window often depends on domain knowledge, but a large time window will lead to unnecessary consumption of computing resources, while a small time window may miss a large amount of late data, thereby affecting the accuracy of data processing results. Furthermore, due to the inability to effectively identify late data, it is difficult to accurately optimize the selection of the time window, and further recalculation measures cannot be taken for late data. In order to make up for this deficiency, data backfilling emerges as a remedy. Although it can effectively deal with challenges such as changes in business logic, the inclusion of new indicators, data loss, or system delays, its execution process is complex and requires high accuracy, and it needs to accurately define the backfilling range and recalculate related data.

[0014] It is worth noting that in order to optimize the incremental data processing process, the traditional strategy usually uses time windows as the natural boundaries of data processing periods to balance the efficient use of computing resources and the timeliness requirements of data processing. However, in practice, the unpredictability of network latency and the difference in system processing capacity often cause some data to not arrive in the preset time window in time, resulting in late data. If these late data are not identified and processed in time, it will directly affect the timeliness and accuracy of data processing results.

[0015] In order to effectively identify the late data in the incremental data processing, and improve the accuracy and timeliness of data processing, the embodiment of the present application provides a processing method of incremental data, an electronic device, a computer readable storage medium and a computer program product, wherein the incremental data and the corresponding business time information are received from the data source, then the incremental data is accurately distributed to the corresponding partition according to the business time, and the first modification time information of each partition is updated synchronously, which accurately records the time point when the data in the partition is last updated, and can be used to indicate the time when the data in the partition is updated. Each partition can be used to store the data within the business time range of the business time information, so that only the data whose business time falls within a specific range is contained in each partition, thereby maintaining the time sequence and logicality of the data. Then, according to the data processing time window, in combination with the business time range and the latest modification time information (i.e. the first modification time information) of each partition, the first target partition in which the late data exists can be determined. That is, according to the comparison and analysis of the information, the first target partition in which the late data exists can be accurately identified, which is the late data that has exceeded the current processing time window due to various reasons. According to the embodiment of the present application, the incremental data is stored in the partition according to the business time, and in combination with the business time range, the modification time information of the partition and the data processing time window, the late data exceeding the time window range is identified and processed by comparison and statistics. Not only can the actual business situation of the data be accurately reflected, but also the limitation of the time window range can be flexibly coped with, the late data in the incremental data processing can be effectively identified, and the accuracy and timeliness of data processing can be improved.

[0016] Based on the above analysis, the embodiment of the present application will be further described in combination with the drawings.

[0017] Reference Figure 1 , Figure 1 is a flowchart of the processing method of incremental data provided by an embodiment of the present application, which can include but is not limited to steps S110 to S130.

[0018] Step S110: receiving incremental data and business time information of the incremental data from a data source; Step S120: writing the incremental data into the corresponding partition according to the business time information, and updating the first modification time information of the partition, wherein the partition is used to store the data within the business time range of the business time information, and the first modification time information is used to indicate the time when the data in the partition is updated; Step S130: determining the first target partition in which the late data exists according to the data processing time window and the business time range and the first modification time information of each partition.

[0019] Exemplarily, the data source can be a relational database (such as MySQL, Oracle), a NoSQL database (such as MongoDB, Cassandra), a log file, a message queue (such as Kafka), an API interface, etc. According to the difference of the data source, different connection technologies or libraries (such as JDBC, ODBC, RESTful API client, etc.) can be used to establish a connection. Once the connection with the data source is successfully established, the incremental data and its business time information can be received. Incremental data, that is, those data that are newly added or changed since the last data collection period, are the key to maintaining the real-time and accuracy of the data. And the business time information of the incremental data refers to the specific time point when the data is actually generated or occurs. Taking an e-commerce system as an example, the time when a user places an order is the business time of the order data.

[0020] Exemplarily, when step S110 is performed, the method of receiving incremental data can be determined based on the type and characteristics of the data source, including but not limited to: based on timestamp: if the data source supports timestamp, the incremental data can be obtained by querying the data within a specific time range; based on change log: some databases (such as MySQL's binlog, Oracle's ChangeData Capture) provide change log function, which can record the change of data. By reading these logs, incremental data can be captured; based on trigger: set a trigger in the database, when the data changes, automatically trigger an event, and send the changed data to a specified location (such as message queue); based on polling: periodically query the data source, and obtain the incremental data by comparing the difference between the current data and the last query result.

[0021] For example, after obtaining the business time information corresponding to incremental data, the system can determine which partition the data should belong to based on this information. Once the partition is determined, the incremental data can be written to the corresponding partition. It's important to note that if the partition doesn't already exist (e.g., for a new time range), it may be necessary to create it first. Consider an e-commerce system that partitions order data by date. Each partition stores all order data within a specific date. After retrieving the latest order data (containing the business time information of the orders, i.e., the time the user placed the order) from data sources such as an order database, message queue, or API interface, the system can parse the order time (e.g., 2023-04-01 15:30:00) for each order. Based on the order time, the system determines which partition the order should belong to. In this example, since the partitions are based on dates, the system can use the date portion (2023-04-01) as the partition key. It's worth noting that the system checks if the partition already exists before writing the order data to it. If the target partition does not yet exist (for example, today is April 2, 2023, but the system has only processed data up to April 1, 2023), the system will create the new partition first. Once the partition exists, the system will write the order data to the corresponding partition.

[0022] For example, when a partition is created, it has a corresponding business time range (also known as partition time). Partition time refers to the actual time range of data occurrence covered by each partition. In other words, each partition can be used to store business time information data within the business time range.

[0023] For example, each partition can maintain a first modification time, which records the time when the data within the partition was last modified. When new data (i.e., incremental data) is written to the partition, the time when the new data was written to the partition can be compared with the current first modification time. If the time when the new data was written to the partition is later than the current first modification time, then the first modification time of that partition is updated to the time when the new data was written to the partition.

[0024] For example, each partition can be stored in a directory. The directory's name or metadata contains the partition's time information, identifying which time periods the partition contains. Specifically, each partition can correspond to a specific directory, whose name or structure reflects the partition's time information. For instance, a partition directory might be named data_20230401, indicating that the partition contains data with a business date of April 1, 2023. When new data (incremental data) is added to the partition directory, the directory's modification time (i.e., the first modification time information) changes. This modification time typically indicates the last time the directory content was modified.

[0025] For example, suppose an e-commerce system stores order data in partitions based on dates, with each partition containing all orders within a specific date. To track the latest modifications to the data within each partition, the system maintains a "first modification time" for each partition. The first modification time refers to the time when the data within the partition was last modified. It records the timestamp of the latest data within the partition. In this e-commerce system, there are currently the following partitions and their corresponding first modification times: Partition data_20230401: First modification time is 2023-04-01 23:59:59 (assuming the last order of the last day in this partition was recorded at this time); Partition data_20230402 (a newly created empty partition): First modification time is empty or set to the partition start time 2023-04-02 00:00:00. When the system receives new order data, its order placement time is 2023-04-01 23:59:59.5 (slightly later than the current partition's 2023-04-01 first modification time). The system parses the order's placement time and determines that the order belongs to the data_20230401 partition. Because the new order's placement time is later than the current first modification time, the system updates the first modification time information of the data_20230401 partition to 2023-04-01 23:59:59.5 and writes the order data to the data_20230401 partition. Furthermore, when the system receives another new order with a placement time of 2023-04-02 10:00:00, the system similarly parses the order's placement time and determines that this order belongs to the data_20230402 partition. It should be noted that because this is a new partition, its first modification time information may be empty or set to the partition's start time. However, regardless of the initial value of the first modification time information, the order placement time of the new order (2023-04-02 10:00:00) is later than it. Therefore, the system updates the first modification time information of the data_20230402 partition to the order placement time of the new order. In this way, the system can track the latest modifications to the data within each partition in real time, providing strong support for subsequent data processing and query optimization.

[0026] For example, in step S130, the data processing time window refers to a time range or period used for processing data. This time window can be fixed (e.g., 00:00 to 23:59 every day) or dynamic (adjusted according to business needs or system load). Setting a processing time window helps the system process data in an orderly and controllable manner, avoiding data processing chaos and conflicts.

[0027] For example, when the business time range corresponding to a partition exceeds the data processing time window, but the first modification time information corresponding to the partition is within the data processing time window, the partition can be identified as the first target partition containing late data. Since the first modification time directly reflects the latest update status of the data within the partition, especially the time when the most recent incremental data was written to the partition, if this time point is within the data processing time window, but the business time range covered by the partition exceeds this window, it can be inferred that at least one incremental data item failed to be processed in time due to some delay, becoming late data. Based on this logic, the partition can be identified as the first target partition containing late data.

[0028] It should be noted that for the determination of late data, if the update time (possibly the first modification time) of data in a partition is within the data processing time window, but the business time of this data is earlier than the start time of the current time window, then this data is considered late data. For example, if the business time of a certain data in a partition is July 25, 2024, and the update time is August 2, 2024, and the current incremental governance time range (i.e., the data processing time window) is the most recent 10 days, i.e., August 1, 2024 to August 5, 2024, since the business time of this data is outside the time window range, but its update time is within the time window, it can be considered late data.

[0029] The following detailed explanation of the incremental data processing method provided in the embodiments of this application will be given with specific examples.

[0030] Suppose that in an e-commerce platform, order data is stored in partitions based on business time (i.e., order creation time). A data processing window is set to periodically process and analyze this data. In this scenario, the data source could be the e-commerce platform's order system, which records detailed information for each order, including order ID, product information, order amount, and order creation time. At a certain time (e.g., 1 AM daily), incremental data begins to be received from the order system. This incremental data refers to order data newly generated since the last processing. Assume the following three incremental order data entries are received: Order ID: 12345, Product Name: A, Order Creation Time: 2023-04-01 15:00:00; Order ID: 12346, Product Name: B, Order Creation Time: 2023-04-02 10:30:00; Order ID: 12347, Product Name: C, Order Creation Time: 2023-04-03 08:15:00. The "Order Creation Time" for each order is its business time information. When storing daily data in a separate partition, the order's creation time can be used to determine which partition each order should be written to. Specifically, order ID 12345 was created on 2023-04-01, so it should be written to the "2023-04-01" partition; order ID 12346 was created on 2023-04-02, so it should be written to the "2023-04-02" partition; and order ID 12347 was created on 2023-04-03, so it should be written to the "2023-04-03" partition. In this example, the "First Modification Time" is the time when the order data was written to the partition. Because it is directly received incremental data and directly written to the corresponding partition, the First Modification Time is usually different from the order creation time, but is the same as (or very close to) the time when the data was written to the partition. However, in practical applications, if order data is modified after being written to the partition (e.g., order status update), the first modification time will be updated to the latest modification time. If the business time of data existing in the partition is earlier than the start time of the current data processing time window, but the first modification time of this data (i.e., the time when the data was written to the partition) is within the current data processing time window, then this data can be considered late data. If some order data is written to the partition some time after the order is created due to system delays or other reasons, and the creation time (i.e., business time) of these orders is earlier than the start time of the current data processing time window, but the time when the data is written to the partition is within the current data processing time window, then this can be considered late data, and the partition storing this order can be identified as the first target partition containing late data.

[0031] In this embodiment, by employing the incremental data processing method including steps S110 to S130 described above, incremental data and its corresponding business time information are first received from the data source. Then, based on these business times, the incremental data is accurately allocated to its respective partitions, and the first modification time information of each partition is updated synchronously. Each partition can be used to store data whose business time information falls within the business time range, ensuring that each partition only contains data whose business time falls within a specific range, thereby maintaining the temporality and logic of the data. Next, based on the data processing time window, combined with the business time range corresponding to each partition and its latest modification time information, the first target partition containing late data can be determined. That is, by comparing and analyzing this information, the first target partition containing late data whose business time has exceeded the current processing time window but has been delayed for various reasons can be accurately identified. This embodiment of the application partitions incremental data according to business time and combines the business time range, modification time information, and data processing time window of the partitions to identify and process late data that exceeds the time window range. It can not only accurately reflect the actual business situation of the data, but also flexibly deal with the limitations of the time window range, effectively avoid the problem that out-of-order data cannot be counted, thereby improving the accuracy and timeliness of data processing.

[0032] For example, before receiving incremental data from the data source, the system can first obtain the second modification time information corresponding to the original data stored in the data source. This second modification time information indicates the time the original data was updated in the data source. Then, the original data whose second modification time information is after the last received time is identified as incremental data. It is worth noting that the system can periodically or on demand query the data source to obtain the last update time (i.e., the second modification time information) of all original data in the data source.

[0033] For example, when storing incremental data to (i.e., writing) the corresponding partition, if the incremental data and historical data in the partition have the same primary key value, the incremental data and historical data can be merged to ensure data integrity and accuracy. Specifically, before writing the incremental data to the partition, it is first checked whether historical data with the same primary key value already exists in the partition. If so, the incremental data and historical data are merged. The merging strategy may vary depending on the data type and business logic, but typically includes the following scenarios: Update: Overwrite the old value in the historical data with the new value in the incremental data. Accumulation: If the data is numeric and represents some kind of accumulative quantity (such as balance, count, etc.), the value in the incremental data is added to the historical data. Retain History: Retain the historical data while storing the incremental data as a new version or new record in the partition (which may require additional version control fields).

[0034] For example, after merging data, it is necessary to ensure that the relevant metadata and information of the partition are updated, such as the first modification time information (if the merge operation causes the last modification time of the data in the partition to change).

[0035] For example, after identifying the first target partition with late data, this partition can be marked as pending data entry. This marking not only intuitively reflects the current processing needs of these partitions, but also provides clear guidance and preparation for subsequent data entry work, ensuring the continuity and efficiency of the data processing workflow.

[0036] For example, the incremental data processing flow also includes a series of supplementary data entry operations for late data, including: obtaining supplementary data entry task information and supplementary data entry time range information input by the user; determining the second target partition based on the supplementary data entry time range information; determining the supplementary data entry task based on the supplementary data entry task information; and executing the supplementary data entry task based on the second target partition. Specifically, the system first receives the supplementary data entry task information and corresponding supplementary data entry time range information input by the user. The supplementary data entry task information describes the specific task the user wishes to perform, such as data cleaning, format conversion, and data merging, aiming to accurately process late data. The supplementary data entry time range information is a time period explicitly specified by the user, indicating the data interval that needs to be focused on. This interval may deviate from the business time range of the first target partition. Based on the supplementary data entry time range information provided by the user, the system performs in-depth filtering and analysis to accurately identify the specific partitions that do indeed require supplementary data entry tasks, i.e., the second target partition. After identifying the second target partition, the system constructs corresponding data processing logic and rules based on the user-defined supplementary data entry task information, generating an instance. Subsequently, by running instances, these tasks will be executed in an orderly manner on the second target partition to ensure that late data is fully and accurately supplemented.

[0037] For example, during the data entry task, to ensure data consistency and security, the system automatically places the second target partition in a data-locked state. This measure effectively prevents concurrent modifications to the data by other processes, ensuring the smooth progress of the data entry task. Once all data entry tasks are successfully completed, the system will promptly restore the partition status from locked to normal, facilitating subsequent data processing or query operations.

[0038] See Figure 2 , Figure 2This is a schematic diagram of the incremental data processing flow of a data system provided in one embodiment of this application. The system integrates a complete chain from efficient data integration from data sources, refined data governance, and accurate indicator calculation, comprehensively ensuring the efficiency and accuracy of data processing. Specifically, the data integration module, as the starting point of the entire data processing flow, is responsible for efficiently integrating data from various data sources. This process consists of two key steps—incremental acquisition and incremental writing to the Operational Data Store (ODS) layer. Incremental acquisition ensures that only newly generated or changed data is captured, reducing data redundancy and improving processing efficiency. Subsequently, this data is incrementally written to the ODS layer, laying a solid foundation for subsequent data governance and indicator calculation. After the data enters the ODS layer, the data governance module performs refined data governance, including data cleaning, transformation, and verification, to ensure the accuracy, integrity, and consistency of the data. In particular, for incremental data, this module implements an incremental governance strategy, processing only newly added or changed data and writing it to the Data Warehouse Detail (DWD) layer. This step not only optimizes data processing performance but also ensures the real-time nature and accuracy of the data in the data warehouse. After data at the DWD layer has undergone governance, the data metrics calculation module further processes it into valuable information. This module can extract data from the DWD layer, generate various business metrics, and write this incrementally governed data into the Data Warehouse Summary (DWS) layer. Data at the DWS layer can be presented in a summarized and aggregated form. In particular, the system has a built-in advanced task scheduling and visualization monitoring mechanism, which can not only flexibly arrange the execution sequence of data processing tasks, but also monitor the status of late data in real time and drive the data supplementation tasks throughout the entire process, thereby optimizing the data processing workflow.

[0039] It's worth noting that the layered structure of ODS, DWD, and DWS allows for the processing and management of data at different levels. Each layer undertakes specific data processing tasks, ranging from capturing raw data to cleaning and transforming it, and finally summarizing and generating reports. Specifically, for the ODS layer, its data sources are typically external databases, log files, API interfaces, etc. Its functions include capturing and storing raw, unprocessed data. The processing flow includes: receiving incremental data and its business time information (such as transaction timestamps) from the external data source; writing the data to the corresponding partitions in the ODS based on the business time information, where each partition stores data within a specific time period; and updating the first modification time information of that partition, indicating the time when the partition was last modified. For the DWD layer, its data source is the ODS layer. Its functions include cleaning, transforming, and normalizing the data from the ODS layer to ensure data consistency and accuracy. The processing flow includes: receiving incremental data and corresponding business time information from the ODS layer; writing the cleaned and transformed data to the corresponding partitions in the DWD layer based on the business time information; and similarly updating the first modification time information of that partition. For the DWS layer, its data source is the DWD layer. Its functions include summarizing and aggregating data based on the DWD layer to generate data for analysis and reporting. The processing flow includes: receiving incremental data and business time information from the DWD layer; writing the summarized data to the corresponding partition in the DWS layer according to the business time information; and updating the first modification time information of the partition. In each layer of ODS, DWD, and DWS, to effectively address the issue of late data, steps S110 to S130 in the above embodiments can be executed. That is, the execution entity of the incremental data processing method provided in this application embodiment can be any one of the ODS, DWD, or DWS layers. For the technical solution of this embodiment and its resulting technical effects, please refer to the relevant descriptions and explanations of steps S110 and S130 in the above embodiments; they will not be repeated here.

[0040] For example, such as Figure 3As shown, the incremental data processing flow is divided into three core stages: data integration, data processing, and indicator calculation. In the data integration stage, the system can use the modification time in external data source records for incremental filtering, accurately capturing data records updated since the last collection, and partitioning each record according to its business time, efficiently writing it to the ODS table. During this process, the system also automatically merges records with the same primary key, ensuring that the data processing stage can directly access the latest and most accurate business data. Entering the data processing stage, the system can retrieve data from the ODS table and execute incremental data processing tasks according to the set data processing time window. This mechanism not only effectively solves the problem of late data but also avoids the heavy burden of full data processing (referring to the periodic processing of all historical data at each stage of the entire data processing process, such as collecting all data once a day and then processing and calculating all indicators), thus improving processing efficiency. In the data indicator calculation stage, the system retrieves data from the DWD table and performs incremental updates according to task scheduling and indicator granularity requirements, accurately calculating various indicators within the specified business time range. It should be noted that task scheduling can determine when to execute which data processing tasks. During the data metric calculation phase, the system needs to be able to schedule corresponding calculation tasks according to a preset schedule or triggering conditions (such as data source updates). Metric granularity refers to the level of detail in which data is divided and calculated. Different business scenarios may require data metrics with different granularities. For example, some scenarios may only require partitioning at a daily granularity, while others may require partitioning at an hourly or even minute granularity. It is worth noting that in some cases, the system may have difficulty autonomously identifying late data and its impact on metrics. Therefore, this embodiment introduces a late data identification module. This module compares and analyzes the time range of data integration and synchronization with the time range of data processing to determine and identify late data that exceeds the data processing time window. Furthermore, combined with monitoring and supplementary recording functions, the system can provide users with real-time feedback on late data and its specific time range, empowering users to specify a time range for data supplementation based on actual conditions, thereby correcting metric accuracy or adjusting time window settings to better cover common late data situations. In addition, this embodiment also introduces a data business time range management method, which unifies the business logic of daily incremental calculation and supplementary recording calculation in data processing and data metric calculation. For example, in the data indicator calculation stage, users can input the time range and calculation method according to their actual needs, and the system can automatically return the precise time range of the data to be read, further improving the flexibility and accuracy of data processing.

[0041] For example, such as Figure 4As shown, incremental data acquisition can filter data based on the change timestamps of records in the source table (obtainable from the data source), ensuring that only records that have changed since the last acquisition are captured each time, effectively avoiding duplicate acquisition and significantly saving resources. This incremental data is then seamlessly integrated with historical data to form the latest ODS dataset. The ODS data can be stored using business time as the partitioning standard; this strategy not only helps improve data access speed but also optimizes the overall efficiency of data governance. During incremental acquisition, the system can dynamically adjust the two variables, `bizStarttime` and `bizEndtime`, based on the scheduling time of the acquisition task. In this scenario, `bizStarttime` and `bizEndtime` are variables used to specify the data time range that the current processing task should consider. For example, during the first execution, `bizStarttime` can be set to January 1, 1970, 00:00:00 (timestamp start), while `bizEndtime` is set to the maximum value to cover all possible historical data. In subsequent schedulings, `bizStarttime` is automatically updated to the end time of the previous acquisition scheduling, ensuring the accuracy of incremental acquisition. During the incremental governance phase, the DWD layer's incremental governance strategy aims to avoid unnecessary processing of the entire dataset. By introducing a data processing time window, it performs refined governance only on incremental data within a specified time range. This strategy not only improves the targeting of data processing but also significantly reduces the consumption of computing resources. During the governance process, the system can dynamically adjust the `bizStarttime` and `bizEndtime` variables based on a pre-configured time window range (e.g., data from the last 10 days). Upon initial execution, to establish a data baseline, `bizStarttime` can also be set to January 1, 1970, 00:00:00, and a full calculation will be performed. Subsequent governance operations strictly adhere to the time window constraints, processing only newly added or updated data. During the incremental metric calculation phase, the system can automatically calculate the data time range using the built-in `CycleTimeTruncS` function, ensuring the consistency of business logic and the accuracy of calculations. The parameters of the `CycleTimeTruncS` function include whether to supplement data, whether to initially submit data, business time value, statistical period, metric statistical granularity, whether to use business start time, and date format, as shown below: CycleTimeTruncS('$isRepairRangeInstance$','$isCommitInstance$',$businessEndDate$ / $businessStartDate$,'$Year$', 'Date',false, 'yyyy-MM-dd').

[0042] For example, by passing the corresponding parameters through normal scheduling and supplementary scheduling, the data time range required for indicator calculation can be returned. For instance, for a statistical period from "last year to this year" (Year#-1 -> Year), taking March 15, 2024 as an example, the CycleTimeTruncS function can accurately calculate: The start time is "2023-01-01", which is obtained by calling CycleTimeTruncS("false", "false", "2024-03-15 00:00:00.000", "$Year#-1$", 'Date', true, "yyyy-MM-dd"), marking the start point of the statistical period.

[0043] End time: "2024-12-31". This function usually automatically calculates the last day of the year based on $Year$, ensuring the completeness of the statistical range.

[0044] For the statistical period of "the most recent year" (Year#-1Date−>Year$$Date$), taking March 15, 2024 as an example, the CycleTimeTruncS function calculates: The start time is "2023-03-15", which is obtained by CycleTimeTruncS("false", "false", "2024-03-15 00:00:00.000", "$Year#-1$$Date$", 'Date', true, "yyyy-MM-dd"), accurate to the same day of last year.

[0045] End time: "2024-03-15", determined by CycleTimeTruncS("false", "false", "2024-03-1500:00:00.000", "$Year$$Date$", 'Date', false, "yyyy-MM-dd"), indicating that the end point of the statistical period is also the same day of this year.

[0046] It is worth noting that when the system is first executed, in order to establish a comprehensive data baseline, bizStarttime is set to the beginning of the timestamp—January 1, 1970, 00:00:00. At this time, the system will perform a full calculation to ensure the integrity and accuracy of the data.

[0047] See Figure 5 , Figure 5This is a flowchart illustrating the late data identification process provided in one embodiment of this application. This process is an automated data processing and monitoring process in a big data processing environment, particularly based on the Hadoop Distributed File System (HDFS) and big data analytics engines (such as Spark and Hive). This process aims to ensure the timeliness and accuracy of data by automatically detecting and identifying data partitions that fail to arrive or update within the expected timeframe, so that appropriate data supplementation or processing can be performed. Figure 5 In the process, incremental data from the ODS layer is collected in real-time or periodically and updated to HDFS to ensure timely updates of the data source. After successful data updates to HDFS, a scheduling hook automatically triggers the system's built-in partition and update time post-processing tasks. It's important to note that the scheduling hook is an automated mechanism used to trigger subsequent tasks when a specific event (such as data update completion) occurs. The post-processing task traverses all partition directories under a specific path on HDFS, collecting path information for each partition and the directory's update time. This information is written to a late data analysis raw data table to prepare for subsequent data analysis. In the late data analysis task, periodic (e.g., daily) update time analysis can be performed on all partitions of the raw data table. The core of the analysis is to determine if any partition's update time exceeds the predetermined incremental processing business time range. If so, these partitions are marked as "late data." Specifically, the analysis results can be displayed on a specific supplementary data monitoring page, listing all tables marked as "data to be supplemented" and their partitions. This display method allows data managers to intuitively see which data partitions have late arrival issues, and thus take appropriate measures.

[0048] See Figure 6 , Figure 6This is a data supplementation flowchart provided in one embodiment of this application. The flowchart illustrates the complete process of data supplementation for a specific data table and time range. The following is a detailed analysis of the flowchart: Users can first specify the supplementation task information and the supplementation time range information, i.e., the specific table to be supplemented, the time range, and the task chain range. Based on the parameters specified by the user, the system can trigger the entire supplementation execution process through a lineage scheduling mechanism. It should be noted that lineage scheduling is a scheduling strategy based on data dependencies (i.e., data lineage), which ensures that the supplementation task is executed in the correct order and according to dependencies. During this process, users can choose to execute some task nodes or all task nodes in the entire chain. This flexibility allows the supplementation operation to be customized according to actual needs, ranging from repairing specific stages to a comprehensive data update. After the supplementation task is submitted, the relevant data will first enter the "submitted" intermediate state. This indicates that the supplementation request has been received by the system and is ready for execution. In the initial stage of data supplementation, the collected data can usually undergo a comprehensive data cleaning process to identify and correct obvious errors, missing values, outliers, and other problems. After initial cleaning, as data analysis deepens and business needs become clearer, more detailed and in-depth data cleaning can be performed. This includes data validation for specific business rules and verification of complex logical relationships. Once the data supplementation task is successfully completed, the data status of the relevant table partitions will change to "normal," indicating that the data has been effectively supplemented and corrected. It's important to note that the data supplementation process follows a unified SQL logic for all tasks, including data collection, governance, and metric calculation. This means the system can automatically replace the `bizStarttime` and `bizEndtime` parameters in the SQL query based on the user-specified data supplementation time range to ensure the query covers the correct data time range. Specifically, during data collection task supplementation, the system replaces `bizStarttime` and `bizEndtime` in the filtering SQL based on the data supplementation time range. During governance task supplementation, the system replaces `bizStarttime` and `bizEndtime` in the execution SQL based on the data supplementation time range. When calculating and supplementing the metric, taking the aforementioned statistical period from last year to this year ($Year#-1$ -> $Year$) and the daily granularity metric task of March 15, 2024 as an example, if the time for issuing the data supplementation task is from March 10, 2024 to March 13, 2024, and the statistical granularity is daily, the time calculated by calling the CycleTimeTruncS function is as follows: Start time: CycleTimeTruncS("true", "false","2024-03-10 00:00:00.000", "$Year#-1$", 'Date', true, "yyyy-MM-dd")="2024-03-10"; End time: CycleTimeTruncS("true", "false","2024-03-13 00:00:00.000", "$Year$", 'Date', false, "yyyy-MM-dd")="2024-03-13".

[0049] It is worth noting that the SQL logic for supplementary data entry remains consistent with that of normal scheduling operations. This helps reduce errors and data inconsistencies caused by logical differences. It also facilitates understanding and operation for system maintenance and management personnel.

[0050] See Figure 7 , Figure 7 This is a specific example of the late data monitoring user interface provided in this application. The left side of the interface is the late data monitoring page, displaying a list of tables in the system, which form the basis for data processing and analysis. Users can browse different tables to understand their status and related information. For each table, the system displays whether late data exists. Specifically, if late data is detected, the system can mark it with a specific identifier (such as a red warning icon) to alert the user. In addition to monitoring late data, the system also displays whether late data has been supplemented, as well as the progress and results of the supplementation. It is worth noting that in some special cases, late data may be considered normal (e.g., delays due to changes in business rules). Users can change the status of these partitions from "pending supplementation" to "normal" through operations on the page (such as "manually ignore" or "cancel ignore"). The right side of the interface provides a function to view table lineage. Table lineage refers to the path of data flow in the system, including the data source, processing process, and final destination. By viewing the table lineage, users can better understand the data flow, facilitating troubleshooting and data processing. For tables with late data, users can specify the table (the current node) and the affected downstream nodes to perform a data replenishment operation. For example... Figure 8The supplementary data entry menu displayed shows basic information about the current node and downstream nodes, including the current supplementary data entry task information, the data entry task name, and the business date. During the scheduling and execution of the supplementary data entry task, the user can specify the time range (i.e., the business date) for the data to be supplemented. This time range should cover all late data to ensure data integrity and accuracy. The system will automatically schedule and replace the time parameters (such as bizStarttime and bizEndtime) in the incremental governance and calculation SQL according to the user-specified time range. Furthermore, the system will automatically trigger the scheduling and execution of supplementary data entry tasks across the entire data chain based on the task lineage (i.e., the data flow path). This means that once the data supplementation operation begins, the system will automatically process all affected downstream nodes to ensure data consistency and accuracy. The entire data supplementation process is highly automated, requiring no frequent user intervention. This significantly reduces the user's workload and improves data processing efficiency.

[0051] In addition, one embodiment of this application discloses an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the incremental data processing method as described in any of the preceding embodiments.

[0052] In addition, one embodiment of this application discloses a computer-readable storage medium storing computer-executable instructions for performing incremental data processing methods as described in any of the preceding embodiments.

[0053] Furthermore, one embodiment of this application discloses a computer program product, including a computer program or computer instructions, which are stored in a computer-readable storage medium. A processor of an electronic device reads the computer program or computer instructions from the computer-readable storage medium and executes the computer program or computer instructions, causing the electronic device to perform the incremental data processing method as described in any of the preceding embodiments.

[0054] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0055] The above is a detailed description of the preferred embodiments of this application. However, this application is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A method of processing incremental data, characterized by, The method comprises: receiving incremental data and service time information of the incremental data from a data source; writing the incremental data into a corresponding partition according to the service time information, and updating first modification time information of the partition, wherein the partition is used to store data of the service time information within a service time range, and the first modification time information is used to indicate a time of updating data of the partition; determining a first target partition with late data according to a data processing time window, and the service time range and the first modification time information corresponding to each partition.

2. The method of claim 1, wherein, Before the receiving incremental data from a data source, the method further comprises: obtaining second modification time information corresponding to original data stored in the data source, wherein the second modification time information is used to indicate a time of updating the original data in the data source; determining original data with a time after a last receiving time as the incremental data according to the second modification time information.

3. The method of claim 1, wherein, The writing the incremental data into a corresponding partition comprises: merging the incremental data and historical data in the partition with a same primary key value.

4. The method of claim 1, wherein, After the determining a first target partition with late data, the method further comprises: identifying the first target partition as a to-be-repaired state.

5. The method of claim 1, wherein, The method further comprises: obtaining a repair task information and a repair time range information input by a user; determining a second target partition according to the repair time range information; determining a repair task according to the repair task information; executing the repair task based on the second target partition.

6. The method of claim 5, wherein, The method further comprises at least one of: identifying the second target partition as a data locking state during execution of the repair task; identifying the second target partition as a normal state after the execution of the repair task ends.

7. The method of claim 1, wherein, The determining a first target partition with late data according to a data processing time window, and the service time range and the first modification time information corresponding to each partition comprises: when the service time range corresponding to the partition exceeds the data processing time window, and the first modification time information corresponding to the partition is within the data processing time window, determining the partition as the first target partition with late data. 8.An electronic device, comprising: one or more processors; a memory having one or more programs stored thereon, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method for processing incremental data according to any one of claims 1-7. 9.A computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method for processing incremental data according to any one of claims 1-7. 10.A computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the method for processing incremental data according to any one of claims 1-7.

Images