Big data batch processing method and system

Abstract

The application discloses a kind of big data batch processing method and system, and the application relates to big data batch processing technical field.Jobs information acquisition module, subgraph splitting module, backup configuration and management module, subgraph execution module, aggregation module, fault monitoring and switching module are included.By dividing subgraph according to the regional attribution of main data set and greedily combining and reducing cutting edge transmission, each subgraph independently calculates using local data as far as possible, significantly reduces the cross-regional network overhead and job completion time;Combined with incremental log real-time synchronization and periodic snapshot, build off-site hot backup, when fault, backup region quickly recovers data and takes over execution, cooperate with the double fault discrimination of heartbeat loss rate and duration, realize high availability and low recovery delay;After fault recovery, residual log is automatically returned, recalculation is corrected and compared, to ensure that global final result is accurate and consistent;Backup region is preferably selected according to available capacity and network delay ratio to ensure replacement performance.

Description

Technical Field

[0001] This invention relates to the field of big data batch processing technology, specifically to a big data batch processing method and system. Background Technology

[0002] In the field of big data batch processing, to extract valuable information from massive amounts of data, it is often necessary to perform complex calculations such as association and aggregation on multiple datasets distributed in different geographical locations. Existing batch processing architectures typically adopt a centralized computing model, either migrating all the scattered data to a single central node or scheduling computing tasks uniformly to the location of the remote data. However, regardless of the approach, large-scale data transfer across wide area networks is required, resulting in long data transmission times and high bandwidth consumption, which significantly extends the overall job completion time. Especially when the data volume far exceeds the computing logic itself, network latency becomes the core bottleneck of system performance.

[0003] To address these issues, some solutions have introduced the concept of data locality, attempting to bring computation closer to data and reduce transmission overhead by allowing computation to move while data remains stationary. However, in actual multi-region collaborative processing, coarse-grained task allocation often fails to fully consider the complex data dependencies within the job, resulting in persistently high amounts of intermediate data transmission across regions. The subgraph partitioning and task merging processes lack effective optimization, making it difficult to minimize the total communication cost of edge cutting.

[0004] Meanwhile, geographically distributed environments face higher risks of node or region-level failures. Traditional fault-tolerance strategies that rely on upstream data recalculation or simple checkpoint recovery suffer from excessively long recovery times in cross-regional scenarios. The integrity and timeliness of reconstructed data from different locations cannot be guaranteed, severely impacting the availability of batch processing jobs and the accuracy of the final results. Therefore, effectively reducing cross-regional data transmission volume and achieving efficient fault recovery has become a pressing technical challenge in this field. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a batch processing method and system for big data, solving the problems mentioned in the background section.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a batch processing method for big data, comprising the following steps:

[0007] Step S100: Obtain the job graph and data distribution: Obtain the directed acyclic graph of the batch processing jobs to be executed, as well as the input datasets and their storage location attributes on which each task in the directed acyclic graph depends.

[0008] For tasks that rely on multiple input datasets, the input dataset with the largest amount of data is determined as the main input dataset for the task, and the storage location attribute of the main input dataset is used as the location of the task.

[0009] Step S200: Greedy Subgraph Decomposition and Merging: Based on the geographical affiliation of each task, the directed acyclic graph is initially split into multiple initial subgraphs; a cross-regional subgraph merging operation is performed, which includes: repeatedly selecting a pair of subgraphs belonging to different regions from the initial subgraphs that can reduce the cross-subgraph data transmission volume after merging and merging them until a preset merging stop condition is met, resulting in multiple final subgraphs; each subgraph corresponds to a region and contains several tasks belonging to that region and complete directed edges connecting these tasks;

[0010] The splitting process includes repeatedly merging subgraph pairs within the same region that reduce the amount of cross-subgraph data transmission after merging, until they can no longer be merged. The task set of all subgraphs constitutes a partition of the original task set, and the directed edges connecting tasks belonging to different subgraphs in the original graph constitute a set of cutting directed edges.

[0011] Step S300: Configure primary and backup region synchronization: For each subgraph, determine its corresponding region as the primary execution region of the subgraph, and select another region as the backup region of the subgraph.

[0012] During the normal execution of the subgraph in the main execution region, the data write operations generated by the subgraph execution are sent to the backup region storage as incremental logs in real time, and the complete dataset snapshot of the subgraph is periodically transferred to the backup region as a base snapshot;

[0013] Step S400, Local Independent Calculation of Subgraphs: Each subgraph is executed independently in its own main execution region using local data to generate the intermediate results of the subgraph and its data volume base.

[0014] The data cardinality refers to the total number of records or the total amount of original input data that each subgraph relies on when calculating its intermediate results. It is used as the statistical weight of the intermediate results of the subgraph during global aggregation.

[0015] Step S500: Global aggregation of intermediate results: Collect the intermediate results of all subgraphs and their data volume cardinality, and aggregate all intermediate results into the global final result according to the preset aggregation logic;

[0016] Step S600, Fault Monitoring and Switching: During the execution of each subgraph in its respective main execution region, monitor the availability of the main execution region; when an unavailability fault is detected in any main execution region, switch the execution of the subgraph corresponding to that main execution region to the backup region designated for it;

[0017] In the backup region, a usable dataset is restored by loading the base snapshot and replaying all subsequent incremental logs, and the complete computational logic of the subgraph is re-executed based on the usable dataset to generate new intermediate results and a new data volume base.

[0018] Replace the original corresponding values ​​with the new intermediate results and the new data volume cardinality, and re-execute the aggregation logic to obtain the updated global final result.

[0019] Preferably, during the normal execution of the subgraph in the main execution region, the data write operations generated by the subgraph execution are sent to the backup region storage as incremental logs in real time. Specifically, the main execution region sends all data write operations generated during the subgraph execution process to the storage nodes of the backup region in real time and asynchronously in the form of ordered logs. The backup region continuously receives and persists the ordered logs, so that the data state held by the backup region consists of the basic snapshot and all incremental logs received thereafter.

[0020] Preferably, the cross-regional subgraph merging operation is performed, which specifically includes: initially treating each task as an independent temporary subgraph; in all current temporary subgraphs, obtaining the sum of the data transmission amounts of the cut directed edges that originally connected the two temporary subgraph pairs belonging to the same region after merging; selecting the same region temporary subgraph pair with the largest sum of data transmission amounts of the original cut directed edges for merging, and repeating this operation until there are no more same region temporary subgraph pairs that can be further merged, thus obtaining the final subgraph.

[0021] Preferably, after the main execution region recovers from the unavailability failure, the process further includes: suspending the subsequent execution of the subgraph in the backup region; obtaining residual incremental logs received by the backup region during the failure period but not synchronized by the main execution region; applying the residual incremental logs to the local data in the main execution region, and having the main execution region re-execute the complete calculation logic of the subgraph to produce a correction intermediate result and a correction data volume base; comparing the correction intermediate result with at least one historical global result generated using the backup region's intermediate result during the failure period; and triggering compensation correction when a discrepancy is found. Specifically, triggering compensation correction includes: recording each version of the global result generated by aggregating the intermediate results of the failed subgraph and its downstream consumption path; pushing the discrepancy value and the affected global result version information to the downstream business system, along with the latest global result containing the correction intermediate result, for the downstream business system to perform difference compensation or data rollback operations; replacing the output base of the subgraph in the aggregation calculation with the correction intermediate result, and switching the execution rights of the subgraph back from the backup region to the main execution region, restoring incremental log sending and snapshot transmission.

[0022] Preferably, the availability of the primary execution region is monitored, and any primary execution region is detected to be unavailable, including: the coordination node maintains periodic heartbeat message interaction with each primary execution region; the number of valid heartbeats received is counted within a preset sliding time window, and the heartbeat loss ratio is calculated based on the expected number of heartbeats and the actual number of valid heartbeats; when the heartbeat loss ratio reaches or exceeds a preset loss ratio threshold, and this state continues for a preset fault confirmation time, the primary execution region is determined to be unavailable.

[0023] Preferably, for tasks that rely on multiple input datasets, when there are multiple input datasets with the largest data volume, one of them is selected as the main input dataset according to a pre-set compliance priority.

[0024] Preferably, when selecting a backup region for each subgraph, for candidate backup regions, their priority score is calculated. The priority score is the available computing capacity of the candidate backup region divided by the sum of its network round-trip latency with the main execution region of the subgraph and a preset positive value. The candidate backup region with the highest priority score is selected as the backup region. The preset positive value is a very small positive number to prevent the denominator from being zero.

[0025] Preferably, each subgraph is executed independently in its own main execution region using local data. When generating intermediate results for the subgraph, the execution time of each task within the subgraph is estimated by adding the fixed startup cost of the task to the product of its unit data processing time and the data volume of all input datasets for the task. The overall execution time of the subgraph depends on the maximum value of the sum of the times of each task on the critical path from any source task to any sink task in the directed acyclic graph of the subgraph.

[0026] Preferably, the preset aggregation logic is determined according to the batch processing calculation type, including: when global summation is required, the global final result is equal to the sum of the intermediate results of all subgraphs; when weighted average is required, the global final result is equal to the sum of the intermediate results of each subgraph multiplied by their corresponding data volume base, divided by the sum of the data volume bases of all subgraphs.

[0027] A batch processing system for big data, the system comprising:

[0028] The job information acquisition module is used to acquire the directed acyclic graph of the batch processing jobs to be executed, as well as the input datasets and their storage location attributes on which each task constituting the directed acyclic graph depends.

[0029] For tasks that rely on multiple input datasets, the input dataset with the largest amount of data is determined as the main input dataset for the task, and the storage location attribute of the main input dataset is used as the location of the task.

[0030] The subgraph splitting module is used to split the directed acyclic graph into multiple subgraphs according to the regional affiliation of each task. Each subgraph corresponds to a region and contains several tasks belonging to that region and complete directed edges connecting these tasks.

[0031] The splitting process includes repeatedly merging subgraph pairs within the same region that reduce the amount of cross-subgraph data transmission after merging, until they can no longer be merged. The task set of all subgraphs constitutes a partition of the original task set, and the directed edges connecting tasks belonging to different subgraphs in the original graph constitute a set of cutting directed edges.

[0032] The backup configuration and management module is used to determine the region corresponding to each subgraph as the main execution region of the subgraph, and select another region as the backup region of the subgraph.

[0033] During the normal execution of the subgraph in the main execution region, the data write operations generated by the execution of the subgraph are sent to the backup region storage in real time as incremental logs, and the complete dataset snapshot of the subgraph is periodically transmitted to the backup region as a base snapshot;

[0034] The subgraph execution module is deployed in each main execution region and is used to independently execute the corresponding subgraph using local data, and generate the intermediate results of the subgraph and its data volume base.

[0035] The aggregation module is used to collect the intermediate results of all subgraphs and their data volume cardinality, and aggregate all intermediate results into the global final result according to the preset aggregation logic;

[0036] The fault monitoring and switching module is used to monitor the availability of the main execution region while each subgraph is executing in its respective main execution region;

[0037] When an unavailability failure is detected in any primary execution region, the execution of the subgraph corresponding to that primary execution region will be switched to the backup region designated for it.

[0038] In the backup region, a usable dataset is restored by loading the base snapshot and replaying all subsequent incremental logs, and the complete calculation logic of the subgraph is re-executed based on the usable dataset to generate new intermediate results and a new data volume base.

[0039] The original corresponding values ​​are replaced with the new intermediate results and the new data volume base, triggering the aggregation module to re-execute the aggregation logic and obtain the updated global final result.

[0040] This invention provides a batch processing method and system for big data. Compared with existing technologies, it has the following advantages:

[0041] This invention identifies the data locality of tasks, splits batch processing jobs into subgraphs based on geographical location, and each subgraph executes independently using local data, greatly reducing cross-regional data transfer, lowering network transmission overhead, and shortening job completion time. It is particularly suitable for big data scenarios with multiple geographical distributions.

[0042] A backup region is pre-established for each subgraph, and data status is saved in real time through incremental log synchronization and periodic snapshots. In the event of a failure in the primary region, the backup region can quickly restore a usable dataset and take over the computation, resulting in almost uninterrupted subgraph execution, significantly improved global job availability, short recovery time, and minimal impact on the overall schedule.

[0043] By repeatedly merging temporary subgraph pairs within the same region, and prioritizing the selection of pairs that minimize the amount of cross-subgraph data transmission, the total transmission amount of the final cut directed edges is minimized, further reducing cross-regional communication costs and improving overall execution efficiency.

[0044] After the faulty region is restored, the residual incremental logs generated during the fault are automatically transmitted back and the intermediate results are recalculated and corrected. The results are compared with the historical results of the backup region. If a discrepancy is found, compensation correction is triggered to ensure that the switchover process does not introduce inconsistencies in the final results and to maintain data accuracy.

[0045] A sliding window heartbeat detection mechanism is adopted to count the heartbeat loss rate and observe the duration, thus making dual judgments on faults. This avoids false alarms due to network jitter and ensures that real faults are quickly identified and triggered for switching, thereby improving system reliability.

[0046] Backup regions are selected based on a comprehensive evaluation of available computing capacity and network latency among candidate regions. Regions with ample capacity and low latency are given priority to ensure high computing performance during failover, optimize resource utilization, and shorten recovery computing time.

[0047] By using an estimation model that estimates the fixed startup overhead of tasks and the time consumed per unit of data processing, combined with critical path analysis of directed acyclic graphs, the execution time of each subgraph can be predicted, providing a basis for job scheduling and resource planning, and helping to further improve processing efficiency. Attached Figure Description

[0048] Figure 1 This is a system block diagram of a batch processing system for big data according to the present invention.

[0049] Figure 2 This is a flowchart illustrating a batch processing method for big data according to the present invention. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] Please see Figure 1 and Figure 2 As shown, the embodiments of the present invention provide the following technical solutions:

[0052] As an embodiment of the present invention:

[0053] This invention relates to a batch processing method for big data, comprising the following steps:

[0054] Step S100: Obtain the work diagram and data distribution:

[0055] Obtain the directed acyclic graph of the batch jobs to be executed;

[0056] A directed acyclic graph consists of a number of vertices and a number of directed edges;

[0057] Each vertex represents an indivisible computation task, and the total number of vertices is equal to the total number of tasks.

[0058] Each directed edge points from its source task to its target task, indicating a dependency where the output of the source task will be used as the input of the target task.

[0059] For each directed edge, obtain the amount of data transmission carried by the directed edge. The amount of data transmission is determined based on the historical execution data of the task connected to the directed edge or the compilation-time estimate.

[0060] Obtain information about the input datasets that all tasks depend on;

[0061] Each task processes one or more input datasets. Each input dataset has a storage region attribute. The value of the storage region attribute corresponds to a data center region participating in the batch processing. The total number of data center regions is a known value.

[0062] For any task, the main input dataset for that task is determined as follows:

[0063] Compare all input datasets for this task according to their respective data size, and select the input dataset with the largest data size as the main input dataset for this task.

[0064] When there are multiple input datasets with the largest data volume, one of them is selected as the main input dataset according to the pre-agreed compliance priority.

[0065] Among them, compliance priority is a sorted list that is pre-defined based on factors such as the regulatory sensitivity of the dataset, network bandwidth cost, or historical access speed.

[0066] After determining the main input dataset for the task, the storage location attribute of the main input dataset is used as the location of the task.

[0067] Step S200: Greedy decomposition and subgraph merging:

[0068] To minimize the total amount of data transfer across subgraphs while satisfying data locality constraints, the following subgraph merging operation is adopted:

[0069] Based on the geographical affiliation of each task, the directed acyclic graph is initialized into multiple temporary subgraphs: vertices corresponding to all tasks belonging to the same geographical region are grouped into the same temporary subgraph, with each temporary subgraph corresponding to one geographical region. The following greedy merging process is then performed on all the resulting temporary subgraphs:

[0070] Sub-step S201: In all the current temporary subgraphs, only consider temporary subgraph pairs belonging to the same region, calculate the sum of the data transmission amount of the cut directed edges that originally connected the two temporary subgraph pairs after merging, and select the pair of temporary subgraphs with the largest sum value for merging. The merged subgraphs still correspond to the same region and do not violate the data locality constraint.

[0071] Sub-step S202: Repeat sub-step S201 until there are no more temporary subgraph pairs within the same region that can be further merged;

[0072] Each subgraph obtained after stopping merging is taken as the final subgraph. The task set of all subgraphs constitutes a partition of the original task set. Each task belongs to one and only one subgraph. The directed edges connecting vertices belonging to different subgraphs in the original graph constitute the set of cutting directed edges. The total amount of data transmission across subgraphs is the sum of the data transmission amounts of each cutting directed edge in the set of cutting directed edges.

[0073] Step S300: Configure primary and backup regional synchronization.

[0074] For each subgraph obtained in step S200, the region corresponding to it is determined as the main execution region of that subgraph;

[0075] At the same time, each subgraph is assigned a backup region different from its main execution region. The backup region is selected from other regions with data storage capacity and computing resources.

[0076] When selecting a backup region, calculate its priority score for each candidate backup region:

[0077] Priority score = available computing capacity of candidate backup region ÷ (network round-trip latency from primary execution region to candidate backup region + preset positive value).

[0078] Select the candidate backup region with the highest priority score as the backup region for this subgraph.

[0079] The default positive value is a very small positive number to prevent the denominator from being zero;

[0080] During normal execution in the main execution region of the subgraph, incremental log synchronization is performed:

[0081] Incremental logs are sequential records of each data write operation, written in chronological order. Each record must include at least the location of the modified data, the new content after modification, and the operation type.

[0082] To ensure the accuracy of calculations after failover, the primary execution region must confirm that all incremental log entries corresponding to its subgraphs have been successfully persisted in the backup region before sending the intermediate results of its subgraphs to the aggregation module. This confirmation mechanism can be implemented through the backup region's acknowledgment of the persisted log sequence numbers.

[0083] The primary execution region sends all data write operations generated during the execution of the subgraph to the storage nodes of the backup region in real time and asynchronously in the form of an ordered log. The backup region continuously receives and persists the ordered log.

[0084] Data write operations refer to operations that change the state of data, such as adding, modifying, or deleting data.

[0085] At the same time, the complete snapshot of the dataset corresponding to the subgraph is transferred from the main execution region to the backup region at a preset period and stored as the base version, so that the backup region retains the combination of the base snapshot and the incremental logs that arrive after the base snapshot.

[0086] A base snapshot is a full copy of the entire dataset involved in a subgraph at a given moment; the base snapshot reflects the data state at that moment.

[0087] Step S400: Local independent calculation of the subgraph:

[0088] The coordinating node distributes the task list of each subgraph to the cluster of the corresponding main execution region. Subgraph execution only uses the data stored locally in the main execution region.

[0089] Within a subgraph, tasks are scheduled and executed according to the topological order of the directed acyclic graph.

[0090] The execution time of a task is estimated in the following way:

[0091] Total task time = Fixed startup overhead + Unit data processing time × Data volume of all input datasets for this task;

[0092] In the formula, fixed startup overhead refers to the average time required for the task to initialize the runtime environment; unit data processing time refers to the average time required to process a single piece or single byte of input data.

[0093] Among them, the fixed startup overhead and unit data processing time are empirical parameters determined in advance through statistical analysis based on the execution logs of historical batch processing jobs, and can be continuously updated as the job is executed.

[0094] The total execution time of a subgraph is the maximum sum of the total time of all tasks on all paths from any source task to any sink task in the directed acyclic graph of that subgraph, which is the critical path length.

[0095] After all tasks within a subgraph are completed, the subgraph outputs local intermediate results and simultaneously calculates the cardinality of the data volume processed by the subgraph.

[0096] Intermediate results refer to the portion of the dataset produced by any subgraph after it has completed all computations within its main execution region, but which has not yet been finally aggregated with the results of other subgraphs.

[0097] The data cardinality refers to the total number of records or the total amount of original input data that each subgraph relies on when calculating its intermediate results. It is used as the statistical weight of the intermediate results of the subgraph during global aggregation.

[0098] The data volume cardinality is determined through the transmission and verification of the data flow input and output relationships between tasks within the subgraph;

[0099] The intermediate results of the subgraph, along with its data volume, are reported to the coordinating node.

[0100] Step S500, Global Aggregation Intermediate Results:

[0101] The coordinating node receives the intermediate results of all subgraphs and the corresponding data volume cardinality, and performs aggregation according to predefined decomposable aggregation logic;

[0102] The decomposable aggregation logic refers to aggregation functions that can directly calculate the global final result from the local aggregation results (intermediate results and data cardinality) of each subgraph, including at least summation, counting, merging weighted averages, and merging variance component calculations. Aggregation logic that cannot be decomposed, such as global sorting and median extraction, or global deduplication, is not applicable to the batch processing calculation types described in this method.

[0103] The aggregation logic is determined based on the batch calculation type and includes at least:

[0104] When the aggregation type is global summation, the global final result is equal to the sum of the intermediate results of all subgraphs;

[0105] When the aggregation type is weighted average, the global final result is equal to the sum of the products of the intermediate results of each subgraph and their corresponding data volume cardinality, divided by the sum of the data volume cardinality of all subgraphs;

[0106] The aggregation process takes place only within the coordinating node.

[0107] Step S600, Fault Monitoring and Switching:

[0108] During job execution, the coordination node maintains periodic heartbeat message exchanges with each primary execution region to monitor the availability of the primary execution regions; the detection process is as follows:

[0109] Sub-step S601: The coordinating node maintains a sliding window with a fixed time length for each primary execution region, and the expected number of heartbeats within the sliding window is a preset value;

[0110] Sub-step S602: At any given time, count the number of valid heartbeats actually sent and successfully received by the coordinating node within the sliding window for the main execution region, and calculate the heartbeat loss rate;

[0111] Heartbeat loss rate = (expected heartbeats - effective heartbeats) ÷ expected heartbeats;

[0112] Sub-step S603: When the heartbeat loss ratio is greater than or equal to the preset loss ratio threshold, and the continuous duration of this state reaches the preset fault confirmation time, the coordinating node determines that the main execution region has an unavailable fault and marks its health status as unavailable.

[0113] For the subgraph corresponding to the main execution region that is determined to be unavailable, the coordinating node performs a failover operation:

[0114] The coordinating node switches the execution target of the subgraph from its primary execution region to the backup region specified for the subgraph in step S300, and sends a re-execution instruction to the backup region. This instruction contains the metadata of the subgraph and the identifier of the dependent dataset.

[0115] After receiving the instruction, the backup region uses the basic snapshot and incremental logs of the subgraph stored locally to restore the data;

[0116] The data recovery method is as follows: First, load the most recent basic snapshot to obtain the basic dataset. Then, replay all incremental logs received after the basic snapshot in log order and apply the incremental changes to the basic dataset one by one to form a usable recovered dataset.

[0117] Because the incremental logs of the primary execution region may not be fully synchronized to the backup region during failover, the state of the recovered usable dataset may be slightly earlier than the instantaneous state when the primary execution region failed.

[0118] The backup region re-executes the complete computation logic of the subgraph based on the restored available dataset, generates new intermediate results and a new data volume base, and sends the new intermediate results and the new data volume base back to the coordinating node;

[0119] The coordinating node replaces the original intermediate result of the subgraph with the new intermediate result, replaces the original data volume base of the subgraph with the new data volume base, and re-executes the aggregation logic according to step S500 to obtain the updated global final result.

[0120] By splitting data locally and using a greedy merging algorithm, cross-regional data transmission is significantly reduced, and each subgraph is computed locally, which significantly reduces network latency and job time. By combining real-time synchronization of incremental logs and periodic snapshots, off-site hot backup is built. In case of failure, the system quickly switches over based on the dual criteria of heartbeat, restores data from the backup region and takes over execution, ensuring high availability of global results and short recovery time, thus improving the overall efficiency and reliability of distributed batch processing.

[0121] As a second embodiment of the present invention:

[0122] In specific implementation, compared with Embodiment 1, the only difference between the technical solution of this embodiment and Embodiment 1 is that this embodiment also includes step S700 and post-fault result correction;

[0123] Once the original execution region returns to normal operation, perform the following correction steps:

[0124] Sub-step S701: The coordinating node suspends the subsequent execution of this subgraph in the backup region and records the latest intermediate result version in the backup region;

[0125] Sub-step S702: The original primary execution region pulls the residual incremental logs that the backup region continuously received during the failure but which the original primary execution region failed to synchronize, from the backup region, and applies the residual incremental logs to the local data of the original primary execution region so that the local data reaches the latest state consistent with the time of the failure.

[0126] Sub-step S703: The original main execution region re-executes the complete calculation logic of the subgraph based on the updated local data, producing intermediate correction results and the base amount of correction data;

[0127] Sub-step S704: The coordinating node compares the corrected intermediate results with the historical global results produced using the backup region during the failure, and triggers compensation correction for the downstream consumed results when a deviation is determined.

[0128] A specific embodiment of the triggering compensation correction is as follows: the coordination node records each global result version generated by the aggregation of intermediate results of the fault subgraph and its downstream consumption path; when the comparison finds that there is a deviation between the correction intermediate result and the corresponding part in a certain historical global result version, the coordination node automatically pushes the deviation value and the affected global result version information to the downstream business system, and attaches the latest global result containing the correction intermediate result, so that the downstream business system can perform difference compensation or data refresh operation.

[0129] Sub-step S705: The coordinating node replaces the output baseline of the subgraph in the aggregation calculation with the correction intermediate result, switches the execution right of the subgraph back from the backup region to the original main execution region, and resumes the incremental log synchronization and periodic snapshot backup process of step S300.

[0130] By adding post-fault recovery correction, after the original primary region recovers, residual incremental logs are automatically retrieved, intermediate results are recalculated and corrected, and compared with historical global results. If a deviation occurs, compensation correction is triggered, and execution control is then switched back. This mechanism compensates for inconsistencies that may be introduced by fault switching, ensuring the accuracy and consistency of the final global result and strengthening the system's fault tolerance and self-healing capabilities.

[0131] As an embodiment of the present invention:

[0132] In its specific implementation, compared to Embodiment 1 and Embodiment 2, the technical solution of this embodiment combines the solutions of Embodiment 1 and Embodiment 2. The only difference between this embodiment and Embodiment 1 and Embodiment 2 is that this embodiment also provides a big data batch processing system, which is used to execute a big data batch processing method. The system includes:

[0133] The job information acquisition module is used to acquire the directed acyclic graph of the batch processing jobs to be executed, as well as the input datasets and their storage location attributes that each task depends on to form the directed acyclic graph.

[0134] For tasks that rely on multiple input datasets, the input dataset with the largest amount of data is determined as the main input dataset for the task, and the storage location attribute of the main input dataset is used as the location of the task.

[0135] The subgraph splitting module is used to split a directed acyclic graph into multiple subgraphs based on the geographical affiliation of each task. Each subgraph corresponds to a geographical region and contains several tasks belonging to that region as well as complete directed edges connecting these tasks.

[0136] The splitting process includes repeatedly merging subgraph pairs within the same region that reduce the amount of cross-subgraph data transmission after merging, until they can no longer be merged. The task set of all subgraphs constitutes a partition of the original task set, and the directed edges connecting tasks belonging to different subgraphs in the original graph constitute a set of cutting directed edges.

[0137] The backup configuration and management module is used to determine the corresponding region as the main execution region for each subgraph and select another region as the backup region for that subgraph.

[0138] During the normal execution of the subgraph in the main execution region, the data write operations generated by the subgraph execution are sent to the backup region storage as incremental logs in real time, and the complete dataset snapshot of the subgraph is periodically transferred to the backup region as a base snapshot;

[0139] The subgraph execution module is deployed in each main execution region and is used to independently execute the corresponding subgraph using local data, and generate the intermediate results of the subgraph and its data volume base.

[0140] The cardinality of the data is transmitted and verified in the following way: an initial cardinality weight of 1 is attached to the original input data record of each subgraph; when each task is executed in the directed acyclic graph of the subgraph, the propagation rules of the cardinality weight are defined according to the operator type of the task.

[0141] The propagation rules include at least the following: for mapping operators, the cardinality weight of the output record is inherited from the cardinality weight of its input record; for filtering operators, the cardinality weight of the output record is equal to the cardinality weight of its input record, and the cardinality weight of the filtered record is not propagated downstream; for aggregation operators, the cardinality weight of the aggregated value of the aggregation result is the sum of the cardinality weights of all aggregated records, and the cardinality weights of other output records of the aggregation result remain unchanged.

[0142] The cardinality of the data corresponding to the intermediate result of the subgraph is equal to the sum of the cardinality weights of the original input data records consumed in generating the intermediate result.

[0143] The aggregation module is used to collect the intermediate results of all subgraphs and their data volume cardinality, and aggregate all intermediate results into the global final result according to the preset aggregation logic;

[0144] The fault monitoring and switching module is used to monitor the availability of the main execution region while each subgraph is executing in its respective main execution region;

[0145] When an unavailability failure is detected in any primary execution region, the execution of the subgraph corresponding to that primary execution region will be switched to the backup region designated for it.

[0146] In the backup region, a usable dataset is restored by loading the base snapshot and replaying all subsequent incremental logs, and the complete computational logic of the subgraph is re-executed based on the usable dataset to generate new intermediate results and a new data volume base.

[0147] The original corresponding values ​​are replaced with the new intermediate results and the new data volume base, triggering the aggregation module to re-execute the aggregation logic and obtain the updated global final result.

[0148] Furthermore, any content not described in detail in this specification is existing technology known to those skilled in the art.

[0149] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0150] The above formulas are all dimensionless calculations. Dimensionless calculation involves introducing a reference benchmark, such as the maximum, minimum, standard deviation, or theoretical extreme value of a physical quantity, to transform the original physical quantity into a dimensionless relative value. This value is usually mapped to a specific interval, such as [0,1] or [-1,1], which eliminates the influence of units while preserving the relative size relationship of the physical quantities. The formula is derived from software simulation based on a large amount of collected data to obtain the most recent real-world results. The preset parameters and thresholds in the formulas are set by those skilled in the art according to the actual situation.

[0151] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

[0152] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A batch processing method for big data, characterized in that, Includes the following steps: Obtain the directed acyclic graph of the batch processing jobs to be executed, as well as the input datasets and their storage location attributes that each task in the directed acyclic graph depends on; for tasks that depend on multiple input datasets, determine the input dataset with the largest data volume as the main input dataset of the task, and use the storage location attribute of the main input dataset as the location of the task. Based on the geographical affiliation of each task, the directed acyclic graph is initially split into multiple regional subgraphs. Subgraphs of the same region are merged to obtain multiple subgraphs. Each subgraph corresponds to a region and contains several tasks belonging to that region and complete directed edges connecting these tasks. For each subgraph, its corresponding region is determined as the primary execution region for that subgraph, and another region is selected as the backup region for that subgraph. During the normal execution of the subgraph in the main execution region, the data write operations generated by the subgraph execution are sent to the backup region storage as incremental logs in real time, and the complete dataset snapshot of the subgraph is periodically transferred to the backup region as a base snapshot; Each subgraph executes independently in its own main execution region using local data, generating the intermediate results of that subgraph and its data volume base. Collect the intermediate results and their data cardinality of all subgraphs, and aggregate all intermediate results into the global final result according to the preset aggregation logic; During the execution of each subgraph in its respective primary execution region, the availability of the primary execution region is monitored; when an unavailability failure is detected in any primary execution region, the execution of the subgraph corresponding to that primary execution region is switched to the backup region designated for it; In the backup region, a usable dataset is restored by loading the base snapshot and replaying all subsequent incremental logs. The complete calculation logic of the subgraph is then re-executed based on the usable dataset to generate new intermediate results and a new data volume cardinality. The original corresponding values ​​are replaced with the new intermediate results and the new data volume cardinality, and the aggregation logic is re-executed to obtain the updated global final result.

2. The batch processing method for big data according to claim 1, characterized in that, The process of merging subgraphs within the same region involves: initially treating each task as an independent temporary subgraph; among all the current temporary subgraphs, obtaining the sum of the data transmission amounts of the cut directed edges connecting the two temporary subgraph pairs belonging to the same region after merging; selecting the temporary subgraph pair within the same region with the largest sum of data transmission amounts of the original cut directed edges for merging, and repeating this operation to minimize the total data transmission amount across subgraphs until there are no more temporary subgraph pairs within the same region that can be merged, thus obtaining the final subgraph.

3. The batch processing method for big data according to claim 1, characterized in that, The data write operations generated during the execution of the subgraph are sent to the backup region storage in real time as incremental logs. Specifically, the main execution region sends all data write operations generated during the execution of the subgraph to the storage nodes of the backup region in real time and asynchronously in the form of ordered logs. The backup region continuously receives and persists ordered logs, so that the data state maintained by the backup region consists of the base snapshot and all incremental logs received thereafter.

4. The batch processing method for big data according to claim 3, characterized in that, After the primary execution region recovers from an unavailability failure and returns to normal operation, a post-failure correction step is also included, as follows: Suspend subsequent execution of the subgraph in the backup region; retrieve the residual incremental logs that the backup region received during the failure but failed to synchronize in the primary execution region; apply the residual incremental logs to the local data in the primary execution region, and have the primary execution region re-execute the complete computation logic of the subgraph, producing correction intermediate results and correction data volume base; compare the correction intermediate results with at least one historical global result generated using the intermediate results from the backup region during the failure, and trigger compensation correction when a deviation is determined; replace the output base of the subgraph in the aggregation calculation with the correction intermediate results, and switch the execution rights of the subgraph back from the backup region to the primary execution region, and resume incremental log sending and snapshot transmission.

5. The batch processing method for big data according to claim 1, characterized in that, The availability of the primary execution region is monitored, and any primary execution region is detected to be unavailable. This includes: the coordination node maintaining periodic heartbeat message exchanges with each primary execution region; counting the number of valid heartbeats received within a preset sliding time window, and calculating the heartbeat loss ratio based on the expected number of heartbeats and the actual number of valid heartbeats; when the heartbeat loss ratio reaches or exceeds a preset loss ratio threshold, and this state continues for a preset fault confirmation time, the primary execution region is determined to be unavailable.

6. The batch processing method for big data according to claim 1, characterized in that, For tasks that rely on multiple input datasets, when there are multiple input datasets with the largest data volume, one of them is selected as the main input dataset according to a pre-set compliance priority.

7. The batch processing method for big data according to claim 1, characterized in that, When selecting a backup region for each subgraph, calculate the priority score for each candidate backup region. The priority score is the sum of the available computing capacity of the candidate backup region and the network round-trip latency between it and the main execution region of the subgraph and a preset positive value. The candidate backup region with the highest priority score is selected as the backup region. The preset positive value is a very small positive number to prevent the denominator from being zero.

8. The batch processing method for big data according to claim 1, characterized in that, Each subgraph executes independently in its own main execution region using local data. When generating intermediate results for the subgraph, the execution time of each task within the subgraph is estimated by adding the fixed startup cost of the task to the product of its unit data processing time and the data volume of all input datasets for the task. The overall execution time of the subgraph depends on the maximum sum of the execution times of all tasks on the critical path from any source task to any sink task in the directed acyclic graph of the subgraph.

9. A batch processing method for big data according to claim 1, characterized in that, The preset aggregation logic is determined based on the batch processing calculation type: When performing a global summation, the final global result equals the sum of the intermediate results of all subgraphs; When using a weighted average, the global final result is equal to the sum of the intermediate results of each subgraph multiplied by the base value of its corresponding data, divided by the sum of the base values ​​of all subgraph data.

10. A batch processing system for big data, characterized in that, This system is used to execute a batch processing method for big data as described in any one of claims 1-9, the system comprising: Job Information Acquisition Module: Used to acquire the directed acyclic graph of batch processing jobs, the input dataset and storage region attributes of each task, and to determine the main input dataset and region affiliation for each task; Subgraph splitting module: Used to split a directed acyclic graph into multiple subgraphs based on geographical affiliation, and to merge subgraphs within the same region to reduce cross-subgraph data transmission. Backup configuration and management module: used to determine the primary execution region and backup region for each subgraph, and to implement incremental log synchronization and periodic snapshot backup during normal execution in the primary execution region; Subgraph execution module: Deployed in each main execution region, responsible for independently executing the corresponding subgraph using local data, generating intermediate results and data volume base; Aggregation module: Used to gather intermediate results and data volume of all subgraphs, and generate the global final result according to preset aggregation logic; Fault monitoring and switching module: Used to monitor the availability of the main execution region. When an unavailability fault is detected, the execution of the subgraph is switched to the backup region to restore data, re-execute the subgraph, and update the global results.

Images