A batch task processing method based on distributed management

Abstract

The application discloses a kind of batch task processing methods based on distributed management, comprising: in response to the batch data processing task submitted by client, it is decomposed into multiple sub-transactions by distributed transaction manager;According to the transaction type of the sub-transaction, the transaction type is called corresponding processing flow, when the transaction type is update transaction, the sub-transaction is stored in partition, and snapshot and local transaction identifier are generated;According to the local transaction identifier, continuous judgment is carried out through consistency coordinator, and global snapshot is generated in combination with the snapshot.The application is applicable to large-scale data migration and enterprise digital employee scene, through the task decomposition of distributed transaction manager, the snapshot generation and LTID management of partition, the global snapshot calculation of consistency coordinator, the performance bottleneck in traditional distributed system, the consistency problem of cross-partition transaction is solved.

Description

Technical Field

[0001] This invention belongs to the field of batch task processing technology, specifically relating to a batch task processing method based on distributed management. Background Technology

[0002] With the rapid development of mobile internet, smart devices, and the Internet of Things (IoT) technologies, the volume of global data has increased dramatically. Enterprises' demand for massive data storage and analysis is constantly rising, requiring database systems not only to have high concurrency processing capabilities but also to maintain consistency during large-scale data operations. Especially in enterprise-level application scenarios, such as financial transactions, e-commerce, and data migration in government agencies, efficient and consistent data processing has become a critical requirement.

[0003] Traditional single-machine databases, due to their architectural limitations, are increasingly unable to meet the growing data processing demands. To address this challenge, distributed database systems have emerged, achieving scalability and high availability through multiple partitions and replicas. However, distributed systems bring new technical challenges, particularly in transaction management and concurrency control. How to improve system concurrency performance while ensuring data consistency has become a key research focus.

[0004] Two-phase locking is a classic concurrency control method suitable for single-machine database environments. However, under high contention loads, especially in long-running read-only transactions, the performance of two-phase locking is limited, which can easily lead to a decrease in system throughput.

[0005] Multi-version concurrency control (MVCC) can effectively reduce lock contention and improve concurrency performance, but it faces challenges when applied in distributed environments. Traditional MVCC can lead to read inconsistencies across partitions, especially at non-serializable isolation levels, where updates from transactions in different partitions may not be visible to other transactions, affecting data consistency.

[0006] In summary, under high-concurrency scenarios, traditional two-phase locking protocols suffer from severe lock contention, leading to a significant decrease in system throughput. Furthermore, multi-version concurrency control struggles to guarantee cross-partition consistency in a distributed environment. Therefore, how to achieve efficient decomposition and management of batch data processing tasks in a distributed environment, and how to ensure cross-partition transaction consistency, have become urgent problems to be solved in batch task processing. Summary of the Invention

[0007] This invention provides a batch task processing method based on distributed management, which is used to achieve efficient batch data processing task decomposition and management in a distributed environment, and to ensure the consistency of cross-partition transactions.

[0008] The technical solution adopted in this invention is as follows:

[0009] A batch task processing method based on distributed management includes:

[0010] In response to the batch data processing tasks submitted by the client, the data is broken down into multiple sub-transactions through a distributed transaction manager;

[0011] Based on the transaction type of the sub-transaction, the corresponding processing flow is invoked for the transaction type. When the transaction type is an update transaction, the sub-transaction is stored in the partition, and a snapshot and local transaction identifier are generated.

[0012] Based on the local transaction identifier, a global snapshot is generated by continuous judgment through the consistency coordinator and combined with the snapshot.

[0013] The batch task processing method based on distributed management disclosed in this invention also has the following additional technical features:

[0014] The sub-transaction is stored in a partition, and a snapshot and local transaction identifier are generated, specifically as follows:

[0015] When the sub-transaction is stored in a partition, a snapshot is generated;

[0016] Local transaction identifiers are generated by sequentially incrementing each partition independently.

[0017] A snapshot association is established through hash table mapping based on the snapshot and the local transaction identifier.

[0018] The transaction type of the sub-transaction is specifically as follows:

[0019] The sub-transactions include at least update transactions and read-only transactions;

[0020] When the transaction type of the sub-transaction is read-only, the corresponding local transaction identifier is determined through the global snapshot;

[0021] The partition is located using the local transaction identifier, and the stored batch data is read.

[0022] The partition is located using the local transaction identifier, specifically as follows:

[0023] Each partition is independent of the others. After locating the partition using the local transaction identifier, an unlock operation is performed on the partition so that the read-only transaction can read the stored batch data.

[0024] For the unlocated partition, it is in a locked state, and the batch data stored therein cannot be read.

[0025] Based on the local transaction identifier, continuous judgment is performed by the consistency coordinator, specifically as follows:

[0026] Sort separately by partition according to the local transaction identifier of each partition;

[0027] When the local transaction identifier of the partition is not continuous, it is determined that the corresponding partition does not meet the continuity requirement;

[0028] When the local transaction identifiers of the partitions are consecutive, it is determined that the corresponding partitions meet the continuity requirement.

[0029] When the corresponding partitions meet the continuity requirement, the maximum value is selected according to the sorting of the local transaction identifiers in each partition to generate the global snapshot.

[0030] The global snapshot is generated as follows:

[0031] The global snapshot is generated by updating the previous global snapshot based on the local transaction identifier corresponding to the new update transaction in each partition.

[0032] When the corresponding partition does not conform to continuity, the sub-transaction corresponding to the non-continuous local transaction identifier is located through a global snapshot;

[0033] The partitions can be made to meet continuity requirements by restarting subtransactions or by manually handling them through alerts.

[0034] The present invention also provides a storage medium,

[0035] The storage medium stores a computer program, which, when executed, implements the steps of the batch task processing method based on distributed management.

[0036] The present invention further provides a processing apparatus, comprising:

[0037] Memory, used to store computer programs;

[0038] A processor is used to implement the steps of the batch task processing method based on distributed management when executing the computer program.

[0039] Due to the adoption of the above technical solution, the beneficial effects achieved by this invention are as follows:

[0040] 1. In this invention, a Distributed Transaction Manager (DTM) decomposes the batch data processing tasks submitted by the client into multiple sub-transactions, and calls the corresponding processing flow according to the transaction type of the sub-transaction (update transaction or read-only transaction). Through task decomposition and parallel execution, the computing resources of the distributed system are fully utilized, significantly improving the efficiency of batch data processing and increasing system throughput. This reduces single-point performance bottlenecks, avoids the concurrency limitations of traditional single-machine databases, supports the high-concurrency processing requirements of large-scale data, and achieves efficient batch task decomposition and parallel processing.

[0041] For update transactions, sub-transactions are stored in partitions, snapshots and Local Transaction Identifiers (LTIDs) are generated, and snapshot associations are established through hash table mapping. Update transaction operations do not involve data item unlocking operations, reducing lock contention and avoiding the performance bottleneck of traditional two-phase locking (2PL) protocols in high-contention scenarios. Update transaction operations do not involve data item unlocking operations, meaning existing data cannot be modified, ensuring transaction isolation and avoiding "dirty reads" and "non-repeatable reads" caused by accidental data modification or deletion. For update transactions, system throughput is stable, outperforming the performance of traditional 2PL at the serializable level. Moreover, partitions operate independently, and failures do not affect data read / write operations in other partitions, enhancing fault tolerance.

[0042] The consistency coordinator performs continuity checks on the Local Transaction Identifiers (LTIDs) of each partition and generates a global snapshot version based on the snapshots. LTID continuity checks ensure snapshot consistency across partitions, avoiding inconsistencies in task updates at non-serializable isolation levels (such as asynchronous updates across partitions). When partition LTIDs are discontinuous, it facilitates timely handling of conflicts, ensuring normal system operation and preventing long wait times or deadlocks caused by discontinuous LTIDs.

[0043] In summary, this invention is applicable to large-scale data migration and enterprise digital employee scenarios. By decomposing tasks in a distributed transaction manager, generating snapshots of partitions and managing LTIDs, and calculating global snapshots in a consistency coordinator, it solves the performance bottlenecks and consistency problems of cross-partition transactions in traditional distributed systems. Attached Figure Description

[0044] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0045] Figure 1 This is a flowchart illustrating the batch task processing method based on distributed management according to one embodiment of the present invention. Detailed Implementation

[0046] To more clearly illustrate the overall concept of the present invention, a detailed description will be provided below with reference to the accompanying drawings and examples.

[0047] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0048] like Figure 1 As shown, a batch task processing method based on distributed management includes:

[0049] S100: In response to a batch data processing task submitted by a client, it is decomposed into multiple sub-transactions through a distributed transaction manager.

[0050] The core purpose of this step is to decompose the batch data processing tasks submitted by the client into multiple sub-transactions so as to achieve efficient parallel processing in a distributed system.

[0051] This step relies on a Distributed Transaction Manager (DTM) to receive batch data processing tasks submitted by clients. The DTM parses the task content and determines the data scope and operation type (e.g., update or read-only). It's understandable that this data type is categorized according to read and write operations on the database. Update transactions only involve write operations, including adding data and modifying existing data in the database. Read-only transactions only involve read operations, i.e., reading existing data from the database.

[0052] Because reading and writing operations on data in a database are fundamentally different, classifying transaction types facilitates the invocation of different processing flows in subsequent operations.

[0053] DTM decomposes a task into multiple sub-transactions, each corresponding to a data partition. For example, if a task involves 1 million records, DTM might split it into 100 sub-transactions, each processing 10,000 records. By decomposing the task, the system can execute multiple sub-transactions in parallel, effectively utilizing multi-node resources, alleviating performance bottlenecks, and significantly improving the throughput of batch data processing.

[0054] Furthermore, sub-transactions are dynamically allocated based on the current load of partitions using load balancing strategies to avoid hotspot partitions. For example, sub-transactions are preferentially assigned to idle or low-load partitions. DTM identifies the transaction type by parsing the operation type (update or read-only) of the sub-transaction, providing a basis for subsequent processing (such as snapshot generation).

[0055] After sub-transactions are decomposed, snapshots of each sub-transaction can be managed independently using Multi-Version Concurrency Control (MVCC). Sub-transactions involve only a single partition, reducing the frequency of global coordination, minimizing cross-partition dependencies, and lowering the complexity of global coordination. Furthermore, traditional two-phase locking protocols are prone to lock contention in long-running read-write transactions. By dividing sub-transactions by operation type (update or read-only), the granularity of task management is refined. Update transactions do not involve lock operations, thus reducing lock contention issues.

[0056] This step decomposes batch tasks into sub-transactions using a distributed transaction manager, solving the performance bottleneck problem of traditional single-machine databases. Through task decomposition, it provides efficient basic data support for subsequent steps such as snapshot management and global consistency coordination.

[0057] S200: Based on the transaction type of the sub-transaction, call the corresponding processing flow for the transaction type. When the transaction type is an update transaction, store the sub-transaction in the partition and generate a snapshot and a local transaction identifier.

[0058] The core objective of this step is to differentiate transaction types, partition and store update transactions, and generate snapshots and Local Transaction Identifiers (LTIDs).

[0059] The Distributed Transaction Manager (DTM) parses the operation type (update or read-only) of sub-transactions. Update transactions (such as INSERT, UPDATE, and DELETE) require the generation of a snapshot and LTID. Read-only transactions (such as SELECT) directly read data based on the global snapshot. INSERT, UPDATE, DELETE, and SELECT are standard SQL operations: INSERT adds data, UPDATE modifies existing data, DELETE deletes data, and SELECT queries data.

[0060] The update transaction is stored in the corresponding partition, and a snapshot is generated when the sub-transaction is committed. Snapshots are only generated upon sub-transaction commit, ensuring that modifications made by uncommitted transactions are not visible. It should be noted that the snapshot content includes the current value of the data item, the operation type, and the associated LTID. The generated LTID is used to return to the DTM and is used in the consistency coordinator for global snapshot calculation.

[0061] For read-only transactions, the snapshot mechanism allows for the rapid location of the target subtransaction, enabling locking operations to be performed only on the located subtransaction data. This further reduces lock contention issues in the traditional two-phase locking protocol (2PL) and improves concurrency performance.

[0062] Snapshot generation is triggered only upon commit, i.e., when the sub-transaction is stored in the partition. Modifications made by uncommitted transactions are not visible in the snapshot, avoiding dirty reads and non-repeatable reads. In addition, LTID provides the basis for subsequent global snapshot calculations to solve the read inconsistency problem of cross-partition transactions, and LTID, instead of full snapshot transmission, also reduces communication load.

[0063] This step resolves the conflict between performance and consistency in traditional concurrency control methods by differentiating transaction types, generating snapshots, and using LTID. It reduces lock contention through transaction type differentiation and snapshot mechanisms, ensures transaction isolation through partitioned storage, and lays the foundation for subsequent global snapshot calculations through LTID.

[0064] S300: Based on the local transaction identifier, the consistency coordinator makes continuous judgments and generates a global snapshot in conjunction with the snapshot.

[0065] The core objective of this step is to use the consistency coordinator to determine the continuity of the local transaction identifiers (LTIDs) of each partition and generate a globally consistent version by combining the snapshots.

[0066] After each subtransaction is committed, each partition sends its generated LTID to the consistency coordinator. The coordinator sorts the LTIDs by partition and determines whether the LTID sequences of each partition are continuous and without gaps. If a partition's LTID sequence contains gaps (e.g., S1: 0→1→3), it is considered discontinuous.

[0067] Traditional MVCC can lead to read inconsistencies in a distributed environment due to inconsistent transaction commit orders across partitions (e.g., a transaction in one partition is committed earlier than one in another). By using LTID continuity checks, the coordinator forces snapshot versions of each partition to align, ensuring that transactions see consistent data states and avoiding data anomalies caused by non-serializable isolation levels (e.g., inconsistent visibility across partition updates).

[0068] When all partitions have consecutive LTIDs, the largest LTID among the partitions is used to generate a global snapshot version. This global snapshot version is associated with the snapshot content of each partition (located by LTID) and is available for subsequent transactions to read.

[0069] It should be noted that the coordinator only interacts with the Distributed Transaction Manager (DTM) when an update transaction begins to commit, i.e., when it is stored in a partition, thus reducing the frequency of communication. Moreover, it only exchanges LTIDs, i.e., lightweight identifiers, significantly reducing the amount of data transmitted.

[0070] This step solves the read inconsistency problem in traditional distributed systems by using the continuity judgment of the consistency coordinator and the generation of global snapshots. Through LTID lightweight management and dynamic optimization strategies, it ensures the global consistency of cross-partition transactions while reducing coordination overhead.

[0071] In a preferred embodiment of the present invention, the sub-transaction is stored in a partition, and a snapshot and a local transaction identifier are generated, specifically as follows:

[0072] When the sub-transaction is stored in a partition, a snapshot is generated;

[0073] Local transaction identifiers are generated by sequentially incrementing each partition independently.

[0074] A snapshot association is established through hash table mapping based on the snapshot and the local transaction identifier.

[0075] The core objective of this implementation method is to achieve efficient data consistency assurance and concurrency control through partitioned storage, snapshot generation, and independent management of Local Transaction Identifiers (LTIDs).

[0076] Sub-transactions are stored in their corresponding partitions, and snapshots are generated when the sub-transactions are committed. This implementation emphasizes generating snapshots only upon sub-transaction commit, ensuring that modifications made by uncommitted transactions are not visible. It is understood that the snapshot content includes the current value of the data item, the operation type (e.g., UPDATE), and the associated LTID. For example, if an update transaction modifies the value of data item X to 20, snapshot X_v2 (LTID = 3) is generated upon commit.

[0077] In this implementation, each partition independently and sequentially generates LTIDs, ensuring the serializability of local transactions. Specifically, the LTID generation rule is that each partition maintains an incrementing counter, and a new LTID is assigned with each commit (e.g., LTID = 0, 1, 2, ...). The incrementing LTIDs provide the basis for subsequent global snapshot calculations. By judging the continuity of LTIDs, the coordinator can generate globally consistent snapshots (e.g., S1:3, S2:3, S3:3), resolving the read inconsistency problem of cross-partition transactions and ensuring the consistency of cross-partition transactions.

[0078] Furthermore, associating LTIDs with snapshots using hash tables (e.g., {3:X_v2}) facilitates rapid subsequent lookups. This LTID-snapshot mapping also enables quick location of sub-transactions, allowing locking operations to be performed only on the identified sub-transactions. This reduces the sudden drop in system throughput caused by lock contention and significantly improves performance.

[0079] This implementation solves the contradiction between performance and consistency in traditional concurrency control methods by using partitioned storage, snapshot generation, and LTID management. It satisfies consistency requirements by using incremental LTIDs and reduces lock contention by using a snapshot-LTID mapping management mechanism.

[0080] In a preferred embodiment of the present invention, the transaction type of the sub-transaction is specifically:

[0081] The sub-transactions include at least update transactions and read-only transactions;

[0082] When the transaction type of the sub-transaction is read-only, the corresponding local transaction identifier is determined through the global snapshot;

[0083] The partition is located using the local transaction identifier, and the stored batch data is read.

[0084] The core objective of this implementation is to enable efficient reading of batch data from a distributed database by read-only transactions through the linkage of global snapshots and local transaction identifiers (LTIDs).

[0085] When a sub-transaction is a read-only transaction, the Distributed Transaction Manager (DTM) requests a global snapshot version from the consistency coordinator and obtains the LTID values ​​for each partition. The read-only transaction locates the corresponding partition based on the LTID and reads the snapshot data for that partition. LTID replaces the full snapshot transmission. As a lightweight identifier, LTID reduces network bandwidth consumption and communication overhead.

[0086] Each partition stores data and snapshots independently. The data version can be obtained simply by looking up the hash table ({LTID: snapshot content}) based on the LTID. After locating the partition, an unlock operation is performed on the partition to allow read-only transactions to access the data.

[0087] Once the partition is located, an unlock operation is performed on the located partition, reducing the number of lock operations, significantly reducing lock contention, reducing performance degradation caused by lock contention, improving system throughput in high contention scenarios, and optimizing resource utilization.

[0088] This implementation solves the contradiction between performance and consistency in traditional concurrency control methods by linking global snapshots with LTID. It reduces lock contention and improves the efficiency of read-only transaction execution by executing read-only transactions in partitioned locations.

[0089] In a preferred embodiment of this implementation, the partition is located using the local transaction identifier, specifically as follows:

[0090] Each partition is independent of the others. After locating the partition using the local transaction identifier, an unlock operation is performed on the partition so that the read-only transaction can read the stored batch data.

[0091] For the unlocated partition, it is in a locked state, and the batch data stored therein cannot be read.

[0092] The core objective of this embodiment is to enable efficient reading of batch data from a distributed database by read-only transactions through the partitioning and locking / unlocking mechanism of the Local Transaction Identifier (LTID).

[0093] Each partition stores data and snapshots independently, requiring only the lookup of the corresponding partition based on its LTID. Partitions store snapshots using a hash table ({LTID: snapshot content}), with the LTID serving as a unique identifier to locate the partition. Read-only transactions directly locate the relevant partitions (e.g., S1 and S2) based on the LTID value in the global snapshot (e.g., S1:3, S2:3). For example, if the global snapshot is S1:3, S2:3, then a read-only transaction only needs to access partitions S1 and S2, without scanning other partitions (e.g., S3).

[0094] An unlock operation is performed on the located partition, allowing read-only transactions to access the data. After locating the partition, the Distributed Transaction Manager (DTM) sends an unlock command to the partition, releasing the lock resources of that partition. After unlocking, read-only transactions can directly read the data in that partition without waiting for locks on other partitions to be released.

[0095] Unlocated partitions are kept locked, preventing read-only transactions from accessing their data. Unlocated partitions (such as S3) remain locked, and their data cannot be accessed by read-only transactions. Locked partitions contain data that is not the target of this read-only transaction. Allowing access not only reduces the efficiency of read-only transactions but may also pollute the read data by reading unnecessary data, leading to dirty reads.

[0096] Moreover, this embodiment only unlocks the partition in question, while keeping other partitions locked. By using a dynamic unlocking mechanism, the lock holding time is reduced, significantly reducing lock contention and improving system throughput.

[0097] In summary, this embodiment resolves the conflict between performance and consistency in traditional concurrency control methods through LTID partition location and locking / unlocking mechanisms. It reduces lock contention by dynamically unlocking and prevents data corruption during reads by locking unlocated partitions.

[0098] In a preferred embodiment of the present invention, continuous judgment is performed by a consistency coordinator based on the local transaction identifier, specifically as follows:

[0099] Sort separately by partition according to the local transaction identifier of each partition;

[0100] When the local transaction identifier of the partition is not continuous, it is determined that the corresponding partition does not meet the continuity requirement;

[0101] When the local transaction identifiers of the partitions are consecutive, it is determined that the corresponding partitions meet the continuity requirement.

[0102] The core objective of this implementation is to sort and determine the continuity of the Local Transaction Identifiers (LTIDs) of each partition through a consistency coordinator, thereby ensuring the generation of a globally consistent snapshot version and resolving the inconsistency problem in cross-partition transaction execution in a distributed system.

[0103] After receiving the LTIDs submitted by each partition, the consistency coordinator sorts the LTIDs individually by partition. Each partition's LTID sequence is generated and maintained independently; for example, the LTID sequence for partition S1 is [0,1,2,3], and the LTID sequence for partition S2 is [0,1,3,4]. The coordinator merges the newly received LTIDs with historical records and sorts them in ascending order of value. For example, if a new LTID = 4 is added to S1, the sorting result will be [0,1,2,3,4].

[0104] For each partition, the LTID sequence is checked for continuity to determine if there are any gaps. If there are no gaps in the LTID sequence of a partition (e.g., [0,1,2,3]), it is considered continuous; if there are gaps (e.g., [0,1,3,4]), it is considered discontinuous.

[0105] All partitions have complete LTID sequences, thus meeting the continuity requirement. At least one partition has a missing LTID sequence, thus failing to meet the continuity requirement.

[0106] It is understandable that traditional MVCC may lead to read inconsistencies in a distributed environment due to inconsistent transaction commit orders across partitions (e.g., a transaction in one partition is committed earlier than that in another partition). This implementation uses LTID continuity judgment to force the snapshot versions of each partition to align, thus avoiding cross-partition data state inconsistencies.

[0107] As one embodiment of this implementation, when the corresponding partitions meet the continuity requirement, the maximum value is selected according to the sorting of the local transaction identifiers in each partition to generate the global snapshot.

[0108] The core objective of this embodiment is to determine that when the corresponding partitions meet the continuity requirement, a global snapshot version is generated by selecting the maximum value of the local transaction identifier (LTID) of each partition.

[0109] When the LTIDs of each partition are consecutive, the coordinator selects the maximum LTID value of each partition and combines them to generate a global snapshot version. The maximum value in the LTID sequence of each partition (e.g., S1:3, S2:3, S3:3) is taken to form global snapshot versions S1:3, S2:3, S3:3.

[0110] The global snapshot is generated as follows:

[0111] The global snapshot is generated by updating the previous global snapshot based on the local transaction identifier corresponding to the new update transaction in each partition.

[0112] The new round of snapshot calculation uses the old snapshot version as the initial value to reduce the computational overhead of repeatedly traversing historical LTIDs. The coordinator uses the global snapshot version calculated in the previous round (such as S1:2, S2:2, S3:2) as the starting point for the new round of judgment and only processes newly added LTIDs.

[0113] In distributed systems, frequently traversing historical LTIDs increases computational complexity. This embodiment selects the maximum value to ensure that the global snapshot version is the latest state of all partitions, avoiding read inconsistencies caused by inconsistent transaction commit orders across partitions. Moreover, selecting the maximum value instead of traversing all historical LTIDs significantly reduces computational complexity. Furthermore, using the global snapshot version calculated in the previous round (e.g., S1:2, S2:2, S3:2) as the starting point for the new round of judgments, and only processing newly added LTIDs, further reduces computational complexity.

[0114] This embodiment generates a global snapshot version by selecting the maximum value of the LTID of each partition, which solves the read inconsistency problem in traditional distributed systems. Through lightweight sorting and dynamic optimization strategies, it ensures the global consistency of cross-partition transactions while reducing coordination overhead.

[0115] As another embodiment of this implementation, when the corresponding partition does not conform to continuity, the subtask corresponding to the discontinuous local transaction identifier is located through a global snapshot;

[0116] The partitions are made to maintain continuity by restarting subtasks or by manually handling alerts.

[0117] The core objective of this embodiment is to locate the subtasks corresponding to discontinuous Local Transaction Identifiers (LTIDs) through global snapshots, and restore the continuity of the partitions through subtask restart or manual alarm mechanisms.

[0118] For discontinuous subtasks identified, a restart operation is performed or a manual alert is triggered. The coordinator sends a restart command to the Distributed Transaction Manager (DTM) to re-execute the uncommitted subtasks. If a subtask was not committed due to network latency or a temporary failure, LTID continuity can be restored after restarting.

[0119] By locating and repairing discontinuous LTIDs, the snapshot versions of each partition are forced to align, avoiding data anomalies caused by inconsistent transaction commit orders across partitions. Furthermore, in high-contention scenarios, some partitions may experience prolonged periods of unclosed LTIDs due to network latency or temporary failures. By restarting subtasks, the system can quickly restore continuity; the automatic restart mechanism rapidly recovers from LTID discontinuities caused by temporary failures, reducing system blocking time.

[0120] If a subtask fails to restart due to a logical error or resource conflict (such as a data conflict), the coordinator sends an alert to the operations personnel. After analyzing the logs or manually correcting the problem, the operations personnel resubmit the subtask. After the subtask restarts or is manually resolved, the coordinator reverifies the LTID continuity of the partition.

[0121] Automatic restarts may not resolve LTID discontinuities caused by logical errors or resource conflicts. Manual intervention allows operations personnel to precisely fix the problem. Manual alerting mechanisms handle complex issues (such as data conflicts), preventing loop failures caused by automatic restarts and avoiding global deadlocks due to long-term LTID discontinuities.

[0122] In one specific embodiment, transactions T1, T2, and T3 are included, along with partitions S1, S2, and S3. Each transaction is divided into sub-transactions involving all three partitions. Because all partitions are multi-threaded databases, a set of transactions may be committed in different orders on each partition, meaning they may generate LTID values ​​in different orders, as shown in the table below.

[0123] Initially, each partition has an initial LTID value of 0, meaning they are in a globally consistent state at the start. When a transaction commits, LTID values ​​are generated for each partition. The consistency coordinator receives the LTID values ​​in the order of T1, T2, and T3. Upon committing transaction T1, the consistency coordinator uses the LTID values ​​to determine if a consistent version can be achieved. The LTID sequence on S1 is (0,1), which is consecutive. However, the LTID sequence on S2 is (0,2), which is discontinuous. This means that a consistent version cannot be obtained at this point.

[0124] Continue committing transaction T2. ​​At this stage, the LTID sequence on S1 is (0,1,2), which is continuous. However, the LTID sequence on S2 is (0,2,3), which is still discontinuous. This also means that a consistent version cannot be obtained at this time.

[0125] Continue committing transaction T3. When the LTID value of T3 arrives, the LTID sequence on S1, S2, and S3 all becomes (0, 1, 2, 3). A globally consistent version is now available. Using the largest LTID value, a globally consistent version (S1:3, S2:3, S3:3) is formed. When the next set of LTID values ​​arrives, the coordinator uses (S1:3, S2:3, S3:3) as the initial LTID value to determine continuity.

[0126]

[0127] It should be noted that the numbering of each sub-transaction within each partition is based on the order in which the sub-transactions were stored in each partition. For partition S1, the storage order of the sub-transactions is T1, T2, T3; for partition S2, the storage order is T3, T1, T2; and for partition S3, the storage order is T2, T3, T1. However, the commit order of each transaction is T1, T2, T3. This discrepancy between the commit order of the individual transactions and the storage order of their corresponding sub-transactions within the partitions leads to a discontinuity issue.

[0128] The present invention also provides a storage medium,

[0129] The storage medium stores a computer program, which, when executed, implements the steps of a batch task processing method based on distributed management.

[0130] Therefore, any effect that can be achieved by batch task processing methods based on distributed management will not be elaborated here.

[0131] The present invention further provides a processing apparatus, comprising:

[0132] Memory, used to store computer programs;

[0133] A processor, used to implement a batch task processing method based on distributed management when executing the computer program.

[0134] Therefore, any effect that can be achieved by batch task processing methods based on distributed management will not be elaborated here.

[0135] For any parts not mentioned in this invention, existing technologies can be used or referenced.

[0136] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0137] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A batch task processing method based on distributed management, characterized in that, include: In response to the batch data processing tasks submitted by the client, the data is broken down into multiple sub-transactions through a distributed transaction manager; Based on the transaction type of the sub-transaction, the corresponding processing flow is invoked for the transaction type. When the transaction type is an update transaction, the sub-transaction is stored in the partition, and a snapshot and local transaction identifier are generated. Based on the local transaction identifier, a global snapshot is generated by continuous judgment through the consistency coordinator and combined with the snapshot. Based on the local transaction identifier, continuous judgment is performed by the consistency coordinator, specifically as follows: Sort separately by partition according to the local transaction identifier of each partition; When the local transaction identifier of the partition is not continuous, it is determined that the corresponding partition does not meet the continuity requirement; When the local transaction identifiers of the partitions are consecutive, it is determined that the corresponding partitions meet the continuity requirement; When the corresponding partitions meet the continuity requirement, the maximum value is selected according to the sorting of the local transaction identifiers in each partition to generate the global snapshot; The global snapshot is generated as follows: The global snapshot is generated by updating the previous global snapshot based on the local transaction identifier corresponding to the new update transaction in each partition. When the corresponding partition does not conform to continuity, the sub-transaction corresponding to the non-continuous local transaction identifier is located through a global snapshot; The partitions can be made to meet continuity requirements by restarting subtransactions or by manually handling them through alerts.

2. The batch task processing method based on distributed management according to claim 1, characterized in that, The sub-transaction is stored in a partition, and a snapshot and local transaction identifier are generated, specifically as follows: When the sub-transaction is stored in a partition, a snapshot is generated; Local transaction identifiers are generated by sequentially incrementing each partition independently. A snapshot association is established through hash table mapping based on the snapshot and the local transaction identifier.

3. The batch task processing method based on distributed management according to claim 1, characterized in that, The transaction type of the sub-transaction is specifically as follows: The sub-transactions include at least update transactions and read-only transactions; When the transaction type of the sub-transaction is read-only, the corresponding local transaction identifier is determined through the global snapshot; The partition is located using the local transaction identifier, and the stored batch data is read.

4. The batch task processing method based on distributed management according to claim 3, characterized in that, The partition is located using the local transaction identifier, specifically as follows: Each partition is independent of the others. After locating the partition using the local transaction identifier, an unlock operation is performed on the partition so that the read-only transaction can read the stored batch data. For the unlocated partition, it is in a locked state, and the batch data stored therein cannot be read.

5. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed, implements the steps of the batch task processing method based on distributed management as described in any one of claims 1 to 4.

6. A processing apparatus, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the batch task processing method based on distributed management as described in any one of claims 1 to 4 when executing the computer program.

Images