Batch task processing method based on distributed management

Abstract

The invention discloses a distributed management-based batch task processing method, which comprises the following steps of: in response to a batch data processing task submitted by a client, decomposing the batch data processing task into a plurality of sub-transactions through a distributed transaction manager; according to the transaction type of the sub-transaction, calling a corresponding processing flow for the transaction type, and when the transaction type is an update transaction, storing the sub-transaction in a partition, and generating a snapshot and a local transaction identifier; and performing continuous judgment through a consistency coordinator according to the local transaction identifier, and generating a global snapshot in combination with the snapshot. The method is suitable for large-scale data migration and enterprise digital employee scenes, and the problems of performance bottleneck and cross-partition transaction consistency in a traditional distributed system are solved through task decomposition of the distributed transaction manager, snapshot generation and LTID management of partitions and global snapshot calculation of the consistency coordinator.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of batch task processing, and particularly relates to a batch task processing method based on distributed management. BACKGROUND

[0002] With the rapid development of mobile Internet, smart devices and Internet of Things technology, global data volume is growing rapidly. Enterprises' demand for mass data storage and analysis is rising, which not only requires database systems to have high concurrent processing capacity, but also needs to maintain consistency in large-scale data operations. Especially in enterprise-level application scenarios such as financial transactions, e-commerce and data migration of government agencies, efficient and consistent data processing has become a key requirement.

[0003] Traditional single-machine databases gradually cannot meet the growing data processing needs due to their architectural limitations. To address this challenge, distributed database systems have emerged, achieving scalability and high availability through multi-partitioning and multi-replication. However, distributed systems bring new technical challenges, especially in transaction management and concurrency control. How to improve the concurrent performance of the system while ensuring data consistency has become the focus of current research.

[0004] The two-phase locking protocol is a classic concurrency control method suitable for single-machine database environments. However, under high contention loads, especially in long read-only transactions, the performance of the two-phase locking protocol is limited, which can easily lead to a decrease in system throughput.

[0005] Multi-version concurrency control can effectively reduce lock contention and improve concurrency performance, but it faces challenges when applied in distributed environments. Traditional multi-version concurrency control can lead to read inconsistency problems across partitions for transactions, especially under non-serializable isolation levels, where transaction updates in different partitions may not be visible to other transactions, affecting data consistency.

[0006] In summary, under high concurrency scenarios, the lock contention of traditional two-phase locking protocols is severe, leading to a significant decrease in system throughput. At the same time, multi-version concurrency control has difficulty ensuring consistency across partitions in distributed environments. How to implement efficient batch data processing task decomposition and management in a distributed environment, and how to ensure the consistency of cross-partition transactions have become urgent problems in batch task processing. SUMMARY

[0007] The present application provides a batch task processing method based on distributed management, which is used to implement efficient batch data processing task decomposition and management in a distributed environment, and ensure the consistency of cross-partition transactions.

[0008] The technical solution adopted by the present application is as follows:

[0009] A batch task processing method based on distributed management, comprising:

[0010] In response to a batch data processing task submitted by a client, decomposing into multiple sub-transactions through a distributed transaction manager;

[0011] According to the transaction type of the sub-transaction, calling a corresponding processing flow according to the transaction type, when the transaction type is an update transaction, storing the sub-transaction in a partition, generating a snapshot and a local transaction identifier;

[0012] According to the local transaction identifier, continuously judging through a consistency coordinator, and generating a global snapshot in combination with the snapshot.

[0013] The batch task processing method based on distributed management disclosed in the present application also has the following additional technical features:

[0014] Storing the sub-transaction in a partition, generating a snapshot and a local transaction identifier, specifically:

[0015] When the sub-transaction is stored in a partition, a snapshot is generated;

[0016] Generating a local transaction identifier through independent sequential increment of each partition;

[0017] According to the snapshot and the local transaction identifier, mapping through a hash table to establish snapshot association.

[0018] The transaction type of the sub-transaction, specifically:

[0019] The transaction type of the sub-transaction at least includes an update transaction and a read-only transaction;

[0020] When the transaction type of the sub-transaction is a read-only transaction, determining a corresponding local transaction identifier through the global snapshot;

[0021] Positioning the partition through the local transaction identifier and reading the stored batch data.

[0022] Positioning the partition through the local transaction identifier, specifically:

[0023] Each partition is independent of each other, and after positioning the partition through the local transaction identifier, performing an unlock operation on the partition, so that the read-only transaction can read the stored batch data;

[0024] For the partition that is not positioned, it is in a locked state, and the stored batch data cannot be read.

[0025] According to the local transaction identifier, continuously judging through a consistency coordinator, specifically:

[0026] sequencing separately according to the local transaction identifiers of each partition;

[0027] when the local transaction identifiers of the partition are discontinuous, judging that the corresponding partition does not meet continuity;

[0028] when the local transaction identifiers of the partition are continuous, judging that the corresponding partition meets continuity.

[0029] when the corresponding partition meets continuity, selecting a maximum value according to the sequencing of the local transaction identifiers in each partition to generate the global snapshot.

[0030] generating the global snapshot, specifically:

[0031] the global snapshot is updated according to a previous global snapshot through the local transaction identifiers corresponding to the newly added update transactions in each partition to generate the global snapshot.

[0032] when the corresponding partition does not meet continuity, locating the sub-transaction corresponding to the discontinuous local transaction identifier through the global snapshot;

[0033] by performing sub-transaction restart or by manual processing through alarm to make the partition meet continuity.

[0034] The application further provides a storage medium,

[0035] the storage medium stores a computer program, and the computer program is executed to realize the steps of the batch task processing method based on distributed management.

[0036] The application further provides a processing device, comprising:

[0037] a memory for storing a computer program;

[0038] a processor for executing the computer program to realize the steps of the batch task processing method based on distributed management.

[0039] Due to the adoption of the above technical solutions, the application has the following beneficial effects:

[0040] 1. In the present application, the distributed transaction manager (DTM) decomposes the batch data processing task submitted by the client into multiple sub-transactions, and calls the corresponding processing flow according to the transaction type (update transaction or read-only transaction) of the sub-transaction. Through task decomposition and parallel execution, the computing resources of the distributed system are fully utilized, the batch data processing efficiency is significantly improved, and the system throughput is improved. It can reduce the single point performance bottleneck, avoid the concurrency limitation of traditional single machine database, support the high concurrency processing demand of large scale data, and realize efficient batch task decomposition and parallel processing.

[0041] For update transactions, the sub-transactions are stored in partitions, snapshots and local transaction identifiers (LTIDs) are generated, and snapshot association is established through hash table mapping. The update transaction operation does not involve data item unlocking operation, reduces lock competition, and avoids the performance bottleneck of traditional two-phase locking protocol (2PL) in high contention scenario. The update transaction operation does not involve data item unlocking operation, that is, it cannot modify the existing data, ensuring the isolation of the transaction and avoiding the problems of "dirty read" and "non-repeatable read" after data modification or deletion. For update transactions, the system throughput is stable, which is better than the performance of traditional 2PL at serializable level. Moreover, the partitions run independently, so that the fault does not affect the data read and write of other partitions, and the fault tolerance is enhanced.

[0042] The consistency coordinator judges the continuity of the local transaction identifier (LTID) of each partition, and generates a global snapshot version combined with the snapshot. Through LTID continuity detection, the snapshot consistency of cross-partition transactions is ensured, and the problem of inconsistent task update at non-serializable isolation level (such as asynchronous cross-partition update) is avoided. When the LTIDs of each partition are not continuous, it is convenient to handle the conflict problem in time, ensure the normal operation of the system, and prevent long waiting or deadlock problems caused by non-continuous LTIDs.

[0043] In summary, the present application is suitable for large-scale data migration and enterprise digital employee scenarios, and solves the performance bottleneck and consistency problem of cross-partition transactions in traditional distributed systems through task decomposition by the distributed transaction manager, snapshot generation and LTID management by the partition, and global snapshot calculation by the consistency coordinator. BRIEF DESCRIPTION OF DRAWINGS

[0044] The drawings described herein are used to provide further understanding of the present application, and form a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application, and do not constitute an improper limitation on the present application. In the drawings:

[0045] Figure 1 The figure is a flowchart of the batch task processing method based on distributed management according to an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to more clearly illustrate the overall concept of the present application, the following detailed description will be made with reference to the accompanying drawings.

[0047] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that the present application can be practiced without the specific details set forth in this description.

[0048] As shown in the figure, a batch task processing method based on distributed management includes: Figure 1

[0049] S100: In response to a batch data processing task submitted by a client, a distributed transaction manager is used to decompose the task into multiple sub-transactions.

[0050] The core purpose of this step is to decompose the batch data processing task submitted by the client into multiple sub-transactions, so as to realize efficient parallel processing in a distributed system.

[0051] The implementation of this step relies on a distributed transaction manager (DTM) that receives a batch data processing task submitted by a client. The DTM parses the task content and determines the data range and operation type (such as update or read-only) involved. It can be understood that the data type is classified according to read and write operations on data in the database. Update transactions only involve write operations, including adding new data and modifying existing data in the database. Read-only transactions only involve read operations, i.e., reading existing data in the database.

[0052] Because read and write operations on data in the database are essentially different, transaction type classification facilitates the calling of different processing flows in subsequent operations.

[0053] The DTM decomposes the task into multiple sub-transactions, each corresponding to a data partition (Partition). For example, if the task involves 1 million records, the DTM may split it into 100 sub-transactions, each handling 10,000 records. By decomposing the task, the system can execute multiple sub-transactions in parallel, effectively utilizing multi-node resources, alleviating performance bottlenecks, and significantly improving the throughput of batch data processing.

[0054] In addition, through a load balancing strategy, sub-transactions are dynamically allocated according to the current load of the partition to avoid hot partitions. For example, sub-transactions are preferentially allocated to idle or low-load partitions. The DTM marks the transaction type by parsing the operation type (update or read-only) of the sub-transaction, providing a basis for subsequent processing flows (such as snapshot generation).

[0055] ​After the sub-transactions are decomposed, the snapshots of each sub-transaction can be managed independently by multi-version concurrency control (MVCC), the sub-transactions only involve a single partition, the frequency of global coordination can be reduced, the dependency across partitions is reduced, and the complexity of global coordination is reduced. In addition, the traditional two-phase locking protocol is prone to lock competition in long read-write transactions, and by dividing the operation types (update or read-only) of the sub-transactions, the task management granularity is refined, and for update transactions, no lock operation is involved, thereby reducing the lock competition problem.

[0056] This step decomposes the batch task into sub-transactions by the distributed transaction manager, solves the performance bottleneck problem of the traditional single-machine database, and provides efficient basic data support for the subsequent snapshot management, global consistency coordination and other steps through task decomposition.

[0057] S200: According to the transaction type of the sub-transaction, a corresponding processing flow is called according to the transaction type, and when the transaction type is an update transaction, the sub-transaction is stored in a partition, a snapshot and a local transaction identifier are generated.

[0058] The core purpose of this step is to perform differential processing according to the transaction type, store the update transaction in a partition, and generate a snapshot and a local transaction identifier (LTID).

[0059] The distributed transaction manager (DTM) analyzes the operation type (update or read-only) of the sub-transaction. Update transactions (such as INSERT, UPDATE, and DELETE) need to generate a snapshot and an LTID. Read-only transactions (such as SELECT) directly read data based on a global snapshot. Among them, INSERT, UPDATE, DELETE, and SELECT are standard SQL operations, INSERT is used to add data, UPDATE modifies existing data, DELETE deletes data, and SELECT queries data.

[0060] The update transaction is stored in the corresponding partition, and a snapshot is generated when the sub-transaction is committed. The snapshot is only generated when the sub-transaction is committed, ensuring that the modifications of uncommitted transactions are not visible. It should be noted that the snapshot content includes the current value of the data item, the operation type, and the associated LTID. The generated LTID is used to return to the DTM and used in the global snapshot calculation in the consistency coordinator.

[0061] For read-only transactions, the snapshot mechanism allows quick positioning of the target sub-transaction to perform lock operations only on the positioned sub-transaction data, further reducing the lock competition problem of the traditional two-phase locking protocol (2PL) and improving concurrency performance.

[0062] The snapshot generation is triggered only when the sub-transactions are stored in the partitions, i.e., at the time of commit, and the modifications of the uncommitted transactions are invisible in the snapshot, avoiding dirty reads and non-repeatable reads. In addition, the LTID provides a basis for subsequent global snapshot calculation to solve the read inconsistency problem of cross-partition transactions, and the LTID replaces the transmission of complete snapshots, also reducing the communication load.

[0063] This step solves the contradiction between performance and consistency in the traditional concurrency control method by distinguishing transaction types, generating snapshots and LTIDs. The lock competition is reduced through transaction type distinction and snapshot mechanism, the transaction isolation is ensured through partition storage, and the foundation for subsequent global snapshot calculation is laid through LTID.

[0064] S300: According to the local transaction identifier, the consistency coordinator is continuously judged to generate a global snapshot in combination with the snapshot.

[0065] The core purpose of this step is to continuously judge the local transaction identifier (LTID) of each partition by the consistency coordinator, and generate a global consistent version in combination with the snapshot.

[0066] After the sub-transactions are submitted, the generated LTID is sent to the consistency coordinator. The coordinator sorts the LTID by partition, and judges whether the LTID sequence of each partition is continuous without gaps. If there is a gap in the LTID sequence of a partition (such as S1:0→1→3), it is determined to be discontinuous.

[0067] The traditional MVCC may cause read inconsistency in a distributed environment due to inconsistent transaction submission sequence between partitions (such as a transaction submitted earlier in one partition than in another partition). Through LTID continuity detection, the coordinator forces the snapshot version alignment of each partition, ensuring that the data state seen by the transaction is consistent, and avoiding data anomalies caused by non-serializable isolation levels (such as cross-partition update visibility inconsistency).

[0068] When the LTIDs of all partitions are continuous, the maximum LTID of each partition is taken to generate a global snapshot version. The global snapshot version is associated with the snapshot content of each partition (located by LTID) for subsequent transaction reading.

[0069] It should be noted that the coordinator only interacts with the distributed transaction manager (DTM) when the update transaction starts to commit, i.e., is stored in the partition, reducing the communication frequency. Moreover, only the LTID, i.e., the lightweight identifier, is interacted with, significantly reducing the amount of transmitted data.

[0070] This step solves the read inconsistency problem in the traditional distributed system through the continuity judgment and global snapshot generation of the consistency coordinator, ensures the global consistency of cross-partition transactions through the lightweight management and dynamic optimization strategy of LTID, and reduces the coordination overhead.

[0071] As a preferred embodiment of the present application, the sub-transaction is stored in a partition, a snapshot is generated, and a local transaction identifier is generated, specifically:

[0072] When the sub-transaction is stored in a partition, a snapshot is generated;

[0073] The local transaction identifier is generated by sequentially incrementing through each partition independently;

[0074] According to the snapshot and the local transaction identifier, a snapshot association is established through a hash table mapping.

[0075] The core purpose of the present embodiment is to achieve efficient data consistency guarantee and concurrency control through partition storage, snapshot generation, and independent management of local transaction identifiers (LTIDs).

[0076] The sub-transaction is stored in a corresponding partition, and a snapshot is generated when the sub-transaction is committed. The present embodiment emphasizes that a snapshot is only generated when a sub-transaction is committed, ensuring that modifications of uncommitted transactions are not visible. It can be understood that the snapshot content includes the current value of the data item, the operation type (such as UPDATE), and the associated LTID. For example, if the value of data item X is modified to 20 in an update transaction, a snapshot X_v2 (LTID=3) is generated when it is committed.

[0077] In the present embodiment, the LTID is generated by sequentially incrementing through each partition independently, ensuring the serializability of local transactions. Specifically, the LTID generation rule is that each partition maintains an incrementing counter, and a new LTID is assigned each time it is committed (such as LTID=0, 1, 2,...). The incrementing LTID provides a basis for subsequent global snapshot calculation, and through LTID continuity judgment, the coordinator can generate a globally consistent snapshot (such as S1:3, S2:3, S3:3), solving the read inconsistency problem of cross-partition transactions and ensuring the consistency of cross-partition transactions.

[0078] In addition, the LTID is associated with the snapshot through a hash table (such as {3:X_v2}), facilitating subsequent rapid lookup. The association mapping of the LTID and the snapshot facilitates rapid positioning of the sub-transaction, so that only the positioned sub-transaction is subjected to lock operation, reducing the problem of sudden drop in system throughput caused by lock competition and significantly improving performance.

[0079] The present embodiment solves the contradiction between performance and consistency in traditional concurrency control methods through partition storage, snapshot generation, and LTID management. The incrementing LTID meets the consistency requirement, and the snapshot and LTID mapping management mechanism reduces lock competition.

[0080] As a preferred embodiment of the present application, the transaction type of the sub-transaction is specifically:

[0081] The transaction type of the sub-transaction at least includes an update transaction and a read-only transaction;

[0082] When the transaction type of the sub-transaction is a read-only transaction, a corresponding local transaction identifier is determined through the global snapshot;

[0083] The partition is located through the local transaction identifier, and the stored batch data is read.

[0084] The core purpose of the embodiment is to realize efficient reading of batch data in a distributed database by the linkage of a global snapshot and a local transaction identifier (LTID) for a read-only transaction.

[0085] When the sub-transaction is a read-only transaction, a distributed transaction manager (DTM) requests a global snapshot version from a consistency coordinator to obtain an LTID value of each partition. The read-only transaction locates the corresponding partition according to the LTID and reads snapshot data of the partition. The LTID replaces complete snapshot transmission, and the LTID as a lightweight identifier can reduce network bandwidth consumption and communication overhead.

[0086] Each partition independently stores data and snapshots, and only needs to find a hash table ({LTID: snapshot content}) according to the LTID to obtain the data version. After locating the partition, an unlock operation is performed on the partition to allow the read-only transaction to access the data.

[0087] After locating the partition, an unlock operation is performed on the located partition, the number of lock operations is reduced, lock contention is significantly reduced, performance degradation caused by lock contention is reduced, system throughput in a high contention scenario is improved, and resource utilization is optimized.

[0088] The embodiment solves the contradiction between performance and consistency of a traditional concurrency control method by the linkage of a global snapshot and an LTID, reduces lock contention by locating a partition to execute a read-only transaction, and improves the execution efficiency of the read-only transaction.

[0089] As a preferred embodiment of the embodiment, the partition is located through the local transaction identifier, specifically:

[0090] The partitions are independent of each other, and after the partition is located through the local transaction identifier, an unlock operation is performed on the partition to enable the read-only transaction to read the stored batch data.

[0091] For the partition that is not located, the partition is in a locked state, and the stored batch data cannot be read.

[0092] The core purpose of the embodiment is to realize efficient reading of batch data in a distributed database by partition positioning and locking / unlocking mechanisms of a local transaction identifier (LTID) for a read-only transaction.

[0093] Each partition stores data and snapshots independently, and only needs to find the corresponding partition according to the LTID. The partition stores the snapshot through a hash table ({LTID: snapshot content}), and the LTID is used as a unique identifier to realize the positioning of the partition through the LTID. The read-only transaction directly locates the involved partitions (such as S1 and S2) according to the LTID value (such as S1:3, S2:3) in the global snapshot. For example, if the global snapshot is S1:3, S2:3, the read-only transaction only needs to access the S1 and S2 partitions, and does not need to scan other partitions (such as S3).

[0094] The unlocking operation is performed on the located partition to allow the read-only transaction to access the data. After locating the partition, the distributed transaction manager (DTM) sends an unlocking instruction to the partition to release the lock resource of the partition. After unlocking, the read-only transaction directly reads the data of the partition without waiting for the lock of other partitions to be released.

[0095] The unlocated partition is kept in a locked state to prohibit the read-only transaction from accessing the data thereof. The unlocated partition (such as S3) is still in a locked state, and the data thereof cannot be accessed by the read-only transaction. The locked partition contains read target data of the read-only transaction, and if the access is allowed, not only the efficiency of the execution of the read-only transaction is reduced, but also the read data can be polluted due to reading unnecessary data, which can cause a dirty read problem.

[0096] Moreover, the embodiment only unlocks the involved partition, and other partitions are kept in a locked state, the holding time of the lock is reduced through the dynamic unlocking mechanism, the lock competition is significantly reduced, and the system throughput is improved.

[0097] In summary, the embodiment solves the contradiction between performance and consistency of the traditional concurrency control method through the partition positioning of the LTID and the locking / unlocking mechanism, reduces the lock competition through the dynamic unlocking, and prevents the pollution of read data through the locking of the unlocated partition.

[0098] As a preferred embodiment of the application, according to the local transaction identifier, a continuous judgment is performed through a consistency coordinator, specifically:

[0099] According to the local transaction identifier of each partition, the partitions are individually sorted;

[0100] When the local transaction identifier of the partition is not continuous, it is judged that the corresponding partition does not meet the continuity;

[0101] When the local transaction identifier of the partition is continuous, it is judged that the corresponding partition meets the continuity.

[0102] The core purpose of the embodiment is to ensure the generation of a globally consistent snapshot version by ordering and continuity judgment of local transaction identifiers (LTIDs) of each partition through a consistency coordinator, and to solve the inconsistency problem of cross-partition transaction execution in a distributed system.

[0103] After the consistency coordinator receives the LTIDs submitted by each partition, it separately orders the LTIDs by partition. The LTID sequence of each partition is independently generated and maintained, for example, the LTID sequence of partition S1 is [0, 1, 2, 3], and the LTID sequence of partition S2 is [0, 1, 3, 4]. The coordinator sorts the newly received LTIDs in ascending order after merging them with the historical records. For example, if S1 adds a new LTID = 4, the sorted result is [0, 1, 2, 3, 4].

[0104] The continuity of the LTID sequence of each partition is detected to determine whether there is a gap. If there is no gap in the LTID sequence of the partition (such as [0, 1, 2, 3]), it is determined to be continuous; if there is a gap (such as [0, 1, 3, 4]), it is determined to be discontinuous.

[0105] If the LTID sequence of all partitions has no gap, it is determined to meet the continuity requirement. If the LTID sequence of at least one partition has a gap, it is determined not to meet the continuity requirement.

[0106] It can be understood that in a distributed environment, the traditional MVCC may cause read inconsistency due to inconsistent transaction submission order between partitions (such as a transaction in one partition being submitted earlier than a transaction in another partition). The embodiment forces the alignment of snapshot versions of each partition through LTID continuity judgment, avoiding the problem of inconsistent data state across partitions.

[0107] As an embodiment under the embodiment, when the corresponding partition meets the continuity, the maximum value of the ordering of the local transaction identifiers in each partition is selected to generate the global snapshot.

[0108] The core purpose of the embodiment is to determine that when the corresponding partition meets the continuity, the global snapshot version is generated by selecting the maximum value of the local transaction identifiers (LTIDs) of each partition.

[0109] When the LTIDs of each partition are continuous, the coordinator selects the maximum LTID value of each partition to generate a global snapshot version. The maximum value in the LTID sequence of each partition is selected (such as S1:3, S2:3, S3:3), forming the global snapshot version S1:3, S2:3, S3:3.

[0110] The global snapshot is generated, specifically:

[0111] The global snapshot is updated according to the last global snapshot by a local transaction identifier corresponding to an added update transaction in each of the partitions, to generate the global snapshot.

[0112] The new round of snapshot calculation takes the old snapshot version as the initial value, reducing the calculation overhead of repeatedly traversing the historical LTID. The coordinator takes the global snapshot version calculated in the last round (such as S1:2, S2:2, S3:2) as the starting point of the new round of judgment, and only processes the added LTID.

[0113] In a distributed system, frequent traversal of historical LTID increases the calculation complexity. The embodiment selects the maximum value to ensure that the global snapshot version is the latest state of all partitions, avoiding read inconsistency caused by inconsistent transaction submission order among partitions. Moreover, selecting the maximum value instead of traversing all historical LTID significantly reduces the calculation complexity. In addition, taking the global snapshot version calculated in the last round (such as S1:2, S2:2, S3:2) as the starting point of the new round of judgment and only processing the added LTID further reduces the calculation complexity.

[0114] The embodiment generates the global snapshot version by selecting the maximum value of the LTID of each partition, solves the read inconsistency problem in the traditional distributed system, ensures the global consistency of cross-partition transactions through lightweight sorting and dynamic optimization strategy, and reduces the coordination overhead.

[0115] As another embodiment under the present embodiment, when the corresponding partition does not meet the continuity, the global snapshot is used to locate the subtask corresponding to the discontinuous local transaction identifier;

[0116] The subtask is restarted or manually processed through an alarm to make the partition meet the continuity.

[0117] The core purpose of the embodiment is to locate the subtask corresponding to the discontinuous local transaction identifier (LTID) through the global snapshot, and restore the continuity of the partition through subtask restart or manual alarm mechanism.

[0118] For the located discontinuous subtask, a restart operation or manual alarm is executed. The coordinator sends a restart instruction to the distributed transaction manager (DTM) to re-execute the uncommitted subtask. If the subtask is uncommitted due to network delay or temporary failure, the LTID continuity can be restored after restart.

[0119] By locating and fixing the discontinuous LTIDs, the snapshot versions of each partition are forced to align, avoiding data anomalies caused by inconsistent transaction commit order across partitions. In addition, in high contention scenarios, some partitions can have long-term non-closed LTIDs due to network latency or temporary failures. By restarting the subtasks, the system can quickly recover continuity, and the automatic restart mechanism quickly recovers the LTID discontinuity caused by temporary failures, reducing system blocking time.

[0120] If the subtask cannot be restarted due to logical errors or resource conflicts (such as data conflicts), the coordinator sends an alarm message to the operation and maintenance personnel. After analyzing the logs or manually correcting the problem, the operation and maintenance personnel resubmit the subtask. After the subtask is restarted or manually processed, the coordinator re-verifies the LTID continuity of the partitions.

[0121] For LTID discontinuity caused by logical errors or resource conflicts, automatic restart may not solve the problem. Through manual intervention, the operation and maintenance personnel can accurately repair the problem. The manual alarm mechanism handles complex problems (such as data conflicts), avoiding the cycle failure caused by automatic restart, and avoiding global deadlock caused by long-term LTID discontinuity.

[0122] In one specific embodiment, there are transactions T1, T2, and T3, and partitions S1, S2, and S3. Each transaction divides subtasks that involve all three partitions. Because all partitions are multi-threaded databases, a set of transactions can be committed in different orders on each partition, i.e., they can generate LTID values in different orders, as shown in the following table.

[0123] At the beginning, the initial LTID value on each partition is 0, meaning that they are in a globally consistent state at the beginning. When a transaction is committed, the LTID value is generated on each partition. The consistency coordinator receives the LTID values in the order of T1, T2, and T3. Commit transaction T1, the consistency coordinator uses the LTID value to determine whether a consistent version can be achieved. The LTID sequence on S1 is (0, 1), which is continuous. However, the LTID sequence on S2 is (0, 2), which is discontinuous. This means that a consistent version cannot be obtained now.

[0124] Continue to commit transaction T2, at this stage, the LTID sequence on S1 is (0, 1, 2), which is continuous. However, the LTID sequence on S2 is (0, 2, 3), which is still discontinuous. Also means that a consistent version cannot be obtained now.

[0125] Continuing with the commit of transaction T3, when the LTID values for T3 arrive, the LTID sequence on S1, S2, and S3 all become (0, 1, 2, 3). A globally consistent version can now be accessed. Using the largest LTID value, a globally consistent version is formed (S1:3, S2:3, S3:3). When the next set of LTID values arrives, the coordinator uses (S1:3, S2:3, S3:3) as the initial LTID values to determine continuity.

[0126]

[0127] It should be noted that for each partition, the numbering of the respective sub-transactions is the order in which the sub-transactions are stored to the respective partition. For the S1 partition, the order in which the sub-transactions are stored is T1, T2, T3; for the S2 partition, the order in which the sub-transactions are stored is T3, T1, T2; and for the S3 partition, the order in which the sub-transactions are stored is T2, T3, T1. The order in which the transactions are committed is T1, T2, T3. The discontinuity arises because the order in which the transactions are committed is not consistent with the order in which the corresponding sub-transactions are stored in the partitions.

[0128] The application further provides a storage medium,

[0129] The storage medium has stored thereon a computer program which, when executed, implements the steps of the batch task processing method based on distributed management.

[0130] Therefore, any effects of the batch task processing method based on distributed management can be achieved, and will not be described here.

[0131] The application further provides a processing apparatus, comprising:

[0132] a memory for storing a computer program;

[0133] a processor for implementing the steps of the batch task processing method based on distributed management when the computer program is executed.

[0134] Therefore, any effects of the batch task processing method based on distributed management can be achieved, and will not be described here.

[0135] Any unmentioned aspects in the application can be implemented by using or referring to the prior art.

[0136] Each embodiment in the specification is described in a progressive manner, and the same or similar parts between each embodiment can be understood by mutual reference. Each embodiment mainly describes the differences from other embodiments.

[0137] The above merely illustrates the embodiments of the present application but should not be taken as limitations. Various changes and modifications can be made by those skilled in the art. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application should be included in the scope of claims of the present application.

Claims

1. A batch task processing method based on distributed management, characterized in that: include: In response to the batch data processing tasks submitted by the client, they are decomposed into multiple sub-transactions through the distributed transaction manager; According to the transaction type of the sub-transaction, the corresponding processing flow is called for the transaction type. When the transaction type is an update transaction, the sub-transaction is stored in a partition and a snapshot and a local transaction identifier are generated. According to the local transaction identifier, a consistency coordinator performs continuous judgment and generates a global snapshot in combination with the snapshot.

2. The batch task processing method based on distributed management according to claim 1, characterized in that: The sub-transaction is stored in a partition, and a snapshot and a local transaction identifier are generated, specifically: When the sub-transaction is stored in the partition, a snapshot is generated; Generate local transaction identifiers by sequentially increasing them independently for each partition; A snapshot association is established according to the snapshot and the local transaction identifier through hash table mapping.

3. The batch task processing method based on distributed management according to claim 1, characterized in that: The transaction type of the sub-transaction is specifically: The transaction type of the sub-transaction includes at least an update transaction and a read-only transaction; When the transaction type of the subtransaction is a read-only transaction, determining a corresponding local transaction identifier through the global snapshot; The partition is located using the local transaction identifier, and the stored batch data is read.

4. The method for batch task processing based on distributed management according to claim 3, characterized in that: Locating the partition using the local transaction identifier is as follows: Each partition is independent of each other. After locating the partition using the local transaction identifier, an unlock operation is performed on the partition so that the read-only transaction can read the stored batch data. The unlocated partition is in a locked state, and the stored bulk data cannot be read.

5. The batch task processing method based on distributed management according to claim 1 is characterized in that: Based on the local transaction identifier, the consistency coordinator performs continuous judgment, specifically: Sort each partition individually according to the local transaction identifier of each partition; When the local transaction identifiers of the partitions are discontinuous, determining that the corresponding partitions do not comply with continuity; When the local transaction identifiers of the partitions are continuous, it is determined that the corresponding partitions comply with continuity.

6. The batch task processing method based on distributed management according to claim 5 is characterized in that: When the corresponding partitions meet the continuity requirement, the maximum value is selected according to the sorting of the local transaction identifiers in each partition to generate the global snapshot.

7. The batch task processing method based on distributed management according to claim 6 is characterized in that: Generate the global snapshot, specifically: The global snapshot is updated based on the previous global snapshot using the local transaction identifier corresponding to the newly added update transaction in each partition to generate the global snapshot.

8. The batch task processing method based on distributed management according to claim 5, characterized in that: When the corresponding partition does not comply with continuity, the sub-transaction corresponding to the discontinuous local transaction identifier is located through the global snapshot; The partition is made consistent by restarting the subtransaction or manually processing through an alarm.

9. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed, implements the steps of the batch task processing method based on distributed management according to any one of claims 1 to 8.

10. A processing device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the batch task processing method based on distributed management as described in any one of claims 1 to 8 when executing the computer program.

Images