Distributed system transaction processing method and device based on two-stage commit protocol, equipment and medium

Abstract

The invention discloses a distributed system transaction processing method and device based on a two-stage commit protocol, equipment and a medium, and relates to the technical field of distributed computing, and the method comprises the steps: obtaining transaction data sent by a coordinator before a preparation stage; sending a first response message to the coordinator and the participants based on the transaction data, so as to send an operation instruction to each participant according to the first response message; receiving a second response message sent by each participant, and storing a target number of copy data based on each second response message; if the operation instruction is not received in the target time period, processing the transaction data based on the copy data; and obtaining a corresponding processing result, and synchronizing the processing result to the target participant so as to optimize the two-stage submission protocol. By sharing the transaction state information among the participants, the participants can still complete transaction submission or rollback under the condition that the coordinator loses efficacy in the system, so that the occurrence of indefinite blockage is avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed computing, and in particular to a distributed system transaction processing method and device based on a two-phase commit protocol, equipment and a medium. BACKGROUND

[0002] In a distributed system, multiple nodes need to maintain data consistency through transactions. Two-phase commit protocol (2PC) is the most commonly used distributed transaction consistency protocol, and is widely used in database management systems (DBMS), distributed storage and financial transaction systems, etc.

[0003] At present, the traditional two-phase commit protocol is usually used for data processing in a distributed system. However, if the coordinator crashes between the preparation phase and the commit phase, all participants will enter a blocked state and cannot continue other transaction operations. Therefore, how to optimize the traditional two-phase commit protocol and avoid the occurrence of the blocked state of each participant has become a technical problem to be solved. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a distributed system transaction processing method and device based on a two-phase commit protocol, which can share transaction state information between participants, so that the participants can complete transaction commit or rollback in the case of coordinator failure, and avoid the occurrence of indefinite blocking. The specific scheme is as follows:

[0005] In a first aspect, the present application provides a distributed system transaction processing method based on a two-phase commit protocol, applied to a node corresponding to any participant, comprising:

[0006] Obtaining target transaction data sent by a coordinator corresponding to a two-phase commit protocol before a preparation phase of the two-phase commit protocol, and caching the target transaction data based on a target storage structure;

[0007] In the preparation phase of the two-phase commit protocol, sending a corresponding first response message to the coordinator and target participants based on the target transaction data cached locally, so that the coordinator receives the first response message and sends a corresponding operation instruction to each participant according to the first response message;

[0008] receive second response messages sent by each of the target participants, and store a target number of copy data based on each of the second response messages; wherein the response message comprises a transaction ID of the target transaction data, a participant ID corresponding to the participant, and state information; and the copy data comprises target state information corresponding to each of the target participants;

[0009] start receiving the operation instruction, and if the operation instruction sent by the coordinator is not received within a target time period, process the target transaction data based on the copy data;

[0010] obtain a processing result corresponding to the target transaction data, and synchronize the processing result to the target participant; wherein the target participant is a participant other than the any participant.

[0011] Optionally, the process of sending the target transaction data to the any participant by the coordinator comprises:

[0012] obtain initial transaction data, perform data verification on the initial transaction data to obtain corresponding verified data, and perform data compression on the verified data to obtain compressed data corresponding to the verified data;

[0013] perform data sharding on the compressed data to obtain the target transaction data, and send the target transaction data to the any participant before a preparation phase of a two-phase commit protocol.

[0014] Optionally, the process of sending the target transaction data to the any participant by the coordinator comprises:

[0015] send the target transaction data to the any participant by using a target data sending tool; wherein the target data sending tool comprises at least one of a high-availability message queue and a distributed log system.

[0016] Optionally, the storing of a target number of copy data based on each of the second response messages comprises:

[0017] determine the target state information corresponding to each of the target participants from each of the second response messages, generate the copy data corresponding to each of the target state information, and store a target number of the copy data based on a distributed hash table and a consistent hashing algorithm.

[0018] Optionally, the processing of the target transaction data based on the copy data comprises:

[0019] statistically determine target state information corresponding to each of the target participants based on the copy data, to determine a first quantity ratio corresponding to first target state information and a second quantity ratio corresponding to second target state information;

[0020] If the first quantity ratio corresponding to the first target state information is greater than a preset proportion threshold, the target transaction data is submitted to a target client;

[0021] If the second quantity ratio corresponding to the second target state information is greater than the preset proportion threshold, the target transaction data is rolled back.

[0022] Optionally, before starting to receive the operation instruction, the method further includes:

[0023] setting an initial time period for timeout judgment of the operation instruction, determining a current network state corresponding to a target network, and adjusting the initial time period according to the current network state to obtain the target time period corresponding to the operation instruction.

[0024] Optionally, after starting to receive the operation instruction, the method further includes:

[0025] If the operation instruction sent by the coordinator is received within the target time period, the target transaction data is directly executed or rolled back according to an instruction type of the operation instruction.

[0026] In a second aspect, the application provides a distributed system transaction processing apparatus based on a two-phase commit protocol, applied to a node corresponding to any participant, including:

[0027] a data caching module, configured to obtain target transaction data sent by a coordinator corresponding to the two-phase commit protocol before a preparation phase of the two-phase commit protocol, and cache the target transaction data based on a target storage structure;

[0028] a message sending module, configured to send corresponding first response messages to the coordinator and target participants based on the locally cached target transaction data in the preparation phase of the two-phase commit protocol, so that the coordinator receives the first response messages and sends corresponding operation instructions to each participant according to the first response messages;

[0029] a message receiving module, configured to receive second response messages sent by each of the target participants, and store copy data of a target quantity based on each of the second response messages; wherein the response message includes a transaction ID of the target transaction data, a participant ID corresponding to a participant, and state information; and the copy data includes target state information corresponding to each of the target participants.

[0030] The data processing module is configured to start receiving the operation instruction, and if the operation instruction sent by the coordinator is not received within a target time period, process the target transaction data based on the copy data.

[0031] The data synchronization module is configured to obtain a processing result corresponding to the target transaction data, and synchronize the processing result to the target participant. The target participant is a participant other than the any participant.

[0032] In a third aspect, the present application provides an electronic device, comprising:

[0033] A memory configured to save a computer program.

[0034] A processor configured to execute the computer program to implement the preceding distributed system transaction processing method based on the two-phase commit protocol.

[0035] In a fourth aspect, the present application provides a computer readable storage medium configured to save a computer program. The computer program is executed by a processor to implement the preceding distributed system transaction processing method based on the two-phase commit protocol.

[0036] The application first acquires target transaction data sent by a coordinator corresponding to a two-phase commit protocol before a preparation phase of the two-phase commit protocol, and caches the target transaction data based on a target storage structure, and then sends a corresponding first response message to the coordinator and target participants based on the locally cached target transaction data in the preparation phase of the two-phase commit protocol, so that the coordinator receives the first response message and sends a corresponding operation instruction to each participant according to the first response message, and then receives a second response message sent by each target participant, and stores target number of copy data based on each second response message; wherein the response message includes a transaction ID of the target transaction data, a participant ID corresponding to the participant, and state information; the copy data includes target state information corresponding to each target participant, and then the receiving of the operation instruction is started, if the operation instruction sent by the coordinator is not received within a target time period, the target transaction data is processed based on the copy data, finally the processing result corresponding to the target transaction data is acquired, and the processing result is synchronized to the target participant; wherein the target participant is a participant other than the any participant. As can be seen, the application makes the coordinator send transaction data to the participant before the preparation phase, reduces the data transmission delay in the commit phase, improves the overall throughput, and thus improves the processing efficiency of the transaction data; by sharing the transaction state information between the participants, the system can complete transaction commit or rollback by the participant itself in the case of coordinator failure, without relying on the coordinator, avoiding the occurrence of indefinite blocking, and improving the availability of the scheme; by the timeout self-determination strategy, the participant is allowed to autonomously determine commit or rollback according to the local transaction log when the participant does not receive the coordinator instruction within the timeout, avoiding the influence of long-term transaction locking on concurrent performance. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute the embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor based on the provided drawings.

[0038] Figure 1 A flow chart of a distributed system transaction processing method based on a two-phase commit protocol disclosed by the present application;

[0039] Figure 2 A comparison chart of transmission efficiency of an optimized two-phase transmission protocol disclosed by the present application;

[0040] Figure 3A distributed system transaction processing device structure based on a two-phase commit protocol disclosed by the application is shown in the figure.

[0041] Figure 4 An electronic device structure disclosed by the application is shown in the figure. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the application will be clearly and completely described in the embodiments of the application in combination with the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by a person of ordinary skill in the art without creative work belong to the protection scope of the application.

[0043] At present, the traditional two-phase commit protocol is usually used for data processing in a distributed system. If the coordinator crashes between the preparation phase and the commit phase, all participants will enter a blocked state and cannot continue other transaction operations. Therefore, the application provides a distributed system transaction processing method based on a two-phase commit protocol. By sharing transaction state information between participants, the system can still complete transaction commit or rollback in the case of coordinator failure, avoiding indefinite blocking.

[0044] Referring to Figure 1 The embodiments of the application disclose a distributed system transaction processing method based on a two-phase commit protocol, applied to a node corresponding to any participant, comprising:

[0045] In step S11, the target transaction data sent by the coordinator corresponding to the two-phase commit protocol before the preparation phase of the two-phase commit protocol is acquired, and the target transaction data is cached based on a target storage structure.

[0046] The embodiments provide an optimized two-phase commit (2PC) protocol to reduce transaction blocking and improve transaction processing efficiency and availability of the system. The traditional 2PC may cause performance bottlenecks in a high-concurrency distributed environment due to single-point failure of the coordinator, transaction timeout waiting and other problems. Based on the 2PC, the present scheme proposes three optimization mechanisms of transaction data pre-transmission, redundant confirmation and timeout self-determination to reduce communication overhead, improve transaction processing efficiency and enhance fault tolerance of the system.

[0047] It should be noted that in the standard 2PC protocol, the participants may still need to obtain data from the coordinator in the commit phase, resulting in transaction delay. Since there is a certain network delay in data transmission, and in a high-concurrency scenario, the load of the coordinator may become a bottleneck, reducing the commit efficiency of the entire transaction.

[0048] Therefore, the embodiment optimizes the traditional 2PC protocol. Before the preparation phase, the coordinator sends transaction-related data to all participants to make them cache in advance, avoiding data transmission delay in the submission phase.

[0049] In the embodiment, the process of sending target transaction data to any participant by the coordinator can specifically include: obtaining initial transaction data, performing data verification on the initial transaction data to obtain corresponding verified data, and performing data compression on the verified data to obtain compressed data corresponding to the verified data; performing data sharding on the compressed data to obtain target transaction data, and sending the target transaction data to any participant before the preparation phase of the two-phase commit protocol.

[0050] Specifically, the transaction data pre-transmission mechanism in the embodiment includes the following steps:

[0051] Preprocessing of coordinator data: Before starting transaction submission, the coordinator preprocesses transaction-related data (i.e., initial transaction data) to ensure integrity. Preprocessing includes data verification, data compression, and data sharding operations (to obtain target transaction data) to reduce network transmission overhead.

[0052] Transaction data distribution: The coordinator sends transaction data to all participants in advance and notifies them to cache. Data distribution is performed in an asynchronous manner to avoid blocking the main thread of the coordinator.

[0053] Cache data: After receiving the data, the participant stores it locally, but does not immediately execute transaction operations, only for subsequent transaction submission. The cached data uses an efficient memory data structure (such as a hash table or a B+ tree) to support fast queries.

[0054] Preparation phase execution: When the coordinator sends the "Prepare" request, the participant can directly execute transaction verification without obtaining data from the coordinator, thereby reducing the time overhead of data transmission and improving transaction submission speed.

[0055] The specific code of the above process is as follows:

[0056] typedef struct {

[0057] int transaction_id;

[0058] char data

[1024] ; / / transaction data

[0059] } TransactionData;

[0060] / / Send transaction data to all participants

[0061] void send_transaction_data(int participant_sockets[], int num_participants, TransactionData *tx) {

[0062] for (int i = 0; i < num_participants; i++) {

[0063] send(participant_sockets[i], tx, sizeof(TransactionData), 0);

[0064] }

[0065] }

[0066] / / Participant receives transaction data and caches (assuming maximum cache of 100 transaction data)

[0067] void cache_transaction_data(TransactionData *tx) {

[0068] if (cache_count < 100) {

[0069] cache[cache_count].transaction_id = tx->transaction_id;

[0070] memcpy(cache[cache_count].data, tx->data, sizeof(tx->data));

[0071] cache_count++;

[0072] printf("Cached transaction data: %d\n", tx->transaction_id);

[0073] } else {

[0074] printf("Cache is full, cannot cache transaction data: %d\n", tx->transaction_id);

[0075] }

[0076] }

[0077] In this embodiment, the process in which the coordinator sends the target transaction data to any participant can specifically include: sending the target transaction data to any participant by using a target data sending tool; wherein the target data sending tool includes at least one of a high-availability message queue and a distributed log system.

[0078] Through the transaction data pre-transmission mechanism, the transaction-related data is distributed to the participants in advance before the commit phase, the data transmission delay in the commit phase is reduced, and the overall throughput is improved.

[0079] In step S12, in the preparation phase of the two-phase commit protocol, the first response message corresponding to the target transaction data based on the local cache is sent to the coordinator and the target participant, so that the coordinator receives the first response message and sends the corresponding operation instruction to each participant according to the first response message.

[0080] The standard 2PC relies on the coordinator to determine the transaction state. Once the coordinator fails, all participants may wait indefinitely, causing system blocking. Therefore, in this embodiment, the above-mentioned redundant confirmation mechanism is used. The participant (i.e. any participant mentioned above) sends a "READY" message to the coordinator in the preparation phase, and also sends the same message to k other randomly selected participants (i.e. target participants), allowing the participant to recover the transaction state through P2P (Peer-to-peer, i.e. point-to-point technology) message exchange, even if the coordinator fails, the transaction can be determined to be committed or rolled back.

[0081] Through the redundant confirmation mechanism, the participants share the transaction state information, so that the system can still complete the transaction commit or rollback in the case of coordinator failure, avoiding indefinite blocking.

[0082] In step S13, the second response message sent by each target participant is received, and the target number of copy data is stored based on each second response message; wherein the response message includes the transaction ID (Identity document, i.e. identity number) of the target transaction data, the participant ID corresponding to the participant and the state information; the copy data includes the target state information corresponding to each target participant.

[0083] In this embodiment, the process of storing the target number of copy data based on each second response message can specifically include: determining the target state information corresponding to each target participant from each second response message, generating the copy data corresponding to each target state information, and storing the target number of copy data based on the distributed hash table and the consistent hash algorithm.

[0084] The redundant confirmation mechanism in this embodiment is as follows:

[0085] Participants send redundant acknowledgements: In the standard 2PC prepare phase, participants usually only send "READY" or "ABORT" messages (i.e., first target state information and second target state information) to the coordinator, this scheme requires participants to send copies of the above messages to k other randomly selected participants at the same time, the copy message content includes transaction ID, participant ID and state information.

[0086] Store redundant state: Each participant stores at least k copies of transaction state locally (i.e., copy data) as a backup solution for transaction consistency recovery. Storage uses a distributed hash table (DHT) or consistent hashing algorithm to improve query efficiency.

[0087] Coordinator failure detection: If a participant does not receive a commit / rollback command from the coordinator within a timeout, it can request redundant acknowledgement information from other participants, and decide the transaction state according to the majority of the confirmation results.

[0088] Transaction recovery: If the majority of participants return the "READY" state, commit the transaction; if some return "ABORT" or the state is unknown, roll back the transaction.

[0089] The code of the above process is as follows:

[0090] typedef struct {

[0091] int transaction_id;

[0092] int participant_id;

[0093] char status

[16] ; / / "READY" or "ABORT"

[0094] } RedundantAck;

[0095] / / Participants send acknowledgement information to other participants

[0096] void send_redundant_ack(int peer_sockets[], int num_peers,RedundantAck *ack) {

[0097] for (int i = 0; i < num_peers; i++) {

[0098] send(peer_sockets[i], ack, sizeof(RedundantAck), 0);

[0099] }

[0100] }

[0101] / / Participants receive redundant acknowledgments

[0102] void receive_redundant_ack(int sockfd, RedundantAck *ack) {

[0103] recv(sockfd, ack, sizeof(RedundantAck), 0);

[0104] printf("Received redundant ACK from participant %d: %s\n", ack->participant_id, ack->status);

[0105] }

[0106] Step S14, start receiving the operation instruction, if the operation instruction sent by the coordinator is not received within the target time period, processing the target transaction data based on the copy data.

[0107] The standard 2PC requires participants to wait for the final decision of the coordinator, and once the coordinator fails, the transaction may be blocked indefinitely, occupying system resources. Therefore, the embodiment introduces a timeout-based decision strategy, allowing participants to make autonomous decisions on transaction submission or rollback after a timeout, avoiding long waiting for the coordinator.

[0108] Correspondingly, in the embodiment, before starting to receive the operation instruction, it also includes: setting an initial time period for timeout judgment of the operation instruction, determining the current network state corresponding to the target network, and adjusting the initial time period according to the current network state to obtain the target time period corresponding to the operation instruction.

[0109] The process of the above timeout-based decision strategy 1 is as follows:

[0110] Set timeout threshold: the participant sets a timeout time (i.e. target time period) in the preparation phase, such as 5 seconds. The timeout threshold can be dynamically adjusted according to the network condition to optimize the transaction processing efficiency.

[0111] Timeout detection: if the final decision of the coordinator is still not received after the timeout, check the redundant confirmation information. The timeout detection uses a polling or event-driven mechanism to reduce CPU overhead.

[0112] Transaction self-decision:

[0113] If all received messages are "READY", commit the transaction directly.

[0114] If an "ABORT" message is received or the participant status is uncertain, roll back the transaction.

[0115] Result feedback: After transaction commit or rollback, participants notify other nodes of the final decision to ensure transaction consistency.

[0116] The specific code for the above process is as follows:

[0117] #define TIMEOUT 5000 / / Timeout time (milliseconds)

[0118] void wait_for_commit_decision(int sockfd) {

[0119] struct timeval timeout;

[0120] timeout.tv_sec = TIMEOUT / 1000;

[0121] timeout.tv_usec = (TIMEOUT % 1000) * 1000;

[0122] fd_set read_fds;

[0123] FD_ZERO(&read_fds);

[0124] FD_SET(sockfd, &read_fds);

[0125] int result = select(sockfd + 1, &read_fds, NULL, NULL, &timeout);

[0126] if (result > 0) {

[0127] char decision

[16] ;

[0128] recv(sockfd, decision, sizeof(decision), 0);

[0129] printf("Received decision: %s\n", decision);

[0130] } else {

[0131] printf("Timeout reached! Making autonomous decision...\n");

[0132] if (can_commit()) {

[0133] printf("All peers are READY, committing transaction.\n");

[0134] } else {

[0135] printf("Some peers aborted, rolling back transaction.\n");

[0136] }

[0137] }

[0138] }

[0139] By the timeout self-decision policy, the participant is allowed to make autonomous decision to commit or rollback according to the local transaction log when no instruction from the coordinator is received within the timeout, so as to avoid the influence of long-term transaction locking on concurrent performance.

[0140] In the embodiment, the process of processing the target transaction data based on the replica data can specifically include: counting the target state information corresponding to each target participant based on the replica data to determine a first quantity proportion corresponding to the first target state information and a second quantity proportion corresponding to the second target state information; if the first quantity proportion corresponding to the first target state information is greater than a preset proportion threshold, submitting the target transaction data to the target client; and if the second quantity proportion corresponding to the second target state information is greater than the preset proportion threshold, rolling back the target transaction data. That is, if most participants return the "READY" state (i.e., the quantity proportion of the first target state information is greater than the preset proportion threshold), the transaction is committed; if some participants return the "ABORT" state or the state is unknown, the transaction is rolled back. The preset proportion threshold in the embodiment can be set according to requirements, which is not specifically limited here.

[0141] It can be understood that in the embodiment, if the operation instruction sent by the coordinator is received within the target time period, it means that the coordinator does not fail at this time and is still in a normal working state, and then the target transaction data is directly executed or rolled back according to the instruction type of the operation instruction.

[0142] Step S15, obtaining a processing result corresponding to the target transaction data, and synchronizing the processing result to the target participant; wherein the target participant is a participant other than the any participant.

[0143] As described in the foregoing steps, in the embodiment, after the transaction is committed or rolled back, the participant notifies other nodes of the final decision, ensuring transaction consistency, thereby completing the processing process of the target transaction data. The effect of the distributed system transaction processing method in the embodiment is as shown in Figure 2

[0144] It should be noted that the embodiment can be applied to the following distributed transaction processing scenarios:

[0145] 1. Distributed database transaction:

[0146] Distributed database systems (such as MySQL Group Replication, TiDB, CockroachDB) often need to perform transaction operations across multiple nodes. The optimized 2PC protocol can play a role in the following scenarios:

[0147] Cross-node data consistency: In a distributed database, data is usually sharded and stored on different nodes. The 2PC protocol can ensure that cross-node transaction operations (such as insertion, update, and deletion) are atomic and consistent.

[0148] High-concurrency transaction processing: Through the transaction data pre-transmission mechanism, the data transmission delay in the commit phase is reduced, and the throughput in high-concurrency scenarios is improved.

[0149] Coordinator fault recovery: Through the redundant confirmation mechanism and timeout self-determination strategy, even if the coordinator fails, the participant can autonomously determine the transaction state, avoiding system blocking.

[0150] 2. Blockchain transaction system

[0151] Blockchain is a typical distributed storage system, and its transaction confirmation process requires consensus among multiple nodes. The optimized 2PC protocol can be applied in the following scenarios:

[0152] Transaction confirmation: In a blockchain, transactions need to be verified and confirmed by multiple nodes. The 2PC protocol can ensure the consistency of transactions across all nodes.

[0153] Smart contract execution: The execution of a smart contract may involve multiple distributed nodes. The 2PC protocol can ensure the atomicity of contract execution.

[0154] Fault recovery: Through the redundant confirmation mechanism, even if some nodes fail, the transaction state can still be recovered through the confirmation information of other nodes. ​

[0155] 3. Distributed File System

[0156] Distributed file systems (e.g., HDFS, Ceph, GlusterFS) often require storing and accessing file data across multiple nodes. The optimized 2PC protocol can be applied to the following scenarios:

[0157] File metadata update: In distributed file systems, updating file metadata (e.g., file size, permissions, block location) needs to be synchronized across multiple nodes. The 2PC protocol can ensure the consistency of metadata updates.

[0158] File write operation: In distributed file systems, file write operations may involve multiple data blocks and replicas. The 2PC protocol can ensure the atomicity of write operations.

[0159] Fault recovery: With the timeout self-decision policy, even if some nodes fail, file operations can continue to be executed, avoiding system blocking.

[0160] 4. Distributed Cache System

[0161] Distributed cache systems (e.g., Redis Cluster, Memcached) are often used to accelerate data access. The optimized 2PC protocol can be applied to the following scenarios:

[0162] Cache data consistency: In distributed cache systems, updating cache data needs to be synchronized across multiple nodes. The 2PC protocol can ensure the consistency of cache data.

[0163] High-concurrency cache update: With the transaction data pre-transmission mechanism, the delay of cache update operations is reduced, and the performance in high-concurrency scenarios is improved.

[0164] Fault recovery: With the redundant confirmation mechanism, even if some nodes fail, cache data can be recovered through the confirmation information of other nodes.

[0165] 5. Distributed Object Storage

[0166] Distributed object storage systems (e.g., Amazon S3, MinIO) are often used to store large-scale unstructured data. The optimized 2PC protocol can be applied to the following scenarios:

[0167] Object metadata update: In distributed object storage systems, updating object metadata (e.g., object size, permissions, storage location) needs to be synchronized across multiple nodes. The 2PC protocol can ensure the consistency of metadata updates.

[0168] Object Write Operation: In a distributed object storage system, an object write operation may involve multiple data blocks and replicas. The 2PC protocol can ensure the atomicity of the write operation.

[0169] Fault Recovery: With the timeout self-decision policy, even if some nodes fail, object operations can still continue to execute, avoiding system blocking.

[0170] 6. Distributed Logging System

[0171] Distributed logging systems (such as Kafka, Apache Pulsar) are commonly used to store and distribute log data. The optimized 2PC protocol can be applied to the following scenarios:

[0172] Log Write Operation: In a distributed logging system, log write operations need to be synchronized across multiple nodes. The 2PC protocol can ensure the atomicity of the write operation.

[0173] Log Partition Consistency: In a distributed logging system, log data is usually stored in different partitions. The 2PC protocol can ensure the consistency of cross-partition operations.

[0174] Fault Recovery: With the redundant confirmation mechanism, even if some nodes fail, log data can be recovered through the confirmation information of other nodes.

[0175] 7. Distributed Message Queue

[0176] Distributed message queue systems (such as RabbitMQ, RocketMQ) are commonly used to decouple system components and implement asynchronous communication. The optimized 2PC protocol can be applied to the following scenarios:

[0177] Message Transaction Processing: In a distributed message queue system, message sending and confirmation need to be synchronized across multiple nodes. The 2PC protocol can ensure the atomicity of message transactions.

[0178] Message Consistency: Through the transaction data pre-transmission mechanism, reduce the delay of message transmission, improve the performance in high concurrency scenarios.

[0179] Fault Recovery: With the timeout self-decision policy, even if some nodes fail, message transactions can still continue to execute, avoiding system blocking.

[0180] 8. Distributed Configuration Management

[0181] Distributed configuration management systems (such as ZooKeeper, etcd) are commonly used to store and manage configuration information of distributed systems. The optimized 2PC protocol can be applied to the following scenarios:

[0182] Configuration update operation: In a distributed configuration management system, the update of configuration information needs to be synchronized across multiple nodes. The 2PC protocol can ensure the consistency of configuration update.

[0183] High-concurrency configuration update: Through the transaction data pre-transmission mechanism, the delay of configuration update operation is reduced, and the performance in high-concurrency scenarios is improved.

[0184] Fault recovery: Through the redundancy confirmation mechanism, even if some nodes fail, the configuration information can be recovered through the confirmation information of other nodes.

[0185] As can be seen, the embodiment reduces the data transmission delay of the commit phase by distributing transaction data to participants in the preparation phase, even if the coordinator fails, the participants can recover the transaction state through P2P message exchange; by allowing participants to autonomously determine the transaction state after timeout, avoiding indefinite waiting for the coordinator, and dispersing part of the load among participants, the pressure on the coordinator is reduced; compared with three-phase commit (3PC), the present application does not additionally increase the commit phase, only through a small amount of message redundancy and logical optimization, the system performance is improved while maintaining low protocol complexity.

[0186] Referring to Figure 3 The embodiment of the application discloses a kind of distributed system transaction processing devices based on two-phase commit protocol, it is applied to any participant corresponding node, including:

[0187] Data cache module 11 is used to obtain the target transaction data that the coordinator corresponding to two-phase commit protocol sends before the preparation phase of two-phase commit protocol, and the target transaction data is cached based on target storage structure;

[0188] Message sending module 12 is used to send corresponding first response message to the coordinator and target participant based on the target transaction data cached locally in the preparation phase of two-phase commit protocol, so that the coordinator receives the first response message, and according to the first response message, each participant sends corresponding operation instruction;

[0189] Message receiving module 13 is used to receive the second response message sent by each target participant, and store target number of copy data based on each second response message;Wherein, response message includes the transaction ID of the target transaction data, the participant ID corresponding to participant and state information;The copy data includes the target state information corresponding to each target participant;

[0190] Data processing module 14 is used to start receiving the operation instruction, if the operation instruction sent by the coordinator is not received in target time period, the target transaction data is processed based on the copy data;

[0191] The data synchronization module 15 is configured to obtain a processing result corresponding to the target transaction data, and synchronize the processing result to the target participant; wherein the target participant is a participant other than the any participant.

[0192] Therefore, the application reduces the data transmission delay of the commit phase by sending transaction data from the coordinator to the participants before the preparation phase, improves the overall throughput, and thus improves the processing efficiency of transaction data; by allowing participants to share transaction state information, the system can complete transaction commit or rollback by the participants themselves in the case of coordinator failure, without relying on the coordinator, avoiding the occurrence of indefinite blocking, and improving the availability of the scheme; by the timeout self-determination strategy, the participants are allowed to autonomously determine commit or rollback according to the local transaction log when the timeout is not received from the coordinator, avoiding the influence of long-term transaction locking on concurrent performance.

[0193] In some specific embodiments, the coordinator can specifically include:

[0194] The data compression module is configured to obtain initial transaction data, perform data verification on the initial transaction data to obtain corresponding verified data, and perform data compression on the verified data to obtain compressed data corresponding to the verified data.

[0195] The data sharding module is configured to perform data sharding on the compressed data to obtain the target transaction data, and send the target transaction data to the any participant before the preparation phase of the two-phase commit protocol.

[0196] In some specific embodiments, the coordinator can specifically include:

[0197] The data sending module is configured to send the target transaction data to the any participant by using a target data sending tool; wherein the target data sending tool includes at least one of a high-availability message queue and a distributed log system.

[0198] In some specific embodiments, the message receiving module 13 can specifically include:

[0199] The data storage unit is configured to determine the target state information corresponding to each of the target participants from each of the second response messages, generate the replica data corresponding to each of the target state information, and store the target number of replica data based on a distributed hash table and a consistent hashing algorithm.

[0200] In some specific embodiments, the data processing module 14 can specifically include:

[0201] An information counting unit is configured to count target state information corresponding to each target participant based on the copy data, to determine a first quantity proportion corresponding to first target state information and a second quantity proportion corresponding to second target state information.

[0202] A data submission unit is configured to submit the target transaction data to a target client if the first quantity proportion corresponding to the first target state information is greater than a preset proportion threshold.

[0203] A data rollback unit is configured to rollback the target transaction data if the second quantity proportion corresponding to the second target state information is greater than the preset proportion threshold.

[0204] In some embodiments, the data processing module 14 further includes:

[0205] A time period acquisition unit is configured to set an initial time period for timeout judgment of the operation instruction, determine a current network state corresponding to a target network, and adjust the initial time period according to the current network state to obtain the target time period corresponding to the operation instruction.

[0206] In some embodiments, the data processing module 14 further includes:

[0207] A data processing unit is configured to directly perform or rollback operation on the target transaction data according to an instruction type of the operation instruction if the operation instruction sent by the coordinator is received within the target time period.

[0208] Further, the embodiment of the present application further discloses an electronic device, Figure 4 is an electronic device 20 structure diagram shown according to an exemplary embodiment, the contents in the figure cannot be considered as any limitation on the use range of the present application.

[0209] Figure 4 A structure schematic diagram of an electronic device 20 provided by the embodiment of the present application. The electronic device 20 specifically can include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input output interface 25 and a communication bus 26. Wherein, the memory 22 is used for storing computer program, the computer program is loaded and executed by the processor 21, to realize the related steps in the preceding any embodiment disclosed distributed system transaction processing method based on two-phase commit protocol. In addition, the electronic device 20 in the embodiment specifically can be electronic computer.

[0210] In this embodiment, the power supply 23 is configured to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 is configured to create a data transmission channel between the electronic device 20 and external devices, and the communication protocol followed by the communication interface 24 can be any communication protocol applicable to the technical solution of the present application, which is not limited specifically herein; the input and output interface 25 is configured to obtain external input data or output data to the outside, and the specific interface type can be selected according to the specific application needs, which is not limited specifically herein.

[0211] In addition, the memory 22 as a carrier of resource storage can be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage mode can be temporary storage or permanent storage.

[0212] The operating system 221 is configured to manage and control each hardware device and the computer program 222 on the electronic device 20, and can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program capable of completing the distributed system transaction processing method based on the two-phase commit protocol executed by the electronic device 20 disclosed in any of the foregoing embodiments, the computer program 222 can further include a computer program capable of completing other specific work.

[0213] Further, the present application further discloses a computer readable storage medium for storing a computer program; wherein the computer program is executed by a processor to implement the foregoing disclosed distributed system transaction processing method based on the two-phase commit protocol. For the specific steps of the method, refer to the corresponding contents disclosed in the foregoing embodiments, which will not be repeated here.

[0214] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. For the same or similar parts between the embodiments, refer to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts refer to the method part.

[0215] Those skilled in the art will further appreciate that the units and algorithm steps of the various examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various examples were described above generally in terms of their functionality, without referring to the corresponding

[0216] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in random access memory (RAM), flash memory, read-only memory (ROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is tangible.

[0217] Finally, it is to be noted that the terms "comprising", "including", and "having" are meant to be interpreted open-ended, i.e. in the sense of specifying the presence of stated features, integers, steps or components, but not precluding the presence of one or more other features, integers, steps, components or groups thereof. It is also to be noted that, as used in the specification and the appended claims, the singular form "a", "an" and "the" include plural references unless the context clearly dictates otherwise. As such, the terms "comprising", "including", "containing", "having" and the like, are to be construed open-ended, i.e. in the sense of specifying features, integers, steps or components that are present, but not in the sense of precluding the presence of one or more other features, integers, steps, components or groups thereof.

[0218] The above detailed description has set forth various examples of the technology disclosed herein. The description is purposefully rendered in sub-patent claims to particularly point out patentable subject matter. The description set forth is intended to enable practitioners to practice the technology as set forth in the claims.

Claims

1. A distributed system transaction processing method based on a two-phase commit protocol, characterized in that: Applicable to the nodes corresponding to any participant, including: Obtain target transaction data sent by a coordinator corresponding to the two-phase commit protocol before a preparation phase of the two-phase commit protocol, and cache the target transaction data based on a target storage structure; In the preparation phase of the two-phase commit protocol, a corresponding first response message is sent to the coordinator and the target participant based on the locally cached target transaction data, so that the coordinator receives the first response message and sends corresponding operation instructions to each participant according to the first response message; Receive a second response message sent by each target participant, and store a target number of replica data based on each second response message; wherein the response message includes a transaction ID of the target transaction data, a participant ID corresponding to the participant, and status information; and the replica data includes target status information corresponding to each target participant; Initiate reception of the operation instruction, and if the operation instruction sent by the coordinator is not received within a target time period, process the target transaction data based on the replica data; Obtain a processing result corresponding to the target transaction data, and synchronize the processing result to the target participant; wherein the target participant is a participant other than any of the participants.

2. The distributed system transaction processing method based on the two-phase commit protocol according to claim 1, characterized in that: The process of the coordinator sending the target transaction data to any of the participants includes: Acquiring initial transaction data, performing data verification on the initial transaction data to obtain corresponding verified data, and performing data compression on the verified data to obtain compressed data corresponding to the verified data; The compressed data is segmented to obtain the target transaction data, and the target transaction data is sent to any one of the participants before the preparation phase of the two-phase commit protocol.

3. The distributed system transaction processing method based on the two-phase commit protocol according to claim 1, characterized in that: The process of the coordinator sending the target transaction data to any of the participants includes: The target transaction data is sent to any of the participants using a target data sending tool; wherein the target data sending tool includes at least one of a high-availability message queue and a distributed log system.

4. The distributed system transaction processing method based on the two-phase commit protocol according to claim 1, characterized in that: The storing a target number of replica data based on each of the second response messages includes: The target state information corresponding to each target participant is determined from each second response message, the replica data corresponding to each target state information is generated, and the target number of replica data is stored based on a distributed hash table and a consistent hash algorithm.

5. The distributed system transaction processing method based on the two-phase commit protocol according to claim 1, characterized in that: The processing of the target transaction data based on the replica data includes: Performing statistics on the target status information corresponding to each target participant based on the copy data to determine a first quantity ratio corresponding to the first target status information and a second quantity ratio corresponding to the second target status information; If the first quantity corresponding to the first target status information accounts for more than a preset ratio threshold, submitting the target transaction data to the target client; If the second quantity corresponding to the second target status information accounts for more than the preset ratio threshold, the target transaction data is rolled back.

6. The distributed system transaction processing method based on the two-phase commit protocol according to claim 1, characterized in that: Before the initiation of receiving the operation instruction, the method further includes: An initial time period for determining a timeout of the operation instruction is set, a current network state corresponding to the target network is determined, and the initial time period is adjusted according to the current network state to obtain the target time period corresponding to the operation instruction.

7. The distributed system transaction processing method based on the two-phase commit protocol according to any one of claims 1 to 6, characterized in that: After the initiation of receiving the operation instruction, the method further includes: If the operation instruction sent by the coordinator is received within the target time period, the target transaction data is directly executed or rolled back according to the instruction type of the operation instruction.

8. A distributed system transaction processing device based on a two-phase commit protocol, characterized in that: Applicable to the nodes corresponding to any participant, including: A data caching module is used to obtain target transaction data sent by the coordinator corresponding to the two-phase commit protocol before the preparation phase of the two-phase commit protocol, and cache the target transaction data based on the target storage structure; a message sending module, configured to send a corresponding first response message to the coordinator and target participant based on the locally cached target transaction data during the preparation phase of the two-phase commit protocol, so that the coordinator receives the first response message and sends corresponding operation instructions to each participant according to the first response message; a message receiving module, configured to receive a second response message sent by each target participant, and store a target number of replica data based on each second response message; wherein the response message includes a transaction ID of the target transaction data, a participant ID corresponding to the participant, and status information; and the replica data includes target status information corresponding to each target participant; a data processing module, configured to initiate reception of the operation instruction, and process the target transaction data based on the replica data if the operation instruction sent by the coordinator is not received within a target time period; A data synchronization module is used to obtain the processing result corresponding to the target transaction data and synchronize the processing result to the target participant; wherein the target participant is a participant other than any of the participants.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the distributed system transaction processing method based on the two-phase commit protocol according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that Used to store a computer program, which, when executed by a processor, implements the distributed system transaction processing method based on the two-phase commit protocol as described in any one of claims 1 to 7.

Images