Transaction processing method and device, equipment and storage medium

Abstract

The invention discloses a transaction processing method and device, equipment and a storage medium, and relates to the field of distributed systems, and the method comprises the steps: in a distributed system, determining a target transaction processing period through employing an initial transaction processing period, a system load, the number of to-be-processed transactions, a network condition and a resource utilization condition of distributed nodes, which are determined based on an epoch mechanism; allocating the to-be-processed transaction to a target transaction processing cycle, and controlling the start of the target transaction processing cycle, so that the distributed node performs asynchronous processing on the to-be-processed transaction in the target transaction processing cycle, and a processed transaction is obtained; obtaining a transaction processing state of the distributed node, and determining a to-be-submitted transaction and a to-be-rolled-back transaction from the processed transactions based on the transaction processing state; and when the target transaction processing cycle is ended, submitting the to-be-submitted transaction, and redistributing the to-be-rollback transaction to the target transaction processing cycle for processing. Therefore, the transaction processing efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the field of distributed systems, and in particular to a transaction processing method, device, equipment and storage medium. Background Art

[0002] In today's distributed systems, transaction processing is a key technology to ensure data consistency and reliability. However, with the continuous expansion of the scale of distributed systems and the sharp increase in business volume, traditional transaction processing mechanisms face serious performance bottlenecks when processing a large number of transactions. In traditional synchronous commit methods such as two-phase commit or three-phase commit, the transaction coordinator needs to wait for the responses of all participants, which will generate a lot of synchronous waiting time. Especially in large-scale distributed systems, there are a large number of nodes, and problems such as network delays and node failures frequently occur, which further aggravates this waiting situation, resulting in a decline in the overall performance of the system, thereby reducing the efficiency of transaction processing.

[0003] Therefore, how to improve the efficiency of transaction processing is a technical problem that needs to be solved urgently. Summary of the invention

[0004] In view of this, the object of the present invention is to provide a transaction processing method, apparatus, device and storage medium, which can improve the efficiency of transaction processing. The specific scheme is as follows:

[0005] In a first aspect, the present application provides a transaction processing method, comprising:

[0006] In a distributed system, the target transaction processing cycle is determined using the initial transaction processing cycle determined based on the epoch mechanism, the system load, the number of pending transactions, the network status, and the resource utilization status of the distributed nodes;

[0007] Allocating the pending transaction to the target transaction processing cycle, and controlling the start of the target transaction processing cycle, so that the distributed node asynchronously processes the pending transaction within the target transaction processing cycle based on the multi-version concurrency control method and the timestamp sorting protocol, and obtains the processed transaction;

[0008] Acquire the transaction processing status of the distributed node, and determine the to-be-committed transaction and the to-be-rolled-back transaction from the processed transaction based on the transaction processing status;

[0009] When the target transaction processing cycle ends, the transaction to be committed is committed, and the transaction to be rolled back is reallocated to the target transaction processing cycle for processing.

[0010] Optionally, determining the target transaction processing cycle based on the initial transaction processing cycle determined by the epoch mechanism, system load, number of transactions to be processed, network condition, and resource utilization status of distributed nodes includes:

[0011] Determine a dynamic attenuation coefficient based on the real-time fluctuation value of the system load, and use the dynamic attenuation coefficient to adjust the cycle length of the initial transaction processing cycle to complete the first length adjustment operation;

[0012] When the network delay value of the network condition is greater than the preset delay threshold, smooth the network delay using the sliding window algorithm, and adjust the cycle length of the initial transaction processing cycle based on the network delay peak within the sliding window during the smoothing operation to complete the second length adjustment operation;

[0013] If the number of transactions to be processed is greater than the first preset quantity threshold, adjust the cycle length of the initial transaction processing cycle based on the number of transactions to be processed to complete the third length adjustment operation;

[0014] Adjust the upper and lower limits of the cycle length of the initial transaction processing cycle based on the CPU utilization rate, memory occupancy rate, and disk I / O throughput of the distributed node to complete the fourth length adjustment operation;

[0015] Determine the target transaction processing cycle based on the first length adjustment operation, the second length adjustment operation, the third length adjustment operation, and the fourth length adjustment operation.

[0016] Optionally, allocating the transactions to be processed to the target transaction processing cycle includes:

[0017] Judge the transaction priority of the transaction to be processed and obtain the judgment result;

[0018] If the judgment result indicates that the transaction priority of the current transaction to be processed meets the preset low priority condition, allocate the current transaction to be processed to the first target transaction processing cycle;

[0019] If the judgment result indicates that the transaction priority of the current transaction to be processed meets the preset high priority condition, allocate the current transaction to be processed to the second target transaction processing cycle;

[0020] Wherein, the transaction processing start time of the first target transaction processing cycle is later than that of the second target transaction processing cycle.

[0021] Optionally, controlling the start of the target transaction processing cycle includes:

[0022] Monitor the status of the transaction queue where the transaction to be processed is located and obtain the monitoring result;

[0023] When the monitoring result indicates that the number of transaction backlogs in the transaction queue is greater than the second preset quantity threshold, trigger the start operation of the target transaction processing cycle;

[0024] If there is a to-be-processed transaction in the transaction queue whose transaction priority meets the preset high-priority condition and the waiting time is greater than the preset waiting threshold, directly trigger the start operation of the target transaction processing cycle.

[0025] Optionally, during the asynchronous processing of the to-be-processed transactions in the target transaction processing cycle based on the multi-version concurrency control method and the timestamp ordering protocol, it includes:

[0026] Allocate version identifiers for the to-be-processed transactions in the target transaction processing cycle based on the multi-version concurrency control method, so as to use the version identifiers for conflict detection;

[0027] Allocate timestamps for the to-be-processed transactions in the target transaction processing cycle based on the timestamp ordering protocol and the processing start time of the to-be-processed transactions, so as to use the timestamps for conflict detection.

[0028] Optionally, determining the to-be-committed transactions and the to-be-rolled-back transactions from the processed transactions based on the transaction processing status includes:

[0029] Obtain transaction version information and transaction timestamp information from the transaction processing status;

[0030] Among them, the transaction version information includes the version identifier information before transaction processing and the version identifier information after transaction processing, and the transaction timestamp information includes the timestamp information before transaction processing and the timestamp information after transaction processing;

[0031] Determine the transactions with version conflicts from the processed transactions based on the transaction version information, and add the transactions with version conflicts to the to-be-rolled-back transactions;

[0032] Determine the transactions with timestamp conflicts from the processed transactions based on the transaction timestamp information, and add the transactions with timestamp conflicts to the to-be-rolled-back transactions.

[0033] Optionally, submitting the to-be-committed transactions and reallocating the to-be-rolled-back transactions to the target transaction processing cycle for processing includes:

[0034] Batch submit the to-be-committed transactions through a distributed transaction coordinator;

[0035] In the transaction to be rolled back, extract the operation sequence and the dependency graph from the transaction log, and determine the rollback transactions without dependencies and the rollback transactions with dependencies from the dependency graph;

[0036] Wherein, the operation sequence is a set of read and write operation steps of the transaction to be rolled back on data items, and the dependency graph represents the data access dependency relationship between the transactions to be rolled back;

[0037] Randomly allocate the rollback transactions without dependencies to the target transaction processing cycle, and allocate the transactions with dependencies to the transaction processing cycles with a continuous relationship in the target transaction processing cycle.

[0038] In a second aspect, the present application provides a transaction processing device, including:

[0039] A cycle determination module, configured to determine a target transaction processing cycle in a distributed system by using an initial transaction processing cycle determined based on the epoch mechanism, system load, the number of transactions to be processed, network conditions, and the resource utilization status of distributed nodes;

[0040] A transaction execution module, configured to allocate the transactions to be processed to the target transaction processing cycle, and control the start of the target transaction processing cycle, so that the distributed nodes asynchronously process the transactions to be processed within the target transaction processing cycle based on the multi-version concurrency control method and the timestamp sorting protocol, and obtain the processed transactions;

[0041] A transaction classification module, configured to obtain the transaction processing status of the distributed nodes, and determine the transactions to be committed and the transactions to be rolled back from the processed transactions based on the transaction processing status;

[0042] A transaction submission module, configured to, when the target transaction processing cycle ends, commit the transactions to be committed, and re-allocate the transactions to be rolled back to the target transaction processing cycle for processing.

[0043] In a third aspect, the present application provides an electronic device, including:

[0044] A memory, configured to store a computer program;

[0045] A processor, configured to process the computer program to implement the foregoing transaction processing method.

[0046] In a fourth aspect, the present application provides a computer-readable storage medium, configured to store a computer program; wherein, when the computer program is processed by a processor, the foregoing transaction processing method is implemented.

[0047] In this application, in a distributed system, the target transaction processing cycle is determined by using the initial transaction processing cycle, system load, number of transactions to be processed, network conditions, and resource utilization status of distributed nodes based on the epoch mechanism; the transactions to be processed are allocated to the target transaction processing cycle, and the start of the target transaction processing cycle is controlled, so that the distributed nodes asynchronously process the transactions to be processed within the target transaction processing cycle based on the multi-version concurrency control method and the timestamp ordering protocol, and obtain the processed transactions; the transaction processing status of the distributed nodes is obtained, and the transactions to be committed and the transactions to be rolled back are determined from the processed transactions based on the transaction processing status; when the target transaction processing cycle ends, the transactions to be committed are committed, and the transactions to be rolled back are re-allocated to the target transaction processing cycle for processing. As can be seen from the above, in a distributed system, the initial transaction processing cycle, system load, number of transactions to be processed, network conditions, and resource utilization situation of distributed nodes determined based on the epoch mechanism are comprehensively considered to determine the target transaction processing cycle. After determining the target transaction processing cycle, the transactions to be processed are allocated to this cycle, and at the same time, the start of the target transaction processing cycle is controlled. The purpose of this is to enable the distributed nodes to asynchronously process the transactions to be processed within the target transaction processing cycle according to the multi-version concurrency control method and the timestamp ordering protocol, and thus obtain the processed transactions. Then, the system obtains the transaction processing status of the distributed nodes, and based on this status, selects the transactions to be committed and the transactions to be rolled back from the processed transactions. When the target transaction processing cycle ends, the transactions to be committed are committed. For the transactions to be rolled back, the system re-allocates them to the target transaction processing cycle for processing again. In this way, this application can improve the efficiency of transaction processing and enhance the user experience. Description of the Drawings

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained according to the provided drawings without creative efforts.

[0049] Figure 1 It is a flowchart of a transaction processing method disclosed in this application;

[0050] Figure 2 It is a flowchart of a specific transaction processing method disclosed in this application;

[0051] Figure 3 It is a schematic structural diagram of a transaction processing device disclosed in this application;

[0052] Figure 4 A structural diagram of an electronic device disclosed in this application. Specific implementation manners

[0053] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0054] Currently, with the continuous expansion of the scale of distributed systems and the sharp increase in the volume of business, the traditional transaction processing mechanism faces serious performance bottlenecks when processing a large number of transactions. In traditional synchronous submission methods such as two-phase commit or three-phase commit, the transaction coordinator needs to wait for the responses of all participants, and a large amount of synchronous waiting time will be generated during this process. Especially in large-scale distributed systems, there are a large number of nodes, and problems such as network latency and node failures frequently occur, further exacerbating this waiting situation, resulting in a decline in the overall performance of the system and thus reducing the efficiency of transaction processing. For this reason, this application provides a transaction processing method, device, equipment, and storage medium, which can improve the efficiency of transaction processing.

[0055] See Figure 1 As shown, an embodiment of the present invention discloses a transaction processing method, including:

[0056] Step S11: In a distributed system, determine a target transaction processing cycle by using the initial transaction processing cycle determined based on the epoch mechanism, system load, number of transactions to be processed, network status, and resource utilization status of distributed nodes.

[0057] In this embodiment, in a distributed system, the efficiency and stability of transaction processing are closely related to the determination of the transaction processing cycle. This application uses the initial transaction processing cycle determined based on the epoch mechanism as the basis, and comprehensively considers multiple factors such as system load, number of transactions to be processed, network status, and resource utilization status of distributed nodes, so as to determine the target transaction processing cycle.

[0058] First, regarding the system load, the fluctuations of the system load can be monitored in real time. By calculating the real-time fluctuation value of the system load, the dynamic attenuation coefficient is then determined. This dynamic attenuation coefficient will be used to adjust the cycle length of the initial transaction processing cycle, that is, to complete the first length adjustment operation. When the system load is high, the dynamic attenuation coefficient will appropriately shorten the length of the initial transaction processing cycle, so that transactions can be scheduled more quickly and the response speed of transaction processing can be improved. When the system load is low, the dynamic attenuation coefficient will appropriately extend the cycle length to reduce unnecessary scheduling overhead.

[0059] Secondly, the network condition also has a certain impact on transaction processing. When the network delay value of the network condition is greater than the preset delay threshold, it indicates that the current network environment may have an adverse impact on transaction processing. At this time, the sliding window algorithm can be used to smooth the network delay. Then, based on the network delay peak value within the sliding window during the smoothing operation, the cycle length of the initial transaction processing cycle is adjusted to complete the second length adjustment operation. The purpose of this is to ensure that transaction processing can be more stable in a high-delay or unstable network environment and reduce problems such as transaction conflicts or commit delays caused by network delays.

[0060] Furthermore, the number of transactions to be processed is also one of the key factors in determining the target transaction processing cycle. If the number of transactions to be processed is greater than the first preset quantity threshold, it indicates that the current transaction volume is large. In this case, the cycle length of the initial transaction processing cycle can be adjusted based on the number of transactions to be processed to complete the third length adjustment operation. That is, if the transaction volume is large and complex, appropriately extending the cycle length can ensure that transactions have enough time to be fully executed. Conversely, in the case of a low transaction volume, the cycle length can be shortened.

[0061] In addition, regarding the resource utilization status of distributed nodes. Specifically, the resource metrics such as the CPU utilization rate, memory occupancy rate, and disk I / O throughput of distributed nodes can be monitored in real time. Based on these metrics, the upper and lower limits of the cycle length of the initial transaction processing cycle are adjusted to complete the fourth length adjustment operation. This can ensure that during the transaction scheduling process, the resources of the nodes can be effectively allocated and avoid performance problems caused by excessive resource occupation or uneven distribution.

[0062] Finally, by integrating the above first length adjustment operation, second length adjustment operation, third length adjustment operation, and fourth length adjustment operation, the target transaction processing cycle can be determined more accurately, so that the target transaction processing cycle can better adapt to the actual operation situation of the distributed system.

[0063] Step S12: Allocate the to-be-processed transaction to the target transaction processing cycle and control the start of the target transaction processing cycle, so that the distributed node asynchronously processes the to-be-processed transaction within the target transaction processing cycle based on the multi-version concurrency control method and the timestamp ordering protocol, and obtains the processed transaction.

[0064] In a specific embodiment, in order to process the to-be-processed transaction more efficiently, it is necessary to judge the priority of the transaction. First, it is necessary to judge the transaction priority of the to-be-processed transaction and obtain the judgment result. The judgment of transaction priority can be based on various factors, such as the business type of the transaction, the impact on the overall system performance, the urgency of the transaction, etc.

[0065] After obtaining the judgment result, if the judgment result indicates that the transaction priority of the current to-be-processed transaction meets the preset low-priority condition, the current to-be-processed transaction will be allocated to the first target transaction processing cycle. The preset low-priority condition may include situations such as lower real-time requirements for the transaction, longer processing time, and larger occupation of system resources. For example, some batch data processing tasks, which usually do not need to be completed immediately and may consume a large amount of system resources, may be judged as low-priority transactions. Allocating low-priority transactions to the first target transaction processing cycle, and the start time of transaction processing in this cycle is later than that of the second target transaction processing cycle. The purpose of doing this is to ensure that high-priority transactions can be processed in a timely manner under sufficient resources and stable environment, and avoid low-priority transactions interfering with high-priority transactions, so as to ensure the response speed and processing efficiency of the system for key services.

[0066] On the contrary, if the judgment result indicates that the transaction priority of the current to-be-processed transaction meets the preset high-priority condition, it will be allocated to the second target transaction processing cycle. High-priority transactions often have higher real-time requirements, and the timely processing of these transactions is crucial for the normal operation of the business and the stability of the system. Allocating high-priority transactions to the second target transaction processing cycle with an earlier start can ensure that the system has sufficient resources and a good environment when processing these transactions, reduce the influence of other transactions, and thus ensure that high-priority transactions can be completed quickly.

[0067] In another specific embodiment, in order to reasonably control the start of the target transaction processing cycle, it is necessary to monitor the status of the transaction queue where the to-be-processed transaction is located and obtain the monitoring result. Among them, the transaction queue is a buffer for storing to-be-processed transactions, and its status can reflect the backlog of current to-be-processed transactions.

[0068] Analyze the obtained monitoring results. When the monitoring results indicate that the number of transaction backlogs in the transaction queue is greater than the second preset quantity threshold, it means that the pending transactions in the transaction queue have accumulated to a certain extent. If not processed in time, it may lead to the problem of transaction processing delay. At this time, it is necessary to trigger the start operation of the target transaction processing cycle. Among them, the second preset quantity threshold is preset according to the processing capacity and performance requirements of the system, which can ensure that when the transaction backlog reaches a certain level, the system can start the transaction processing cycle in time to avoid the negative impact on the system caused by excessive transaction backlogs.

[0069] In addition, if there are pending transactions in the transaction queue whose transaction priorities meet the preset high-priority conditions and the waiting time is greater than the preset waiting threshold, then the start operation of the target transaction processing cycle will be directly triggered. This is to ensure that high-priority transactions can be processed within a reasonable time and to avoid affecting the normal operation of the business due to excessive waiting time.

[0070] Step S13: Obtain the transaction processing status of the distributed node, and determine the transactions to be committed and the transactions to be rolled back from the processed transactions based on the transaction processing status.

[0071] In this embodiment, first obtain the transaction version information and transaction timestamp information from the transaction processing status of the distributed node. Specifically, the transaction version information includes the version identification information before transaction processing and the version identification information after transaction processing. That is to say, each transaction will have an initial version identification when it starts to be processed, and new version identifications may be generated during the processing. By comparing the version identifications before and after processing, it is possible to understand the degree of data modification by the transaction and whether there are conflicts with other transactions.

[0072] Moreover, the transaction timestamp information includes the timestamp information before transaction processing and the timestamp information after transaction processing. During the transaction processing, the timestamp can reflect the start time and execution progress of the transaction, which can help to judge the sequence and concurrency relationship between transactions.

[0073] In a specific implementation manner, based on the obtained transaction version information, start to find the transactions with version conflicts from the processed transactions. In a multi-version concurrency control environment, multiple transactions may operate on the same data simultaneously. If a transaction, when submitting, finds that its version identification after processing is inconsistent with the data version modified by other transactions, it means that a version conflict has occurred.

[0074] For example, transaction A and transaction B operate on data X simultaneously. Transaction A first obtains version V1 of data X and modifies it, and transaction B also obtains version V1 and makes different modifications. When transaction A is about to commit, it is found that the version of data X has become version V2 modified by transaction B, and at this time, a version conflict occurs. Then, such transactions with version conflicts will be added to the list of transactions to be rolled back.

[0075] In another specific implementation, the transaction timestamp information provides another dimension for conflict detection. Specifically, based on the transaction timestamp information, transactions with timestamp conflicts will be determined from the processed transactions. According to the timestamp sorting protocol, transactions should be executed in the order of their timestamps. If the timestamp of a transaction indicates that it should be executed after another transaction, but it modifies the same data in advance, a timestamp conflict will occur.

[0076] For example, the timestamp of transaction C is earlier than that of transaction D, but transaction D completes the modification of data Y first. When transaction C commits, it will be found that the state of data Y is inconsistent with the state that should be obtained according to the timestamp order, thus determining that there is a timestamp conflict. Then, such transactions with timestamp conflicts will be added to the list of transactions to be rolled back.

[0077] After determining the transactions to be rolled back, the remaining processed transactions are the transactions to be committed. These transactions have not detected conflicts in terms of version information and timestamp information, indicating that their execution processes comply with the system's concurrency control rules and can be safely committed.

[0078] Step S14, when the target transaction processing cycle ends, commit the transactions to be committed, and re - allocate the transactions to be rolled back to the target transaction processing cycle for processing.

[0079] When the target transaction processing cycle ends, the distributed transaction coordinator will batch - commit the transactions to be committed. Moreover, by batch - committing the transactions to be committed, the system's scheduling overhead and the number of network communications can be significantly reduced. Compared with committing transactions one by one, batch - committing can integrate the commit requests of multiple transactions and send them to each relevant distributed node at one time. Such a method can improve the performance of the entire system.

[0080] During the batch - commit process, the distributed transaction coordinator can communicate with each distributed node to ensure that all nodes are ready to receive and execute the commit operation. Once all nodes confirm that they are ready, the coordinator will send a batch - commit instruction, and each node will complete the commit of the transaction according to the instruction and save the execution result of the transaction to the local storage.

[0081] For the transactions to be rolled back, certain classification is required to reallocate and process them. First, extract the operation sequence and dependency graph in the transaction log from the transactions to be rolled back. Among them, the operation sequence records the set of read and write operation steps of the transactions to be rolled back on data items, which describes the specific operation process of the transactions on data during execution. The dependency graph represents the data access dependencies between the transactions to be rolled back, which reflects the associations generated between different transactions due to access to the same data.

[0082] Specifically, by analyzing the operation sequence and dependency graph, the transactions to be rolled back can be classified into independent rollback transactions and dependent rollback transactions. Among them, independent rollback transactions refer to those transactions that have no data access dependencies on other transactions during execution. Their execution results will not be affected by other transactions, nor will they affect the execution of other transactions. Dependent rollback transactions refer to those transactions that have data access dependencies on other transactions. Their execution results may be affected by other transactions, or they may affect the execution of other transactions.

[0083] According to the classification results, the independent rollback transactions will be randomly allocated to the target transaction processing cycles. Since independent rollback transactions are independent of each other, random allocation can make full use of the idle resources within the target transaction processing cycles, improve the parallelism of transaction processing. At the same time, random allocation can also avoid overloading of certain nodes and achieve balanced allocation of resources.

[0084] For the dependent rollback transactions, they will be allocated to the transaction processing cycles with consecutive relationships in the target transaction processing cycles. This is because there are data access dependencies between dependent rollback transactions, and they need to be executed in a certain order sequentially to ensure data consistency. Therefore, allocating them to the target transaction processing cycles with consecutive relationships can ensure that these transactions can be executed sequentially in the order of the dependency relationships during execution, avoiding data inconsistency problems caused by disordered order.

[0085] As can be seen above, in a distributed system, the target transaction processing cycle is determined by comprehensively considering the initial transaction processing cycle determined based on the epoch mechanism, the system load, the number of transactions to be processed, the network status, and the resource utilization of distributed nodes. After determining the target transaction processing cycle, the transactions to be processed are allocated to this cycle, and at the same time, the start of the target transaction processing cycle is controlled. The purpose of this is to enable distributed nodes to asynchronously process the transactions to be processed within the target transaction processing cycle according to the multi-version concurrency control method and the timestamp ordering protocol, and then obtain the processed transactions. After that, the system obtains the transaction processing status of the distributed nodes, and based on this status, filters out the transactions to be committed and the transactions to be rolled back from the processed transactions. When the target transaction processing cycle ends, the transactions to be committed are committed. For the transactions to be rolled back, the system reallocates them to the target transaction processing cycle for processing again. In this way, this application can improve the efficiency of transaction processing and enhance the user experience.

[0086] See Figure 2 As shown, in order to efficiently perform conflict detection on the transactions to be processed in a distributed system, an embodiment of this application further provides a specific transaction processing method, including:

[0087] Step S21: Start the target transaction processing cycle.

[0088] In this embodiment, the target transaction processing cycle is started so that the distributed nodes can asynchronously process the transactions to be processed according to the multi-version concurrency control method and the timestamp ordering protocol.

[0089] Step S22: During the target transaction processing cycle, the distributed nodes allocate version identifiers to the transactions to be processed within the target transaction processing cycle based on the multi-version concurrency control method, so as to perform conflict detection using the version identifiers.

[0090] In this embodiment, the distributed nodes allocate version identifiers to the transactions to be processed within the target transaction processing cycle based on the multi-version concurrency control method. During the execution of each transaction, a new version is created instead of directly overwriting the old data version. Specifically, the version identifier is the unique identifier of the transaction, which records the modification history and status of the transaction to the data. Through the version identifier, lock-free reads can be achieved between multiple concurrent transactions, allowing different transactions to access different versions of the same data. Correspondingly, when a transaction is ready to commit, it will determine whether there is a conflict with other transactions based on the version identifier. If the versions are the same, it means there is no conflict and the transaction can continue to execute. If there is a version conflict, it needs to be processed through a conflict detection mechanism, such as rolling back the conflicting transaction and rescheduling the execution.

[0091] Step S23: Based on the timestamp ordering protocol and the processing start time of the transaction to be processed, assign a timestamp to the transaction to be processed within the target transaction processing cycle, so as to perform conflict detection using the timestamp.

[0092] In this embodiment, the distributed node, based on the timestamp ordering protocol, assigns a timestamp to the transaction to be processed within the target transaction processing cycle according to the processing start time of the transaction to be processed. The timestamp is the time identifier when the transaction starts, and it provides a unique time order for each transaction. Before the transaction is committed, the system detects whether there are conflicts between transactions according to the timestamp ordering mechanism.

[0093] Therefore, during the transaction processing, the multi-version concurrency control method and the timestamp ordering protocol cooperate with each other to jointly ensure the concurrent processing of transactions and data consistency in the distributed system. The multi-version concurrency control method realizes the multi-version management of data through version identifiers, improving the concurrency performance, while the timestamp ordering protocol provides sequential control for transactions through timestamps, avoiding data conflicts. The combination of the two enables the distributed system to efficiently and stably process transactions in a high-concurrency environment, providing strong support for the development of distributed applications.

[0094] In addition, to further optimize the transaction processing process, the multi-version concurrency control method and the timestamp ordering protocol can also be dynamically adjusted. For example, according to the system load situation and the concurrency degree of transactions, adjust the generation strategy of version identifiers and the allocation algorithm of timestamps to improve the adaptability and performance of the system. At the same time, the system can also optimize the conflict detection mechanism, such as adopting more efficient algorithms and data structures, reducing the time overhead of conflict detection, and improving the efficiency of transaction processing.

[0095] As can be seen from the above, in the distributed system of this application, the target transaction processing cycle is first started, and the distributed node asynchronously processes transactions based on the multi-version concurrency control method and the timestamp ordering protocol. When processing transactions, the multi-version concurrency control method is used to assign version identifiers to transactions for conflict detection, and timestamps are assigned based on the timestamp ordering protocol according to the transaction start time to detect conflicts. In this way, this application can efficiently and stably process transactions in a high-concurrency environment, effectively detect conflicts, and ensure data consistency.

[0096] Correspondingly, as shown in Figure 3 this application embodiment provides a transaction processing apparatus, including:

[0097] A cycle determination module 11, configured to determine a target transaction processing cycle in a distributed system by using an initial transaction processing cycle determined based on the epoch mechanism, system load, the number of transactions to be processed, network conditions, and the resource utilization status of distributed nodes;

[0098] A transaction execution module 12, configured to allocate the to-be-processed transaction to the target transaction processing cycle, and control the start of the target transaction processing cycle, so that the distributed node asynchronously processes the to-be-processed transaction within the target transaction processing cycle based on the multi-version concurrency control method and the timestamp ordering protocol, and obtains the processed transaction;

[0099] A transaction classification module 13, configured to obtain the transaction processing status of the distributed node, and determine the to-be-committed transaction and the to-be-rolled-back transaction from the processed transactions based on the transaction processing status;

[0100] A transaction submission module 14, configured to, when the target transaction processing cycle ends, commit the to-be-committed transaction, and re-allocate the to-be-rolled-back transaction to the target transaction processing cycle for processing.

[0101] As can be seen from the above, in a distributed system, the target transaction processing cycle is determined by comprehensively considering the initial transaction processing cycle determined based on the epoch mechanism, the system load condition, the number of to-be-processed transactions, the network condition, and the resource utilization situation of the distributed node. After determining the target transaction processing cycle, the to-be-processed transaction is allocated to this cycle, and at the same time, the start of the target transaction processing cycle is controlled. The purpose of this is to enable the distributed node to asynchronously process the to-be-processed transaction within the target transaction processing cycle according to the multi-version concurrency control method and the timestamp ordering protocol, so as to obtain the processed transaction. Then, the system obtains the transaction processing status of the distributed node, and based on this status, filters out the to-be-committed transaction and the to-be-rolled-back transaction from the processed transactions. When the target transaction processing cycle ends, the to-be-committed transaction is committed. For the to-be-rolled-back transactions, the system re-allocates them to the target transaction processing cycle for processing again. In this way, the present application can improve the efficiency of transaction processing and enhance the user experience.

[0102] In some specific embodiments, the cycle determination module 11 specifically includes:

[0103] A first length adjustment unit, configured to determine a dynamic decay coefficient based on the real-time fluctuation value of the system load, and use the dynamic decay coefficient to adjust the cycle length of the initial transaction processing cycle to complete the first length adjustment operation;

[0104] A second length adjustment unit, configured to, when the network delay value of the network condition is greater than a preset delay threshold, smooth the network delay by using a sliding window algorithm, and adjust the cycle length of the initial transaction processing cycle based on the network delay peak value within the sliding window during the smoothing operation to complete the second length adjustment operation;

[0105] A third length adjustment unit, configured to, if the number of transactions to be processed is greater than a first preset number threshold, adjust the cycle length of the initial transaction processing cycle based on the number of transactions to be processed, so as to complete a third length adjustment operation;

[0106] A fourth length adjustment unit, configured to adjust the upper and lower limits of the cycle length of the initial transaction processing cycle based on the CPU utilization rate, memory occupancy rate, and disk I / O throughput of a distributed node, so as to complete a fourth length adjustment operation;

[0107] A cycle determination unit, configured to determine a target transaction processing cycle based on the first length adjustment operation, the second length adjustment operation, the third length adjustment operation, and the fourth length adjustment operation.

[0108] In some specific embodiments, the transaction execution module 12 specifically includes:

[0109] A priority judgment unit, configured to judge the transaction priority of the transaction to be processed and obtain a judgment result;

[0110] A first transaction allocation unit, configured to, if the judgment result indicates that the transaction priority of the current transaction to be processed meets a preset low priority condition, allocate the current transaction to a first target transaction processing cycle;

[0111] A second transaction allocation unit, configured to, if the judgment result indicates that the transaction priority of the current transaction to be processed meets a preset high priority condition, allocate the current transaction to a second target transaction processing cycle;

[0112] Wherein, the transaction processing start time of the first target transaction processing cycle is later than that of the second target transaction processing cycle.

[0113] In some specific embodiments, the transaction execution module 12 specifically includes:

[0114] A status monitoring unit, configured to monitor the status of the transaction queue where the transaction to be processed is located and obtain a monitoring result;

[0115] A first cycle start unit, configured to, when the monitoring result indicates that the number of backlogged transactions in the transaction queue is greater than a second preset number threshold, trigger a start operation of the target transaction processing cycle;

[0116] A second cycle start unit, configured to, if there is a transaction to be processed in the transaction queue whose transaction priority meets the preset high priority condition and the waiting time is greater than a preset waiting threshold, directly trigger the start operation of the target transaction processing cycle.

[0117] In some specific embodiments, the transaction execution module 12 specifically includes:

[0118] An identifier allocation unit, configured to allocate a version identifier for the to-be-processed transaction within the target transaction processing cycle based on the multi-version concurrency control method, so as to perform conflict detection by using the version identifier;

[0119] A timestamp allocation unit, configured to allocate a timestamp for the to-be-processed transaction within the target transaction processing cycle based on the timestamp ordering protocol and the processing start time of the to-be-processed transaction, so as to perform conflict detection by using the timestamp.

[0120] In some specific embodiments, the transaction classification module 13 specifically includes:

[0121] An information acquisition unit, configured to acquire transaction version information and transaction timestamp information from the transaction processing status;

[0122] Wherein, the transaction version information includes version identifier information before transaction processing and version identifier information after transaction processing, and the transaction timestamp information includes timestamp information before transaction processing and timestamp information after transaction processing;

[0123] A first transaction determination unit, configured to determine, from the processed transactions, the transactions with version conflicts based on the transaction version information, and add the transactions with version conflicts to the to-be-rolled-back transactions;

[0124] A second transaction determination unit, configured to determine, from the processed transactions, the transactions with timestamp conflicts based on the transaction timestamp information, and add the transactions with timestamp conflicts to the to-be-rolled-back transactions.

[0125] In some specific embodiments, the transaction submission module 14 specifically includes:

[0126] A transaction submission unit, configured to batch-submit the to-be-submitted transactions through a distributed transaction coordinator;

[0127] A transaction classification unit, configured to extract the operation sequence and the dependency graph in the transaction log from the to-be-rolled-back transactions, and determine the rollback transactions without dependencies and the rollback transactions with dependencies from the dependency graph;

[0128] Wherein, the operation sequence is a set of read / write operation steps of the to-be-rolled-back transactions on data items, and the dependency graph represents the data access dependency relationship between the to-be-rolled-back transactions;

[0129] A third transaction allocation unit, configured to randomly allocate the independent rollback transactions to the target transaction processing cycle, and allocate the dependent transactions to the transaction processing cycles with a continuous relationship in the target transaction processing cycle.

[0130] Furthermore, an embodiment of the present application also discloses an electronic device. Figure 4 It is a structural diagram of an electronic device 20 shown according to an exemplary embodiment. The content in the figure should not be considered as any limitation to the scope of use of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. Among them, the memory 22 is used to store a computer program, and the computer program is loaded and processed by the processor 21 to implement the relevant steps in the transaction processing method disclosed in any of the foregoing embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0131] In this embodiment, the power supply 23 is used to provide operating voltages for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows is any communication protocol applicable to the technical solution of the present application, and specific limitations are not imposed here; the input / output interface 25 is used to obtain external input data or output data to the outside, and its specific interface type can be selected according to specific application needs, and no specific limitations are made here.

[0132] In addition, as a carrier for resource storage, the memory 22 may be a read-only memory, a random access memory, a magnetic disk, or an optical disc, etc. The resources stored thereon may include an operating system 221, a computer program 222, etc., and the storage method may be temporary storage or permanent storage.

[0133] Among them, the operating system 221 is used to manage and control each hardware device and the computer program 222 on the electronic device 20, and it may be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program that can be used to complete the transaction processing method processed by the electronic device 20 disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program that can be used to complete other specific tasks.

[0134] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, the computer program, when processed by a processor, implements the foregoing disclosed transaction processing method. For the specific steps of this method, reference may be made to the corresponding content disclosed in the foregoing embodiments, and details are not described herein again.

[0135] In this specification, the various embodiments are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the embodiments, reference can be made to each other. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple. For the relevant parts, reference can be made to the description in the method section.

[0136] Those skilled in the art can further realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been generally described according to their functions in the above description. Whether these functions are processed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0137] The steps of the methods or algorithms described in combination with the embodiments disclosed herein can be directly implemented by hardware, software modules processed by a processor, or a combination of the two. The software modules can be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium well-known in the technical field.

[0138] Finally, it should also be noted that in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, article or device including the said element.

[0139] The above has introduced the technical solution provided by this application in detail. Specific examples are used herein to elaborate on the principle and implementation manner of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application; at the same time, for those of ordinary skill in the art, according to the idea of this application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to this application.

Claims

1. A transaction processing method, characterized in that, Including: In a distributed system, determining a target transaction processing period based on an initial transaction processing period determined by an epoch mechanism, system load, the number of transactions to be processed, network conditions, and the resource utilization status of distributed nodes; Allocating the transactions to be processed to the target transaction processing period and controlling the start of the target transaction processing period, so that the distributed nodes asynchronously process the transactions to be processed within the target transaction processing period based on the multi-version concurrency control method and the timestamp ordering protocol, and obtain processed transactions; Obtaining the transaction processing status of the distributed nodes and determining the transactions to be committed and the transactions to be rolled back from the processed transactions based on the transaction processing status; When the target transaction processing period ends, committing the transactions to be committed and reallocating the transactions to be rolled back to the target transaction processing period for processing.

2. The transaction processing method according to claim 1, characterized in that The determining the target transaction processing period by using the initial transaction processing period determined by the epoch mechanism, system load, the number of transactions to be processed, network conditions, and the resource utilization status of distributed nodes includes: Determining a dynamic decay coefficient based on the real-time fluctuation value of the system load, and using the dynamic decay coefficient to adjust the period length of the initial transaction processing period to complete the first length adjustment operation; When the network delay value of the network conditions is greater than a preset delay threshold, smoothing the network delay by using a sliding window algorithm, and adjusting the period length of the initial transaction processing period based on the network delay peak within the sliding window during the smoothing operation to complete the second length adjustment operation; If the number of transactions to be processed is greater than a first preset number threshold, adjusting the period length of the initial transaction processing period based on the number of transactions to be processed to complete the third length adjustment operation; Adjusting the upper and lower limits of the period length of the initial transaction processing period based on the CPU utilization rate, memory occupancy rate, and disk I / O throughput of the distributed nodes to complete the fourth length adjustment operation; Determining the target transaction processing period based on the first length adjustment operation, the second length adjustment operation, the third length adjustment operation, and the fourth length adjustment operation.

3. The transaction processing method according to claim 1, wherein The allocating the transactions to be processed to the target transaction processing period includes: Judging the transaction priority of the transactions to be processed and obtaining a judgment result; If the judgment result indicates that the transaction priority of the current transaction to be processed meets a preset low priority condition, allocating the current transaction to be processed to a first target transaction processing period; If the judgment result indicates that the transaction priority of the current transaction to be processed meets a preset high priority condition, allocating the current transaction to be processed to a second target transaction processing period; Wherein, the start time of the transaction processing in the first target transaction processing period is later than that in the second target transaction processing period.

4. The transaction processing method according to claim 3, wherein The controlling the start of the target transaction processing period includes: Monitoring the status of the transaction queue where the transactions to be processed are located and obtaining a monitoring result; When the monitoring result indicates that the number of transaction backlogs in the transaction queue is greater than the second preset quantity threshold, the start operation of the target transaction processing cycle is triggered; If there are pending transactions in the transaction queue whose transaction priorities meet the preset high-priority conditions and the waiting time is greater than the preset waiting threshold, the start operation of the target transaction processing cycle is directly triggered.

5. The transaction processing method according to claim 1, wherein During the process of asynchronously processing the pending transactions within the target transaction processing cycle based on the multi-version concurrency control method and the timestamp ordering protocol, it includes: Allocating version identifiers for the pending transactions within the target transaction processing cycle based on the multi-version concurrency control method, so as to perform conflict detection using the version identifiers; Allocating timestamps for the pending transactions within the target transaction processing cycle based on the timestamp ordering protocol and the processing start time of the pending transactions, so as to perform conflict detection using the timestamps.

6. The transaction processing method according to any one of claims 1 to 5, characterized in that Determining the transactions to be committed and the transactions to be rolled back from the processed transactions based on the transaction processing status, includes: Obtaining transaction version information and transaction timestamp information from the transaction processing status; Among them, the transaction version information includes the version identifier information before transaction processing and the version identifier information after transaction processing, and the transaction timestamp information includes the timestamp information before transaction processing and the timestamp information after transaction processing; Determining the transactions with version conflicts from the processed transactions based on the transaction version information, and adding the transactions with version conflicts to the transactions to be rolled back; Determining the transactions with timestamp conflicts from the processed transactions based on the transaction timestamp information, and adding the transactions with timestamp conflicts to the transactions to be rolled back.

7. The transaction processing method according to claim 1, characterized in that, Committing the transactions to be committed and reallocating the transactions to be rolled back to the target transaction processing cycle for processing, includes: Batch committing the transactions to be committed through a distributed transaction coordinator; In the transactions to be rolled back, extracting the operation sequence and the dependency graph in the transaction log, and determining the rollback transactions without dependencies and the rollback transactions with dependencies from the dependency graph; Among them, the operation sequence is a set of read and write operation steps of the transactions to be rolled back on data items, and the dependency graph represents the data access dependency relationship between the transactions to be rolled back; Randomly allocating the rollback transactions without dependencies to the target transaction processing cycle, and allocating the transactions with dependencies to the transaction processing cycles with consecutive relationships in the target transaction processing cycle.

8. A transaction processing device, characterized in that, Includes: A cycle determination module, which is used to determine the target transaction processing cycle in a distributed system by using the initial transaction processing cycle determined based on the epoch mechanism, the system load, the number of pending transactions, the network status, and the resource utilization status of distributed nodes; A transaction execution module, configured to allocate the to-be-processed transaction to the target transaction processing cycle and control the start of the target transaction processing cycle, so that the distributed node asynchronously processes the to-be-processed transaction within the target transaction processing cycle based on the multi-version concurrency control method and the timestamp ordering protocol, and obtains the processed transaction; A transaction classification module, configured to obtain the transaction processing status of the distributed node, and determine the to-be-committed transaction and the to-be-rolled-back transaction from the processed transactions based on the transaction processing status; A transaction submission module, configured to, when the target transaction processing cycle ends, submit the to-be-committed transaction and re-allocate the to-be-rolled-back transaction to the target transaction processing cycle for processing.

9. An electronic device, characterized in that, Comprising: A memory, configured to store a computer program; A processor, configured to process the computer program to implement the transaction processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, For storing a computer program; wherein, when the computer program is processed by the processor, the transaction processing method according to any one of claims 1 to 7 is implemented.

Images