Data operation sequencing methods and systems, and data operation submission methods

Abstract

This application discloses a data operation sequencing method and system. The method includes: responding to a client's broadcast request for a data operation, each log server determines a logical timestamp based on the type of the broadcast request and records the broadcast request and the logical timestamp of the data operation; and sequencing different data operations based on the broadcast requests and logical timestamps of different data operations recorded by each log server. This application decouples the replication logic and sequencing logic of data operations. The client's broadcast request for a data operation directly reaches each node of the write-ahead log system, implementing the replication logic. Each log server in the write-ahead log system asynchronously collects the broadcast requests and logical timestamps recorded by each log server to implement the sequencing logic of different data operations. Therefore, the write-ahead log system does not need to elect a master node as a single point of failure, which can improve the stability of client access and avoid service unavailability caused by master node failure triggering election.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a data operation sequencing method and system, and a data operation submission method. Background Technology

[0002] The Write-Ahead Log (WAL) mechanism, commonly used in distributed storage systems, can guarantee the atomicity and durability of data operations. The principle behind this is that data operations do not directly affect the storage system, but are first written to the write-ahead log. If the data operation fails, the operation record in the write-ahead log is ignored and the data operation is rolled back. If the data operation succeeds, the data operation recorded in the write-ahead log will be written to the storage system at a later time and the data operation will be committed.

[0003] Distributed storage systems ensure data reliability by storing data in multiple replicas. The WAL mechanism must not only guarantee the atomicity and durability of data operations, but also ensure the consistency of data across multiple replicas. Therefore, the WAL mechanism needs to order concurrent data operations from multiple clients, and then transfer the ordered data operations to the storage system. This allows the storage system's multiple replicas to execute the concurrent data operations from multiple clients in the ordered order, thus ensuring the consistency of data across multiple replicas. Summary of the Invention

[0004] The purpose of this application is to provide a data operation sequencing method and system, and a data operation submission method, in order to address the shortcomings of the prior art. This purpose is achieved through the following technical solutions.

[0005] The first aspect of this application proposes a data operation sequencing method applied to a write-ahead log system, the write-ahead log system comprising at least two log servers, the method comprising:

[0006] In response to a client initiating a broadcast request for a data operation, each of the log servers determines the logical timestamp of the data operation based on the type of the broadcast request, and records the broadcast request and the logical timestamp of the data operation; the type of the broadcast request is used to identify the number of hops in a request for a data operation initiated by the client;

[0007] Based on the broadcast requests and logical timestamps of different data operations recorded by each of the log servers, the different data operations are ordered.

[0008] The data manipulation ordering method described in the first aspect above has at least the following beneficial effects or advantages:

[0009] Since the write-ahead log system receives broadcast requests for data operations initiated by clients, each log server in the write-ahead log system can receive these broadcast requests. Each log server then records a logical timestamp for each data operation based on the type of broadcast request. Compared to existing solutions that rely on selecting a master node in the write-ahead log system to replicate data operations among other nodes and to order different data operations, resulting in single-point performance jitter, this application decouples the replication and ordering logic of data operations. The broadcast requests for data operations initiated by clients directly reach each node in the write-ahead log system, implementing the replication logic. Each log server in the write-ahead log system asynchronously collects the broadcast requests and logical timestamps recorded by each log server to implement the ordering logic for different data operations. Therefore, this application eliminates the need for a master node election as a single point of failure in the write-ahead log system, improving client access stability and avoiding service unavailability windows caused by master node failure triggering the election process.

[0010] A second aspect of this application proposes a data operation submission method applied to a client, the method comprising:

[0011] Send a first-hop type broadcast request for data operation to the write-ahead log system, and receive the first response returned by the write-ahead log system within a preset timeout period;

[0012] If the number of first responses reaches the number of log servers included in the write-ahead log system and the logical timestamp included in the first response meets the predetermined conditions, the data operation is determined to have been successfully submitted.

[0013] If the number of first responses does not reach the number of log servers but reaches the first target number, or if the logical timestamp contained in the first response does not meet the predetermined conditions, a target logical timestamp is determined based on the logical timestamp contained in the first response, and a second-hop type broadcast request is sent to the write-ahead log system using the target logical timestamp, and a second response returned by the write-ahead log system is received within the timeout period.

[0014] The data operation is considered successfully submitted once the number of second responses reaches the second target number.

[0015] The data operation submission method described in the second aspect above has at least the following beneficial effects or advantages:

[0016] The client actively uses a multi-write approach to broadcast a first-hop request for data operations to the write-ahead log system. If the number of responses returned by the write-ahead log system is exactly the number of nodes it contains, and the logical timestamps in the first responses also meet predetermined conditions, the data operation can be directly confirmed as successfully committed. For the client, the data operation is committed in one hop, reducing client request latency. However, if the first responses returned by the write-ahead log system do not meet the above criteria, but the number of responses reaches a certain target number, considering the possibility of some nodes failing, a target logical timestamp is further determined based on the logical timestamps of the first responses, and a second-hop request for data operations is broadcast to the write-ahead log system. If the number of responses returned by the write-ahead log system meets certain conditions, the data operation can be confirmed as successfully committed. Therefore, in most cases, the client can complete the data operation commit process in one hop, and in rare cases, two hops are required. Thus, the overall client request latency is very low.

[0017] A third aspect of this application proposes a data manipulation sequencing system, characterized in that the system comprises:

[0018] A write-ahead log system for performing the steps of the method described in the first aspect above;

[0019] A client for performing the steps of the method described in the second aspect above.

[0020] The fourth aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to perform the steps of the method as described in the first or second aspect above.

[0021] The fifth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to perform the steps of the method as described in the first or second aspect above. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0023] Figure 1 This is a schematic diagram illustrating an implementation architecture of a write-ahead log mechanism with master consensus according to an exemplary embodiment;

[0024] Figure 2 This is a schematic diagram illustrating an implementation architecture of a write-ahead log mechanism for a masterless consensus according to an exemplary embodiment;

[0025] Figure 3 This is a flowchart illustrating an embodiment of a data operation submission method according to an exemplary embodiment;

[0026] Figure 4 According to Figure 3 An embodiment illustrates an overall implementation flowchart of a data operation submission process;

[0027] Figure 5 This is a flowchart illustrating an embodiment of a data manipulation sequencing method according to an exemplary embodiment;

[0028] Figure 6 According to Figure 5 An embodiment illustrates a schematic diagram of displaying sequencing results on a logical timeline;

[0029] Figure 7 This is a flowchart illustrating the reentrancy implementation of a write-ahead log system when node reentrancy occurs, according to an exemplary embodiment.

[0030] Figure 8 This is a schematic diagram of the hardware structure of an electronic device according to an exemplary embodiment;

[0031] Figure 9 This is a schematic diagram illustrating the structure of a storage medium according to an exemplary embodiment. Detailed Implementation

[0032] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of systems and methods consistent with some aspects of this application as detailed in the appended claims.

[0033] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0034] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0035] As mentioned earlier, in a distributed storage system using a write-ahead log (WAP) mechanism, the WAP mechanism not only needs to ensure the atomicity and durability of data operations, but also needs to order concurrent data operations from multiple clients, and then transfer the ordered data operations to the storage system to ensure consistency of multi-replica data.

[0036] In existing technologies, a write-ahead log mechanism based on master consensus is used. First, a management system such as Zookeeper or ETCD is introduced into the write-ahead log system to elect a master node. The master node is responsible for replicating data operations among the remaining nodes and ordering different data operations. See also Figure 1 As shown, the client submits data operation requests to the master node of the write-ahead log system. The master node replicates the requests among the other nodes and reorders the requests according to the order in which they are received. After replication and reordering are completed, the master node replies to the client and then stores the replicated and ordered requests in the distributed storage system.

[0037] In practical applications, the above implementation scheme has several drawbacks. First, the master node becomes a single point of failure. If the master node fails, it will trigger a re-election process. The entire system will be unavailable until a new master node is elected. Furthermore, if the master node experiences performance fluctuations due to network or load issues, it will directly affect the latency and throughput of client data operations. Second, the data replication and reordering processes are both on the client's data submission path, which introduces at least two hops of time delay for the client to submit the data operation.

[0038] To address the aforementioned technical issues, this application proposes a write-ahead log mechanism with masterless consensus to decouple the replication logic from the ordering logic of data operations. (See [link to relevant documentation]). Figure 2As shown, for the replication logic, the client broadcasts a request for data operation to the write-ahead log system. Once the write-ahead log system confirms that it has performed the corresponding persistence operation, the request is considered successfully submitted. For the ordering logic, any node in the write-ahead log system asynchronously collects the persistence information recorded by other nodes during their persistence operations. Based on this persistence information, it orders multiple requests and stores the ordered requests in the distributed storage system. The distributed storage system then executes each request sequentially according to the ordered sequence. For example, the distributed storage system can be any system among block storage systems, file storage systems, and object storage systems.

[0039] Based on the masterless consensus write-ahead log mechanism described above, since there is no need to elect a master node as a single point of failure in the write-ahead log system, the single point of failure in master node performance is eliminated, improving client access stability and avoiding service unavailability windows caused by master node failure triggering the election process. Furthermore, when a client broadcasts a data operation request to the write-ahead log system, after confirming that the write-ahead log system has performed the corresponding persistence operation, the request is considered successfully submitted without waiting for the request reordering process to complete, thus reducing request submission latency for the client.

[0040] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings, focusing on the data operation sequencing on the write-ahead log system side and the data operation submission on the client side.

[0041] Example 1:

[0042] Figure 3 This is a flowchart illustrating an embodiment of a data operation submission method according to an exemplary embodiment, applied to the above. Figure 2 On the client side, such as Figure 3 As shown, the data operation submission method includes the following steps:

[0043] Step 301: Send a first-hop type broadcast request for the data operation to the write-ahead log system, and receive the first response returned by the write-ahead log system within the preset timeout period.

[0044] In this process, the client sends a broadcast request to the write-ahead logging system. All nodes on the write-ahead logging system receive this broadcast request, persist it, and then return a first response. The first-hop type indicates that the client initiates the first request for a data operation, and the broadcast request of the first-hop type contains the details of this data operation.

[0045] It is understandable that client-side data operations can be either write or read operations.

[0046] Furthermore, the timeout period is the threshold set by the client for request timeouts; the first response received within this timeout period is considered a normal reply. Normally, the number of first responses should match the number of log servers deployed in the write-ahead logging system.

[0047] It should be noted that the first response carries a logical timestamp assigned by the log server to the received broadcast request. The logical timestamp is assigned independently by each log server and is guaranteed to be monotonically increasing. Due to the differences in the logical clock timing on different log servers, the logical timestamps carried by different first responses may be the same or different.

[0048] Step 302: If the number of first responses reaches the number of log servers included in the write-ahead log system and the logical timestamps included in the first responses meet the predetermined conditions, the data operation is determined to have been successfully submitted.

[0049] If the number of first responses equals the number of log servers, it means that all log servers deployed in the write-ahead logging system have responded, and there are no faulty log server nodes in the write-ahead logging system. The predetermined condition is that the logical timestamps in the first responses must meet the corresponding requirements. If both the number of first responses and the logical timestamps included in the first responses meet the corresponding requirements, the client can directly confirm that the data operation has been successfully submitted. Thus, for the client, the data operation submission process can be completed in one hop, and the data operation can be considered to have been completed, resulting in a relatively high system response rate.

[0050] Optionally, the predefined condition can be that at least a third target number of first responses have a logical timestamp that is the maximum logical timestamp of all first responses, indicating that the timing of the local logical clocks of a certain number of log servers in the write-ahead log system is consistent.

[0051] In practice, the logical timestamp is represented by a fixed-length string, which can be divided into three parts from left to right: the highest bit, the high bit, and the low bit. Considering the individuality of the log server's logical clock, the low bit of different log server logical clocks may be inconsistent. Therefore, the predetermined condition is set to have at least a third target number of first responses whose high bit of the logical timestamp is the maximum value of the high bit of the logical timestamp in all first responses.

[0052] For example, assuming the logical timestamp is 128 bits long, bits 1-32 are used as the most significant part (epoch), bits 33-64 are used as the high-order part (H), and bits 65-128 are used as the low-order part (L).

[0053] Step 303: If the number of first responses does not reach the number of log servers but reaches the first target number, or if the logical timestamp contained in the first response does not meet the predetermined conditions, determine the target logical timestamp based on the logical timestamp contained in the first response, and send a second-hop type broadcast request to the write-ahead log system using the target logical timestamp, and receive the second response returned by the write-ahead log system within the timeout period.

[0054] If the number of initial responses is less than the number of log servers, it indicates that not all log servers deployed in the write-ahead logging system have responded, meaning there are faulty log server nodes. The first target number is a minimum response threshold set by the client based on the total number of log servers in the write-ahead logging system to ensure the fault tolerance performance of the write-ahead logging system.

[0055] For example, assuming the entire system is fault-tolerant, the total number of deployed log servers is N = 2F + 1, where F is the number of concurrent node failures that the system can tolerate. Then the first target number is M = F + 1, which means the number of responses is just over half of the total number of nodes.

[0056] In an optional embodiment, the process of determining the target logical timestamp can be achieved by sorting the logical timestamps contained in the first response in ascending order, and then using the logical timestamps with a sorting number equal to the first target number as the target logical timestamp.

[0057] Since the first target quantity is a response number that guarantees the system's fault tolerance performance, the target logical timestamp determined by the first target quantity is also a logical timestamp that guarantees the system's fault tolerance performance. The target logical timestamp is used by the client to indicate the unified sequence number of data operations to the write-ahead log system, ensuring that the write-ahead log system achieves correct ordering.

[0058] In one specific implementation, as described above, when sending a second-hop type broadcast request using the target logical timestamp, the high-order byte of the target logical timestamp can be boosted and added to the broadcast request to better ensure the correct ordering of the write-ahead log system.

[0059] Step 304: Once the number of second responses reaches the second target number, confirm that the data operation has been successfully submitted.

[0060] The second target number is also a response number set by the client to ensure the system's fault tolerance performance.

[0061] It is understandable that the number of the first target, the number of the second target, and the number of the third target described above may be equal or unequal.

[0062] It should be noted that, in order to ensure the synchronization and promotion of the logical timestamp of the local logical clock of the log server with the target logical timestamp, the client can also broadcast a synchronization request containing the promoted target logical timestamp to the write-ahead log system when sending a second-hop type broadcast request, so as to synchronize the local logical clock of each log server to the promoted target logical timestamp and maintain the consistency of the local logical clock of each log server.

[0063] Furthermore, in addition to considering that the number of second responses reaches the second target number, the client also needs to consider that the number of replies to the synchronization request also reaches the second target number before confirming that the data operation has been successfully submitted.

[0064] This completes the above. Figure 3 The data operation submission process shown involves the client actively using multiple writes to broadcast a first-hop request for the data operation to the write-ahead log system. If the number of responses returned by the write-ahead log system is exactly the number of nodes it contains, and the logical timestamps in the first responses also meet predetermined conditions, the data operation submission can be directly confirmed as successful. For the client, the data operation submission process is completed in one hop, reducing client request latency. However, if the first responses returned by the write-ahead log system do not meet the above judgment conditions, but the number of first responses reaches a certain target number, considering the possibility of partial node failures in the write-ahead log system, a target logical timestamp is further determined based on the logical timestamps of the first responses, and a second-hop request for the data operation is broadcast to the write-ahead log system. If the number of second responses returned by the write-ahead log system meets certain conditions, the data operation submission can be confirmed as successful. Therefore, it can be seen that in most cases, the client can complete the data operation submission process in one hop, and in rare cases, two hops are required. Thus, the overall client request latency is very small.

[0065] Based on the processes described in steps 301 to 304 above, the following is an overall implementation flow description for data operation submission. It is assumed that the number of log servers in the write-ahead log system is N = 2F + 1, and the number of the first target, second target, and third target are all M = F + 1. The most significant bit (epoch), the high-order bit (H), and the least significant bit (L) of the logical timestamp are described in [reference needed]. Figure 4 As shown, it includes the following steps:

[0066] Step 401: Send a first-hop type broadcast request for the data operation to the write-ahead log system and receive the first response within the timeout period.

[0067] Step 402: Determine if N first responses have been received. If yes, proceed to step 403; otherwise, proceed to step 405.

[0068] Step 403: Determine if at least M of the N first responses have a logical timestamp H equal to Hn. If so, proceed to step 404; otherwise, proceed to step 406.

[0069] Where Hn represents the maximum value of the high-order part of the logical timestamp in N first responses.

[0070] Step 404: Confirm that the data operation was successfully submitted.

[0071] Step 405: Determine if the number of first responses is greater than or equal to M. If yes, proceed to step 406; otherwise, proceed to step 410.

[0072] Step 406: Send a second-hop type broadcast request to the write-ahead log system and receive a second response within the timeout period.

[0073] Specifically, the broadcast request carries the enhanced target logical timestamp:<epoch,Hm+1,Lm> .

[0074] Step 407: Determine whether the number of second responses is greater than or equal to M. If yes, proceed to step 408; otherwise, proceed to step 410.

[0075] Step 408: Send a synchronization request to the write-ahead log system.

[0076] This synchronization request also carries the enhanced target logical timestamp:<epoch,Hm+1,Lm> .

[0077] Step 409: Determine whether the number of responses to the synchronization request is greater than or equal to M. If yes, proceed to step 404; otherwise, proceed to step 410.

[0078] Step 410: Determine that the data operation failed to submit.

[0079] It should be noted that this application does not restrict the order in which broadcast requests and synchronization requests of the second hop type are sent.

[0080] This completes the overall process of submitting a data operation.

[0081] Example 2:

[0082] Figure 5 This is a flowchart illustrating an embodiment of a data operation sequencing method according to an exemplary embodiment, applied to the above. Figure 2 On the write-ahead logging system side, this write-ahead logging system deploys at least two log servers, such as... Figure 5 As shown, this data manipulation ordering method includes the following steps:

[0083] Step 501: In response to a broadcast request for a data operation initiated by the client, each log server determines the logical timestamp of the data operation based on the type of the broadcast request, and records the broadcast request and the logical timestamp of the data operation.

[0084] The broadcast request type identifies the number of hops in a single data operation initiated by the client. As mentioned earlier, broadcast request types include first-hop type and second-hop type. The first-hop type identifies the client's first-hop request, and the second-hop type identifies the client's second-hop request. The process of the log server processing broadcast requests is the persistence operation.

[0085] After the broadcast request is sent to each log server, each log server independently determines the logical timestamp and records it.

[0086] In one optional embodiment, regarding the process of the log server determining the logical timestamp of a data operation, if the broadcast request is of the first hop type, the log server uses the current logical timestamp of the local logical clock as the logical timestamp of the data operation; if the broadcast request is of the second hop type, the log server uses the logical timestamp carried by the broadcast request as the logical timestamp of the data operation.

[0087] As mentioned earlier, the client carries a logical timestamp in the second-hop type broadcast request it sends. This logical timestamp is determined by the client based on the logical timestamps carried in the responses to the first-hop type broadcast request returned by each log server, that is, the client determines it based on the logical timestamp carried in the first response.

[0088] Therefore, if the log server receives a first-hop type broadcast request, it records the current logical timestamp of the local logical clock. Since each log server allocates its own logical time, the logical timestamps recorded by each log server may be the same or different. If the log server receives a second-hop type broadcast request, it records the logical timestamp carried by the client in the broadcast request. Each log server records the same logical timestamp. Specifically, the broadcast request carries the boosted logical timestamp.

[0089] It should be noted that, as mentioned earlier, if the client triggers a second-hop type broadcast request, it will also trigger a logical timestamp synchronization request at the same time. Therefore, each log server in the write-ahead log system will receive the synchronization request broadcast by the client, and thus each log server will synchronize the logical timestamp of its local logical clock to the logical timestamp carried in the synchronization request, so as to maintain the consistency of logical clock timing among different log servers as much as possible.

[0090] Step 502: Sequence the different data operations based on the broadcast requests and logical timestamps of the different data operations recorded by each log server.

[0091] Among these, different data operations are concurrent operations performed by multiple clients.

[0092] In one optional embodiment, a master node is elected among the various log servers, and the master node obtains broadcast requests and logical timestamps of different data operations from the various log servers. Then, the different data operations are ordered according to the obtained broadcast requests and logical timestamps.

[0093] Optionally, as mentioned above, the system can tolerate the failure of at least F log server nodes, so the master node can collect broadcast requests and logical timestamps of different data operations from M=NF log servers.

[0094] It is important to note that although this embodiment also requires the election of a master node, the data operation request submission process and the master node ordering process are decoupled in this embodiment. That is to say, the master node election and ordering process are unknown to the client. Ordering is an asynchronous task, so the master node election and ordering process is not on the client's data operation submission path.

[0095] Optionally, the master node election for ordered tasks in the write-ahead log system can be implemented using management systems such as Zookeeper or ETCD.

[0096] In one specific implementation, the sequential implementation process for different data operations is as follows:

[0097] First, for each data operation, if all the collected broadcast requests are of the first-hop type, the master node selects the maximum value from the collected logical timestamps as the target logical timestamp for the data operation; if there are second-hop type broadcast requests among the collected broadcast requests, the master node uses the logical timestamp carried by the second-hop type broadcast request as the target logical timestamp for the data operation.

[0098] In practice, if all collected broadcast requests are of the first-hop type, it means the client completes the data operation submission with a single request. In this case, the logical timestamps recorded on each log server are inconsistent. Therefore, the largest logical timestamp among the collected logical timestamps can be selected as the target logical timestamp for the data operation. If the collected broadcast requests include second-hop type requests, it means the client completes the data operation submission with two requests. In this case, the logical timestamps recorded on each log server are consistent, i.e., the logical timestamp carried by the second-hop type broadcast request. Therefore, this logical timestamp can be directly used as the target logical timestamp for the data operation.

[0099] Then, the master node orders the different data operations based on the target logical timestamp. Specifically, the different data operations are arranged in descending order of the target logical timestamp, and the resulting data operation sorting achieves the ordering.

[0100] Regarding the aforementioned ordering logic, for data operations involving two-hop requests, the logical timestamp used for ordering can be directly determined, i.e., the logical timestamp carried in the second-hop type broadcast request; for data operations involving one-hop requests, there are multiple logical timestamps used for ordering, which cannot be directly determined, so the maximum value is taken as the logical timestamp for ordering. See [link to relevant documentation]. Figure 6 The data operation sequence results shown on the logical timeline are monotonically increasing. Assume the first-hop broadcast request is defined as TYPE1, and the second-hop broadcast request is defined as TYPE2. Figure 6 It can be seen that data operations #1, #4, #6, and #8 use the logical timestamp ordering of TYPE2, while data operations #2, #3, #5, #7, #9, and #10 use the logical timestamp ordering of TYPE1. The shaded area corresponding to each data operation represents the time range of multiple logical timestamps. The maximum value among these logical timestamps is selected as the target logical timestamp for comparison and sorting with the logical timestamps of other data operations. Figure 6 It can be seen that the final order of these 10 data operations is: #1 data operation, #2 data operation, #3 data operation, #4 data operation, #5 data operation, #6 data operation, #7 data operation, #8 data operation, #9 data operation, and #10 data operation.

[0101] This completes the above. Figure 5The data operation sequencing process shown in this paper involves a write-ahead log system that receives broadcast requests for data operations initiated by clients. Therefore, each log server in the write-ahead log system receives these broadcast requests, and each log server records a logical timestamp for each data operation based on the type of broadcast request. Compared to existing solutions that rely on selecting a master node in the write-ahead log system to replicate data operations among other nodes and address the single-point performance jitter caused by different data operation sequencing, this application decouples the replication logic from the sequencing logic. The broadcast requests for data operations initiated by clients directly reach each node in the write-ahead log system, implementing the replication logic. Each log server in the write-ahead log system asynchronously collects the broadcast requests and logical timestamps recorded by each log server to implement the sequencing logic for different data operations. Therefore, this application eliminates the need for a master node election as a single point of failure in the write-ahead log system, improving client access stability and avoiding service unavailability windows caused by master node failure triggering the election process.

[0102] Example 3:

[0103] Based on the above embodiments, the write-ahead log system involves scenarios such as repairing and re-entering nodes after a failure, or adding new nodes during expansion. Therefore, the write-ahead log system involves a node re-entry process.

[0104] Figure 7 The flowchart illustrates a reentrancy implementation process for a write-ahead log system when node reentrancy occurs, according to an exemplary embodiment, and includes the following steps:

[0105] Step 701: The log server newly added to the write-ahead log system queries the logical timestamp of the write-ahead log system.

[0106] The logical timestamp of the write-ahead log system represents the current logical time of the logical clock of each log server. Optionally, a unified logical timestamp of the write-ahead log system can be managed and maintained by a management system such as Zookeeper or ETCD. That is, the management system and each log server of the write-ahead log system always maintain a heartbeat connection to sense the local logical clock of each log server.

[0107] In practical implementation, since the logical timestamp consists of three parts: the highest bit, the high bit, and the low bit, the highest bit will only change after the high bit and the low bit have been counted during the logical clock timing process. In most cases, it remains unchanged. Considering the consistency of the highest bit and the high bit of the logical clock timing of different log servers, the highest bit of the logical timestamp of the pre-written log system, i.e., the epoch, can be queried from the management system.

[0108] Step 702: Increase the logical timestamp found in the query, and use the increased logical timestamp to update the logical timestamp of the write-ahead log system. In this way, when other log servers detect that the logical timestamp of the write-ahead log system has changed, they will also update the logical timestamp of their local logical clock, so that newly added log servers can work normally.

[0109] Specifically, when increasing the logical timestamp, the highest bit of the retrieved logical timestamp is incremented by 1, i.e., epoch+1.

[0110] It should be noted that during the process of adding the log server to the write-ahead logging system, the entire write-ahead logging system does not stop. In other words, the client is unaware of the log server's re-entry process.

[0111] It is worth noting that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0112] Corresponding to the aforementioned embodiments of the data operation sequencing method, this application also provides embodiments of the data operation sequencing system.

[0113] The data manipulation sequencing system includes the features described above. Figure 2 The client and write-ahead logging system shown are described, and the write-ahead logging system contains at least two log servers.

[0114] The client is used to execute the above. Figure 3 and Figure 4 The data operation submission method provided in the embodiment is used by the write-ahead log system to perform the above. Figure 5 The data operation sequencing method provided in the embodiment.

[0115] This application also provides an electronic device corresponding to the data operation sequencing method provided in the foregoing embodiments, for executing the above-described data operation sequencing method or data operation submission method.

[0116] Figure 8The present invention illustrates a hardware structure diagram of an electronic device according to an exemplary embodiment. The electronic device includes a communication interface 601, a processor 602, a memory 603, and a bus 604. The communication interface 601, processor 602, and memory 603 communicate with each other via the bus 604. The processor 602 can execute the data operation sequencing method or data operation submission method described above by reading and executing machine-executable instructions in the memory 603 corresponding to the control logic of the data operation sequencing method or data operation submission method.

[0117] The memory 603 mentioned in this application can be any electronic, magnetic, optical, or other physical storage system, and can contain stored information such as executable instructions, data, etc. Specifically, the memory 603 can be RAM (Random Access Memory), flash memory, storage drive (such as hard disk drive), any type of storage disk (such as optical disc, DVD, etc.), or similar storage media, or combinations thereof. Communication between this system network element and at least one other network element is achieved through at least one communication interface 601 (which can be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc., can be used.

[0118] Bus 604 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory 603 is used to store programs, and the processor 602 executes the programs after receiving execution instructions.

[0119] Processor 602 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 602 or by instructions in software form. The processor 602 can be a general-purpose processor, including a network processor (NP), digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor.

[0120] The electronic device provided in this application embodiment and the data operation sequencing method or data operation submission method provided in this application embodiment are based on the same inventive concept and have the same beneficial effects as the methods they adopt, operate or implement.

[0121] This application also provides a computer-readable storage medium corresponding to the data operation sequencing method or data operation submission method provided in the foregoing embodiments. Please refer to... Figure 9 As shown, the computer-readable storage medium shown is an optical disc 30, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it executes the data operation sequencing method or the data operation submission method provided in any of the foregoing embodiments.

[0122] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0123] The computer-readable storage medium provided in the above embodiments of this application and the data operation sequencing method or data operation submission method provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0124] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0125] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0126] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for determining the order of data operations, characterized in that, Applied to a write-ahead logging system, the write-ahead logging system comprising at least two log servers, the method includes: In response to a client initiating a broadcast request for a data operation, each of the log servers determines the logical timestamp of the data operation based on the type of the broadcast request, and records the broadcast request and the logical timestamp of the data operation; the type of the broadcast request is used to identify the number of hops in a request for a data operation initiated by the client; Based on the broadcast requests and logical timestamps of different data operations recorded by each of the log servers, the different data operations are ordered.

2. The method according to claim 1, characterized in that, The logical timestamp of the data operation is determined by each of the log servers based on the type of the broadcast request, including: Each of the log servers uses the current logical timestamp of its local logical clock as the logical timestamp of the data operation, based on the broadcast request being of the first hop type. Since the broadcast request is of the second hop type, the logical timestamp carried by the broadcast request is used as the logical timestamp of the data operation; the logical timestamp carried by the broadcast request is determined by the client based on the logical timestamp carried by the responses returned by each of the log servers to the first hop type broadcast request.

3. The method according to claim 1, characterized in that, Based on the broadcast requests and logical timestamps of different data operations recorded by each of the log servers, the different data operations are ordered, including: A master node is elected among the various log servers; The master node obtains the broadcast requests and logical timestamps of the different data operations from each log server; The master node orders the different data operations based on the acquired broadcast requests and logical timestamps.

4. The method according to claim 3, characterized in that, The master node, based on the acquired broadcast request and logical timestamp, orders the different data operations, including: For each of the different data operations, the master node selects the maximum value from the logical timestamps of the data operation as the target logical timestamp of the data operation, based on the fact that the broadcast requests of the data operation are all of the first hop type; and if there are broadcast requests of the second hop type in the broadcast requests of the data operation, the logical timestamp carried by the broadcast requests of the second hop type is used as the target logical timestamp of the data operation. The master node orders the different data operations based on the target logical timestamp.

5. The method according to any one of claims 1-4, characterized in that, Before ordering the different data operations based on the broadcast requests and logical timestamps of the different data operations recorded by each of the log servers, the method further includes: In response to a synchronization request sent by the client, each of the log servers adjusts the logical timestamp of its local logical clock to the logical timestamp carried in the synchronization request.

6. The method according to any one of claims 1-4, characterized in that, The method further includes: The log server newly added to the write-ahead log system queries the logical timestamp of the write-ahead log system; The newly added log server in the write-ahead log system raises the queried logical timestamp and updates the logical timestamp of the write-ahead log system with the raised logical timestamp, so that when the existing log server detects that the logical timestamp of the write-ahead log system has changed, it updates the logical timestamp of its local logical clock.

7. A data operation submission method, characterized in that, Applied to a client, the method includes: Send a first-hop type broadcast request for data operation to the write-ahead log system, and receive the first response returned by the write-ahead log system within a preset timeout period; If the number of first responses reaches the number of log servers included in the write-ahead log system and the logical timestamp included in the first response meets the predetermined conditions, the data operation is determined to have been successfully submitted. If the number of first responses does not reach the number of log servers but reaches the first target number, or if the logical timestamp contained in the first response does not meet the predetermined conditions, a target logical timestamp is determined based on the logical timestamp contained in the first response, and a second-hop type broadcast request is sent to the write-ahead log system using the target logical timestamp, and a second response returned by the write-ahead log system is received within the timeout period. The data operation is considered successfully submitted once the number of second responses reaches the second target number.

8. The method according to claim 7, characterized in that, The predetermined condition is that at least one third target number of first responses has a logical timestamp that is the maximum logical timestamp of all first responses.

9. The method according to claim 7, characterized in that, Determining the target logical timestamp based on the logical timestamp contained in the first response includes: Sort the logical timestamps contained in the first response in ascending order; Use the logical timestamp with the sorting number of the first target quantity as the target logical timestamp.

10. A data manipulation sequencing system, characterized in that, The system includes: A write-ahead log system for performing the steps of the method according to any one of claims 1-6; A client for performing the steps of the method according to any one of claims 7-9.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the program to implement the steps of the method as claimed in any one of claims 1-9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by a processor to implement the steps of the method as described in any one of claims 1-9.

Images