Distributed log persistence methods, systems, devices, storage media, and electronic devices

By setting a shared buffer in the database client, log data is automatically persisted by the replica server after being written, which solves the performance limitation problem of the database client in the log persistence process and achieves efficient log persistence.

CN123068474APending Publication Date: 2026-11-10CHENGDU FANLIAN ZHICUN TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611593162.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-15
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

In existing technologies, database clients bear too much responsibility for network communication and coordination during log persistence, which limits the improvement of write performance.

Method used

The persistence process shifts from the database client to the replica server. By setting up a shared buffer on the database client side, log data is received and automatically persisted. The database client does not need to wait for network communication or persistence confirmation, and the write operation terminates at the local memory copy.

Benefits of technology

It improves log persistence efficiency, removes write performance limitations, and reduces latency and resource consumption.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123068474A_ABST
    Figure CN123068474A_ABST
Patent Text Reader

Abstract

This application provides a distributed log persistence method, system, apparatus, storage medium, and electronic device, relating to the field of computer science. In this scheme, after the database client receives log data sent by the database service in response to a database access request, it writes the data to a log buffer. The replica server automatically reads the log data from the log buffer for persistence. Since the primary focus of persistence shifts from the database client to the replica server, the database client does not need to initiate network communication, coordinate data replication, or wait for persistence confirmation on the write path. The write operation terminates at the local memory copy, and then a write success response is returned to the database service. This frees up limited write performance and improves log persistence efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of computers, and more specifically, to a distributed log persistence method, system, device, storage medium, and electronic device. Background Technology

[0002] Database systems (such as RocksDB, PostgreSQL, MySQL, etc.) must write and persist the transaction modifications to the log before returning a successful commit to the application when a transaction is committed. This mechanism is commonly referred to as the Write-Ahead Log (WAL) or Redo Log.

[0003] Log writing is characterized by strict sequential appending, high-frequency small writes, synchronous blocking, and the need for multiple replicas. Strict sequential appending means that log records are appended in the order in which transactions occur; high-frequency small writes mean that each transaction generates only tens to hundreds of bytes of log records; synchronous blocking means that transactions must wait for the log to be successfully persisted before committing; and the need for multiple replicas means that to ensure data reliability, logs need to be stored as copies on multiple physical nodes.

[0004] Research has found that, due to the aforementioned characteristics of log writing, whether it is a traditional local file system solution or a remote storage solution based on Remote Direct Memory Access (RDMA), the client assumes too many responsibilities in the log writing path when persisting logs, including initiating network communication, coordinating data replication, and waiting for persistence confirmation. This client-dominated architecture limits further improvements in write performance. Summary of the Invention

[0005] To overcome at least one deficiency in the prior art, this application provides a distributed log persistence method, system, device, storage medium, and electronic device, which can shift the dominant object of persistence from the database client side to the replica server side. The database client does not need to initiate network communication, coordinate data replication, or wait for persistence confirmation on the write path. The write operation terminates at the copy in local memory, and then a write success response can be returned to the database service, thereby releasing the limited write performance and improving the log persistence efficiency.

[0006] In a first aspect, this application provides a distributed log persistence method applied to a database client, wherein the database client communicates with a database service and a replica server, and the database client side is provided with a shared buffer shared with the replica server, the method comprising: Receive log data sent by the database service in response to database access requests; The log data is written to the log buffer of the shared buffer so that the replica server can automatically read the log data from the log buffer for persistence. After writing the log data to the shared buffer, a write success response is returned to the database service.

[0007] Secondly, this application provides a distributed log persistence system, which includes a database client, a database service communicating with the database client, and a replica server. The database client is provided with a shared buffer shared with the replica server. The database client receives log data sent by the database service in response to a database access request; and writes the log data into the log buffer of the shared buffer. The replica server automatically reads the log data from the log buffer and persists it. After the log data is written to the shared buffer, the database client returns a write success response to the database service.

[0008] Thirdly, this application provides a distributed log persistence device applied to a database client, wherein the database client communicates with a database service and a replica server, and the database client side is provided with a shared buffer shared with the replica server; the device includes: The log interaction module is used to receive log data sent by the database service in response to database access requests; The log buffer module is used to write the log data into the log buffer of the shared buffer, so that the replica server can automatically read the log data from the log buffer for persistence; The log interaction module is also used to return a write success response to the database service after writing the log data to the shared buffer.

[0009] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the distributed log persistence method.

[0010] Fifthly, this application provides an electronic device, which includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, implements the distributed log persistence method.

[0011] Compared with the prior art, this application has the following beneficial effects: In the distributed log persistence method, system, device, storage medium, and electronic device provided in this application, after the database client receives log data sent by the database service in response to a database access request, it writes the data to a log buffer. The replica server then automatically reads the log data from the log buffer for persistence. Because the primary focus of persistence shifts from the database client to the replica server, the database client does not need to initiate network communication, coordinate data replication, or wait for persistence confirmation along the write path. The write operation terminates at the local memory copy, and then a write success response is returned to the database service. This frees up the limited write performance and improves the efficiency of log persistence. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 One of the flowcharts of the distributed log persistence method provided in the embodiments of this application; Figure 2 A schematic diagram of the structure of the shared buffer provided in the embodiments of this application; Figure 3 This is a schematic diagram of the copy server startup process provided in an embodiment of this application; Figure 4 The second flowchart illustrates the distributed log persistence method provided in this application embodiment. Figure 5 A schematic diagram illustrating the interaction between the copy server and the shared buffer provided in an embodiment of this application; Figure 6 The third flowchart illustrating the distributed log persistence method provided in this application embodiment; Figure 7 This is a schematic diagram of the structure of the distributed log persistence device provided in the embodiments of this application; Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of the embodiments of this application (hereinafter referred to as "the embodiments") clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0015] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0016] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0017] In the description of this application, it should be noted that the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0018] Based on the above statement, as described in the background section, the client-dominated architectural pattern limits further improvements in write performance.

[0019] Taking the traditional local file system scheme as an example, when a transaction is committed, the database process first writes the log records to the local file, and then forces a flush to the persistent storage medium through the fsync system call. In this process, each fsync involves kernel mode switching, file system metadata updates, block device I / O scheduling, and other overheads, with latency typically ranging from hundreds of microseconds to milliseconds. In addition, this scheme has a high risk of single point of failure; once the local storage is damaged, the data cannot be recovered.

[0020] In RDMA-based remote storage solutions, a push mode is used. In this process, the database client sends log data to the remote storage node via RDMA SEND / RECV bilateral operations. Upon receiving the request, the storage node first persists the data to its local disk and then returns a persistence completion response via RDMA SEND. The client considers the log write successful only after receiving this response. However, in practical applications, this solution still suffers from problems such as client network round-trip overhead, client CPU involvement, multi-replica fan-out overhead, and complex synchronization semantics. Specifically, each log append consumes at least one RDMA network round-trip time (RTT, typically 1-5 microseconds); the client needs to execute RDMA verbs calls and manage the send queue, consuming application CPU resources; to ensure multiple replicas, the client also needs to perform an RDMA WRITE operation for each replica, with overhead increasing linearly with the number of replicas; furthermore, the client needs to wait for persistence confirmation from each replica, further increasing latency.

[0021] It should be noted that the defects in the solutions in the prior art are the result of practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed by the embodiments of this application in the following text should be regarded as contributions to this application in the process of invention and creation, and should not be understood as technical content known to those skilled in the art.

[0022] Based on the discovery of the above-mentioned technical problems, this embodiment provides a distributed log persistence method applied to a database client. The database client communicates with the database service and the replica server. The database client has a shared buffer shared with the replica server. Figure 1 As shown, the method includes: S1, Receive log data sent by the database service in response to the database access request; S2, write log data to the log buffer of the shared buffer so that the replica server can automatically read log data from the log buffer for persistence; S3, after writing the log data to the shared buffer, returns a write success response to the database service.

[0023] This can be understood as follows: after the database client receives the log data sent by the database service in response to the database access request, it writes it to the log buffer. The replica server then automatically reads the log data from the log buffer for persistence. Since the main body of persistence shifts from the database client to the replica server, the database client does not need to initiate network communication, coordinate data replication, or wait for persistence confirmation on the write path. The write operation terminates at the copy in local memory, and then the database client can return a write success response to the database service.

[0024] This allows the limited write performance to be released, improving the efficiency of log persistence.

[0025] In this embodiment, the database service is provided by a running database instance. The database client can be understood as a device running a log client program. The database instance and the log client program can run on the same device or on different devices. When both are deployed on the same device, the database instance is the one on that device that responds to database access requests and generates log data accordingly. The transfer of log data and the write success response between the database instance and the log client program are all completed within the local machine. It should be noted that the replica server is located on an independent node outside this device, which automatically reads log data from the shared buffer's log buffer and performs persistence.

[0026] To make the solution provided in this embodiment clearer, the following is combined with... Figure 1 Each step of the method is described in detail. However, it should be understood that the operations in the flowchart may not be implemented in sequence, and steps without logical contextual relationships may be reversed in order or implemented simultaneously. Furthermore, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowchart, or remove one or more operations from the flowchart.

[0027] See also Figure 1 First, let's talk about... Figure 1 Step S1 will be explained in detail: S1 receives log data sent by the database service in response to the database access request.

[0028] In practical applications, database access requests can be issued by the application. In the process of responding to this request, the database service modifies the data and generates log data that needs to be written to disk. The party that receives this log data is the database client.

[0029] Specifically, after the database service records the data changes caused by a certain access as log data, it calls the log writing interface provided by the database client to send the log data to the database client, which then receives it. It should be noted that in the scenario of transaction commit, the database service must first persist the log data before returning a successful commit to the application. Therefore, in the response process of the database access request, the log data is sent out before the successful result returned to the application.

[0030] Based on the above description of log data in the embodiments, the following will continue to discuss... Figure 1 Step S2 will be explained in detail: S2 writes log data to the log buffer of the shared buffer so that the replica server can automatically read the log data from the log buffer for persistence.

[0031] like Figure 2 As shown, in practical applications, the database client can pre-allocate a contiguous block of physical memory as a shared buffer, for example, 64MB. A segment of this memory can be designated as a log buffer, which can specifically be a circular buffer that stores log data sequentially. The database client writes the received log data to the write position in the log buffer using a memory copy method. After writing, it updates the write position and uses a memory barrier to ensure that the log data is published (visible to the replica server) before the updated write position. Furthermore, the write operation is completed locally and does not involve network communication.

[0032] Optionally, the log data written to the log buffer can be in a self-describing format, with a fixed-length header, for example, 4 bytes, before each log data entry. This header indicates the byte length of the log data entry, allowing the replica server to identify the boundary of a single log data entry from a continuous byte stream without the need for external pointers.

[0033] Optionally, in addition to the log buffer, the memory area can also include a replica progress table and a global metadata area. The global metadata area is used to store a small amount of control information, including the end pointer of the log data and the truncation position of the log data.

[0034] The replica server automatically performs read operations after log data is written, without requiring database clients to notify or wait for read results. In practical applications, the replica server can read fixed-length data blocks (e.g., 64KB) from the log buffer via Remote Direct Memory Access (RDMA) read operations. It then parses each record according to the specified length header, stopping parsing when it encounters records with abnormal lengths, failed validations, or records exceeding the current data block size. The unparsed portions are carried over to the next round of processing.

[0035] like Figure 3 As shown, to retrieve data promptly, the replica server's read operations can be performed continuously and repeatedly, without a fixed polling cycle or sleep interval. The read actions of each replica server are staggered at startup, and the first read can be determined by the following formula. The startup delay of each replica server is expressed as:

[0036] In the formula, Values , Indicates the number of replica servers. This indicates the interval between the startup times of two adjacent replica servers, for example, tens of microseconds.

[0037] After the replica server reads the log data, it can batch write the data to the local solid state drive (SSD) when the segment buffer is full (for example, when the segment buffer reaches 256KB) or when the time threshold is reached, and then write back the persisted progress, thereby completing the persistence.

[0038] The above embodiments described the log data writing operation. Next, we will continue with... Figure 1 Step S3 will be explained in detail: S3, after writing the log data to the shared buffer, returns a write success response to the database service.

[0039] This can be understood as the database client limiting the scope of a single log append operation to the step of writing log data to the shared buffer. Once the write operation is complete, it can return a write success response to the database service without waiting for the replica server to read the log data from the log buffer, or for the log data to be stored on the replica server's local storage medium. This response is the database client's return to the log write interface call, occurring within the runtime environment of the database client and the database service, and does not involve any network communication.

[0040] It should be noted that since the return condition depends only on the writing of log data, the database client does not need to be aware of the number of replica servers on the write path, and the reading and persistence processes of multiple replica servers can be transparent to the database client.

[0041] For example, in a three-replica deployment scenario, the latency of a single log append can be limited to around 100 nanoseconds, keeping the processor usage of the database client at a very low level. Once the database service receives the write success response, it can continue processing subsequent requests.

[0042] It's important to note that after the database client writes log data to the shared buffer, it immediately returns a success message to the upper layer. Its write speed is determined by the volume of business requests, while the speed at which the replica server reads the log buffer and completes persistence depends on its own processing capacity. There is no inherent matching relationship between the two. If the replica server slows down its reads due to network congestion or slow storage flushing, the space occupied in the log buffer will continue to grow. It should be understood that if the database client ignores this and continues writing, the circularly used log buffer will eventually fill up, and log data not yet retrieved by the replica server will risk being overwritten by new data.

[0043] To address the aforementioned issues, the shared buffer also includes a replica schedule, based on which, such as Figure 4 As shown, after writing log data to the shared buffer, the distributed log persistence method provided in this embodiment further includes: S4 retrieves the persistence progress from the replica progress table.

[0044] The persistence progress is used to indicate the position of the persisted log data in the log buffer on the replica server.

[0045] like Figure 5 As shown, during implementation, a replica schedule can also be drawn in the shared buffer, which reserves one slot for each replica server. For example, in a distributed storage cluster consisting of three replica servers, there are three slots, and the number of slots is the same as the number of replica servers.

[0046] After the replica server flushes the log data to the local storage medium, the position of its persisted log data in the log buffer will advance accordingly. This position can be written back to the aforementioned slot by the replica server through a Remote Direct Memory Access (RDMA) write operation. The write-back position of each replica server in the replica progress table can be recorded as follows: , This indicates the position in the log buffer where the log data that has been persisted by the replica server has been reached. Log data before this position has been persisted by the replica server, while log data after this position has not yet been persisted by the replica server.

[0047] The database client can thus read the current values ​​of each slot from the replica progress table and use them as the persistence progress. For example, the replica server can batch write log data when its segment buffer is full or when a time threshold is reached, and update the values ​​in the slots after the write is complete. Therefore, the persistence progress read by the database client reflects the result of the most recent write by the replica server.

[0048] Regarding step S4, it should be noted that multiple replica servers independently read the log data from the log buffer and complete the persistence, without any agreed-upon order. Therefore, their positions in the replica progress table at any given time are often different. However, if the database client only takes the position written back by one replica server, or uses the latest position as the basis for overall persistence, then when that replica's progress is ahead while other replicas lag behind, the log data on the lagging replicas that has not yet been persisted is considered safely saved, posing a risk of log loss.

[0049] To address the above problems, this embodiment provides the following optional implementation methods for step S4: S4-1, retrieve the persistent individual progress of each replica server from the replica progress table; S4-2: Based on the persistence progress of each replica server, the smallest one is determined as the persistence progress.

[0050] This can be understood as follows: when the database client reads data, it can retrieve the persistence progress of each replica server from the replica progress table one by one, and determine the smallest one as the persistence progress. This value relationship can be expressed as:

[0051] In the formula, Indicates the persistence progress. Indicates the first The persistent individual progress of each replica server. This indicates the number of replica servers.

[0052] Because each replica server independently reads log data from the log buffer and completes its own persistence, their individual persistence progress can vary in time. For example, if a replica server lags behind due to slow storage flushing, its value in the replica progress table will be lower than that of the other replica servers. In this case, the result obtained by taking the minimum value is determined by the replica server that lagged behind. Therefore, the persistence progress only advances after all replica servers have completed the persistence of their corresponding log data.

[0053] Based on the above embodiments regarding the replication progress table and its persistent recording progress, the following will continue to... Figure 4 Step S5 will be explained in detail: S5 determines the percentage of remaining log data in the log buffer based on the persistence progress.

[0054] The remaining log data refers to log data that is cached in the log buffer but has not yet been persisted.

[0055] This can be understood as follows: the log data in the log buffer that has not yet been persisted by any replica server constitutes the remaining log data. The length of the bytes between the beginning and end of the log data increases as log data is written and decreases as persistence progresses.

[0056] The database client can read the write position of log data in the log buffer, i.e., the address reached by the most recent write, and combine this with the persistence progress obtained from the replica progress table. The length in bytes between these two values ​​is taken as the length of the remaining log data. This length is then compared with the capacity of the log buffer to obtain the proportion of remaining log data in the log buffer. This proportion can be expressed as:

[0057] In the formula, This indicates the aforementioned percentage. This indicates the position where log data is written in the log buffer. Indicates the persistence progress. This indicates the capacity of the log buffer.

[0058] For example, in a scenario with a shared buffer of 64MB, if the write volume continues to increase while the persistence progress fails to keep up, and The distance between them thus increases. Gradually increases from near zero and approaches At this point, almost all the space in the log buffer is occupied by the remaining log data. For example, the preset threshold for this percentage can be 75%, 90%, and 95% as comparison nodes.

[0059] Therefore, the above percentage describes the extent to which the log buffer is occupied; the larger the value, the more log data is backed up in the log buffer waiting to be persisted.

[0060] Based on the above examples regarding the proportion of remaining log data in the log buffer, the following will continue to discuss... Figure 4 Step S6 will be explained in detail: S6. If the percentage is greater than the preset threshold, execute the back pressure operation.

[0061] The backpressure operation is used to limit the continued writing of log data to the log buffer.

[0062] During implementation, when the proportion of remaining log data in the log buffer exceeds the preset threshold, it indicates that the log data writing speed has exceeded the persistence speed of the replica server, and the available space in the log buffer is becoming insufficient. The database client can then perform a backpressure operation to limit the continued writing of log data into the log buffer.

[0063] It should be understood that this backpressure operation does not change the way log data is written, but rather constrains the writing rhythm, coordinating the writing speed with the persistence progress, thereby preventing log data that has not yet been persisted by the replica server from being overwritten by subsequently written log data. The above triggering condition can be expressed as follows: .

[0064] For example, the preset threshold The database client can set the write percentage to 75%, 90%, and 95%, each corresponding to different levels of backpressure operations. When the percentage exceeds 75%, the database client can log an alert and notify the database service to trigger a batch commit, merging multiple small batch writes and reducing the number of writes and the amount of log buffer space used. When the percentage exceeds 90%, the database client can block writes and wait for the persistence progress to continue before resuming writing. When the percentage exceeds 95%, the database client can force synchronization and terminate the log stream to prevent writing from continuing if the log buffer space is exhausted.

[0065] In this way, the persistence progress of the replica server is fed back to the write side, the write rhythm can be adapted to the consumption capacity of the replica server, the log data that has not yet been persisted will not be overwritten, and the space occupation of the shared buffer is kept within a controllable range.

[0066] In practical applications, after log data is written to the shared buffer, although the database service has received a write success response, this response only indicates that the log data has entered the log buffer of the shared buffer. Whether the replica server has retrieved this log data and stored it on the local storage medium is still in an unconfirmed state.

[0067] To solve the above problems, such as Figure 6 As shown, the distributed log persistence method provided in this embodiment also includes: S7 responds to the transaction commit request initiated by the database service and obtains the persistence progress written back by the replica server.

[0068] The persistence progress is used to indicate the position of the persisted log data in the log buffer on the replica server.

[0069] During implementation, the database service sends a transaction commit request to the database client upon transaction commit. This request can be understood as a synchronous call initiated by the database service to the database client. Upon receiving this request, the database client reads the persistence progress written back by the replica server from the replica progress table in the shared buffer.

[0070] S8. If the persistence progress reaches the target position required by the transaction commit request, then return a response to the database service indicating that the target position has been reached.

[0071] S9. If the persistence progress has not reached the target position required by the transaction commit request, continue to obtain the persistence progress until the persistence progress reaches the target position.

[0072] In practical applications, the database client then compares the persistence progress with the target position, which can be the location of any persisted log data in the log buffer. This judgment condition can be expressed as:

[0073] In the formula, Indicates the persistence progress. This indicates the target position that this transaction commit requires to be achieved, that is, the position of the log data in the shared buffer.

[0074] when It has been reached When the database client can return immediately; when Not yet reached At this time, the database client can enter a polling wait, repeatedly reading the replica progress table until the conditions are met or the preset timeout period is reached, thereby obtaining confirmation of the log data being written to disk at the moment of transaction commit.

[0075] In this way, the confirmation conditions on which the transaction commit depends correspond to the actual state of the log data being persisted to disk. The database service only needs to initiate a transaction commit request once and wait for a response. The process of confirming whether the log data has been persisted is undertaken by the database client.

[0076] Based on the same inventive concept as the distributed log persistence method provided in this embodiment, this embodiment also provides a distributed log persistence system. The distributed log persistence system includes a database client, a database service that communicates with the database client, and a replica server. The database client side is provided with a shared buffer shared with the replica server.

[0077] The database client receives log data sent by the database service in response to the database access request; and writes the log data to the shared log buffer. The replica server automatically reads log data from the log buffer and persists it. After writing the log data to the shared buffer, the database client returns a write success response to the database service.

[0078] This can be understood as follows: since the main body of persistence shifts from the database client side to the replica server side, the database client does not need to initiate network communication, coordinate data replication, or wait for persistence confirmation on the write path. The write operation terminates at the copy in local memory, and then a write success response can be returned to the database service.

[0079] This allows the limited write performance to be released, improving the efficiency of log persistence.

[0080] Optionally, the number of replica servers can be multiple; When log data is initially read from the log buffer for persistence, each replica server performs a round-robin reading according to a preset time interval.

[0081] It should be understood that multiple replica servers need to retrieve log data from the same log buffer. If they are allowed to initiate reads simultaneously after startup without differentiation, the read operations at the same time will be concentrated on the database client's memory and the network resources connected to it. The instantaneous contention will affect the reads of each replica server.

[0082] Therefore, in this embodiment, when each replica server reads log data from the log buffer and persists it for the first time, it performs polling reads in sequence according to a preset time interval. That is, after the first replica server starts reading, the second replica server starts reading after the preset time interval, and so on for the remaining replica servers.

[0083] For example, the preset time interval can be tens of microseconds. In this way, the reading actions of each replica server are distributed across the timeline, avoiding the situation where multiple replica servers start reading at the same time and fall into the same reading rhythm, and the log data temporarily stored in the log buffer is read by replica servers throughout the entire timeline.

[0084] It should also be noted that the replica servers read log data from the shared buffer on the database client side. Once the database client encounters an error, the shared buffer can no longer provide log data, and the replica servers lose a unified source of data to read. At this point, the persistence levels of the replica servers are often inconsistent. If the replica servers do not refer to each other, the replica servers lagging behind will not know which log data they are missing, nor will they have anywhere to retrieve this data, and will simply remain where they are, resulting in long-term uneven persistence levels among the replica servers.

[0085] To address the aforementioned issues, multiple replica servers are used. If a database client encounters an error, a primary replica is selected from among the replica servers, with the primary replica being the one with the highest persistence progress. The remaining replica servers then obtain the out-of-sync log data from the primary replica for persistence.

[0086] During implementation, each replica server independently reads log data from the log buffer and completes persistence. The persistence progress can be recorded as follows:

[0087] In the formula, This indicates the number of replica servers.

[0088] When a database client encounters an exception, the shared buffer stops providing new log data. The replica servers can compare the values ​​mentioned above and select the one with the highest persistence progress as the primary replica. This primary replica is the one with the most complete persisted log data among all replica servers.

[0089] The remaining replica servers then retrieve the unsynchronized log data from the primary replica and continue persisting it. For example, in a three-replica deployment scenario, if one replica server falls behind due to slow storage flushing, and its persistence progress is less than that of the other two replica servers, the primary replica server with the maximum progress is selected as the one with the leading progress. The lagging replica servers request the missing log data from the primary replica server, thereby catching up with the primary replica server's position in terms of persistence progress.

[0090] In this way, each replica server can still replenish the log data through the primary replica server after an anomaly occurs in the database client, and the distribution of the log data that has been taken from the log buffer tends to be consistent among the replica servers.

[0091] Optionally, if a replica server fails, its persistence progress can be temporarily excluded from the persistence progress comparison, while the other replica servers continue to read and persist as usual. After the replica server recovers, it can read the current shared buffer of the database client and fill in its missing log data.

[0092] Based on the same inventive concept as the distributed log persistence method provided in this embodiment, this embodiment also provides a distributed log persistence device. This device includes at least one software functional module that can be stored in memory or embedded in an electronic device. A processor in the electronic device executes the executable module stored in memory. For example, the software functional module and computer program included in this device. Please refer to... Figure 7 Functionally, the device may include: The log interaction module is used to receive log data sent by the database service in response to database access requests; The log buffer module is used to write log data to the log buffer of the shared buffer so that the replica server can automatically read the log data from the log buffer for persistence. The log interaction module is also used to return a write success response to the database service after writing log data to the shared buffer.

[0093] Optionally, the shared buffer also includes a replica progress table. After the log data is written to the shared buffer, the log buffer module is also used to obtain the persistence progress from the replica progress table, wherein the persistence progress is used to indicate the position of the log data that has been persisted by the replica server in the log buffer. Based on the persistence progress, determine the proportion of remaining log data in the log buffer. The remaining log data refers to the log data cached in the log buffer that has not yet been persisted. If the percentage exceeds a preset threshold, a backpressure operation is executed. The backpressure operation is used to restrict the continued writing of log data to the log buffer.

[0094] Optionally, there are multiple replica servers, and the replica progress table records the progress of the persistent individual write-back by each replica server. The log buffer module obtains the persistence progress from the replica progress table in the following ways: Retrieve the persistent individual progress of each replica server from the replica progress table; Based on the persistence progress of each replica server, the smallest one is determined as the persistence progress.

[0095] Optionally, the log interaction module is also used to respond to a transaction commit request initiated by the database service and obtain the persistence progress of the replica server's write-back, wherein the persistence progress is used to indicate the position of the log data that has been persisted by the replica server in the log buffer; If the persistence progress reaches the target position required by the transaction commit request, a response indicating that the target position has been reached is returned to the database service. If the persistence progress has not reached the target position required by the transaction commit request, the persistence progress will continue to be acquired until the persistence progress reaches the target position.

[0096] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0097] It should also be understood that if the above embodiments are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0098] Therefore, this embodiment also provides a storage medium, which is a computer-readable storage medium. This storage medium stores a computer program, which, when executed by a processor, implements the distributed log persistence method provided in this embodiment. The storage medium can be any medium capable of storing program code, such as a USB flash drive, external hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0099] Please refer to Figure 8 This embodiment also provides an electronic device, which may include a processor 22 and a memory 21. The memory 21 stores a computer program, and the processor implements the distributed log persistence method provided in this embodiment by reading and executing the computer program in the memory 21 corresponding to the above embodiments.

[0100] See also Figure 8 The electronic device also includes a communication unit 23. The memory 21, processor 22 and communication unit 23 are electrically connected to each other directly or indirectly through system bus 24 to realize data transmission or interaction.

[0101] The memory 21 can be an information recording device based on any electronic, magnetic, optical, or other physical principles, used to record execution instructions, data, etc. In some embodiments, the memory 21 can be, but is not limited to, volatile memory, non-volatile memory, memory drive, etc.

[0102] In some embodiments, the volatile memory may be random access memory (RAM); in some embodiments, the non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, etc.; in some embodiments, the storage drive may be a disk drive, solid-state drive, any type of storage disk (such as optical disc, DVD, etc.), or similar storage media, or a combination thereof.

[0103] The communication unit 23 is used to send and receive data over a network. In some embodiments, the network may include a wired network, a wireless network, a fiber optic network, a telecommunications network, an intranet, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a public switched telephone network (PSTN), a Bluetooth network, a ZigBee network, or a near field communication (NFC) network, or any combination thereof. In some embodiments, the network may include one or more network access points. For example, the network may include wired or wireless network access points, such as base stations and / or network switching nodes, through which one or more components of the service request processing system can connect to the network to exchange data and / or information.

[0104] The processor 22 may be an integrated circuit chip with signal processing capabilities, and may include one or more processing cores (e.g., a single-core processor or a multi-core processor). By way of example only, the processor described above may include a Central Processing Unit (CPU), an Application Specific Integrated Circuit (ASIC), an Application Specific Instruction-set Processor (ASIP), a Graphics Processing Unit (GPU), a Physics Processing Unit (PPU), a Digital Signal Processor (DSP), a Field Programmable Gate Array (FPGA), a Programmable Logic Device (PLD), a controller, a microcontroller unit, a Reduced Instruction Set Computing (RISC) computer, or a microprocessor, or any combination thereof.

[0105] Understandable. Figure 8The structure shown is for illustrative purposes only. Electronic devices may also have more advanced features. Figure 8 Showing more or fewer components, or having with Figure 8 The different configurations shown. Figure 8 The components shown can be implemented using hardware, software, or a combination thereof.

[0106] It should be understood that the apparatus and methods disclosed in the above embodiments can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0107] The above descriptions are merely various embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A distributed log persistence method, characterized in that, Applied to a database client, the database client communicates with a database service and a replica server. The database client has a shared buffer shared with the replica server. The method includes: Receive log data sent by the database service in response to database access requests; The log data is written to the log buffer of the shared buffer so that the replica server can automatically read the log data from the log buffer for persistence. After writing the log data to the shared buffer, a write success response is returned to the database service.

2. The distributed log persistence method according to claim 1, characterized in that, The shared buffer also includes a replication schedule. After writing the log data to the shared buffer, the method further includes: Obtain the persistence progress from the replica progress table, wherein the persistence progress is used to indicate the position of the log data that has been persisted by the replica server in the log buffer; Based on the persistence progress, determine the proportion of remaining log data in the log buffer, wherein the remaining log data represents log data cached in the log buffer that has not been persisted. If the percentage is greater than a preset threshold, a backpressure operation is performed, wherein the backpressure operation is used to restrict the continued writing of log data into the log buffer.

3. The distributed log persistence method according to claim 2, characterized in that, There are multiple replica servers, and the replica progress table records the persistent individual write-back progress of each replica server. Obtaining the persistence progress from the replica progress table includes: Obtain the persistent individual progress of each replica server from the replica progress table; Based on the persistence progress of each replica server, the smallest one is determined as the persistence progress.

4. The distributed log persistence method according to claim 1, characterized in that, The method further includes: In response to the transaction commit request initiated by the database service, the persistence progress written back by the replica server is obtained, wherein the persistence progress is used to indicate the position of the log data that has been persisted by the replica server in the log buffer; If the persistence progress reaches the target position that the transaction commit request is intended to achieve, then a response indicating that the target position has been reached is returned to the database service. If the persistence progress has not reached the target position required by the transaction commit request, the persistence progress will continue to be acquired until the persistence progress reaches the target position.

5. A distributed log persistence system, characterized in that, The distributed log persistence system includes a database client, a database service that communicates with the database client, and a replica server. The database client has a shared buffer that is shared with the replica server. The database client receives log data sent by the database service in response to a database access request; and writes the log data into the log buffer of the shared buffer. The replica server automatically reads the log data from the log buffer and persists it. After the log data is written to the shared buffer, the database client returns a write success response to the database service.

6. The distributed log persistence system according to claim 5, characterized in that, The number of replica servers is multiple; When the log data is first read from the log buffer for persistence, each replica server performs a polling read in sequence according to a preset time interval.

7. The distributed log persistence system according to claim 5, characterized in that, The number of replica servers is multiple; If the database client encounters an error, a primary replica is selected from among the replica servers, wherein the primary replica is the replica server with the highest persistence progress. The remaining replica servers obtain the out-of-synchronization log data from the primary replica server and persist it.

8. A distributed log persistence device, characterized in that, An apparatus for use in database clients, which communicate with both a database service and a replica server. The database client has a shared buffer provided to the replica server. The apparatus includes: The log interaction module is used to receive log data sent by the database service in response to database access requests; The log buffer module is used to write the log data into the log buffer of the shared buffer, so that the replica server can automatically read the log data from the log buffer for persistence; The log interaction module is also used to return a write success response to the database service after writing the log data to the shared buffer.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the distributed log persistence method according to any one of claims 1-4.

10. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing a computer program that, when executed by the processor, implements the distributed log persistence method according to any one of claims 1-4.