Data backup system, method and apparatus, and device, storage medium and program product

Abstract

A data backup system, comprising a first data server, a log server and at least one second data server. The first data server is configured for: acquiring at least one operation log, wherein the operation log is used for recording an operation behavior for first data; updating the first data on the basis of the at least one operation log; and sending the at least one operation log to the second data server and the log server during the process of updating the first data. The at least one second data server is configured for receiving the at least one operation log to perform backup storage, and updating, on the basis of the at least one operation log, second data stored in the second data server and used for backing up the first data. The log server is configured for: receiving the at least one operation log sent by the first data server, so as to perform backup storage; and after successfully performing backup storage on the at least one operation log, feeding back to the first data server a log backup result corresponding to the at least one operation log. The first data server is further configured for receiving the log backup result.

Description

Data backup system, method, device, equipment, storage medium and program product

[0001] Related applications

[0002] This application claims priority to Chinese patent application number 202410334042.2, filed on March 22, 2024, entitled “Data backup system, method, device, equipment, storage medium and program product,” the entire text of which is hereby incorporated by reference. Technical Field

[0003] The present application relates to the field of data synchronization technology, and in particular to a data backup system, method, apparatus, device, storage medium, and program product. Background Art

[0004] System continuity is crucial for many organizations and businesses. High system availability ensures rapid recovery and business continuity in the event of a failure. To achieve high system availability, multiple replicas are typically configured, with synchronous replication between replicas to ensure that a secondary database can be immediately switched to provide business data in the event of a primary database failure.

[0005] In related technologies, since asynchronous replication (Async Replication) supported by database languages ​​such as MySQL has data consistency issues, semi-synchronous replication (Semi-Sync Replication) is usually adopted. After the primary database receives a write operation and writes it to the local log, the primary database will wait for at least one slave database to confirm that it has received and written the same data change operation before returning the completion status of the write operation to the application.

[0006] However, the above method will degenerate into asynchronous replication when the master database cannot wait for the slave database's response due to network delay or slave database failure. There are still data consistency issues, and the slave database consumes a lot of resources and has low data synchronization efficiency. Summary of the Invention

[0007] Embodiments of the present application provide a data backup system, method, apparatus, device, storage medium, and program product.

[0008] In one aspect, a data backup system is provided, the system comprising a first data server, a second data server, and a log server;

[0009] The first data server is configured to obtain at least one operation log, the operation log being used to record operation behavior on the first data; update the first data based on the at least one operation log; and send the at least one operation log to the second data server and the log server during the process of updating the first data;

[0010] The log server is configured to receive the at least one operation log sent by the first data server for backup storage; after successfully backing up and storing the at least one operation log, feedback a log backup result corresponding to the at least one operation log to the first data server;

[0011] The first data server is further configured to receive the log backup result.

[0012] In another aspect, a data backup method is provided. The method is performed by a first data server in a data backup system, and the method includes:

[0013] Obtain at least one operation log, where the operation log is used to record an operation behavior on the first data;

[0014] updating the first data based on the at least one operation log; in the process of updating the first data, sending the at least one operation log to a log server and at least one second data server, the second data server being configured to back up the operation log and the first data stored in the first data server, and the log server being configured to back up the operation log stored in the first data server;

[0015] Receive a first log backup result sent by the log server after successfully storing the at least one operation log.

[0016] In another aspect, a data backup device is provided, comprising:

[0017] a processing module, configured to obtain at least one operation log, where the operation log is used to record an operation behavior on the first data;

[0018] a sending module, configured to update the first data based on the at least one operation log; in the process of updating the first data, sending the at least one operation log to a log server and at least one second data server, the second data server being configured to back up the operation log and the first data stored in the first data server, and the log server being configured to back up the operation log stored in the first data server;

[0019] The receiving module is configured to receive a first log backup result sent by the log server after the at least one operation log is successfully stored.

[0020] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement a data backup method as described in any of the above-mentioned embodiments of the present application.

[0021] On the other hand, a computer-readable storage medium is provided, in which at least one instruction, at least one program, a code set or an instruction set is stored. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the data backup method as described in any of the above embodiments of the present application.

[0022] In another aspect, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data backup method described in any of the above embodiments.

[0023] The details of one or more embodiments of the present application are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the present application will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the conventional technology, the following briefly introduces the drawings required for use in the embodiments or the conventional technology descriptions. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the disclosed drawings without any creative work.

[0025] FIG1 is a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application;

[0026] FIG2 is an interactive flow chart of a data backup system provided by an exemplary embodiment of the present application;

[0027] FIG3 is a flow chart of a first data server interaction method provided by an exemplary embodiment of the present application;

[0028] FIG4 is a flow chart showing an effect comparison provided by an exemplary embodiment of the present application;

[0029] FIG5 is a flow chart of an interaction method of a second data server provided by an exemplary embodiment of the present application;

[0030] FIG6 is a flow chart of a data backup method for a log server provided by an exemplary embodiment of the present application;

[0031] FIG7 is a flow chart of a data backup method provided by an exemplary embodiment of the present application;

[0032] FIG8 is a diagram of a distributed database architecture provided by an exemplary embodiment of the present application;

[0033] FIG9 is a schematic diagram of a data backup system provided by an exemplary embodiment of the present application;

[0034] FIG10 is a structural block diagram of a data backup device provided by an exemplary embodiment of the present application;

[0035] FIG11 is a structural block diagram of a terminal provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0037] It should be understood that although the terms first, second, etc. may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, a first parameter may also be referred to as a second parameter, and similarly, a second parameter may also be referred to as a first parameter without departing from the scope of this disclosure. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0038] For many organizations and enterprises, system continuity is crucial. The high availability of the system can ensure that the system can recover quickly and maintain business continuity when a failure occurs. In order to meet the high availability of the system, it is usually necessary to configure multiple replicas, and the replicas need to be synchronized and replicated immediately so that when the master database fails, the slave database can be switched to provide business data for the system. In the related art, since asynchronous replication supported by database languages ​​such as MySQL itself has data consistency issues, semi-synchronous replication is usually adopted. After the master database receives a write operation and writes it to the local log, the master database will wait for at least one slave database to confirm that it has received and written the same data change operation, and then return the completion status of the write operation to the application. However, when the master database cannot wait for a response from the slave database due to network delays or slave database failures, the above method will degenerate into asynchronous replication. There are still data consistency issues, and the slave database consumes a lot of resources, and the data synchronization efficiency is low.

[0039] Data consistency refers to the property that data in multiple replicas (such as the primary and backup databases) remains in the same state during data backup and synchronization. In this application, to ensure system stability and data integrity, strong data consistency is required. That is, all slave databases (including secondary data servers and log servers) can receive all transactions, so that the data in the primary and backup databases always remains consistent.

[0040] High availability refers to the ability of a system to continue to operate normally for a certain period of time. High availability is crucial for the systems of organizations and enterprises. It can ensure that the system can be quickly restored in the event of a failure and maintain business continuity. In the data backup system of this application, the high availability requirements of the system are met by configuring multiple replicas (such as a first data server, a second data server, and a log server) and implementing instant synchronous replication of data, so that when the primary database fails, the secondary database can be switched in time to provide business data to the system.

[0041] The main synchronization methods for replicas are asynchronous synchronization, semi-synchronization, and strong synchronization, which are introduced below:

[0042] Asynchronous synchronization: In asynchronous synchronization mode, after executing a transaction and writing it to the log, the master database immediately returns the transaction to the client, indicating that the transaction is complete, without waiting for the log to be replicated to any slave databases. This approach offers high performance because the master database does not need to wait. However, if the master database crashes before the log is replicated to the slave database, the unreplicated transactions will be lost.

[0043] Semi-synchronous mode: In semi-synchronous mode, after executing a transaction and writing logs, the master database waits for at least one slave database to confirm receipt of the logs before returning a message to the client indicating the transaction is complete. This approach has the advantage of ensuring that at least one slave database receives all transactions, allowing recovery from a master database crash. However, performance is slightly lower than asynchronous synchronization because it requires waiting for confirmation from the slave database.

[0044] Strong synchronization: In strong synchronization mode, after executing a transaction and writing it to the log, the master database waits for all slave databases to confirm receipt of the log before returning a response to the client indicating the transaction is complete. This approach offers the best data consistency, as all slave databases receive all transactions. However, it also offers the worst performance, as it requires waiting for confirmation from all slave databases.

[0045] To achieve strong data synchronization, at least three servers are typically required to participate in data backup, meaning at least two backup nodes are required. This avoids single points of failure and split-brain issues, improving system availability and data synchronization reliability. This architecture of at least three servers ensures strong data consistency, ensuring system stability and data integrity. Related technologies require at least two secondary data servers to participate in data backup to achieve strong data synchronization.

[0046] The split-brain problem occurs when a distributed system loses connection between nodes due to network failures or other factors, resulting in the system splitting into multiple independent parts. Each part believes it is a functioning part of the entire system, which can lead to data inconsistencies and conflicts. To avoid split-brain problems in data backup systems, at least three servers are typically required to participate in data backup. Reasonable architecture design and coordination mechanisms ensure system stability and data consistency.

[0047] The data backup system provided in the embodiment of the present application can be used to improve data synchronization efficiency in a strong synchronization scenario, and improve synchronization efficiency while strictly ensuring the consistency of master and backup data. On this basis, the data backup system provided in the embodiment of the present application performs data backup by adopting a data backup system including a first data server, a log server and at least one second data server, wherein the first data server acts as a master node, and the log server and at least one second data server act as backup nodes. It is possible to achieve strong data synchronization in the data backup process and ensure strong data consistency. When the first data server synchronizes data with the second data server, at least one operation log is sent to the log server through the first data server, and the log server receives and backs up and stores the at least one operation log, and feeds back the log to the first data server. In order to improve the efficiency of data synchronization, a log server is introduced to back up only the storage operation logs during the data synchronization process. Since the storage data of the log server is greatly reduced compared with the second data server, the load pressure of the log server is relatively small. The log server is relatively idle compared with the second data server, and can promptly feedback the response to the first data server, reducing the time the first data server waits for the response, thereby improving the efficiency of data synchronization. At the same time, in the scenario of strong data synchronization, by using the log server to replace at least one second data server, only at least one second data server is retained, and there is no need for at least two second data servers to achieve strong data synchronization. Since the load pressure of the log server is relatively small, the overall resource consumption demand of the data backup system is reduced, and the hardware configuration requirements of the log server are reduced, so that strong data synchronization can be achieved with fewer resources.

[0048] First, the application scenarios of the data backup system provided in the embodiments of the present application are explained. The data backup system provided in the embodiments of the present application can be applied to any application scenarios involving data backup requirements, such as read-write separation, cross-regional data synchronization, database upgrade and testing, load balancing, data backup and disaster recovery, data analysis and report generation, risk management, etc. This application does not limit this.

[0049] Cross-regional data synchronization refers to the real-time or periodic synchronization of data between storage devices or servers in different geographical locations. In this application, the data backup system can be applied to cross-regional data synchronization scenarios to ensure that data in different regions remains consistent, meeting the data access needs of users in different regions and improving data availability and consistency. For example, branches in different regions can all obtain the latest business data.

[0050] In some embodiments, the application of a data backup system in the field of read-write separation is taken as an example for description.

[0051] Schematically, during the use of the application, the first data server in the data backup system is used as the main database, and the second data server and the log server are used as the backup database. The main database receives any write operation during the use of the application, obtains the user request through the first data server to generate an operation log, and updates the data. The first data server synchronizes the operation log to the second data server and the log server to realize master-slave replication in the data backup system. The backup database receives the read operation. When the user needs to read data or operation logs, the read operation can be distributed to the backup database. For example, data is read in the second data server, and further data analysis and other operations can be performed on the second data server based on the read data. The log reading operation log is performed in the log server, and further historical operation records can be checked based on the read operation log to realize read-write separation, thereby reducing the read pressure of the main database and improving the read performance of the application. At the same time, through read-write separation, the resources of the backup database can also be effectively utilized to improve the overall performance and concurrency of the application.

[0052] In some embodiments, the application of the data backup system in the field of data backup and disaster recovery is taken as an example for description.

[0053] Schematically, during the operation of the user terminal, data services are provided to the user terminal through the data backup system, the first data server in the data backup system is used as the main database, the second data server and the log server are used as backup databases, and the first data server synchronizes the operation log to the second data server and the log server to realize master-slave replication in the data backup system. By performing regular data backup on the backup database, the security and reliability of the data are improved. When the main database fails or data is lost, it can be quickly switched to the backup database for data recovery. At the same time, during the operation of the user terminal, the backup database can also be switched in time to provide data services to the user terminal to ensure the user terminal experience.

[0054] It is worth noting that the above application scenarios are only illustrative examples and this application does not limit them.

[0055] In conjunction with the above application scenarios, the implementation environment of this application is introduced. Please refer to FIG1 , which shows a schematic diagram of an implementation environment provided by an exemplary embodiment of this application, and the implementation environment includes: a terminal 110 and a data backup system 120 .

[0056] The terminal 110 is a user terminal, and is used to send a user request to the data backup system 120. The data backup system 120 is used to provide background data services for the terminal 110.

[0057] The data backup system 120 includes a first data server 121, a second data server 122 and a log server 123, wherein the first data server 121, the second data server 122 and the log server 123 have a master-slave relationship, and the master-slave relationship is used to indicate that the second data server 122 and the log server 123 are used to provide data backup function for the first data server 121.

[0058] The first data server 121 is a master node server in the data backup system 120, and the second data server 122 and the log server 123 are standby node servers in the data backup system 120. Optionally, the data backup system 120 may have multiple master node servers and multiple standby node servers.

[0059] First data server 121 is the data backend server for terminal 110 and is used to process user requests initiated by terminal 110. Second data server 122 and log server 123 are used to back up the operation logs generated by first data server 121 based on user requests. If first data server 121 fails, second data server 122 and log server 123 can serve as backup servers to restore the operation logs and data corresponding to the user requests.

[0060] During data backup, first data server 121 concurrently sends at least one operation log to second data server 122 and log server 123. Upon receiving a backup result from either second data server 122 or log server 123, first data server 121 determines the backup status of the operation log corresponding to that backup result. Since log server 123 typically only backs up operation logs, log server 123 is more efficient in providing backup result feedback. Therefore, in a semi-synchronous scenario, first data server 121 only needs to wait for the backup result from log server 123 before providing feedback on the completion status of the operation log to user terminal 110, improving data synchronization efficiency. In a strong synchronization scenario, first data server 121 waits for both second data server 122 and log server 123 to provide feedback on the backup result before determining that the operation log backup is complete and providing feedback on the completion status to user terminal 110. Since log server 123 only backs up operation logs, the load is lower, reducing the overall resource consumption of data backup system 120 and lowering the hardware requirements of log server 123. This allows for strong data synchronization with fewer resources, improving data synchronization efficiency.

[0061] The above-mentioned terminal is optional. The terminal can be a desktop computer, a laptop computer, a mobile phone, a tablet computer, an e-book reader, a Moving Picture Experts Group Audio Layer III (MP3) player, a Moving Picture Experts Group Audio Layer IV (MP4) player, a smart TV, a smart car, and other terminal devices in various forms. The embodiments of the present application are not limited to this.

[0062] It is worth noting that the above-mentioned servers can be independent physical servers, or they can be server clusters or distributed systems composed of multiple physical servers. They can also be cloud servers that provide basic cloud computing services such as cloud services, cloud security, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), as well as big data and artificial intelligence platforms.

[0063] Among them, cloud technology refers to a hosting technology that unifies hardware, software, network and other resources within a wide area network or local area network to achieve data calculation, storage, processing and sharing.

[0064] In some embodiments, the above-mentioned server can also be implemented as a node in a blockchain system.

[0065] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, storage, display, etc.), and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant region. For example, the user request and operation data involved in this application are all obtained with full authorization.

[0066] The data backup system provided in the embodiment of the present application includes a first data server, a log server, and at least one second data server. This embodiment of the present application uses the data backup system 120 shown in Figure 1 as an example. The data backup system includes a first data server 121, a second data server 122, and a log server 123. There can be one or more log servers 123.

[0067] In some embodiments, a master-slave relationship exists between the servers in the data backup system 120. Optionally, the master-slave relationship is used to indicate that the first data server 121 acts as the master node in the data backup system, and the second data server 122 and the log server 123 act as backup nodes in the data backup system 120.

[0068] A master-slave relationship refers to a collaborative relationship between different server nodes in a data backup system. In the data backup system of this application, the first data server acts as the master node (master database), responsible for processing user requests, updating data, and generating operation logs, etc. The second data server and log server act as standby nodes (standby databases) to back up the data and operation logs in the master node. The master-slave relationship indicates that the standby node provides data backup function for the master node. When the master node fails, the standby node can be used as an alternative server to restore operation logs and data.

[0069] Schematically, taking the data backup system 120 used to provide data services for the user terminal 110 as an example, the first data server 121 is connected to the user terminal 110 as a master node, and stores operation logs and business data in the process of providing background data services for the user terminal 110. The second data server 122 is used as a backup node to back up the operation logs and business data stored in the first data server 121, and the log server 123 is used as a backup node to back up the operation logs stored in the first data server 121.

[0070] Optionally, the number of the first data server, the second data server and the log server in the data backup system can be one or more, such as a master-slave relationship of one master and one slave, one master and multiple slaves, multiple masters and multiple slaves, etc., which is not limited in this application. In order to achieve strong data synchronization, the data backup system provided by the embodiment of the present application includes at least three servers, wherein the first data server acts as a master node, and the log server and at least one second data server act as standby nodes. The data backup system provided by the embodiment of the present application adopts a log server to replace at least one second data server in the related art, and does not require at least two second data servers to participate in data backup, thereby reducing the hardware resource consumption of the second data server in the standby node.

[0071] Optionally, physical replication between the master node and the standby node is achieved by synchronizing the operation log from the master node to the standby node. Physical replication is a data replication technology that synchronizes data by replicating the original data bits on the storage medium. In the data backup system of the present application, physical replication between the master and slave nodes is achieved by synchronizing the operation log from the master node (first data server) to the standby node (second data server and log server). During the physical replication process, the bits of the source data are directly copied to the target storage medium without considering the logical structure or format of the data, thereby ensuring that the operation logs stored in the log server and at least one second data server are consistent with the operation logs stored in the first data server. In physical replication, the bits of the source data are directly copied to the target storage medium without considering the logical structure or format of the data. That is, the operation log in the first data server is directly copied to at least one second data server and the log server without considering the logical structure or format of the operation log, and the operation logs stored in the log server and at least one second data server are consistent with the operation logs stored in the first data server.

[0072] In some embodiments, the first data server and the second data server can switch with each other. For example, during the operation of the system, the first data server fails and crashes, and the second data server is switched to the main machine. The second data server is used to provide data services for the system based on the second data backed up by the first data stored in the first data server.

[0073] In some embodiments, the master node and the backup node can be responsible for different functions of the application. For example, the master node and the backup node can be separated in read and write, with the master node performing write operations based on the application and the backup node performing read operations based on the application. Through division of labor and cooperation, the load pressure of each node in the data backup system is evenly distributed, the load pressure of the master node is reduced, the possibility of the master node crashing due to excessive load pressure is reduced, and the data backup efficiency is improved.

[0074] Optionally, the hardware configurations of the master node and the backup node may be the same or different.

[0075] Illustratively, in order to ensure data consistency between the first data server and the second data server, the same hardware configuration is set for the first data server and the second data server.

[0076] Each node of the data backup system can be an independent server or a server cluster.

[0077] Schematically, in order to ensure the data storage capacity of the first data server and the second data server, the first data server and the second data server are set as a server cluster. Based on the characteristic that the log server only stores operation logs, the log server is set as an independent server. In the strong synchronization scenario, while ensuring the data synchronization efficiency, the resources required for the data backup system are reduced.

[0078] Schematically, please refer to Figure 2, which shows an interaction flow chart of a data backup system provided by an exemplary embodiment of the present application. As shown in Figure 2, the system includes a first data server, a second data server and a log server, and the system interaction includes the following steps.

[0079] Step 211: The first data server obtains at least one operation log.

[0080] The operation log is used to record operation behaviors on the first data, where the first data is data stored in the first data server.

[0081] An operation log is a detailed record of operations performed by a system or application. Operation logs typically include information such as the user, time, operation performed, and operation results.

[0082] Optionally, the functions of the operation log include but are not limited to reviewing operation history, troubleshooting, security auditing and performance analysis. Among them, reviewing operation history means that the operation log can be used to determine the operation status within the historical time period, helping users understand the system's operation status and operation records; troubleshooting means that when a problem or failure occurs in the system, the operation log can help the system administrator or developer determine the root cause of the problem and find the cause of the failure; security auditing means that the operation behavior in the system can be detected through the operation log, and abnormal operations or security threats can be discovered in a timely manner; performance analysis means that by analyzing the operation log, the performance bottleneck of the system can be understood and the system operation efficiency can be optimized.

[0083] In the computer technology field, operation logs are typically stored as text files or database records, allowing for retrieval, analysis, and archiving as needed. The granularity of operation logs can be configured to suit individual needs, allowing for recording detailed operation information or only critical operation events.

[0084] Taking the data backup system providing data services to user terminals as an example, the operation log is generated based on a user request initiated by the user terminal. The first data server receives the user request initiated by the user terminal and generates and obtains at least one operation log based on the user request.

[0085] The user request is used to request the data server to perform a data operation on the first data.

[0086] Optionally, data operations include, but are not limited to, Data Definition Language (DDL) operations and Data Manipulation Language (DML) operations. Data Definition Language (DDL) is a database language used to create, modify, and delete database objects (such as tables, views, indexes, etc.). Data Manipulation Language (DML) is a language used to operate on data in a database, including but not limited to operations such as insert, update, delete, and query, and is used to add, modify, delete, and retrieve data in database tables.

[0087] DDL operations are typically used to create, modify, and delete database objects, such as tables, views, and indexes. DML operations are used to manipulate data in a database, including but not limited to insert, update, delete, and query operations, and are used to add, modify, delete, and retrieve data in database tables.

[0088] In some embodiments, an operation log is associated with a Log Sequence Number (LSN), an ascending sequence number used to uniquely identify the operation log. The LSN can be used to identify the corresponding operation log. The LSN occupies less storage space and allows for quick identification of the corresponding operation log, improving the efficiency of operation log processing.

[0089] Step 212: The first data server updates the first data based on the at least one operation log; in the process of updating the first data, the first data server sends the at least one operation log to the second data server and the log server.

[0090] Specifically, for the second data server and the log server, step 212 includes step 212a, in which the first data server sends at least one operation log to the second data server during the process of updating the first data based on at least one operation log; and step 212b, in which the first data server sends at least one operation log to the log server during the process of updating the first data based on at least one operation log.

[0091] Optionally, step 212a and step 212b may be performed synchronously or asynchronously, which is not limited in this application.

[0092] In some embodiments, when the first data server obtains at least one operation log, the first data server will copy the at least one operation log to the log buffer, run the log writing thread, and write at least one operation log in the log buffer to the log file through the log writing thread. In the process of writing the operation log, the first data is updated based on the at least one operation log, the updated first data is stored, and the updated first data file is obtained.

[0093] The log buffer is an area in the first data server used to store operation logs to be written locally to the first data server. The first data server copies at least one operation log to the area through the first write thread, and then updates the first data stored in the first data server based on the operation log therein.

[0094] Illustratively, the first data stored in the first data server is "User A's account balance: 100", and the operation log is used to indicate the operation "User A's balance increases by 100". While storing the operation log, the first data server performs the calculation corresponding to the operation to obtain the updated first data "User A's account balance: 200". The first data server is used to store the full amount of data, that is, the first data server stores the first data before the update, the operation log for operating the first data, and the updated first data.

[0095] During the data update process, the first data server synchronizes the operation logs with the second data server and the log server by sending the at least one operation log to the second data server and the log server. The second data server contains the full data, meaning that the second data server is used to back up the full data in the first data server, while the log server is used only to back up the operation logs in the first database.

[0096] For example, the second data server has the same initial configuration as the first data server. Assume that both the first data server and the second data server store the data "User A's account balance: 100". The first data server obtains the operation log "User A's balance increases by 100" and sends the operation log to the second data server and the log server. The second data server backs up and stores the operation log, and calculates and stores the updated data "User A's account balance: 200" based on the operation log. The log server does not store the relevant data, but directly backs up and stores the operation log "User A's balance increases by 100".

[0097] Step 221: The second data server receives at least one operation log for backup storage.

[0098] The second data server runs a second writing thread during the data backup process. The second writing thread is a log writing thread for writing operation logs locally to the second data server.

[0099] The second write thread (Second Write Thread) is a log write thread in the second data server used to write the received operation log to the local second data server. The second data server uses this thread to back up and store at least one received operation log to obtain a second log file, which is used to store the full operation log corresponding to the first data server.

[0100] The second data server backs up and stores the received at least one operation log through the second writing thread to obtain a second log file, where the second log file is used to store the full operation log corresponding to the first data server.

[0101] In some embodiments, after writing at least one operation log, the second data server feeds back a log backup result to the first data server, where the log backup result is used to indicate that storage of the at least one operation log in the second data server is complete.

[0102] Step 222: The second data server updates the second data used to back up the first data stored in the second data server based on at least one operation log.

[0103] In some embodiments, the second data server updates the second data based on the at least one operation log by performing the same computing operation as in the first data server.

[0104] When the second data server feeds back the log backup result to the first data server based on the storage status of at least one operation log, the second data server can execute the log backup result feedback process and the second data update process in parallel.

[0105] Optionally, the second data server may also alternately perform the log backup result feedback process and the second data update process at a preset frequency.

[0106] Step 231: The log server receives at least one operation log for backup storage.

[0107] In some embodiments, the log server is used to run a log writing thread, and the log writing thread backs up and stores the received at least one operation log to obtain a log file.

[0108] The log writing thread is used to write log information to a log file. In computer systems, logging is used to record important data such as the system's operating status, operation history, and error messages. The log writing thread's role is to write this log information to the log file according to specific rules for subsequent review, analysis, and troubleshooting.

[0109] In many systems, the log writing thread is usually a separate thread responsible for asynchronously writing log information to disk or other storage media to avoid affecting system performance. By making log writing operations asynchronous, the system can respond to user requests more quickly while ensuring that important log information is not lost.

[0110] Log writing threads typically implement optimization strategies, such as batch writing, buffer management, and log rolling, to improve writing efficiency and minimize the impact on system performance. Furthermore, to ensure log integrity and reliability, log writing threads typically implement fault-tolerance mechanisms, such as retrying write failures and logging errors.

[0111] By properly designing and optimizing log writing threads, the stability, reliability, and performance of the system can be improved.

[0112] During the process of writing the operation log, the log server directly writes to the operation log through the log writing thread. There is no need to execute the calculation process of updating the first data based on the operation log in the first data server, and there is no need to roll back the operation log. The storage of the operation log can be completed quickly, the load pressure on the log server is small, the required computing resources are small, and the response can be quickly fed back to the first data server.

[0113] In step 232 , after successfully backing up and storing the at least one operation log, the log server feeds back a log backup result corresponding to the at least one operation log to the first data server.

[0114] In some embodiments, the log server generates a log backup result each time it successfully stores an operation log, and immediately feeds back the log backup result to the first data server.

[0115] Schematically, at least one operation log includes a first operation log. After the log server writes the first operation log in the log file, it generates a first log backup result corresponding to the first operation log and sends the first log backup result to the first data server. The first log backup result is used to indicate that the first operation log is backed up in the log server.

[0116] In some embodiments, the operation log corresponds to a log sequence number, and the first operation log corresponds to a first log sequence number. The first log result may include the first log sequence number, and the first data server can determine that the first operation log is backed up in the log server through the first log sequence number in the first log backup result.

[0117] Schematically, the log backup result can be implemented as an acknowledgment packet (ACK), which includes but is not limited to a sequence number (Sequence Number), an acknowledgment number (Acknowledgment Number), status information, a checksum, and other data. The sequence number is used to identify the data packet sent by the first data server, the acknowledgment number is used to indicate the sequence number of the next data packet that the log server expects to receive, and the status information is used to indicate the current operating status of the log server, including but not limited to the processing result, synchronization status, delay status, etc. of the log server on at least one operation log. The checksum is used to verify the integrity and correctness of the acknowledgment packet.

[0118] Acknowledgment (ACK) is the data packet that is used to confirm data reception situation in the data communication process.In the data backup system of the present application, log server can feed back the log backup result to the first data server after the operation log is backed up and stored successfully, and this result can be implemented as the confirmation packet.Confirmation packet generally comprises sequence number (can be implemented as log sequence number), confirmation number, status information, checksum and other data, is used to identify the data packet that the first data server sends, represents the sequence number of the next data packet that the log server expects to receive, indicates the current running state of the log server and the integrity and the correctness of the verification confirmation packet.

[0119] The Acknowledgment Number (ACK) is a data item in an ACK packet that indicates the sequence number of the next data packet that the recipient (e.g., the log server) expects to receive. In the data backup system of this application, when the log server sends an ACK packet back to the first data server, the ACK number indicates that the log server has successfully received the data packet with the previous sequence number and expects to receive the next operation log data packet with the specified sequence number, thereby achieving orderly and reliable data transmission.

[0120] A sequence number is a number used to identify a data packet in data communication. In the data backup system of the present application, when the first data server sends an operation log, the operation log is assigned a log sequence number, which can be used as the sequence number in an acknowledgment packet (ACK) to uniquely identify the operation log. Through the sequence number, the first data server can determine the corresponding operation log, as well as the sending and receiving conditions of the operation log, thereby achieving effective management and search of the operation log.

[0121] A checksum is a technology used to verify data integrity. In the data backup system of the present application, an acknowledgment packet (ACK) contains a checksum, which is a value calculated based on the data content in the acknowledgment packet. After receiving the acknowledgment packet, the first data server recalculates the checksum of the acknowledgment packet and compares it with the received checksum. If the two are the same, it indicates that no data errors occurred during the transmission of the acknowledgment packet, and the content of the acknowledgment packet is complete and correct; if they are different, it indicates that the data may have an error during transmission and needs to be processed accordingly, such as retransmission, to ensure data reliability.

[0122] When the operation log corresponds to a log sequence number, the sequence number in the ACK can be implemented as the log sequence number.

[0123] Step 213: The first data server receives the log backup result.

[0124] In some embodiments, the first data server receives the log backup result via a communication network with the log server.

[0125] The first data server may decode the log backup result to obtain a log sequence number in the log backup result, determine a corresponding operation log based on the log sequence number, and determine a backup completion status of the operation log based on the log backup result.

[0126] Illustratively, the first data server determines that the log server has completed backing up the first operation log based on the received first log backup result, and feeds back the completion status of the first operation log to the user terminal.

[0127] The log backup result is used to indicate that at least one operation log is successfully backed up in the log server, and the log backup result is directly generated after the log server writes the operation log, without the log server calculating and storing data based on the operation log.

[0128] In a semi-synchronous scenario, the first data server can feedback the completion status of the operation log to the user terminal based on the log backup results fed back by at least one second data server and / or any one of the log servers. On this basis, due to the characteristic that the log server only stores operation logs, the storage data volume and load pressure are smaller than those of the second data server. Under the same hardware configuration, the log server is usually more efficient in feeding back the log backup results to the first data server than to the second data server. In the semi-synchronous scenario, when the first data server receives the log backup results fed back by the log server, it can directly feed back the completion status of the operation log to the user terminal without waiting for feedback from the second data server, thereby improving data synchronization efficiency.

[0129] In the strong synchronization scenario, after receiving the log backup results for a certain operation log from all backup nodes in the data backup system (including the log server and at least one second data server), the first data server will feedback the completion status of the operation log to the user terminal, which can ensure strong consistency of the data. Based on the characteristic that the log server only stores operation logs, the storage data volume and load pressure are smaller than those of the second data server, which reduces the overall resource consumption demand of the data backup system and the hardware configuration requirements of the log server, and can achieve strong data synchronization with fewer resources.

[0130] In summary, the data backup system provided by the embodiment of the present application performs data backup by adopting a data backup system including a first data server, a log server and at least one second data server, wherein the first data server acts as a master node, and the log server and at least one second data server act as standby nodes, which can achieve strong data synchronization during the data backup process and ensure strong consistency of the data. When the first data server synchronizes data with the second data server, at least one operation log is sent to the log server through the first data server, and the log server receives and backs up and stores the at least one operation log, and feeds back the log backup result to the first data server, introducing a log server that only backs up and stores the operation log during the data synchronization process. Since the log server stores significantly less data than the second data server, the load pressure of the log server is relatively small. The log server is relatively idle compared to the second data server and can promptly feedback responses to the first data server, reducing the time the first data server waits for responses, thereby improving data synchronization efficiency. At the same time, in the scenario of strong data synchronization, by using the log server to replace at least one second data server, only at least one second data server is retained, and there is no need for at least two second data servers to achieve strong data synchronization. Since the load pressure of the log server is relatively small, the overall resource consumption demand of the data backup system is reduced, and the requirements of the log server on the hardware configuration are reduced, so that strong data synchronization can be achieved with fewer resources.

[0131] In some embodiments, the first data server includes a log writing function, a log synchronization function, and a backup detection function during the data backup process, wherein the log writing function is used to write operation logs and data updated based on the operation logs locally on the first data server, the log synchronization function is used to synchronize the operation logs to the second data server and the log server, and the backup detection function is used to detect the backup storage status of the operation logs by the second data server and the log server. Schematically, please refer to Figure 3, which is a flow chart of the first data server interaction method provided by an exemplary embodiment of the present application. As shown in Figure 3, the first data server 300 implements the log synchronization function through the log dump thread 310, implements the log writing function through the first writing thread 320, and implements the backup detection function through the result detection thread 330. Next, multiple functions implemented by the first data server during the data backup process will be described in conjunction with Figure 3.

[0132] (1) Log writing function.

[0133] The first data server runs a first write thread 320 , which is a log write thread in the first data server for writing operation logs locally.

[0134] The first data server copies at least one operation log to the log buffer 301 through the first writing thread. The log buffer is used to store the operation log to be written locally to the first data server.

[0135] The first data server updates the first data stored in the first data server based on at least one operation log in the log buffer 301 , and obtains a full log file 302 storing the operation log and the updated first data.

[0136] The first data server copies at least one operation log to the copy buffer 303 through the first write thread. The copy buffer is used to store the operation logs to be backed up to other servers.

[0137] Illustratively, in the data backup system provided in an embodiment of the present application, the aforementioned other servers include a log server and at least one second data server, and the replication buffer is used to store operation logs to be backed up to the log server and the at least one second data server. The replication buffer is a buffer in the first data server used to store operation logs to be backed up to the log server and the at least one second data server. The first data server copies the operation logs to the replication buffer via a first write thread, and the log dump thread reads the operation logs from the replication buffer and sends them to the log server and the second data server.

[0138] In some embodiments, the first data server further runs a log flushing (log_flusher) thread 340 , which is used to store operation logs on a local disk of the first data server.

[0139] The log flush (log_flusher) thread is a thread in the first data server that stores operation logs on the local disk of the first data server. The local disk of the first data server is a storage medium for persistently storing related data or operation logs. This thread ensures that even in the event of a system crash, the operation logs in the replication buffer can be retained on the local disk, ensuring data reliability and persistence.

[0140] The local disk of the first data server is a storage medium used by the first data server to persistently store relevant data or operation logs. When a system failure occurs, the data in the memory is easily lost, and the relevant data or operation logs stored in the local disk belong to persistent storage. Therefore, the first data server refreshes the operation logs in the replication buffer to the local disk of the first data server through the log refresh thread, thereby ensuring that even in the event of a system crash, the operation logs in the replication buffer can be retained on the local disk, thereby ensuring data reliability and persistence in the event of a system failure or power outage.

[0141] In some embodiments, before running the first write thread, the first data server is further configured to run a user thread 350 . The first data server receives user requests initiated by user terminals through the user thread, and the user thread is further configured to process user requests.

[0142] The first data server generates at least one operation log based on the user request through the user thread. The user thread is a thread in the first data server used to receive user requests initiated by the user terminal and process the user requests. At the same time, the thread is also used to generate at least one operation log based on the user request.

[0143] (2) Log synchronization function.

[0144] The first data server runs a log dump thread 310 , which is used to back up the operation log stored in the first data server to the log server and at least one second data server.

[0145] The log dump thread is a thread in the first data server used to back up the operation log stored in the first data server to the log server and at least one second data server. Based on the communication connection relationship established with the second data server and the log server, the thread reads the operation log from the copy buffer and sends it to the corresponding server.

[0146] The first data server may establish a communication connection relationship with the second data server and the log server based on the log dump thread, and send at least one operation log to the second data server and the log server based on the communication connection relationship.

[0147] The first data server reads at least one operation log from a copy buffer through the log dump thread. The copy buffer is a buffer for storing operation logs to be backed up to the log server and at least one second data server.

[0148] The first data server sends at least one operation log to the second data server and the log server through the log dump thread.

[0149] In order to improve log synchronization efficiency, the first data server can run the first dump thread and the second dump thread in parallel, send at least one operation log to be backed up to the second data server through the first dump thread, and send at least one operation log to be backed up to the log server through the second dump thread.

[0150] In some embodiments, the first data server reads at least one operation log from the copy buffer through a log dump thread according to the log sequence number stored in a sending queue. The sending queue is used to store the log sequence number corresponding to the operation log to be sent. The log sequence number is a sequence number generated when obtaining at least one operation log. The log sequence number is used to identify the above-mentioned at least one operation log.

[0151] The send queue (Send Queue) is a queue in the first data server used to store the log sequence number corresponding to the operation log to be sent. The log sequence number is generated when obtaining at least one operation log and is used to identify the operation log. The log dump thread reads the log sequence number from the queue in sequence, and reads the operation log from the copy buffer according to the read log sequence number for sending.

[0152] Schematically, when the first data server obtains at least one operation log, a log sequence number is generated to identify the operation log, and the log sequence number is stored in a sending queue. The log dump thread reads the log sequence numbers from the sending queue in sequence, and reads at least one operation log from the copy buffer according to the read log sequence number for sending.

[0153] Optionally, after the log dump thread sends at least one operation log to the log server and at least one second data server, the log sequence number corresponding to the at least one operation log is deleted from the sending queue.

[0154] Since the copy buffer is usually used to store the latest operation logs to be backed up, the number of logs is small, and the operation logs corresponding to the log sequence numbers can be read quickly, which improves the log sending speed during the data backup process and improves the data backup efficiency.

[0155] When the copy buffer 303 does not include at least one operation log corresponding to the log sequence number, at least one operation log corresponding to the log sequence number is read from the full log file 302 according to the log sequence number, and the full log file includes the full operation logs stored by the first data server during the historical time period.

[0156] In some embodiments, when the first data server sends at least one operation log to the standby node (including the second data server and the log server), it compresses and encodes the at least one operation log to obtain a log coding sequence, which includes a log sequence number corresponding to the operation log; the first data server sends the log coding sequence corresponding to the at least one operation log to the second data server and the log server through the log dump thread to synchronize at least one operation log to the standby node.

[0157] Since the data backup system provided in the embodiment of the present application includes at least two backup nodes, the master node (first data server) needs to communicate data with each backup node during the data backup process. The communication pressure and network overhead are large, which can easily reduce the data synchronization performance. By further compressing and encoding the operation log and transmitting the log coding sequence to the backup node, the network overhead can be reduced, the transmission efficiency can be improved, and thus the data synchronization performance can be improved.

[0158] (3) Backup detection function.

[0159] In some embodiments, the first data server stores the log sequence number corresponding to at least one operation log in the waiting queue 304 through the user thread 350. The waiting queue is used to store the log sequence numbers of operation logs waiting for backup results.

[0160] The first data server runs a result detection (listener) thread 330, which is used to receive the backup result returned by the second data server and / or the log server based on at least one operation log.

[0161] The result detection (listener) thread is a thread in the first data server used to receive the backup results returned by the second data server and / or the log server based on at least one operation log. The thread parses the log sequence number of the operation log corresponding to the backup result and determines the processing result of the operation log based on whether the sequence number exists in the waiting queue.

[0162] The waiting queue (Wait Queue) is a queue in the first data server used to store the log sequence numbers of operation logs waiting for backup results. The user thread stores the log sequence numbers corresponding to the operation logs in the queue. The result detection thread parses the log sequence numbers of the operation logs corresponding to the backup results. When the sequence number exists in the waiting queue, it determines the processing result of the operation log and can delete the sequence number from the queue.

[0163] The first data server parses the log sequence number of the operation log corresponding to the backup result through the result detection thread. When the waiting queue includes the first log sequence number obtained based on the first backup result analysis, the processing result of the first operation log is determined. The first log sequence number is used to identify the first operation log.

[0164] Schematically, the log sequence number of each operation log is different. During the processing of the operation log, the log sequence number increases sequentially. The first data server will record the log sequence number of the operation log corresponding to the current latest feedback processing result as the submission sequence number. Since the log sequence number always increases, when the first data server detects that the log sequence number obtained by parsing the backup result is greater than or equal to the submission sequence number, the first data server reads the user session corresponding to the log sequence number, and based on the user session, feedback the processing result of the operation log to the corresponding user terminal.

[0165] In some embodiments, after determining the processing result of the first operation log, the first log sequence number is deleted from the waiting queue, and the storage space of the waiting queue can be adjusted immediately to avoid the log sequence number of the processed operation log still occupying storage space, causing resource waste and jamming.

[0166] To ensure high performance, asynchronous optimization is performed on the first data server. Specifically, a response waiting (User ACK) thread pool is introduced to improve concurrent processing capabilities and reduce waiting time.

[0167] The answer waiting (User ACK) thread pool is a thread pool introduced in the first data server to improve concurrent processing capabilities and reduce waiting time. The answer waiting thread corresponds to the user request initiated by the user terminal. The thread waits for the result detection thread to determine the processing result of the operation log corresponding to the user request, and after determining the result, it feeds back to the user terminal and releases the thread corresponding to the user request, so that the user thread can continue to process other operations without waiting for the backup result.

[0168] The answer waiting thread (User ACK Thread) is a thread in the answer waiting (User ACK) thread pool. It corresponds to the user request initiated by the user terminal. It is used to wait for the result detection thread to determine the processing result of the operation log corresponding to the user request. After determining the processing result, it feeds back the result to the user terminal and releases itself to improve the concurrent processing capability of the first data server.

[0169] The first data server runs a response waiting thread 360 , which corresponds to the user request initiated by the user terminal.

[0170] Optionally, the first data server may run a response waiting thread corresponding to the operation log after the user thread stores the log sequence number corresponding to the at least one operation log in the waiting queue.

[0171] The first data server determines the processing result of the operation log corresponding to the user request through the response waiting thread and the waiting result detection thread. After determining the processing result corresponding to the user request, the first data server feeds back the processing result corresponding to the user request to the user terminal and releases the response waiting thread corresponding to the user request.

[0172] In some embodiments, the first data server can release the corresponding user thread based on the response waiting thread, so that the user thread does not need to wait for the backup result and can continue to process other operation logs after pulling up the response waiting thread, thereby improving concurrent processing capabilities.

[0173] Illustratively, a first data server runs the i-th user thread based on the i-th user request initiated by a user terminal; runs the i-th response waiting thread corresponding to the i-th user request based on the i-th user thread; releases the i-th user thread after generating at least one operation log based on the i-th user request and storing it locally; runs the i+1-th user thread based on the i+1-th user request; runs the i+1-th response waiting thread corresponding to the i+1-th user request based on the i+1-th user thread; and releases the i-th response waiting thread after feeding back a processing result corresponding to the i-th user request to the user terminal. Where i is a positive integer.

[0174] During the strong synchronous replication process, the steps of waiting for the response from the standby machine, which originally needed to be executed serially, are now switched to a specific thread in the User ACK thread pool. In this way, the main working thread (user thread) can be released in advance to handle other user connections without waiting for the response from the standby machine. The introduction of this thread pool model enables the first data server to more effectively utilize system resources and improve concurrent processing capabilities. The process of waiting for the response from the standby machine is carried out in a dedicated thread, avoiding blocking the main working thread, thereby reducing waiting time and improving overall performance and response speed. In general, by introducing the thread pool model, the first data server parallelizes the steps of waiting for the response from the standby machine, which were originally executed serially, thereby improving concurrent processing capabilities, reducing waiting time and optimizing performance. This optimization measure enables the first data server to handle write operations more efficiently when implementing the strong synchronous replication solution.

[0175] For illustration, please refer to FIG4 , which is a flow chart showing a comparison of effects provided by an exemplary embodiment of the present application. As shown in FIG4 , the concurrency of the first data server can be improved by introducing a response waiting thread to release the user thread. Specifically, the following two process comparisons are included.

[0176] (1) In process 410, for operation log T1, the master database needs to wait for multiple solid line interactions to complete before releasing the user thread. Multiple rounds of waiting are required to release the user thread (solid line part), and cross-machine interactions must be completed before a response can be given. This includes the master user thread writing the redo log, the log dump thread initiated by the standby database reading the redo log and sending it to the standby database's input / output (IO) thread, and the standby database IO thread writing the relay log. After writing, it confirms (ACK) to the master user thread before it is considered complete and the user thread can be released.

[0177] A relay log is a file in the standby database that stores logs received from the primary database. During data backup and synchronization, the standby database's log dump thread reads logs from the primary database's replication buffer and sends them to the standby database's input / output (IO) thread. The standby database's IO thread then writes the received logs to the relay log. The standby database then processes the contents of the relay log, parsing the logs and updating the standby database's data based on the log contents to synchronize data with the primary database. It is a crucial component of the standby database's storage and processing of primary database logs during the data synchronization process.

[0178] The redo log is a log used to record data change operations in the primary database. During the primary database's operational flow, the host user thread records data change operations as redo logs. These logs record data modification operations such as inserts, updates, and deletes. Redo logs are an important component for ensuring data consistency and recoverability. In the event of a system failure or data recovery, these data change operations can be re-executed using the redo logs to restore the data to its correct state. The redo logs in the primary database are subsequently copied to the global log buffer and further sent to the standby database and log server via the log dump thread to achieve data backup and synchronization.

[0179] (2) In process 420, after asynchronous optimization, the master database only needs to acknowledge the waiting thread (User ACK thread) to maintain the session after completing the write operation (write), and the user thread can be released, thereby improving concurrency performance. The thread can be released after completing the write operation for operation log T2. Other processes (dashed line) do not need to wait for threads. This includes the master database user thread (user thread) writing the redo log, and the worker thread can be released after the writing is completed.

[0180] Through the asynchronous transformation process, the data backup system's performance can be brought close to that of native asynchronous replication. Compared to semi-synchronous replication, the transformed strong synchronous replication adds almost no additional time overhead in the event of network jitter in Internet Data Centers (IDCs).

[0181] To sum up, the system provided by the embodiment of the present application realizes the log writing function, log synchronization function and backup detection function through the first data server serving as the main node in the data backup system. By introducing a response waiting thread instead of the user thread waiting for the backup node to feedback the log backup result, while ensuring data consistency, the user thread can be released in time, the parallelism of the user thread is improved, and the efficiency of the user thread in processing user requests is improved.

[0182] In some embodiments, a full data backup of a first data server is performed via a second data server. For illustration, please refer to FIG. 5 , which is a flow chart of an interaction method for a second data server provided in an exemplary embodiment of the present application. As shown in FIG. 5 , the data backup system includes a first data server, a second data server, and a log server. The second data server performs the following steps during the data backup interaction process.

[0183] Step 501: Receive at least one operation log for backup storage.

[0184] The second data server runs a second writing thread 510 during the data backup process. The second writing thread is a log writing thread for writing operation logs locally to the second data server.

[0185] The second data server backs up and stores the received at least one operation log through the second writing thread to obtain a second log file, where the second log file is used to store the full operation log corresponding to the first data server.

[0186] Step 502: Update second data used to back up the first data stored in the second data server based on at least one operation log.

[0187] The second data is backup data of the first data stored in the first data server. The second data is kept consistent with the first data through data backup, and the second data can be updated based on at least one operation log.

[0188] The second data server runs a log application (log apply) thread 520 during the data backup process. The log application thread includes a coordination thread and at least one working thread. The coordination thread can be implemented as a log coordinator (Log Coordinator).

[0189] The log apply thread is a thread running on the second data server during the data backup process, which includes a coordination thread and at least one worker thread. The coordination thread can be implemented as a log coordinator, which is used to read and parse at least one operation log in batches from the second log file, and assign the operation log to the worker thread. The worker thread then updates the second data stored in the second data server based on the corresponding operation log.

[0190] The coordinator thread is part of the log apply thread and can be implemented as a log coordinator. It is used in the second data server to read and parse at least one operation log in batches from the second log file, and assign these operation logs to at least one worker thread to update the second data.

[0191] A worker thread is part of a log apply thread and is used in the second data server to update the second data stored in the second data server based on the corresponding operation log. Multiple worker threads can be executed concurrently to improve data update efficiency, and ultimately obtain an updated second data file so that it is consistent with the updated first data file in the first data server.

[0192] The second data server reads and parses at least one operation log in batches from the second log file through the coordination thread, and distributes the at least one operation log to at least one working thread.

[0193] The second data server updates the second data stored in the second data server based on the corresponding operation log through at least one working thread to obtain an updated second data file.

[0194] The updated second data file is consistent with the updated first data file in the first data server.

[0195] In some embodiments, after the second data server stores at least one operation log, it feeds back a log backup result to the first data server.

[0196] Schematically, at least one operation log includes a first operation log. After storing the first operation log, the second data server generates a first backup result corresponding to the first operation log and sends the first backup result to the first data server. The first backup result is used to indicate that the first operation log is backed up in the second data server.

[0197] Optionally, feeding back the log backup result to the first data server and updating the second data based on the operation log can be performed in parallel, thereby improving backup feedback efficiency.

[0198] Illustratively, in the process of updating the second data based on at least one operation log, a log backup result corresponding to the at least one operation log is fed back to the first data server.

[0199] The system provided by the embodiment of the present application uses a second data server as a backup node in the data backup system to perform full data backup during the log backup process based on at least one operation log sent by the first data server, thereby reducing the communication pressure on the first data server and improving data backup efficiency.

[0200] In some embodiments, an embodiment of the present application provides a data backup method for a log server based on the above-mentioned data backup system. Please refer to Figure 6. Figure 6 is a flow chart of the data backup method for a log server provided by an exemplary embodiment of the present application. The method can be executed by a terminal, or by a server, or by both a terminal and a server. The embodiment of the present application takes the method as an example of being executed by a log server in a data backup system. As shown in Figure 6, the method includes the following steps.

[0201] Step 610: Receive at least one operation log sent by the first data server.

[0202] The first data server is used to obtain at least one operation log, and the operation log is used to record operation behavior on the first data.

[0203] The first data server is a main server in the data backup system, and the log server is used to perform a full operation log backup on the first data server.

[0204] Step 620: Back up and store at least one operation log.

[0205] In some embodiments, the log server runs a log writing thread during the data backup process, and the log writing thread is used to write an operation log locally on the log server.

[0206] The log server backs up and stores the received at least one operation log through a log writing thread to obtain a log file.

[0207] Step 630: Generate a log backup result after completing the backup storage of at least one operation log.

[0208] Illustratively, at least one operation log includes a first operation log. After the first operation log is written into the log file, a first log backup result corresponding to the first operation log is generated.

[0209] Step 640: Send the log backup result to the first data server.

[0210] The log backup result is used to indicate that at least one operation log has been backed up in the log server.

[0211] Illustratively, a first log backup result is sent to the first data server, where the first log backup result is used to indicate that the first operation log is completely backed up in the log server.

[0212] Since the log server only synchronizes logs, the central processing unit (CPU) of the log server consumes very low resources, about 80% of a single core and less than 8G of memory.

[0213] To sum up, the data backup method provided in the embodiment of the present application, through the log server in the data backup system, only backs up the operation log during the data backup process. Under the same hardware configuration, it can achieve fast backup response compared to the second data server, thereby improving the backup response efficiency; in the strong synchronization scenario, while cooperating with the second data server and the first data server to ensure strong data consistency, due to the low resource consumption of the log server, the hardware configuration requirements are low, thereby reducing the overall overhead of the data backup system and improving data synchronization efficiency.

[0214] In some embodiments, an embodiment of the present application provides a data backup method based on the above-mentioned data backup system. Please refer to Figure 7. Figure 7 is a flow chart of a data backup method provided by an exemplary embodiment of the present application. The method can be executed by a terminal, or by a server, or by both a terminal and a server. The embodiment of the present application takes the method as an example of being executed by the first data server in the data backup system. As shown in Figure 7, the method includes the following steps.

[0215] Step 710: Obtain at least one operation log.

[0216] The operation log is used to record operation behaviors on the first data.

[0217] In some embodiments, the first data server generates the at least one operation log based on a user request initiated by a user terminal.

[0218] For details, please refer to the relevant description of the first data server in the above data backup system, which will not be repeated here.

[0219] Step 720: Update the first data based on the at least one operation log; in the process of updating the first data, send the at least one operation log to the log server and the at least one second data server.

[0220] The second data server is used to back up the operation log and the first data stored in the first data server, and the log server is used to back up the operation log stored in the first data server.

[0221] The first data is data stored in the first data server. After writing at least one operation log, the first data server needs to perform corresponding data operations on the first data based on the operation log to update the first data.

[0222] For details, please refer to the relevant instructions of the above-mentioned data backup system, which will not be repeated here.

[0223] Step 730: Receive a first log backup result sent by the log server after successfully storing at least one operation log.

[0224] The first log backup result is used to indicate that at least one operation log is completely backed up in the log server.

[0225] In some embodiments, after step 720 above, the first data server further receives a second log backup result sent by at least one second data server.

[0226] The second log backup result is used to indicate that the backup of at least one operation log in the second data server is complete.

[0227] In some embodiments, the first data server further determines a backup status of at least one operation log based on the log backup result.

[0228] The first data server performs subsequent data processing based on the backup status of the operation log, or feeds back the processing result of the operation log to the user terminal.

[0229] Taking the data backup method provided in the embodiment of the present application as an example of being applied to a data semi-synchronization scenario, the first data server can immediately feedback the processing result of the operation log to the user terminal upon receiving the log backup result feedback from any standby node (including the log server and at least one second data server). There is no need to wait for all standby nodes to feedback the log backup result before feedbacking the processing result to the user terminal. This can improve data synchronization efficiency, avoid the user terminal waiting for a long time for the first data server to feedback the processing result, and ensure user experience.

[0230] Taking the data backup method provided in the embodiment of the present application as an example of being applied to a data strong synchronization scenario, the first data server needs to wait for the log backup results fed back by all standby nodes (including the log server and at least one second data server). Upon receiving the first log backup result and the second log backup result, the processing result of at least one operation log is fed back to the user terminal. Strong data synchronization is achieved through the first data server, the log server and at least one second data server, ensuring strong data consistency. Since at least two second data servers are usually required to participate in data backup in a data strong synchronization scenario, the data backup method provided in the embodiment of the present application uses a log server to replace at least one second data server. Based on the characteristic that the log server only stores operation logs, the resource consumption of the log server is less than that of the second data server, which reduces the resource consumption of the data backup method provided in the embodiment of the present application during the data strong synchronization process as a whole, ensures data consistency, and improves data synchronization efficiency.

[0231] To sum up, the method provided in the embodiment of the present application synchronizes operation logs to the log server and at least one second data server through the first data server in the data backup system, and reduces the response waiting time of the first data server in the data semi-synchronization scenario by receiving the log backup results fed back by the log server, thereby improving the data synchronization efficiency; in the data strong synchronization scenario, it ensures data consistency, reduces the resource consumption of data synchronization, and improves the data synchronization efficiency.

[0232] Schematically, please refer to FIG8 , which is a diagram of a distributed database architecture provided by an exemplary embodiment of the present application. As shown in FIG8 , the distributed database includes the following modules.

[0233] Management Module 810 is a module in a distributed database, consisting of a scheduler cluster and a decision-making (Zookeeper) cluster. The scheduler cluster helps database administrators (DBAs) or database users automatically schedule and run various types of jobs, such as database backups, master-slave switchover, and task scheduling. The decision-making cluster's primary functions include configuration maintenance, election decisions, and route synchronization. It also assists with storing routing and control information. The management module is also used to pull and execute DDL tasks from the decision-making cluster, obtain status and generate expansion tasks, control master-slave switchover, and implement disaster recovery for multiple schedulers.

[0234] The Scheduler cluster is part of the management module and is used to help database administrators (DBAs) or database users automatically schedule and run various types of jobs, such as database backups, master-slave switchover, task scheduling, resource scheduling, resource increase and decrease management in replication instances, collection and detection, report generation, or business process execution. By integrating with Zookeeper and the Operations Support System (OSS), it uses time windows to activate specified resource plans, meeting the database's complex needs in resource management and job scheduling.

[0235] The decision-making (Zookeeper) cluster is part of the management module. Its main functions are configuration maintenance, election decision-making, routing synchronization, etc. It assists in storing routing, control information, task information, heartbeat information, and supports the creation, deletion, and replacement of database node groups (shards). Cluster deployment requires more than or equal to 3 groups and cross-computer room deployment. The management module can obtain status from the decision-making, pull DDL tasks, and perform related operations.

[0236] The management module includes: (1) Scheduler cluster helps database administrators (DBAs) or database users automatically schedule and run various types of jobs, such as database backup, master-slave switching, task scheduling, resource scheduling, resource increase and decrease management in replication instances, collection and detection, generation of various reports, or execution of business processes, etc. The distributed database combines Scheduler, Zookeeper, and Operations Support System (OSS) to activate specified resource plans through time windows to meet the various complex requirements of the database in resource management and job scheduling. (2) Decision-making (Zookeeper) The main functions of the cluster are configuration maintenance, election decision-making, routing synchronization, etc. It also assists in storing routing, control information, task information, heartbeat information, etc. Zookeeper supports the creation, deletion, and replacement of database node groups (shards). Cluster deployment requires more than or equal to 3 groups and is deployed across computer rooms.

[0237] The Operations Support System (OSS) is a system used in conjunction with the scheduler cluster in a distributed database. It activates specified resource plans through time windows to assist in completing various complex database resource management and job scheduling requirements, such as database backup, master-slave switching, task scheduling, resource scheduling, etc., helping database administrators (DBAs) or database users manage and operate databases more efficiently.

[0238] In some embodiments, the management module 810 is used to: 1. pull DDL tasks from decisions and execute them on actual database instances; 2. obtain status from decisions and generate capacity expansion tasks; 3. control the master-slave switching within the data set unit; 4. multiple schedulers themselves achieve disaster recovery through decision elections.

[0239] Database (DB) module 820: This module consists of data set (Set) units, including data nodes and agent software. Data nodes are data storage nodes. In high-availability scenarios, a Set contains one primary node and multiple secondary nodes. The agent is a bypass module that primarily monitors the health of Data nodes and reports heartbeats to Zookeeper. It also reports on Data node resource usage, table size, access counts, and other information. It monitors master-slave replication and data synchronization, and performs master-slave switchover, migration, table consistency checks, and mirror backups.

[0240] A data node (Data Node) is a data storage node in the data set (Set) unit of the database (Data Base, DB) module. In a high-availability scenario, a Set contains one primary node and multiple secondary nodes for data storage. The primary node accepts all read and write requests, while the secondary node accepts read requests and backs up data. The secondary node can also detect instance and table status, pull and execute migration tasks, and participate in the primary-backup switching process.

[0241] A table consistency check is an operation that verifies the data consistency of tables in a database. In the database (DB) module of this application, the agent software (Agent) performs table consistency checks to ensure that the data in the tables on the primary and backup nodes remains consistent, avoiding data inconsistencies and ensuring database system stability and data accuracy.

[0242] Schematically, as shown in FIG8 , the database module 820 includes M data set units, each of which includes 1 master node and n backup nodes, where M and n are positive integers.

[0243] The database (DB) module is a component of a distributed database and consists of dataset (Set) units, including data nodes and agent software. Data nodes are data storage nodes. In high-availability scenarios, a Set consists of one primary node and multiple secondary nodes. The agent is a bypass module, primarily responsible for monitoring the health of Data nodes, reporting heartbeats and node resource information, monitoring master-slave replication and data synchronization, and performing master-slave switchovers. Standby nodes are also used to monitor instance and table status, pull and execute migration tasks, and participate in the master-slave switchover process.

[0244] In some embodiments, the above-mentioned standby node is used to: 1. detect instance status and report it to the decision-making office; 2. detect table status and report it to the decision-making office; 3. pull migration tasks from the decision-making office and execute them; 4. participate in the active-standby switching process.

[0245] Optionally, the above-mentioned multiple data set units can be used to process the same data backup task, or can be divided to process different data backup tasks based on different system functions. For example, data set unit 1 is used to back up the financial account data of the user terminal, and data set unit M is used to back up the game data of the user terminal.

[0246] Database engine (SQLEngine) module 830: The main functions are account authentication, connection management, SQL parsing, routing allocation, etc. SQL Engine can be deployed on a database node (Data Node) in a mixed manner, or it can be deployed independently on a physical machine. SQLEngine also adopts a distributed architecture design to provide parallel load capabilities. It contains a proxy module that supports the MySQL protocol and encapsulates functions such as permission verification, read-write separation, lexical analysis, syntax analysis, routing selection, protocol adaptation, distributed transaction control, and aggregation functions. In a distributed scenario, the SQL engine needs to handle distributed transactions and maintain a global auto-increment field to ensure that multiple data and multiple storage nodes share a globally auto-incrementing sequence; if it is distributed, it will restrict some syntax, including lexical and syntax parsing; and in some complex calculations, it also needs to do some SQL pushdown and final data aggregation.

[0247] Read-Write Separation is a database architecture design strategy that separates database read and write operations onto different nodes for processing. In the data backup system of this application, the first data server serves as the primary database, responsible for handling write operations; the second data server and log server serve as backup databases, responsible for handling read operations. Read-Write Separation can reduce the read pressure on the primary database, improve the read performance of the application, and effectively utilize the resources of the backup database, improving the overall performance and concurrency of the application.

[0248] Account authentication is a key function of the SQLEngine module, used to verify the legitimacy and permissions of user accounts. When a user accesses the database, the SQLEngine module verifies the provided account and password to confirm that the user has access permissions. Only users who have passed account authentication can perform subsequent database operations, which helps ensure database security and prevent unauthorized access and data leaks.

[0249] SQL parsing is one of the core functions of the SQLEngine module, used to analyze and understand SQL statements entered by users. This process includes lexical analysis and syntax analysis, breaking down SQL statements into lexical units, checking their syntax for correctness, and then converting them into an internal representation that the database can execute. Through SQL parsing, the database engine module can determine the SQL statement type (such as query, insert, update, etc.) and the operation target, providing the basis for subsequent operations such as routing selection and execution plan generation.

[0250] Route allocation is a function of the SQLEngine module. It distributes user requests to appropriate data nodes for processing based on the SQL parsing results and the database architecture. In a distributed database system, different data nodes are responsible for storing and processing different data. Route allocation considers factors such as data distribution and node load to ensure efficient request processing and improve system performance and availability.

[0251] A distributed architecture is an architectural design approach that distributes the different components of a system across multiple computer nodes. In this application, the database engine (SQLEngine) module uses a distributed architecture design, which enables parallel load balancing. This architecture distributes the workload across multiple nodes, improving the system's concurrent processing capabilities and scalability while enhancing its reliability and fault tolerance. Each node communicates and collaborates over a network to jointly complete the system's functions.

[0252] A distributed transaction refers to a transaction operation involving multiple database nodes or services in a distributed system. In the distributed scenario of the SQLEngine module, a distributed transaction occurs when a transaction operates on data from multiple data nodes. The database engine module handles distributed transactions, such as maintaining global auto-increment fields to ensure that multiple data nodes share a global auto-increment sequence to ensure data consistency and integrity. It also requires distributed transaction control, coordinating operations between nodes to ensure that transactions are either successfully committed on all nodes or rolled back on all nodes, avoiding situations where some nodes succeed while others fail, thereby ensuring data consistency and reliability.

[0253] Connection management is a function of the SQLEngine module, primarily responsible for managing connections between users and the database. This includes operations such as establishing connections, maintaining connection status, and releasing connections. When multiple users are accessing the database concurrently, connection management effectively controls the number and usage of connections, preventing system resource exhaustion caused by excessive connections. It also ensures that each user's connection can interact with data stably and efficiently, improving database system performance and availability.

[0254] The Proxy module, part of the SQLEngine database engine, supports proxying of the MySQL protocol. This module encapsulates functionality such as permission verification, read-write splitting, lexical analysis, syntax analysis, routing, protocol adaptation, distributed transaction control, and aggregate functions. Within the database system, it connects and processes client requests, coordinates internal database operations, and facilitates efficient database operation and management, meeting various data access and processing requirements in distributed scenarios. An aggregate function is a database function that performs calculations on a set of data and returns a single result. Within the Proxy module, aggregate functions are encapsulated to perform statistical analysis on database data, such as calculating sums, averages, maximums, and minimums. Aggregate functions allow users to aggregate and process large amounts of data, providing more valuable information and meeting data analysis and report generation needs.

[0255] Lexical analysis is a step in the SQL statement processing process by the proxy and database engine modules. After receiving the SQL statement from the client, the proxy module performs lexical analysis on the SQL statement, breaking it down into lexical units (tokens), such as keywords, identifiers, constants, and operators. Lexical analysis converts the textual form of the SQL statement into an internal representation that the computer can understand and process, providing the foundation for subsequent steps such as syntax analysis and semantic analysis. Lexical analysis is a crucial step in ensuring that SQL statements can be correctly parsed and executed. It identifies the various components of the SQL statement for further syntax and semantic verification and processing.

[0256] Syntax analysis is a step that further processes SQL statements based on lexical analysis. After completing lexical analysis and obtaining the lexical units of the SQL statement, the proxy module or database engine module performs syntax analysis. The purpose of syntax analysis is to check whether the SQL statement conforms to the grammatical rules of the SQL language, such as whether the statement structure is correct and whether keywords are used appropriately. If the SQL statement contains syntax errors, the syntax analysis process will detect and report the error, preventing further execution of the SQL statement. Syntax analysis ensures the correctness and legality of the SQL statement, providing a reliable foundation for subsequent semantic analysis and execution.

[0257] Route selection is a function of the proxy and SQLEngine modules. In a distributed database system, when a client sends a request, the proxy or database engine module must select an appropriate data node to process the request based on the request content and the database architecture. For example, a read request might be handled by a less-loaded backup node to achieve load balancing; write requests are typically sent to the primary node for processing. The route selection process considers multiple factors, such as data node load and data distribution, to ensure efficient request processing while maintaining system performance and availability. Appropriate route selection can optimize database system resource utilization, improve system response speed, and enhance concurrent processing capabilities.

[0258] Protocol adaptation is one of the functions of the proxy module, used to convert and adapt between different protocols. In a database system, clients and database nodes may communicate using different protocols. The proxy module uses protocol adaptation to convert client requests into the protocol supported by the database node and convert responses returned by the database node into the protocol expected by the client. This enables normal communication between the client and database node, ensuring compatibility and interoperability between different protocols.

[0259] SQL Pushdown is an operational strategy used by the SQLEngine module in distributed scenarios. When performing complex calculations, the database engine module pushes down portions of the SQL statement's computational tasks to the data nodes for execution. This approach leverages the computing resources of each data node, reduces data transmission across the network, and improves data processing efficiency. For example, when performing operations such as data aggregation, the aggregation calculations are pushed down to the data nodes. After preliminary local data processing, the data nodes return only the results to the database engine module, rather than transmitting all the raw data to the database engine module for processing. This reduces network overhead and improves overall system performance.

[0260] Permission Verification is a function of the Proxy module, used to verify client access rights to the database. In a database system, different users or clients may have different access rights. For example, some users may only be able to read data, while others may be able to insert, update, and delete data. When the Proxy module receives a request from a client, it performs a permission check to determine whether the client has the required permissions. If the client's permissions are insufficient, the Proxy module rejects the request to ensure database security and data integrity, preventing unauthorized access and manipulation from damaging the database.

[0261] A global auto-increment field is a globally unique, incremental field shared by multiple data nodes in a distributed database system. In the distributed scenario of the database engine (SQLEngine) module of this application, it is necessary to maintain a global auto-increment field to ensure that the value of this field remains globally unique and increments when multiple data nodes perform operations such as data insertion, avoiding data conflicts and duplications and ensuring data consistency and integrity.

[0262] In some embodiments, the database engine module 830 is used to: 1. identify DDL operations and save them in the decision as tasks; 2. identify DML operations, perform database language conversion, and send them to the master node or backup node in the corresponding data set unit; 3. collect responses from each node in the data set unit, combine and process them, and return them to the front-end database interface protocol; 4. detect decisions, pull routing, permissions and other information.

[0263] Schematically, the data backup system provided in an embodiment of the present application can be implemented as a data set unit in the above-mentioned database module 820. Please refer to Figure 9, which is a schematic diagram of a data backup system provided by an exemplary embodiment of the present application. As shown in Figure 9, the data backup system includes a main database 910, a backup database 920 and a log server 930.

[0264] The primary replica 910 has the full data and accepts all read and write requests. Log copying, writing logs to disk, transaction commit waiting, and log confirmation processes are performed in the primary replica 910. Log copying: The logs generated by the transaction are copied to the global log buffer. Writing logs to disk: A log writing thread writes the logs in the log buffer to disk and copies them to a circular queue in the replication buffer. Transaction commit waiting: When a transaction is ready to commit, it does not immediately return an OK packet. Instead, the transaction is added to a pending commit queue, waiting for the log to be sent to any standby database or log server. Log confirmation: Once the standby database or log server returns an ACK, the transaction is actually committed and an OK packet is sent to the client.

[0265] The Global Log Buffer (GLB) is an area within the master database used to store transaction logs. In the master database, transaction logs are first copied to this GLB. The log writing thread then writes the logs from the GLB to disk and copies them to the replication buffer's circular queue. It is a crucial component of the master database's log processing process, temporarily storing log data and acting as a data buffer and transfer point, facilitating subsequent log processing and storage.

[0266] The Pending Commit Queue (Pending Commit Queue) is a queue in the primary database that stores transactions ready for commit. When a transaction is ready to commit, it does not immediately return an OK packet. Instead, it is added to the Pending Commit Queue, awaiting log transmission to any slave database or log server. Only after the slave database or log server returns an ACK is the transaction actually committed from the Pending Commit Queue and an OK packet is sent to the client. This queue manages the transaction commit process, ensuring that transactions are committed only after logs have been successfully backed up to the slave database or log server, thereby ensuring data consistency and reliability.

[0267] The secondary replica 920 holds the full data set and can accept read requests. Log storage, log processing, and log application are performed in the secondary replica 920. Log storage: After receiving the log, the secondary replica stores it on its local disk and returns an ACK to the primary replica. Log processing: When the secondary replica receives a new log, the local log coordination thread reads and parses the new log. It then hashes the logs based on page numbers and assigns them to different worker threads. Log application: Worker threads concurrently apply the logs to update the data in the secondary replica.

[0268] In some embodiments, to avoid data loss, the standby database 920 writes data to disk. The standby database 920 includes a buffer pool for storing data pages read from disk by the standby database 920. When the database needs to read data, the data page is first loaded into the buffer pool. If the data page already exists in the buffer pool, the standby database 920 can read the data directly from memory without having to access the disk again. The existence of the buffer pool can improve database performance because memory access speed is much faster than disk access speed. By caching data pages in memory, the database can reduce the number of disk accesses, thereby speeding up data reading.

[0269] The buffer pool is an area in the standby database (secondary data server) used to store data pages read from disk. When the database needs to read data, the data page is first loaded into the buffer pool. If the data page already exists in the buffer pool, the standby database can read the data directly from memory without having to access the disk again. Its size is usually configurable and is used to improve database performance, reduce the number of disk accesses, and speed up data reading.

[0270] The size of the buffer pool is usually configurable. The standby database 920 can adjust the size of the buffer pool according to actual needs and system resources to improve database performance.

[0271] In some embodiments, a log dump thread is created on the master database 910 through the slave database 920 . The log dump thread is responsible for reading logs from the Copy Buffer and sending the read logs from the master database to the slave database.

[0272] The log replica 930 backs up logs. Similar to the backup database 920 in log processing, it does not require log application and is primarily used for high availability. Because the log replica only synchronizes logs, CPU usage is very low, around 80% per core, and memory usage is within 8GB.

[0273] The data backup system provided by the embodiment of the present application realizes the strong synchronous replication function of concurrent execution. Through concurrent execution, multiple write operations can be processed simultaneously, the waiting time of write operations is reduced, and the concurrency performance of the system is improved. This is particularly important for database systems under high-load environments and can better meet the needs of users. Concurrent execution can reduce the delay of write operations because there is no need to wait for the previous write operation to be completed before the next write operation can be executed. In this way, data can be copied to the backup database faster, improving the real-time and consistency of data. Strong synchronous replication with concurrent execution can make better use of system resources and improve the scalability of the system by processing multiple write operations in parallel, which means that the system can better cope with increased load and data volume and maintain high performance and reliability. Concurrent execution can reduce user waiting time and improve user response speed and experience. Users can complete write operations faster and see the results immediately without having to wait for a long time. Strong synchronous replication with concurrent execution enables the database system to process write operations more efficiently and maintain data consistency and reliability by improving the concurrent performance of the system, reducing delays, improving scalability and enhancing user experience.

[0274] In summary, the data backup system includes a first data server, a log server, and at least one second data server. The first data server receives a user request initiated by a user terminal and generates an operation log based on the request. The operation log records the operation behavior on the first data. The first data server then updates the first data based on the operation log. During the update process, the operation log is sent to the second data server and the log server. The second data server receives the operation log for backup storage and, based on these operation logs, updates the second data stored on itself for backing up the first data. The log server receives the operation log sent by the first data server for backup storage. After successful storage, it feeds back the log backup results to the first data server, which then receives the log backup results. Through this system architecture, strong data synchronization is achieved during the data backup process, ensuring strong data consistency. The introduction of a log server that only backs up and stores operation logs has a low load pressure and can provide timely feedback, reducing the waiting time of the first data server and improving data synchronization efficiency. In addition, using the log server to replace at least one second data server reduces resource consumption and lowers the requirements for hardware configuration, achieving strong data synchronization with fewer resources.

[0275] The first data server runs a log dump thread, which is responsible for backing up the operation logs stored in the first data server to the log server and at least one second data server. The log dump thread reads the operation logs from the copy buffer, which specifically stores the operation logs to be backed up to the log server and at least one second data server. After reading, the log dump thread sends the operation logs to the second data server and the log server. Through the setting of the log dump thread and the copy buffer, the backup transmission of the operation logs in the first data server to other servers is realized, and the copy buffer stores the operation logs to be backed up, which can quickly read the operation logs for sending, thereby improving the log sending speed during the data backup process and thus improving the data backup efficiency.

[0276] The first data server reads the operation log from the copy buffer through the log dump thread according to the log sequence number stored in the sending queue. The sending queue stores the log sequence number corresponding to the operation log to be sent. These log sequence numbers are generated in association when the operation log is obtained and are used to identify the operation log. When the copy buffer does not include the operation log corresponding to the log sequence number, the first data server reads the corresponding operation log from the full log file according to the log sequence number. The full log file contains the full operation log stored by the first data server in the historical time period. By using the log sequence number to identify the operation log and storing the sequence number of the operation log to be sent in the sending queue, the operation log can be read from the copy buffer in a more orderly manner. When the corresponding log does not exist in the copy buffer, it is read from the full log file, which ensures the integrity of the operation log reading, thereby ensuring the normal progress of data backup and improving data backup efficiency.

[0277] The first data server compresses and encodes the operation log to generate a log code sequence containing the corresponding log sequence number. This log code sequence is then sent to the second data server and the log server via the log dump thread. By compressing and encoding the operation log, the amount of data transmitted is reduced, lowering the network overhead for data communication between the primary and backup nodes, improving transmission efficiency, and thus enhancing data synchronization performance.

[0278] The first data server runs a first write thread, which copies the operation log to the log buffer through the thread. The log buffer stores the operation log to be written locally to the first data server. Then, based on the operation log in the log buffer, the first data stored in the first data server is updated to obtain a full log file storing the operation log and the updated first data. At the same time, the first write thread copies the operation log to the copy buffer, which is used to store the operation log to be backed up to the log server and at least one second data server. Through the setting of the first write thread, the log buffer and the copy buffer, the writing of the operation log locally to the first data server and the update of the first data are realized, and preparations are made for the backup storage of the operation log to other servers, ensuring the orderly data processing and data backup of the first data server and improving the efficiency of data processing and backup.

[0279] The first data server runs a log refresh thread, which flushes the operation logs in the replication buffer to the first data server's local disk. This log refresh thread ensures that even in the event of a system crash, the operation logs in the replication buffer are retained on the local disk. This ensures data reliability and durability, and prevents data backup and recovery impacted by the loss of operation logs due to system failures.

[0280] The first data server runs a user thread, which receives user requests initiated by the user terminal and generates an operation log based on the user requests. This user thread allows the first data server to receive and process user requests and generate an operation log, providing a foundation for subsequent data updates and backups, enabling the data backup system to respond to user operations and perform corresponding data processing.

[0281] The first data server stores the log sequence number corresponding to the operation log in a waiting queue through a user thread, and the waiting queue is used to store the log sequence number of the operation log waiting for the backup result. Then, the first data server runs a result detection thread, which receives the backup result returned by the second data server or the log server based on the operation log, and parses the log sequence number of the operation log corresponding to the backup result. When the waiting queue includes the first log sequence number parsed based on the first backup result, the processing result of the first operation log is determined, and then the first log sequence number is deleted from the waiting queue. Through the setting of the waiting queue and the result detection thread, the backup result of the operation log can be managed in an orderly manner, the processing status of the operation log can be determined in time, and the sequence number of the processed operation log can be deleted at the same time, so as to avoid taking up storage space and improve the efficiency of data processing and resource utilization.

[0282] The first data server runs a response waiting thread, which corresponds to the user request initiated by the user terminal. The response waiting thread waits for the result detection thread to determine the processing result of the operation log corresponding to the user request. After determining the processing result, it feeds back the result to the user terminal and releases itself. By setting up the response waiting thread, while ensuring data consistency, the response waiting thread replaces the user thread in waiting for the backup node to feedback the log backup result. This can timely release the user thread, improve the parallelism of the user thread, and thus improve the efficiency of the user thread in processing user requests, thereby enhancing the concurrent processing capability of the system.

[0283] The first data server runs the i-th user thread based on the i-th user request initiated by the user terminal, runs the i-th response waiting thread corresponding to the i-th user request based on the i-th user thread, and releases the i-th user thread after generating at least one operation log based on the i-th user request for local storage. Then, the first data server runs the i+1-th user thread based on the i+1-th user request, runs the i+1-th response waiting thread corresponding to the i+1-th user request based on the i+1-th user thread, and releases the i-th response waiting thread after feeding back the processing result corresponding to the i-th user request to the user terminal, where i is a positive integer. Through this thread running and releasing mechanism, the processing flow of the first data server for user requests is further optimized, the concurrent processing capability is improved, the thread waiting time is reduced, and the system can process multiple user requests more efficiently, thereby improving the user experience.

[0284] The second data server runs a second write thread, which backs up and stores the received operation logs to obtain a second log file. At the same time, the second data server runs a log application thread, which includes a coordination thread and at least one working thread. The coordination thread reads and parses the operation logs from the second log file in batches, and assigns the operation logs to the working threads. The working threads update the second data stored in the second data server based on the corresponding operation logs to obtain an updated second data file. Through the setting of the second write thread and the log application thread, the second data server can effectively back up and store the operation logs, and update the second data based on the operation logs, thereby ensuring the consistency of the second data with the first data. At the same time, through the division of labor and cooperation between the coordination thread and the working thread, the efficiency of data updating is improved, the communication pressure of the first data server is reduced, and the data backup efficiency is improved.

[0285] After storing the first operation log, the second data server generates a first backup result corresponding to the first operation log and sends the first backup result to the first data server. This first backup result indicates that the first operation log has been backed up on the second data server. By providing timely feedback on the backup result of the first operation log, the first data server can promptly understand the backup status of the operation log on the second data server, facilitating subsequent data processing and decision-making, further improving the efficiency and reliability of data backup.

[0286] The log server runs a log writing thread, and backs up and stores the received operation logs through the thread to obtain a log file. After the first operation log is written in the log file, a first log backup result corresponding to the first operation log is generated, and the first log backup result is sent to the first data server. The first log backup result is used to indicate that the first operation log is backed up in the log server. Since the log server only synchronizes logs, the CPU resource consumption of the log server is very low, about 80% for a single core and less than 8G of memory. By running the log writing thread on the log server and only backing up the operation log, under the same hardware configuration, it can achieve fast backup response and improve backup response efficiency compared to the second data server. In a strong synchronization scenario, the second data server and the first data server are cooperated to ensure strong data consistency. Since the log server consumes less resources and has low hardware configuration requirements, the overall overhead of the data backup system is reduced and the data synchronization efficiency is improved.

[0287] The first data server obtains an operation log, which records the operation behavior on the first data, and then updates the first data based on the operation log. During the update process, the operation log is sent to the log server and at least one second data server. Then, the first log backup result sent by the log server after the operation log is successfully stored is received, and the second log backup result sent by at least one second data server is also received. In the data semi-synchronization scenario, when the first data server receives the log backup result feedback from any standby node (including the log server and at least one second data server), it immediately feeds back the processing result of the operation log to the user terminal, without having to wait for all standby nodes to feed back the log backup result before feeding back the processing result to the user terminal. This can improve data synchronization efficiency, avoid the user terminal waiting for a long time for the first data server to feed back the processing result, and ensure user experience. In the data strong synchronization scenario, the first data server needs to wait for the log backup result feedback from all standby nodes (including the log server and at least one second data server). When the first log backup result and the second log backup result are received, the processing result of the operation log is fed back to the user terminal. Strong data synchronization is achieved through the first data server, the log server and at least one second data server, ensuring strong data consistency. Since at least two second data servers are usually required to participate in data backup in the data strong synchronization scenario, this solution uses a log server to replace at least one second data server. Based on the characteristic that the log server only stores operation logs, the resource consumption of the log server is less than that of the second data server. As a whole, the resource consumption of this solution in the data strong synchronization process is reduced, data consistency is guaranteed, and data synchronization efficiency is improved.

[0288] FIG10 is a block diagram of a data backup device according to an exemplary embodiment of the present application. As shown in FIG10 , the device includes the following parts:

[0289] The processing module 1010 is configured to obtain at least one operation log, where the operation log is configured to record an operation on the first data;

[0290] a sending module 1020 configured to update the first data based on the at least one operation log; and during the process of updating the first data, sending the at least one operation log to a log server and at least one second data server, the second data server being configured to back up the operation log and the first data stored in the first data server, and the log server being configured to back up the operation log stored in the first data server;

[0291] The receiving module 1030 is configured to receive a first log backup result sent by the log server after the at least one operation log is successfully stored.

[0292] In some embodiments, the receiving module 1030 is further used to receive a second log backup result sent by the at least one second data server, where the second log backup result is used to indicate that the at least one operation log is completely backed up in the second data server.

[0293] In some embodiments, the processing module 1010 is further configured to feed back a processing result of the at least one operation log to the user terminal upon receiving the first log backup result and the second log backup result.

[0294] To sum up, the device provided in the embodiment of the present application, through the log server in the data backup system, only backs up the operation log during the data backup process. Under the same hardware configuration, it can achieve fast backup response compared to the second data server, thereby improving the backup response efficiency; in the strong synchronization scenario, while cooperating with the second data server and the first data server to ensure strong data consistency, due to the low resource consumption of the log server, the hardware configuration requirements are low, thereby reducing the overall overhead of the data backup system and improving data synchronization efficiency.

[0295] It should be noted that the data backup device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0296] Figure 11 shows a block diagram of a terminal 1100 according to an exemplary embodiment of the present application. Terminal 1100 may be a smartphone, tablet computer, MP3 player, MP4 player, laptop computer, or desktop computer. Terminal 1100 may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other similar names.

[0297] Typically, the terminal 1100 includes a processor 1101 and a memory 1102 .

[0298] The processor 1101 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1101 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 1101 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1101 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1101 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0299] The memory 1102 may include one or more computer-readable storage media, which may be non-transitory. The memory 1102 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1102 is used to store at least one instruction, which is executed by the processor 1101 to implement the data backup method provided in the method embodiment of the present application.

[0300] In some embodiments, the terminal 1100 also includes other components. Those skilled in the art will understand that the structure shown in Figure 11 does not constitute a limitation on the terminal 1100, and it may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0301] An embodiment of the present application further provides a computer device, which can be implemented as a terminal or server as shown in FIG1 . The computer device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set, or instruction set, and the at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor to implement the data backup methods provided in the above-mentioned method embodiments.

[0302] An embodiment of the present application also provides a computer-readable storage medium, which stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the data backup method provided by the above-mentioned method embodiments.

[0303] Embodiments of the present application further provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data backup methods provided in the above-mentioned method embodiments.

[0304] Optionally, the computer-readable storage medium may include: Read Only Memory (ROM), Random Access Memory (RAM), Solid State Drives (SSD), or an optical disk. Among them, the random access memory may include resistance random access memory (ReRAM) and dynamic random access memory (DRAM). The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0305] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0306] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0307] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A data backup system, comprising a first data server, a log server, and at least one second data server; The first data server is configured to obtain at least one operation log, where the operation log is configured to record an operation on the first data; updating the first data based on the at least one operation log; During the process of updating the first data, sending the at least one operation log to the second data server and the log server; The at least one second data server is configured to receive the at least one operation log for backup storage; updating second data for backing up the first data stored in the second data server based on the at least one operation log; The log server is configured to receive the at least one operation log sent by the first data server for backup storage; After successfully backing up and storing the at least one operation log, feeding back a log backup result corresponding to the at least one operation log to the first data server; and The first data server is further configured to receive the log backup result.

2. According to the system of claim 1, the first data server is further used to run a log dump thread, and the log dump thread is used to back up the operation log stored in the first data server to the log server and the at least one second data server; read the at least one operation log from a copy buffer through the log dump thread, and the copy buffer is a buffer for storing operation logs to be backed up to the log server and the at least one second data server; and send the at least one operation log to the second data server and the log server through the log dump thread.

3. According to the system of claim 2, the first data server is further used to read the at least one operation log from the copy buffer through the log dump thread according to the log sequence number stored in the sending queue, the sending queue is used to store the log sequence number corresponding to the operation log to be sent, the log sequence number is a sequence number associated and generated when obtaining the at least one operation log, and the log sequence number is used to identify the at least one operation log; when the at least one operation log corresponding to the log sequence number is not included in the copy buffer, the at least one operation log corresponding to the log sequence number is read from the full log file according to the log sequence number, and the full log file includes the full operation log stored by the first data server in the historical time period.

4. According to the system of claim 3, the first data server is further used to compress and encode the at least one operation log to obtain a log coding sequence, wherein the log coding sequence includes the log sequence number corresponding to the operation log; and send the log coding sequence corresponding to the at least one operation log to the second data server and the log server through the log dump thread.

5. According to the system described in any one of claims 1 to 4, the first data server is also used to run a first write thread; copy the at least one operation log to a log buffer through the first write thread; update the first data stored in the first data server based on the at least one operation log in the log buffer to obtain a full log file storing the operation log and the updated first data; copy the at least one operation log to a copy buffer through the first write thread, and the copy buffer is used to store the operation log to be backed up to the log server and the at least one second data server.

6. The system according to claim 5, wherein the first data server is further configured to run a log refresh thread, and refresh the operation log in the copy buffer to the local disk of the first data server through the log refresh thread.

7. The system according to any one of claims 1 to 5, wherein the first data server is further configured to run a user thread; receive a user request initiated by a user terminal through the user thread, and generate the at least one operation log based on the user request.

8. The system according to claim 7, wherein the first data server is further configured to store the log sequence number corresponding to the at least one operation log in a waiting queue through the user thread; The first data server is further configured to run a result detection thread, wherein the result detection thread is configured to receive a backup result returned by the second data server or the log server based on the at least one operation log; Parsing the log sequence number of the operation log corresponding to the backup result through the result detection thread; When the waiting queue includes a first log sequence number obtained based on the first backup result analysis, determining a processing result of the first operation log, the first log sequence number is used to identify the first operation log; and deleting the first log sequence number from the waiting queue.

9. According to the system of claim 8, the first data server is also used to run a response waiting thread, and the response waiting thread has a corresponding relationship with the user request initiated by the user terminal; the response waiting thread waits for the result detection thread to determine the processing result of the operation log corresponding to the user request; after determining the processing result corresponding to the user request, the processing result corresponding to the user request is fed back to the user terminal; and the response waiting thread corresponding to the user request is released.

10. The system according to claim 9, wherein the first data server is further configured to run an i-th user thread based on the i-th user request initiated by the user terminal; run an i-th response waiting thread corresponding to the i-th user request based on the i-th user thread; release the i-th user thread after generating at least one operation log based on the i-th user request for local storage; run an i+1-th user thread based on the i+1-th user request; run an i+1-th response waiting thread corresponding to the i+1-th user request based on the i+1-th user thread; and release the i-th response waiting thread after feeding back a processing result corresponding to the i-th user request to the user terminal; i is a positive integer.

11. The system according to any one of claims 1 to 10, wherein the second data server is further configured to run a second writing thread; and the second writing thread backs up and stores the at least one received operation log to obtain a second log file; The second data server is also used to run a log application thread, which includes a coordination thread and at least one worker thread; the coordination thread reads and parses the at least one operation log in batches from the second log file, and assigns the at least one operation log to the at least one worker thread; the at least one worker thread updates the second data stored in the second data server based on the corresponding operation log to obtain an updated second data file.

12. The system according to claim 11, wherein the at least one operation log comprises a first operation log; The second data server is further used to generate a first backup result corresponding to the first operation log after storing the first operation log; and send the first backup result to the first data server, where the first backup result is used to indicate that the first operation log is backed up in the second data server.

13. The system according to any one of claims 1 to 12, wherein the at least one operation log comprises a first operation log; The log server is further configured to run a log writing thread; the log writing thread backs up and stores the received at least one operation log to obtain a log file; After writing the first operation log into the log file, generating a first log backup result corresponding to the first operation log; The first log backup result is sent to the first data server, where the first log backup result is used to indicate that the first operation log is completely backed up in the log server.

14. A data backup method, the method being executed by a first data server in a data backup system, the method comprising: Obtain at least one operation log, where the operation log is used to record an operation behavior on the first data; updating the first data based on the at least one operation log; During the process of updating the first data, sending the at least one operation log to a log server and at least one second data server, the second data server being used to back up the operation log and the first data stored in the first data server, and the log server being used to back up the operation log stored in the first data server; and Receive a first log backup result sent by the log server after successfully storing the at least one operation log.

15. The method according to claim 14, further comprising: A second log backup result sent by the at least one second data server is received, where the second log backup result is used to indicate that the at least one operation log is completely backed up in the second data server.

16. The method according to claim 15, further comprising: When the first log backup result and the second log backup result are received, a processing result of the at least one operation log is fed back to the user terminal.

17. A data backup device, comprising: a processing module, configured to obtain at least one operation log, where the operation log is used to record an operation behavior on the first data; a sending module, configured to update the first data based on the at least one operation log; During the process of updating the first data, sending the at least one operation log to a log server and at least one second data server, the second data server being used to back up the operation log and the first data stored in the first data server, and the log server being used to back up the operation log stored in the first data server; and The receiving module is configured to receive a first log backup result sent by the log server after the at least one operation log is successfully stored.

18. A computer device comprising a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the data backup method according to any one of claims 14 to 16.

19. A computer-readable storage medium, wherein at least one computer program is stored in the storage medium, and the at least one computer program is loaded and executed by a processor to implement the data backup method according to any one of claims 14 to 16.

20. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the data backup method according to any one of claims 14 to 16 is implemented.

Images