Data synchronization method, device and server-side equipment of distributed table system

Abstract

The present application provides a data synchronization method, device and server-side equipment for a distributed table system, which relates to the field of data processing technology and improves the data synchronization efficiency between multiple copies in a distributed table. The method includes: receiving a log synchronization request sent by a storage server where a master copy is located, the log synchronization request includes the first maximum number of the log to be synchronized and the log applied in the storage server where the master copy is located; synchronizing the log to be synchronized based on the log synchronization request; sending a synchronization success message to the storage server where the master copy is located, so that the storage server where the master copy is located applies the log according to the synchronization success message, and the synchronization success message includes the second maximum number of the locally synchronized log.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data synchronization method, device and server-side equipment of a distributed table system. Background Art

[0002] At present, in the process of data storage, in order to ensure data security and alleviate the problems caused by bad disks and server downtime, multiple copies are generally used to store data. A distributed table with multiple copies can use one data copy as the master copy and other data copies as slave copies.

[0003] However, in distributed tables, the degree of data synchronization between multiple copies is difficult to control, resulting in low data synchronization efficiency between multiple copies, which can easily affect the service availability of the master-slave state system in the distributed database. Summary of the invention

[0004] The object of the present invention is to provide a data synchronization method, apparatus and server device for a distributed table system, so as to improve the data synchronization efficiency between multiple copies in a distributed table.

[0005] In a first aspect, an embodiment of the present application provides a data synchronization method for a distributed table system, wherein the distributed table system includes multiple storage servers, and the method is applied to the storage servers, and the method includes:

[0006] Receiving a log synchronization request sent by the storage server where the primary copy is located, the log synchronization request including the log to be synchronized and the first largest number of the logs applied in the storage server where the primary copy is located;

[0007] Synchronize the log to be synchronized based on the log synchronization request;

[0008] Send synchronization success information to the storage server where the primary copy is located, so that the storage server where the primary copy is located applies the log according to the synchronization success information, wherein the synchronization success information includes the second largest number of the locally synchronized logs.

[0009] In a possible implementation, after the step of synchronizing the log to be synchronized based on the log synchronization request, the method further includes:

[0010] The logs in the local logs whose numbers are less than or equal to the first maximum number are applied, and the second maximum number of the locally synchronized logs is determined.

[0011] In a second aspect, a data synchronization method for a distributed table system is provided, wherein the distributed table system includes a plurality of storage servers, and the method is applied to the storage servers, and the method includes:

[0012] Receive a write request sent by the client;

[0013] Generate a master copy log according to the write request, and determine a log to be synchronized based on the master copy log;

[0014] Sending a log synchronization request to the storage server where the slave copy is located in the distributed table system, the log synchronization request including the first largest number of the log to be synchronized and the locally applied log, so that the storage server where the slave copy is located synchronizes the log to be synchronized;

[0015] Receiving synchronization success information sent by the storage server where the slave copy is located, the synchronization success information includes the second largest number of the logs synchronized in the storage server where the slave copy is located;

[0016] The log is applied according to the synchronization success information.

[0017] In a possible implementation, the step of sending a log synchronization request to the storage server where the slave copy is located in the distributed table system includes:

[0018] The logs to be synchronized are sent in batches according to a preset number to the storage server where the slave copies are located in the distributed table system.

[0019] In a possible implementation, the step of sending the logs to be synchronized in batches according to a preset number to the storage server where the slave copies are located in the distributed table system includes:

[0020] According to the ascending order of the numbers of the logs to be synchronized, the logs to be synchronized are sent in batches according to a preset number to the storage server where the slave copy is located.

[0021] In a possible implementation, the step of applying the log according to the synchronization success information includes:

[0022] According to the synchronization success information, the application is performed on the logs in the local logs whose numbers are greater than the first maximum number and less than or equal to the second maximum number.

[0023] In a possible implementation, the number of storage servers where the slave replicas sending the synchronization success information are located is multiple; among all the storage servers where the slave replicas are located, the second largest number of the synchronized logs in the storage servers that meet the preset conditions is determined as the target maximum number;

[0024] After receiving the synchronization success information sent from the storage server where the replica is located, the method further includes:

[0025] It is determined that the logs corresponding to the numbers less than or equal to the target maximum number are synchronized successfully, so as to apply the synchronization to the logs in the local logs whose numbers are greater than the first maximum number and less than or equal to the target maximum number.

[0026] In a possible implementation, after the step of determining that the logs corresponding to the logs with numbers less than or equal to the target maximum number are successfully synchronized and applying the steps to the logs in the local logs with numbers greater than the first maximum number and less than or equal to the target maximum number, the step further includes:

[0027] The value of the first maximum number is updated to the value of the target maximum number, and a write success response is returned to the client.

[0028] In a third aspect, a data synchronization device for a distributed table system is provided, wherein the distributed table system includes a plurality of storage servers, and the device is applied to the storage servers, including:

[0029] A receiving module, configured to receive a log synchronization request sent by a storage server where a primary copy is located, wherein the log synchronization request includes a log to be synchronized and a first maximum number of logs applied in the storage server where the primary copy is located;

[0030] A synchronization module, used for synchronizing the log to be synchronized based on the log synchronization request;

[0031] The sending module is used to send synchronization success information to the storage server where the primary copy is located, so that the storage server where the primary copy is located applies the log according to the synchronization success information, and the synchronization success information includes the second largest number of the locally synchronized logs.

[0032] In a fourth aspect, a data synchronization device for a distributed table system is provided, wherein the distributed table system includes a plurality of storage servers, and the device is applied to the storage servers, including:

[0033] A first receiving module, used for receiving a write request sent by a client;

[0034] A generation module, configured to generate a primary copy log according to the write request, and determine a log to be synchronized based on the primary copy log;

[0035] A first sending module is used to send a log synchronization request to a storage server where a slave copy is located in the distributed table system, wherein the log synchronization request includes a first maximum number of the log to be synchronized and the locally applied log, so that the storage server where the slave copy is located synchronizes the log to be synchronized;

[0036] A second receiving module, configured to receive synchronization success information sent by the storage server where the slave copy is located, wherein the synchronization success information includes a second maximum number of the logs synchronized in the storage server where the slave copy is located;

[0037] An application module is used to apply the log according to the synchronization success information.

[0038] In the fifth aspect, an embodiment of the present application further provides a server-side device, including a memory and a processor, wherein the memory stores a computer program that can be executed on the processor, and when the processor executes the computer program, the method described in the first aspect or the second aspect is implemented.

[0039] In the sixth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores machine-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to execute the method described in the first aspect or the second aspect above.

[0040] The embodiments of the present application bring the following beneficial effects:

[0041] The embodiments of the present application provide a data synchronization method, apparatus and server-side device for a distributed table system, which are capable of receiving a log synchronization request including the first maximum number of logs to be synchronized and logs applied in the storage server where the primary copy is located, and then synchronizing the logs to be synchronized. After that, a synchronization success message including the second maximum number of locally synchronized logs is sent to the storage server where the primary copy is located, so that the storage server where the primary copy is located can apply the logs according to the synchronization success message. In this scheme, while synchronizing log records between the storage server where the primary copy is located and the storage server where the slave copy is located, the first maximum number of applied logs in the storage server where the primary copy is located and the second maximum number of synchronized logs in the storage server where the slave copy is located are used to realize log update and application synchronization between the primary copy and the slave copy, thereby enhancing the real-time synchronization of data between the primary copy and the slave copy, and improving the data synchronization efficiency between multiple copies in the distributed table. In this way, the synchronization method of the application log is optimized, so that the service availability of the master-slave state system in the distributed table is not affected.

[0042] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 A schematic diagram of a data synchronization method for a distributed table system provided in an embodiment of the present application;

[0045] Figure 2 A flowchart of another method for synchronizing data in a distributed table system provided in an embodiment of the present application;

[0046] Figure 3 A schematic diagram of the structure of a data synchronization device for a distributed table system provided in an embodiment of the present application;

[0047] Figure 4 A schematic diagram of the structure of another data synchronization device for a distributed table system provided in an embodiment of the present application;

[0048] Figure 5 The present invention shows a schematic diagram of the structure of a server device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0050] The terms "including" and "having" and any variations thereof mentioned in the embodiments of the present application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0051] At present, in distributed table systems, in order to ensure data security and tolerate bad disks and server downtime, multiple copies are generally used to store data. In a distributed table system with multiple copies, one data copy can be used as the primary copy (leader) and other data copies as followers. In order to maintain data consistency between the primary copy and the follower copy, the raft consistency protocol can be used for data synchronization. The data write operation first writes the log of the primary copy. The storage server where the primary copy is located synchronizes the log to the storage server where the follower copy is located. After most of the copies in the distributed table system have successfully written the log, the storage server where the primary copy is located will apply the log to the data engine of the storage server where the primary copy is located, and then return the write success to the client.

[0052] In various types of raft-based distributed table systems, the following solutions can be used for the process of synchronizing logs from the storage server where the primary replica is located to the storage server where the secondary replica is located, and for the process of applying logs to the data engine after the storage server where the secondary replica is located synchronized logs.

[0053] Solution 1: No matter how the storage server where the primary copy is located synchronizes logs with the storage server where the secondary copy is located, the storage server where the secondary copy is located always returns success to the storage server where the primary copy is located immediately after the log synchronization is successful, and the application of the log is handled by the background thread.

[0054] Solution 2: The storage server where the primary copy is located synchronizes at most one log to the storage server where the secondary copy is located at a time. After the storage server where the secondary copy is located successfully synchronizes the logs, it starts to apply the logs, applies the applicable logs to the data engine, and then returns the successful synchronization of the logs to the storage server where the primary copy is located.

[0055] Solution 3: The storage server where the primary copy is located synchronizes as many logs as possible to the storage server where the secondary copy is located each time. After the storage server where the secondary copy is located successfully synchronizes the logs, it starts to apply the logs, applies the applicable logs to the data engine, and then returns the successful synchronization of the logs to the storage server where the primary copy is located.

[0056] However, the above solutions all have their own shortcomings.

[0057] Solution 1: When switching to the master, a slave becomes the new master. After the new master is generated, it starts to accept write requests. Each write request writes a record to its own log, and then synchronizes it to the storage server where the slave is located. After confirming that most replicas have this log record, it starts to apply the log and then returns it to the client. Each time the storage server where the master is located applies a log, it needs to apply all previously unapplied logs to the data engine. However, the log application of the storage server where the slave is located was always completed by the background thread. There may be a lot of unapplied logs accumulated on the storage server where the slave is located. At this time, the slave is switched to the master, and a large number of accumulated logs need to be applied immediately, which takes too much time. As a result, the write request at this time fails due to a long period of time without response, affecting the availability of the service.

[0058] Solution 2: Even if it is possible to avoid a decrease in service availability when switching to the primary replica, when the amount of data written per second is very large, at most one log is synchronized to the storage server where the secondary replica is located each time, resulting in a waste of network and CPU resources, making the storage server where the primary replica is located respond more slowly to write requests and increasing system write latency.

[0059] Solution 3: When the amount of data written per second is very large, the storage server where the master copy is located may synchronize too many logs to the storage server where the slave copy is located at one time. This causes two effects: 1. Because the storage server where the master copy is located synchronizes too much data at one time, the synchronization thread takes too long, affecting the synchronization efficiency of other storage servers where the master copies are located, causing the write request delay on other storage servers where the master copies are located to increase (the synchronization tasks of the storage server where the master copy is located and other storage servers where the master copies are located may share a synchronization thread); 2. The next time the storage server where the master copy is located synchronizes logs to the storage server where the slave copy is located, the storage server where the slave copy is located synchronizes logs successfully, opens the application log, and applies the last synchronized log to the data engine, causing too many logs to be applied at once, affecting the efficiency of the next log synchronization of the storage server where the master copy is located, causing the write delay to increase or even write timeout and failure.

[0060] Furthermore, in a distributed table, the degree of data synchronization between multiple copies is difficult to control, resulting in low efficiency of data synchronization between multiple copies, which can easily affect the service availability of the master-slave state system in the distributed table.

[0061] Based on this, an embodiment of the present application provides a data synchronization method, apparatus, and server device for a distributed table system, through which the data synchronization efficiency between multiple copies in a distributed table can be improved.

[0062] The embodiments of the present invention are further described below in conjunction with the accompanying drawings.

[0063] Figure 1The present invention provides a flow chart of a data synchronization method for a distributed table system. The distributed table system includes multiple storage servers, and the method is applied to the storage servers. Figure 1 As shown, the method includes:

[0064] Step S110, receiving a log synchronization request sent by a storage server where a primary copy in a distributed table is located.

[0065] The log synchronization request includes the first largest number of the logs to be synchronized and the logs applied in the storage server where the primary copy is located. The number can also be in the form of a label, such as applied_index, which is used to indicate the generation order of the logs in the storage server where the primary copy is located from old to new in time, and the first largest number indicates the latest log applied in the storage server where the primary copy is located.

[0066] In this step, the storage server where the slave copy in the distributed table system is located can receive the logs to be synchronized sent by the storage server where the master copy is located and the first maximum number of the logs applied in the storage server where the master copy is located. For example, the storage server where the master copy is located has a total of 600 logs, and their corresponding numbers are 1 to 600, respectively. The logs to be synchronized are logs numbered from 301 to 450 (301-450), and the first maximum number of the logs applied in the storage server where the master copy is located is 300.

[0067] Of course, the first maximum number can also be any number less than 301 (the minimum number of the log to be synchronized), such as 50, 100, 150 or 200. Since the logs that have not been synchronized cannot be applied, the first maximum number of the applied logs in the storage server where the primary copy is located (such as 50, 100, 150, 200 or 300, etc.) is less than the number of the current log to be synchronized (301 to 450 above). The specific value of the first maximum number of the applied logs is not limited here.

[0068] The numbers of the logs to be synchronized are not specifically limited here. However, it should be noted that the numbers of the logs to be synchronized (such as 301 to 450) are less than the total number of all logs in the storage server where the primary copy is located (such as the 600 logs). After the last batch of logs to be synchronized (such as the last batch of logs to be synchronized are numbered 451 to 600), all logs in the storage server where the primary copy is located (the above 600 logs) are synchronized.

[0069] Step S120: Synchronize the log to be synchronized based on the log synchronization request.

[0070] It should be noted that after receiving a log synchronization request from the storage server where the replica is located, the local log is synchronized and updated according to the log synchronization request, that is, the log to be synchronized (such as the logs numbered 301-450 mentioned above) is recorded and written to the local log file of the storage server where the replica is located, so that the log of the storage server where the replica is located is consistent with the log of the storage server where the primary copy is located (such as synchronized to the logs numbered 301-450).

[0071] After synchronization and update from the storage server where the copy is located, the second maximum number of the locally synchronized logs can be determined from the storage server where the copy is located. For example, after it is determined that all logs with numbers less than or equal to the first maximum number 300 and logs between numbers 301 and 450 have been synchronized and updated, the second maximum number of the synchronized logs in the storage server where the copy is located can be determined to be 450.

[0072] Step S130, sending synchronization success information to the storage server where the primary copy is located, so that the storage server where the primary copy is located applies the log according to the synchronization success information.

[0073] The synchronization success information includes the second largest number of the locally synchronized logs. Exemplarily, the synchronization success information is returned from the storage server where the slave copy is located to the storage server where the master copy is located, and carries the second largest number (450) of the logs synchronized by the storage server where the slave copy is located, such as the number label max_index of the latest log.

[0074] The data synchronization method of the distributed table system in the embodiment of the present application can be used as a data synchronization and application optimization method for a distributed table system. By synchronizing log records between the storage server where the master copy is located and the storage server where the slave copy is located, the first maximum number of the applied log in the storage server where the master copy is located and the second maximum number of the synchronized log in the storage server where the slave copy is located are used to realize log update and application synchronization between the master copy and the slave copy, thereby enhancing the real-time synchronization of data between the master copy and the slave copy, and improving the data synchronization efficiency between multiple copies in the distributed table. In this way, the synchronization method of the application log is optimized, so that the service availability of the master-slave state system in the distributed table is not affected.

[0075] The above steps are described in detail below.

[0076] In some embodiments, after the above step S120, the step may further include: applying the logs in the local logs whose numbers are less than or equal to the first maximum number, and determining the second maximum number of the locally synchronized logs.

[0077] Exemplarily, the storage server where the slave copy is located can also apply logs (such as logs numbered 1-299) whose log numbers are less than or equal to the first maximum number (such as 300 mentioned above) to the local data engine of the storage server where the slave copy is located, that is, the storage server where the slave copy is located applies previously unapplied logs to the data engine of the storage server where the slave copy is located, and thereby applies the logs that have been applied to the data engine of the storage server where the primary copy is located to the data engine of the storage server where the slave copy is located.

[0078] In some embodiments, the master replica may be determined by raft election. As an example, the master replica is a replica determined by raft algorithm based on multiple replicas in a distributed table. In practical applications, a distributed table includes multiple replicas, each replica is used to provide services to the outside world. In practical applications, the master replica can be elected from these replicas by raft algorithm, and the other replicas are slave replicas.

[0079] It should be noted that raft provides a general method for distributing state machines in a computing system cluster, ensuring that each node in the cluster agrees on the same series of state transitions. Raft reaches consensus through the elected master replica. The master replica is responsible for copying logs to the slave replicas, and raft achieves consensus through the leadership method. The cluster has only one elected master replica, which is responsible for managing log replication on other servers in the cluster. The raft algorithm selects the master replica from multiple replicas in the distributed table, so as to quickly and effectively determine the master and slave replicas in the distributed table, so as to effectively carry out the log writing process.

[0080] Figure 2 The present invention provides a flow chart of a data synchronization method for a distributed table system. The distributed table system includes multiple storage servers, and the method is applied to the storage servers. Figure 2 As shown, the method includes:

[0081] Step S210: receiving a write request sent by a client.

[0082] In actual applications, the storage server where the primary copy of the distributed table is located can receive write requests sent by the client.

[0083] Step S220: Generate a primary copy log according to the write request, and determine the log to be synchronized based on the primary copy log.

[0084] Exemplarily, the storage server where the primary copy is located can generate a log record based on the received write request, write it to the local log of the storage server where the primary copy is located, and then start the log synchronization task. The background thread is responsible for synchronizing the log to the storage server where the secondary copy is located.

[0085] Step S230 , sending a log synchronization request to the storage server where the slave copy is located in the distributed table system, so that the storage server where the slave copy is located synchronizes the log to be synchronized.

[0086] The log synchronization request includes the first maximum number of the log to be synchronized and the locally applied log, so that the storage server where the replica is located can apply the log whose number is less than or equal to the first maximum number.

[0087] For example, the storage server where the master copy is located can send the logs to be synchronized (such as the logs numbered 301-450 in the above embodiment) and the first largest number applied_index (such as number 300 in the above embodiment) of the logs that have been locally applied to the data engine of the storage server where the master copy is located to the storage server where the slave copy is located. The logs are applied in order from small to large numbers (that is, from old to new).

[0088] Step S240, receiving synchronization success information sent from the storage server where the replica is located.

[0089] The synchronization success information includes the second largest number of the logs synchronized in the storage server where the slave copy is located. When the storage server where the primary copy is located receives the synchronization success information of the storage server where the slave copy is located, it can obtain the largest number of the logs synchronized in the storage server where the slave copy is located.

[0090] Step S250: applying the log according to the synchronization success information.

[0091] The storage server where the master copy is located applies the local log according to the synchronization success information, so that the consistency of log application between the storage server where the master copy is located and the storage server where the slave copy is located can be achieved.

[0092] By synchronizing log records between the storage server where the master copy is located and the storage server where the slave copy is located, the first maximum number of the applied logs in the storage server where the master copy is located and the second maximum number of the synchronized logs in the storage server where the slave copy is located are used to realize the synchronization of log updates and applications between the storage server where the master copy is located and the storage server where the slave copy is located, so that the real-time synchronization of data between the master copy and the slave copy is enhanced. In this way, the synchronization method of application logs is optimized, so that the service availability of the master state system in the distributed table system is not affected.

[0093] In some embodiments, the storage server where the primary copy is located may send the logs to be synchronized to the storage server where the secondary copy is located according to a preset number of logs. As an example, the above step S230 may include the following steps:

[0094] Step a), sending the logs to be synchronized in batches according to a preset number to the storage server where the slave copy is located in the distributed table system.

[0095] For the sending process of a large number of logs to be synchronized, they can be sent in a certain order. As an example, the above step a) may include the step of sending the logs to be synchronized in batches according to the preset number in the order of the log numbers to be synchronized from small to large to the storage server where the slave copy is located.

[0096] Exemplarily, when the background thread starts synchronizing logs, the logs that have not been synchronized to the storage server where the slave copy is located can be synchronized to the storage server where the slave copy is located in order from old to new, and a maximum of 150 log records are sent to the storage server where the slave copy is located, and the applied_index of the maximum log record applied to the data engine by the storage server where the master copy is located is carried, and the log records are applied in order from small to large (that is, from old to new). The preset number can be adjusted according to the actual application and thread usage, and is not limited here.

[0097] By sending the logs to be synchronized according to the set maximum number of synchronized logs (such as 150) during the log synchronization process between the storage server where the master copy is located and the storage server where the slave copy is located, there will be at most 150 accumulated logs to be applied on the storage server where the slave copy is located when the master is switched, which will not affect the system availability. In addition, by synchronizing logs in batches, the online write delay is reduced, which can reduce the write delay. By controlling the entries of the batch write logs, the write delay can be controlled, so that the write delay will not be too long in extreme cases.

[0098] In some embodiments, the above step S250 may include the following steps: according to the synchronization success information, applying to the logs in the local log whose numbers are greater than the first maximum number and less than or equal to the second maximum number.

[0099] The storage server where the primary copy is located can check for omissions in the application of logs by applying logs in the local log whose numbers are greater than the first maximum number and less than or equal to the second maximum number, so as to achieve consistency in log application between the storage server where the primary copy is located and the storage server where the secondary copy is located.

[0100] In some embodiments, there are multiple storage servers where the slave replicas sending the synchronization success information are located; among all the storage servers where the slave replicas are located, the second largest number of the synchronized logs in the storage servers that meet the preset conditions is determined as the target largest number; after the above step S240, the method further includes the following steps:

[0101] Step b) determines that the logs corresponding to the numbers less than or equal to the target maximum number are successfully synchronized, so as to apply the synchronization to the logs in the local logs whose numbers are greater than the first maximum number and less than or equal to the target maximum number.

[0102] Among them, among all the storage servers where the slave copies are located that send synchronization success information that meet the preset conditions, the preset condition can be to select the second maximum number of the synchronized logs in the storage servers where the slave copies are located that meet the second preset number, and determine the second maximum number as the target maximum number. Exemplarily, after the storage server where the master copy is located receives the synchronization success information (logs numbered 301-450) sent by the storage server where the slave copies are located and the second maximum number of the synchronized logs in the storage server where the slave copies are located, it can determine the logs that meet the second preset number of the storage servers where the slave copies are located from the synchronized maximum numbers sent by all the storage servers where the slave copies are located, and the maximum number of the synchronized logs is committed_index (for example, the above number is 450), for example, the second preset number can be more than 80% of the maximum number of the synchronized logs of the storage servers where the slave copies are located; in actual applications, when the storage server where the master copy is located receives the synchronization success information sent by multiple storage servers where the slave copies are located, it can determine the maximum log index applied by most of the storage servers where the slave copies are located, that is, committed_index, according to the maximum number max_index of the applied logs sent by most of the storage servers where the slave copies are located. The storage server where the primary copy is located then applies the log records in the range of [applied_index, committed_index] (such as logs numbered 301-450) to the data engine of the storage server where the primary copy is located.

[0103] By determining the maximum log number synchronized from the storage server where the replica is located that meets the second preset number from all synchronized maximum numbers sent from the storage server where the replica is located, the second maximum number can be determined more quickly and effectively, thereby improving the synchronization efficiency of the log application.

[0104] In some embodiments, after the above step b), the method may further include the following steps:

[0105] Step c), updating the value of the first maximum number to the value of the target maximum number, and returning a write success response to the client.

[0106] For the above step c), illustratively, after the storage server where the primary copy is located applies the log records in the range of [applied_index, committed_index] to the primary copy data engine, the applied index is updated to be equal to committed_index; finally, a successful write response can be returned to the client.

[0107] In one possible implementation, if the storage server where the master copy is located determines that the log of a storage server where a slave copy is located has been synchronized to the log numbered 450, then when sending log synchronization requests to the storage server where the slave copy is located, it only needs to start from 451, that is, logs numbered before 450 are no longer considered in the synchronization process of the storage server where the slave copy is located.

[0108] The storage server where the primary copy is located can check whether all logs before the maximum log number applied to the storage server where the secondary copy is located have been applied. If not, the application will be executed in the storage server where the primary copy is located. If they have been applied, the application will not be executed again. This can more efficiently achieve consistency in the application level of logs between the storage server where the primary copy is located and the storage server where the secondary copy is located.

[0109] The data synchronization method for a distributed table system provided in the embodiment of the present application has the same technical features as the data synchronization method for a distributed table system provided in the above embodiment, so it can also solve the same technical problems and achieve the same technical effects.

[0110] Figure 3 A structural diagram of a data synchronization device for a distributed table system is provided. The distributed table system includes multiple storage servers, and the device is applied to the storage servers. Figure 3 As shown, the data synchronization device 300 of the distributed table system includes:

[0111] The receiving module 301 is used to receive a log synchronization request sent by the storage server where the primary copy is located, where the log synchronization request includes the log to be synchronized and the first largest number of the logs applied in the storage server where the primary copy is located;

[0112] A synchronization module 302, configured to synchronize the log to be synchronized based on the log synchronization request;

[0113] The sending module 303 is used to send synchronization success information to the storage server where the primary copy is located, so that the storage server where the primary copy is located applies the log according to the synchronization success information. The synchronization success information includes the second largest number of the locally synchronized logs.

[0114] In some embodiments, the apparatus further comprises:

[0115] The application module is used for synchronizing the logs to be synchronized based on the log synchronization request, applying the logs with numbers less than or equal to the first maximum number in the local logs, and determining the second maximum number of the locally synchronized logs.

[0116] The data synchronization device for a distributed table system provided in the embodiment of the present application has the same technical features as the data synchronization method for a distributed table system provided in the above embodiment, so it can also solve the same technical problems and achieve the same technical effects.

[0117] Figure 4 A structural diagram of a data synchronization device for a distributed table system is provided. The distributed table system includes multiple storage servers, and the device is applied to the storage servers. Figure 4 As shown, the data synchronization device 400 of the distributed table system includes:

[0118] The first receiving module 401 is used to receive a write request sent by a client;

[0119] A generating module 402 is used to generate a primary copy log according to a write request, and determine a log to be synchronized based on the primary copy log;

[0120] The first sending module 403 is used to send a log synchronization request to the storage server where the slave replica is located in the distributed table system, where the log synchronization request includes the first maximum number of the log to be synchronized and the locally applied log, so that the storage server where the slave replica is located synchronizes the log to be synchronized;

[0121] The second receiving module 404 is used to receive synchronization success information sent from the storage server where the replica is located, where the synchronization success information includes the second largest number of the logs synchronized from the storage server where the replica is located;

[0122] The application module 405 is used to apply the log according to the synchronization success information.

[0123] In some embodiments, the first sending module 403 is specifically used for:

[0124] The logs to be synchronized are sent in batches according to the preset number to the storage server where the slave replica is located in the distributed table system.

[0125] In some embodiments, the first sending module 403 is further used to:

[0126] In order of the numbers of the logs to be synchronized from small to large, the logs to be synchronized are sent in batches according to the preset number to the storage server where the slave copy is located.

[0127] In some embodiments, the application module 405 is specifically used to:

[0128] According to the synchronization success information, the logs in the local logs whose numbers are greater than the first maximum number and less than or equal to the second maximum number are applied.

[0129] In some embodiments, the number of storage servers where the slave replicas sending the synchronization success information are located is multiple; among all the storage servers where the slave replicas are located, the second largest number of the synchronized logs in the storage servers that meet the preset conditions is determined as the target largest number; the device also includes:

[0130] The determination module is used to determine that the logs corresponding to the log numbers less than or equal to the target maximum number are successfully synchronized after receiving the synchronization success information sent from the storage server where the slave copy is located, so as to apply the logs in the local logs whose numbers are greater than the first maximum number and less than or equal to the target maximum number.

[0131] In some embodiments, the apparatus further comprises:

[0132] The return module is used to update the value of the first maximum number to the value of the target maximum number after determining that the log synchronization corresponding to the number is less than or equal to the target maximum number is successful, and apply it to the logs in the local log whose numbers are greater than the first maximum number and less than or equal to the target maximum number, and return a write success response to the client.

[0133] The data synchronization device for a distributed table system provided in the embodiment of the present application has the same technical features as the data synchronization method and device for a distributed table system provided in the above-mentioned embodiment, so it can also solve the same technical problems and achieve the same technical effects.

[0134] The present application embodiment provides a server device, such as Figure 5 As shown, the server device 500 includes a memory 501 and a processor 502. The memory stores a computer program that can be run on the processor. When the processor executes the computer program, the steps of the method provided in the above embodiment are implemented.

[0135] See also Figure 5 The server device further includes: a bus 503 and a communication interface 504, a processor 502, a communication interface 504 and a memory 501 are connected via the bus 503; the processor 502 is used to execute an executable module stored in the memory 501, such as a computer program.

[0136] The memory 501 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 504 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0137] The bus 503 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0138] Among them, the memory 501 is used to store programs, and the processor 502 executes the program after receiving the execution instruction. The method executed by the device defined by the process disclosed in any embodiment of the present application can be applied to the processor 502 or implemented by the processor 502.

[0139] The processor 502 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 502. The above processor 502 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 501, and the processor 502 reads the information in the memory 501 and completes the steps of the above method in combination with its hardware.

[0140] Corresponding to the data synchronization method of the above-mentioned distributed table system, an embodiment of the present application also provides a computer-readable storage medium, which stores machine-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to execute the steps of the data synchronization method of the above-mentioned distributed table system.

[0141] The data synchronization device of the distributed table system provided in the embodiment of the present application can be specific hardware on the device or software or firmware installed on the device. The implementation principle and technical effects of the device provided in the embodiment of the present application are the same as those of the aforementioned method embodiment. For the sake of brief description, the parts not mentioned in the device embodiment can refer to the corresponding contents in the aforementioned method embodiment. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can all refer to the corresponding processes in the aforementioned method embodiment, and will not be repeated here.

[0142] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0143] For another example, the flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the device, method and computer program product according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0144] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0145] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0146] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence or in other words, the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the data synchronization method of the distributed table system described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0147] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0148] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solution of the present application, rather than to limit it. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the aforementioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solution recorded in the aforementioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solution deviate from the scope of the technical solution of the embodiment of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A data synchronization method for a distributed table system, wherein the distributed table system comprises a plurality of storage servers, characterized in that: The method is applied to the storage server, comprising: Receive a write request sent by the client; Generate a master copy log according to the write request, and determine a log to be synchronized based on the master copy log; Sending a log synchronization request to the storage server where the slave copy is located in the distributed table system, the log synchronization request including the first largest number of the log to be synchronized and the locally applied log, so that the storage server where the slave copy is located synchronizes the log to be synchronized; Receiving synchronization success information sent by the storage server where the slave copy is located, the synchronization success information includes the second largest number of the logs synchronized in the storage server where the slave copy is located; Applying logs according to the synchronization success information; The step of applying the log according to the synchronization success information includes: According to the synchronization success information, applying the logs in the local logs whose numbers are greater than the first maximum number and less than or equal to the second maximum number; There are multiple storage servers where the slave replicas sending the synchronization success information are located; among all the storage servers where the slave replicas are located, the second largest number of the synchronized logs in the storage servers that meet the preset conditions is determined as the target maximum number; After receiving the synchronization success information sent from the storage server where the replica is located, the method further includes: Determine that the logs corresponding to the numbers less than or equal to the target maximum number are successfully synchronized, so as to apply the synchronization to the logs in the local logs whose numbers are greater than the first maximum number and less than or equal to the target maximum number; After the step of determining that the logs corresponding to the numbers less than or equal to the target maximum number are successfully synchronized, and applying the steps to the logs in the local logs whose numbers are greater than the first maximum number and less than or equal to the target maximum number, the method further includes: The value of the first maximum number is updated to the value of the target maximum number, and a write success response is returned to the client.

2. The method according to claim 1, characterized in that The step of sending a log synchronization request to the storage server where the replica is located in the distributed table system includes: The logs to be synchronized are sent in batches according to a preset number to the storage server where the slave copies are located in the distributed table system.

3. The method according to claim 2, characterized in that The step of sending the logs to be synchronized in batches according to a preset number to the storage server where the slave copies are located in the distributed table system includes: According to the ascending order of the numbers of the logs to be synchronized, the logs to be synchronized are sent in batches according to a preset number to the storage server where the slave copy is located.

4. A data synchronization device for a distributed table system, the distributed table system comprising a plurality of storage servers, characterized in that: The device is applied to the storage server and includes: A first receiving module, used for receiving a write request sent by a client; A generation module, configured to generate a primary copy log according to the write request, and determine a log to be synchronized based on the primary copy log; A first sending module is used to send a log synchronization request to a storage server where a slave copy is located in the distributed table system, wherein the log synchronization request includes a first maximum number of the log to be synchronized and the locally applied log, so that the storage server where the slave copy is located synchronizes the log to be synchronized; A second receiving module, configured to receive synchronization success information sent by the storage server where the slave copy is located, wherein the synchronization success information includes a second maximum number of the logs synchronized in the storage server where the slave copy is located; An application module, used for applying logs according to the synchronization success information; The application modules are specifically used for: According to the synchronization success information, apply the logs in the local logs whose numbers are greater than the first maximum number and less than or equal to the second maximum number; The number of storage servers where the slave replicas sending the synchronization success information are located is multiple; among all the storage servers where the slave replicas are located, the second largest number of the synchronized logs in the storage servers that meet the preset conditions is determined as the target largest number; the device also includes: A determination module, configured to determine that the synchronization of logs corresponding to numbers less than or equal to the target maximum number is successful after receiving the synchronization success information sent from the storage server where the slave copy is located, so as to apply the logs in the local logs whose numbers are greater than the first maximum number and less than or equal to the target maximum number; The device also includes: The return module is used to update the value of the first maximum number to the value of the target maximum number after determining that the log synchronization corresponding to the number is less than or equal to the target maximum number is successful, and apply it to the logs in the local log whose numbers are greater than the first maximum number and less than or equal to the target maximum number, and return a write success response to the client.

5. A server device, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 3 are implemented.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores machine executable instructions. When the computer executable instructions are called and executed by a processor, the computer executable instructions prompt the processor to execute the method according to any one of claims 1 to 3.

Images