Data processing method, system, electronic device and computer storage medium

Abstract

An embodiment of the present application provides a data processing method, system, electronic device and computer storage medium, wherein the method is applied to a master node of a distributed system, the distributed system includes a master node, several slave nodes, and a persistent storage area independent of the master node and the slave nodes, and the method includes: in response to a received log synchronization request, assigning a log number to the log to be synchronized corresponding to the log synchronization request according to a rule of increasing number sequence, so as to store the log to be synchronized and the corresponding log number in the persistent storage area in an unordered manner according to the log number; after the log to be synchronized and the corresponding log number are stored, confirming that the log to be synchronized is submitted, and sending the log number and the log to be synchronized to several slave nodes, so that the several slave nodes apply the log to be synchronized.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a data processing method, system, electronic device, and computer storage medium. Background Art

[0002] Typically, to ensure data consistency in a distributed system, a state replicator is used. This means that the state replicator synchronizes logs to enable multiple nodes to start from the same initial state, execute the same commands, and generate a consistent state.

[0003] In a distributed system, both the master and slave nodes can have corresponding stateful replicas. When log synchronization is required, the master node receives the log to be synchronized and sends it to multiple slave nodes. The slave nodes' corresponding state replicas then apply the logs and notify the master node of the completed log application results. Once the master node determines that more than half of the slave nodes have applied the logs and that log rollbacks are not possible, it confirms that the log commit is complete.

[0004] However, when ensuring data consistency through this method, the master node needs to ensure that more than half of the slave nodes will successfully apply the logs. In addition, in order to ensure data accuracy, only the master node can receive the logs to be synchronized, which greatly affects the efficiency and scalability of the distributed system. Summary of the Invention

[0005] In view of this, an embodiment of the present application provides a database data processing solution to at least partially solve the above problems.

[0006] According to a first aspect of an embodiment of the present application, a data processing method is provided, which is applied to a master node of a distributed system, wherein the distributed system includes the master node, several slave nodes, and a persistent storage area independent of the master node and the slave nodes. The method includes: in response to a received log synchronization request, assigning a log number to the log to be synchronized corresponding to the log synchronization request according to a rule of increasing numbering sequence, so as to store the log to be synchronized and the corresponding log number in the persistent storage area in an unordered manner according to the log numbers; after the log to be synchronized and the corresponding log number are stored, confirming that the log to be synchronized is submitted, and sending the log number and the log to be synchronized to the several slave nodes, so that the several slave nodes apply the log to be synchronized.

[0007] According to a second aspect of an embodiment of the present application, there is provided a slave node applied to a distributed system, the distributed system comprising the master node, a storage area, and several slave nodes, the method comprising: generating a number generation request in response to a received log synchronization request, and sending the number generation request to the master node, so that the master node, in response to the number generation request, assigns a log number to the log to be synchronized corresponding to the log synchronization request according to a rule of increasing number sequence; receiving the log number returned by the master node, and storing the log to be synchronized and the corresponding log number in the persistent storage area in an unordered manner according to the log numbers; informing the master node that the storage of the log to be synchronized is completed, so that the master node confirms that the submission of the log to be synchronized is completed, and sending the log number and the log to be synchronized to the master node and several other slave nodes except the current slave node, so that the master node and other slave nodes except the current slave node apply the log to be synchronized in an increasing order of the log numbers.

[0008] According to the third aspect of the embodiment of the present application, a distributed system is provided, including: a master node, several slave nodes, and a storage area; the master node and the slave node are used to receive log synchronization requests; the master node is used to assign log numbers to the logs to be synchronized corresponding to the log synchronization requests according to the rule of increasing number sequence, so that the master node or the slave node that receives the log synchronization request stores the logs to be synchronized and the corresponding log numbers in the storage area in an unordered manner according to the log numbers; the master node is used to confirm that the logs to be synchronized are submitted after the logs to be synchronized and the corresponding log numbers are stored, and send the log numbers and the logs to be synchronized to the master nodes or the slave nodes other than the master node or the slave node that received the log synchronization request, so that other master nodes or the slave nodes apply the logs to be synchronized in the order of increasing log numbers.

[0009] According to the fourth aspect of the embodiments of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the method described in the first aspect.

[0010] According to a fifth aspect of an embodiment of the present application, a computer storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method described above is implemented.

[0011] According to the solution provided by the embodiment of the present application, the master node assigns a log number to the log to be synchronized corresponding to the log synchronization request according to the rule of increasing number sequence, and the log number can be used to ensure that the log is continuous and not missed, and the master node and the slave node can both receive the log synchronization request and write the log to be synchronized and the log number to the persistent storage area in an unordered manner according to the log number. The order requirements of the log numbers of the stored logs are low and concurrent writing is supported, and the log submission is determined to be completed after the write is successful. Compared with receiving the log synchronization request only through the master node, the solution provided by this embodiment greatly improves the log synchronization efficiency and achieves low latency and high concurrency; and, by storing the log to be synchronized and the corresponding log number in a persistent storage area independent of the master node and the slave node, it is ensured that the log will not roll back, so that the process of determining log synchronization is decoupled from the master node and the slave node, thereby making the number of slave nodes have a lower impact on the log synchronization process, thereby improving the scalability of the distributed system. After determining that the log to be synchronized has been stored in the persistent storage area in an unordered manner according to the log number, the slave node can apply the log to be synchronized to ensure data consistency between the master and slave nodes. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0013] Figure 1 A schematic diagram of an exemplary system applicable to the data processing method of an embodiment of the present application;

[0014] Figure 2 is a structural block diagram of a distributed system according to an embodiment of the present application;

[0015] Figure 3 Schematic diagram of information flow of a data processing method according to an embodiment of the present application;

[0016] Figure 4 This is a schematic diagram of information flow of another data processing method according to an embodiment of the present application;

[0017] Figure 5 is a structural block diagram of a distributed system according to an embodiment of the present application;

[0018] Figure 6 1 is a flow chart of a data processing method applied to a master node according to an embodiment of the present application;

[0019] Figure 71 is a flow chart of another data processing method applied to a master node according to an embodiment of the present application;

[0020] Figure 8 1 is a flow chart of a data processing method applied to a slave node according to an embodiment of the present application;

[0021] Figure 9 Schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field should fall within the scope of protection of the embodiments of the present application.

[0023] The specific implementation of the embodiment of the present application is further explained below in conjunction with the accompanying drawings of the embodiment of the present application.

[0024] Figure 1 An exemplary system for the data processing method of the embodiment of the present application is shown. Figure 1 As shown, the system may include a cloud service 102, a communication network 104 and / or one or more user devices 106. Figure 1 The example in the figure is for multiple user devices.

[0025] The cloud server 102 may be any suitable device for storing information, data, programs, and / or any other suitable type of content, including but not limited to a distributed storage system, a server cluster, a computing cloud server cluster, etc. In some embodiments, the cloud server 102 may perform any suitable function. For example, in some embodiments, the cloud server 102 may include a distributed system, which may specifically include a distributed database, etc.

[0026] In some embodiments, the communication network 104 can be any suitable combination of one or more wired and / or wireless networks. For example, the communication network 104 can include any one or more of the following: the Internet, an intranet, a wide area network (WAN), a local area network (LAN), a wireless network, a digital subscriber line (DSL) network, a frame relay network, an asynchronous transfer mode (ATM) network, a virtual private network (VPN), and / or any other suitable communication network. The user device 106 can be connected to the communication network 104 via one or more communication links (e.g., communication link 112), and the communication network 104 can be linked to the cloud service end 102 via one or more communication links (e.g., communication link 114). The communication link can be any communication link suitable for transmitting data between the user device 106 and the cloud service end 102, such as a network link, a dial-up link, a wireless link, a hard-wired link, any other suitable communication link, or any suitable combination of such links.

[0027] The user device 106 may include any one or more user devices suitable for operating a distributed system. In some embodiments, the user device 106 may include any suitable type of device. For example, in some embodiments, the user device 106 may include a mobile device, a tablet computer, a laptop computer, a desktop computer, a wearable computer, a game console, a media player, a vehicle entertainment system, and / or any other suitable type of user device. For example, the distributed system includes a master node and a slave node, and the user device 106 may send a log synchronization request to the master node or the slave node to perform data processing through the distributed system.

[0028] Generally, to ensure data consistency across nodes in a distributed system, the master node converts the log in a log synchronization request into a write-ahead log (WAL) and assigns a corresponding number to the converted WAL. The master node then synchronizes the WAL to each slave node. The slave nodes apply the WAL through a state replica, ensuring data consistency between the slave nodes and the master node.

[0029] When the master node determines that more than half of the slave nodes have applied the log and that the log will not be rolled back, it confirms that the log to be synchronized has been committed and can provide feedback on the success of the log submission.

[0030] However, as the number of nodes increases, more resources are consumed to achieve the condition of "more than half of the slave nodes have completed the application", and there is a high possibility of synchronization failure due to interference from the network, etc. In addition, the master node needs to ensure that more than half of the slave nodes will successfully apply the logs. In order to ensure data accuracy, only the master node can receive the logs to be synchronized, which greatly affects the synchronization efficiency and scalability of the distributed system.

[0031] In order to solve the above problems, the embodiments of the present application provide a data processing solution, which is described below through multiple embodiments.

[0032] See also Figure 2 , shows a distributed system, including: a master node 201, a plurality of slave nodes 202 (m are used as an example in the figure), and a persistent storage area 203 independent of the master node and the slave nodes.

[0033] In this embodiment, the master node 201 and the slave node 202 may be used to receive log synchronization requests.

[0034] The solution provided in this embodiment can be any distributed system, and this embodiment does not limit this.

[0035] For example, the received log synchronization request may come from a client or other end, which is not limited in this embodiment.

[0036] In this embodiment, the master node 201 can respond to the received log synchronization request and assign a log number to the log to be synchronized corresponding to the log synchronization request according to the rule of increasing number sequence, so as to store the log to be synchronized and the corresponding log number in the persistent storage area in an unordered manner according to the log numbers.

[0037] Specifically, in this embodiment, when the master node 201 receives a log synchronization request, it can directly assign a log number to the log to be synchronized corresponding to the log synchronization request according to the rule of increasing number sequence, and the master node stores the log to be synchronized and the corresponding log number in the persistent storage area in an unordered manner according to the log numbers.

[0038] When a slave node receives a log synchronization request, it can request the master node to assign a log number to the log to be synchronized corresponding to the log synchronization request in ascending order and return the number to the slave node. The slave node that receives the log synchronization request stores the log to be synchronized and the corresponding log number in the persistent storage area in an unordered manner.

[0039] The numbering sequence can be increased by one by assigning each log number to the value of the previous log number. Of course, other values ​​can also be added as long as the log numbering sequence can be increased. This embodiment does not limit this.

[0040] The log number is the unique code of the log during the synchronization process and can be used to ensure data consistency between the master node and the slave node. When the master node or the slave node finds that the log number corresponding to its own log does not increase sequentially, it can be determined that there is an error in data consistency.

[0041] For example, the master node can assign log number LSN1 to log a in ascending order and store it in a non-sequential manner in the persistent storage area (corresponding to Figure 2 In step 1) of FIG, multiple blocks in the persistent storage area represent multiple logs and their corresponding log numbers.

[0042] The master node completes the storage of the log to be synchronized and the corresponding log number (corresponding to Figure 2 Step 2) in the above example) confirms that the log to be synchronized will not be rolled back, that is, determines that the log to be synchronized is submitted, and sends the log number and the log to be synchronized to the several slave nodes (corresponding to Figure 2 In step 3) of the log synchronization request, the slave nodes apply the logs to be synchronized in ascending order of log numbers, ensuring data consistency between the slave nodes and the master node. Specifically, the master node may also return a log submission completion result to the sender of the log synchronization request, such as a client.

[0043] In the embodiment of the present application, the master node assigns a log number to the log to be synchronized corresponding to the log synchronization request according to the rule of increasing number sequence, the log number can be used to ensure that the log is continuous and not missed, and the master node and the slave node can both receive the log synchronization request and write the log to be synchronized and the log number to the persistent storage area, and by storing the log to be synchronized and the corresponding log number in the persistent storage area in an unordered manner according to the log number, the order of the log numbers of the stored logs is relatively low and concurrent writing is supported, thereby ensuring the storage efficiency of the log to be synchronized, and after determining that the storage is successful, it can be guaranteed that the log will not be rolled back, thereby determining that the log submission is completed, which is different from only using the master node. Compared with the node receiving log synchronization request, the solution provided by this embodiment greatly improves the log synchronization efficiency, achieves low latency and high concurrency; and, by storing the log to be synchronized and the corresponding log number in a persistent storage area independent of the master node and the slave node, it is ensured that the log will not roll back, so that the process of determining log synchronization is decoupled from the master node and the slave node, thereby making the number of slave nodes have a lower impact on the log synchronization process, thereby improving the scalability of the distributed system. After determining that the log submission is completed, the log number and the log to be synchronized can be sent to the several slave nodes, so that the several slave nodes apply the log to be synchronized, ensuring data consistency between the master node and the slave nodes.

[0044] The following is an introduction to partitioning for distributed systems

[0045] Service Layer

[0046] like Figure 2 As shown, in this embodiment, the service layer may belong to the coordination service of the distributed system, and the service layer may include several service nodes, each of which may be a replica, and the service nodes may include the above-mentioned master node and slave nodes.

[0047] Both master and slave nodes can receive log synchronization requests. Furthermore, they can also receive log query and log deletion requests. When a master or slave node receives a log query request, it can retrieve the corresponding log data from the local historical application logs.

[0048] When the master node or slave node receives a log deletion request, it can determine the corresponding log number based on the log deletion request, and delete the logs with number values ​​less than or equal to the log number locally, as well as from other slave master nodes and slave nodes. The logs in the persistent storage area can be deleted or retained, and this embodiment does not limit this.

[0049] For example, Figure 3As shown, when the master node receives a log synchronization request w(a), that is, a request to write write log a, it can assign log number LSN1 to the log a to be synchronized according to the rule of increasing number sequence; and write the log a to be synchronized and its corresponding log number LSN1 into the persistent storage area in an unordered manner according to the log numbers; when it is determined that the writing is completed, it can be determined that log a will not be rolled back, that is, log a is committed, and the synchronization result is returned to the sender of the log synchronization request w(a), and log a and its number are synchronized to several slave nodes, so that the several slave nodes apply the logs in the order of increasing log number sequence.

[0050] like Figure 4 As shown, when the slave node receives a log synchronization request w(b), that is, a request to write to the write log b, it can send a number generation request to assign a log number to the log b to be synchronized to the master node. The master node can assign the log number LSN2 to the log b to be synchronized according to the rule of increasing number sequence, and send the log number LSN2 to the slave node.

[0051] The slave node may store the log b and its corresponding log number LSN2 in the persistent storage area in an unordered manner according to the log numbers, and feed back a successful storage result to the slave node.

[0052] After the slave node determines that the storage is successful, it can synchronize log b and its log number LSN2 to the master node and several other slave nodes except the current slave node, so that the master node and several other slave nodes except the current slave node apply log b in the order of increasing log numbers, and the slave node can return the synchronization result to the sender of the log synchronization request w(b).

[0053] Storage Area:

[0054] Optionally, in this embodiment, if Figure 5 As shown, in order to back up the log, the persistent storage area may include 2n+1 storage nodes, where n is a positive integer. The log to be synchronized and the corresponding log number are stored in the persistent storage area, including: writing the log to be synchronized and the corresponding log number to at least n+1 storage nodes.

[0055] This ensures that each log to be synchronized includes n+1 file copies in the persistent storage area, avoiding the situation where storage errors in the persistent storage area cause errors in the log to be synchronized. It should be noted that when writing the log to be synchronized and the corresponding log number to at least n+1 storage nodes, the writing is also carried out in an unordered manner according to the log number, that is, the order in which the logs are written to the storage nodes is not limited.

[0056] In addition, since the logs to be synchronized stored in the persistent storage area are used as log copies, and since both the master node and the slave node can write logs and corresponding log numbers to the persistent storage area, that is, concurrent writing is supported, the logs and corresponding log numbers can be stored in the persistent storage area in an unordered manner.

[0057] Optionally, in this embodiment, in order to further improve the log synchronization efficiency and the fault tolerance of the logs stored in the distributed system, the number of storage nodes may be increased.

[0058] Since the log to be synchronized and the corresponding log number are written to at least n+1 storage nodes when storing the log to be synchronized, when the log needs to be queried, the corresponding log can also be queried from at least n+1 storage nodes according to the log number to ensure that the log can be queried successfully.

[0059] Furthermore, if Figure 5 As shown, to more conveniently manage the persistent storage area, the persistent storage area includes multiple storage node groups 2031. The figure uses two storage node groups as an example for illustration, namely storage node groups a and b. Each storage node group includes 2n+1 storage nodes (storage nodes a1 to a2n+1, and storage nodes b1 to b2n+1 in the figure). Writing the log to be synchronized and the corresponding log number to at least n+1 storage nodes includes writing the log to be synchronized and the corresponding log number to at least n+1 storage nodes in the multiple data storage node groups. For example, when there are k storage node groups, each storage node group includes 2n+1 storage nodes, i.e., a total of k*(2n+1) storage nodes, when writing, the log to be synchronized and the corresponding log number can be written to at least n+1 storage nodes among the k*(2n+1) storage nodes.

[0060] Masternode election

[0061] In this embodiment, when the slave node does not receive a message from the master node for a preset period of time, it indicates that the master node may have crashed, or information transmission may have been interrupted due to a network failure. At this time, the slave node can determine itself as a candidate master node.

[0062] In this embodiment, it is necessary to write the log to be synchronized and the corresponding log number to at least n+1 storage nodes to ensure successful log submission. When electing the master node, several candidate master nodes can send election information to the 2n+1 storage nodes respectively, and the storage nodes are used to elect one of the candidate master nodes as the master node; each candidate master node receives the election confirmation message returned by the storage node, and determines the candidate master node confirmed by at least n+1 storage nodes as the master node.

[0063] The specific method for determining the candidate master node can be referred to the relevant technology and will not be repeated here.

[0064] In addition, when there are multiple storage node groups, the candidate master node may send election information to 2n+1 storage nodes in the multiple storage node groups respectively, and the storage nodes are used to elect one of the candidate master node as the master node.

[0065] Afterwards, each candidate master node can receive the election confirmation message returned by the storage node, and the candidate master node confirmed by each storage node group will be the master node. At least n+1 candidate master nodes confirmed by the storage nodes in each storage node group will be the master node determined by the storage node group.

[0066] It should also be noted that each elected master node may have a version number determined based on a sequentially increasing rule. When storing the logs to be synchronized and their corresponding log numbers in the persistent storage area, the version number of the master node is also written.

[0067] When the master node or the slave node finds that the value of the log version number decreases when the log number increases, it indicates that there is data inconsistency.

[0068] Node crash recovery

[0069] In this embodiment, if a slave node crashes and causes data inconsistency, the slave node can directly obtain the corresponding log from the master node or other slave nodes according to the rule of increasing number sequence, thereby completing data recovery.

[0070] If the master node crashes and causes data inconsistency, in order to enable recovery after the master node crashes, in this embodiment, the application logs and corresponding log numbers can be backed up to the log backup area by the master node or the slave node. When the master node crashes, recovery can be performed by querying the application logs and corresponding log numbers in the log backup area.

[0071] See also Figure 5, the log and the corresponding log number can be backed up to a distributed file system (DFS) through the master node or the slave node. Of course, in other implementations, they can also be backed up to the disk of the master node or the slave node, which is not limited in this embodiment.

[0072] Furthermore, in the embodiment of the present application, in order to reduce the workload of the master node, the application log and the corresponding log number can be preferably backed up to the log backup area through the slave node. In addition, the backup process can be asynchronous backup.

[0073] In this embodiment, when the master node crashes, some log synchronization errors may occur. To this end, in this embodiment, if the previous master node crashes and the current master node is re-elected, the backed-up log can be obtained from the log backup area through the current master node, and several increasing first log numbers can be obtained from the log; through the current master node, several second log numbers are obtained from the persistent storage area, and the second log number is greater than the first log number; through the current master node, based on the rule of increasing number sequence, the third log number corresponding to the log of the synchronization error caused by the crash of the previous master node is located from the first log number and the second log number; through the current master node, the log to be synchronized with a number value greater than the third log number and the corresponding log number are re-stored in the persistent storage area. In this way, the operation of the distributed system can be quickly and conveniently restored when the master node crashes.

[0074] When the master node crashes, the logs may include the following: logs that have completed synchronization and have been backed up to the log backup area, logs that have completed synchronization but have not been backed up to the log backup area, and logs that have failed to synchronize, with the majority of logs already synchronized and backed up to the log backup area. In this embodiment, by obtaining a plurality of incremental first log numbers from the logs, it is possible to determine the majority of logs that have completed synchronization. By obtaining a plurality of second log numbers from the persistent storage area, where the second log numbers are greater than the first log numbers, it is possible to obtain from the persistent storage area some logs that have completed synchronization but have not been backed up, as well as logs that have failed to synchronize, with a smaller number of second log numbers that need to be read.

[0075] Furthermore, because the master node assigns log numbers to logs to be synchronized based on an ascending numbering rule, when the current master node recovers, it can also locate the third log number corresponding to the log with the synchronization error caused by the previous master node crash from the obtained first and second logs based on the ascending numbering rule. It should be noted that the third log number is generally one of the second log numbers.

[0076] After the third log number is located, the master node can restore the logs with number values ​​greater than or equal to the third log number and the corresponding log numbers to the persistent storage area to perform a secondary synchronization operation on them.

[0077] If storage is successful, a message indicating successful log submission can be returned to the sender of the log synchronization request. If storage fails, a message indicating failed log submission can be returned to the sender of the log synchronization request. The sender can request the failed log data again or abandon the failed log data, which is not limited in this embodiment.

[0078] The solution of this application is exemplified below through a specific method flow.

[0079] See also Figure 6 , which shows a flow chart of a data processing method. The method provided in this embodiment is executed by a master node of a distributed system, as shown in the figure, and includes:

[0080] S601. In response to a received log synchronization request, a log number is assigned to the log to be synchronized corresponding to the log synchronization request according to the rule of increasing number sequence, so that the log to be synchronized and the corresponding log number are stored in the persistent storage area in an unordered manner according to the log number.

[0081] Optionally, in this embodiment, the log synchronization request can be received by the master node or the slave node; when the log synchronization request is received by the slave node, step S601 may include: receiving a number allocation request from the slave node, the number allocation request being generated by the slave node based on the received log synchronization request; in response to the number allocation request, allocating the log number to the log to be synchronized corresponding to the log synchronization request according to the rule of increasing number sequence; returning the log number to the slave node, so that the slave node stores the log to be synchronized and the corresponding log number in the persistent storage area in an unordered manner according to the log numbers.

[0082] Optionally, in this embodiment, the persistent storage area includes 2n+1 storage nodes, where n is a positive integer. Storing the log to be synchronized and the corresponding log number in the persistent storage area includes writing the log to be synchronized and the corresponding log number to at least n+1 storage nodes. This ensures that each log to be synchronized includes n+1 file replicas in the persistent storage area, preventing errors in the log to be synchronized due to storage errors in the persistent storage area.

[0083] Furthermore, in order to more conveniently manage the persistent storage area, the persistent storage area includes multiple storage node groups, each of the storage node groups includes 2n+1 storage nodes, and writing the log to be synchronized and the corresponding log number to at least n+1 storage nodes includes: writing the log to be synchronized and the corresponding log number to at least n+1 storage nodes in at least one of the storage node groups.

[0084] Optionally, in this embodiment, the master node is determined in the following manner: several candidate master nodes send election information to the 2n+1 storage nodes respectively, and the storage nodes are used to elect one of the candidate master nodes as the master node; each of the candidate master nodes receives the election confirmation message returned by the storage node, and determines the candidate master node confirmed by at least n+1 storage nodes as the master node.

[0085] S602: After the log to be synchronized and the corresponding log number are stored, confirm that the log to be synchronized is submitted, and send the log number and the log to be synchronized to the multiple slave nodes, so that the multiple slave nodes apply the log to be synchronized.

[0086] In an embodiment of the present application, a master node assigns a log number to the log to be synchronized corresponding to the log synchronization request according to a rule of increasing number sequence. The log number can be used to ensure that the log is continuous and not missed. The master node and the slave node can both receive the log synchronization request and write the log to be synchronized and the log number to the persistent storage area in an unordered manner according to the log number. The order of the log numbers of the stored logs is low and concurrent writing is supported. After the write is successful, the log to be synchronized is submitted to the state replica of the master node, thereby determining that the log submission is complete. Compared with only receiving the log synchronization request through the master node, the solution provided by this embodiment greatly improves the log synchronization efficiency and achieves low latency and high concurrency. Moreover, by storing the log to be synchronized and the corresponding log number in a persistent storage area independent of the master node and the slave node, it is ensured that the log will not roll back, so that the process of determining log synchronization is decoupled from the master node and the slave node, thereby reducing the impact of the number of slave nodes on the log synchronization process, thereby improving the scalability of the distributed system. After determining that the log to be synchronized has been stored in the persistent storage area in an unordered manner according to the log number, the slave node can apply the log to be synchronized in an increasing order of the log number to ensure log synchronization.

[0087] See also Figure 7 , shows a flow chart of another data processing method. The method provided in this embodiment is executed by the master node of the distributed system, as shown in the figure, and includes:

[0088] S701. In response to a received log synchronization request, a log number is assigned to the log to be synchronized corresponding to the log synchronization request according to the rule of increasing number sequence, so that the log to be synchronized and the corresponding log number are stored in the persistent storage area in an unordered manner according to the log number.

[0089] S702: After the log to be synchronized and the corresponding log number are stored, confirm that the log to be synchronized is submitted, and send the log number and the log to be synchronized to the plurality of slave nodes, so that the plurality of slave nodes apply the log to be synchronized;

[0090] S703: Back up the application logs and corresponding log numbers to the log backup area.

[0091] If the previous master node crashes and the current master node is re-elected, the method further includes:

[0092] S704. Obtaining a backed-up log from the log backup area through the current master node, and obtaining a plurality of incremental first log numbers from the log;

[0093] S705. Obtain, from the persistent storage area through the current master node, a plurality of second log numbers, where the second log numbers are greater than the first log numbers.

[0094] S706: Using the current master node, based on the rule of increasing number sequence, locate the third log number corresponding to the log of the synchronization error caused by the crash of the previous master node from the first log number and the second log number;

[0095] S707. Restore the logs with number values ​​greater than or equal to the third log number and the corresponding log numbers to the persistent storage area through the current master node.

[0096] When the master node crashes, the logs may include the following: logs that have completed synchronization and have been backed up to the log backup area, logs that have completed synchronization but have not been backed up to the log backup area, and logs that have failed to synchronize, with the majority of logs already synchronized and backed up to the log backup area. In this embodiment, by obtaining a plurality of incremental first log numbers from the logs, it is possible to determine the majority of logs that have completed synchronization. By obtaining a plurality of second log numbers from the persistent storage area, where the second log numbers are greater than the first log numbers, it is possible to obtain from the persistent storage area some logs that have completed synchronization but have not been backed up, as well as logs that have failed to synchronize, with a smaller number of second log numbers that need to be read.

[0097] After locating the third log number from several first log numbers and several second log numbers, the master node can restore the logs with number values ​​greater than or equal to the third log number and the corresponding log numbers to the persistent storage area to perform a secondary synchronization operation on them. This can conveniently and efficiently restore the operation of the distributed system when the master node crashes.

[0098] If storage succeeds, a submission success message can be returned to the sender of the log synchronization request. If storage fails, a submission failure message can be returned to the sender of the log synchronization request. The sender can request the failed log data again or abandon the failed log data, which is not limited in this embodiment.

[0099] See also Figure 8 , shows a flow chart of a data processing method. The method provided in this embodiment is executed by a slave node of a distributed system, as shown in the figure, and includes:

[0100] S801. In response to the received log synchronization request, generate a number generation request, and send the number generation request to the master node, so that the master node responds to the number generation request and assigns a log number to the log to be synchronized corresponding to the log synchronization request according to the rule of increasing number sequence.

[0101] S802: Receive the log number returned by the master node, and store the log to be synchronized and the corresponding log number in the persistent storage area in an unordered manner according to the log numbers.

[0102] S803. Inform the slave node that the storage of the log to be synchronized is completed, so that the slave node confirms that the submission of the log to be synchronized is completed, and sends the log number and the log to be synchronized to the master node and several other slave nodes except the current slave node, so that the master node and several other slave nodes except the current slave node apply the log to be synchronized.

[0103] In an embodiment of the present application, a master node assigns a log number to the log to be synchronized corresponding to the log synchronization request according to a rule of increasing number sequence. The log number can be used to ensure that the log is continuous and not missed. The master node and the slave node can both receive the log synchronization request and write the log to be synchronized and the log number to the persistent storage area in an unordered manner according to the log number. The order of the log numbers of the stored logs is low and concurrent writing is supported. After the write is successful, the log to be synchronized is submitted to the state replica of the master node, thereby determining that the log submission is complete. Compared with only receiving the log synchronization request through the master node, the solution provided by this embodiment greatly improves the log synchronization efficiency and achieves low latency and high concurrency. Moreover, by storing the log to be synchronized and the corresponding log number in a persistent storage area independent of the master node and the slave node, it is ensured that the log will not roll back, so that the process of determining log synchronization is decoupled from the master node and the slave node, thereby reducing the impact of the number of slave nodes on the log synchronization process, thereby improving the scalability of the distributed system. After determining that the log to be synchronized has been stored in the persistent storage area in an unordered manner according to the log number, the slave node can apply the log to be synchronized in an increasing order of the log number to ensure log synchronization.

[0104] Reference Figure 9 , shows a structural diagram of an electronic device according to an embodiment of the present application. The specific embodiment of the present application does not limit the specific implementation of the electronic device.

[0105] like Figure 9 As shown, the electronic device may include: a processor (processor) 902 , a communication interface (Communications Interface) 904 , a memory (memory) 906 , and a communication bus 908 .

[0106] in:

[0107] The processor 902 , the communication interface 904 , and the memory 906 communicate with each other via a communication bus 908 .

[0108] The communication interface 904 is used to communicate with other electronic devices or servers.

[0109] The processor 902 is configured to execute the program 910 , and specifically may execute the relevant steps in the above-mentioned data processing method embodiment.

[0110] Specifically, the program 910 may include program codes, which include computer operation instructions.

[0111] Processor 902 may be a CPU, an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs, or processors of different types, such as one or more CPUs and one or more ASICs.

[0112] The memory 906 is used to store the program 910. The memory 906 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage.

[0113] The program 910 can be specifically used to enable the processor 902 to execute operations corresponding to the data processing method described in any of the aforementioned method embodiments.

[0114] Specifically, program 910 can be used to enable the processor 902 to perform the following steps: in response to a received log synchronization request, assign a log number to the log to be synchronized corresponding to the log synchronization request according to the rule of increasing number sequence, so as to store the log to be synchronized and the corresponding log number in the persistent storage area in an unordered manner according to the log numbers; after the log to be synchronized and the corresponding log number are stored, confirm that the submission of the log to be synchronized is completed, and send the log number and the log to be synchronized to the several slave nodes, so that the several slave nodes apply the log to be synchronized.

[0115] Optionally, in any embodiment of the present application, the application log and the corresponding log number are backed up to the log backup area through the master node or the slave node.

[0116] Optionally, in any embodiment of the present application, if the previous master node crashes and the current master node is re-elected, the method further includes: obtaining a backed-up log from the log backup area through the current master node, and obtaining several increasing first log numbers from the log; obtaining several second log numbers from the persistent storage area through the current master node, the second log number being greater than the first log number; locating the third log number corresponding to the log of the synchronization error caused by the crash of the previous master node from the first log number and the second log number based on the rule of increasing numbering sequence through the current master node; and re-storing the log with a number value greater than or equal to the third log number and the corresponding log number to the persistent storage area through the current master node.

[0117] Optionally, in any embodiment of the present application, in response to the received log synchronization request, a log number is assigned to the log to be synchronized corresponding to the log synchronization request according to the rule of increasing number sequence, so as to store the log to be synchronized and the corresponding log number in the persistent storage area, including: receiving the number assignment request of the slave node, the number assignment request is generated by the slave node according to the received log synchronization request; in response to the number assignment request, the log number is assigned to the log to be synchronized corresponding to the log synchronization request according to the rule of increasing number sequence; and the log number is returned to the slave node, so that the slave node stores the log to be synchronized and the corresponding log number in the persistent storage area.

[0118] Optionally, in any embodiment of the present application, the persistent storage area includes 2n+1 storage nodes, where n is a positive integer, and storing the log to be synchronized and the corresponding log number in the persistent storage area includes: writing the log to be synchronized and the corresponding log number to at least n+1 storage nodes.

[0119] Optionally, in any embodiment of the present application, the persistent storage area includes multiple storage node groups, each of the storage node groups includes 2n+1 storage nodes, and writing the log to be synchronized and the corresponding log number to at least n+1 of the storage nodes includes: writing the log to be synchronized and the corresponding log number to at least n+1 of the storage nodes in the multiple storage node groups.

[0120] Optionally, in any embodiment of the present application, the master node is determined in the following manner: several candidate master nodes send election information to the 2n+1 storage nodes respectively, and the storage nodes are used to elect one of the candidate master nodes as the master node; each of the candidate master nodes receives the election confirmation message returned by the storage node, and determines the candidate master node confirmed by at least n+1 storage nodes as the master node.

[0121] Specifically, the program 910 can also be used to enable the processor 902 to perform the following steps: in response to the received log synchronization request, generate a number generation request, and send the number generation request to the master node, so that the master node responds to the number generation request and assigns a log number to the log to be synchronized corresponding to the log synchronization request according to the rule of increasing number sequence; receives the log number returned by the master node, and stores the log to be synchronized and the corresponding log number in the persistent storage area in an unordered manner according to the log number; informs the master node that the storage of the log to be synchronized is completed, so that the master node confirms that the submission of the log to be synchronized is completed, and sends the log number and the log to be synchronized to the master node and several other slave nodes except the current slave node, so that the master node and several other slave nodes except the current slave node apply the log to be synchronized.

[0122] The specific implementation of each step in program 910 can refer to the corresponding description of the corresponding steps and units in the above-mentioned method embodiment, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working process of the above-mentioned devices and modules can refer to the corresponding process description in the above-mentioned method embodiment, and will not be repeated here.

[0123] An embodiment of the present application further provides a computer storage medium on which a computer program is stored, and when the program is executed by a processor, the method described above is implemented.

[0124] An embodiment of the present application also provides a computer program product, including computer instructions, which instruct a computing device to execute operations corresponding to any one of the above-mentioned multiple method embodiments.

[0125] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0126] The methods according to the embodiments of the present application described above can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code originally stored in a remote recording medium or non-transitory machine-readable medium downloaded via a network and then stored in a local recording medium. Thus, the methods described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It will be understood that a computer, processor, microprocessor controller, or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the methods described herein are implemented. In addition, when a general-purpose computer accesses the code for implementing the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for performing the methods shown herein.

[0127] Those skilled in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of this application.

[0128] The above implementation methods are only used to illustrate the embodiments of the present application, and are not intended to limit the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of the present application, and the scope of patent protection of the embodiments of the present application should be defined by the claims.

Claims

1. A data processing method, applied to a master node of a distributed system, wherein the distributed system includes the master node, a plurality of slave nodes, and a persistent storage area independent of the master node and the slave nodes, the method comprising: In response to a received log synchronization request, assigning a log number to the log to be synchronized corresponding to the log synchronization request according to a rule of increasing number sequence, so as to store the log to be synchronized and the corresponding log number in the persistent storage area in an unordered manner according to the log number; After the log to be synchronized and the corresponding log number are stored, confirming that the log to be synchronized is submitted, and sending the log number and the log to be synchronized to the plurality of slave nodes, so that the plurality of slave nodes apply the log to be synchronized; receiving a number allocation request from the slave node, where the number allocation request is generated by the slave node according to the received log synchronization request; In response to the number allocation request, allocating the log number to the to-be-synchronized log corresponding to the log synchronization request according to a rule of increasing number sequence; The log number is returned to the slave node, so that the slave node stores the log to be synchronized and the corresponding log number in the persistent storage area.

2. The method according to claim 1, wherein The method further includes: backing up the application log and the corresponding log number to a log backup area through the master node or the slave node.

3. The method according to claim 2, wherein: If the previous master node crashes and the current master node is re-elected, the method further includes: Obtaining a backed-up log from the log backup area through the current master node, and obtaining a plurality of incremental first log numbers from the log; Obtaining, through the current master node, a plurality of second log numbers from the persistent storage area, where the second log numbers are greater than the first log numbers; Using the current master node, based on a rule of increasing number sequence, locate, from the first log number and the second log number, a third log number corresponding to a log of a synchronization error caused by a crash of the previous master node; Through the current master node, logs with number values ​​greater than or equal to the third log number and the corresponding log numbers are restored to the persistent storage area.

4. The method according to claim 1, wherein The persistent storage area includes 2n+1 storage nodes, where n is a positive integer. The step of storing the log to be synchronized and the corresponding log number in the persistent storage area includes: Write the log to be synchronized and the corresponding log number to at least n+1 storage nodes.

5. The method according to claim 4, wherein The persistent storage area includes a plurality of storage node groups, each of the storage node groups includes 2n+1 storage nodes, and writing the to-be-synchronized log and the corresponding log number to at least n+1 storage nodes includes: The log to be synchronized and the corresponding log number are written to at least n+1 of the storage nodes in the plurality of storage node groups.

6. The method according to claim 4, wherein: The master node is determined in the following manner: Several candidate master nodes send election information to the 2n+1 storage nodes respectively, and the storage nodes are used to elect one of the candidate master nodes as the master node; The election confirmation message returned by the storage node is received by each of the candidate master nodes, and the candidate master node confirmed by at least n+1 of the storage nodes is determined as the master node.

7. An electronic device comprising: A processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the data processing method according to any one of claims 1 to 6.

8. A computer storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images