Data processing method, system and device

Abstract

The embodiments of the present application provide a data processing method, system and device. Among them, the method includes: saving the master-slave synchronization status information of multiple computing units. The computing unit includes a master node and at least one standby node, and the master-slave synchronization status information includes the playback log identifier of the standby node; the playback log identifier is determined after the standby node performs data playback using the master node synchronization log belonging to the same computing unit; according to the log identifier and the master-slave synchronization status information carried in the data processing request, the target computing unit is determined from multiple computing units. Receive a read request, send the read request to at least one standby node in the target computing unit, so that the at least one standby node responds to the read request to feedback the corresponding target data. The technical solution provided by the embodiment of the present application realizes the data processing effect of one write and multiple reads and ensures strong consistency of data processing based on the coordination node, the master node and at least one standby node, which can effectively improve the data processing efficiency.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to data processing methods, systems, and devices. Background Art

[0002] With the development of cloud computing technology, the scale of databases is getting larger and larger. In order to meet the needs of big data and high-efficiency database applications, a distributed database architecture is proposed based on the existing centralized database.

[0003] In existing distributed databases, a coordination node contains multiple storage nodes and computing units. To ensure data security and reliability, backup data is usually created for each node. For example, in a computing unit, each master node is assigned at least one backup node for data backup. However, the data in the backup node can only be restored if the master node loses data or fails. During normal operation, when the coordination node receives a data read request or data write request, the master node in the corresponding computing unit completes the request. The backup node in the computing unit does not participate in the data reading and writing work. Therefore, although the backup node stores data synchronized from the master node, this data is idle most of the time. When the data processing demand is relatively large, it is difficult to meet the data processing needs of the database by only using a limited number of master nodes to perform the corresponding data read and write operations. Summary of the Invention

[0004] To solve or improve the problems existing in the prior art, the embodiments of the present application provide data processing methods, systems and devices.

[0005] In one aspect, in one embodiment of the present application, a data processing method is provided. The method includes:

[0006] Storing master-slave synchronization status information of multiple computing units; wherein each computing unit includes a master node and at least one slave node, and the master-slave synchronization status information includes a playback log identifier of each slave node; the playback log identifier is determined by the slave node after performing data playback using the synchronization log of the master node belonging to the same computing unit;

[0007] In response to the received data processing request, determining a target computing unit from the multiple computing units according to a log identifier carried in the data processing request and master / slave synchronization status information of the multiple computing units;

[0008] If the data processing request is a read request, sending the read request to at least one standby node in the target computing unit, so that the at least one standby node responds to the read request to feed back corresponding target data;

[0009] If the data processing request is a write request, the write request is sent to the master node in the target computing unit.

[0010] In a second aspect, in one embodiment of the present application, another data processing method is provided. The method includes:

[0011] Receive log information sent by the master node;

[0012] Performing data synchronization operations according to the log information to obtain synchronized data with the master node;

[0013] After receiving the read request, obtaining the target data specified by the read request from the synchronized data;

[0014] Feedback the target data to the read request sender.

[0015] In a third aspect, in one embodiment of the present application, a data processing system is provided. The system includes:

[0016] A coordination node device, configured to store master-slave synchronization status information of a plurality of computing units; wherein any computing unit includes a master node and at least one standby node, and the master-slave synchronization status information includes a playback log identifier of any standby node; the playback log identifier is determined by the standby node after performing data playback using the synchronization log of the master node belonging to the same computing unit; in response to a received data processing request, determining a target computing unit from the plurality of computing units based on the log identifier carried in the data processing request and the master-slave synchronization status information of the plurality of computing units; if the data processing request is a read request, sending the read request to at least one standby node in the target computing unit, so that the at least one standby node responds to the read request to feed back corresponding target data; if the data processing request is a write request, sending the write request to the master node in the target computing unit;

[0017] A standby node device is used to receive log information sent by the master node; based on the log information, perform data synchronization operations to obtain synchronized data synchronized with the master node; after receiving a read request, obtain the target data specified by the read request from the synchronized data; and feed back the target data to the sender of the read request.

[0018] In a fourth aspect, in one embodiment of the present application, an electronic device is provided, including a memory and a processor; wherein,

[0019] The memory is used to store programs;

[0020] The processor is coupled to the memory and is used to execute the program stored in the memory to implement the data processing method described in the first aspect or the other data processing method described in the second aspect.

[0021] In the fifth aspect, in one embodiment of the present application, a non-transitory machine-readable storage medium is provided, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor executes the method described in the first aspect or the method described in the second aspect.

[0022] In the sixth aspect, in one embodiment of the present application, a computer program product is provided, comprising a computer program / instruction, which, when executed by a processor, enables the processor to implement the method described in the first aspect or the method described in the second aspect.

[0023] The technical solution provided by the embodiment of the present application stores master-backup synchronization status information of multiple computing units; wherein each computing unit includes a master node and at least one backup node, and the master-backup synchronization status information includes a playback log identifier of any backup node; the playback log identifier is determined by the backup node after performing data playback using the synchronization log of the master node belonging to the same computing unit. In response to a received data processing request, a target computing unit is determined from the multiple computing units based on the log identifier carried in the data processing request and the master-backup synchronization status information of the multiple computing units. If the data processing request is a read request, the read request is sent to at least one backup node in the target computing unit, so that the at least one backup node responds to the read request and returns the corresponding target data. If the data processing request is a write request, the write request is sent to the master node in the target computing unit. Using the above technical solution, by including a master node and multiple backup nodes in the computing unit, the master-backup synchronization status information of the master and backup nodes is sent to the coordination node in real time. After receiving the data processing request, the coordination node selects the corresponding backup node or master node to perform the corresponding data processing action based on the type of data processing request. For example, when data is required to be read, the appropriate backup node is selected based on the log identifier carried in the read request, and the backup node then provides the target data corresponding to the log identifier. While utilizing the backup node for secure data backup, it also enables the backup node to satisfy the data read requirements of the coordination node. Based on the coordination node, the master node, and at least one backup node, a single-write, multiple-read data processing effect is achieved, ensuring strong data processing consistency and effectively improving data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0025] Figure 1 A schematic diagram of the overall structure of a data processing system provided in an embodiment of the present application;

[0026] Figure 2 A flowchart of a data processing method provided in an embodiment of the present application;

[0027] Figure 3 A schematic diagram of a process for reporting status information via heartbeat messages provided in an embodiment of the present application;

[0028] Figure 4 A schematic diagram of the process of reporting status information through transaction submission provided in an embodiment of the present application;

[0029] Figure 5 A schematic diagram of a target data acquisition process provided in an embodiment of the present application;

[0030] Figure 6 A flowchart of another data processing method provided in an embodiment of the present application;

[0031] Figure 7 A schematic diagram of the data processing system structure provided in an embodiment of the present application;

[0032] Figure 8 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0033] Figure 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0034] Figure 10 A schematic diagram of the structure of another data processing device provided in an embodiment of the present application;

[0035] Figure 11 A schematic structural diagram of another electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] A distributed database consists of coordination nodes and computing units. The coordination node (master) schedules tasks for each computing unit (segment). Generally speaking, to ensure the reliability and stability of computing units, each computing unit includes both a master node (primary) and at least one backup node (mirror). However, the backup node serves only to replace the master node in the event of a failure. If the master node is functioning properly, the backup node will update based on data updates from the master node. Especially when there are a large number of computing units, data backup costs are high, but backup node utilization is low, resulting in wasted storage resources.

[0037] Figure 1 This is a schematic diagram of the overall structure of a data processing system provided in an embodiment of the present application. Figure 1 As can be seen in the figure, the coordination node is associated with at least two computing units (in actual applications, the number of computing units associated with the coordination node is not limited). The coordination node here is Figure 1 The master node in the calculation unit is Figure 1 In the segment. Specifically, the computing unit segment includes a primary node that plays a reading and writing role, and at least one backup node mirror that plays a backup and read-only role. Since the data backed up in the backup node mirror is the same as the data in the primary node, the data in the backup node mirror can be fully utilized, that is, the backup node mirror is not only used to provide external services when the primary node fails, but also can meet the data reading of the coordination node when the primary node works normally, so that the data in the backup node mirror is fully utilized, reducing the workload of the primary node. In addition, Figure 1 It can also be seen that a one-write-multiple-read service is provided, where the master node and multiple backup nodes that serve as writers all use the same shared storage to store data. In the technical solution of this application, the specific working process will be described in the following embodiments.

[0038] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0039] In some processes described in the specification, claims and the above-mentioned figures of this application, multiple operations appearing in a specific order are included. These operations may not be executed in the order in which they appear in this document or may be executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any order of execution. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequence, nor do they limit "first" and "second" to different types. In addition, the embodiments described below are only some of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0040] like Figure 2 This is a flow chart of a data processing method provided in an embodiment of the present application. In actual application scenarios, the execution subject of the method can be a coordination node (e.g., a computer, server, etc.) in a distributed database. The method specifically includes the following steps:

[0041] 201: Save the master-slave synchronization status information of multiple computing units; wherein, any computing unit includes a master node and at least one standby node, and the master-slave synchronization status information includes the replay log identifier of any standby node; the replay log identifier is determined after the standby node uses the synchronization log of the master node belonging to the same computing unit to perform data replay.

[0042] 202: In response to the received data processing request, determine a target computing unit from the multiple computing units according to the log identifier carried in the data processing request and the master-slave synchronization status information of the multiple computing units.

[0043] 203: If the data processing request is a read request, send the read request to at least one standby node in the target computing unit, so that the at least one standby node responds to the read request to feed back corresponding target data.

[0044] 204: If the data processing request is a write request, send the write request to the master node in the target computing unit.

[0045] Combine Figure 1As shown, the coordination node master can obtain the master-slave synchronization status information of multiple computing units and store it in the segment status in the coordination node master. The master-slave synchronization status information mentioned here includes: active, sync LSN, apply LSN, replay LSN, etc. In order to facilitate the coordination node to timely and accurately grasp the master-slave synchronization status information of the master node and the standby node in the computing unit, the coordination node master can directly obtain the required master-slave synchronization status information from each standby node when obtaining the master-slave synchronization status information. The log identifier mentioned here can be understood as the LSN (Log sequence number) of the log, such as sync LSN, apply LSN, and replay LSN.

[0046] Through the master-slave synchronization status information, the coordination node can timely grasp the latest master-slave synchronization status information of each computing unit and the master node and at least one backup node within the computing unit. As a result, when receiving a write request or read request from the client, it can timely and accurately find the appropriate master node or backup node to perform the corresponding data read and write task. Specifically, after the coordination node receives the read request, it can extract the log identifier (for example, sync LSN) carried in the read request from the read request. Based on the log identifier, the master-slave synchronization status information of each current backup node can also be understood. For example, the log replay progress of the backup node can be understood through the replay LSN, and the data synchronization progress of the backup node can be understood through the sync LSN.

[0047] As mentioned above, the backup node mirror in the computing unit is used for safe backup. During normal use, it needs to maintain synchronization with the primary node. When the primary node has a problem and cannot work normally, the backup node can replace the primary node to continue working. In other words, when the primary node is not faulty, although the backup node has been performing data backup and file backup, the coordination node cannot timely understand the synchronization progress and synchronization status information between each backup node and the primary node. The backup data and backup files in the backup node are idle and cannot be fully utilized. Therefore, in order to make full use of the backup data and backup files in the backup node, when the coordination node receives the read request from the client, it can determine the target computing unit from multiple computing units based on the log identifier and the primary-backup synchronization status information. Then, the target data is provided by the backup node in the target computing unit.

[0048] Through the above solution, the data in the standby node in the computing unit can be fully utilized, while ensuring the basic security backup requirements of the standby node, it can also well meet the client's demand for data reading.

[0049] Combine Figure 1 As you can see, the coordinator node can obtain the master-slave synchronization status information of the primary and backup nodes in the computing unit. In actual applications, a new data structure can be created in the coordinator node to store the information status of the primary node and the mirror node in the computing unit. For example:

[0050]

[0051] The corresponding global instance is static SegmentSyncStatusArray*allSegSyncStatus

[0052] It should be noted that the coordinating node master can obtain the master-slave synchronization status information by proactively reporting it through a computing unit (segment), and the reporting method may include reporting through heartbeat messages or reporting during the distributed transaction submission process. Specific examples are given below to illustrate each.

[0053] like Figure 3 A flow chart of reporting master-slave synchronization status information via heartbeat messages provided in an embodiment of the present application. Figure 3 As can be seen in the figure, the specific process of reporting the master-slave synchronization status information via heartbeat messages is as follows: 301: Receive the heartbeat message from the computing unit. 302: Obtain the playback log identifier carried in the heartbeat message. 303: If the playback log identifier is greater than any stored log identifier, obtain the master-slave synchronization status information of the backup node carried in the heartbeat message.

[0054] For example, the master / slave synchronization status information of the computing unit segment is added to the fts_result message.

[0055]

[0056]

[0057] Through the above embodiment, it can be seen that the master-slave synchronization state information contained in the heartbeat message has a playback log identifier. Since the transmission order of heartbeat messages may be disordered in actual applications, when updating information, it is necessary to ensure that the apply LSN represented by applyPtr (that is, the playback log identifier mentioned above) is greater than the LSN (stored log identifier) ​​already stored in the master collaborative node. Only then is the master-slave synchronization state information received from the standby node allowed to be saved in the current data structure to complete the update of the master-slave synchronization state information.

[0058] like Figure 4A flow chart of reporting master-slave synchronization status information through transaction submission provided in an embodiment of the present application. Figure 4 As can be seen in the figure, the master-slave synchronization status information is reported through distributed transaction submission. The specific steps are as follows: 401: Determine the transaction status of the master node and the backup node in the computing unit. 402: If the transaction status of the master node and the at least one backup node is in the prepared state, send a transaction submission request to the master node and the at least one backup node in the prepared state. 403: Receive the master-slave synchronization status information of the computing unit based on the feedback of the transaction submission request.

[0059] For example, two-phase commit (2PC) is used in the distributed transaction processing process, including the prepare phase and the commit phase. When the segment processes the distributed transaction commit protocol, exec_mpp_dtx_protocol_command is used to commit the transaction. Before the End Command, the latest master-slave synchronization status information of the current computing unit segment is sent to the coordination node master, so that the master can save the master-slave synchronization status information through the data structure mentioned above. This can ensure the strong consistency of the coordination node master, the primary node, and the backup node mirror when processing data. The data played back in the backup node mirror can participate in the data reading work and meet the data reading requirements of the coordination node.

[0060] As mentioned above, in a distributed system, a coordination node master can be associated with multiple computing unit segments at the same time. The backup nodes in different computing units have different data file backup contents, and the backup progress is also different. It should be noted that in actual applications, it is necessary to ensure that the log playback progress and data synchronization progress of each backup node can meet the client's data reading needs. Specifically as follows: Receive the data processing request. Obtain the latest playback log identifier contained in the master-slave synchronization status information, and the log identifier carried in the data processing request. If the latest playback log identifier is greater than the log identifier, determine the target computing unit corresponding to the log identifier from the multiple computing units.

[0061] In actual applications, data on standby nodes is updated promptly during distributed transaction execution. After the update, the log identifier is updated on the standby node. Therefore, when the coordinating node receives a read request, it synchronously updates the log identifier, resulting in a new replay log identifier for the current compute unit. To accurately identify the appropriate target compute unit, the coordinating node compares the log identifiers of multiple target compute units to find the most recent replay log identifier.

[0062] During the comparison, if it is found that the latest playback log identifier is greater than the log identifier carried in the read request, the computing unit corresponding to the latest playback log identifier is used as the target computing unit.

[0063] For example, when the coordination node master selects a computing unit segment, it will select the standby node of the corresponding segment based on the information of curSegSyncStatus. Specifically, the coordination node master obtains a snapshot of the master-standby synchronization status information of the computing unit, saves the global variables of the current process, and serializes the SnapshotData, and serializes the playback master-standby synchronization status information of all segments and sends it to the corresponding standby node. Generally speaking, a computing unit contains one standby node. If a computing unit contains multiple standby nodes, the standby node with the fastest playback speed or the standby node that is currently idle can be selected from the multiple standby nodes as the node that can read the target data.

[0064] When comparing the latest playback log identifier with the log identifier carried in the read request, if the latest playback log identifier is found to be no greater than the log identifier carried in the read request, then the current playback speed of each computing unit cannot meet the target data reading requirement of the read request. For example, because the backup node is slow in data text synchronization or playback, the data and logs in the primary node are not synchronized to the backup node in a timely manner. In this case, the primary node in the computing unit can meet the reading requirement. In other words, the required target data is read from the primary node based on the log identifier carried in the read request.

[0065] As mentioned above, multiple computing units associated with the same coordination node store different data and corresponding log identifiers. For example, computing unit segment 1 stores log identifiers ranging from LSN001 to LSN100, while computing unit segment 2 stores log identifiers ranging from LSN101 to LSN200. The log identifier carried in the read request is LSN105. Therefore, computing unit segment 2 is the target computing unit. After receiving the master-slave synchronization status snapshot from the standby node, the primary node serializes it to obtain playback-related master-slave synchronization status information. It then sends this serialized playback-related master-slave synchronization status information to the standby node in computing unit segment 2. The standby node then deserializes the information and compares the log identifier with the latest playback log identifier. If it determines that the "apply LSN" in the read request's log identifier is less than the "replay LSN" on the standby node, the read request is processed.

[0066] In one or more embodiments of the present application, if the data processing request is a read request, the read request is sent to the standby node that has completed the data synchronization operation on the target data corresponding to the log identifier, so that the standby node can feedback the target data based on the log identifier.

[0067] In actual applications, the standby node can synchronize data from the master node. For example, the primary sender process sends a message to the standby node Mirror, and then the standby node Mirror's mirror consumer and other processes parse the message and execute the change. XLOG performs the same operation through the XLogWrite function (write XLOG), synchronizing the XLOG update. In addition, the standby node HIA needs to synchronize the WAL log from the master node and use the WAL log to replay the data and configuration files. Since the WAL log stores the modified data content, the required data and configuration files can be obtained by replaying the WAL log. After receiving the LSN of the read request, the standby node in the computing unit can replay the WAL log. After the replay is completed, the coordinating node master receives the target data fed back by the standby node.

[0068] The target data is obtained by the standby node replaying the log information corresponding to the log identifier when the replay log identifier in the standby node is not less than the log identifier.

[0069] Because distributed transactions are processed very quickly in actual applications, there is a possibility of a sudden failure of the standby node or computing unit during the process from the coordination node responding to the read request to sending the read request to the standby node of the target computing unit. If a failure occurs, data rollback will occur, causing the latest playback log identifier in the current standby node to change to an earlier playback log identifier, resulting in the inability to satisfy the read request. Therefore, in order to ensure the reliable reading of data on the standby node, before sending the target data, the standby node is also required to compare the size relationship between the playback log identifier and the log identifier in the read request. Only when the comparison confirms that the playback log identifier is greater than the log identifier in the read request can the required target data be provided by the standby node.

[0070] For example, using the above solution, to ensure data consistency across all computing units, you need to set synchronous_commit to on. This ensures consistency across all computing unit segments. It should be noted that if synchronous_commit is on, during transaction processing, the standby node must replay the WAL log provided by the primary node before committing the transaction.

[0071] In actual application, if the data processing request is a write request, the write request is sent to the master node in the target computing unit. Figure 1 As shown in the figure, each computing unit segment has a primary node with both read and write capabilities. Since read requests can be completed by at least one backup node, but data writes cannot be completed, when a write request is received from the coordinating node, it is assigned to the primary node in the corresponding computing unit. Therefore, in a distributed database, multiple computing units, each containing a primary node and multiple backup nodes, achieve a single write, multiple reads effect. This effectively improves data processing capabilities while ensuring data security through the use of backup nodes.

[0072] In one or more embodiments of the present application, it also includes: receiving the target data fed back by the standby node. The target data is obtained by the standby node replaying the log information corresponding to the log identifier when the replay log identifier obtained by replaying after the standby node receives at least one log information is not less than the log identifier. In other words, in order to improve reading efficiency, if synchronous_commit is set to off, then during the transaction processing, multiple sync lsns are accumulated and then the WAL log is read for replay, that is, when receiving the sync lsn, the standby node will perform counting statistics, and when the number of received sync lsns is not less than the counting threshold, the WAL will be replayed. Specifically,

[0073] like Figure 5 A schematic diagram of a target data acquisition process provided by an embodiment of the present application. Figure 5 As you can see, the system includes the master coordinating node, the primary node in the computing unit segment, and the standby mirror node. The standby mirror node is divided into the log receiving module mirror walreceiver, the log playback module mirrorstartup, and the data reading module mirrorread according to its function. The specific process is as follows:

[0074] First, assume the count threshold is 2. The primary node in the computing unit synchronizes the log identifier, sync lsn1 (data synchronization log identifier), with the standby node. After the mirror walreceiver in the standby node receives the sync message, it replies "sync lsn1 ack" to the primary node. Furthermore, the primary node sends sync lsn2 to the mirror walreceiver. At this point, the standby node has received two consecutive sync lsns, satisfying the count threshold. Therefore, the standby node can proceed with WAL log playback.

[0075] If a read request is received from the coordinating node at this time, and the read request includes sync lsn2, the mirror must wait until the WAL log corresponding to sync lsn2 is replayed before continuing. At this point, mirror startup replays replaylsn2 and sends the replayed data to the data reading module mirror read. Mirror read, based on the read request for sync lsn2, feeds the corresponding target data back to the coordinating node.

[0076] If the mirror read receives the read request "read, get sync lsn2" from the coordinating node master after replaying the WAL logs for sync lsn2, the mirror can directly provide the required target data to the coordinating node because the mirror has already completed replaying the WAL logs for sync lsn1 and sync lsn2.

[0077] It should be noted that in Figure 5During execution, the illustrated solution uses a centralized playback method that accumulates multiple sync LSNs before performing centralized playback. To avoid errors caused by read requests seeking data corresponding to logs that have not yet been played back before playback is complete, after receiving a read request from the master, the mirror compares the size of the mirror's replay LSN with the size of the sync LSN. For example, after receiving a "read, get sync lsn2" command, the mirror read compares sync LSN2 with the reply LSN of the most recently completed replay, ensuring that the reply LSN is greater than or equal to sync LSN2 in the read request. This effectively mitigates the adverse impact of low mirror write efficiency when replaying each sync LSN individually.

[0078] As can be seen from the above embodiments, each master is configured with multiple segments, and each segment is configured with multiple mirrors. During use, the mirror does not provide data read services to user clients and only replaces the primary when the primary fails. Therefore, when the system is running stably, although the data in the mirror is updated synchronously, it remains idle. Therefore, these idle resources can be utilized without changing the original mirror's safety function of emergency primary replacement, and the utilization rate of the mirror can be improved.

[0079] Based on the same idea, the technical solution of this application also provides a data processing method. The execution subject of this method can be a standby node in the computing unit. Figure 6 A flow chart of another data processing method provided in an embodiment of the present application. Figure 6 As can be seen in , the method includes the following steps:

[0080] 601: Receive log information sent by the master node.

[0081] 602: Perform a data synchronization operation according to the log information to obtain synchronized data with the master node.

[0082] 603: After receiving the read request, obtain the target data specified by the read request from the synchronized data.

[0083] 604: Feedback the target data to the read request sender.

[0084] During the distributed transaction processing, the master node has read and write functions, and meets the read and write requirements of the coordination node during normal operation. After the master node receives the log information and data that need to be processed, the standby node also needs to back up the master node. In the technical solution of the present application, the standby node can be used to meet the read-only requirements of the coordination node. The master node will send new log information to the standby node at any time. After receiving the log information provided by the master node, the standby node performs data synchronization operations. During the synchronization operation, data can be directly synchronized, and some configuration files can be obtained by replaying the log information, so that the standby node can obtain synchronized data with the master node.

[0085] Furthermore, after the standby node receives the read request sent by the coordination node, the standby node will search for the target data from the data that has been replayed in the standby node according to the log identifier carried in the read request, and then send the target data to the corresponding read request sender. For example, the client (the client mentioned here can be understood as the end that has the need to read data from the standby node) sends a data acquisition request to the coordination node according to its own needs. After the coordination node receives the data acquisition request from the client, it selects a suitable computing unit for providing the target data based on the data synchronization and data playback conditions in each computing unit, and sends a read request to the computing unit. The read request carries the corresponding log identifier, and then the required target data is read from the corresponding standby node based on the log identifier. The details are as follows:

[0086] Upon receiving the read request, the method retrieves the target data specified by the read request from the synchronized data, including: upon receiving the read request, querying the synchronized data to determine whether the target data matches the log identifier carried in the read request. If the target data is not found, the method waits until the synchronized data contains the target data, and then feeds back the target data to the sender of the read request.

[0087] In actual applications, the way a standby node searches for target data based on the sync LSN carried in a received read request is related to the standby node's playback method. As previously mentioned, after receiving the sync LSN and corresponding WAL log in the log information, the standby node replays the WAL log and then commits the transaction. However, this process of replaying each sync LSN and corresponding WAL log and then writing it to disk is time-consuming and prevents the timely receipt of new log information. Therefore, a method can be used to accumulate multiple sync LSNs before replaying and then writing to disk. This allows for the timely receipt of new log information and ensures consistent data synchronization.

[0088] If multiple sync LSNs are accumulated before playback, the sync LSN carried in the read request may precede the reply LSN of the currently completed playback. In other words, the standby node cannot query the target data that satisfies the read request. Therefore, after receiving the sync LSN of a read request, the standby node will wait for the playback operation to be performed until the reply LSN corresponding to the sync LSN is obtained.

[0089] In practical applications, querying whether target data matching the log identifier carried in the read request exists in the synchronized data includes: obtaining the log identifier carried in the read request; receiving at least one log message synchronized by the master node; replaying the at least one log message to obtain a replay log identifier and the synchronized data; and finding the target data corresponding to the log identifier in the synchronized data when the replay log identifier is not less than the log identifier.

[0090] Before the standby node provides the target data, it needs to compare the reply lsn of the log currently played back by the standby node with the sync lsn in the received read request. Only when the reply lsn is not less than the sync lsn can the target data required by the coordinating node be provided. For details, please refer to the previous article Figure 5 The corresponding embodiments will not be repeated here.

[0091] It should be noted that in the technical solution of the present application, when the coordinating node has a data read request, if it is determined that a certain standby node in the computing unit can provide the data, the coordinating node will send the read request directly to the corresponding standby node in the computing unit without the need for the master node as a transit. This can effectively improve the data processing efficiency of the distributed system. In order to be able to respond to the user's read request more promptly and efficiently, when it is found after query that the standby node is temporarily unable to provide the target data required for the read request, the master node can provide the target data if the master node is not very busy. If the master node is always busy, the standby node can wait for the standby node to complete the WAL log playback corresponding to the target data, and then provide the corresponding target data.

[0092] In order to implement the technical solution of this application, the master node and the backup node in the computing unit need to send their master-backup synchronization status information to the coordination node in a timely manner. In actual applications, the master node and the backup node can each send their own master-backup synchronization status information, or the backup node can send its own master-backup synchronization status information to the master node, and then the master node sends its own master-backup synchronization status information and the backup node's master-backup synchronization status information to the coordination node. The method of sending the master-backup synchronization status information is as follows: Figures 1 to 5The explanation has been made in the illustrated embodiment and will not be repeated here.

[0093] Based on the same idea, the embodiment of the present application also provides a data processing system. Figure 7 This is a schematic diagram of the data processing system structure provided in the embodiment of the present application. Figure 7 As can be seen in the figure, the system includes:

[0094] A coordination node device 71 is used to save the master-slave synchronization status information of multiple computing units; wherein, any computing unit includes a master node and at least one standby node, and the master-slave synchronization status information includes the playback log identifier of any standby node; the playback log identifier is determined after the standby node performs data playback using the synchronization log of the master node belonging to the same computing unit; in response to a received data processing request, a target computing unit is determined from the multiple computing units based on the log identifier carried in the data processing request and the master-slave synchronization status information of the multiple computing units; if the data processing request is a read request, the read request is sent to at least one standby node in the target computing unit, so that the at least one standby node responds to the read request to feedback the corresponding target data; if the data processing request is a write request, the write request is sent to the master node in the target computing unit.

[0095] The standby node device 72 is configured to receive log information sent by the master node. Based on the log information, it performs data synchronization operations to obtain synchronized data with the master node. Upon receiving a read request, it retrieves the target data specified in the read request from the synchronized data. It then feeds the target data back to the sender of the read request.

[0096] Based on the same idea, the embodiment of the present application also provides a data processing device. Figure 8 This is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application. The data processing device includes:

[0097] The saving module 81 is used to save the master-slave synchronization status information of multiple computing units; wherein, any computing unit includes a master node and at least one standby node, and the master-slave synchronization status information includes the replay log identifier of any standby node; the replay log identifier is determined after the standby node uses the synchronization log of the master node belonging to the same computing unit to perform data playback.

[0098] The determination module 82 is configured to respond to a received data processing request and determine a target computing unit from the multiple computing units according to a log identifier carried in the data processing request and the master-slave synchronization status information of the multiple computing units.

[0099] The reading module 83 is configured to send the read request to at least one standby node in the target computing unit if the data processing request is a read request, so that the at least one standby node responds to the read request to feed back corresponding target data.

[0100] The writing module 84 is configured to send the write request to the master node in the target computing unit if the data processing request is a write request.

[0101] Optionally, the acquisition module 81 is also used to receive the heartbeat message of the computing unit; obtain the replay log identifier carried in the heartbeat message; if the replay log identifier is greater than any stored log identifier, obtain the master-slave synchronization status information of the standby node carried in the heartbeat message.

[0102] Optionally, the acquisition module 81 is also used to determine the transaction status of the master node and the backup node in the computing unit; if the transaction status of the master node and the at least one backup node is in the ready state, send a transaction submission request to the master node and the at least one backup node in the ready state; and receive the master-backup synchronization status information of the computing unit based on the feedback of the transaction submission request.

[0103] Optionally, the determination module 82 is also used to receive the data processing request; obtain the latest playback log identifier contained in the master-slave synchronization status information, and the log identifier carried in the data processing request; if the latest playback log identifier is greater than the log identifier, determine the target computing unit corresponding to the log identifier from the multiple computing units.

[0104] Optionally, the sending module 83 is also used to send the read request to the standby node that has completed the data synchronization operation on the target data corresponding to the log identifier if the data processing request is a read request, so that the standby node can feedback the target data based on the log identifier.

[0105] Optionally, a receiving module 85 is further included for receiving the target data fed back by the standby node; wherein, the target data is obtained by the standby node replaying the log information corresponding to the log identifier when the replay log identifier in the standby node is not less than the log identifier.

[0106] Optionally, a receiving module 85 is also included, which is also used to receive the target data fed back by the standby node; wherein, the target data is obtained by the standby node replaying the log information corresponding to the log identifier when the replay log identifier obtained by replaying after the standby node receives at least one log information is not less than the log identifier.

[0107] An embodiment of the present application also provides an electronic device. The electronic device is a master node electronic device in a computing unit. Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device includes a memory 901, a processor 902 and a communication component 903; wherein,

[0108] The memory 901 is used to store programs;

[0109] The processor 902 is coupled to the memory and is configured to execute the program stored in the memory to:

[0110] Storing master-slave synchronization status information of multiple computing units; wherein each computing unit includes a master node and at least one slave node, and the master-slave synchronization status information includes a playback log identifier of each slave node; the playback log identifier is determined by the slave node after performing data playback using the synchronization log of the master node belonging to the same computing unit;

[0111] In response to the received data processing request, determining a target computing unit from the multiple computing units according to a log identifier carried in the data processing request and master / slave synchronization status information of the multiple computing units;

[0112] If the data processing request is a read request, sending the read request to at least one standby node in the target computing unit, so that the at least one standby node responds to the read request to feed back corresponding target data;

[0113] If the data processing request is a write request, the write request is sent to the master node in the target computing unit.

[0114] The memory 901 can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0115] Furthermore, the processor 902 in this embodiment may specifically be: a programmable switching processing chip, which is configured with a data replication engine and can replicate received data.

[0116] When the processor 902 executes the program in the memory, in addition to the above functions, it can also realize other functions, for details, please refer to the description of the above embodiments. Figure 9 As shown, the electronic device further includes: a power supply component 904 and other components.

[0117] Based on the same idea, the embodiment of the present application also provides a data processing device. Figure 10 This is a schematic diagram of the structure of another data processing device provided in an embodiment of the present application. The data processing device includes:

[0118] The receiving module 1001 is used to receive log information sent by the master node.

[0119] The execution module 1002 is used to perform a data synchronization operation according to the log information to obtain synchronized data with the master node.

[0120] The receiving module 1001 is further configured to, after receiving a read request, obtain the target data specified by the read request from the synchronized data.

[0121] The sending module 1003 is configured to feed back the target data to the read request sender.

[0122] The receiving module 1001 is further configured to, upon receiving a read request, query whether there is target data in the synchronized data that matches the log identifier carried in the read request;

[0123] If no data is found, the system is in a waiting state until the data synchronization operation is performed and the synchronized data contains the target data, and the target data is fed back to the sender of the read request.

[0124] The receiving module 1001 is further configured to obtain the log identifier carried in the read request;

[0125] Receiving at least one log information synchronized by the master node;

[0126] replaying the at least one log information to obtain a replay log identifier and the synchronized data;

[0127] When the playback log identifier is not less than the log identifier, the target data corresponding to the log identifier is found from the synchronized data.

[0128] The embodiment of the present application further provides a non-transitory machine-readable storage medium, wherein the non-transitory machine-readable storage medium stores executable code, and when the executable code is executed by a processor of an electronic device, the processor executes Figures 1 to 5 The method described in the corresponding embodiment.

[0129] The present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, enables the processor to implement Figures 1 to 5 The method described in the corresponding embodiment.

[0130] An embodiment of the present application also provides an electronic device. The electronic device is a standby node electronic device in a computing unit. Figure 11 This is a schematic diagram of the structure of another electronic device provided in an embodiment of the present application. The electronic device includes a memory 1101, a processor 1102, and a communication component 1103; wherein,

[0131] The memory 1101 is used to store programs;

[0132] The processor 1102 is coupled to the memory and configured to execute the program stored in the memory to:

[0133] Receive log information sent by the master node;

[0134] Performing data synchronization operations according to the log information to obtain synchronized data with the master node;

[0135] After receiving the read request, obtaining the target data specified by the read request from the synchronized data;

[0136] Feedback the target data to the read request sender.

[0137] The memory 1101 can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0138] Furthermore, the processor 1102 in this embodiment may specifically be: a programmable switching processing chip, which is configured with a data replication engine and can replicate received data.

[0139] When the processor 1102 executes the program in the memory, in addition to the above functions, it can also realize other functions, for details, please refer to the description of the above embodiments. Figure 11 As shown, the electronic device further includes: a power supply component 1104 and other components.

[0140] The embodiment of the present application further provides a non-transitory machine-readable storage medium, wherein the non-transitory machine-readable storage medium stores executable code, and when the executable code is executed by a processor of an electronic device, the processor executes Figure 6 The method described in the corresponding embodiment.

[0141] The present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, enables the processor to implement Figure 6 The method described in the corresponding embodiment.

[0142] Based on the above embodiment, master-slave synchronization status information for multiple computing units is stored; each computing unit includes a master node and at least one slave node, and the master-slave synchronization status information includes a playback log identifier for each slave node; the playback log identifier is determined by the slave node after performing data playback using the synchronization log of the master node belonging to the same computing unit. In response to a received data processing request, a target computing unit is determined from the multiple computing units based on the log identifier carried in the data processing request and the master-slave synchronization status information of the multiple computing units. If the data processing request is a read request, the read request is sent to at least one slave node in the target computing unit, so that the at least one slave node responds to the read request and returns the corresponding target data. If the data processing request is a write request, the write request is sent to the master node in the target computing unit. Using the above technical solution, for a computing unit containing one master node and multiple slave nodes, the master-slave synchronization status information of the master and slave nodes is sent to the coordination node in real time. After receiving the data processing request, the coordination node selects the corresponding slave node or master node to perform the corresponding data processing action based on the type of the data processing request. For example, when data is required to be read, the appropriate backup node is selected based on the log identifier carried in the read request, and the backup node then provides the target data corresponding to the log identifier. While utilizing the backup node for secure data backup, it also enables the backup node to satisfy the data read requirements of the coordination node. Based on the coordination node, the master node, and at least one backup node, a single-write, multiple-read data processing effect is achieved, ensuring strong data processing consistency and effectively improving data processing efficiency.

[0143] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0144] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, applied to a distributed database, comprising a coordinating node and multiple computing units, each of which comprises a master node and at least one standby node to replace the master node in the event of a failure of the master node, wherein data stored in the standby node is obtained by replaying log information sent by the master node. The method, executed by the coordinating node, comprises: Save the master-slave synchronization status information of multiple computing units, including the playback log identifier of any standby node; The replay log identifier is determined by the standby node after performing data replay using the synchronization log of the master node belonging to the same computing unit; In response to the received data processing request, determining a target computing unit from the plurality of computing units; If the data processing request is a read request and it is determined according to the master-slave synchronization status information of the target computing unit that the log replay identifier of the standby node of the target computing unit is not less than the log identifier of the read request, the read request is sent to the standby node whose replay log identifier is not less than the log identifier of the read request, so that the standby node responds to the read request to feed back the corresponding target data; if the data processing request is a read request and it is determined according to the master-slave synchronization status information of the target computing unit that the log replay identifier of the standby node of the target computing unit is less than the log identifier of the read request, the read request is sent to the master node, so that the master node responds to the read request to feed back the corresponding target data; Alternatively, if the data processing request is a read request, a standby node is selected from at least one standby node of the target computing node according to the master-standby synchronization state information of the target computing unit, and the read request is sent to the selected standby node, so that the standby node responds to the read request and directly feeds back the corresponding target data when its log replay identifier is not less than the log identifier of the read request, or feeds back the corresponding target data after waiting for the completion of playback of the log information corresponding to the log identifier when its replay log identifier is less than the log identifier of the read request; If the data processing request is a write request, the write request is sent to the master node in the target computing unit.

2. The method according to claim 1, wherein the storing of the master-slave synchronization status information of the plurality of computing units comprises: Receiving a heartbeat message from the computing unit; Obtain the playback log identifier carried in the heartbeat message; If the playback log identifier is greater than any stored log identifier, the master-slave synchronization state information carried in the heartbeat message is saved.

3. The method according to claim 1, wherein the storing of the master-slave synchronization status information of the plurality of computing units comprises: Determining the transaction status of the master node and the backup node in the computing unit; If the transaction status of the master node and the at least one standby node is in the prepared state, sending a transaction commit request to the master node and the at least one standby node in the prepared state; The master-slave synchronization state information of the computing unit fed back based on the transaction commit request is received.

4. The method according to claim 1, further comprising: Receive the target data fed back by the standby node.

5. A data processing method, applied to a distributed database, comprising a coordinating node and multiple computing units, each of which comprises a master node and at least one standby node to replace the master node in the event of a failure of the master node. The data stored in the standby node is obtained by replaying log information sent by the master node. The method, executed by the standby node, comprises: Receive log information sent by the master node; Performing data synchronization operations according to the log information to obtain synchronized data with the master node; Receive a read request; The read request is processed by the coordinating node by executing the following processing method: if the received data processing request is a read request and it is determined based on the master-slave synchronization status information of the target computing unit that the log replay identifier of the standby node of the target computing unit is not less than the log identifier of the read request, then the read request is sent to the standby node whose replay log identifier is not less than the log identifier of the read request; if the data processing request is a read request and it is determined based on the master-slave synchronization status information of the target computing unit that the log replay identifier of the standby node of the target computing unit is less than the log identifier of the read request, then the read request is sent to the master node; or, if the data processing request is a read request, a standby node is selected from at least one standby node of the target computing node based on the master-slave synchronization status information of the target computing unit, and the read request is sent to the selected standby node; In response to the read request, when its log playback identifier is not less than the log identifier of the read request, directly obtain the target data specified by the read request from the synchronized data; or when its playback log identifier is less than the log identifier of the read request, wait for the completion of playback of the log information corresponding to the log identifier, and then obtain the target data specified by the read request from the synchronized data; Feedback the target data to the read request sender.

6. A data processing system, applied to a distributed database, comprising a coordinating node and multiple computing units, each of which includes a master node and at least one standby node to replace the master node in the event of a failure of the master node. The data stored in the standby node is obtained by replaying log information sent by the master node. The coordinating node is configured to store master-slave synchronization status information of the multiple computing units, the master-slave synchronization status information including a replay log identifier of any standby node; the replay log identifier is determined by the standby node after performing data replay using the synchronization log of the master node belonging to the same computing unit; The coordination node is further configured to determine a target computing unit from the plurality of computing units in response to a received data processing request; The coordination node is also used to send the read request to the standby node whose playback log identifier is not less than the log identifier of the read request if the data processing request is a read request and it is determined based on the master-slave synchronization status information of the target computing unit that the log playback identifier of the standby node of the target computing unit is not less than the log identifier of the read request, so that the standby node responds to the read request to feedback the corresponding target data; if the data processing request is a read request and it is determined based on the master-slave synchronization status information of the target computing unit that the log playback identifier of the standby node of the target computing unit is less than the log identifier of the read request, then the read request is sent to the standby node. the master node, so that the master node responds to the read request to feed back the corresponding target data; or, if the data processing request is a read request, selecting a standby node from at least one standby node of the target computing node according to the master-standby synchronization status information of the target computing unit, and sending the read request to the selected standby node, so that the standby node responds to the read request and directly feeds back the corresponding target data when its log playback identifier is not less than the log identifier of the read request, or waits for the log information corresponding to the log identifier to be played back after completion of playback, and then feeds back the corresponding target data when its playback log identifier is less than the log identifier of the read request; The coordinating node is further configured to send the write request to the master node in the target computing unit if the data processing request is a write request; The standby node is configured to receive log information sent by the master node; perform data synchronization operations based on the log information to obtain synchronized data synchronized with the master node; receive the read request; respond to the read request, and when its log playback identifier is not less than the log identifier of the read request, directly obtain the target data specified by the read request from the synchronized data, or when its playback log identifier is less than the log identifier of the read request, wait for the completion of playback of the log information corresponding to the log identifier, and then obtain the target data specified by the read request from the synchronized data; and feed back the target data to the sender of the read request.

7. An electronic device comprising a memory and a processor; wherein: The memory is used to store programs; The processor is coupled to the memory, and is configured to execute the program stored in the memory to implement the method according to any one of claims 1 to 4; or the method according to claim 5.

8. A non-transitory machine-readable storage medium having executable code stored thereon, wherein when the executable code is executed by a processor of an electronic device, the processor is caused to execute the method according to any one of claims 1 to 4; or the method according to claim 5 above.

9. A computer program product comprising a computer program / instruction, which, when executed by a processor, causes the processor to implement the method according to any one of claims 1 to 4; or the method according to claim 5.

Images