Data processing method and device, node and storage medium

Abstract

The embodiment of the invention discloses a data processing method and device, a node and a storage medium, relates to the technical field of computers, and is used for improving the problem that a response process in a distributed system is relatively large in time delay due to a general technology so as to improve the throughput of the distributed system. The data processing method is applied to a data processing node in a distributed system, and comprises the following steps: receiving a data processing request from a client; the data processing request is used for requesting to process to-be-processed data; the data processing request comprises a write request or a read request; in response to the data processing request, determining distribution information of to-be-processed data according to the data state set; the data state set is used for recording distribution information of the data in the distributed system; executing a corresponding processing strategy based on the distribution information of the to-be-processed data to respond to the data processing request; wherein the processing strategy comprises not indicating the distributed node to process the to-be-processed data, or indicating the distributed node to process the to-be-processed data.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, device, node and storage medium. Background Art

[0002] In a distributed system, data consistency is required between nodes.

[0003] In order to improve the consistency of data between nodes, the master-slave coordination method can be used to implement the distributed system's response to data processing requests. This method requires multiple steps such as master-slave coordination and data processing during the response process of data processing requests, which can easily cause a large delay in the response process and may lead to a decrease in the throughput of the distributed system. Summary of the invention

[0004] The present application provides a data processing method, device, node and storage medium, which are used to improve the problem that general technology easily causes a large response process delay in a distributed system, so as to improve the throughput of the distributed system.

[0005] In a first aspect, the present application provides a data processing method, which is applied to a data processing node in a distributed system, the method comprising: receiving a data processing request from a client; the data processing request is used to request processing of data to be processed; the data processing request comprises a write request or a read request; in response to the data processing request, determining distribution information of the data to be processed according to a data state set; the data state set is used to record distribution information of data in a distributed system; executing a corresponding processing strategy based on the distribution information of the data to be processed to respond to the data processing request; wherein the processing strategy comprises not instructing the distributed node to process the data to be processed, or instructing the distributed node to process the data to be processed.

[0006] In the present technical solution, since the data processing node stores a data state set, and the data state set is used to record the distribution information of the data in multiple distributed nodes. Therefore, after receiving the data processing request from the client, it can be determined whether to instruct the distributed node to process the data to be processed based on the data identifier and the data state set to ensure the consistency of the data in multiple distributed nodes. If it is determined that the distributed node is not instructed to process the data to be processed, the response can be ended directly. In this case, compared to the general technology that requires master-slave coordination before ending the response, resulting in two or more round-trip delays, the present application can support the delay of the entire response process as a round-trip delay between the client and the data processing node, which can effectively reduce the delay of the response process and improve the throughput of the distributed system.

[0007] If it is determined to instruct the distributed node to process the data to be processed, the data processing node can further instruct the distributed node to process the data to be processed in response to the data processing request. In this case, compared to the general technology that requires master-slave coordination before processing the data to be processed, resulting in two or more round-trip delays, the present application can support the delay of the entire response process as a round-trip delay between the client and the distributed node, which can effectively reduce the delay of the response process and improve the throughput of the distributed system. Therefore, the present application can be used to improve the problem that the general technology easily causes a large delay in the response process in the distributed system, so as to improve the throughput of the distributed system.

[0008] In one possible implementation, a data status set is used to record the correspondence between the data identification and distribution information of the data; the distribution information includes status information and write node information; the status information is used to indicate whether the data is consistent in the distributed system, and the write node information is used to indicate the distributed nodes in the distributed system that have not written data and / or the distributed nodes that have written data.

[0009] In one possible implementation, a method for determining distribution information of data to be processed based on a data state set specifically includes: querying the data state set with the data identifier of the data to be processed as an index; if there is no distribution information corresponding to the data identifier of the data to be processed, the distribution information is empty; if there is distribution information corresponding to the data identifier of the data to be processed, the distribution information corresponding to the data identifier of the data to be processed is used as the distribution information of the data to be processed.

[0010] In one possible implementation, the data processing request is a write request, and a method for responding to the data processing request by executing a corresponding processing strategy based on the distribution information of the data to be processed specifically includes: when the distribution information of the data to be processed indicates that the data to be processed has not reached a consensus in the distributed system, not instructing the distributed nodes to write the data to be processed, and sending a first response to the client to indicate that the data to be processed will not be written; when the distribution information of the data to be processed indicates that the data to be processed has reached a consensus in the distributed system, or the distribution information is empty, instructing the distributed nodes to write the data to be processed.

[0011] In one possible implementation, a data processing node stores a current index, which is greater than other indexes, and corresponding data are consistent in the distributed system; a method for instructing a distributed node to write data to be processed specifically includes: generating a to-be-committed index corresponding to a write request, which is greater than the current index; sending a data write indication to multiple distributed nodes in the distributed system, the data write indication carrying the to-be-committed index and the to-be-processed data; when the to-be-processed data is consistent in the distributed system, sending a second response to the client to characterize that the to-be-processed data has been written.

[0012] In a possible implementation, the data processing method further includes: storing in a data status set a correspondence between an index to be submitted, a data identifier, status information on which a plurality of distributed nodes have not reached consensus, and write node information.

[0013] In one possible implementation, the data processing method also includes: when receiving write confirmation information corresponding to the index to be submitted sent by the distributed node, updating the write status of the distributed node in the write node information corresponding to the index to be submitted to written; when the current ratio is greater than or equal to a preset ratio threshold, updating the status information on which multiple distributed nodes corresponding to the index to be submitted have not reached a consensus to status information on which multiple distributed nodes have reached a consensus; the current ratio is used to characterize the ratio between the number of distributed nodes whose write status is written and the number of multiple distributed nodes.

[0014] In a possible implementation, the data processing method further includes: when the number of distributed nodes in the write state that have been written is equal to the number of the plurality of distributed nodes, deleting information corresponding to the to-be-committed index in the data state set.

[0015] In a possible implementation, when the current ratio is greater than or equal to a preset ratio threshold, the method further includes: updating the to-be-submitted index to the current index.

[0016] In a possible implementation, the method for sending a data write indication to multiple distributed nodes in a distributed system specifically includes: sending the data write indication to the multiple distributed nodes by multicast.

[0017] In a possible implementation, the data processing method further includes: storing log information corresponding to the write request in a preset cache location; the log information includes a correspondence between the data to be processed, the data identifier, and the index to be submitted.

[0018] In one possible implementation, the data processing node performs heartbeat communications with multiple distributed nodes in the distributed system respectively; the method also includes: when the waiting time corresponding to the distributed node is greater than or equal to a first time threshold and the heartbeat communication with the distributed node is normal, reading the log information corresponding to the write request from a preset cache location, and re-sending the data write indication to the distributed node based on the log information; the waiting time is used to represent the time for waiting for the distributed node to send the write confirmation information corresponding to the index to be submitted.

[0019] In one possible implementation, the data processing method also includes: when the interruption duration of heartbeat communication between the distributed nodes in the distributed system is greater than or equal to the second duration threshold, deleting the configuration information corresponding to the distributed node and updating the number of distributed nodes in the distributed system.

[0020] In one possible implementation, the data processing request is a read request, and a method for responding to the data processing request by executing a corresponding processing strategy based on the distribution information of the data to be processed specifically includes: when the distribution information of the data to be processed indicates that the data to be processed has not reached a consensus in the distributed system, or the distribution information is empty, instructing a first distributed node to read the data to be processed, and the first distributed node is included in multiple distributed nodes in the distributed system; when the distribution information of the data to be processed indicates that the data to be processed has reached a consensus in the distributed system, instructing a second distributed node to read the data to be processed, and the second distributed node is included in the distributed nodes that have written the data to be processed.

[0021] In one possible implementation, a method for instructing a first distributed node to read data to be processed specifically includes: determining a first distributed node that meets the load balancing condition among multiple distributed nodes, and sending a data read indication carrying a current index to the first distributed node, so that the first distributed node responds to the read request; a method for instructing a second distributed node to read data to be processed specifically includes: determining a second distributed node that has written the data to be processed and meets the load balancing condition, as indicated by write node information of the data to be processed, and sending a data read indication carrying the current index to the second distributed node, so that the second distributed node responds to the read request.

[0022] In a second aspect, a data processing device is provided, which is applied to a data processing node in a distributed system, and the device includes: a receiving unit, a determining unit and an executing unit; the receiving unit is used to receive a data processing request from a client; the data processing request is used to request processing of data to be processed; the data processing request includes a write request or a read request; the determining unit is used to determine the distribution information of the data to be processed according to a data state set in response to the data processing request received by the receiving unit; the data state set is used to record the distribution information of the data in the distributed system; the executing unit is used to execute a corresponding processing strategy based on the distribution information of the data to be processed determined by the determining unit, in response to the data processing request; wherein the processing strategy includes not instructing the distributed node to process the data to be processed, or instructing the distributed node to process the data to be processed.

[0023] In one possible implementation, a data status set is used to record the correspondence between the data identification and distribution information of the data; the distribution information includes status information and write node information; the status information is used to indicate whether the data is consistent in the distributed system, and the write node information is used to indicate the distributed nodes in the distributed system that have not written data and / or the distributed nodes that have written data.

[0024] In one possible implementation, the determination unit is specifically used to: query the data state set using the data identifier of the data to be processed as an index; if there is no distribution information corresponding to the data identifier of the data to be processed, the distribution information is empty; if there is distribution information corresponding to the data identifier of the data to be processed, the distribution information corresponding to the data identifier of the data to be processed is used as the distribution information of the data to be processed.

[0025] In one possible implementation, the execution unit is specifically used to: when the distribution information of the data to be processed indicates that the data to be processed has not reached a consensus in the distributed system, not instruct the distributed nodes to write the data to be processed, and send a first response to the client to indicate that the data to be processed will not be written; when the distribution information of the data to be processed indicates that the data to be processed has reached a consensus in the distributed system, or the distribution information is empty, instruct the distributed nodes to write the data to be processed.

[0026] In one possible implementation, a data processing node stores a current index, which is greater than other indexes and corresponding data are consistent in the distributed system; an execution unit is specifically used to: generate a to-be-committed index corresponding to a write request, which is greater than the current index; send a data write indication to multiple distributed nodes in the distributed system, the data write indication carries the to-be-committed index and the to-be-processed data, and the execution unit is also used to send a second response to the client to represent the written to-be-processed data when the to-be-processed data is consistent in the distributed system.

[0027] In a possible implementation, the data processing device further includes: a storage unit; a storage unit configured to store, in a data status set, a correspondence between an index to be submitted, a data identifier, status information on which multiple distributed nodes have not reached consensus, and write node information.

[0028] In one possible implementation, the data processing device also includes: an update unit; an update unit, which is used to update the write status of the distributed node in the write node information corresponding to the index to be submitted to written when receiving write confirmation information corresponding to the index to be submitted sent by the distributed node; the update unit is also used to update the status information on which multiple distributed nodes corresponding to the index to be submitted have not reached a consensus to status information on which multiple distributed nodes have reached a consensus when the current ratio is greater than or equal to a preset ratio threshold; the current ratio is used to characterize the ratio between the number of distributed nodes whose write status is written and the number of multiple distributed nodes.

[0029] In one possible implementation, the data processing device also includes: a deletion unit; a deletion unit, which is used to delete the information corresponding to the index to be submitted in the data status set when the number of distributed nodes in the write status is equal to the number of multiple distributed nodes.

[0030] In a possible implementation, the updating unit is further configured to update the to-be-submitted index to the current index.

[0031] In a possible implementation, the execution unit is specifically configured to: multicast a data write indication to multiple distributed nodes.

[0032] In a possible implementation, the storage unit is further used to store log information corresponding to the write request in a preset cache location; the log information includes a correspondence between the data to be processed, the data identifier, and the index to be submitted.

[0033] In one possible implementation, the data processing node performs heartbeat communication with multiple distributed nodes in the distributed system respectively; the execution unit is also used to read the log information corresponding to the write request from a preset cache location when the waiting time corresponding to the distributed node is greater than or equal to the first time threshold and the heartbeat communication with the distributed node is normal, and re-send the data write indication to the distributed node based on the log information; the waiting time is used to represent the time for waiting for the distributed node to send the write confirmation information corresponding to the index to be submitted.

[0034] In one possible implementation, the deletion unit is also used to delete the configuration information corresponding to the distributed node and update the number of distributed nodes in the distributed system when the interruption duration of the heartbeat communication between the distributed nodes in the distributed system is greater than or equal to the second duration threshold.

[0035] In one possible implementation, the execution unit is specifically used to: when the distribution information of the data to be processed indicates that the data to be processed has not reached a consensus in the distributed system, or the distribution information is empty, instruct a first distributed node to read the data to be processed, and the first distributed node is included in the multiple distributed nodes of the distributed system; when the distribution information of the data to be processed indicates that the data to be processed has reached a consensus in the distributed system, instruct a second distributed node to read the data to be processed, and the second distributed node is included in the distributed nodes that have written the data to be processed.

[0036] In one possible implementation, the execution unit is specifically used to: determine a first distributed node that meets the load balancing condition among multiple distributed nodes, and send a data read indication carrying a current index to the first distributed node, so that the first distributed node responds to the read request; the execution unit is specifically used to: determine a second distributed node that has written the data to be processed as indicated by the write node information of the data to be processed and that meets the load balancing condition, and send a data read indication carrying the current index to the second distributed node, so that the second distributed node responds to the read request.

[0037] In a third aspect, the present application provides a data processing node, which may include: a memory and a processor, the memory being used to store a computer program, the processor being used to call the computer program to execute the operating steps of the method in the first aspect or any possible implementation of the first aspect.

[0038] It should be noted that the memory and processor described in this application can be integrated on a single chip or can be separately arranged on different chips. This application does not limit the type of memory and the arrangement of the memory and the processor.

[0039] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program runs on a data processing node, the data processing node executes the operating steps of the method in the first aspect or any possible implementation of the first aspect.

[0040] In a fifth aspect, the present application provides a computer program product, which, when executed on a data processing node, enables the operation steps of the method in the first aspect or any possible implementation of the first aspect to be executed.

[0041] It can be understood that any of the data processing nodes or computer-readable storage media or computer program products provided above are used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of a response process provided in an embodiment of the present application;

[0043] Figure 2 A schematic diagram of another response process provided in an embodiment of the present application;

[0044] Figure 3 A schematic diagram of the structure of a distributed system provided in an embodiment of the present application;

[0045] Figure 4 A schematic diagram of the structure of another distributed system provided in an embodiment of the present application;

[0046] Figure 5 A schematic diagram of the structure of a data processing node provided in an embodiment of the present application;

[0047] Figure 6 A flowchart of a data processing method provided in an embodiment of the present application;

[0048] Figure 7 A flowchart of another data processing method provided in an embodiment of the present application;

[0049] Figure 8 A flowchart of another data processing method provided in an embodiment of the present application;

[0050] Fig. 9 A flowchart of another data processing method provided in an embodiment of the present application;

[0051] Fig.10 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The terms "first", "second", "third" and "fourth" etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects rather than to limit a specific order.

[0053] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0054] The following is a brief introduction to the terms and technologies involved in the embodiments of the present application to facilitate the reader's understanding.

[0055] Round trip time (RTT): refers to the time required for data to be transmitted from the sender to the receiver and then received by the sender to confirm the data. Optionally, there may be an intermediate node responsible for data forwarding between the sender and the receiver. The round trip time may include the processing delay of each intermediate node and the transmission delay when forwarding data.

[0056] Heartbeat mechanism: It is a mechanism used by both parties to ensure the validity of the connection. One of the two parties can send a custom structure (or heartbeat packet) to the other party at a preset time interval, so that the other party can determine whether the connection between the two parties is valid, and judge whether the connection between the two parties is valid by the other party's reply. Among them, the preset time can be 500 milliseconds, etc. When the two parties are connected based on the heartbeat mechanism, it can also be understood that the two parties have established heartbeat communication. This is a unified explanation here, and it will not be repeated below.

[0057] Distributed system: A software system built on a network, which can be composed of multiple dispersed computers and has the distribution of data processing. Optionally, the dispersed multiple computers include a master computer and slave computers, the master computer can be called a master node, and the slave computer can be called a slave node. The master node and the slave node can be connected through a heartbeat mechanism.

[0058] The data between nodes in a distributed system is consistent. In order to improve the consistency of data between nodes, the master-slave coordination method can be used to implement the distributed system's response process to data processing requests. For example, distributed databases such as Zookeeper and extended distributed key value (ETCD) based on Paxos and Raft consensus algorithms can achieve data consistency in distributed systems. However, this method is prone to large delays in the response process, which may lead to a decrease in the throughput of the distributed system. The following uses ETCD as an example to illustrate the delay in the distributed system reaching consistency.

[0059] ETCD is a consistent key-value storage system. With its advantages of ease of use and high reliability, it is widely used in scenarios such as service discovery, message publishing and subscription, metadata management, and distributed locks. ETCD can include clients and servers. The client can initiate write requests or read requests to the server. The server can receive write requests or read requests from the client and respond to the client. The server usually includes an odd number of nodes. Among the odd number of nodes, there is a master node (Leader) and multiple slave nodes (Follower).

[0060] The process of reaching consistency in the distributed system in ETCD specifically includes: all write requests initiated by the client need to reach the master node first, and the master node configures a globally unique number for each write request, and stores the data in each write request locally in the form of logs in sequence. In addition, the master node can synchronize the logs to the slave nodes. After the slave node completes the synchronization of the logs, it can return a synchronization confirmation to the master node. When the master node confirms that more than half of the nodes (including the master node) have completed the log synchronization, it can determine that the data in the write request has been successfully written to the distributed system, and send a write completion response to the client. At this time, the write request initiated by the client is confirmed to be completed, and the condition for confirming consistency is that more than half of the nodes on the server have completed the write. In addition, the master node can determine the globally unique number of the write request as the highest log number in the distributed system and store the highest log number. The highest log number can also be called Commit Index.

[0061] like Figure 1 As shown, a schematic diagram of a response process of a read request in ETCD provided by an embodiment of the present application. Figure 1 As shown, it includes a client, a master node, and two slave nodes. Figure 1In the response process of the read request shown, the client initiates a read request to the master node. After the master node receives the read request from the client, it can confirm whether it still has the master node identity. The master node can send a heartbeat packet to all slave nodes. Each slave node can respond to the heartbeat packet. After the master node receives the heartbeat packets responded by more than half of the nodes, it can confirm that it still has the master node identity, and read the current highest log number, thereby determining the log number corresponding to the read request based on the highest log number, and reading the data based on the log number corresponding to the read request. After the master node reads the data, it can send a response carrying the read data to the client.

[0062] Figure 1 The response process of the read request shown needs two RTTs, namely the RTT of the response data between the client and the master node, and the RTT of the identity confirmation between the master node and the slave node. In addition, when the client initiates a write request to the master node, the response process of the write request is similar to the response process of the read request, and also requires two RTTs.

[0063] like Figure 2 As shown, another schematic diagram of the response process of a read request in ETCD provided by an embodiment of the present application. Figure 2 As shown, it includes a client, a master node, and two slave nodes. Figure 2 In the response process of the read request shown, the client initiates a read request to the slave node. After the slave node receives the read request from the client, it can send a numbered read request for requesting the current highest log number to the master node. After the master node receives the numbered read request from the slave node, it needs to first confirm whether it still has the master node identity. The master node can send a heartbeat packet to all slave nodes. Each slave node can respond to the heartbeat packet. After the master node receives the heartbeat packet responded by more than half of the nodes, it can be confirmed that it still has the master node identity, and reads the current highest log number, and sends the current highest log number read to the slave node. Based on this, the slave node can determine the log number corresponding to the read request according to the highest log number, and read the data according to the log number corresponding to the read request, so as to further send a response carrying the read data to the client.

[0064] Figure 2 The response process of the read request shown needs to go through three RTTs, namely the RTT of the response data between the client and the slave node, the RTT of the identity confirmation between the master node and the slave node, and the RTT of the highest log number confirmation between the master node and the slave node. In addition, when the client initiates a write request to the slave node, the response process of the write request is similar to the response process of the read request, and also requires three RTTs. Compared with the response process of the client initiating a write request to the master node, there is an additional RTT for the slave node to forward the write request to the master node.

[0065] It can be seen that the general technology requires 2-3 RTTs to complete the read and write operations, which easily causes a large delay in the response process and may limit the throughput of the distributed system. In addition, in the general technology, both read and write operations need to be processed by the master node, and the master node is limited by the processing power of the central processing unit (CPU), which can easily become the bottleneck of the distributed system, resulting in poor scalability of the distributed system.

[0066] To this end, an embodiment of the present application provides a data processing method, device, node and storage medium. The method provided by the present application may include: after receiving a data processing request from a client, the distribution information of the data to be processed can be determined in response to the data processing request according to a data state set used to record the distribution information of the data in the distributed system, and whether to instruct the distributed node to process the data to be processed can be determined based on the distribution information of the data to be processed, and the corresponding processing strategy is executed to respond to the data processing request. If it is determined not to instruct the distributed node to process the data to be processed, a response of not executing this processing can be directly returned to the client. If it is determined to instruct the distributed node to process the data to be processed, the distributed node can be directly instructed to process the data to be processed in response to the data processing request.

[0067] Based on this, the present application can support the entire response process delay to be a round-trip delay between the client and the data processing node, or the entire response process delay to be a round-trip delay between the client and the distributed node. Compared with the general technology that requires master-slave coordination to determine whether to process the data to be processed, resulting in two or more round-trip delays, the present application can support one round-trip delay to complete the response process in multiple situations. Therefore, the present application can be used to improve the problem that general technology easily causes large response process delays in distributed systems, so as to improve the throughput of distributed systems.

[0068] The data processing method is applied to the data processing node in the distributed system. The distributed system provided by the embodiment of the present application is described below.

[0069] Figure 3 A schematic diagram of a distributed system provided in an embodiment of the present application. The distributed system includes a client 10, a data processing node 20 and a plurality of distributed nodes 30.

[0070] The client 10 may be used to send a data processing request, such as a write request or a read request, to the data processing node 20 .

[0071] The data processing node 20 can be used to respond to data processing requests based on the stored data state set and send relevant instructions to the distributed node 30. The data processing node 20 can be a network device that has cache resources and can perform basic operations such as reading, writing and comparing data in the cache resources. In addition, in the present application, the data processing node 20 can also have a data state control function, a log cache function and a heartbeat communication function. The network device can be a device such as a switch and a router that provides network connection services in the network, and has the functions of receiving network messages, extracting fields in network messages, and updating fields in network messages after comparison and calculation and forwarding network messages.

[0072] The distributed node 30 may be used to store data, or may be used to read data and send the read data to the client 10 .

[0073] Optionally, the client 10 and the distributed node 30 may be a terminal, a server, or other types of electronic devices with corresponding reading and writing functions.

[0074] When the client 10 and the distributed node 30 are terminals, the terminal can be a personal computer such as a desktop, tablet computer and laptop computer, or a remote terminal, a user terminal (terminal equipment, TE) and a mobile device, etc. It can also be a wireless terminal in industrial control, a wireless terminal in self driving, a wireless terminal in remote medical, a wireless terminal in a smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, etc.

[0075] This application does not limit the form of the terminal. The device for implementing the function of the terminal can be a terminal, or a device that can support the terminal to implement the function, such as a chip system. The device can be installed in the terminal or used in conjunction with the terminal. In the embodiment of this application, the chip system can be composed of a chip, or it can include a chip and other discrete devices.

[0076] When the client 10 and the distributed node 30 are servers, the server may be a single server or a server cluster consisting of multiple servers. In some implementations, the server cluster may also be a distributed cluster. This application does not limit the form of the server.

[0077] It should be noted that Figure 3The distributed system shown is only an example, and does not limit the system architecture to which the data processing method provided in the embodiment of the present application is applicable. For example, the distributed system may include multiple clients 10. For another example, the distributed system may include multiple data processing nodes 20. For another example, multiple distributed nodes 30 may be in the same cluster, or in two or more independent clusters.

[0078] To implement the data processing method provided by the present application, specifically, the functional components included in the data processing node 20 and the distributed node 30 are as follows: Figure 4 shown. Figure 4 A schematic diagram of another distributed system provided for this application, such as Figure 4 As shown, the data processing node 20 may include a control plane and a data plane. The distributed node 30 may include an intermediate layer, a consistency module and a storage module. The functional components included in the data processing node 20 and the distributed node 30 are described below:

[0079] The control plane in the data processing node 20 may include a data state set controller, a log cache module, and a heartbeat transmitter. The data plane may include functions such as data state set operation, read request parsing, query distribution, load balancing, write request multicast, counting, and state judgment. The control plane is used to control various functions on the data plane to support the data processing node to implement the response of read requests and write requests in the distributed system, reduce the delay of data processing, and improve the consistency of data.

[0080] The data state set controller can be used to add, delete and modify the data state set to dynamically update the distribution information about the data. The data state set can also be called a transient state table, which can be used to record the distribution information of data in multiple distributed nodes 30. For example, the corresponding relationship between the key (also called an identifier) ​​of each data, the state information and the write node information is recorded. The state information can be used to indicate whether a consensus is reached in multiple distributed nodes 30. The write node information can be used to characterize the distributed nodes 30 that have written data, and the distributed nodes 30 that have not written data.

[0081] For example, the data processing node 20 can update the data state set through the data state set controller, record the state information of the data that has not reached a consensus in the distributed system in the instantaneous state in the data state set, and can also delete the state information of the data from the data state set when the data reaches a consensus, so as to maintain the distribution state of the data, thereby more conveniently maintaining the linear consistency of the data and realizing the data state control function. At the same time, the volume of the data state set will not expand with the increase of the number of write requests and the number of distributed nodes, and has good scalability.

[0082] The data state set operation function can be used to perform operations such as query on the data state set to support the call and judgment of data distribution information.

[0083] The log cache module can be used to cache write requests and read requests from the client 10, as well as related indication information sent to the distributed node 30 as log information, and perform management operations such as reading or deleting these log information to facilitate data recovery and data retransmission.

[0084] The heartbeat transmitter can be used to exchange heartbeat packets with the distributed nodes 30 to support the data processing node 20 in determining whether the working status of the distributed node 30 is abnormal based on whether the heartbeat packet interaction is interrupted, so as to facilitate the data processing node 20 to delete the distributed node 30 with abnormal working status from the distributed system.

[0085] The read request parsing function may be used to parse the read request from the client 10 and obtain the data identifier of the data to be processed in the read request, so as to support the data processing node 20 to respond to the read request.

[0086] The query distribution function can be used to query data distribution information, as well as information such as the working status and load conditions of distributed nodes.

[0087] The load balancing function can be used to evenly distribute read requests on multiple distributed nodes 30 to achieve full utilization and efficient operation of resources in the distributed system.

[0088] The write request multicast function can be used to send indication information corresponding to the write request to each distributed node 30 in a multicast manner, so as to improve the efficiency of data writing and reduce the delay of data processing.

[0089] The counting function may be used to count the number of received write confirmation messages, that is, the number of distributed nodes 30 that have completed the write operation.

[0090] The state judgment function can judge the consistency state of the data according to the number of distributed nodes 30 that have completed the write operation.

[0091] The data processing node 20 can achieve efficient response to read requests and write requests based on the functions of the above control plane and data plane. For example, after receiving a write request from the client 10 for requesting to write the pending data, the data processing node 20 can query the distribution information of the pending data through the data status set operation function, and when it is determined that multiple distributed nodes 30 are instructed to write the pending data, the data writing instruction is sent to multiple distributed nodes 30 through the write request multicast function, and the number of distributed nodes 30 that have completed the write operation is counted through the counting function, and the consistency state of the data is judged according to the number of distributed nodes 30 that have completed the write operation through the state judgment function.

[0092] For example, after the data processing node 20 receives a read request from the client 10 for requesting to read the data to be processed, it can parse the read request through the read request parsing function to obtain information such as the data identification of the data to be processed, and query the working status and load status of the distributed node through the query distribution function, and call the data status set operation function to query the distribution information of the data to be processed, so as to further distribute the read request to the appropriate distributed node 30 through the load balancing function.

[0093] It should be noted that by configuring the above-mentioned log cache module, heartbeat transmitter, and read request parsing, query distribution, load balancing, write request multicast, counting and status judgment functions in the data processing node 20, the function separation and unloading of the master node in the general technology can be realized. That is, the storage function of the traditional master node as a data copy and the consistency function as the master node are separated, the storage function of the traditional master node as a data copy is retained, and the consistency function of the master node is unloaded to the data processing node 20. In this case, when the data processing node 20 is deployed in the form of a physical device such as a switch or a router, it is not necessary to store too much data, reducing the demand for hardware resources to support reducing the delay of the data processing process. Therefore, the present application can effectively support the unloading of consistency logic to the data transmission process in the network, and realize 1 RTT to complete the read operation and write operation, breaking through the limitation of 2-3 RTT in the general technology, while having practicality.

[0094] In one possible manner, when a data processing node 20 fails and cannot support data consistency implementation, data consistency implementation can be supported by redeploying consistency functions on slave nodes.

[0095] The client 10 can be adapted to the protocol packet header corresponding to the data processing node 20, and can conveniently connect to the data plane of the data processing node 20 to initiate read requests and write requests, etc. within the distributed system, thereby realizing seamless client-side access to the distributed system provided in this application.

[0096] The distributed node 30 can be configured with a network intermediary layer (also called a Shim layer), a consistency module, a storage module, etc. The network intermediary layer can support the parsing and conversion of multiple protocols and has good versatility. The consistency module can be implemented based on consistency algorithms such as Raft, and can support data replication, hash calculation, checksum, etc., for achieving data synchronization and consistency between distributed nodes, and efficiently reading the latest data, etc. The storage module can be a storage medium that supports long-term storage, such as a disk and an optical disk, for storing backup data.

[0097] The distributed node 30 can realize data processing through various configured modules. For example, the distributed node 30 can receive a data write instruction from the data processing node 20 through the network intermediary layer, and perform data replication through the consistency module to write the data to the storage module. For another example, the distributed node 30 can receive a data read instruction from the data processing node 20 through the network intermediary layer, and determine the latest data through the consistency module to read data from the storage module.

[0098] In one possible manner, the distributed node 30 can be located at the server end of a key-value storage system such as ETCD, and can support key-value storage systems such as ETCD to conveniently access the consistency logic configured in the data processing node 20 through a configured network intermediary layer, thereby achieving seamless access of the server side to the distributed system provided in this application.

[0099] Optionally, the above functional deployment of data processing nodes, clients, and distributed nodes can be implemented in software and / or hardware form. This application does not limit this. Taking the data processing node as an example, the data processing node 20 may include: Figure 4 The functional components shown may also include Figure 5 The functional components are shown in hardware form without limitation.

[0100] Figure 5 The hardware structure diagram of a data processing node 20 provided in an embodiment of the present application is shown in FIG. The data processing node 20 may include a processor 2001, a memory 2002, and a transceiver 2003. The processor 2001, the memory 2002, and the transceiver 2003 may be interconnected via a bus 2004.

[0101] The processor 2001 may be one or more CPUs. In the case where the processor 2001 is a CPU, the CPU may be a single-core CPU or a multi-core CPU. The processor 2001 may be used to read data stored in the memory 2002, perform data processing operations, and specifically Figure 4 The functions implemented by the data state collection controller and log cache module in the control plane shown.

[0102] The memory 2002 may be a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM) or a portable read-only memory (CD-ROM). The memory 2002 is used to store a data state set and related program codes.

[0103] The transceiver 2003 is used to receive / send data in response to instructions from the processor 2001 and the memory 2002 .

[0104] The embodiment of the present application does not limit the number of these components. For example, the data processing node 20 may include one or more processors 2001. In another example, the data processing node 20 may include one or more memories 2002 and the like.

[0105] The following, combined Figure 3 or Figure 4 The distributed system architecture shown illustrates the data processing method provided in the embodiment of the present application. Figure 6 A flowchart of a data processing method provided in an embodiment of the present application. Figure 6 The method shown can be applied to data processing nodes in a distributed system. Figure 6 The data processing node in the method shown may specifically include Figure 4 The functional components shown or Figure 5 As shown in the parts. Figure 6 As shown, the data processing method may include: S101-S104.

[0106] S101, a client sends a data processing request to a data processing node. The data processing node receives the data processing request from the client.

[0107] The data processing request may be used to request processing of data, which may be referred to as data to be processed. The data processing request may include a data identifier of the data to be processed. The data to be processed may be data of a type such as text, image, audio, or video. The data identifier of the data to be processed may be used to represent the data to be processed. Optionally, the data identifier of the data to be processed may be a key or other identifier of the data to be processed, etc., without limitation.

[0108] Optionally, the processing data described in the present application may include read data or write data, and the data processing request may be a write request or a read request. A write request may be used to request operations such as adding, modifying or deleting data. A read request may be used to request querying data. In the case where the data processing request is a write request, the data processing request may also include data to be processed.

[0109] In one possible manner, the client may send a data processing request to the data processing node according to the user's requirements. For example, the client may receive a processing operation of the user, and in response to the processing operation, the client may send a data processing request to the data processing node. Accordingly, the data processing node may receive the data processing request from the client. Subsequently, the data processing node may respond to the data processing request and determine whether to instruct the distributed node to perform the data processing operation directed to the data to be processed.

[0110] Exemplarily, the client may send a data processing request directly to the data processing node, or may send a data processing request to the data processing node through an intermediate node (or relay node), etc., and this application does not impose any restrictions on this.

[0111] S102. The data processing node queries the data status set in response to the data processing request to obtain the distribution information of the data to be processed. The data status set may include the data identifier and the distribution information corresponding to the data identifier, and the data status set may be used to indicate or record the distribution information of the data in multiple distributed nodes. The distribution information may include status information and write node information. The status information may be used to characterize whether the data is consistent across multiple distributed nodes. The write node information may be used to characterize the writing status of the data on each distributed node, that is, which distributed node has been written and which distributed node has not been written. Optionally, the data status set may be pre-stored in the data processing node. For example, Figure 4 As shown, the data state set can be pre-stored in the data plane of the data processing node.

[0112] In the initial state, the data status set is empty. After receiving a write request for data initiated by the client and the write request is successful, the data processing node can store the data identifier, status information, and write node information of the data in the data status set. At this time, the status information is not consistent, and the write node information is used to indicate that each distributed node has not written the data. After receiving a response from the distributed node confirming the completion of the write operation, the data processing node can update the write status corresponding to the distributed node in the write node information of the data to written. In addition, the data processing node can update the number of distributed nodes that have been written. When the number of distributed nodes that have been written meets the consistency requirement (for example, half of them), the data processing node can update the status information of the data to reach a consensus. When the number of distributed nodes that have been written is equal to the number of multiple distributed nodes (that is, all distributed nodes are written), the data processing node can delete the data identifier of the data in the data status set and the corresponding relationship between the status information and write node information.

[0113] S103: The data processing node determines a processing strategy corresponding to the data to be processed according to the distribution information of the data to be processed.

[0114] The processing strategy may include not instructing the distributed nodes to process the data to be processed, or instructing the distributed nodes to process the data to be processed.

[0115] Exemplarily, after receiving a data processing request from a client, a data processing node can parse the data processing request to obtain the data identifier of the data to be processed, and use the data identifier of the data to be processed as an index to query the data status set to determine whether the data status set contains distribution information corresponding to the data identifier of the data to be processed, so as to further determine whether instructing the distributed node to process the data to be processed will affect the consistency of the data.

[0116] If it is determined that instructing the distributed nodes to process the data to be processed will affect the consistency of the data, the data processing node can determine the processing strategy to not instruct the distributed nodes to process the data to be processed. If it is determined that instructing the distributed nodes to process the data to be processed will not affect the consistency of the data, the data processing node can determine the processing strategy to instruct the distributed nodes to process the data to be processed. Specifically, the process can refer to the description in S201-S206 and S501-S505 below, which will not be repeated here.

[0117] S104: The data processing node executes a processing strategy corresponding to the data to be processed to respond to the data processing request.

[0118] If the processing strategy corresponding to the data to be processed is not to instruct the distributed node to process the data to be processed, the data processing node can directly send a response message to the client indicating that the operation is not performed, and the response process ends.

[0119] If the processing strategy corresponding to the data to be processed is to instruct the distributed node to process the data to be processed, the data processing node can send a corresponding instruction to the distributed node so that the distributed node performs the data processing operation and returns corresponding response information to the client.

[0120] Based on the description of S101-S104 above, it can be known that after receiving the data processing request from the client, it can be determined whether to instruct the distributed node to process the data to be processed based on the data identifier and the data status set to ensure the consistency of the data in multiple distributed nodes. If it is determined not to instruct the distributed node to process the data to be processed, the response can be ended directly. In this case, compared to the general technology that requires master-slave coordination before ending the response, resulting in two or more round-trip delays, the present application can support the delay of the entire response process as a round-trip delay between the client and the data processing node, which can effectively reduce the delay of the response process and improve the throughput of the distributed system.

[0121] If it is determined to instruct the distributed node to process the data to be processed, the data processing node can further instruct the distributed node to process the data to be processed in response to the data processing request. In this case, compared to the general technology that requires master-slave coordination before processing the data to be processed, resulting in two or more round-trip delays, the present application can support the delay of the entire response process as a round-trip delay between the client and the distributed node, which can effectively reduce the delay of the response process and improve the throughput of the distributed system. Therefore, the present application can be used to improve the problem that the general technology easily causes a large delay in the response process in the distributed system, so as to improve the throughput of the distributed system.

[0122] Combination Figure 6 For the response process of the data processing request, if the data processing request is a write request, such as Figure 7 As shown, an embodiment of the present application provides a flow chart of another data processing method. Figure 7 The method shown can be applied to data processing nodes in a distributed system. For example, Figure 3 or Figure 4 The data processing node 20 in the distributed system is shown. The data processing method may include: S201-S206.

[0123] S201: The client sends a write request to a data processing node, and the data processing node receives the write request from the client.

[0124] The write request may be used to request to write data into a distributed node. The data may be referred to as data to be processed or data to be written. The write request may include a data identifier of the data to be processed.

[0125] In one possible manner, the client may receive a data write operation from a user (e.g., click a write button), and in response to the data write operation, the client may send a write request to a data processing node. Accordingly, the data processing node may receive the write request from the client. Subsequently, the data processing node may determine, in response to the write request, whether to instruct the distributed node to perform a write operation directed to the data to be processed.

[0126] S202: The data processing node queries the data status set to determine the distribution information of the data to be processed in response to the write request.

[0127] Optionally, after receiving a write request from a client, the data processing node may parse the write request to obtain the data identifier of the data to be processed, and use the data identifier of the data to be processed as an index to query the data state set to determine whether the distribution information corresponding to the data identifier of the data to be processed exists in the data state set. If not, the distribution information is empty. If so, the distribution information corresponding to the data identifier is used as the distribution information of the data to be processed.

[0128] As mentioned above, the distribution information includes status information and write node information. Status information can be used to characterize whether the data is consistent across multiple distributed nodes. Write node information can be used to characterize the writing status of data on each distributed node, that is, which distributed node has been written and which distributed node has not been written. In other words, the distribution information of the data to be processed may exist in the following situations: 1. The data to be processed is consistent across multiple distributed nodes and has been written to some distributed nodes. The number of this part of distributed nodes is greater than or equal to half of the number of multiple distributed nodes and less than the number of multiple distributed nodes. 2. The data to be processed has not reached a consensus on multiple distributed nodes and has not been written to any distributed node. 3. The data to be processed has not reached a consensus on multiple distributed nodes and has been written to less than half of the distributed nodes.

[0129] Exemplarily, the preset distributed system includes five distributed nodes, namely distributed node 1, distributed node 2, distributed node 3, distributed node 4 and distributed node 5. For data A being written to multiple distributed nodes, the status information in the corresponding distribution information may be a consensus or a disagreement, and the corresponding write node information may be "00100". Among them, "0" means not written, and "1" means written.

[0130] S203: The data processing node determines a processing strategy corresponding to the data to be processed based on the distribution information of the data to be processed.

[0131] In one example, the distribution information of the data to be processed does not exist, which means that the data to be processed has never been written, and this write may be the first write. Alternatively, the data to be processed has been written to all distributed nodes, and this write may be a data update. In this case, writing the data to be processed to multiple distributed nodes does not cause data inconsistency. The data processing node can then determine the processing strategy to instruct the distributed nodes to write the data to be processed.

[0132] In one example, if the distribution information of the data to be processed indicates that the data to be processed is consistent across multiple distributed nodes, it can indicate that the data value corresponding to the data identifier of the data to be processed in the previous write request is being written to multiple distributed nodes, and multiple distributed nodes have reached a consensus, that is, more than half of the distributed nodes have completed the previous write operation for the data to be processed. In this case, writing the data to be processed to multiple distributed nodes does not cause data inconsistency problems. The data processing node can then determine the processing strategy to instruct the distributed nodes to write the data to be processed.

[0133] In one example, the distribution information of the data to be processed indicates that the data to be processed is not consistent across multiple distributed nodes. In this case, if the write operation corresponding to this write request is executed, it is easy to cause data inconsistency in multiple distributed nodes. Then the data processing node can determine the processing strategy to not instruct the distributed node to write the data to be processed, so as to improve the consistency of the data in multiple distributed nodes.

[0134] S204: The data processing node does not instruct the distributed node to write the data to be processed, and sends a first response to the client indicating that the data to be processed will not be written.

[0135] After determining that the processing strategy is not to instruct the distributed node to write the pending data, the data processing node does not send the relevant instructions corresponding to the current write request to the distributed node, and can send a first response to the client to indicate that the pending data is not written. Accordingly, the client can receive the first response from the data processing node and determine that the current write request is not executed. Subsequently, the client can resend the write request to the data processing node.

[0136] It should be understood that after the data processing node executes S203, if S204 is executed, the response process for this write request ends and the following S205 is no longer executed. If S204 is not executed, S205 is executed to respond to this write request.

[0137] Based on this, when the data processing request is a write request, if it is determined that the distributed node is not instructed to write the data to be processed, the response can be terminated directly. In this case, compared with the general technology that requires master-slave coordination before terminating the response, resulting in two or more round-trip delays, this application can support the delay of the entire response process as a round-trip delay between the client and the data processing node, which can effectively reduce the delay of the response process and improve the throughput of the distributed system.

[0138] S205: The data processing node instructs the distributed node to write the data to be processed.

[0139] After determining that the processing strategy is to instruct the distributed nodes to write the data to be processed, the data processing node may send write instruction information carrying the data to be processed to each distributed node to instruct each distributed node to write the data to be processed.

[0140] The process in which the data processing node instructs the distributed node to write the data to be processed may specifically include: S2051-S2052.

[0141] S2051. The data processing node generates a to-be-committed index corresponding to the write request.

[0142] It should be noted that the data processing node can also store the current index. The current index can be larger than other indexes, and the corresponding data has been agreed upon by multiple distributed nodes. That is, the current index is the index of the data that multiple distributed nodes have most recently agreed upon at the current moment. The index can be a globally unique identifier assigned by the data processing node to a write request that needs to be processed by a distributed node. After multiple distributed nodes agree on the data requested to be written by a write request, the index of the write request can also be used to query the data requested to be written by the write request.

[0143] The index to be submitted can be larger than the current index.

[0144] In a possible embodiment, the data processing node can increase the index value of the current index by a preset value in a monotonically increasing manner to obtain the index to be submitted. The index to be submitted can be a globally unique identifier of this write request in the distributed system. Through the monotonically increasing index generation method, the data processing node can sort all write requests, so as to write them to each distributed node in a fixed order, thereby improving data consistency.

[0145] In one possible embodiment, combining Figure 4The data processing node may be pre-configured with a log cache module. The data processing node may store log information corresponding to the write request in a preset cache location through the log cache module. The log information may include the corresponding relationship between the data to be processed, the data identifier of the data to be processed, and the index to be submitted.

[0146] S2052: The data processing node sends a data write instruction to multiple distributed nodes in the distributed system.

[0147] The data writing instruction may carry the index to be submitted and the data to be processed.

[0148] Optionally, the data processing node may send a data write instruction carrying the index to be submitted and the data to be processed to multiple distributed nodes in a multicast, multicast or broadcast manner. For example, the data processing node may send a data write instruction to multiple distributed nodes by multicast. In this multicast manner, the utilization rate of network resources can be improved, and the distributed system can be made more flexible and easier to expand.

[0149] Based on this, in the case where the data processing request is a write request, if it is determined to instruct the distributed node to write the data to be processed, the data processing node can further instruct the distributed node to process the data to be processed based on the current index to respond to the data processing request. In this case, compared to the general technology that requires master-slave coordination before processing the data to be processed, resulting in two or more round-trip delays, the present application can support the delay of the entire response process as a round-trip delay between the client and the distributed node, which can effectively reduce the delay of the response process and improve the throughput of the distributed system.

[0150] S206. When the data to be processed reaches a consensus in the distributed system, the data processing node sends a second response to the client to indicate that the data to be processed has been written.

[0151] After the data processing node generates a pending index that is larger than the current index corresponding to the write request, the pending index, the data identifier of the data to be processed, the state information that multiple distributed nodes have not reached consensus on, and the corresponding relationship between the write node information can be stored in the data status set to facilitate determining the processing progress of this write request, thereby improving data consistency. The write node information can be used to characterize the write status of the data to be processed in each distributed node. The write status can include not written or written.

[0152] In one possible manner, after receiving the data write instruction from the data processing node, each distributed node can parse the data write instruction to obtain the to-be-committed index and the to-be-processed data, and execute the operation of writing the to-be-processed data in response to the data write instruction, and send write confirmation information to the data processing node after completing the writing. The write confirmation information can also be understood as an acknowledgement character (ACK) corresponding to the data write instruction.

[0153] When the data processing node receives the write confirmation information corresponding to the index to be submitted sent by any distributed node, it can update the write status of any distributed node in the write node information corresponding to the index to be submitted to be written, and count the number of received write confirmation information, that is, the number of distributed nodes that have completed the write operation of the data to be processed, to further determine whether the data to be processed has reached a consensus among multiple distributed nodes. When the data to be processed has reached a consensus among multiple distributed nodes, the data processing node can send a second response to the client to indicate that the data to be processed has been written.

[0154] Optionally, the data processing node may use methods including but not limited to the following two methods to determine whether consensus is reached on multiple distributed nodes for the data to be processed.

[0155] The first method: When the number of distributed nodes that have completed the write operation of the data to be processed is greater than or equal to half of the number of multiple distributed nodes, the data processing node can determine that the data to be processed has reached a consensus among multiple distributed nodes, and update the status information on which the multiple distributed nodes corresponding to the index to be submitted have not reached a consensus to the status information on which the multiple distributed nodes have reached a consensus.

[0156] The second method: The data processing node can determine the current ratio between the number of distributed nodes that have completed the write operation of the data to be processed and the number of multiple distributed nodes, that is, the current ratio between the number of distributed nodes whose write status is written and the number of multiple distributed nodes, and compare the current ratio with the preset ratio threshold. When the current ratio is greater than or equal to the preset ratio threshold, the data processing node can determine that the data to be processed has been agreed upon by multiple distributed nodes, and update the state information on which multiple distributed nodes corresponding to the index to be submitted have not reached a consensus to the state information on which multiple distributed nodes have reached a consensus. Among them, the preset ratio threshold can be reasonably set by the staff. For example, the preset ratio threshold can be 0.5.

[0157] Exemplarily, the preset distributed system includes five distributed nodes, namely distributed node 1, distributed node 2, distributed node 3, distributed node 4 and distributed node 5, and the condition for data in the distributed system to reach a consensus among the five distributed nodes is to write to more than half of the nodes. If the data has been written to three or more distributed nodes, it can be indicated that the data has reached a consensus among the five distributed nodes. If the data has been written to one or two distributed nodes, it can be indicated that the data has not reached a consensus among the five distributed nodes.

[0158] In a possible embodiment, when the current ratio is greater than or equal to a preset ratio threshold, that is, when it is determined that the data to be processed has reached a consensus among multiple distributed nodes, it can be indicated that the index to be submitted is the index corresponding to the latest data that has reached a consensus among multiple distributed nodes. In this case, the data processing node can update the index to be submitted to the current index, so as to better support write operations and read operations on data in multiple distributed nodes.

[0159] In a possible embodiment, when the number of distributed nodes in the write state that have been written is equal to the number of multiple distributed nodes, it can indicate that multiple distributed nodes have completed the write operation corresponding to this write request. In this case, the data processing node can delete the information corresponding to the index to be submitted in the data state set, so that the table items of the data that have been agreed in the data state set can be deleted in time, thereby avoiding the problem that the size of the data state set expands with the increase of the amount of data and / or the number of distributed nodes, and reducing the storage resource occupation of the data state set in the data processing node.

[0160] In one possible embodiment, combining Figure 7 The data processing method in the write request response process shown, in order to avoid data inconsistency problems caused by data packet loss, after the data processing node sends a data write indication to multiple distributed nodes in the distributed system, the data processing method provided in the embodiment of the present application also includes: S301.

[0161] S301. When the waiting time corresponding to the distributed node is greater than or equal to the first time threshold and the heartbeat communication between the data processing node and the distributed node is normal, the data processing node reads the log information corresponding to the write request from the preset cache location, and re-sends the data write indication to the distributed node based on the log information.

[0162] The waiting time may be used to represent the time for waiting for the distributed node to send the write confirmation information corresponding to the index to be submitted.

[0163] In one possible embodiment, combining Figure 4,The data processing node can perform heartbeat communication with multiple distributed nodes through the configured heartbeat ,sender.

[0164] After the data processing node sends a data write indication carrying the index to be submitted and the data to be processed to multiple distributed nodes, it can also determine the waiting time corresponding to each distributed node in real time or periodically, and compare the waiting time corresponding to each distributed node with the first time threshold to confirm whether a packet loss problem occurs. Among them, the waiting time can be used to characterize the time to wait for the distributed node to send the write confirmation information corresponding to the index to be submitted. The first time threshold can be reasonably set by the staff. For example, the first time threshold can be 1 second.

[0165] If the data processing node determines that the waiting time corresponding to any distributed node is greater than or equal to the first time threshold, and the heartbeat communication with the any distributed node is normal, it may indicate that a packet loss problem occurs between the data processing node and the any distributed node. The packet loss problem may be that the data write indication sent by the data processing node to the any distributed node has packet loss, that is, the any distributed node has not received the data write indication from the data processing node, resulting in no write confirmation information being sent to the data processing node. The packet loss problem may also be that the any distributed node has received the data write indication from the data processing node, but the write confirmation information sent by the any distributed node to the data processing node has packet loss.

[0166] In this case, if the data processing node can read the log information corresponding to the write request from the preset cache location and resend the data write instruction to any distributed node based on the log information until any distributed node completes the write operation corresponding to the write request, the stability of the distributed system and the consistency of the data can be improved.

[0167] In one possible embodiment, combining Figure 7 The data processing method in the write request response process shown is to avoid the problem of data state set expansion caused by distributed node failure and improve the stability of the distributed system. The data processing method provided in the embodiment of the present application also includes: S401.

[0168] S401. When the interruption duration of the heartbeat communication between the data processing node and the distributed nodes in the distributed system is greater than or equal to the second duration threshold, the data processing node deletes the configuration information corresponding to the distributed node and updates the number of distributed nodes in the distributed system.

[0169] During the heartbeat communication between the data processing node and each distributed node, if the time the data processing node waits for the heartbeat packet from any distributed node is longer than the preset time, it can indicate that any distributed node may be in an abnormal state and cannot perform write operations normally. Then, for the data being written, the data processing node cannot determine that the number of distributed nodes writing these data is equal to the number of multiple distributed nodes, and it is also impossible to delete the table items related to these data in the data state set, which easily causes the data state set to accumulate a large number of items in a short period of time and expand.

[0170] In this case, the data processing node can determine that the heartbeat communication with any distributed node is interrupted, and count the interruption duration of the heartbeat communication with any distributed node to further compare the interruption duration with the second duration threshold. If the interruption duration of the heartbeat communication with any distributed node is greater than or equal to the second duration threshold, the data processing node can delete the configuration information corresponding to any distributed node, and update the number of multiple distributed nodes to delete any distributed node from the distributed system, update the judgment logic of the data state set, thereby avoiding the expansion of the data state set and improving the stability of the distributed system.

[0171] Combination Figure 6 For the response process of the data processing request, if the data processing request is a read request, such as Figure 8 As shown, an embodiment of the present application provides a flow chart of another data processing method. Figure 8 The method shown can be applied to data processing nodes in a distributed system. For example, Figure 3 or Figure 4 The distributed system and data processing node 20 shown. The data processing method may include: S501-S505.

[0172] S501: The client sends a read request to a data processing node, and the data processing node receives the read request from the client.

[0173] The read request may include a data identifier of the data to be processed.

[0174] In one possible manner, the client may receive a data query operation from a user (e.g., click a query button), and in response to the data query operation, the client may send a read request to a data processing node. Accordingly, the data processing node may receive the read request from the client. Subsequently, the data processing node may determine, in response to the read request, whether to instruct the distributed node to perform a read operation directed to the data to be processed.

[0175] S502: The data processing node queries the data status set to determine the distribution information of the data to be processed in response to the read request.

[0176] It should be understood that for the specific implementation of S502, reference may be made to the description of S202 above, which will not be repeated here.

[0177] S503: The data processing node determines a processing strategy corresponding to the data to be processed based on the distribution information of the data to be processed.

[0178] In one example, if the data state set records the distribution information corresponding to the data identifier, and the distribution information includes the state information that multiple distributed nodes do not agree on, it can indicate that the new data value corresponding to the data identifier of the data to be processed is being written to each distributed node, and the write operation has not been completed. In order to meet the consistency of the data, the new data value cannot be used as the return value corresponding to the data identifier of the data to be processed. When multiple distributed nodes have written data values ​​corresponding to the data identifier of the data to be processed at a historical time, any of the multiple distributed nodes can determine the old data as the return value corresponding to the data identifier of the data to be processed. The old data is the data value corresponding to the data identifier of the data to be processed that is agreed upon in multiple distributed nodes at a historical time. Alternatively, when multiple distributed nodes have not written data values ​​corresponding to the data identifier of the data to be processed at a historical time, any of the multiple distributed nodes can determine empty data as the return value corresponding to the data identifier of the data to be processed.

[0179] If the data processing node determines that the data status set does not record the distribution information corresponding to the data identifier, it can indicate that the data to be processed may have reached a consensus among multiple distributed nodes. At this time, any distributed node can read the data to be processed without affecting the consistency of the data. Alternatively, the data to be processed may not have been written to multiple distributed nodes. At this time, any distributed node can return empty data without affecting the consistency of the data.

[0180] In the above two cases, the data processing node can determine the processing strategy to instruct the first distributed node to read the data to be processed, that is, the second strategy of selecting a distributed node from multiple distributed nodes to read the data to be processed. In other words, the first distributed node is included in the multiple distributed nodes of the distributed system.

[0181] In one example, if the data status set records the distribution information corresponding to the data identifier, and the distribution information includes status information that multiple distributed nodes have reached a consensus on, it can indicate that the new data value corresponding to the data identifier of the data to be processed is being written to each distributed node, and this write operation has reached a consensus. In this case, using the new data value as the return value corresponding to the data identifier of the data to be processed does not affect the consistency of the data. In order to reduce the data reading latency, the distributed node that has completed the write operation corresponding to the new data value can be instructed to respond to the read request. The data processing node can then determine the processing strategy to instruct the second distributed node to read the data to be processed, that is, to select a second distributed node from the distributed nodes whose write status is written in the write node information included in the distribution information to respond to the read request. In other words, the second distributed node is included in the distributed nodes that have written the data to be processed.

[0182] S504: The data processing node instructs the first distributed node to read the data to be processed.

[0183] In one possible manner, the data processing node may determine a first distributed node that meets the load balancing condition among multiple distributed nodes, and send a data read instruction carrying the current index to the first distributed node, so that the first distributed node responds to the read request. Figure 4 , the data processing node can select a first distributed node that meets the load balancing condition from multiple distributed nodes through the configured load balancing function, and send a data read instruction carrying the current index to the first distributed node. Correspondingly, the first distributed node can receive the data read instruction from the data processing node, parse the data read instruction to obtain the current index and the data identifier of the data to be processed, and read the data value corresponding to the data identifier of the data to be processed from the data with an index less than or equal to the current index to respond to the read request.

[0184] If the data value corresponding to the data identifier of the data to be processed is read, the first distributed node can send the read data value to the client as a return value. If the data value corresponding to the data identifier of the data to be processed is not read, the first distributed node can send empty data to the client as a return value. In this way, even if the new data value corresponding to the data identifier of the data to be processed is being written to each distributed node, since the current index is less than the index corresponding to the new data value, the first distributed node will not read the new data value, thereby improving data consistency.

[0185] In one possible manner, the first distributed node may send a return value to the client through any data processing node that can provide a network connection service, for example, the data processing node that sends a data read instruction to the first distributed node.

[0186] Optionally, the load balancing condition may be set based on a load balancing algorithm such as a round-robin scheduling algorithm or a random method, which will not be described in detail here.

[0187] It should be understood that after the data processing node executes S503, if S504 is executed, the response process for this read request ends and the following S505 is no longer executed. If S504 is not executed, S505 is executed to respond to this read request.

[0188] Based on this, in the case where the data processing request is a read request, if the new data value corresponding to the data identifier of the data to be processed is being written to each distributed node and no consensus is reached, or there is no new data value being written to the data identifier of the data to be processed, then the data processing node can further instruct any distributed node to respond to the data processing request based on the current index. In this case, compared to the general technology that requires master-slave coordination before processing the data to be processed, resulting in two or more round-trip delays, the present application can support the delay of the entire response process as a round-trip delay between the client and the distributed node, which can effectively reduce the delay of the response process and improve the throughput of the distributed system.

[0189] S505: The data processing node instructs the second distributed node to read the data to be processed.

[0190] In one possible manner, the data processing node may determine a second distributed node that has written the data to be processed and meets the load balancing condition as indicated by the write node information of the data to be processed, and send a data read instruction carrying the current index to the second distributed node, so that the second distributed node responds to the read request. Figure 4 , the data processing node can determine the distributed node whose write status is written in the write node information included in the distribution information and meets the load balancing condition as the second distributed node through the configured load balancing function, and send a data read instruction carrying the current index to the second distributed node. Correspondingly, the second distributed node can receive the data read instruction from the data processing node, parse the data read instruction to obtain the current index and the data identifier of the data to be processed, and read the new data value from the data whose index is less than or equal to the current index, so as to send the new data value to the client as a return value.

[0191] In one possible manner, the second distributed node may send a return value to the client through any data processing node that can provide a network connection service, for example, the data processing node that sends a data read instruction to the second distributed node.

[0192] Based on this, in the case where the data processing request is a read request, if the new data value corresponding to the data identifier of the data to be processed is being written to each distributed node and a consensus has been reached, the data processing node can further respond to the data processing request based on the current index indication of the distributed node that has written the new data value to improve the consistency of the data. In this case, compared to the general technology that requires master-slave coordination before processing the data to be processed, resulting in two or more round-trip delays, the present application can support the delay of the entire response process as a round-trip delay between the client and the distributed node, which can effectively reduce the delay of the response process and improve the throughput of the distributed system.

[0193] For example, Fig. 9 As shown, it is a schematic diagram of a read data processing flow provided by an embodiment of the present application. The preset distributed system includes distributed node 1, distributed node 2 and distributed node 3. The data with data identifier A is being written to the three distributed nodes, and the indicator character of the status information is "C" indicating that a consensus has been reached, and the write node information is "011", that is, the write status of distributed node 1 is not written, the write status of distributed node 2 is written, and the write status of distributed node 3 is written. In this case, after the data processing node receives a read request from the client carrying the data identifier A, it can look up the table to obtain the status information "C" and write node information "011" corresponding to the data identifier A in the data status set, and use "C" and "011" as parameters for load balancing, so as to select a second distributed node that meets the load balancing conditions between distributed node 2 and distributed node 3, thereby responding to the read request.

[0194] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. In order to realize the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0195] The embodiment of the present application can divide the data processing node into functional modules according to the above method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0196] Combined with the above Figures 6 to 9 , describes in detail the data processing method provided according to the embodiment of the present application, and will be combined with Fig.10 , describing a data processing device provided according to an embodiment of the present application.

[0197] Fig.10 The data processing device 40 is a schematic diagram of a structure of a data processing device 40 provided in an embodiment of the present application. The data processing device 40 can be used to execute Figure 6-Figure 9 The data processing method shown in any of the figures in the figure. The data processing device 40 can be applied to a data processing node in a distributed system. The data processing device 40 can include: a receiving unit 4001, a determining unit 4002 and an executing unit 4003. As an example, the data processing device 40 can be a switch. For example, in combination Figure 6 , the receiving unit 4001 can be used to execute S101; the determining unit 4002 can be used to execute S102, and the executing unit 4003 can be used to execute S103 and S104.

[0198] A receiving unit 4001 is used to receive a data processing request from a client; the data processing request is used to request processing of data to be processed; the data processing request includes a write request or a read request; a determining unit 4002 is used to determine the distribution information of the data to be processed according to a data state set in response to the data processing request received by the receiving unit 4001; the data state set is used to record the distribution information of the data in a distributed system; an executing unit 4003 is used to execute a corresponding processing strategy based on the distribution information of the data to be processed determined by the determining unit 4002, in response to the data processing request; wherein the processing strategy includes not instructing the distributed nodes to process the data to be processed, or instructing the distributed nodes to process the data to be processed.

[0199] In one possible implementation, a data status set is used to record the correspondence between the data identification and distribution information of the data; the distribution information includes status information and write node information; the status information is used to indicate whether the data is consistent in the distributed system, and the write node information is used to indicate the distributed nodes in the distributed system that have not written data and / or the distributed nodes that have written data.

[0200] In one possible implementation, the determination unit 4002 is specifically used to: query the data state set using the data identifier of the data to be processed as an index; if there is no distribution information corresponding to the data identifier of the data to be processed, the distribution information is empty; if there is distribution information corresponding to the data identifier of the data to be processed, the distribution information corresponding to the data identifier of the data to be processed is used as the distribution information of the data to be processed.

[0201] In one possible implementation, the execution unit 4003 is specifically used to: when the distribution information of the data to be processed indicates that the data to be processed has not reached a consensus in the distributed system, not instruct the distributed nodes to write the data to be processed, and send a first response to the client to indicate that the data to be processed will not be written; when the distribution information of the data to be processed indicates that the data to be processed has reached a consensus in the distributed system, or the distribution information is empty, instruct the distributed nodes to write the data to be processed.

[0202] In one possible implementation, the data processing node stores a current index, which is greater than other indexes and the corresponding data is consistent in the distributed system; the execution unit 4003 is specifically used to: generate a to-be-committed index corresponding to the write request, which is greater than the current index; send a data write indication to multiple distributed nodes in the distributed system, the data write indication carries the to-be-committed index and the to-be-processed data, and the execution unit 4003 is also used to send a second response to the client to represent the written to-be-processed data when the to-be-processed data is consistent in the distributed system.

[0203] In a possible implementation, the data processing device also includes: a storage unit 4004; the storage unit 4004 is used to store the correspondence between the to-be-committed index, data identification, state information on which multiple distributed nodes have not reached consensus, and write node information in the data state set.

[0204] In one possible implementation, the data processing device also includes: an update unit 4005; the update unit 4005 is used to update the write status of the distributed node in the write node information corresponding to the index to be submitted to written when receiving write confirmation information corresponding to the index to be submitted sent by the distributed node; the update unit 4005 is also used to update the status information on which multiple distributed nodes corresponding to the index to be submitted have not reached a consensus to status information on which multiple distributed nodes have reached a consensus when the current ratio is greater than or equal to a preset ratio threshold; the current ratio is used to characterize the ratio between the number of distributed nodes whose write status is written and the number of multiple distributed nodes.

[0205] In one possible implementation, the data processing device also includes: a deletion unit 4006; the deletion unit 4006 is used to delete the information corresponding to the index to be submitted in the data status set when the number of distributed nodes in the write status is equal to the number of multiple distributed nodes.

[0206] In a possible implementation, the updating unit 4005 is further configured to update the to-be-submitted index to the current index.

[0207] In a possible implementation, the execution unit 4003 is specifically configured to: multicast a data write indication to multiple distributed nodes.

[0208] In a possible implementation, the storage unit 4004 is further used to store log information corresponding to the write request in a preset cache location; the log information includes the correspondence between the data to be processed, the data identifier and the index to be submitted.

[0209] In one possible implementation, the data processing node performs heartbeat communications with multiple distributed nodes in the distributed system respectively; the execution unit 4003 is also used to read the log information corresponding to the write request from the preset cache location and re-send the data write indication to the distributed node based on the log information when the waiting time corresponding to the distributed node is greater than or equal to the first time threshold and the heartbeat communication between the distributed nodes is normal; the waiting time is used to represent the time for waiting for the distributed node to send the write confirmation information corresponding to the index to be submitted.

[0210] In one possible implementation, the deletion unit 4006 is also used to delete the configuration information corresponding to the distributed node and update the number of distributed nodes in the distributed system when the interruption duration of the heartbeat communication between the distributed nodes in the distributed system is greater than or equal to the second duration threshold.

[0211] In one possible implementation, the execution unit 4003 is specifically used to: when the distribution information of the data to be processed indicates that the data to be processed has not reached a consensus in the distributed system, or the distribution information is empty, instruct a first distributed node to read the data to be processed, and the first distributed node is included in the multiple distributed nodes of the distributed system; when the distribution information of the data to be processed indicates that the data to be processed has reached a consensus in the distributed system, instruct a second distributed node to read the data to be processed, and the second distributed node is included in the distributed nodes that have written the data to be processed.

[0212] In one possible implementation, the execution unit 4003 is specifically used to: determine a first distributed node that meets the load balancing condition among multiple distributed nodes, and send a data read indication carrying the current index to the first distributed node, so that the first distributed node responds to the read request; the execution unit 4003 is specifically used to: determine a second distributed node that has written the data to be processed as indicated by the write node information of the data to be processed and that meets the load balancing condition, and send a data read indication carrying the current index to the second distributed node, so that the second distributed node responds to the read request.

[0213] The explanation of the relevant contents and the description of the beneficial effects in this embodiment can all be referred to the above method embodiment, which will not be repeated here. Figure 4 In this embodiment, the data processing device 40 may be Figure 4 The data processing node 20 in the above-mentioned receiving unit 4001, determining unit 4002 and executing unit 4003 can be used to implement Figure 4 The data processing node 20 shown in FIG. 4 has various functions. For example, the receiving unit 4001 can be used to implement functions such as read request parsing. The determining unit 4002 can be used to implement a data state set controller, a query distribution function, and a counting function. The executing unit 4003 can be used to implement functions such as write request multicast and load balancing.

[0214] As an example, combining Figure 5 In this embodiment, the data processing device 40 may be Figure 5 In the data processing node 20, part or all of the above-mentioned modules such as the receiving unit 4001, the determining unit 4002 and the executing unit 4003 can also be implemented by the processor 2001.

[0215] It should be understood that the data processing device 40 of the embodiment of the present application can be implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), and the PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof. Figures 6 to 9 When the data processing method is shown, the data processing device 40 and its various modules may also be software modules.

[0216] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing computer execution instructions on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. Computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, data center, etc. that contains one or more servers that can be integrated with a medium. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (eg, a solid state disk (SSD)).

[0217] The above is only a specific implementation of the present application. Those skilled in the art may conceive of changes or substitutions based on the specific implementation provided by the present application, which should all be included in the protection scope of the present application.

Claims

1. A data processing method, characterized in that: Applied to a data processing node in a distributed system, the method comprises: Receiving a data processing request from a client; the data processing request is used to request processing of data to be processed; the data processing request includes a write request or a read request; In response to the data processing request, determining distribution information of the data to be processed according to a data state set; the data state set is used to record distribution information of the data in the distributed system; A corresponding processing strategy is executed based on the distribution information of the data to be processed to respond to the data processing request; wherein the processing strategy includes not instructing the distributed nodes to process the data to be processed, or instructing the distributed nodes to process the data to be processed.

2. The data processing method according to claim 1, characterized in that: The data state set is used to record the corresponding relationship between the data identification and distribution information of the data; The distribution information includes status information and write node information; the status information is used to indicate whether the data is consistent in the distributed system, and the write node information is used to indicate the distributed nodes in the distributed system that have not written the data and / or the distributed nodes that have written the data.

3. The data processing method according to claim 1 or 2, characterized in that: The determining, according to the data state set, distribution information of the data to be processed includes: Using the data identifier of the data to be processed as an index, querying the data state set; If there is no distribution information corresponding to the data identifier of the data to be processed, the distribution information is empty; If there is distribution information corresponding to the data identifier of the data to be processed, the distribution information corresponding to the data identifier of the data to be processed is used as the distribution information of the data to be processed.

4. The data processing method according to claim 3, characterized in that: The data processing request is the write request, and the executing a corresponding processing strategy based on the distribution information of the data to be processed to respond to the data processing request includes: When the distribution information of the data to be processed indicates that the data to be processed has not reached a consensus in the distributed system, not instructing the distributed node to write the data to be processed, and sending a first response to the client indicating that the data to be processed is not written; When the distribution information of the data to be processed indicates that the data to be processed reaches a consensus in the distributed system, or when the distribution information is empty, the distributed nodes are instructed to write the data to be processed.

5. The data processing method according to claim 4, characterized in that: The data processing node stores a current index, the current index is greater than other indexes, and corresponding data are consistent in the distributed system; The instructing the distributed node to write the data to be processed includes: Generate a to-be-committed index corresponding to the write request, where the to-be-committed index is greater than the current index; Sending a data write instruction to multiple distributed nodes in the distributed system, wherein the data write instruction carries the to-be-committed index and the to-be-processed data, The method further includes: when the data to be processed reaches a consensus in the distributed system, sending a second response to the client indicating that the data to be processed has been written.

6. The data processing method according to claim 5, characterized in that: The method further comprises: The correspondence between the to-be-committed index, the data identifier, the state information on which the plurality of distributed nodes have not reached a consensus, and the write node information is stored in the data state set.

7. The data processing method according to claim 5 or 6, characterized in that: The method further comprises: When receiving write confirmation information corresponding to the to-be-committed index sent by the distributed node, updating the write status of the distributed node in the write node information corresponding to the to-be-committed index to written; When the current ratio is greater than or equal to a preset ratio threshold, the status information on which the multiple distributed nodes corresponding to the index to be submitted have not reached a consensus is updated to the status information on which the multiple distributed nodes have reached a consensus; the current ratio is used to characterize the ratio between the number of distributed nodes whose write status is already written and the number of the multiple distributed nodes.

8. The data processing method according to claim 7, characterized in that: The method further comprises: When the write status is that the number of distributed nodes that have been written is equal to the number of the multiple distributed nodes, the information corresponding to the to-be-committed index in the data status set is deleted.

9. The data processing method according to claim 7, characterized in that: In the case where the current ratio is greater than or equal to the preset ratio threshold, the method further includes: Update the to-be-submitted index to the current index.

10. The data processing method according to any one of claims 5 to 9, characterized in that: The sending of data write instructions to multiple distributed nodes in the distributed system includes: The data write indication is sent by multicast to the multiple distributed nodes.

11. The data processing method according to any one of claims 5 to 10, characterized in that: Also includes: Log information corresponding to the write request is stored in a preset cache location; the log information includes a correspondence between the data to be processed, the data identifier and the index to be submitted.

12. The data processing method according to any one of claims 5 to 11, characterized in that: The data processing node performs heartbeat communication with a plurality of distributed nodes in the distributed system respectively; the method further comprises: When the waiting time corresponding to the distributed node is greater than or equal to the first time threshold and the heartbeat communication with the distributed node is normal, the log information corresponding to the write request is read from the preset cache location, and the data write indication is re-sent to the distributed node based on the log information; the waiting time is used to represent the time for waiting for the distributed node to send the write confirmation information corresponding to the index to be submitted.

13. The data processing method according to any one of claims 1 to 12, characterized in that: Also includes: When the interruption duration of the heartbeat communication with the distributed nodes in the distributed system is greater than or equal to the second duration threshold, the configuration information corresponding to the distributed node is deleted, and the number of distributed nodes in the distributed system is updated.

14. The data processing method according to claim 3, characterized in that: The data processing request is a read request, and the executing a corresponding processing strategy based on the distribution information of the data to be processed to respond to the data processing request includes: Instructing a first distributed node to read the data to be processed when the distribution information of the data to be processed indicates that the data to be processed has not reached a consensus in the distributed system, or the distribution information is empty, wherein the first distributed node is included in the plurality of distributed nodes in the distributed system; When the distribution information of the data to be processed indicates that the data to be processed reaches a consensus in the distributed system, a second distributed node is instructed to read the data to be processed, and the second distributed node is included in the distributed nodes that have written the data to be processed.

15. The data processing method according to claim 14, characterized in that: The instructing the first distributed node to read the data to be processed includes: Determine the first distributed node that meets the load balancing condition among the multiple distributed nodes, and send a data read indication carrying a current index to the first distributed node, so that the first distributed node responds to the read request; The instructing the second distributed node to read the data to be processed includes: determining a second distributed node that has written the data to be processed and meets the load balancing condition, as indicated by the write node information of the data to be processed, and sending a data read indication carrying the current index to the second distributed node so that the second distributed node responds to the read request.

16. A data processing device, characterized in that: Applied to a data processing node in a distributed system, the device comprises: a receiving unit, a determining unit and an executing unit; The receiving unit is used to receive a data processing request from a client; the data processing request is used to request processing of data to be processed; the data processing request includes a write request or a read request; The determining unit is used to determine the distribution information of the data to be processed according to the data state set in response to the data processing request received by the receiving unit; the data state set is used to record the distribution information of the data in the distributed system; The execution unit is used to execute a corresponding processing strategy based on the distribution information of the data to be processed determined by the determination unit to respond to the data processing request; wherein the processing strategy includes not instructing the distributed nodes to process the data to be processed, or instructing the distributed nodes to process the data to be processed.

17. The data processing device according to claim 16, characterized in that: The data state set is used to record the corresponding relationship between the data identification and distribution information of the data; The distribution information includes status information and write node information; the status information is used to indicate whether the data is consistent in the distributed system, and the write node information is used to indicate the distributed nodes in the distributed system that have not written the data and / or the distributed nodes that have written the data.

18. The data processing device according to claim 16 or 17, characterized in that: The determining unit is specifically configured to: Using the data identifier of the data to be processed as an index, querying the data state set; If there is no distribution information corresponding to the data identifier of the data to be processed, the distribution information is empty; If there is distribution information corresponding to the data identifier of the data to be processed, the distribution information corresponding to the data identifier of the data to be processed is used as the distribution information of the data to be processed.

19. The data processing device according to claim 18, characterized in that: The execution unit is specifically used for: When the distribution information of the data to be processed indicates that the data to be processed has not reached a consensus in the distributed system, not instructing the distributed node to write the data to be processed, and sending a first response to the client indicating that the data to be processed is not written; When the distribution information of the data to be processed indicates that the data to be processed reaches a consensus in the distributed system, or when the distribution information is empty, the distributed nodes are instructed to write the data to be processed.

20. The data processing device according to claim 19, characterized in that The data processing node stores a current index, the current index is greater than other indexes, and corresponding data are consistent in the distributed system; the execution unit is specifically used to: Generate a to-be-committed index corresponding to the write request, where the to-be-committed index is greater than the current index; Sending a data write instruction to multiple distributed nodes in the distributed system, wherein the data write instruction carries the to-be-committed index and the to-be-processed data, The execution unit is further configured to send a second response to the client indicating that the data to be processed has been written when a consensus is reached on the data to be processed in the distributed system.

21. The data processing device according to claim 20, characterized in that: The device further comprises: a storage unit; The storage unit is used to store the correspondence between the to-be-committed index, the data identifier, the state information on which the multiple distributed nodes have not reached a consensus, and the write node information in the data state set.

22. The data processing device according to claim 19 or 20, characterized in that: The device further comprises: an updating unit; The updating unit is configured to update the write status of the distributed node in the write node information corresponding to the index to be submitted to written when receiving the write confirmation information corresponding to the index to be submitted sent by the distributed node; The update unit is also used to update the state information on which the multiple distributed nodes corresponding to the index to be submitted have not reached a consensus to the state information on which the multiple distributed nodes have reached a consensus when the current ratio is greater than or equal to a preset ratio threshold; the current ratio is used to represent the ratio between the number of distributed nodes whose write status is written and the number of the multiple distributed nodes.

23. The data processing device according to claim 22, characterized in that: The device further comprises: a deleting unit; The deleting unit is used to delete the information corresponding to the to-be-committed index in the data status set when the number of distributed nodes in the write status is equal to the number of the multiple distributed nodes.

24. The data processing device according to claim 22, characterized in that: The updating unit is further used to update the to-be-submitted index to the current index.

25. The data processing device according to any one of claims 20 to 24, characterized in that: The data processing node performs heartbeat communication with a plurality of distributed nodes in the distributed system respectively; The execution unit is further configured to read the log information corresponding to the write request from a preset cache location when the waiting time corresponding to the distributed node is greater than or equal to the first time threshold and the heartbeat communication with the distributed node is normal, and resend the data write indication to the distributed node based on the log information; The waiting time is used to represent the time for waiting for the distributed node to send the write confirmation information corresponding to the index to be submitted.

26. The data processing device according to any one of claims 16 to 25, characterized in that: The deleting unit is also used to delete the configuration information corresponding to the distributed node and update the number of distributed nodes in the distributed system when the interruption duration of the heartbeat communication between the distributed node and the distributed system is greater than or equal to the second duration threshold.

27. The data processing device according to claim 18, characterized in that: The execution unit is specifically used for: Instructing a first distributed node to read the data to be processed when the distribution information of the data to be processed indicates that the data to be processed has not reached a consensus in the distributed system, or the distribution information is empty, wherein the first distributed node is included in the plurality of distributed nodes in the distributed system; When the distribution information of the data to be processed indicates that the data to be processed reaches a consensus in the distributed system, a second distributed node is instructed to read the data to be processed, and the second distributed node is included in the distributed nodes that have written the data to be processed.

28. The data processing device according to claim 27, characterized in that: The execution unit is specifically used for: Determine the first distributed node that meets the load balancing condition among the multiple distributed nodes, and send a data read indication carrying a current index to the first distributed node, so that the first distributed node responds to the read request; The execution unit is specifically used to: determine a second distributed node that has written the data to be processed and meets the load balancing conditions, as indicated by the write node information of the data to be processed, and send a data read indication carrying the current index to the second distributed node so that the second distributed node responds to the read request.

29. A data processing node, characterized in that: The method comprises a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to call the computer program to implement the data processing method according to any one of claims 1 to 15.

30. A computer-readable storage medium, characterized in that: The method comprises computer program instructions, which, when executed by a data processing node, implement the data processing method according to any one of claims 1 to 15.

Images