Data processing method and device based on raft protocol, and computer storage medium

Abstract

The embodiment of the application discloses a data processing method based on a RAFT algorithm, which is applied to a master node, the master node is configured with a first RDMA module, and the method comprises the following steps: receiving a current task request; wherein the current task request carries current task data; and in response to the current task request, sending a write instruction of the current task data to an SCM module of a slave node by using the first RDMA module according to a RAFT protocol.

Description

Technical Field

[0001] This invention relates to the field of data processing, and more particularly to a data processing method and apparatus based on the RAFT protocol, as well as a computer storage medium. Background Technology

[0002] Currently, the RAFT protocol has been widely used in many fields, mainly to maintain data consistency among multiple nodes, including its application in distributed networks for distributed storage.

[0003] However, for data transmission in distributed networks, the RAFT protocol requires waiting for a successful response message from the Follower when submitting data, and only when more than half of the responses are successful is the data considered successfully submitted. If the Follower cannot return a successful response message in time, it will lead to a decrease in the throughput of the entire data transmission system, resulting in high data transmission latency. Summary of the Invention

[0004] To address the aforementioned technical problems, embodiments of the present invention aim to provide a data processing method and apparatus based on the RAFT protocol, as well as a computer storage medium.

[0005] The technical solution of this invention is implemented as follows: This invention provides a data processing method based on the RAFT algorithm, applied to a master node. The master node is configured with a first RDMA module and receives a current task request, wherein the current task request carries current task data. In response to the current task request, the master node sends a write instruction for the current task data to the SCM module of the slave node using the first RDMA module according to the RAFT protocol.

[0006] This invention provides a data processing method based on the RAFT algorithm, applied to a slave node. The slave node is configured with a second RDMA module and an SCM module. The slave node uses the second RDMA module to receive a write instruction for task data sent by the master node; in response to the write instruction, the slave node writes the task data into the SCM module; when it is determined that the SCM module contains a current data successful submission sequence number corresponding to the task data, the task data is determined to be usable data.

[0007] This invention provides a data processing device based on the RAFT protocol, characterized in that the data processing device is configured with a first RDMA module, and the data processing device includes a receiving unit and a transmitting unit. The receiving unit is used to receive a task request sent by the client; wherein the task request carries task data. The sending unit is used to respond to the current task request and send a write instruction for the current task data to the SCM module of the slave node using the first RDMA module in accordance with the RAFT protocol.

[0008] This invention provides a data processing device based on the RAFT protocol, characterized in that the data processing device is configured with a second RDMA module and an SCM module, and the data processing device includes a receiving unit, a writing unit, and a determining unit. The receiving unit is used to receive write instructions for task data sent by the master node using the second RDMA module; The write unit is used to respond to the write command and write the task data into the SCM module; The determining unit is used to determine the task data as applicable data when it is determined that there is a current data successful submission sequence number corresponding to the task data in the SCM module.

[0009] This invention provides a data processing device based on the RAFT protocol, comprising: a processor and a memory storing processor-executable instructions. When the instructions are executed by the processor, the data processing method based on the RAFT protocol described above is implemented.

[0010] This invention provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the data processing method based on the RAFT protocol as described above.

[0011] The technical solution provided by this invention allows the master node to directly write data to the memory of the slave node using the RDMA module, without waiting for a response from the slave node or requiring the slave node's CPU to participate, thereby further improving system throughput and increasing data transmission efficiency. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating a data processing method based on the RAFT protocol in related technologies. Figure 2 This is a schematic diagram of the structure of the distributed storage system proposed in an embodiment of the present invention; Figure 3 A schematic diagram of the implementation flow of the data processing method based on the RAFT protocol provided in this embodiment of the invention. Figure 1 ; Figure 4 This is a schematic diagram of the implementation process of the data processing method based on the RAFT protocol proposed in this embodiment of the invention. Figure 2 ; Figure 5 A schematic diagram of the implementation flow of the data processing method based on the RAFT protocol provided in this embodiment of the invention. Figure 3 ; Figure 6 A schematic diagram of the implementation flow of the data processing method based on the RAFT protocol provided in this embodiment of the invention. Figure 4 ; Figure 7 A schematic diagram of the implementation flow of the data processing method based on the RAFT protocol provided in this embodiment of the invention. Figure 5 ; Figure 8 A schematic diagram of the implementation flow of the data processing method based on the RAFT protocol provided in this embodiment of the invention. Figure 6 ; Figure 9 A schematic diagram of the implementation flow of the data processing method based on the RAFT protocol provided in this embodiment of the invention. Figure 7 ; Figure 10 A schematic diagram of the implementation flow of the data processing method based on the RAFT protocol provided in this embodiment of the invention. Figure 8 ; Figure 11 This is a flowchart illustrating the data processing method performed by the distributed storage system proposed in an embodiment of the present invention. Figure 12 This is a schematic diagram of the composition structure of the data processing device based on the RAFT protocol proposed in an embodiment of the present invention. Figure 1 ; Figure 13 This is a schematic diagram of the composition structure of the data processing device based on the RAFT protocol proposed in an embodiment of the present invention. Figure 2 ; Figure 14 This is a schematic diagram of the composition structure of the data processing device based on the RAFT protocol proposed in an embodiment of the present invention. Figure 3 ; Figure 15 This is a schematic diagram of the composition structure of the data processing device based on the RAFT protocol proposed in an embodiment of the present invention. Figure 4 . Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0015] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of the invention described herein can be implemented in an order other than that illustrated or described herein.

[0016] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0017] Before providing a further detailed description of the embodiments of the present invention, the nouns and terms involved in the embodiments of the present invention will be explained, and the nouns and terms involved in the embodiments of the present invention shall be interpreted as follows.

[0018] 1) Distributed Storage: Distributed storage is a data storage technology that utilizes the disk space on every machine within an enterprise via a network, combining these dispersed storage resources into a virtual storage device. Data is stored across various locations within the enterprise. In other words, a distributed storage system distributes data across multiple independent devices. Traditional network storage systems use a centralized storage server to store all data, making the storage server a bottleneck for system performance and a focal point for reliability and security, failing to meet the needs of large-scale storage applications. Distributed network storage systems employ a scalable system architecture, utilizing multiple storage servers to share the storage load and location servers to locate stored information. This not only improves system reliability, availability, and access efficiency but also facilitates expansion. Therefore, distributed storage systems typically maintain multiple replicas for fault tolerance, improving system availability.

[0019] 2) RAFT Protocol: RAFT is a distributed consistency protocol used in distributed storage systems to maintain consistency among multiple replicas. For example, in a cluster with consistency properties, RAFT ensures that all nodes at any given time have the same result for a given value stored therein, i.e., they maintain consistency across their shared storage. The cluster has automatic recovery properties; the failure of a few nodes does not affect the normal operation of the cluster, but when a majority of nodes in the cluster fail, the cluster will stop providing services (without returning an incorrect result). Therefore, the RAFT protocol is used to ensure that the system can still provide services even if some (more precisely, a small portion) of the replicas fail. The RAFT protocol is typically based on replicated state machines, meaning that all nodes start from the same state, go through the same sequence of operations (log), and finally arrive at the same state.

[0020] In the RAFT protocol, each node is in one of three states: Leader, Follower, or Candidate.

[0021] Leader: The handler of all requests. The Leader accepts update requests from the client, processes them locally, and then synchronizes them to multiple other slave nodes. Follower: The passive updater that receives update requests from the Leader and then writes them to the local log file.

[0022] Candidate: If a Follower does not receive a heartbeat signal from the Leader within a certain period of time, it is determined that the Leader may have failed. At this time, the leader election process is initiated, and the node will become a Candidate until the leader election is completed.

[0023] 3) Remote Direct Memory Access (RDMA) technology: Through RDMA, the local node can directly access the memory of the remote node. "Direct" means that data can be directly transferred to the remote node's memory over the network, just like accessing local memory. This bypasses the complex Transmission Control Protocol (TCP) / Internet Protocol (IP) network protocol stack of traditional Ethernet for reading and writing remote memory. It easily achieves ultra-low latency data processing and ultra-high throughput transmission without requiring the intervention of resources such as the remote node's Central Processing Unit (CPU), and avoids consuming excessive resources for data processing and migration.

[0024] 4) Persistent memory: also called non-volatile memory (NVM) or storage class memory (SCM), it is a storage hardware that supports byte-addressable storage, can be directly operated by CPU instructions, and does not lose data after power failure.

[0025] Currently, the RAFT protocol has been widely used in many fields, mainly to maintain data consistency among multiple nodes, including its application in distributed networks for distributed storage.

[0026] The specific process of distributed data storage according to the RAFT protocol currently includes: Leader: The Leader sends data to the Followers via Append Message, and after receiving a majority of successful responses from the Followers, it determines that the data has been successfully submitted. At this point, the Leader can return a success response to the client.

[0027] Follower: After receiving a message from the Leader, the Follower can persist the message and return the result to the Leader. Upon receiving the Leader's next message (heartbeat data), the Follower performs a LogApply operation, which informs the upper-layer application that the data has been saved and can be used for related processing.

[0028] For example, Figure 1 This is a flowchart illustrating a data processing method based on the RAFT protocol in related technologies, such as... Figure 1As shown, the network interface controller (NIC) of the master node and the NIC of the slave node first establish a communication connection. When the master node receives a client request (Sa1), it uses an append message to carry data and sends the data to the NIC of the slave node via the NIC (Sa2). After receiving the data (Sa3), the slave node writes the data to persistent memory (Sa4) and sends a data write success response message to the master node via the NIC. After receiving this success response (Sa6), the master node checks whether more than half of the slave nodes have responded successfully (Sa7). If not, it continues to receive response messages until more than half have responded successfully, or it indicates that the data submission to the slave node has failed. If more than half have responded successfully, it means that the data submission to the slave node has been successful, i.e., the data consistency storage is successful. Furthermore, the master node can update the data submission sequence number based on the current data number (Sa8) and send heartbeat data to the slave node via the NIC (Sa9).

[0029] Furthermore, after receiving the heartbeat data, the slave node can determine whether there is a corresponding commit sequence number (Sa10) in the heartbeat data. If not, it continues to wait to receive data; if so, it performs log application (S11) on the data written to persistent memory.

[0030] It is evident that the data processing method based on the RAFT protocol in related technologies requires waiting for a successful response message from the Follower when submitting data, and the data is only considered successfully submitted when more than half of the responses are successful. If the Follower cannot return a successful response message in a timely manner, it will lead to a decrease in the throughput of the entire data transmission system, resulting in high data transmission latency.

[0031] This invention provides a data processing method, apparatus, device, and computer storage medium based on the RAFT protocol. The data processing device, acting as the master node, is configured with a first RDMA module. The master node can receive current task requests, each carrying current task data. In response to the current task request, the master node uses the first RDMA module to send a write instruction for the current task data to the slave node's SCM module according to the RAFT protocol. Correspondingly, the data processing device, acting as the slave node, is configured with a second RDMA module and an SCM module. The slave node uses the second RDMA module to receive the write instruction for the task data sent by the master node. In response to the write instruction, the slave node writes the task data to the SCM module. When it is determined that the SCM module contains a successful submission sequence number corresponding to the current data for the task data, the task data is determined to be usable data.

[0032] In this way, the master node can use the RDMA module to directly write data to the slave node's memory based on the RAFT protocol and RDMA messages, without waiting for the slave node's return response or requiring the slave node's CPU to participate, thereby further improving the system throughput and making data transmission more efficient.

[0033] The data processing method based on the RAFT protocol provided in this invention is applied to a data processing device based on the RAFT protocol. The following describes exemplary applications of the data processing device based on the RAFT protocol provided in this invention. This data processing device can be implemented in any device with data storage capabilities, such as mobile phones, laptops, tablets, desktop computers, servers, virtual machines, smart TVs, in-vehicle devices, wearable devices, industrial equipment, and cloud devices.

[0034] The technical solutions of the present invention will now be clearly and completely described with reference to the accompanying drawings in the embodiments of the present invention.

[0035] One embodiment of the present invention provides a data processing method based on the RAFT protocol, applicable to, for example... Figure 2 The distributed storage system 100 shown includes a data processing device 10 as a master node and a data processing device 20 as a slave node. Device 10 is equipped with a first RDMA module 11; device 20 is equipped with a second RDMA module 21 and an SCM module 22. The first RDMA module 11 of device 10 is used to send write commands to the SCM module 22; the second RDMA module 21 of device 20 is used to receive write commands from device 10 and write the data to the SCM module 22.

[0036] Figure 3 This is a schematic diagram of the implementation process of the data processing method based on the RAFT protocol proposed in this embodiment of the invention. Figure 1 ,like Figure 3 As shown, in an embodiment of the present invention, the method for the master node to perform data processing based on the RAFT protocol may include the following steps: S101. Receive the current task request; wherein the current task request carries the current task data.

[0037] In this embodiment of the invention, the master node can receive task requests carrying task data.

[0038] It should be understood that, in this embodiment of the invention, the current task request can be a request sent by a client received by the master node, and the current task data is the data value carried in the task request. This request can be viewed as a data write request or a data change request, and the data carried in the request is considered as newly written data or changed data. For example, writing a new data value from the client to a service node in a distributed storage system.

[0039] It should be noted that, in this embodiment of the invention, when a client initiates a data change requiring the writing of a new value to a service node in the distributed storage system, the client can first connect to the master node and send a write request carrying the changed data to the master node. In other words, the client can first send all requests carrying changed data to the master node in the system.

[0040] It should be understood that, in embodiments of the present invention, the client may be other devices different from the master node, such as mobile phones, laptops, tablets, desktop computers, servers, virtual machines, smart TVs, in-vehicle devices, wearable devices, industrial equipment, etc.

[0041] In some embodiments, the master node can receive multiple task requests sent by the client and place these multiple task requests in a buffer in the form of a message queue in chronological order.

[0042] In some embodiments, the master node can receive a set of task requests sent by the client, which includes multiple requests; the master node can merge these multiple requests and package them into a single task request package. Alternatively, the master node can directly receive a task request package sent by the client; wherein the task request package contains multiple requests.

[0043] In some embodiments, the master node can receive a set of task requests from a client, each containing multiple requests. These multiple requests correspond to the same transaction ID, which links the requests together. If the write of modified data carried by one task request to a service node in the distributed storage system fails, all associated task requests also fail simultaneously. Conversely, if the write of data in one task request succeeds, the write of data in other associated task requests succeeds simultaneously.

[0044] Furthermore, in this embodiment of the invention, after receiving a request from the client carrying task data, the master node can further write the task data to the slave node.

[0045] S102. In response to the current task request, the first RDMA module sends a write instruction for the current task data to the SCM module of the slave node according to the RAFT protocol.

[0046] In this embodiment of the invention, after receiving the task request, the master node can further respond to the request by directly sending a data write instruction to the slave node's SCM.

[0047] It is understood that the RDMA module supports remote direct memory data access. Therefore, in the embodiments of the invention, the master node can be configured with an RDMA module so that the master node can support direct memory data access of the data sent by the client to the slave node.

[0048] In addition, the slave node also needs to be configured with an RDMA module. After the master node and slave node establish communication based on the RDMA module, data can be directly written to the slave node's memory.

[0049] Furthermore, to ensure that data is not lost after a power outage, the slave node can be configured with a persistent memory module, namely the SCM module. In this way, the master node can directly send write commands to the slave node's SCM module, that is, write data directly to the slave node's SCM without the involvement of the CPU.

[0050] In summary, both the master node and the slave node are configured with RDMA modules, and a communication connection is established based on the RDMA modules to enable the master node to directly write data into the slave node's SCM module.

[0051] It should be understood that in this embodiment of the invention, data is written to the service nodes in the system according to the RAFT protocol. Therefore, after receiving each task request (such as data change) from the client, in response to the request, the master node will add each change data as an entry to the master node's log. This entry has a corresponding log number. At this time, the log status is uncommitted. Next, the master node needs to commit the entry, that is, copy the change data to all slave nodes.

[0052] Specifically, the master node can implement replication via Append Message according to the RAFT protocol. Furthermore, the master node can also commit entries via Append Message and based on RDMA Write technology supported by the RDMA module, directly writing each changed data to the slave node's SCM module to achieve data replication to the slave node.

[0053] Therefore, the data processing method based on the RAFT protocol for master nodes provided in this embodiment of the invention allows the master node to directly write data to the SCM module of the slave node using the RDMA module according to the RAFT protocol, without the need for CPU involvement, thereby further improving system throughput and data transmission efficiency.

[0054] In other embodiments of the present invention Figure 4A schematic diagram of the implementation flow of the data processing method based on the RAFT protocol provided in this embodiment of the invention. Figure 2 ,like Figure 4 As shown, after responding to the task request and sending the write instruction for the task data to the SCM module of the slave node using the first RDMA module according to the RAFT protocol, i.e. after step 102, the method for the master node to perform data processing based on the RAFT protocol may include the following steps: S103. In response to the write instruction that successfully sends task data to a preset number of slave nodes, the task number corresponding to the task data is determined as the current data successful submission sequence number.

[0055] Understandably, since the master node directly writes data to the slave node's memory using RDMA Write technology supported by the RDMA module, the master node can determine that the data has been successfully written to the slave node as long as the data write command is successfully sent.

[0056] In addition, when the RDMA module is exchanging spatial information, if a task is completed, it will place the completed task in the Completion Queue (CQ). The master node can query the Completion Queue (CQ) based on the underlying message to know which tasks have been successfully completed.

[0057] In this embodiment of the invention, when the master node performs a write operation to the slave node, the master node does not need to wait for the slave node to return a response message indicating that the data has been successfully written. The master node can use the completion queue (CQ) to confirm whether the number of slave nodes that have successfully sent the write instruction for the current task data meets the preset number, thereby determining whether the task data has been successfully copied in the distributed storage system.

[0058] Specifically, after the master node successfully sends the write instruction for the task data to a preset number of slave nodes, if the master node determines, based on the completion queue (CQ), that the write instruction for the task data has been successfully sent to a preset number of slave nodes, then it indicates that the changed data sent by the client has been successfully replicated to the service nodes of the distributed storage system.

[0059] It should be noted that the preset number can refer to more than half of the nodes in the entire distributed storage system. That is, after the master node confirms that it has successfully sent the write command for the task data to more than half of the slave nodes, the master node can determine that the task data has been successfully replicated to more than half of the slave nodes in the distributed system, that is, the changed data sent by the client has been successfully replicated to the distributed storage system.

[0060] It should be noted that, in this embodiment of the invention, the task number corresponding to the task data refers to the log number corresponding to the entry of that task data in the log. The data successful submission sequence number refers to the sequence number of the entry where the data was successfully copied.

[0061] Furthermore, after the master node confirms that the changed data sent by the client has been successfully replicated to the distributed storage system, that is, the entry corresponding to the data of this task has been successfully committed, the master node can determine the log number corresponding to the above entry as the current data successful commit sequence number, i.e., the commit number.

[0062] For example, the log contains multiple entries corresponding to various data changes, each with its own number, such as 1, 2, 3, 4, and 5. When the data change for entry 1 is successfully replicated to more than half of the slave nodes, it indicates that the data change for entry 1 has been successfully replicated to the distributed storage system, and the current data's successful commit sequence number is updated to 1. Furthermore, when the data change for entry 2 is successfully replicated to more than half of the slave nodes, the current data's successful commit sequence number is updated to 2. However, if the data change for entry 3 is not successfully replicated to more than half of the slave nodes, the current data's successful commit sequence number remains 3.

[0063] Furthermore, in this embodiment of the invention, after the master node determines that the task data has been successfully replicated to a majority of the service nodes in the distributed storage system, the master node determines that the entry corresponding to the changed data has been successfully committed, and at this time the log status corresponding to the entry is committed.

[0064] Furthermore, in this embodiment of the invention, after obtaining the current data successful submission sequence number, the master node can further send the data successful submission sequence number to each of the majority of slave nodes that have successfully written the task data.

[0065] S104. Using the first RDMA module, a write instruction indicating the current data successful submission sequence number is sent to each SCM module of each of the slave nodes that meet the preset number of slave nodes.

[0066] In this embodiment of the invention, after the master node obtains the current data successful submission sequence number, it can send a write instruction of the data successful submission sequence number to the SCM module of each of the multiple slave nodes that have successfully written the task data through the RDMA module. That is, the RDMA module is used to directly write the data successful submission sequence number into the SCM module of the slave node.

[0067] In some embodiments, the master node can write the data successful submission sequence number to the slave node's SCM module using the Append Message format in the RAFT protocol and based on the RDMA Write technology supported by the RDMA module.

[0068] The master node can send a write command to the slave node with the current data successful submission sequence number, or it can merge the current data successful submission sequence number with the next task data sent by the client, and write the merged content to the slave node's SCM module using RDMA Write technology in the form of an Append Message.

[0069] It should be understood that in a distributed storage system based on the RAFT protocol, the master node can periodically trigger heartbeat data to inform the slave node of the latest committed entry number in the log. Therefore, in the implementation of this invention, after obtaining the current data successful commit sequence number, the master node can provide the data successful commit sequence number to the slave node in the next heartbeat data based on the RAFT protocol. That is, it can use RDMA Write technology to directly write the current data successful commit sequence number into the slave node's SCM module, or use RDMA Send technology to send it to the slave node; wherein, the heartbeat data only contains the latest data successful commit sequence number.

[0070] Furthermore, in this embodiment of the invention, the master node can return a message indicating that the task was successful to the client.

[0071] Furthermore, in this embodiment of the invention, the master node can perform LogApply on the current task data that has been successfully written to a majority of slave nodes, that is, determine that the task data is applicable data, and inform the upper-layer application that the data has been effectively stored in a majority of service nodes in the distributed storage system, and can be used by the upper-layer application for some related processing.

[0072] As can be seen, in this embodiment of the invention, the master node can directly write data into the persistent memory of the slave node and directly determine the success of data writing through the underlying message, without waiting for the return response sent by the slave node, thereby reducing communication content and reducing data transmission latency.

[0073] Based on the above embodiments, in another embodiment of the present invention, Figure 5 A schematic diagram of the implementation flow of the data processing method based on the RAFT protocol provided in this embodiment of the invention. Figure 3 ,like Figure 5 As shown, in an embodiment of the present invention, the method for the master node to perform data processing based on the RAFT protocol may include the following steps: S201. Receive multiple task requests and cache them in a message queue.

[0074] S202. After sending a write instruction for at least one task data carried by at least one historical task request in the message queue to the SCM module of the slave node, in response to the current task request, the first RDMA module sends a write instruction for the current task data to the SCM module of the slave node.

[0075] It should be understood that, in order to overcome the drawback of the RAFT protocol requiring strict ordering and thus unable to achieve concurrent submissions, in this embodiment of the invention, the master node sets up a message queue buffer, which is located in the master node's memory. When the master node receives a task request sent by a client, the master node can put it into the message queue buffer sequentially according to the time order.

[0076] Furthermore, the master node can use the aforementioned message queue to concurrently submit task data carried in task requests.

[0077] Specifically, if the master node has not successfully sent write instructions for at least one task request carrying task data to the SCM modules of more than a preset number (e.g., more than half) of the slave nodes' message queues, the master node can continue to send write instructions for the next task request carrying task data to the SCM modules of the slave nodes through Append Message messages and based on RDMA Write technology.

[0078] In other words, in the log entries of task data arranged sequentially by number, once the entry corresponding to at least one of the first numbered task data is submitted, the task is suspended. Before it is determined whether at least one of its entries has been successfully submitted, the master node can continue to submit the entry corresponding to the next task data.

[0079] As can be seen, the master node caches the data change requests sent by the client in a queue manner. When executing the current data change request sent by the client, it does not need to wait for the task to be completed successfully. It can continue to perform the write operation of data to the slave node's memory based on the next data change request initiated by the client. That is, the master node can submit multiple requests in sequence, realizing concurrent data submission, further improving the system throughput and data transmission efficiency.

[0080] Based on the above embodiments, in another embodiment of the present invention, Figure 6 This is a schematic diagram of the implementation process of the data processing method based on the RAFT protocol proposed in this embodiment of the invention. Figure 4 ,like Figure 6 As shown, the method for the master node to perform data processing based on the RAFT protocol may include the following steps: S301: Receive multiple data sent by multiple slave nodes to be written to the virtual address.

[0081] S302. Based on the length of the task data, determine the first virtual address from the virtual address to be written to the data of each slave node.

[0082] S303. Use the first RDMA module to send a write instruction for task data to the first physical address corresponding to the first virtual address in each SCM module of each slave node.

[0083] In this embodiment of the invention, the master node can receive data to be written to the virtual address, i.e., the available free space of data, sent by the slave node in the form of private messages or custom commands in real time or periodically. This virtual address is the same as the physical address of the SCM module in the slave node. The master node can actively determine the corresponding storage location for the current task data from the available free space of data.

[0084] Specifically, the master node can determine the length of the current task data and calculate the virtual address space corresponding to the current task data, i.e., the first virtual address, from the virtual address to be written to the data based on the data length.

[0085] Optionally, the master node can also determine the first virtual address corresponding to the current task data from the virtual addresses to be written to the data according to the preset address allocation rules. For example, the preset address allocation rules can be that each data is allocated 1k of space.

[0086] Furthermore, after determining the first virtual address corresponding to the data, the master node determines the virtual memory location where the current task data is written. Then, the master node can write the task data into the SCM module and the physical address corresponding to that virtual address, that is, directly send the write instruction of the current task data to the SCM module and the physical address corresponding to that virtual address.

[0087] As can be seen, in this embodiment of the invention, when the master node performs a write operation on task data based on the RDMA Write technology supported by the RDMA module, the master node can actively determine the memory write location of the data and perform a write operation on the slave node SCM module based on the memory write location.

[0088] In other embodiments of the present invention Figure 7 A schematic diagram of the implementation flow of the data processing method based on the RAFT protocol provided in this embodiment of the invention. Figure 5 ,like Figure 7 As shown, a method for performing data processing based on the RAFT protocol from a node may include the following steps: S401. Utilize the second RDMA module to receive the write command for the task data sent by the master node.

[0089] S402, respond to the write command and write the task data to the SCM module.

[0090] S403. When it is determined that there is a successful submission sequence number for the current data corresponding to the task data in the SCM module, the task data is determined to be applicable data.

[0091] It should be noted that in this embodiment of the invention, both the master node and the slave node are configured with RDMA modules. The communication connection established based on the RDMA module enables the master node to directly write data into the SCM module of the slave node, that is, to directly write the task data into persistent memory based on the write command sent by the master node.

[0092] It should be understood that after the master node actively sends a write instruction to the slave node's SCM module based on the RDMA module, the slave node can receive the write instruction using the RDMA module and directly write the data to persistent memory (i.e., the SCM module) without CPU intervention.

[0093] Optionally, in this embodiment of the invention, the task data may be the task data corresponding to a single task request from the client, or the corresponding task data after packaging a group of task requests including multiple requests; or the task data corresponding to any one of a group of task requests with the same transaction number and an association relationship.

[0094] Furthermore, in this embodiment of the invention, based on the feature in S103 to S104 that if the master node determines that the task data has been successfully replicated to more than half of the slave nodes, then the master node can send a write instruction to the SCM module of the slave node indicating the successful data submission sequence number. It can be seen that the slave node can periodically query the successful data submission sequence number written by the SCM module. As long as the slave node finds the successful data submission sequence number corresponding to the latest written task data in the SCM module, the slave node can perform LogApply on the task data, that is, determine that the task data is determined to be applicable data. The slave node can inform the upper-layer application that the data has been effectively stored in most of the service nodes in the distributed storage system and can be used by the upper-layer application for some related processing.

[0095] In other embodiments, the slave node can also receive heartbeat data sent by the master node. The slave node can determine the data successful submission sequence number carried in the heartbeat data. If it is the data successful submission sequence number corresponding to the latest written task data, the slave node can execute LogApply on the data, that is, determine that the task data is determined to be applicable data and can be used by the upper layer application for some related processing.

[0096] As can be seen, in this embodiment of the invention, the slave node does not require CPU participation and can directly write data to persistent memory according to the write instructions sent by the master node, without having to return a response message to the master node, thus reducing communication content and lowering data transmission latency.

[0097] Based on the above embodiments, Figure 8 A schematic diagram of the implementation flow of the data processing method based on the RAFT protocol provided in this embodiment of the invention. Figure 6 ,like Figure 8 As shown, a method for performing data processing based on the RAFT protocol from a node may include the following steps: S601. Perform memory mapping on the physical address corresponding to the SCM module to obtain the virtual address corresponding to the physical address.

[0098] S602. Register the virtual address to the second RDMA module so that the second RDMA module can write task data to the physical address in the SCM module by accessing the virtual address.

[0099] Understandably, to enable the master node to directly write data to the persistent memory of the slave node using the RDMA module, as described in this invention, data writing to persistent memory needs to be achieved through communication between the master node's RDMA module and the slave node's RDMA module. Therefore, it is first necessary to map the storage space corresponding to the SCM module to the virtual memory of the device operating system and register the virtual memory with the RDMA module. Only then can the RDMA module have permission to access this virtual memory area.

[0100] As can be seen, in this embodiment of the invention, memory mapping and registration need to be completed before data processing.

[0101] Specifically, in this embodiment of the invention, the slave node can first perform memory mapping processing on the physical address corresponding to the SCM module, mapping it to the device's local memory space, that is, the virtual address space, thereby establishing a mapping relationship between physical addresses and virtual addresses, such as an address mapping table, so as to obtain the virtual address corresponding to the physical address.

[0102] Optionally, the entire or partial space of the SCM module can be mapped to the device's virtual address space using dev-dax or fs-dax.

[0103] Furthermore, after completing the mapping between the physical address space and the virtual address space, the slave node can register all or part of the virtual address space to the local RDMA module. After successful registration, the RDMA module can obtain the corresponding virtual address and the memory reference returned by the registration (this memory reference can be regarded as a "key" used to control and manage memory access permissions), and thus has the right to access the virtual address space.

[0104] The RDMA module can access the virtual address space based on the MR returned by the registration, and write data to the physical address in the SCM module. It should be understood that when the RDMA module writes the log to the virtual address in memory, it is actually writing the data to the physical address corresponding to that virtual address, i.e., persistent memory.

[0105] It should be noted that, in this embodiment of the invention, after the slave node completes the memory mapping and registers it with the RDMA module, the slave node can also provide the virtual address corresponding to its own machine and the registered MR to other nodes (not limited to the master node or other slave nodes) in the form of private messages through the RDMA module.

[0106] The slave node can be configured with a timing module, such as a timer, which can send the current free space (i.e., the virtual address to which data is to be written) to the master node in real time or periodically in the form of a custom command or a private message.

[0107] As can be seen, the slave node can map the physical address space of the SCM module to virtual memory and register it with the RDMA module, enabling the RDMA module to write task data to the physical address in the SCM module through memory access.

[0108] Based on the above embodiments, Figure 9 A schematic diagram of the implementation flow of the data processing method based on the RAFT protocol provided in this embodiment of the invention. Figure 7 ,like Figure 9 As shown, a method for performing data processing based on the RAFT protocol from a node may include the following steps: S701: Receive the first write virtual address sent by the master node using the second RDMA module.

[0109] S702. Determine the first physical address corresponding to the first virtual address from the SCM module.

[0110] S703: Use the second RDMA module to write the task data to the first physical address.

[0111] Understandably, the master node needs to receive the data write command through the RDMA module in the slave node and then write the data to persistent memory. Therefore, the master node needs to send the memory address of the data in the slave node to the slave node so that the slave node can write the task data to persistent memory based on that memory address.

[0112] In this embodiment of the invention, while the slave node is using the RDMA module to receive the write instruction for the task data sent by the master node, it can also use the RDMA module to receive the first virtual address corresponding to the task data sent by the master node.

[0113] Furthermore, the RDMA module can determine the first physical address corresponding to the first virtual address from the SCM module based on the above-mentioned mapping relationship between virtual addresses and physical addresses, such as an address mapping table, and then write the data into the first physical address to complete the data writing to persistent memory.

[0114] Therefore, it can be seen that the slave node can receive the data sent by the master node and the virtual address determined by the master node for the data, and write the task data into persistent memory based on the address mapping relationship.

[0115] Based on the above embodiments, Figure 10 A schematic diagram of the implementation flow of the data processing method based on the RAFT protocol provided in this embodiment of the invention. Figure 8 ,like Figure 10 As shown, a method for performing data processing based on the RAFT protocol from a node may include the following steps: S801: Use the second RDMA module to receive the write command of the current data submission sequence number sent by the master node.

[0116] S802, respond to the write command and write the current data successful submission sequence number to the SCM module.

[0117] In this embodiment of the invention, based on S103 and S104, after the master node determines that the write instruction to send data to the slave nodes that meet the preset number is successful, and sends the write instruction to the slave nodes with the data successful submission sequence number, the slave nodes can directly write the data successful submission sequence number into the slave node's SCM module, that is, write the data successful submission sequence number to persistent memory based on the write instruction sent by the master node.

[0118] Specifically, the slave node can use the RDMA module to receive the write instruction sent by the master node indicating that the current data has been successfully committed. At the same time, the slave node can also use the RDMA module to receive the target virtual address (i.e., the second virtual address) determined by the master node from the virtual address to be written for the data with that sequence number. Then, based on the mapping relationship between virtual addresses and physical addresses, such as an address mapping table, the RDMA module can determine the second physical address corresponding to the second virtual address from the SCM module, and then write the current data successfully committed sequence number to the second physical address, completing the write of the committed sequence number to persistent memory.

[0119] Therefore, it can be seen that the slave node can receive the current data successful submission sequence number sent by the master node and the virtual address determined by the master node for the sequence number, and write the sequence number into persistent memory based on the address mapping relationship.

[0120] For example, Figure 11This is a flowchart illustrating the data processing method of the distributed storage system proposed in this embodiment of the invention. Both the master node and the slave node in the distributed system are equipped with Ethernet cards with RDMA engine modules, namely (RDMA Network Interface Controller, RNIC). The RNIC in the master node establishes a communication connection with the RNIC in the slave node to realize the interaction of spatial information.

[0121] like Figure 11 As shown, the master node receives a client request (Sb1), carries data via an Append Message, and uses RDMA Write technology to send the data to the slave node's RNIC via RNIC (Sb2). At this point, it doesn't wait for a successful response from the slave node. Instead, it queries the completion queue (CQ) based on the RDMA underlying messages to obtain a message indicating whether data was successfully sent to the slave node (Sb3), and determines whether more than half of the slave nodes have successfully sent the data (Sb4). If not, it continues querying the CQ completion queue until more than half have successfully sent the data; or it indicates that the data submission to the slave node failed. If more than half have successfully submitted the data, it means the data submission to the slave node was successful, i.e., data consistency storage was successful. Furthermore, the master node can update the data submission sequence number based on the current task data number (Sb5) and send it to the slave's RNIC via RNIC using an Append Message and RDMA Write technology. Simultaneously, the master node can also perform LogApply on this data (Sb6).

[0122] On the other hand, the slave node's RNIC directly writes the data sent by the master node into persistent memory (Sb7), without CPU involvement and without requiring any response messages from the master node. Furthermore, the slave node can periodically check the successful commit sequence number of the data written to persistent memory (Sb8) and determine whether there is a commit sequence number corresponding to newly committed task data in persistent memory (Sb9). If so, the corresponding data is applied to the log (Sb10); otherwise, the persistent memory is checked periodically.

[0123] In summary, by configuring RDMA modules on both the master and slave nodes in a distributed storage system, the master node can utilize RDMA Write technology to directly write data to the persistent memory of the slave nodes. This eliminates the need to wait for a response from the slave nodes and requires no CPU intervention from the slave nodes, thereby further improving system throughput and increasing data transfer efficiency.

[0124] Based on the above embodiments, in one embodiment of the present invention, Figure 12 This is a schematic diagram of the composition structure of the data processing device based on the RAFT protocol proposed in an embodiment of the present invention. Figure 1 ,like Figure 12 As shown, the data processing device 10 based on the RAFT protocol includes a receiving unit 12, a sending unit 13, a buffering unit 14, and a determining unit 15. The receiving unit 12 is used to receive a current task request; wherein the current task request carries current task data; The sending unit 13 is used to respond to the current task request and send a write instruction for the current task data to the SCM module of the slave node using the first RDMA module in accordance with the RAFT protocol.

[0125] In some embodiments, the receiving unit 12 is further configured to receive a plurality of the task requests; The caching unit 14 is used to cache the multiple task requests to the message queue; The sending unit 13 is further configured to, after sending a write instruction for at least one task data carried by at least one historical task request in the message queue to the SCM module of the slave node, respond to the current task request and use the first RDMA module to send a write instruction for the current task data to the SCM module of the slave node.

[0126] In some embodiments, the receiving unit 12 is further configured to receive multiple data to be written to a virtual address sent by multiple slave nodes; The determining unit 15 is used to determine a first virtual address from the virtual address to be written to the data of each slave node according to the length of the task data; The sending unit 13 is specifically used to send a write instruction for the task data to the first physical address corresponding to the first virtual address in each of the SCM modules of each of the slave nodes using the first RDMA module.

[0127] In some embodiments, the determining unit 15 is further configured to, in response to the task request, send a write instruction for the task data to the SCM module of the slave node using the first RDMA module according to the RAFT protocol, and in response to successfully sending the write instruction for the task data to a preset number of slave nodes, determine the task number corresponding to the task data as the current data successful submission sequence number. The sending unit 13 is further configured to use the first RDMA module to send a write instruction for the current data successful submission sequence number to each of the SCM modules of each of the slave nodes that meet the preset number of slave nodes.

[0128] Based on the above embodiments, in one embodiment of the present invention, Figure 13 This is a schematic diagram of the composition structure of the data processing device based on the RAFT protocol proposed in an embodiment of the present invention. Figure 2 ,like Figure 13 As shown, the data processing device 20 based on the RAFT protocol includes a receiving unit 23, a writing unit 24, a determining unit 25, a memory mapping unit 26, and a registration unit 27. The receiving unit 23 is used to receive a write instruction for task data sent by the master node using the second RDMA module; The write unit 24 is used to respond to the write command and write the task data into the SCM module; The determining unit 25 is used to determine the task data as applicable data when it is determined that there is a current data successful submission sequence number corresponding to the task data in the SCM module.

[0129] In some embodiments, the receiving unit 23 is further configured to use the second RDMA module to receive a write instruction sent by the master node indicating that the current data has been successfully committed. The write unit 24 is also used to respond to the write instruction and write the current data successful submission sequence number into the SCM module.

[0130] In some embodiments, the memory mapping unit 26 is used to perform memory mapping processing on the physical address corresponding to the SCM module to obtain the virtual address corresponding to the physical address; The registration unit is used to register the virtual address to the second RDMA module, so that the second RDMA module can write the task data into the physical address in the SCM module by accessing the virtual address.

[0131] In some embodiments, the receiving unit 23 is further configured to receive the first write virtual address sent by the master node using the second RDMA module; The write unit 24 is specifically used to determine the first write physical address corresponding to the first write virtual address from the SCM module; and to write the task data to the first write physical address using the second RDMA module.

[0132] In an embodiment of the present invention, further, Figure 14 This is a schematic diagram of the composition structure of the data processing device based on the RAFT protocol proposed in this invention. Figure 3 ,like Figure 14 As shown, the data processing device 10 based on the RAFT protocol proposed in this embodiment of the invention may further include a first RDMA module 11, a processor 16, and a memory 17 storing executable instructions of the processor 16. Further, the data processing device 10 based on the RAFT protocol may also include a communication interface 18 and a bus 19 for connecting the processor 16, the memory 17 and the communication interface 18.

[0133] In embodiments of the present invention, the processor 16 may be at least one of the following: Application-Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field-Programmable Gate Array (FPGA), Central Processing Unit (CPU), Controller, Microcontroller, and Microprocessor. It is understood that for different devices, the electronic device used to implement the above-mentioned processor function may also be other types, and the embodiments of the present invention do not specifically limit this. The data processing device 10 based on the RAFT protocol may further include a memory 17, which may be connected to the processor 16. The memory 17 is used to store executable program code, which includes computer operation instructions. The memory 17 may include high-speed RAM memory and may also include non-volatile memory, such as at least two disk drives.

[0134] In an embodiment of the present invention, bus 19 is used to connect communication interface 18, processor 16 and memory 17 and the mutual communication between these devices.

[0135] In an embodiment of the present invention, memory 17 is used to store instructions and data.

[0136] Furthermore, in an embodiment of the present invention, the processor 16 is configured to receive a current task request; wherein the current task request carries current task data; and in response to the current task request, to send a write instruction of the current task data to the SCM module of the slave node using the first RDMA module according to the RAFT protocol.

[0137] In practical applications, the aforementioned memory 17 can be volatile memory, such as random-access memory (RAM); or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 16.

[0138] In an embodiment of the present invention, further, Figure 15 This is a schematic diagram of the composition structure of the data processing device based on the RAFT protocol proposed in this invention. Figure 4 ,like Figure 15 As shown, the data processing device 20 based on the RAFT protocol proposed in this embodiment of the invention may further include a second RDMA module 21, an SCM module 22, a processor 28, and a memory 29 storing executable instructions of the processor 28. Further, the data processing device 20 based on the RAFT protocol may also include a communication interface 210 and a bus 211 for connecting the processor 28, the memory 29 and the communication interface 210.

[0139] In embodiments of the present invention, the processor 28 may be at least one of the following: Application-Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field-Programmable Gate Array (FPGA), Central Processing Unit (CPU), Controller, Microcontroller, and Microprocessor. It is understood that for different devices, the electronic device used to implement the above-mentioned processor function may also be other types, and the embodiments of the present invention do not specifically limit this. The data processing device 20 based on the RAFT protocol may further include a memory 29, which may be connected to the processor 28. The memory 29 is used to store executable program code, which includes computer operation instructions. The memory 29 may include high-speed RAM memory and may also include non-volatile memory, such as at least two disk drives.

[0140] In an embodiment of the present invention, bus 211 is used to connect communication interface 210, processor 28 and memory 29 and the mutual communication between these devices.

[0141] In an embodiment of the present invention, memory 29 is used to store instructions and data.

[0142] Furthermore, in an embodiment of the present invention, the processor 28 is configured to receive a write instruction for task data sent by the master node using the second RDMA module; respond to the write instruction by writing the task data into the SCM module; and when it is determined that there is a current data successful submission sequence number corresponding to the task data in the SCM module, the task data is determined to be usable data.

[0143] In practical applications, the aforementioned memory 29 can be volatile memory, such as random-access memory (RAM); or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 28.

[0144] Furthermore, in this embodiment, the functional modules can be integrated into a single recommended unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit. The integrated unit can be implemented in hardware or as a software functional module.

[0145] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method of this embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0146] This invention provides a data processing device based on the RAFT protocol. The data processing device acting as the master node is configured with a first RDMA module, which can receive current task requests. These current task requests carry current task data. In response to the current task request, the master node sends a write instruction for the current task data to the SCM module of the slave node using the first RDMA module, according to the RAFT protocol. Correspondingly, the data processing device acting as the slave node is configured with a second RDMA module and an SCM module. The slave node uses the second RDMA module to receive the write instruction for the task data sent by the master node. In response to the write instruction, the slave node writes the task data to the SCM module. When it is determined that the SCM module contains a successful submission sequence number corresponding to the current data for the task data, the task data is determined to be usable data. Thus, data can be directly written to the slave node's memory using the RDMA module without waiting for a response from the slave node or requiring the slave node's CPU, thereby further improving system throughput and data transmission efficiency.

[0147] This invention provides a computer-readable storage medium storing a program that, when executed by a processor, implements the data processing method based on the RAFT protocol applied to a master node or a slave node as described above.

[0148] Specifically, the program instructions corresponding to a data processing method based on the RAFT protocol in this embodiment can be stored on storage media such as optical discs, hard disks, and USB flash drives.

[0149] When program instructions corresponding to a RAFT-based data processing method applied to a master node, stored in a storage medium, are read or executed by an electronic device, the following steps are included: Receive current task request; wherein, the current task request carries current task data; In response to the current task request, the first RDMA module sends a write instruction for the current task data to the SCM module of the slave node according to the RAFT protocol.

[0150] When an electronic device reads or executes program instructions in a storage medium corresponding to a RAFT-based data processing method applied to a slave node, the following steps are included: The second RDMA module is used to receive write commands for task data sent by the master node. In response to the write command, the task data is written to the SCM module; When it is determined that the current data corresponding to the task data has a successful submission sequence number in the SCM module, the task data is determined to be applicable data.

[0151] Accordingly, this embodiment of the invention further provides a computer program product, which includes computer-executable instructions for implementing the steps in the data processing method based on the RAFT protocol for a master node or a slave node provided in this embodiment of the invention.

[0152] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0153] This invention is described with reference to schematic and / or block diagrams illustrating the implementation of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the schematic and / or block diagrams can be implemented by computer program instructions, as well as combinations of blocks in the schematic and / or block diagrams. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the schematic and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0154] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the implementation flow diagram. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0155] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the functions specified in one or more boxes. The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention.

Claims

1. A data processing method based on the distributed consensus RAFT protocol, characterized in that, The data processing method is applied to a master node, which is configured with a first Remote Direct Data Access (RDMA) module. The method includes: Receive multiple task requests and cache the multiple task requests sequentially into a message queue; Send a write instruction for at least one task data carried by at least one historical task request in the message queue to the SCM module of the slave node. When it is not determined whether at least one historical task request was successful, the next task request in the message queue is sent sequentially to the slave node. Receive current task request; wherein, the current task request carries current task data; In response to the current task request, the first RDMA module sends a write instruction for the current task data to the first physical address corresponding to the first virtual address in the SCM module of the slave node, according to the RAFT protocol.

2. The method according to claim 1, characterized in that, The method further includes: Receive multiple data packets from multiple slave nodes to be written to the virtual address; Based on the length of the task data, a first virtual address is determined from the virtual address to be written to the data of each slave node.

3. The method according to claim 2, characterized in that, After responding to the task request and sending a write instruction for the task data to the SCM module of the slave node using the first RDMA module according to the RAFT protocol, the method further includes: In response to successfully sending a write instruction for the task data to a preset number of slave nodes, the task number corresponding to the task data is determined as the current data successful submission sequence number. The first RDMA module sends a write instruction indicating the current data successful submission sequence number to each SCM module of each of the preset number of slave nodes.

4. A data processing method based on the RAFT protocol, characterized in that, The data processing method is applied to a slave node, the slave node being configured with a second RDMA module and a storage-class memory (SCM) module, the method comprising: The first physical address corresponding to the first virtual address in the second RDMA module is used to receive write instructions for task data sent sequentially by the master node when it is uncertain whether at least one historical task request has been successful; wherein, the at least one historical task request carries a write instruction for at least one task data. In response to the write command, the task data is written to the SCM module; When it is determined that the current data corresponding to the task data has a successful submission sequence number in the SCM module, the task data is determined to be applicable data.

5. The method according to claim 4, characterized in that, The method further includes: The second RDMA module is used to receive the write instruction of the current data submission sequence number sent by the master node; In response to the write command, the current data successful submission sequence number is written to the SCM module.

6. The method according to claim 4, characterized in that, The method further includes: The physical address corresponding to the SCM module is memory-mapped to obtain the virtual address corresponding to the physical address. The virtual address is registered to the second RDMA module so that the second RDMA module can write the task data to the physical address in the SCM module by accessing the virtual address.

7. The method according to claim 6, characterized in that, The method further includes: The second RDMA module is used to receive the first virtual address sent by the master node. The step of responding to the write command and writing the task data into the SCM module includes: The first physical address corresponding to the first virtual address is determined from the SCM module; The task data is written to the first physical address using the second RDMA module.

8. A data processing device based on the RAFT protocol, characterized in that, The data processing device is configured with a first RDMA module, and the data processing device includes a buffer unit, a transmitting unit, and a receiving unit. The caching unit is used to receive multiple task requests and cache the multiple task requests sequentially into a message queue; The sending unit is configured to send a write instruction for at least one task data carried by at least one historical task request in the message queue to the SCM module of the slave node; and is also configured to send the next task request in the message queue to the slave node sequentially when it is not determined whether the at least one historical task request was successful. The receiving unit is configured to receive a current task request; wherein the current task request carries current task data; The sending unit is further configured to respond to the current task request and, in accordance with the RAFT protocol, use the first RDMA module to send a write instruction for the current task data to the first physical address corresponding to the first virtual address in the SCM module of the slave node.

9. A data processing device based on the RAFT protocol, characterized in that, The data processing device is configured with a second RDMA module and an SCM module, and includes a receiving unit, a writing unit, and a determining unit. The receiving unit is configured to receive, using the first physical address corresponding to the first virtual address in the second RDMA module, a write instruction for task data sent sequentially by the master node when it is uncertain whether at least one historical task request has been successful, wherein the at least one historical task request carries a write instruction for at least one task data. The write unit is used to respond to the write command and write the task data into the SCM module; The determining unit is used to determine the task data as applicable data when it is determined that there is a current data successful submission sequence number corresponding to the task data in the SCM module.

Images