Multi-copy storage system and multi-copy storage method

Abstract

The embodiment of the invention provides a multi-copy storage system and a multi-copy storage method, the multi-copy storage system comprises a request node, a master node, a slave node and a shared memory pool based on a high-speed interconnection protocol, and the shared memory pool comprises a metadata area, a slave node feedback area and a data block area; the request node is configured to obtain a target write-in request, write the target write-in request into the metadata area and the data block area of the shared memory pool, and transmit the target write-in request to the main node through the shared memory pool; the master node is configured to access the shared memory pool to obtain a target write-in request, write the target write-in request into the first local storage unit, and transmit the target write-in request to the slave node through the shared memory pool; and the slave node is configured to access the shared memory pool to obtain the target write-in request, write the target write-in request into the second local storage unit, and write a feedback signal into a slave node feedback area of the shared memory pool. And the cross-node network overhead is saved.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of data storage technology, and in particular to a multi-copy storage system and a multi-copy storage method. Background Technology

[0002] With the rapid development of computer technology and data storage technology, multi-replica storage systems (MRS) have become an indispensable component in distributed services (such as distributed databases and distributed file systems), meeting the requirements for high fault tolerance and data availability.

[0003] In existing technologies, multi-replica storage systems require replica write operations, which persist the corresponding metadata and data to multiple nodes. In a multi-replica storage system, a designated central node is responsible for coordinating the replica write process. Specifically, after the central node receives a write request from the client via network communication, it propagates the write request to a group of follower nodes via network communication. Both the central node and the follower nodes unpack the metadata and data from the write request and persist them locally. If each follower node completes local persistence, it sends an acknowledgment message to the central node via network communication. After determining that a given number of nodes have completed local persistence, the central node reports the completion of the request to the client via network communication.

[0004] However, the write process of replica write operations in the aforementioned multi-replica storage system involves multiple network round trips across nodes, resulting in significant network communication overhead and high write latency. Therefore, there is an urgent need for a multi-replica storage solution with low network communication overhead and low write latency. Summary of the Invention

[0005] In view of this, embodiments of this specification provide a multi-copy storage system. One or more embodiments of this specification also relate to another multi-copy storage system, a multi-copy storage method, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0006] According to a first aspect of the embodiments of this specification, a multi-replica storage system is provided, including a request node, a master node and a slave node, and a shared memory pool based on a high-speed interconnect protocol, the shared memory pool including a metadata area, a slave node feedback area and a data block area; The request node is configured to acquire a target write request, write the target write request into the metadata area and data block area of ​​the shared memory pool, and pass the target write request to the master node through the shared memory pool; The master node is configured to access the shared memory pool to obtain the target write request, write the target write request to the first local storage unit, and pass the target write request to the slave node through the shared memory pool; The slave node is configured to access the shared memory pool to obtain the target write request, write the target write request to the second local storage unit, and write a feedback signal to the slave node feedback area of ​​the shared memory pool.

[0007] According to a second aspect of the embodiments of this specification, a multi-replica storage system is provided, including a request node, a master node, and a slave node, as well as a shared memory pool constructed by a switch connected to a memory server based on a high-speed interconnect protocol; the request node, master node, and slave node are configured with a distributed service, a node-side interconnect manager, and a local storage unit. The distributed service of the request node is configured to access the switch based on the high-speed interconnection protocol through the node-side interconnection manager, access the corresponding memory server through the switch, and write the target write request into the shared memory pool. The distributed service of the master node and slave node is configured to access the switch based on the high-speed interconnection protocol through the node-side interconnection manager, access the corresponding memory server through the switch, obtain the target write request from the shared memory pool, and write it to the corresponding local storage unit.

[0008] According to a third aspect of the embodiments of this specification, a multi-replica storage method is provided, applied to a master node in a multi-replica storage system, the multi-replica storage system further comprising requesting nodes, slave nodes, and a shared memory pool based on a high-speed interconnect protocol, the shared memory pool comprising a metadata area, a slave node feedback area, and a data block area; the method includes: Access the shared memory pool to obtain a target write request, wherein the target write request is the requesting node writing to the metadata area and data block area of ​​the shared memory pool, and is transmitted to the master node through the shared memory pool; The target write request is written to the first local storage unit, and the target write request is passed to the slave node through the shared memory pool, so that the slave node can access the shared memory pool to obtain the target write request, write the target write request to the second local storage unit, and write a feedback signal to the slave node feedback area of ​​the shared memory pool.

[0009] According to a fourth aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above-described multiple copy storage method are implemented.

[0010] According to a fifth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the above-described multiple copy storage method.

[0011] According to a sixth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described multiple copy storage method.

[0012] This specification provides a multi-replica storage system in one embodiment, including a request node, a master node, and slave nodes, as well as a shared memory pool based on a high-speed interconnect protocol. The shared memory pool includes a metadata area, a slave node feedback area, and a data block area. The request node obtains a target write request, writes the target write request to the metadata area and data block area of ​​the shared memory pool, and transmits the target write request to the master node through the shared memory pool. The master node accesses the shared memory pool to obtain the target write request, writes the target write request to a first local storage unit, and transmits the target write request to the slave node through the shared memory pool. The slave node accesses the shared memory pool to obtain the target write request, writes the target write request to a second local storage unit, and writes a feedback signal to the slave node feedback area of ​​the shared memory pool. Thus, a system architecture is provided in which request nodes, master nodes, and slave nodes all directly access the shared memory pool and perform multi-replica persistent writing of target write requests based on the shared memory pool. The request node writes the target write request to the shared memory pool, and the master node and each slave node directly access the shared memory pool to read the target write request and write it to the local storage unit for local persistence. The single producer and multiple consumer access mode based on the shared memory pool replaces multiple network round trips across nodes, saving cross-node network overhead and reducing write latency. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the system architecture of a multi-replica storage system provided in one embodiment of this specification; Figure 2 This is a schematic diagram of the architecture of any node in a multi-replica storage system provided in one embodiment of this specification; Figure 3 This is a schematic diagram of a shared memory pool provided in one embodiment of this specification; Figure 4 This is a schematic diagram of the system architecture of another multi-replica storage system provided in one embodiment of this specification; Figure 5 This is a flowchart illustrating a multi-copy storage method provided in one embodiment of this specification; Figure 6 This is a schematic diagram illustrating the copy writing process of a multi-copy storage system according to one embodiment of this specification; Figure 7 This is a schematic diagram of a write bandwidth test result provided in one embodiment of this specification; Figure 8 This is a schematic diagram of a write latency test result provided in one embodiment of this specification; Figure 9 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0014] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0015] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0016] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0017] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0018] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0019] Multi-Replica Storage System (MRS): A distributed storage architecture that achieves high availability and fault tolerance by storing multiple complete copies of the same data on multiple physical nodes. When a node fails, the system can seamlessly switch to other replicas to continue providing services. It is widely used in distributed databases, distributed file systems, etc.

[0020] CXL (Compute Express Link) is a high-speed interconnect protocol based on the PCIe physical layer. It supports cache-consistent low-latency communication between CPUs (Central Processing Units) and devices (such as memory expansion cards, GPUs (Graphics Processing Units), and accelerators). It includes three sub-protocols: CXL.io (Input / Output, I / O), CXL.cache (cache consistency), and CXL.mem (memory semantic access). It aims to achieve memory pooling, resource decoupling, and heterogeneous computing collaboration.

[0021] RDMA (Remote Direct Memory Access) is a network technology that allows one computer (the initiating computer) to directly read or write memory data to another computer (the target computer) over a network without the involvement of the target operating system kernel or the need for data copying between the CPUs of the two communicating parties. This reduces communication latency (in microseconds) and CPU overhead. It is commonly used in high-performance computing, AI (artificial intelligence) clusters, and distributed storage, but requires explicit programming by the application and does not provide cache coherency.

[0022] DMA (Direct Memory Access) is an I / O mechanism in computer systems that allows certain hardware subsystems (such as disk controllers, network cards, graphics cards, and other peripheral devices) to directly read and write data to the system's main memory without continuous processor intervention. Without DMA, the CPU would need to handle every byte of data transfer manually. The DMA controller takes over this intensive task, notifying the CPU only at the start and end of data transfers, thus freeing the CPU from heavy I / O operations and allowing it to focus on computational tasks.

[0023] NT-Store (Non-Temporal Store): This is an optimized operation for CPU memory writes. It bypasses the cache by using special instructions (such as MOVNTI) to write data directly to main memory or device memory, avoiding the CPU cache from being occupied by writing large amounts of data. It can significantly improve throughput and reduce cache coherency overhead in scenarios such as large data writes, persistent memory, and CXL.

[0024] It should be noted that the write process in a multi-replica storage system, including multiple network round trips across nodes and the persistence of copies to local storage devices, constitutes a latency-sensitive I / O service path. Therefore, optimizing write latency is one of the goals of a multi-replica storage system. To achieve this goal, one implementation uses Remote Direct Memory Access (RDMA) to replace the network protocol stack (e.g., TCP / IP), thereby accelerating network communication between the central node and followers for request propagation and acknowledgment. Specifically, in an RDMA-based multi-replica storage system, the client initiates a write request by directly transferring the copy to the central node's memory through RDMA operations (i.e., RDMA Send or Write). The central node then forwards the request to followers using RDMA operations. Once the copy is persisted, each follower sends an acknowledgment to the central node via RDMA.

[0025] Although RDMA offers certain performance advantages over the network protocol stack, the overhead of RDMA operations is gradually becoming a major bottleneck as storage I / O latency decreases dramatically (e.g., 12 µs for a 4 KB sequential write in PCIe 5.0 SSDs (Solid-State Drives)). In tests, the propagation of 4 KB of data took approximately 4.65 µs, while a 64-byte acknowledgment round trip took 2.81 µs. Therefore, the cumulative latency of multiple network round trips dominates the end-to-end write latency, and this growing performance issue highlights the need for latency optimization.

[0026] As can be seen from the above, the rapid development of storage hardware has made microsecond-level RDMA round-trip communication a bottleneck for write operations. Therefore, one embodiment of this specification provides a multi-replica storage system based on the CXL protocol, which utilizes the sub-microsecond communication latency of CXL to replace the RDMA-based communication model with a model based on CXL native shared memory. Specifically, the multi-replica storage system makes data visible to all nodes through simple memory storage operations, thereby achieving low-latency replication. It employs a cache-line aligned metadata layout, transmitting metadata via lightweight, cache-line-sized memory loading and storage operations, reducing network round-trip overhead. Furthermore, to address the limitations of CXL hardware, it introduces the single-writer principle and a decoupled communication-worker thread model. This model separates batch request sending from independent request acknowledgment, effectively avoiding false sharing and contention, and balancing high throughput with low latency. In addition, in the absence of cross-node consistency, it uses DMA-based data persistence and NT-Store to bypass CPU caching, thereby reducing software synchronization overhead and implementing a low-latency multi-replica storage system based on the CXL shared memory pool. Low-latency end-to-end writing is achieved through direct memory operations and DMA persistence.

[0027] This specification provides a multi-replica storage system, and also relates to another multi-replica storage system, a multi-replica storage method, a computing device, and a computer-readable storage medium, which will be described in detail in the following embodiments.

[0028] See Figure 1 , Figure 1 A schematic diagram of a system architecture for a multi-replica storage system according to an embodiment of this specification is shown, such as... Figure 1 As shown, the multi-replica storage system includes a request node 102, a master node 104, and a slave node 106, as well as a shared memory pool 108 based on a high-speed interconnect protocol. The shared memory pool 108 includes a metadata area 1082, a slave node feedback area 1084, and a data block area 1086. The request node 102 is configured to obtain the target write request, write the target write request to the metadata area 1082 and the data block area 1086 of the shared memory pool 108, and pass the target write request to the master node 104 through the shared memory pool 108. The master node 104 is configured to access the shared memory pool 108 to obtain the target write request, write the target write request to the first local storage unit 1042, and pass the target write request to the slave node 106 through the shared memory pool 108. The slave node 106 is configured to access the shared memory pool 108 to obtain the target write request, write the target write request to the second local storage unit 1062, and write a feedback signal to the slave node feedback area 1084 of the shared memory pool 108.

[0029] Specifically, a multi-replica storage system refers to a distributed storage architecture that ensures high data reliability and system availability by storing multiple copies of data across different nodes. It can be applied to distributed databases, distributed file systems, and more. A request node is the initiator of read / write requests in a multi-replica storage system; it acts as the client. The master node is the coordination and control center of the system, responsible for determining the placement of data replicas (e.g., selecting which slave nodes to use) and coordinating the order and consistency of the multi-replica write process. The master node can also store copies of write requests, ensuring its own state is recoverable during the coordination process. A slave node is the node responsible for persistent data storage in a multi-replica storage system. A multi-replica storage system can have multiple slave nodes, each with independent physical storage resources. They are responsible for writing received data to their local disk and maintaining data replicas; they act as followers in the system.

[0030] A shared memory pool based on a high-speed interconnect protocol is an architecture that aggregates the memory resources of multiple physical nodes into a logically unified memory pool that can be accessed concurrently by multiple hosts through low-latency, high-bandwidth interconnect technologies (such as CXL, NVLink, or InfiniBand). It is a shared memory area that provides data between different nodes in a multi-replica storage system. It acts as a bridge for asynchronous communication between nodes, enabling different computing nodes to directly read and write shared data at near-local memory speeds, achieving cache consistency (such as CXL.cache) or efficient remote access (such as RDMA), thereby avoiding the overhead of network protocol stacks.

[0031] The metadata area refers to a structured memory area located in the shared memory pool that stores data request description information and control instructions; that is, the area that stores the metadata of the target write request. The data block area refers to a memory area located in the shared memory pool that stores the actual data of the write request. The data is stored here in the form of raw byte streams or blocks, and its location and size can be described by the metadata.

[0032] The slave node feedback area refers to the memory area located in the shared memory pool used to store the local persistence results of each slave node. It is a follower ACK (Acknowledgement) area. Its data structure is usually indexed by a combination of the slave node identifier and the request identifier. Each entry records the local persistence status of the corresponding slave node for a specific request (such as success, failure, progress, etc.).

[0033] The first / second local storage unit refers to the non-volatile storage media (such as SSD (Solid-State Drive), HDD (Hard Disk Drive)) local to the master node and slave node, respectively, used to achieve local data persistence.

[0034] Feedback signals refer to the status information written by the slave node to the slave node's feedback area. They are used to indicate the local persistence result of a specific request. They can typically include information such as request identifier, slave node identifier, status code (such as success / failure), and timestamp. They are the medium for achieving distributed transaction state synchronization.

[0035] In the embodiments described in this specification, a multi-replica storage system architecture built on a shared memory pool is introduced, such as... Figure 1 As shown, its hardware infrastructure consists of multiple computing nodes, including requesting nodes (i.e., clients), master nodes, and slave nodes (i.e., followers).

[0036] In actual implementation, both the master and slave nodes, in addition to their own CPUs and main memory, are equipped with a set of high-speed NVMe SSDs (Non-Volatile Memory Express Solid-State Drives) as persistent backends for the replication process, i.e., local storage units. The local storage unit of the master node is called the first local storage unit, and the local storage unit of the slave node is called the second local storage unit. Requesting nodes, master nodes, and slave nodes interact through a shared memory pool built in the multi-replica storage system.

[0037] Specifically, in a multi-replica storage system, specific roles can be assigned to each compute node through software, designating it as a requesting node (client), master node, or slave node (follower). Role assignment is carried out during the system initialization phase based on configuration files or preset parameters. One feasible configuration strategy is random assignment, or assignment based on the node performance of each compute node.

[0038] It should be noted that, in order to meet the requirements of a multi-replica storage system, the three access phases of the multi-replica storage workflow are configured, and the network fan-out of RDMA is replaced by a single-producer, multi-consumer access mode (i.e., "write once, read many times") for data transfer during replication operations.

[0039] In practice, the master node no longer explicitly transmits write requests received from the requesting nodes to the slave nodes. Instead, when a requesting node sends a write request, it performs a separate write operation to the shared memory pool. After the write operation, the data from the write request and the request metadata are visible to all other nodes, including the master and slave nodes. The master and slave nodes then read the requested data from the shared memory pool. Thus, the fan-out replication process required in an RDMA-based multi-replica storage system is replaced by an efficient memory write.

[0040] In one optional implementation of this embodiment, the shared memory pool 108 is constructed by connecting the memory server to a switch based on a high-speed interconnect protocol. Each node in the multi-replica storage system is configured with a node-side interconnect manager. The node-side interconnect manager is used for the corresponding node to access the shared memory pool 108 and map the memory address of the shared memory pool 108 to the local storage unit 1042 / 1062 on the node side.

[0041] Specifically, a high-speed interconnect switch refers to a high-speed switching device that supports cache coherence protocols (such as CXL or CCIX (CacheCoherent Interconnect for Accelerators)). It allows multiple processors (or nodes) to jointly access shared memory resources in a way that maintains memory data consistency, ensuring that modifications to memory by any node are promptly detected by other nodes. A memory server refers to an independent server node that provides memory capacity. Its main hardware components are large-capacity DRAM or persistent memory, and it connects to the system via cache coherence interconnect to provide memory pool services.

[0042] The node-side interconnect manager is a hardware module or firmware / driver layer software integrated within each node (including requesting nodes, master nodes, and slave nodes) of a multi-replica storage system. It manages the connection between the node and the cache-coherent interconnect switch and performs critical address translation and mapping operations, enabling seamless mapping and integration of remote shared memory pool address spaces into the node's local memory address space. For example, in the case of the CXL cache-coherent protocol, this node-side interconnect manager is the CXL root complex. The CXL root complex is the control and initiation component in CXL; essentially, it is a PCIe / CXL controller integrated within the CPU that supports the CXL protocol. It is responsible for initiating transactions, managing the topology, configuring devices, and acting as the "root" of the entire CXL hierarchy.

[0043] In practice, a globally unified, low-latency, and highly consistent shared memory pool is constructed using a hardware architecture based on cache-coherent interconnects (such as CXL), serving as the data flow interaction channel. Specifically, multiple memory servers connect to one or more cache-coherent interconnect switches through their supported cache-coherent interfaces (such as CXL.mem). These memory servers contribute their large-capacity DRAM or persistent memory, which is collaboratively managed by the switches and system software to form a logically continuous and physically distributed unified shared memory address space. This address space is transparent to other nodes within the system. Each compute node (requesting, scheduling, and slave nodes) is equipped with a node-side interconnect manager, which implements access to the shared memory pool and address translation at the hardware level.

[0044] Example, Figure 2 This is a schematic diagram of the architecture of any node in a multi-replica storage system provided in one embodiment of this specification, such as... Figure 2 As shown, the structure of any node (such as a requesting node (client), master node, or slave node (follower)) includes at least a CPU (controller), memory, and a storage unit (SSD). The CPU is configured with a node-side interconnect manager (such as a CXL root complex). The CXL root complex is used to access the shared memory pool in the multi-replica storage system, obtain replica data, and write it to the local storage unit (SSD) to achieve local persistence.

[0045] In the embodiments described in this specification, the requesting node, master node, and slave node are interconnected through a switch based on a high-speed interconnect protocol. This switch is connected to a memory server to form a shared memory pool. The node-side interconnect manager is the basic control unit that implements the hardware communication link in the multi-replica storage system. At the underlying physical layer, the node-side interconnect manager is responsible for initiating transactions, converting protocols, and executing operations that allow each node to access the shared memory pool (including establishing memory mapping, data reading and writing, etc.). It is the hardware foundation that supports the implementation of "direct memory semantic access," allowing each node to map the shared memory pool to its address space and supporting direct memory semantic access. Through hardware-level memory pooling and consistent interconnect, the network-based discrete data transmission in the distributed system is updated to direct memory access based on a unified memory address space.

[0046] In one optional implementation of this embodiment, the request identifier of the write request can be transmitted between the request node, the master node, and the slave node based on the communication queue. That is, the shared memory pool 108 also includes a communication queue, which is a queue of request identifiers to be transmitted stored in the circular buffer of the shared memory pool 108. The request node 102 is further configured to write the target write request to the metadata area and data block area of ​​the shared memory pool 108, and to transmit the target request identifier of the target write request to the master node 104 through the communication queue. The master node 104 is further configured to obtain the target request identifier by accessing the communication queue of the shared memory pool 108; access the metadata area and data block area of ​​the shared memory pool 108 based on the target request identifier to obtain the target write request; write the target write request to the first local storage unit 1042; and transmit the target request identifier to the slave node 106 through the communication queue of the shared memory pool 108. The slave node 106 is further configured to obtain the target request identifier by accessing the communication queue of the shared memory pool 108; access the metadata area and data block area of ​​the shared memory pool 108 based on the target request identifier to obtain the target write request, and write the target write request to the second local storage unit 1062.

[0047] Specifically, the communication queue refers to the ring buffer located in the shared memory pool, which is used to store and transmit request identifiers of multiple write requests in an orderly manner. It is a notification and control channel for transmitting work instructions between nodes, and it does not transmit data itself, but only the request identifiers.

[0048] The metadata area and data block area refer to the memory areas in the shared memory pool used to store complete data request content (such as the data block of the target write request, metadata, etc.). It is the data channel for transmitting the data body between nodes.

[0049] A request identifier is a unique identifier (such as a pointer, index ID, or transaction ID) used to represent a specific write request in the communication queue. Nodes obtain and locate the complete request data to be processed in the metadata and data block areas by passing and retrieving this request identifier.

[0050] A ring buffer is a fixed-size, first-in-first-out (FIFO) data structure that uses a contiguous block of memory to simulate a queue. It achieves circular writing and reading of data by maintaining producer and consumer pointers, avoiding data movement and serving as a high-performance inter-process communication queue.

[0051] In practice, when a requesting node needs to send a new write request, it collects the corresponding request identifiers and writes them to the communication queue in a dedicated circular buffer within the shared memory pool. The receiving nodes (master and slave nodes) can poll the tail pointer of this communication queue to detect new write requests. Upon detecting a new write request, they copy the request identifier to their local buffer for subsequent local persistent writing.

[0052] It should be noted that the ring buffer, built upon contiguous memory space, exhibits excellent spatial locality. This allows the CPU to efficiently utilize the cache prefetch mechanism during read and write operations, significantly reducing cache miss rates and maximizing the utilization of the CXL bus bandwidth for high memory access efficiency. Furthermore, as a fixed-size, cyclically used data structure, the ring buffer eliminates the need for frequent memory allocation and deallocation after initialization, avoiding system jitter and latency caused by dynamic memory management and avoiding the overhead of dynamic memory allocation. Moreover, the ring buffer supports lock-free operations. Combined with the subsequent SPSC (Single Producer, Single Consumer) mode, the ring buffer can achieve thread synchronization through simple head / tail pointer atomic operations without using expensive mutexes. In a cross-node CXL latency-sensitive environment, this lock-free design can achieve high throughput and low latency communication. Therefore, the embodiments in this specification use a ring buffer as a communication queue to transmit write request identifiers between nodes.

[0053] In the embodiments described in this specification, the communication queue is responsible for transmitting lightweight request identifiers (control flow) to guide the workflow. The lightweight request identifiers are transmitted through an efficient circular buffer queue, achieving nanosecond-level synchronization of work instructions and task scheduling. Meanwhile, the heavyweight request data is statically stored in the storage area for each node to access directly as needed, eliminating the overhead of multiple serializations and copies when data flows between nodes. By adopting an architecture that separates control flow and data flow, the communication between nodes is optimized.

[0054] In one optional implementation of this embodiment, the communication queue includes a first shared communication queue shared between the requesting node 102 and the master node 104, and a second shared communication queue between the master node 104 and each slave node 106. The request node 102 is further configured to write the target request identifier into the first shared communication queue, wherein the first shared communication queue is used to transmit the target request identifier between the request node 102 and the master node 104. The master node 104 is further configured to obtain the target request identifier by accessing the first shared communication queue of the shared memory pool 108; and to write the target request identifier into each of the second shared communication queues of the shared memory pool 108, wherein any second shared communication queue is used to transmit the target request identifier between the master node 104 and the corresponding slave node 106. The slave node 106 is further configured to access the second shared communication queue corresponding to the slave node 106 in the shared memory pool 108 to obtain the target request identifier.

[0055] Specifically, the first shared communication queue refers to a circular buffer queue located in the shared memory pool, exclusively shared by the requesting node and the master node. It is specifically used to transmit request identifiers unidirectionally between the two nodes. In other words, the first shared communication queue is a [requesting node-master node shared queue] to transmit request identifiers between the requesting node and the master node, thus forming the first-level pipeline for task distribution.

[0056] The second shared communication queue refers to a ring-shaped buffer queue located in the shared memory pool, exclusively shared by the master node and each slave node. Each slave node has its own independent second shared communication queue, used to implement parallel, point-to-point task instruction transmission from the master node to different slave nodes. That is, there is a corresponding second shared communication queue for each master node and slave node. For example, the second shared communication queue is [Master Node-Slave Node 1 Shared Queue] to transmit request identifiers between the master node and slave node 1, [Master Node-Slave Node 2 Shared Queue] to transmit request identifiers between the master node and slave node 2, ..., [Master Node-Slave Node n Shared Queue] to transmit request identifiers between the master node and slave node n, which constitutes the second-level parallel pipeline for data persistence.

[0057] It should be noted that the communication queue is a single producer single consumer queue (SPSC queue), meaning that each communication queue is shared by only one request sender and one receiver to implement the single writer principle, that is, for each communication queue, only one user can write the request identifier to the queue.

[0058] In practice, to ensure efficient lock-free communication, a defined communication relationship (such as from a requesting node to the master node, or from the master node to any slave node) is established through a dedicated shared communication queue. Specifically, the requesting node writes the target request identifier of the target write request into the first shared communication queue dedicated to itself and the master node. The master node continuously listens to this first shared communication queue and retrieves the target request identifiers of the target write requests written by the requesting node in sequence.

[0059] The master node retrieves the target request identifier from the first shared communication queue and reads the complete request content from the storage area based on the identifier. Then, according to the replication strategy, the master node determines which slave nodes should participate in this write operation. The same target request identifier is written in parallel to the second shared communication queue corresponding to each slave node, achieving one-to-many task broadcasting. Each slave node has an independent command channel, preventing them from blocking each other. Each slave node continuously listens to its own and the master node's second shared communication queue. When it obtains the target request identifier from its own second shared communication queue, it becomes aware of the replica write task assigned to it. The slave node can then access the storage area in the shared memory pool based on the target request identifier, read the complete request data, and write it to its own second local storage unit, achieving local persistent storage.

[0060] For example, taking a multi-replica storage system containing a request node, a master node, and two slave nodes as an example, the transmission process of the target request identifier for a target write request is as follows: First, the request node writes the generated target request identifier into the first shared communication queue shared with the master node; then, the master node reads the target request identifier from the first shared communication queue and writes it into two other independent second shared communication queues, one dedicated to connecting the master node and slave node A, and the other dedicated to connecting the master node and slave node B; finally, the two slave nodes read the request identifier from their respective second shared communication queues.

[0061] In the embodiments described in this specification, each pair of nodes that need to interact uses an independent shared communication queue. Through this cascading method, the target request identifier can be transmitted sequentially from the requesting node to the master node and each slave node in an independent point-to-point channel without conflict, triggering subsequent processing. By establishing two levels of dedicated shared communication queues, a clear hierarchical and highly parallel architecture is constructed, avoiding head-of-queue blocking between slave nodes, improving the overall throughput and task processing concurrency of the multi-replica storage system, and enhancing the scalability and stability of the system through point-to-point shared queue isolation, so that the performance fluctuation or latency of any node will not spread and affect the normal operation of other nodes.

[0062] In an optional implementation of this embodiment, the request node 102 is further configured to write multiple target request identifiers into the first shared communication queue in batches. After writing, a first memory barrier instruction is executed for each target request identifier written in batches in the first shared communication queue. The first memory barrier instruction is used to make each target request identifier visible to the master node 104. The master node 104 is further configured to write multiple target request identifiers into the second shared communication queue in batches. After writing, a second memory barrier instruction is executed for each target request identifier written in batches in the second shared communication queue. The second memory barrier instruction is used to make each target request identifier visible to the slave node 106.

[0063] Specifically, memory barrier instructions are CPU-level synchronization instructions (such as mfence and sfence in x86 architecture, or dmb in ARM architecture). They ensure that all memory write operations (stores) before the instruction must be globally visible and complete (i.e., the data has actually arrived in memory or the cache coherence domain of the other CPU) before memory read or write operations after the instruction can be executed. Their core function is to prevent memory operation reordering and ensure write visibility.

[0064] Batch writing refers to the producer node (request node or master node) writing multiple request identifiers as a batch into the circular buffer of the communication queue, and then executing a memory barrier instruction only once. This changes the pattern of "write one identifier - execute one memory barrier instruction".

[0065] Write visibility refers to the ability of a CPU core to correctly and timely observe the result of a write operation when the data written by one CPU core is temporarily stored in its own cache in a multi-core, multi-CPU system. Memory barriers are the hardware mechanisms that enable this cross-core, cross-node visibility.

[0066] It should be noted that, in order to improve throughput, the sender can adopt a batch processing strategy. Before initiating a single, batch transmission to the communication queue, multiple request identifiers can be collected. After writing the multiple request identifiers into the data area of ​​the communication queue, a memory barrier instruction can be executed to ensure that this batch of data is immediately visible to the receiver.

[0067] In actual implementation, the requesting node aggregates multiple target request identifiers and writes them in batches to the first shared communication queue. Then, it executes the first memory barrier instruction to make the batch of writes globally visible, thereby notifying the master node at once. After the master node obtains and processes these requests in batches, it writes multiple target request identifiers in parallel in batches to the second shared communication queues corresponding to each slave node. After each batch write, it executes an independent memory barrier instruction to ensure that the slave nodes can promptly and reliably perceive their own task batches.

[0068] In the embodiments described in this specification, the entire batch of write requests is made visible to the receiver through a single memory barrier operation. This batch processing mechanism allows the high fixed cost of a single memory barrier to be amortized across the entire batch of requests, thereby significantly reducing the average synchronization overhead and achieving a high-throughput communication path. The receiver is accordingly designed to read and process these incoming request identifiers in batches to maintain high efficiency.

[0069] In one optional implementation of this embodiment, The request node 102 is further configured to write the target control information of the target write request into the metadata area 1802, and write the target request data of the target write request into the data block area 1086. The master node 104 is further configured to access the metadata area 1082 based on the target request identifier, obtain the target control information of the target write request; access the data block area 1086 based on the target control information, copy the target request data of the data block area 1086 to obtain the corresponding target replica data, and write the target replica data into the first local storage unit 1042. The slave node 106 is further configured to access the metadata area 1082 based on the target request identifier to obtain the target control information of the target write request; access the data block area 1086 based on the target control information, copy the target request data of the data block area 1086 to obtain the corresponding target replica data, and write the target replica data into the second local storage unit 1062.

[0070] Specifically, target control information refers to the metadata instance corresponding to the target write request. It is a specific entry or structure in the metadata area that contains the command information required to process the request. Target request data refers to the actual user data content in the target write request that needs to be persistently stored. It is stored in one or more contiguous memory blocks in the data block area.

[0071] Target replica data refers to a copy of the target requested data that the master or slave node reads from the data block area and copies to its local memory.

[0072] It's important to note that, to manage the exchange of control messages, the control information for write requests can be stored in a separate area of ​​a shared memory pool, distinct from the data blocks—the metadata area. Control information (such as status flags and address pointers) is typically small (e.g., the size of a cache line) and requires frequent access (e.g., polling for status). If stored alongside large data blocks, the CPU would have to load adjacent large data blocks into the cache when reading control information, wasting cache space and increasing inefficient memory bandwidth usage. Separating it allows the CPU to complete status checks by loading only the smaller control fields. Furthermore, cache coherence protocols (such as CXL) support byte-level or cache line-level load / store instructions, allowing the system to directly and quickly obtain signaling using lightweight CPU instructions, without navigating complex protocol stacks or handling irrelevant data payloads.

[0073] In actual implementation, the requesting node first stores the actual data payload of the write request into the data block area, and then writes its description information (such as data pointer, length, etc.) as target control information into the metadata area. After the master node and slave node obtain the target request identifier from their respective queues, they first access the metadata area to obtain the target control information, and then locate and read the target request data in the data block area according to the guidance therein, copy it to the local memory to form the target replica data, and write it into their respective local storage units.

[0074] It should be noted that by physically separating control information from actual data, latency-sensitive message delivery is ensured. For example, sending a new request or confirming completion is performed through lightweight memory load / store instructions, rather than heavyweight network transmissions. This fully leverages the low-latency characteristics of the shared memory pool for fine-grained data access.

[0075] In the embodiments of this specification, by dividing the storage area of ​​the shared memory pool into a metadata area and a data block area, an efficient data processing architecture is constructed, realizing the separate management of control information and data body, improving cache hit rate and processing speed, improving cache efficiency, avoiding cache pollution, and reducing control path latency, thus leveraging the fine-grained access advantages of the shared memory pool based on the cache consistency protocol.

[0076] In one optional embodiment of this example, the slave node feedback area 1084 is used to store the feedback signals corresponding to each slave node 106, and the feedback signals are based on the slave node identifier and the request identifier index; The slave node 106 is further configured to write the target replica data into the second local storage unit 1062, and then write the feedback signal corresponding to the slave node identifier and the target request identifier of the slave node 106 into the slave node feedback area 1084. The master node 104 is further configured to, after polling and determining that the requesting node 102 has written the target write request to the shared memory pool 108, switch to polling the slave node feedback area 1084. If it is determined that the feedback signal written in the slave node feedback area 1084 meets the replica storage constraint conditions, switch to write mode and update the target control information of the target request identifier in the metadata area 1082. The request node 102 is further configured to poll the target control information of the target request identifier in the metadata area 1082, and determine the writing process of the target write request based on the target control information.

[0077] Specifically, the slave node feedback area refers to the memory area located in the shared memory pool that is used to store the local persistence results of each slave node. It is a follower ACK (Acknowledgement) area. Its data structure is usually indexed by a combination of the slave node identifier and the request identifier. Each entry records the local persistence status of the corresponding slave node for a specific request (such as success, failure, progress, etc.).

[0078] Feedback signals refer to the status information written by the slave node to the slave node's feedback area. They are used to indicate the local persistence result of a specific request. They can typically include information such as request identifier, slave node identifier, status code (such as success / failure), and timestamp. They are the medium for achieving distributed transaction state synchronization.

[0079] Replica storage constraints refer to the conditions corresponding to the number of replicas required for a multi-replica storage system to complete successfully. For example, in a 3-replica system, the constraint might be "at least 2 replicas succeed". Once this condition is met, the write operation can be considered successful at the system level.

[0080] Write mode is a working state of the master node, in which the master node can update the control information in the metadata area, indicating that the copy of the write request has been persistently stored on multiple slave nodes as needed.

[0081] In actual implementation, the request release writes the target request data to the data block area, the target control information to the metadata area, and pushes the target request identifier into the first shared communication queue. At this time, the target control information in the metadata is marked as "processing". After the master node obtains the target request identifier through the first shared communication queue, it reads the metadata to obtain the target control information. After obtaining and writing the target request data to the first local storage unit, it dispatches the task to each slave node through the second shared communication queue. Each slave node reads the target request data from the data block area of ​​the shared memory pool storage area and writes it to the second local storage unit. Immediately afterward, it writes the corresponding feedback signal to the slave node feedback area. Each slave node writes a signal containing the "success" / "failure" status at the index position corresponding to its own node identifier and the request identifier.

[0082] After dispatching the task, the master node switches from listening to the communication queue to polling the slave node feedback area. It continuously checks the feedback signals from each slave node corresponding to the target request identifier. When the number of successful responses meets the replica storage constraints (e.g., two successful responses out of three replicas), the master node switches to write mode, updates the target control information status of the write request in the metadata area to "committed," and may add metadata such as a list of successful replica nodes. The requesting node continuously polls the control information status of the write request in the metadata area. When it detects a change from "processing" to "committed," it confirms a successful write. If no status change is observed within a timeout, a retry or error handling process can be initiated.

[0083] It should be noted that the slave node feedback area includes multiple storage feedback slots. These storage feedback slots are independent, non-overlapping memory areas reserved for each slave node, used for writing feedback signals to the corresponding slave node. Each storage feedback slot is indexed by both the slave node identifier and the request identifier. Spatially, in scenarios where multiple slave nodes need to report status simultaneously (i.e., followers send ACKs), only the corresponding slave node has write access to a specific storage feedback slot. Even if all slave nodes need to write status at the same time, they write to their respective dedicated storage feedback slots, thus ensuring that each storage feedback slot physically satisfies the "single-writer" principle.

[0084] Example, Figure 3 This is a schematic diagram of a shared memory pool structure provided in one embodiment of this specification, as shown below. Figure 3As shown, the shared memory pool includes a metadata area, a slave node feedback area (i.e., the follower ACK area), and a data block area. The metadata area stores target control information for target write requests, the slave node feedback area stores feedback signals from each slave node performing local persistent writes, and the data block area stores the specific target request data for the target write request. The slave node feedback area includes multiple storage feedback slots, each corresponding to one cache line. This storage feedback slot stores the feedback signal from the corresponding slave node for a specific request identifier, such as the write request identifier (i.e., request ID) and feedback status.

[0085] In the embodiments described in this specification, each slave node directly writes its local persistent write results to the slave node feedback area of ​​the shared memory pool in the form of feedback signals. This allows the master node to quickly obtain the replica status by accessing the slave node feedback area without polling each slave node, thus reducing coordination overhead. Furthermore, after the master node satisfies the replica constraints, it updates the metadata status. The requesting node can perceive the replica write results by polling the metadata. This simplifies the complex distributed state synchronization to accessing the shared memory pool, ensuring not only the consistency and high availability of multi-replica writes but also accelerating the state synchronization process through the shared memory pool, reducing network overhead and latency.

[0086] In one optional implementation of this embodiment, the metadata area 1082 includes multiple metadata slots, each metadata slot corresponding to a cache line for writing control information of a write request. The control information includes a write request identifier, a global request status, a logical block address, and a data block pointer. The request node 102 is further configured to write the target request data of the target write request into a free data block in the data block area 1086, and write the write request identifier, global request status, logical block address and data block pointer of the target write request into the target metadata slot in the metadata area 1082. After the target request data is written, the global request status in the target metadata slot is updated and switched to read mode. The global request status is used to transmit the request status of the target write request across nodes. The master node 104 is further configured to poll the target metadata slot of the metadata region 1082 based on the target request identifier. When the global request status in the target metadata slot indicates that the request node 102 has completed writing, the master node 104 obtains the logical block address and data block pointer of the target write request in the target metadata slot based on the target request identifier. Based on the logical block address and data block pointer of the target write request, the master node 104 obtains the target replica data and writes it into the first local storage unit 1042. The slave node 106 is further configured to access the target metadata slot of the metadata region 1082 based on the target request identifier, obtain the logical block address and data block pointer of the target write request, and obtain the target replica data and write it to the second local storage unit 1062 based on the logical block address and data block pointer of the target write request.

[0087] Specifically, a metadata slot refers to a fixed-size storage unit in the metadata area aligned to the cache line size. It is used to store complete control information for a single write request. Each slot corresponds to a cache line (usually 64 bytes), allowing the CPU to atomically read or write the entire block of control information in an efficient manner. A cache line is the smallest unit of data transfer between the CPU cache and memory (usually 64 bytes). Designing metadata slots to be cache line aligned avoids false sharing, which is the overhead of invalid cache synchronization caused when different CPU cores modify different variables in the same cache line.

[0088] The global request status refers to the request processing progress indicators (such as "Initial", "Data Preparation", "Data Ready", "Processing", "Completed", etc.) stored in the metadata slot, visible to all nodes and interpreted uniformly. This global request status is updated through atomic operations and is a signal for synchronizing the request lifecycle across nodes.

[0089] Logical block address (LBA) refers to the logical data address (such as LBA) from the perspective of the user or upper-level system. It identifies the logical location of data in the storage system, rather than the physical disk address, and is used by master and slave nodes to determine the final storage location of data.

[0090] A data block pointer is a memory address (or offset) that points to a specific data block in the data block region. Together with the logical block address, it forms complete information about "where the data is (memory pointer)" and "where it is stored (logical address)".

[0091] It's important to note that the logical organization of the shared memory pool is crucial for achieving correctness and high performance. Shared memory pools built on CXL switches do not provide hardware-managed cache consistency across nodes. This limitation means that data written to one node will not automatically become visible to another. Therefore, software needs to explicitly manage data visibility and prevent race conditions. Specifically, firstly, the control information for shared data structures (control information for individual write requests) should be aligned to a single CPU cache line (e.g., 64 bytes); secondly, updates to a given cache line should be performed as a single 64-byte write, prohibiting partial modifications. These two principles combined prevent false sharing, ensuring that different nodes do not simultaneously hold conflicting, partially updated views of the same cache line.

[0092] Furthermore, for concurrently accessed data and metadata, it is ensured that only one writer exists at a given time to avoid multi-writer contention. Specifically, the single-writer principle refers to write operations to the shared memory pool. This principle is ensured through a combination of temporal role switching and spatial region partitioning. Temporally, role switching logic controls the process. For example, during the writing of requested metadata and data to the shared memory pool by a requesting node, the requesting node holds the writer identity, while the master and slave nodes are in poller (read) mode, only performing read checks. Once the requesting node completes the write and updates the global request status (i.e., the flag) in the metadata slot, it switches to poller mode, waiting for the result, and no longer writes to that slot. The master node only switches to writer mode after read confirmation. There is no temporal overlap between the writer modes of the two nodes, thus avoiding the possibility of multiple parties writing to the same address simultaneously. Spatially, the single-writer principle is implemented through the storage feedback slots corresponding to each slave node in the aforementioned slave node feedback region.

[0093] Continuing with the previous example, such as Figure 3 As shown, the shared memory pool includes a metadata area, a follower ACK area, and a data block area. The metadata area is constructed as an array of metadata slots, where each metadata slot corresponds to a single write request, and its size and alignment are consistent with a single 64-byte cache line, ensuring that no two metadata slots will be located on the same cache line, thereby avoiding the false sharing problem.

[0094] Each metadata slot contains control information such as write request identifier (64 bits for global request ID), global request status (8 bits to describe the current processing status, such as ISSUED, COMPLETED), logical block address (LBA), and data block pointer (64 bits to point to the requested payload in the data area).

[0095] The slave node feedback area is designed to maintain the single-writer principle during the acknowledgment phase. Specifically, the slave node feedback area contains multiple cache line-sized storage feedback slots (i.e., ACK slots). Each metadata slot is associated with a set of storage feedback slots within this area, and each slave node corresponds to one storage feedback slot. Therefore, different slave nodes can concurrently send acknowledgments to the master node by writing different cache lines, thereby preventing write contention and false sharing issues.

[0096] Additionally, the data block region is a large, contiguous memory area used to store the payload of requested data. Logically, it is organized as an array of fixed-size data blocks, with each write request occupying one block within this region. These regions are statically allocated during system initialization within the shared memory pool. However, shared memory pool management can also include dynamic allocation and reclamation. This shared memory can be integrated with other high-performance shared memory managers. Dynamic allocation and reclamation refers to a management mechanism where, during system operation, the software flexibly requests new shared memory pool storage areas or releases idle shared memory pool storage areas to the operating system based on real-time load demands. The shared memory pool architecture and process based on CXL in this specification's embodiments are not limited by the underlying memory management method. A static allocation strategy (i.e., pre-allocating all memory regions) can be used during the initialization phase. However, since memory management and copying logic are decoupled by design, it can be extended to support dynamic memory allocation mechanisms to adapt to more complex scenarios without altering the copying workflow.

[0097] In the embodiments described in this specification, the control information for write requests is encapsulated in independent cache line units, enabling requesting nodes to complete metadata publication and state updates atomically, avoiding the false sharing problem under multi-core concurrent access. The master node can accurately trigger subsequent processing flows by polling the global request status, without relying on additional communication overhead. Slave nodes can directly read the ready metadata to obtain complete data location information. The cache line-aligned metadata slot design achieves atomicity and efficiency in cross-node state synchronization, significantly reducing inter-node communication latency and ensuring linear scalability in high-concurrency scenarios through a hardware-friendly memory layout.

[0098] In an optional implementation of this embodiment, the master node 104 is further configured to initiate a first asynchronous direct memory access operation based on the data block pointer and the logical block address, wherein the first asynchronous direct memory access operation is used to copy the target request data corresponding to the data block pointer in the data block area, obtain the target copy data and write it to the first local storage unit 1042 indicated by the logical block address; The slave node 106 is further configured to initiate a second asynchronous direct memory access operation based on the data block pointer and the logical block address. The second asynchronous direct memory access operation is used to copy the target requested data corresponding to the data block pointer in the data block area, obtain the target copy data, and write it to the second local storage unit 1062 indicated by the logical block address.

[0099] In actual implementation, when the worker threads of the master / slave nodes perform local persistence, they access the metadata area in the storage area of ​​the shared memory pool based on the request ID, find the metadata slot corresponding to the request ID, and record the data block pointer and the logical address of the SSD in the metadata slot. Based on the data block pointer, they find the corresponding data block in the corresponding data area and persist the data block to the SSD through DMA based on the logical address of the SSD.

[0100] Specifically, asynchronous direct memory access (ADI) refers to a master or slave node (the initiator) issuing a command to its local storage unit controller (such as an NVMe SSD controller) that does not require the CPU to wait continuously. The command instructs the node to directly access a data block region within the shared memory pool's storage area and copy the data to a specified logical address in the local storage unit. This operation is executed asynchronously by the hardware controller, allowing the CPU to continue processing other tasks immediately after initiation.

[0101] The first asynchronous direct memory access operation refers to a DMA operation initiated by the master node. Its operation object is the first local storage unit, the data source is the target requested data pointed to by the data block pointer in the shared memory pool, and the destination is the location in the first local storage unit indicated by the logical block address (such as the location used to record operation logs or metadata).

[0102] The second asynchronous direct memory access operation refers to a DMA operation initiated by the node. Its operation object is the second local storage unit, the data source is the target requested data pointed to by the data block pointer in the shared memory pool, and the destination is the physical storage location in the second local storage unit indicated by the logical block address.

[0103] In practice, the direct memory access (DMA) capabilities supported by cache coherency protocol switches (such as CXL2.0 switches) and local storage units (such as SSDs) can be leveraged to eliminate the expensive explicit data copy from the shared memory pool to local memory when preparing data blocks for persistence. Specifically, when a master or slave node is ready to persist a data block, it can submit an asynchronous I / O operation that specifies the source address in the shared memory pool (the address of the data block region indicated by the data block pointer) and the target logical block address on the local storage unit (SSD). Both are extracted from the metadata slots. The data transfer is then handled by hardware, copying the data from the data block region in the shared memory pool's storage region to the local storage unit without CPU intervention.

[0104] It should be noted that in one implementation, the master node's CPU needs to execute instructions to read data from the shared memory pool into the local DRAM (Dynamic Random-Access Memory), and then write it into the first local storage unit through a system call.

[0105] In the embodiments described in this specification, the master node only needs to send the first asynchronous direct memory access operation, and subsequent data transfer is automatically completed by the DMA hardware without the CPU's involvement in data movement. The same applies to the slave node, thereby significantly freeing up CPU computing resources and transferring the data movement task from the CPU to dedicated DMA hardware. This offloading eliminates a software-based memory copy, directly reducing request processing time and thus lowering end-to-end latency. Secondly, the master / slave node is immediately released after issuing asynchronous I / O, allowing it to continue processing other pending requests. This effectively overlaps high-latency storage I / O with inter-node communication, and significantly improves system throughput under high concurrency.

[0106] In one optional implementation of this embodiment, each node in the multi-replica storage system includes a communication thread and a worker thread; The request node 102 is further configured to obtain a target write request from the pending request queue through a first communication thread, allocate a corresponding first worker thread to the target write request through the first communication thread, write the target write request to the shared memory pool 108 through the first worker thread, and transmit the target write request to the master node 104 based on the shared memory pool 108 through the first communication thread. The master node 104 is further configured to access the shared memory pool 108 through a second communication thread, obtain the target write request, allocate a corresponding second working thread for the target write request through the second communication thread, write the target write request to the first local storage unit 1042 through the second working thread, and transmit the target write request to the slave node 106 based on the shared memory pool 108 through the second communication thread. The slave node 106 is configured to access the shared memory pool 108 through a third communication thread, obtain the target write request, allocate a corresponding third worker thread for the target write request through the third communication thread, and write the target write request to the second local storage unit 1062 through the third worker thread.

[0107] Specifically, a communication thread is a dedicated thread responsible for network communication or message passing between nodes. Its main responsibilities are to listen for external requests, read or write notification messages from the shared memory pool, and manage connections with other nodes. It is an I / O-intensive thread that handles lightweight control flow.

[0108] A worker thread is a dedicated thread responsible for data processing and computation tasks. Its main responsibilities are to perform CPU-intensive operations such as data reading and writing, computation, compression, and encryption, as well as access local storage units. It is a computationally intensive or storage I / O intensive thread that handles heavy data streams.

[0109] The first, second, and third communication threads refer to the communication threads running on the request node, master node, and slave node, respectively. They have similar functions but serve different nodes and objects. The first, second, and third worker threads refer to the worker threads allocated by the communication threads on the request node, master node, and slave node for a specific request, and which execute the processing tasks of that node.

[0110] The pending request queue is a memory queue located inside the request node, used to temporarily store write requests that have not yet been processed.

[0111] It should be noted that this multi-replica storage system eliminates the static binding between worker threads and metadata slots. Write requests are assigned to a dynamic metadata slot indexed by their request identifier, which is also stored in the metadata slot. When a worker thread accesses a metadata slot, it uses the stored request identifier to verify whether the metadata slot corresponds to the expected write request, thereby avoiding contention in a dynamic dispatch environment.

[0112] In actual implementation, the first communication thread of the request node continuously listens to the queue of pending requests and retrieves one or more target write requests currently pending. After retrieving a target write request from the queue, the first communication thread performs two actions in parallel. One action is that the first communication thread dynamically allocates or wakes up an idle first worker thread from the worker thread pool of the request node and allocates the context (or pointer) of the target write request to it. The allocated first worker thread begins to perform data operations, writing the target request data to the data block area of ​​the shared memory pool, writing the target control information to the metadata area, and updating the global request status in the metadata slot corresponding to the target request identifier in the metadata area. The other action is that the first communication thread writes the target request identifier of the target write request to the first shared communication queue in the shared memory pool, so as to pass the target request identifier to the master node through the first shared communication queue.

[0113] The master node's second communication thread continuously polls the first shared communication queue. When it obtains a new target request identifier, similar to the requesting node, the second communication thread also performs two actions in parallel. One action is: the second communication thread dynamically allocates or wakes up an idle second worker thread from the master node's worker thread pool, passes the target request identifier to the second worker thread, and the allocated second worker thread begins to perform data operations, accesses the metadata slot corresponding to the target request identifier in the metadata area of ​​the shared memory pool, obtains the logical address and data block pointer corresponding to the target request identifier, initiates an asynchronous direct memory access operation, and copies the target request data from the data block area of ​​the shared memory pool to the first local storage unit. The other action is: the second communication thread writes the target request identifier of the target request to the second shared communication queue corresponding to each slave node in the shared memory pool, so that the target request identifier can be passed to each slave node through the second shared communication queue corresponding to each slave node.

[0114] The third communication thread of any slave node continuously polls the corresponding second shared communication queue. When a new target request identifier is obtained, similar to the master node, the third communication thread dynamically allocates or wakes up an idle third worker thread from the slave node's worker thread pool, passes the target request identifier to the third worker thread, and the allocated third worker thread starts to perform data operations. It accesses the metadata slot corresponding to the target request identifier in the metadata area of ​​the shared memory pool, obtains the logical address and data block pointer corresponding to the target request identifier, initiates an asynchronous direct memory access operation, copies the target request data from the data block area of ​​the shared memory pool and writes it to the second local storage unit. After the writing is completed, the third worker thread of the slave node writes a feedback signal to the corresponding storage feedback slot in the slave node feedback area of ​​the shared memory pool.

[0115] In the embodiments described in this specification, based on high-throughput notification delivery and low-latency task execution, two conflicting responsibilities are assigned to different threads: a communication thread (IPC thread) and a worker thread. A decoupled communication-worker thread model is introduced to reduce the overhead of software polling.

[0116] In an optional implementation of this embodiment, the request node 102 is further configured to write the target request data of the target write request to the shared memory pool 108 through the first worker thread based on a non-cached storage operation, wherein the non-cached storage operation is an operation that bypasses the cache and directly writes data to memory.

[0117] Specifically, non-cached storage operation refers to a storage operation initiated by a CPU instruction that bypasses the CPU's cache levels and directly writes data into memory. Non-cached storage operation is an NT-Store instruction, which is a specific CPU instruction that implements non-cached storage operation.

[0118] It should be noted that if a normal write instruction is used, the data will be temporarily stored in the CPU's private cache. An explicit flush instruction (such as clflush) must be executed to push it to the shared memory pool for other nodes to read. Non-cache storage operations can instruct the CPU to skip the cache level and directly write the requested data and metadata to the storage area of ​​the shared memory pool.

[0119] In practice, to accelerate the writing of data and metadata to the shared memory pool, non-cached storage operations are used to directly write the data and metadata to the shared memory pool, bypassing the cache. Once the data is written to the shared memory pool, it is physically visible to each node, thus naturally eliminating the need for subsequent explicit cache flushing. Furthermore, although non-cached storage operations exist, they are usually only performance indicators for the processor. Therefore, to ensure that new data is retrieved from the shared memory pool in a single read operation, explicit cache line flushing before loading is still necessary.

[0120] In the embodiments described in this specification, when a requesting node writes to the shared memory pool, a non-cached storage operation is used to directly write the data of the write request and metadata to the shared memory pool, bypassing the cache, removing a source of delay from the data write path of the shared content, and improving the data write efficiency to the shared memory pool.

[0121] In one optional implementation of this embodiment, the worker threads in each node concurrently process multiple write requests and create a state machine for each write request. The state machine is used to implement asynchronous concurrent processing of multiple write requests within the node. The master node 104 is further configured to create a first state machine for the target write request through the second worker thread when the state machine queue of the second worker thread is not full. The first state machine is used to record the processing status of the second worker thread of the master node for the target write request. The slave node 106 is further configured to create a second state machine for the target write request through the third worker thread when the state machine queue of the third worker thread is not full. The second state machine is used to record the processing status of the third worker thread of the slave node for the target write request.

[0122] Specifically, a state machine is an abstract data model used to represent the states a write request may experience during its lifecycle, and / or the events that trigger state transitions. In program implementation, it is typically a data structure (such as a struct or object) that contains current state flags and context data (such as request ID, data pointer, progress information, etc.).

[0123] The first state machine refers to the state machine instance created by the second worker thread of the master node for the target write request. It manages the processing flow of the target write request within the master node. The states may include: initial, reading metadata from the shared memory pool, initiating the first asynchronous DMA, waiting for the first DMA to complete, dispatching to the second shared communication queue, waiting for replica ACK, updating metadata status, completion / error, etc.

[0124] The second state machine refers to the state machine instance created by the third worker thread of the slave node for the target write request. It manages the processing flow of the target write request within the slave node. The states may include: initial, reading metadata from the shared memory pool, initiating the second asynchronous DMA, waiting for the second DMA to complete, writing ACK to the feedback area of ​​the slave node, completion / error, etc.

[0125] A state machine queue is a fixed-size queue (usually a lock-free circular queue) maintained internally by each worker thread. It is used to store the state machine instances corresponding to the multiple write requests that the worker thread is currently processing concurrently. The "full" and "not full" status of the state machine queue is the mechanism for the worker thread to perform flow control.

[0126] It's important to note that the global request state in the metadata slot is global information stored in the metadata area of ​​the shared memory pool, used to synchronize the lifecycle of write requests (e.g., published, completed) across different nodes. In contrast, the state machine created by the worker thread for a write request is a control logic structure stored in the node's local private memory, used only within that node's worker thread to track the specific execution steps of the write request (e.g., waiting for local SSD writes, waiting for queue sending, etc.). The former is used for global coordination across nodes, while the latter is used for asynchronous concurrency control within a node.

[0127] Worker threads are responsible for executing the actual copy write logic. A worker thread dynamically retrieves request identifiers (i.e., metadata indexes) from a local queue populated by communication threads. This dynamic dispatching allows a single worker thread to concurrently handle a large number of write requests, each associated with a unique metadata slot. To efficiently handle multiple concurrent write requests, each worker thread maintains a lightweight state machine for each write request it processes. This state machine-based execution approach, rather than relying on a centralized scheduler like coroutines, is crucial for achieving low latency. It allows worker threads to proactively and directly poll their own request state machines. This localized polling method ensures that completion status is detected with minimal latency, avoiding the latency bottlenecks found in centralized polling. By striking a balance between the number of worker threads and the write request capacity of each worker thread, the system can achieve high concurrency while maintaining low latency.

[0128] In actual implementation, when the second / third worker thread receives the target write request from the corresponding second / third communication thread, it creates a corresponding state machine instance for the target write request if the state machine queue in local memory is not full (the master node creates the first state machine, and the slave node creates the second state machine) to ensure that the load is controllable. If the state machine queue is full, it waits for the currently processed write request to be completed and the corresponding state machine to be released before processing.

[0129] It should be noted that by introducing a state machine queue for worker threads and creating and maintaining a state machine for write requests, efficient multiplexing within a single worker thread is achieved, enabling non-blocking concurrent management of the lifecycle of multiple write requests. This builds a high-concurrency asynchronous processing engine and improves system throughput.

[0130] The multi-replica storage system provided in the embodiments of this specification allows requesting nodes, master nodes, and slave nodes to directly access a shared memory pool. Based on this shared memory pool, the system architecture directly performs multi-replica persistent writing of target write requests. The requesting node writes the target write request to the shared memory pool, and the master node and each slave node directly access the shared memory pool to read the target write request and write it to their local storage unit for local persistence. The single producer and multiple consumer access mode based on the shared memory pool replaces multiple network round trips across nodes, saving cross-node network overhead and reducing write latency.

[0131] See Figure 4 , Figure 4 A schematic diagram of the system architecture of another multi-replica storage system according to an embodiment of this specification is shown, such as... Figure 4As shown, the multi-replica storage system includes request nodes, master nodes, and slave nodes, as well as a shared memory pool constructed by connecting memory servers via switches based on high-speed interconnect protocols; the request nodes, master nodes, and slave nodes are configured with distributed services, node-side interconnect managers, and local storage units. Figure 4 (Taking the master node as an example) The distributed service of the request node is configured to access the switch based on the high-speed interconnection protocol through the node-side interconnection manager, access the corresponding memory server through the switch, and write the target write request into the shared memory pool. The distributed service of the master node and slave node is configured to access the switch based on the high-speed interconnection protocol through the node-side interconnection manager, access the corresponding memory server through the switch, obtain the target write request from the shared memory pool, and write it to the corresponding local storage unit.

[0132] In actual implementation, after the requesting node obtains the target write request, it accesses the switch based on the high-speed interconnect protocol through the node-side interconnect manager based on its distributed service, accesses the corresponding memory server through the switch, writes the target write request into the metadata area and data block area of ​​the shared memory pool, and then passes the target write request to the master node through the shared memory pool.

[0133] The master node's distributed service accesses the switch based on the high-speed interconnect protocol through its node-side interconnect manager, accesses the corresponding memory server through the switch, obtains the target write request from the shared memory pool, writes the target write request to the first local storage unit, and passes the target write request to the slave node through the shared memory pool.

[0134] The distributed service of the slave node accesses the switch based on the high-speed interconnect protocol through its node-side interconnect manager, accesses the corresponding memory server through the switch, obtains the target write request from the shared memory pool, writes the target write request to the second local storage unit, and writes a feedback signal to the slave node feedback area of ​​the shared memory pool.

[0135] The multi-replica storage system provided in the embodiments of this specification allows requesting nodes, master nodes, and slave nodes to directly access a shared memory pool built by a switch-connected memory server based on a high-speed interconnect protocol through their distributed services and node-side interconnect managers. Based on this shared memory pool, the system architecture directly performs multi-replica persistent writing of target write requests. The requesting node writes the target write request to the shared memory pool, and the master node and each slave node directly access the shared memory pool to read the target write request and write it to their local storage unit for local persistence. This single-producer, multi-consumer access mode based on the shared memory pool replaces multiple network round trips across nodes, saving cross-node network overhead and reducing write latency.

[0136] The above is an illustrative scheme of another multi-replica storage system according to this embodiment. It should be noted that the technical solution of this multi-replica storage system is similar to that described above. Figure 1 The technical solutions of the multi-replica storage system shown belong to the same concept. Details not described in detail in this multi-replica storage system technical solution can be found in the above-mentioned... Figure 1 The technical solution of the multi-replica storage system shown is described.

[0137] See Figure 5 , Figure 5 A flowchart of a multi-replica storage method according to an embodiment of this specification is shown, applied to the master node of a multi-replica storage system. The multi-replica storage system also includes requesting nodes, slave nodes, and a shared memory pool based on a high-speed interconnect protocol. The shared memory pool includes a metadata area, a slave node feedback area, and a data block area. The method specifically includes the following steps.

[0138] Step 502: Access the shared memory pool to obtain the target write request, wherein the target write request is the requesting node to write to the metadata area and data block area of ​​the shared memory pool, and is passed to the master node through the shared memory pool.

[0139] Step 504: Write the target write request to the first local storage unit, and pass the target write request to the slave node through the shared memory pool so that the slave node can access the shared memory pool to obtain the target write request, write the target write request to the second local storage unit, and write a feedback signal to the slave node feedback area of ​​the shared memory pool.

[0140] The above is an illustrative scheme of a multi-replica storage method according to this embodiment. It should be noted that the technical solution of this multi-replica storage method belongs to the same concept as the technical solution of the multi-replica storage system described above. For details not described in detail in the technical solution of this multi-replica storage method, please refer to the description of the technical solution of the multi-replica storage system described above.

[0141] The following is in conjunction with the appendix Figure 6 This section further explains the complete copy writing process of a multi-replica storage system. Figure 6 This diagram illustrates the replica write process of a multi-replica storage system according to an embodiment of this specification. The CXL shared memory pool includes a metadata area (each write request corresponds to a metadata slot, and a metadata slot includes a write request identifier, a global request status, a logical block address, and a data block pointer), a slave node feedback area (also known as the ACK area, where the slave node ID and the request ID jointly correspond to the ACK slot (storage feedback slot), and a storage feedback slot includes a write request identifier and a feedback status), and a data area (the specific data blocks).

[0142] The requesting node (client) sends write requests to a local queue for processing (where multiple write requests initiated by the requesting node can be temporarily stored).

[0143] The request node's communication thread (i.e., IPC thread) retrieves write request 1 (any one pending in the queue) from the queue and executes the two operations in parallel.

[0144] Operation 1: The communication thread of the requesting node dispatches the write request 1 to the worker thread of the requesting node. The worker thread of the requesting node writes the data of the write request 1 to a free data block in the data block area of ​​the CXL shared memory pool (e.g., write data block 1) through the NT-Store instruction, and prepares the contents of the corresponding metadata slot in the metadata area (e.g., metadata slot 1). The metadata slot 1 includes a write request identifier, such as "write request 1", a global request status "initial", a logical block address (LBA), and a pointer to the data block. After the data writing is completed and ready, the global request status in the metadata slot 1 is updated to "published" through a single NT-Store instruction.

[0145] Operation 2: The communication thread of the requesting node will write Request 1 to the CXL shared memory pool [Requesting Node - Master Node Shared Queue].

[0146] The master node's communication thread accesses the "Request Node-Master Node Shared Queue" in the CXL shared memory pool and can obtain the newly written write request 1 from the request node. At this time, the master node's communication thread also performs two operations.

[0147] Operation 1: The master node communication thread writes the write request 1 to the CXL shared memory pool [Master Node-Slave Node 1 Shared Queue], ..., [Master Node-Slave Node n Shared Queue].

[0148] Operation 2: The master node's communication thread assigns write request 1 to the local worker thread. After receiving write request 1, the master node's worker thread creates state machine 1 for write request 1, polls metadata slot 1 in the CXL shared memory pool, and when it finds that the global state of metadata slot 1 is "published", it means that the data is ready. At this time, the master node's worker thread accesses metadata slot 1 in the CXL shared memory pool based on the identifier of write request 1, accesses the data block area based on the data block 1 pointer in metadata slot 1, obtains the corresponding data block 1, performs local persistent writing (initiating asynchronous DMA I / O), and modifies state machine 1 to "data ready, persistent writing in progress", and then polls the ACK slot in the CXL shared memory pool.

[0149] From node 1 (follower): When the communication thread of node 1 accesses the [Master-Slave Node 1 Shared Queue] in the CXL shared memory pool, it can obtain a newly written write request 1. At this time, the communication thread of slave node 1 assigns write request 1 to its local worker thread. After receiving write request 1, the worker thread of slave node 1 creates state machine 1 for write request 1, and accesses metadata slot 1 in the metadata area of ​​the CXL shared memory pool based on the identifier of write request 1. Based on the data block 1 pointer in metadata slot 1, it retrieves the corresponding data block 1 from the data block area and performs local persistent writing. After the writing is completed, it writes an ACK feedback signal to the slave node feedback area (i.e., ACK area) of the CXL shared memory pool (write request 1 and completion status are stored in the corresponding slave node ACK slot jointly indexed by slave node ID and request ID), updates the state machine of write request 1 to "local persistence completed", and deletes the state machine of write request 1.

[0150] The other slave nodes operate in the same way as slave node 1 described above, and will not be repeated here in the embodiments of this specification.

[0151] Master node: The master node's worker thread polls the slave node feedback area of ​​the CXL shared memory pool. If the polling shows that the slave node's write is complete, the "global state" of the metadata slot 1 of write request 1 in the CXL shared memory pool is updated to "completed". The state machine of write request 1 in the master node's worker thread is also updated to "completed", and the state machine of write request 1 is deleted.

[0152] Request node (client): The worker thread of the request node polls the metadata slot 1 of the write request 1 in the CXL shared memory pool. Based on the "global status" of the metadata slot 1 being "completed", it can receive a completion signal and mark the write request 1 as completed.

[0153] In the embodiments of this specification, benchmark tests were used to compare the write operation performance of the CXL shared memory pool-based multi-replica storage system provided in the embodiments of this specification with that of an RDMA system. End-to-end write latency and aggregate bandwidth of the system were measured at request data sizes of 4KB and 16KB, while varying the I / O depth from 1 to 32. Figure 7 and Figure 8 As shown, Figure 7 This is a schematic diagram illustrating a write bandwidth test result provided in one embodiment of this specification. Figure 8 This is a schematic diagram of a write latency test result provided in one embodiment of this specification.

[0154] Comparison of aggregate write bandwidth as follows: Figure 7As shown, in the 4KB and 16KB tests, the multi-replica storage system based on the CXL shared memory pool provided in this specification consistently outperforms the RDMA benchmark. For example, at an I / O depth of 32, the multi-replica storage system achieves a bandwidth of 3.63 GB / s for 4KB requests, while the RDMA benchmark only achieves a throughput of 2.80 GB / s. For 16KB writes, the performance gap is even larger; for example, at an I / O depth of 16, the multi-replica storage system achieves a bandwidth of 5.39 GB / s, 30.6% higher than the RDMA benchmark. Furthermore, the difference in scalability becomes significant in the 16KB test. Specifically, the bandwidth of the RDMA benchmark gradually expands with increasing I / O depth, reaching a saturation point at the storage backend at a higher I / O depth (i.e., 32), approximately 5.39 GB / s. In contrast, the multi-replica storage system saturates the same storage backend at a lower I / O depth (i.e., 16), demonstrating its higher efficiency.

[0155] Figure 8 A comparison of write latency is shown. At a queue depth of one (representing the best-case single-request latency), the CXL shared memory pool-based multi-replica storage system provided in this specification demonstrates a significant advantage. For 4KB and 16KB requests, the latencies are 24.1µs and 26.8µs, respectively, representing 38.3% and 46.7% lower than the RDMA benchmark. As I / O depth increases, the latency of both RDMA and this multi-replica storage system rises. This phenomenon can be attributed to the increased queuing latency at the storage layer. However, the multi-replica storage system maintains its latency advantage at almost all tested I / O depths. A significant exception occurs with a 16KB request at an I / O depth of 32, where the latency of the multi-replica storage system increases sharply. This is because, at this data size and I / O depth setting, the multi-replica storage system places a much higher load on the storage backend than RDMA, causing storage backend overload, which in turn leads to a significant increase in I / O latency.

[0156] Figure 9 A structural block diagram of a computing device 900 according to one embodiment of this specification is shown. The components of the computing device 900 include, but are not limited to, a memory 910 and a processor 920. The processor 920 is connected to the memory 910 via a bus 930, and a database 950 is used to store data.

[0157] The computing device 900 also includes an access device 940, which enables the computing device 900 to communicate via one or more networks 960. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 940 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0158] In one embodiment of this specification, the aforementioned components of the computing device 900 and Figure 9 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 9 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0159] The computing device 900 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 900 can also be a mobile or stationary server.

[0160] The processor 920 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described multiple-copy storage method.

[0161] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device belongs to the same concept as the technical solution of the multi-copy storage system described above. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the multi-copy storage system described above.

[0162] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described multiple-copy storage method.

[0163] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solution of the multi-copy storage system described above. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the multi-copy storage system described above.

[0164] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described multiple copy storage method.

[0165] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the aforementioned multi-copy storage system belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the aforementioned multi-copy storage system.

[0166] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0167] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0168] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0169] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0170] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A multi-replica storage system, comprising requesting nodes, master nodes, and slave nodes, and a shared memory pool based on a high-speed interconnect protocol, wherein the shared memory pool comprises a metadata area, a slave node feedback area, and a data block area; The request node is configured to acquire a target write request, write the target write request into the metadata area and data block area of ​​the shared memory pool, and pass the target write request to the master node through the shared memory pool; The master node is configured to access the shared memory pool to obtain the target write request, write the target write request to the first local storage unit, and pass the target write request to the slave node through the shared memory pool; The slave node is configured to access the shared memory pool to obtain the target write request, write the target write request to the second local storage unit, and write a feedback signal to the slave node feedback area of ​​the shared memory pool.

2. The multi-replica storage system according to claim 1, wherein the shared memory pool further includes a communication queue, the communication queue being a queue of request identifiers to be transmitted stored in a circular buffer within the shared memory pool; The request node is further configured to write the target write request to the metadata area and data block area of ​​the shared memory pool, and to transmit the target request identifier of the target write request to the master node through the communication queue; The master node is further configured to obtain the target request identifier by accessing the communication queue of the shared memory pool; Based on the target request identifier, access the metadata area and data block area of ​​the shared memory pool, obtain the target write request, write the target write request to the first local storage unit, and transmit the target request identifier to the slave node through the communication queue of the shared memory pool; The slave node is further configured to obtain the target request identifier by accessing the communication queue of the shared memory pool; Based on the target request identifier, access the metadata area and data block area of ​​the shared memory pool, obtain the target write request, and write the target write request to the second local storage unit.

3. The multi-replica storage system according to claim 2, wherein the communication queue includes a first shared communication queue shared between the requesting node and the master node, and a second shared communication queue between the master node and each slave node; The requesting node is further configured to write the target request identifier into the first shared communication queue, wherein... The first shared communication queue is used to transmit the target request identifier between the requesting node and the master node; The master node is further configured to obtain the target request identifier by accessing a first shared communication queue of the shared memory pool; and to write the target request identifier into each of the second shared communication queues of the shared memory pool, wherein any second shared communication queue is used to transmit the target request identifier between the master node and the corresponding slave node; The slave node is further configured to access the second shared communication queue corresponding to the slave node in the shared memory pool to obtain the target request identifier.

4. The multi-replica storage system according to claim 3, wherein the request node is further configured to batch write multiple target request identifiers into the first shared communication queue, and after writing, execute a first memory barrier instruction for each target request identifier batch-written in the first shared communication queue, wherein, The first memory barrier instruction is used to make each target request identifier visible to the master node; The master node is further configured to write multiple target request identifiers into the second shared communication queue in batches. After writing, for each target request identifier written in batches in the second shared communication queue, a second memory barrier instruction is executed, wherein the second memory barrier instruction is used to make each target request identifier visible to the slave node.

5. The multi-replica storage system according to claim 2, wherein the request node is further configured to write the target control information of the target write request into the metadata area, and write the target request data of the target write request into the data block area; The master node is further configured to access the metadata area based on the target request identifier and obtain the target control information of the target write request; Access the data block area based on the target control information, copy the target request data of the data block area to obtain the corresponding target copy data, and write the target copy data into the first local storage unit. The slave node is further configured to access the metadata area based on the target request identifier to obtain the target control information of the target write request; Access the data block area based on the target control information, copy the target request data of the data block area to obtain the corresponding target copy data, and write the target copy data into the second local storage unit.

6. The multi-replica storage system according to claim 5, wherein the slave node feedback area is used to store the feedback signals corresponding to each slave node, and the feedback signals are based on the slave node identifier and the request identifier index; The slave node is further configured to write the target replica data into the second local storage unit, and then write the slave node identifier and the feedback signal corresponding to the target request identifier into the slave node feedback area. The master node is further configured to, after polling and determining that the requesting node has written the target write request to the shared memory pool, switch to polling the feedback area of ​​the slave node. If it is determined that the feedback signal written in the feedback area of ​​the slave node meets the replica storage constraint, then switch to write mode and update the target control information of the target request identifier in the metadata area. The request node is further configured to poll the target control information of the target request identifier in the metadata area, and determine the write process of the target write request based on the target control information.

7. The multi-replica storage system according to claim 5, wherein the metadata area includes multiple metadata slots, one metadata slot corresponds to one cache line, and is used to write control information for a write request, wherein the control information includes a write request identifier, a global request status, a logical block address, and a data block pointer; The request node is further configured to write the target request data of the target write request into a free data block in the data block area, and write the write request identifier, global request status, logical block address and data block pointer of the target write request into the target metadata slot in the metadata area. After the target request data is written, the global request status in the target metadata slot is updated and switched to read mode. The global request status is used to transmit the request status of the target write request across nodes. The master node is further configured to poll the target metadata slot of the metadata region based on the target request identifier. When the global request status in the target metadata slot indicates that the request node has completed writing, the master node obtains the logical block address and data block pointer of the target write request in the target metadata slot based on the target request identifier. Based on the logical block address and data block pointer of the target write request, the master node obtains the target replica data and writes it to the first local storage unit. The slave node is further configured to access the target metadata slot of the metadata region based on the target request identifier, obtain the logical block address and data block pointer of the target write request, and obtain the target replica data and write it to the second local storage unit based on the logical block address and data block pointer of the target write request.

8. The multi-replica storage system according to claim 7, wherein the master node is further configured to initiate a first asynchronous direct memory access operation based on the data block pointer and the logical block address, wherein, The first asynchronous direct memory access operation is used to copy the target requested data corresponding to the data block pointer in the data block area, obtain the target copy data, and write it to the first local storage unit indicated by the logical block address; The slave node is further configured to initiate a second asynchronous direct memory access operation based on the data block pointer and the logical block address, wherein the second asynchronous direct memory access operation is used to copy the target requested data corresponding to the data block pointer in the data block area, obtain the target copy data, and write it to the second local storage unit indicated by the logical block address.

9. The multi-replica storage system according to claim 1, wherein each node in the multi-replica storage system includes a communication thread and a working thread; The request node is further configured to retrieve a target write request from a queue of pending requests via a first communication thread, allocate a corresponding first worker thread to the target write request via the first communication thread, write the target write request to the shared memory pool via the first worker thread, and transmit the target write request to the master node via the shared memory pool via the first communication thread. The master node is further configured to access the shared memory pool through a second communication thread, obtain the target write request, allocate a corresponding second worker thread for the target write request through the second communication thread, and write the target write request to the first local storage unit through the second worker thread. And through the second communication thread, the target write request is transmitted to the slave node based on the shared memory pool; The slave node is configured to access the shared memory pool through a third communication thread, obtain the target write request, allocate a corresponding third worker thread for the target write request through the third communication thread, and write the target write request to the second local storage unit through the third worker thread.

10. The multi-replica storage system according to claim 9, wherein the request node is further configured to write the target request data of the target write request to the shared memory pool through the first worker thread based on a non-cached storage operation, wherein, The non-cached storage operation is the operation of writing data directly into memory without caching.

11. The multi-replica storage system according to claim 9, wherein the worker threads in each node concurrently process multiple write requests and create a state machine for each write request, the state machine being used to implement asynchronous concurrent processing of multiple write requests within the node; The master node is further configured to create a first state machine for the target write request through the second worker thread when the state machine queue of the second worker thread is not full. The first state machine is used to record the processing status of the second worker thread of the master node for the target write request. The slave node is further configured to create a second state machine for the target write request through the third worker thread when the state machine queue of the third worker thread is not full. The second state machine is used to record the processing status of the third worker thread of the slave node for the target write request.

12. The multi-replica storage system according to claim 1, wherein the shared memory pool is constructed based on a switch connected to a memory server using a high-speed interconnect protocol, and each node in the multi-replica storage system is configured with a node-side interconnect manager, the node-side interconnect manager being used by the corresponding node to access the shared memory pool and map the memory address of the shared memory pool to the local storage unit on the node side.

13. A multi-replica storage system, comprising request nodes, master nodes, and slave nodes, and a shared memory pool constructed by connecting memory servers via a switch based on a high-speed interconnect protocol; wherein the request nodes, master nodes, and slave nodes are configured with distributed services, node-side interconnect managers, and local storage units; The distributed service of the request node is configured to access the switch based on the high-speed interconnection protocol through the node-side interconnection manager, access the corresponding memory server through the switch, and write the target write request into the shared memory pool. The distributed service of the master node and slave node is configured to access the switch based on the high-speed interconnection protocol through the node-side interconnection manager, access the corresponding memory server through the switch, obtain the target write request from the shared memory pool, and write it to the corresponding local storage unit.

14. A multi-replica storage method applied to a master node in a multi-replica storage system, the multi-replica storage system further comprising requesting nodes, slave nodes, and a shared memory pool based on a high-speed interconnect protocol, the shared memory pool comprising a metadata area, a slave node feedback area, and a data block area; the method comprising: Access the shared memory pool to obtain a target write request, wherein the target write request is the requesting node writing to the metadata area and data block area of ​​the shared memory pool, and is transmitted to the master node through the shared memory pool; The target write request is written to the first local storage unit, and the target write request is passed to the slave node through the shared memory pool, so that the slave node can access the shared memory pool to obtain the target write request, write the target write request to the second local storage unit, and write a feedback signal to the slave node feedback area of ​​the shared memory pool.

15. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the multiple copy storage method of claim 14.

16. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the multiple copy storage method of claim 14.

17. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the multiple copy storage method of claim 14.

Images