A distributed storage method, device, equipment and readable storage medium

Abstract

This application discloses a distributed storage method, apparatus, device, and readable storage medium, comprising: mapping NVMe devices of other nodes to each node in a cluster as virtual NVMe devices; combining the local physical NVMe device and the virtual NVMe devices corresponding to the physical NVMe devices of other nodes in the cluster into an independent redundant disk array volume; creating a shared memory buffer; redirecting write requests from upper-layer applications to the pre-allocated shared memory buffer; and using RDMA technology to batch store data in the shared memory buffer into the local physical NVMe device and the virtual NVMe device in the independent redundant disk array volume. This application expands storage capacity by establishing virtual mappings of NVMe devices of other nodes on each node, and simultaneously enables direct data transmission between NVMe devices on each node using RDMA technology, thus achieving distributed data storage.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a distributed storage method, apparatus, device, and readable storage medium. Background Technology

[0002] In large-scale model applications such as large language models and generative AI, the scale of data processing grows exponentially, placing stringent demands on the single-machine storage capacity and cross-node storage read / write performance of storage systems. Existing storage solutions are struggling to meet the efficient data requirements of large-scale model training and inference. Specifically, single-machine NVMe storage capacity is insufficient: the capacity of a single NVMe SSD on a mainstream server node is typically 3-7TB, while the dataset required for large-scale model training often exceeds 20TB. The limited NVMe storage capacity of a single node cannot directly support the local loading of ultra-large-scale datasets, leading to cross-node scheduling for data access and further exacerbating performance degradation. Furthermore, the mainstream solution for sharing NVMe devices across nodes in existing technologies is the Network File System (NFS). NFS uses the TCP / IP protocol stack for data transmission, requiring the CPU to participate in protocol parsing and data copying processes throughout, resulting in significant CPU resource consumption and a substantial increase in end-to-end latency for data read / write. Therefore, distributed data storage has been a key research area. Summary of the Invention

[0003] In view of this, this application provides a distributed storage method, apparatus, device, and readable storage medium to facilitate distributed storage of data.

[0004] To achieve the above objectives, the following solution is proposed:

[0005] A distributed storage method, comprising:

[0006] For each node in the cluster, the physical non-volatile memory host controller interface specification NVMe device of other nodes in the cluster is mapped as a virtual non-volatile memory host controller interface specification NVMe device, so that each node contains its own local physical non-volatile memory host controller interface specification NVMe device and the virtual non-volatile memory host controller interface specification NVMe device corresponding to the physical non-volatile memory host controller interface specification NVMe devices of other nodes in the cluster.

[0007] For each node in the cluster, its local physical non-volatile memory host controller interface (NVMe) device and the virtual non-volatile memory host controller interface (NVMe) devices corresponding to the physical non-volatile memory host controller interface (NVMe) devices of other nodes in the cluster are combined into an independent redundant disk array volume.

[0008] Create a shared memory buffer;

[0009] Redirect write requests from upper-layer applications to a pre-allocated shared memory buffer;

[0010] By utilizing Remote Direct Memory Access (RDMA) technology, data in the shared memory buffer is stored in batches to local physical non-volatile memory host controller interface (NVMe) devices in independent disk redundant array volumes and virtual non-volatile memory host controller interface (NVMe) devices corresponding to physical non-volatile memory host controller interface (NVMe) devices on other nodes in the cluster, thereby achieving distributed data storage.

[0011] Optionally, before using Remote Direct Memory Access (RDMA) technology to batch store data from the shared memory buffer to the local physical non-volatile memory host controller interface (NVMe) device in the independent disk redundant array volume and the virtual non-volatile memory host controller interface (NVMe) devices corresponding to the physical non-volatile memory host controller interface (NVMe) devices of other nodes in the cluster, the following steps are also included:

[0012] Get the current values ​​of shared memory utilization and I / O latency of the shared memory buffer;

[0013] Adjust the I / O queue depth for batch storage of data in the shared memory buffer to the independent disk redundant array volume based on the current shared memory utilization and I / O latency;

[0014] The method of using Remote Direct Memory Access (RDMA) technology to batch store data in the shared memory buffer to local physical non-volatile memory host controller interface (NVMe) devices in an independent disk redundant array volume and to virtual non-volatile memory host controller interface (NVMe) devices corresponding to physical non-volatile memory host controller interface (NVMe) devices on other nodes in the cluster includes:

[0015] Based on the I / O queue depth, using Remote Direct Memory Access (RDMA) technology, data in the shared memory buffer is batch stored to the local Physical Non-Volatile Memory Host Controller Interface (NVMe) device in the independent disk redundant array volume and the virtual Non-Volatile Memory Host Controller Interface (NVMe) devices corresponding to the Physical Non-Volatile Memory Host Controller Interface (NVMe) devices of other nodes in the cluster.

[0016] Optionally, adjusting the I / O queue depth for batch storing data in the shared memory buffer to the independent disk redundant array volume based on the shared memory utilization and current I / O latency includes:

[0017] Determine whether the shared memory utilization rate has reached a preset shared memory utilization rate threshold;

[0018] If the shared memory utilization rate reaches the preset shared memory utilization rate threshold, the I / O queue depth is adjusted to the preset maximum I / O queue depth.

[0019] If the shared memory utilization rate does not reach the preset shared memory utilization rate threshold, the I / O queue depth is dynamically adjusted based on the current I / O latency value, the preset target I / O latency value, and the preset maximum and minimum I / O queue depths.

[0020] Optionally, the method for determining the preset target value of the I / O delay includes:

[0021] Based on historical I / O latency distribution, a preset target value for I / O latency is determined.

[0022] A distributed storage device, comprising:

[0023] The device mapping module is used to map the physical non-volatile memory host controller interface specification NVMe devices of other nodes in the cluster to each node in the cluster, and then to virtual non-volatile memory host controller interface specification NVMe devices. This allows each node to contain its own local physical non-volatile memory host controller interface specification NVMe device and the virtual non-volatile memory host controller interface specification NVMe devices corresponding to the physical non-volatile memory host controller interface specification NVMe devices of other nodes in the cluster.

[0024] The disk striping module is used to combine the local physical non-volatile memory host controller interface (NVMe) device of each node in the cluster with the virtual non-volatile memory host controller interface (NVMe) devices corresponding to the physical non-volatile memory host controller interface (NVMe) devices of other nodes in the cluster into an independent redundant disk array volume.

[0025] The buffer creation module is used to create shared memory buffers;

[0026] The data offloading module is used to redirect write requests from upper-layer applications to a pre-allocated shared memory buffer.

[0027] The distributed storage module is used to utilize Remote Direct Memory Access (RDMA) technology to batch store data in the shared memory buffer to local physical non-volatile memory host controller interface (NVMe) devices in independent disk redundant array volumes and virtual non-volatile memory host controller interface (NVMe) devices corresponding to physical non-volatile memory host controller interface (NVMe) devices of other nodes in the cluster, thereby realizing distributed data storage.

[0028] Optional, also includes:

[0029] The data acquisition module is used to obtain the current values ​​of shared memory utilization and I / O latency of the shared memory buffer;

[0030] The write strategy adjustment module is used to adjust the I / O queue depth of batch storing data in the shared memory buffer to the independent disk redundant array volume based on the current value of shared memory utilization and I / O latency.

[0031] The distributed storage module performs the process of using Remote Direct Memory Access (RDMA) technology to batch store data from the shared memory buffer to the local physical non-volatile memory host controller interface (NVMe) devices in the independent disk redundant array volume and the virtual non-volatile memory host controller interface (NVMe) devices corresponding to the physical non-volatile memory host controller interface (NVMe) devices of other nodes in the cluster, including:

[0032] Based on the I / O queue depth, using Remote Direct Memory Access (RDMA) technology, data in the shared memory buffer is batch stored to the local Physical Non-Volatile Memory Host Controller Interface (NVMe) device in the independent disk redundant array volume and the virtual Non-Volatile Memory Host Controller Interface (NVMe) devices corresponding to the Physical Non-Volatile Memory Host Controller Interface (NVMe) devices of other nodes in the cluster.

[0033] Optionally, the write strategy adjustment module performs a process of adjusting the I / O queue depth of the shared memory buffer to store data in batches to the independent disk redundant array volume based on the shared memory utilization and the current value of I / O latency, including:

[0034] Determine whether the shared memory utilization rate has reached a preset shared memory utilization rate threshold;

[0035] If the shared memory utilization rate reaches the preset shared memory utilization rate threshold, the I / O queue depth is adjusted to the preset maximum I / O queue depth.

[0036] If the shared memory utilization rate does not reach the preset shared memory utilization rate threshold, the I / O queue depth is dynamically adjusted based on the current I / O latency value, the preset target I / O latency value, and the preset maximum and minimum I / O queue depths.

[0037] Optionally, the write strategy adjustment module determines the preset target value for I / O latency in the following ways:

[0038] Based on historical I / O latency distribution, a preset target value for I / O latency is determined.

[0039] A distributed storage device includes: a memory and a processor;

[0040] The memory is used to store programs;

[0041] The processor is used to execute the program to implement the various steps of the distributed storage method as described in any of the preceding embodiments.

[0042] A readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the distributed storage method as described in any of the preceding claims.

[0043] As can be seen from the above technical solutions, the distributed storage method, apparatus, device, and readable storage medium provided in this application embodiment include: for each node in the cluster, mapping the physical non-volatile memory host controller interface specification (NVMe) devices of other nodes in the cluster as virtual non-volatile memory host controller interface specification (NVMe) devices, such that each node includes its own local physical non-volatile memory host controller interface specification (NVMe) device and the virtual non-volatile memory host controller interface specification (NVMe) devices corresponding to the physical non-volatile memory host controller interface specification (NVMe) devices of other nodes in the cluster; for each node in the cluster, mapping its own local physical non-volatile memory host controller interface specification (NVMe) device... This application specifies that NVMe devices and virtual NVMe devices corresponding to the physical non-volatile memory host controller interface specifications of other nodes in the cluster are combined into an independent redundant disk array volume. A shared memory buffer is created; write requests from upper-layer applications are redirected to the pre-allocated shared memory buffer; and data in the shared memory buffer is batch-stored into the local physical non-volatile memory host controller interface specification NVMe device and the virtual non-volatile memory host controller interface specification NVMe device corresponding to the physical non-volatile memory host controller interface specification NVMe device of other nodes in the cluster using RDMA technology. This application expands storage capacity by establishing virtual mappings of other node NVMe devices on each node, and simultaneously achieves distributed data storage by combining RDMA technology to enable direct data transfer between NVMe devices on each node. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0045] Figure 1 A flowchart of a distributed storage method provided in an embodiment of this application;

[0046] Figure 2 An organizational architecture diagram of a distributed storage device provided in this application embodiment;

[0047] Figure 3 This is a schematic diagram of a distributed storage device structure provided in an embodiment of this application;

[0048] Figure 4 This is a hardware structure block diagram of a distributed storage device provided in an embodiment of this application. Detailed Implementation

[0049] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0050] Figure 1 This application provides a flowchart of a distributed storage method, which includes:

[0051] Step S100: For each node in the cluster, map the physical non-volatile memory host controller interface specification (Non-Volatile Memory Express, NVMe) device of other nodes in the cluster as a virtual NVMe device.

[0052] Specifically, for each node in the cluster, other nodes act as servers. A Storage Performance Development Kit (SPDK) is deployed on the server, and the ko driver is loaded to initialize the server-side NVMe device. The node itself acts as a client, deploying the nvme-cli program. This involves loading kernel modules, creating NVMe targets, adding namespaces to bind physical disks, configuring Remote Direct Memory Access (RDMA) ports, and optimizing NVMe-RDMA configurations. For example, increasing inline data allows small I / O requests to carry data directly in the RDMA transport layer, avoiding additional memory registration operations and reducing microsecond-level latency to some extent. Increasing queue depth improves RDMA concurrent processing capabilities and matches the bandwidth of high-performance network cards, ultimately completing the creation of the virtual NVMe device. Using the NVMe-RDMA kernel module and the nvme-cli tool, the virtual NVMe device is mapped from the remote physical NVMe device to the local device, building a peer-to-peer interconnect architecture. This allows a single node to access the resources of all NVMe devices within the cluster, achieving horizontal linear scaling of storage capacity, with the scaling factor equal to the number of cluster nodes.

[0053] like Figure 2 As shown, Figure 2 This application provides a distributed storage device organizational architecture diagram, in which each node acts as both a server and a client, such that each node contains its own local physical NVMe device and virtual NVMe devices corresponding to the physical NVMe devices of other nodes in the cluster.

[0054] For example Figure 2 In the example, node-1 itself has one physical NVMe device, nvme01. After mapping the physical NVMe devices on nodes-2 and-3 to node-1, node-1 will have one physical NVMe device and two virtual NVMe devices. Similarly, nodes-2 and-3 will also have one physical NVMe device and two virtual NVMe devices. At this point, each node has three NVMe devices. Assuming each NVMe device has 7.7TB of storage, after completing the above operations, the total capacity of the NVMe devices on each node reaches 3 x 7.7TB = 23.1TB, thus achieving NVMe storage capacity expansion.

[0055] Step S101: For each node in the cluster, combine its own local physical NVMe device and the virtual NVMe devices corresponding to the physical NVMe devices of other nodes in the cluster into an independent redundant disk array volume.

[0056] Specifically, each node's one physical NVMe device and multiple virtual NVMe devices are combined into an independent redundant disk RAID volume. For example, disk striping can be used to merge them into a single RAID-0 volume. Local physical NVMe devices and virtual NVMe devices are combined into a RAID-0 striped volume. Striping technology is used to aggregate bandwidth across multiple devices, and combined with XFS file system optimization, the aggregated bandwidth is maximized. Linux's dma_map_page technology accelerates disk I / O through the Direct Memory Access (DMA) engine, achieving memory-to-disk mapping. This achieves aggregated bandwidth; for example, if each NVMe device has a read / write bandwidth of 4GB / s, three devices aggregated together can provide a total NVMe read / write bandwidth of up to 12GB / s. Simultaneously, storage capacity is doubled; a single NVMe device has 7.7TB, and the aggregated capacity reaches 23.1TB.

[0057] By using RAID-0 striping to aggregate the bandwidth of multiple NVMe devices, the problem of insufficient bandwidth of a single NVMe device is solved, meeting the high-parallel data access requirements of large model training. Furthermore, DMA mapping technology enables direct data transfer between memory and storage devices, avoiding CPU involvement in copying. This, combined with RDMA technology, further optimizes end-to-end latency and ensures that data is evenly distributed across multiple nodes, avoiding single-node storage hotspots and improving system stability.

[0058] Step S102: Create a shared memory buffer.

[0059] Specifically, during system startup, shared memory space is reserved through kernel parameters to ensure that the buffer is exclusively occupied and free of memory fragmentation.

[0060] Step S103: Redirect the write request of the upper layer application to the pre-allocated shared memory buffer.

[0061] Specifically, by utilizing the data offloading function supported by the NVMe protocol, write requests are redirected to a shared memory buffer. Data is quickly written to the shared memory buffer, and a write success response is given immediately. By leveraging the high bandwidth and low latency characteristics of shared memory, a fast response from the client can be achieved.

[0062] Step S104: Using RDMA technology, the data in the shared memory buffer is stored in batches to the local physical NVMe device in the independent disk redundant array volume and the virtual NVMe device corresponding to the physical NVMe device of other nodes in the cluster.

[0063] Specifically, leveraging NVMe over RDMA enables seamless and transparent data transfer to upper-layer applications. By replacing the traditional TCP protocol with RDMA technology and combining SPDK user-space drivers with dma_map_page DMA mapping technology, direct data transfer between client and server NVMe devices is achieved. The entire process involves no CPU involvement in protocol parsing or data copying, achieving end-to-end microsecond-level latency. By avoiding the CPU overhead of TCP protocol parsing and data copying, as well as the switching latency between kernel and user modes, end-to-end data transfer latency can be reduced to the microsecond level. Furthermore, RDMA supports high-bandwidth, low-packet-loss large-scale data transfer. Combined with the high IOPS characteristics of NVMe devices, aggregated bandwidth is maximized, such as 12GB / s aggregated read / write bandwidth, reducing CPU usage and freeing up CPU resources for core business processing, thereby improving the overall system resource utilization.

[0064] After the data is written to the virtual NVMe device, it is then directly written to the physical NVMe device on the server using RDMA technology, such as... Figure 2 As shown, after data is written to the virtual NVMe02 on node1, it is directly written to the physical NVMe device on the server, i.e., the physical NVMe02 device on node2, via RDMA technology. This process bypasses CPU relay and TCP network, utilizing the RDMA network to achieve data offloading, accelerate the data read and write process, and realize distributed data storage. A shared memory buffer is used, combined with a two-stage asynchronous write mechanism for background batch writes, enabling fast response to application write requests through shared memory.

[0065] By overcoming the capacity limitations of single-node NVMe devices through the above methods, the problem that the 3-7TB NVMe capacity of a single node in existing technologies cannot support large models with datasets of more than 20TB is solved. Through capacity aggregation, storage expansion of 23.1TB and above can be achieved on a single node, with the expansion factor equal to the number of cluster nodes. Moreover, upper-layer applications do not need to modify any code and can access distributed NVMe resources in the same way as accessing local NVMe devices, reducing the cost of technology implementation, making full use of existing NVMe devices in the cluster, eliminating the need to purchase additional high-performance storage devices, and reducing hardware investment costs.

[0066] As can be seen from the above embodiments, the distributed storage method provided in this application includes: mapping the physical NVMe devices of other nodes in the cluster to virtual NVMe devices for each node in the cluster, so that each node contains its own local physical NVMe device and the virtual NVMe devices corresponding to the physical NVMe devices of other nodes in the cluster; combining the local physical NVMe device and the virtual NVMe devices corresponding to the physical NVMe devices of other nodes in the cluster into an independent redundant disk array volume for each node in the cluster; creating a shared memory buffer; redirecting write requests from upper-layer applications to the pre-allocated shared memory buffer; and using RDMA technology to batch store the data in the shared memory buffer into the local physical NVMe device and the virtual NVMe devices corresponding to the physical NVMe devices of other nodes in the independent redundant disk array volume. This application expands storage capacity by establishing virtual mappings of NVMe devices of other nodes on each node, and at the same time, combines RDMA technology to realize direct data transmission between the NVMe devices of each node, thereby achieving distributed data storage.

[0067] In some embodiments of this application, since the load varies, different I / O queue depths can be used for different data volumes during distributed data storage to balance latency and throughput and ensure the stability of storage performance. Based on this, before step S104, which uses RDMA technology to batch store the data in the shared memory buffer to the local physical NVMe device in the independent disk redundant array volume and the virtual NVMe device corresponding to the physical NVMe device of other nodes in the cluster, the distributed storage method may further include the following steps:

[0068] S11. Get the current values ​​of shared memory utilization and I / O latency of the shared memory buffer.

[0069] Specifically, the shared memory usage rate can be obtained using the statfs function to retrieve information about the shared memory directory.

[0070] S12. Based on the current shared memory utilization and I / O latency, adjust the I / O queue depth for batch storage of data in the shared memory buffer to the independent disk redundant array volume.

[0071] Specifically, through the asynchronous I / O engine, after batch submitting data from the shared memory buffer to the independent disk redundant array volume, the asynchronous I / O queue depth can be dynamically adjusted based on the shared memory utilization and current I / O latency. Dynamic I / O queue depth scheduling can adjust the write strategy according to real-time load conditions, such as memory utilization and latency, to avoid latency waste under light loads and buffer overflows under heavy loads, ensuring performance stability under different load scenarios.

[0072] Based on this, step S104, which utilizes RDMA technology to batch store the data in the shared memory buffer to the local physical NVMe device in the independent disk redundant array volume and the virtual NVMe device corresponding to the physical NVMe devices of other nodes in the cluster, may include:

[0073] Based on the I / O queue depth, using RDMA technology, data in the shared memory buffer is stored in batches to the local physical NVMe device in the independent disk redundant array volume and the virtual NVMe device corresponding to the physical NVMe device of other nodes in the cluster.

[0074] In some embodiments of this application, S12, adjusting the I / O queue depth for batch storing data in the shared memory buffer to the independent disk redundant array volume based on the shared memory utilization rate and the current value of I / O latency, may include:

[0075] S21. Determine whether the shared memory utilization rate has reached the preset shared memory utilization rate threshold.

[0076] Specifically, if the shared memory utilization rate reaches the preset shared memory utilization rate threshold, then S22 is executed; if the shared memory utilization rate does not reach the preset shared memory utilization rate threshold, then S23 is executed.

[0077] S22. Adjust the I / O queue depth to the preset maximum I / O queue depth.

[0078] Specifically, a preset shared memory utilization threshold can be set to 80%. When the shared memory utilization reaches 80%, the maximum I / O queue depth is used to improve batch write efficiency and quickly release the shared memory buffer. The preset maximum I / O queue depth can be set according to hardware limitations.

[0079] S23. Based on the current I / O latency value, the preset target I / O latency value, and the preset maximum and minimum I / O queue depths, dynamically adjust the I / O queue depth.

[0080] Specifically, the preset target value for I / O latency can be determined based on historical I / O latency distribution. If the shared memory utilization rate does not reach the preset shared memory utilization rate threshold, the I / O queue depth can be dynamically adjusted based on the current I / O latency value, the preset target value for I / O latency, and the preset maximum and minimum I / O queue depths. When the shared memory buffer contains a large amount of data, the I / O queue depth can be increased to accelerate writing and increase throughput; when the amount of shared memory data is small, the I / O queue depth can be decreased to avoid waiting and reduce latency. The specific I / O queue depth iodepth can be calculated according to the following formula:

[0081]

[0082] in, Set the minimum depth for the preset I / O queue; Set the maximum depth of the preset I / O queue; Preset target values ​​for I / O latency; This is the current value of the I / O delay.

[0083] The distributed storage device provided in the embodiments of this application is described below. The distributed storage device described below can be referred to in correspondence with the distributed storage method described above.

[0084] Figure 3 This application provides a schematic diagram of a distributed storage device structure, which may include:

[0085] The device mapping module 10 is used to map the physical NVMe devices of other nodes in the cluster to each node in the cluster as virtual NVMe devices, so that each node contains its own local physical NVMe device and the virtual NVMe devices corresponding to the physical NVMe devices of other nodes in the cluster.

[0086] The disk striping module 20 is used to combine the local physical NVMe device of each node in the cluster with the virtual NVMe devices corresponding to the physical NVMe devices of other nodes in the cluster into an independent redundant disk array volume.

[0087] Buffer creation module 30 is used to create shared memory buffers;

[0088] The data offloading module 40 is used to redirect write requests from upper-layer applications to a pre-allocated shared memory buffer.

[0089] The distributed storage module 50 is used to utilize RDMA technology to batch store data in the shared memory buffer to the local physical NVMe device in the independent disk redundant array volume and the virtual NVMe device corresponding to the physical NVMe device of other nodes in the cluster, thereby realizing distributed data storage.

[0090] As can be seen from the above embodiments, the distributed storage device provided in this application may include: a device mapping module 10, used to map the physical NVMe devices of other nodes in the cluster to each node in the cluster as virtual NVMe devices, so that each node contains its own local physical NVMe device and the virtual NVMe devices corresponding to the physical NVMe devices of other nodes in the cluster; a disk striping module 20, used to combine the local physical NVMe device and the virtual NVMe devices corresponding to the physical NVMe devices of other nodes in the cluster into an independent redundant disk array volume for each node in the cluster; a buffer creation module 30, used to create a shared memory buffer; a data offloading module 40, used to redirect write requests from upper-layer applications to the pre-allocated shared memory buffer; and a distributed storage module 50, used to use RDMA technology to batch store data in the shared memory buffer to the local physical NVMe devices and the virtual NVMe devices corresponding to the physical NVMe devices of other nodes in the independent redundant disk array volume. This application expands storage capacity by establishing virtual mappings of NVMe devices of other nodes on each node, and at the same time, combined with RDMA technology, realizes direct data transmission between NVMe devices on each node, thereby achieving distributed data storage.

[0091] Optionally, the distributed storage device may also include:

[0092] The data acquisition module is used to obtain the current values ​​of shared memory utilization and I / O latency of the shared memory buffer;

[0093] The write strategy adjustment module is used to adjust the I / O queue depth of batch storing data in the shared memory buffer to the independent disk redundant array volume based on the current value of shared memory utilization and I / O latency.

[0094] The distributed storage module 50 performs the process of using RDMA technology to batch store data in the shared memory buffer to the local physical NVMe device in the independent disk redundant array volume and the virtual NVMe device corresponding to the physical NVMe device of other nodes in the cluster. This process may include:

[0095] Based on the I / O queue depth, using RDMA technology, data in the shared memory buffer is stored in batches to the local physical NVMe device in the independent disk redundant array volume and the virtual NVMe device corresponding to the physical NVMe device of other nodes in the cluster.

[0096] Optionally, the write strategy adjustment module performs the process of adjusting the I / O queue depth of the shared memory buffer to store data in batches to the independent disk redundant array volume based on the current value of shared memory utilization and I / O latency. This process may include:

[0097] Determine whether the shared memory utilization rate has reached a preset shared memory utilization rate threshold;

[0098] If the shared memory utilization rate reaches the preset shared memory utilization rate threshold, the I / O queue depth is adjusted to the preset maximum I / O queue depth.

[0099] If the shared memory utilization rate does not reach the preset shared memory utilization rate threshold, the I / O queue depth is dynamically adjusted based on the current I / O latency value, the preset target I / O latency value, and the preset maximum and minimum I / O queue depths.

[0100] Optionally, the method by which the write strategy adjustment module determines the preset target value for I / O latency may include:

[0101] Based on historical I / O latency distribution, a preset target value for I / O latency is determined.

[0102] This application also provides a distributed storage device. Figure 4 The hardware structure block diagram of the distributed storage device is shown, with reference to... Figure 4 The hardware structure of a distributed storage device may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4;

[0103] In this embodiment of the application, the number of processor 1, communication interface 2, memory 3, and communication bus 4 is at least one, and processor 1, communication interface 2, and memory 3 communicate with each other through communication bus 4;

[0104] Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0105] Memory 3 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;

[0106] The memory stores a program, and the processor can call the program stored in the memory. The program is used to implement the various processing flows in the aforementioned distributed storage method.

[0107] This application embodiment also provides a storage medium that can store a program suitable for execution by a processor, the program being used to implement various processing flows in the aforementioned distributed storage method.

[0108] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0109] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined with each other, and the same or similar parts can be referred to each other.

[0110] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A distributed storage method, characterized in that, include: For each node in the cluster, the physical non-volatile memory host controller interface specification NVMe device of other nodes in the cluster is mapped as a virtual non-volatile memory host controller interface specification NVMe device, so that each node contains its own local physical non-volatile memory host controller interface specification NVMe device and the virtual non-volatile memory host controller interface specification NVMe device corresponding to the physical non-volatile memory host controller interface specification NVMe devices of other nodes in the cluster. For each node in the cluster, its local physical non-volatile memory host controller interface (NVMe) device and the virtual non-volatile memory host controller interface (NVMe) devices corresponding to the physical non-volatile memory host controller interface (NVMe) devices of other nodes in the cluster are combined into an independent redundant disk array volume. Create a shared memory buffer; Redirect write requests from upper-layer applications to a pre-allocated shared memory buffer; Get the current values ​​of shared memory utilization and I / O latency of the shared memory buffer; Adjust the I / O queue depth for batch storage of data in the shared memory buffer to the independent disk redundant array volume based on the current shared memory utilization and I / O latency; Based on the I / O queue depth, using Remote Direct Memory Access (RDMA) technology, data in the shared memory buffer is batch-stored to the local physical non-volatile memory host controller interface (NVMe) device in the independent disk redundant array volume and the virtual non-volatile memory host controller interface (NVMe) devices corresponding to the physical non-volatile memory host controller interface (NVMe) devices of other nodes in the cluster, thereby realizing distributed data storage.

2. The method according to claim 1, characterized in that, The step of adjusting the I / O queue depth for batch storage of data in the shared memory buffer to the independent disk redundant array volume based on the current value of shared memory utilization and I / O latency includes: Determine whether the shared memory utilization rate has reached a preset shared memory utilization rate threshold; If the shared memory utilization rate reaches the preset shared memory utilization rate threshold, the I / O queue depth is adjusted to the preset maximum I / O queue depth. If the shared memory utilization rate does not reach the preset shared memory utilization rate threshold, the I / O queue depth is dynamically adjusted based on the current I / O latency value, the preset target I / O latency value, and the preset maximum and minimum I / O queue depths.

3. The method according to claim 2, characterized in that, The method for determining the preset target value of I / O latency includes: Based on historical I / O latency distribution, a preset target value for I / O latency is determined.

4. A distributed storage device, characterized in that, include: The device mapping module is used to map the physical non-volatile memory host controller interface specification NVMe devices of other nodes in the cluster to each node in the cluster, and then to virtual non-volatile memory host controller interface specification NVMe devices. This allows each node to contain its own local physical non-volatile memory host controller interface specification NVMe device and the virtual non-volatile memory host controller interface specification NVMe devices corresponding to the physical non-volatile memory host controller interface specification NVMe devices of other nodes in the cluster. The disk striping module is used to combine the local physical non-volatile memory host controller interface (NVMe) device of each node in the cluster with the virtual non-volatile memory host controller interface (NVMe) devices corresponding to the physical non-volatile memory host controller interface (NVMe) devices of other nodes in the cluster into an independent redundant disk array volume. The buffer creation module is used to create shared memory buffers; The data offloading module is used to redirect write requests from upper-layer applications to a pre-allocated shared memory buffer. The data acquisition module is used to obtain the current values ​​of shared memory utilization and I / O latency of the shared memory buffer; The write strategy adjustment module is used to adjust the I / O queue depth of batch storing data in the shared memory buffer to the independent disk redundant array volume based on the current value of shared memory utilization and I / O latency. The distributed storage module is used to store data in batches from the shared memory buffer to local physical non-volatile memory host controller interface (NVMe) devices in an independent disk redundant array volume and virtual non-volatile memory host controller interface (NVMe) devices corresponding to physical non-volatile memory host controller interface (NVMe) devices of other nodes in the cluster, based on the I / O queue depth and using Remote Direct Memory Access (RDMA) technology, thereby realizing distributed data storage.

5. The apparatus according to claim 4, characterized in that, The write strategy adjustment module performs the process of adjusting the I / O queue depth of batch storing data in the shared memory buffer to the independent disk redundant array volume based on the shared memory utilization and the current I / O latency, including: Determine whether the shared memory utilization rate has reached a preset shared memory utilization rate threshold; If the shared memory utilization rate reaches the preset shared memory utilization rate threshold, the I / O queue depth is adjusted to the preset maximum I / O queue depth. If the shared memory utilization rate does not reach the preset shared memory utilization rate threshold, the I / O queue depth is dynamically adjusted based on the current I / O latency value, the preset target I / O latency value, and the preset maximum and minimum I / O queue depths.

6. The apparatus according to claim 5, characterized in that, The write strategy adjustment module determines the preset target value for I / O latency in the following ways: Based on historical I / O latency distribution, a preset target value for I / O latency is determined.

7. A distributed storage device, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is configured to execute the program to implement the steps of the distributed storage method as described in any one of claims 1-3.

8. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the distributed storage method as described in any one of claims 1-3.

Images