Data storage method and device and storage medium

Abstract

The embodiment of the invention provides a data storage method and device and a storage medium. An innovative network card device is assembled on each storage server in a storage cluster, and a nonvolatile cache is arranged in the network card device. After the network card equipment receives the to-be-stored data, the to-be-stored data can be written into a nonvolatile cache on the network card equipment; and then, directly writing one of a plurality of to-be-persistent data constructed for the to-be-stored data and used for persistent storage into a storage disk on the storage server where the to-be-persistent data is located by the network card equipment, and distributing the other to network card equipment assembled on other storage servers, thereby completing persistent storage. It can be known that in the persistent storage process, dependence on a CPU in the storage server is decoupled, multiple times of memory copying do not need to be carried out in the storage server, data transfer is directly carried out through the network card equipment, construction processing of the to-be-persistent data is completed nearby, and therefore the persistent storage efficiency of the storage cluster can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of storage technology, and in particular to a data storage method, device and storage medium. Background Art

[0002] Compared with local storage, distributed storage clusters can support dynamic expansion or contraction on demand, and have high availability and strong consistency of data persistence storage. In recent years, high-bandwidth RDMA networks have shortened network transmission delays, and the design of high-performance storage servers has significantly reduced the end-to-end delay of distributed storage while adapting to high throughput, making the performance of distributed storage clusters close to that of local storage disks, narrowing the gap, improving user experience, and meeting the access requirements of application software.

[0003] Currently, after the computing node sends the data to be stored to the storage server, the storage server's network card will first store the data in the storage server's system memory DRAM, and then copy the data in the system memory to the persistent memory through memory copying, and return a successful write notification to the computing node. After that, the storage server needs to copy the data from the persistent memory PMEM back to the system memory, and the storage server's CPU needs to read the data from the system memory and build multiple copies of the data to be persisted, which will then be distributed to multiple storage servers and finally stored in the solid-state hard disks and other storage disks in each storage server. Only after the above complex process can the storage cluster complete the persistent storage of data.

[0004] It can be seen that in the existing persistent storage solutions, although the network transmission delay is shortened, multiple memory copies are required inside the storage server, and it also relies on the participation of the CPU. Therefore, the storage delay inside the storage server is still high, which leads to the overall delay of data persistent storage based on the storage cluster still being different from the delay of local storage. Summary of the invention

[0005] Multiple aspects of the present application provide a data storage method, device, and storage medium to reduce the latency of persistent storage in a storage cluster.

[0006] The embodiment of the present application provides a data storage method, which is applicable to a network card device installed on a storage server in a storage cluster, wherein the network card device is provided with a non-volatile cache, and the method comprises:

[0007] Writing the received data to be stored into the non-volatile cache;

[0008] Constructing multiple copies of the data to be persisted for persistent storage for the data to be stored;

[0009] Writing one copy of the data to be persisted among the multiple copies of the data to be persisted into a storage disk on the storage server;

[0010] The other copies of the data to be persisted are distributed to the network card devices installed on the other storage servers, so that the network card devices installed on the other storage servers respectively write the received data to be persisted into the storage disk of the storage server where they are located.

[0011] The embodiment of the present application also provides a network card device, including a memory, a processor, a communication component and a non-volatile cache;

[0012] The memory is used to store one or more computer instructions;

[0013] The processor is coupled to the memory, the communication component and the non-volatile cache, and is configured to execute the one or more computer instructions to perform the aforementioned data storage method.

[0014] An embodiment of the present application further provides a storage cluster, including multiple storage servers, each of which is equipped with a network card device, and the network card device is used to execute the aforementioned data storage method to persistently store the data to be stored in the storage cluster.

[0015] An embodiment of the present application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the aforementioned data storage method.

[0016] In the embodiment of the present application, an innovative network card device is installed on each storage server in the storage cluster, and a non-volatile cache is provided in the network card device. On this basis, after receiving the data to be stored, the network card device can write the data to be stored into the non-volatile cache on it, which can effectively shorten the response delay to the corresponding storage request; afterwards, the network card device can construct multiple copies of the data to be stored for persistent storage for the data to be stored, and directly write one of the data to be persisted into the storage disk on the storage server where it is located by the network card device, and distribute the other copies of the data to be persisted to the network card devices installed on other storage servers in the storage cluster, so that these network card devices respectively write the received data to be persisted into the storage disk on the storage server where they are located, thereby completing the persistent storage. It can be seen that in the process of persistent storage, the dependence on the CPU in the storage server is decoupled, and it is no longer necessary to perform multiple memory copies in the storage server. Instead, the network card device directly transfers the data and completes the construction and processing of the data to be persisted nearby, which can effectively reduce the storage delay on the storage server side, thereby effectively improving the persistent storage efficiency of the storage cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 This is a logical diagram of persistent storage based on a storage cluster;

[0019] Figure 2 This is a logical diagram of the traditional persistent storage solution inside the storage server;

[0020] Figure 3 A schematic diagram of the structure of a network card device and a schematic diagram of its assembly on a storage server provided for an exemplary embodiment of the present application;

[0021] Figure 4 An exemplary structural diagram of a network card device provided for an exemplary embodiment of the present application;

[0022] Figure 5 A logical schematic diagram of an optional implementation scheme for constructing data to be persisted in a network card device provided by an exemplary embodiment of the present application;

[0023] Figure 6 A schematic diagram of a data storage method provided by another exemplary embodiment of the present application;

[0024] Figure 7a An exemplary flow chart of writing data to be stored into a storage cluster is provided;

[0025] Figure 7b An exemplary flow chart for completing persistence of data to be stored in a storage cluster is provided. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0027] Figure 1 This is a logical diagram of persistent storage based on a storage cluster. Figure 1, the computing node can send the data to be stored to the storage cluster through the network switch. In actual applications, computing nodes usually adopt the method of submitting multiple backup data in parallel to store data based on the storage cluster. Taking three backups as an example, the computing node can initiate storage requests to the three storage servers in the storage cluster in parallel. The storage request carries the backup data, and the backup data can be used by the storage server as the data to be stored. After the three backups are successfully written, the storage cluster can select one of the three storage servers as the primary server, and the primary server will persist the backup data stored on it to multiple storage servers in the storage cluster to complete the persistent storage of data for the computing node.

[0028] Figure 2 This is a logical diagram of the traditional persistent storage solution inside the storage server. Figure 2 In traditional persistent storage solutions, in order to shorten the write path, storage servers usually use persistent memory PMEM rather than bus PCIe storage disks with greater latency. Based on this, after the backup data submitted by the computing node enters the network card of the storage server, it will be transmitted to the system memory DRAM of the storage server through the bus PCIe link, and then the data will be copied to the persistent memory PMEM through the system memory DRAM; at this time, the computing node will be fed back that the backup has been written successfully, and the write path of the backup data has ended; the computing node can declare the write successful after the multiple backups it submitted have been written successfully. After that, continue to refer to Figure 2 , the storage server needs to read the backup data from the persistent memory PMEM to the system memory DRAM. The CPU of the storage server then constructs data for persistence based on the backup data read from the system memory DRAM and sends it to the network card. After that, the network card can distribute these data for persistence to multiple storage servers in the storage cluster through the network switch. Each storage server needs to store the received data in its own SSD or other storage disk to complete the persistent storage of the data. When the persistent storage is successful, the corresponding data can be deleted from the persistent memory PMEM of the storage server to release space for subsequent data.

[0029] During the research, the inventors found that the traditional persistent storage solution has at least the following limitations:

[0030] 1. Persistent memory PMEM requires CPU support and stable software and hardware adaptation with the CPU, including but not limited to adaptation in BIOS, firmware, transmission protocol, SI, PI, etc.

[0031] 2. Memory replication is required between the network card and the system memory DRAM, and between the system memory DRAM and the persistent memory PMEM, which increases storage latency.

[0032] 3. The persistent memory PMEM has a single supply and has been announced to be discontinued in the future, which cannot meet the supply.

[0033] 4. Persistent memory PMEM can only be used for CPUs of specified manufacturers, and requires the CPU to comply with it and cannot be universally adapted to CPUs of all manufacturers.

[0034] To this end, this embodiment provides an innovative network card device to shorten the persistent storage delay on the storage server side. The network card device provided by this embodiment can be installed on each storage server in the storage cluster. Figure 3 A schematic diagram of the structure of a network card device and a schematic diagram of its assembly on a storage server are provided for an exemplary embodiment of the present application. Figure 3 The network card device provided in this embodiment can be installed on the storage server in the form of bus PCIe interconnection. The network card device provided in this embodiment can be connected based on its communication component using the standard Ethernet protocol. Figure 1 A network switch in the network to receive / send data over the network.

[0035] refer to Figure 3 The network card device provided in this embodiment may include a processor, a communication component and a non-volatile cache. In this embodiment, the communication component in the network card device may retain the original network card communication function, including but not limited to the network interface NIC and the media access control MAC, etc., which will not be described in detail here. Figure 3 The non-volatile cache is a component added to the network card device in this embodiment, and the non-volatile cache can be coupled to the processor in the network card device through the bus PCIe. In practical applications, the assembly of the non-volatile cache can be achieved by patching on the circuit board of the network card device or by plugging in an expansion card. In addition, in the network card device provided in this embodiment, in addition to the aforementioned basic functions such as NIC+MAC+PCIe, it can also provide enhanced functions such as multi-queue reading and writing and direct memory access DMA. Moreover, based on the independent processor in the network card device, in this embodiment, one or more computer instructions can be preset in the memory (such as system memory) of the network card device to enable its processor to execute the data storage method in this embodiment.

[0036] Based on this, reference Figure 3 , the network card device can use its communication component to receive the data to be stored. Figure 1 The multi-backup storage method of computing nodes mentioned in the above, the device to be stored here can be the backup data sent by the computing node. Of course, the computing node can also adopt other storage methods, which are not limited here. Here, the data to be stored received by the network card device can be understood as data that needs to be persistently stored based on the storage cluster.

[0037] refer to Figure 3, after receiving the data to be stored, the network card device no longer writes it into the system memory of the storage server as in the traditional solution, but writes it into the non-volatile cache in the online device. Compared with the traditional solution, this not only does not require the participation of the system memory in the storage server, but also effectively shortens the data writing path, so the data writing can be completed with low latency. Preferably, after the data to be stored is written into the non-volatile cache, the network card device can return a write success notification to the storage requester of the data to be stored (such as the aforementioned computing node). In actual applications, from the perspective of the storage requester, after receiving the write success notification, it will think that the data to be stored has been successfully written into the storage cluster, and the storage requester defaults that the storage cluster will autonomously perform subsequent persistent storage processes. Therefore, the shorter write path provided in this embodiment can feedback the write success notification to the storage requester with lower latency, thereby effectively improving the storage efficiency perceived by the storage requester.

[0038] Figure 4 An exemplary structural diagram of a network card device provided by an exemplary embodiment of the present application. Figure 4 , in addition to the relevant functional components in the communication components mentioned above (such as Figure 4 NIC, MAC, etc.), the network card device also has a non-volatile storage controller, such as Figure 4 As shown, the non-volatile storage controller can be connected to the network transmission path in a bypass manner. In this way, the non-volatile storage controller can intercept the data to be stored from the network transmission path, and write the data to be stored into the non-volatile cache by performing a write operation on the non-volatile cache.

[0039] Continue to refer Figure 3 In order to achieve persistent storage of the data to be stored, the network card device in this embodiment can also construct persistent data for the data to be stored. The construction logic can be preset in the processor of the network card device. Figure 2 In the traditional solution in the present embodiment, when constructing the data to be persisted, the CPU in the storage server is no longer required to participate, nor is it required to rely on the aforementioned persistent storage PMEM. Instead, the network card device constructs the data to be persisted for the data to be stored, which can effectively avoid the dependence on the CPU type. In this way, the persistent storage process on the storage server side can be completely decoupled from the CPU in the storage server. Therefore, the data storage solution implemented by the network card device in the present embodiment is compatible with various CPU types in the storage server.

[0040] In addition, since the data to be stored is stored in the non-volatile cache of the network card device, compared with the PCU in the storage server, the processor of the network card device is closer to the data to be stored, so higher construction efficiency can be obtained.

[0041] In this embodiment, the construction mechanism of the data to be persisted is not limited, and the data to be persisted can be constructed by using an erasure coding mechanism or a multiple backup mechanism as needed. The detailed implementation method will be described in detail later.

[0042] After completing the construction of the data to be persisted, refer to Figure 3 , the network card device can write one of the multiple copies of data to be persisted into the storage disk on the storage server; and distribute the other copies of data to be persisted to the network card devices installed on other storage servers. The network card devices installed on other storage servers can write the data to be persisted received by themselves into the storage disk of the storage server where they are located. In this embodiment, the storage disk in the storage server can be a solid-state hard disk SSD, etc., and this embodiment does not limit the type of storage disk.

[0043] It should be understood that in actual applications, it is usually necessary to build multiple copies of the data to be persisted for the data to be stored, and store the multiple copies of the data to be persisted on different storage servers in the storage cluster to ensure that the data to be stored can still be completely restored in the event of failures such as partial server downtime or data loss.

[0044] Preferably, in this embodiment, direct memory access can be used to write one of the multiple copies of data to be persisted into the memory of the storage disk on the storage server where the network card device is located. Direct memory access means that the network card device directly writes the copy of the data to be persisted into the memory of the storage disk without the participation of the CPU in the storage server where the network card device is located.

[0045] In this preferred solution, both the network card device and the storage disk are used as bus PCIe devices of the storage server. Based on this, the network card device can support direct memory access to the storage disk by establishing a direct connection relationship between the two PCIe devices. In an exemplary solution: the memory address space of the storage disk on the storage server can be mapped to the logical address space of the network card device. Among them, the memory address space of the storage disk is the BAR (Base Address Register) address space. In this way, through the BAR address space mapping technology, the memory address space of the storage disk can be exposed to the network card device, and the network card device can initiate read and write operations on the memory address space of the storage disk like accessing the local storage space. Based on this, the network card device can perform a write operation on the memory address space of the storage disk based on the mapping relationship between the memory address space of the storage disk and the logical address space of the network card device, so as to write one of the multiple copies of the data to be persisted to the memory in the storage disk on the storage server. Of course, other solutions can also be used in this embodiment to realize the direct connection between the two PCIe devices of the network card device and the storage disk, and no more examples are given here.

[0046] The storage disk in the storage server usually uses flash memory as a persistent storage medium. Based on this, the storage disk can also continue to write the data to be persisted written in its memory into its flash memory. Among them, flash memory can also be called flash memory, which is a non-volatile storage device, usually composed of flash memory chips. It uses electronic storage technology to store data in internal flash memory units; unlike memory, the data in flash memory remains unchanged after power failure. In practical applications, the access unit of flash memory is usually a data group batch. For this reason, after the data to be persisted in the memory of the storage disk reaches the data group specification required by its flash memory, the persistent data in the memory of the storage disk can be spliced ​​into data groups and written to the flash memory.

[0047] After other copies of data to be persisted are distributed to network card devices installed on other storage servers, these network card devices can also write the data to be persisted they receive directly into the storage disk on the storage server according to the above-mentioned direct memory access method.

[0048] In the above, the network card device writes the data to be persisted directly to the storage disk of the storage server where it is located through direct memory access. There is no need for memory replication or the CPU participation of the storage server, which can effectively shorten the write delay and reduce the overhead of memory replication.

[0049] In summary, in this embodiment, an innovative network card device can be installed on each storage server in the storage cluster, and a non-volatile cache is provided in the network card device. On this basis, after receiving the data to be stored, the network card device can write the data to be stored into the non-volatile cache thereon, which can effectively shorten the response delay to the corresponding storage request; thereafter, the network card device can construct multiple copies of the data to be persisted for persistent storage for the data to be stored, and directly write one of the copies of the data to be persisted to the storage disk on the storage server where it is located by the network card device, and distribute the other copies of the data to be persisted to the network card devices installed on other storage servers in the storage cluster, so that these network card devices can respectively write the received data to be persisted into the storage disk on the storage server where they are located, thereby completing persistent storage. It can be seen that in the persistent storage process, the dependence on the CPU in the storage server is decoupled, and multiple memory copies are no longer required in the storage server. Instead, the network card device directly transfers data and completes the construction and processing of the data to be persisted nearby. This can effectively reduce the storage delay on the storage server side, thereby effectively improving the persistent storage efficiency of the storage cluster.

[0050] In the above or below embodiments, the network card device may use a variety of implementation methods to construct data to be persisted for data to be stored. Figure 5A logical schematic diagram of an optional implementation scheme for constructing data to be persisted in a network card device is provided for an exemplary embodiment of the present application.

[0051] refer to Figure 5 In this implementation, an erasure code mechanism is used to construct the data to be persisted: the data to be stored can be read from the non-volatile cache; the data to be stored is subjected to erasure code encoding processing to obtain multiple erasure code slices corresponding to the data to be stored, which are respectively used as the data to be persisted. The erasure code mechanism may refer to: dividing the data to be stored into k original data, and adding m redundant data to the k original data; on this basis, the k original data can be restored through any k data among the k+m parts.

[0052] refer to Figure 4 , a computation offloading unit may be pre-installed in the processor of the network card device, and an erasure code encoder may be deployed in the computation offloading unit to perform erasure code encoding processing on the data to be stored.

[0053] In this implementation, the non-volatile cache in the network card device can adopt a sequential read-write mode, which can also be called a log read-write mode. In this mode, the non-volatile cache can be regarded as a continuous address space for sequential read and write. The write pointer points to the current write address and stores the latest data being written. The read pointer points to the current read address, and the read data will be used for erasure coding and subsequent processing. The data that has been read will no longer be retained in the non-volatile cache, so refer to Figure 5 ,The data between the read pointer and the write pointer is the current valid data and needs to be encoded with erasure code. This sequential read and write mode can ensure the accuracy of data reading and writing in the non-volatile cache.

[0054] Considering that the read / write unit of the non-volatile cache may not match the amount of data to be stored, that is, it may not be possible to read the complete data to be stored from the non-volatile cache at one time, for this reason, a data temporary storage area may also be set in the network card device in this embodiment. In actual applications, the data temporary storage area may be implemented by a dynamic random access memory, etc., and this embodiment does not limit the hardware type used by the data temporary storage area.

[0055] Based on this, in this optional implementation, on the one hand, the network card device can write the data to be stored into the non-volatile cache in a sequential write manner. On the other hand, after the data to be stored is written, the network card device can read the data from the non-volatile cache into the data temporary storage area in a sequential read manner; after the data to be stored is read into the data temporary storage area, the data to be stored is extracted from the data temporary storage area to perform erasure coding processing operations. Figure 4After the data to be stored is completely read into the data temporary storage area, the data to be stored can be transmitted to the erasure code encoder to generate multiple erasure code slices for the data to be stored in the erasure code encoder, respectively serving as the data to be persisted.

[0056] It should be understood that the above-mentioned implementation method of using the erasure code mechanism to construct the data to be persisted is illustrative. In this embodiment, the network card device can also use other implementation methods to realize the construction of the data to be persisted. For example, a multi-backup mechanism can be used to construct the data to be persisted. Under this mechanism, a backup generator can be deployed in the computing offloading unit of the aforementioned network card device to generate multiple backup data for the data to be stored, each as the data to be persisted. No more examples are given here. Reference Figure 4 ,Compression / decompression function modules, network offload engines, etc. can also be deployed in the computing offload unit of the network card device as optional configuration items. Configurable processing modules can also be deployed to provide optional functions such as data filtering and data format preprocessing to more effectively serve scenarios such as big data, reduce the amount of data read and transmitted, and provide near storage computing capabilities.

[0057] In this embodiment, an improved solution for the distribution process of data to be persisted is also provided.

[0058] In this improved solution, based on the idea of ​​setting a data temporary storage area in the network card device mentioned above, reference Figure 5 , data block buffers can be set up separately for different storage servers in the data temporary storage area.

[0059] Based on this, the network card device can obtain the persistent storage address information corresponding to each of the multiple copies of the data to be persisted, and the persistent storage location information indicates the identification of the storage server; the multiple copies of the data to be persisted corresponding to the storage device are written into the data block buffer in the data temporary storage area of ​​the corresponding storage server; when the data to be persisted in any data block buffer reaches the transmission volume requirement, the data to be persisted in the data block buffer is encapsulated as a data block and transmitted to the network card device installed on the corresponding storage server.

[0060] In this way, each data block buffer can store multiple copies of data to be persisted from different data to be stored; and from the perspective of the data to be stored, the corresponding multiple copies of data to be persisted will be distributed to multiple data block caches.

[0061] refer to Figure 5When the data to be persisted in the data block buffer corresponding to the storage server where the network card device is located reaches the transmission volume requirement, the network card device can be triggered to encapsulate the data to be persisted in the data block buffer into data blocks, and write them into the memory of the storage disk in the storage server through the aforementioned direct memory access and other methods. After the data blocks in the memory accumulate to the specified data group specification, these data blocks can be spliced ​​into a data group batch and written into the flash memory.

[0062] refer to Figure 5 When the data to be persisted in the data block buffer corresponding to other storage servers reaches the transmission volume requirement, a network transmission can be triggered, that is, the data to be persisted in the data block buffer is encapsulated into data blocks, and the data blocks are transmitted to the network card device installed on the corresponding storage server through the network switch.

[0063] In this improved solution, by setting a data block buffer, persistent data can be transmitted in blocks, thereby making full use of network bandwidth, reducing the number of network transmissions, and reducing the network overhead within the storage cluster.

[0064] Figure 6 A flowchart of a data storage method provided for another exemplary embodiment of the present application. The data storage method can be implemented by a network card device, in which a non-volatile cache is provided, and the network card device can be installed on a storage server in a storage cluster. Figure 6 , the method may include:

[0065] Step 600: writing the received data to be stored into the non-volatile cache;

[0066] Step 601: construct multiple copies of the data to be stored for persistent storage for the data to be stored;

[0067] Step 602: write one copy of the data to be persisted among the multiple copies of the data to be persisted into a storage disk on the storage server;

[0068] Step 603: distribute other copies of the data to be persisted to the network card devices installed on other storage servers, so that the network card devices installed on the other storage servers can write the received data to be persisted into the storage disk of the storage server where they are located.

[0069] In an optional embodiment, the method may further include:

[0070] After the data to be stored is written into the non-volatile cache, a write success notification is returned to the storage requester of the data to be stored.

[0071] In an optional embodiment, for the data to be stored, multiple copies of the data to be persisted for persistent storage are constructed, including:

[0072] Reading the data to be stored from the non-volatile cache;

[0073] Erasure coding is performed on the data to be stored to obtain multiple erasure code slices corresponding to the data to be stored, which are respectively used as data to be persisted.

[0074] In an optional embodiment, the non-volatile cache adopts a sequential read-write mode, and the network card device is further provided with a data temporary storage area, and the received data to be stored is written into the non-volatile cache, including:

[0075] Writing the data to be stored into the non-volatile cache in a sequential writing manner;

[0076] Reading the data to be stored from the non-volatile cache includes:

[0077] Reading data from the non-volatile cache to the data temporary storage area in a sequential reading manner;

[0078] After the data to be stored is read into the data temporary storage area, the data to be stored is extracted from the data temporary storage area to perform the erasure code encoding processing operation.

[0079] In an optional embodiment, a data temporary storage area is set in the network card device, and data block buffers are respectively set for different storage servers in the data temporary storage area, and the method further includes:

[0080] Determine persistent storage address information corresponding to each of the plurality of copies of data to be persisted, wherein the persistent storage location information indicates an identifier of a storage server;

[0081] Writing the plurality of copies of data to be persisted into the data block buffer of the corresponding storage server in the data temporary storage area respectively;

[0082] When the data to be persisted in any data block buffer reaches the transmission volume requirement, the data to be persisted in the data block buffer is encapsulated as data blocks and transmitted to the network card device equipped with the corresponding storage server.

[0083] In an optional embodiment, the data temporary storage area uses a dynamic random access memory.

[0084] In an optional embodiment, writing one copy of the plurality of copies of data to be persisted to a storage disk on the storage server includes:

[0085] By adopting a direct memory access method, one copy of the plurality of copies of data to be persisted is written into the memory of the storage disk on the storage server, so that the storage disk continues to write the persistent data in its memory into its flash memory.

[0086] In an optional embodiment, the method may further include:

[0087] Mapping the memory address space of the storage disk on the storage server to the logical address space of the network card device;

[0088] The method of writing one copy of the plurality of copies of data to be persisted into the memory of the storage disk on the storage server by using direct memory access includes:

[0089] Based on the mapping relationship between the memory address space of the storage disk and the logical address space of the network card device, a write operation is performed on the memory address space of the storage disk to write one of the multiple copies of data to be persisted into the memory of the storage disk on the storage server.

[0090] In an optional embodiment, after the data to be persisted in the memory of the storage disk reaches the data group specification required by its flash memory, the data to be persisted in the memory of the storage disk is spliced ​​into a data group and written into the flash memory.

[0091] In summary, in this embodiment, the data storage method can be roughly divided into two steps: writing the data to be stored into the storage cluster and completing the persistence of the data to be stored in the storage cluster.

[0092] Figure 7a An exemplary flow chart of writing data to be stored into a storage cluster is provided. Figure 7a In this link, after the multiple copies of backup data submitted by the computing node enter the network card devices installed on each storage server, the network card device can write the backup data (as the data to be stored) into the non-volatile cache integrated in the network card device in a sequential write manner, and return a write success notification to the computing node. When all the backup data submitted by the computing node are written successfully, the storage request initiated by the computing node this time will be marked as successful; if the write timeout is still unsuccessful, the computing node can re-initiate the storage request, and the multiple servers re-selected may not be completely consistent with the previous attempt. In other words, when the computing node initiates a storage request again, the multiple servers selected for writing are not exactly the same as the last time. When the data is retried and written successfully, this write operation is successful, and the computing node can initiate subsequent storage requests.

[0093] Figure 7b Provides an exemplary process diagram for completing the persistence of data to be stored in a storage cluster. Figure 7b , select one of the storage servers that have been written to multiple backups as the primary node, and the network card device on the storage server reads the data to be stored from its internal non-volatile cache. After that, the data to be stored is processed by erasure coding to construct multiple erasure code slices for it, and can be distributed to the n storage servers corresponding to this erasure code storage group (one of which is in the local SSD), and the local erasure code slice can be transmitted to the SSD via direct memory access. After receiving the erasure code slice, the network card devices of the remaining n-1 storage servers can also write it to the local SSD of the storage server where it is located via direct memory access. If the n erasure code slices corresponding to the data to be stored are all successfully written, it is determined that the persistent storage is successful, and the corresponding data in the non-volatile cache can be deleted, and its capacity can be used for subsequent data writing. However, if the persistent storage is unsuccessful and times out, the primary node restarts the above-mentioned persistent storage logic, that is, repeating the whole process of reading the non-volatile cache, erasure code encoding, network transmission, local writing, etc., until the n erasure code slices are successfully written.

[0094] It is worth noting that the technical details of the above data storage method can be referred to the relevant description in the aforementioned network card device embodiment. In order to save space, they will not be repeated here, but this should not cause loss of the protection scope of this application.

[0095] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement each step that can be executed by the network card device in the above method embodiment.

[0096] It should be noted that in some of the processes described in the above embodiments and the accompanying drawings, multiple operations appearing in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this document or may be executed in parallel, and the sequence numbers of the operations, such as 601, 602, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel.

[0097] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0098] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0099] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0100] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0101] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0102] The above is only the embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A data storage method, applicable to a network card device mounted on a storage server in a storage cluster, wherein the network card device is provided with a non-volatile cache, the method comprising: Writing the received data to be stored into the non-volatile cache; Constructing multiple copies of the data to be persisted for persistent storage for the data to be stored; Writing one copy of the data to be persisted among the multiple copies of the data to be persisted into a storage disk on the storage server; The other copies of the data to be persisted are distributed to the network card devices installed on the other storage servers, so that the network card devices installed on the other storage servers respectively write the received data to be persisted into the storage disk of the storage server where they are located.

2. The method according to claim 1, further comprising: After the data to be stored is written into the non-volatile cache, a write success notification is returned to the storage requester of the data to be stored.

3. The method according to claim 1, constructing multiple copies of the data to be stored for persistent storage for the data to be stored, comprising: Reading the data to be stored from the non-volatile cache; Erasure coding is performed on the data to be stored to obtain multiple erasure code slices corresponding to the data to be stored, which are respectively used as data to be persisted.

4. The method according to claim 3, wherein the non-volatile cache adopts a sequential read-write mode, and the network card device is further provided with a data temporary storage area, and the received data to be stored is written into the non-volatile cache, comprising: Writing the data to be stored into the non-volatile cache in a sequential writing manner; Reading the data to be stored from the non-volatile cache includes: Reading data from the non-volatile cache to the data temporary storage area in a sequential reading manner; After the data to be stored is read into the data temporary storage area, the data to be stored is extracted from the data temporary storage area to perform the erasure code encoding processing operation.

5. The method according to claim 1, wherein a data temporary storage area is set in the network card device, and data block buffers are respectively set for different storage servers in the data temporary storage area, and the method further comprises: Determine persistent storage address information corresponding to each of the plurality of copies of data to be persisted, wherein the persistent storage location information indicates an identifier of a storage server; Writing the plurality of copies of data to be persisted into the data block buffer of the corresponding storage server in the data temporary storage area respectively; When the data to be persisted in any data block buffer reaches the transmission volume requirement, the data to be persisted in the data block buffer is encapsulated as data blocks and transmitted to the network card device equipped with the corresponding storage server.

6. The method according to any one of claims 4 or 5, wherein the data temporary storage area uses a dynamic random access memory.

7. The method according to claim 1, writing one copy of the plurality of copies of data to be persisted into a storage disk on the storage server, comprising: By adopting a direct memory access method, one copy of the plurality of copies of data to be persisted is written into the memory of the storage disk on the storage server, so that the storage disk continues to write the persistent data in its memory into its flash memory.

8. The method according to claim 7, further comprising: Mapping the memory address space of the storage disk on the storage server to the logical address space of the network card device; The method of writing one copy of the plurality of copies of data to be persisted into the memory of the storage disk on the storage server by using direct memory access includes: Based on the mapping relationship between the memory address space of the storage disk and the logical address space of the network card device, a write operation is performed on the memory address space of the storage disk to write one of the multiple copies of data to be persisted into the memory of the storage disk on the storage server.

9. According to the method of claim 7, after the data to be persisted in the memory of the storage disk reaches the data group specification required by its flash memory, the data to be persisted in the memory of the storage disk is spliced ​​into a data group and written into the flash memory.

10. A network card device, comprising a memory, a processor, a communication component and a non-volatile cache; The memory is used to store one or more computer instructions; The processor is coupled to the memory, the communication component and the non-volatile cache, and is configured to execute the one or more computer instructions to execute the data storage method according to any one of claims 1 to 9.

11. A storage cluster, comprising a plurality of storage servers, each storage server being equipped with a network card device, the network card device being used to execute the data storage method according to any one of claims 1 to 9, so as to persistently store the data to be stored in the storage cluster.

12. A computer-readable storage medium storing computer instructions, which, when executed by one or more processors, causes the one or more processors to execute the data storage method according to any one of claims 1 to 9.

Images