A data storage method, device, equipment and computer readable storage medium

Abstract

The application discloses a data storage method, device and equipment and a computer readable storage medium, and belongs to the field of distributed storage and is used for storing data to a distributed storage system. If erasure correction calculation is performed on a storage node, all the data to be stored needs to be first sent to a master storage node, and the master storage node needs to distribute the data to be stored after the erasure correction calculation to other storage nodes, so that the data transmission amount is large. The application transfers the erasure correction calculation work to the client of the distributed storage system, and then can transmit each piece of data to be stored to the multiple storage nodes of the distributed storage system one by one, so that the process of forwarding by the master storage node is omitted, the data transmission amount is reduced, and the storage efficiency and user experience are improved.

Description

Technical Field

[0001] This invention relates to the field of distributed storage, and in particular to a data storage method. This invention also relates to a data storage device, apparatus, and computer-readable storage medium. Background Technology

[0002] Distributed storage systems are commonly used storage systems that can distribute data across multiple independent devices. Distributed storage systems adopt a scalable system architecture and utilize multiple storage servers to share the storage load. This not only improves the system's reliability, availability, and access efficiency, but also makes it easy to expand.

[0003] The client can store data to a distributed storage system. However, there is a lack of mature data storage methods in the current technology, which results in low efficiency when storing data to a distributed storage system through the client, thus affecting the user experience.

[0004] Therefore, how to provide a solution to the above-mentioned technical problems is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to provide a data storage method that transfers erasure calculations to the client side of a distributed storage system. Each copy of data to be stored can then be transmitted one-to-one to multiple storage nodes in the distributed storage system, eliminating the forwarding process handled by the primary storage node. This reduces data transmission volume and improves storage efficiency and user experience. Another purpose of this invention is to provide a data storage device, apparatus, and computer-readable storage medium that transfers erasure calculations to the client side of a distributed storage system. Each copy of data to be stored can then be transmitted one-to-one to multiple storage nodes in the distributed storage system, eliminating the forwarding process handled by the primary storage node. This reduces data transmission volume and improves storage efficiency and user experience.

[0006] To address the aforementioned technical problems, this invention provides a data storage method applied to a client of a distributed storage system, comprising:

[0007] Perform erasure calculations on the data to be stored;

[0008] The data to be stored, after correction calculation, is divided into a preset number of parts;

[0009] The divided data to be stored is transmitted one by one to multiple storage nodes of the distributed storage system.

[0010] Preferably, the deletion calculation for the data to be stored specifically includes:

[0011] Request remote direct data access (RDMA) memory;

[0012] Copy the data to be stored to the requested RDMA memory;

[0013] Perform erasure calculations on the data to be stored located in the RDMA memory;

[0014] The specific steps of transmitting the divided portions of the data to be stored to multiple storage nodes of the distributed storage system in a one-to-one correspondence are as follows:

[0015] The storage addresses of each copy of the data to be stored in the RDMA memory are sent to the distributed storage system, so that the distributed storage system can control its multiple storage nodes to read each copy of the data to be stored one by one.

[0016] Preferably, copying the data to be stored to the requested RDMA memory specifically involves:

[0017] The data to be stored is copied to the allocated RDMA memory using multiple copy threads.

[0018] Preferably, the step of performing erasure calculations on the data to be stored located in the RDMA memory specifically involves:

[0019] Multiple computation threads, distinct from the copy thread, are used to perform erasure calculations on the data to be stored in the RDMA memory.

[0020] Preferably, the step of using multiple computation threads, distinct from the copy thread, to perform erasure calculations on the data to be stored in the RDMA memory specifically involves:

[0021] When using multiple copy threads to copy the data to be stored to the allocated RDMA memory, it is determined whether the RDMA memory has received new unit data from the data to be stored.

[0022] If received, the idle computing thread among the multiple computing threads that are different from the copy thread is used to perform erasure calculations on the latest received unit data in the RDMA memory;

[0023] Determine whether all unit data in the data to be stored has completed the deletion calculation;

[0024] If all steps have been completed, then the erasure calculation of the data to be stored is considered complete.

[0025] Preferably, the data storage method further includes:

[0026] When it is unloaded, all the copy threads and computation threads are released.

[0027] Preferably, the number of computing threads is greater than the number of copying threads.

[0028] To address the aforementioned technical problems, the present invention also provides a data storage device, applied to a client of a distributed storage system, comprising:

[0029] The calculation module is used to perform erasure calculations on the data to be stored;

[0030] The segmentation module is used to divide the data to be stored, after erasure calculation, into a preset number of parts;

[0031] The transmission module is used to transmit the divided portions of the data to be stored to multiple nodes of the distributed storage system in a one-to-one correspondence.

[0032] To address the aforementioned technical problems, the present invention also provides a data storage device, comprising:

[0033] Memory, used to store computer programs;

[0034] A processor for implementing the data storage method described above when executing the computer program.

[0035] To address the aforementioned technical problems, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the data storage method described above.

[0036] This invention provides a data storage method. Considering that if erasure calculation is performed on the storage node, all data to be stored must first be sent to the main storage node. After the main storage node completes the erasure calculation, it also needs to distribute the erasure-processed data to other storage nodes, resulting in a large data transmission volume. This application transfers the erasure calculation work to the client of the distributed storage system. Subsequently, each piece of data to be stored can be transmitted one-to-one to multiple storage nodes of the distributed storage system, eliminating the forwarding process by the main storage node, thereby reducing the data transmission volume and improving storage efficiency and user experience.

[0037] The present invention also provides a data storage device, apparatus, and computer-readable storage medium, which have the same beneficial effects as the data storage method described above. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the prior art and embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 A flowchart illustrating a data storage method provided by the present invention;

[0040] Figure 2 A schematic diagram of the structure of a data storage device provided by the present invention;

[0041] Figure 3 This is a schematic diagram of the structure of a data storage device provided by the present invention. Detailed Implementation

[0042] The core of this invention is to provide a data storage method that transfers erasure calculations to the client side of a distributed storage system. Each copy of data to be stored can then be transmitted one-to-one to multiple storage nodes in the distributed storage system, eliminating the forwarding process handled by the primary storage node. This reduces data transmission volume and improves storage efficiency and user experience. Another core aspect of this invention is to provide a data storage device, apparatus, and computer-readable storage medium that transfers erasure calculations to the client side of a distributed storage system. Each copy of data to be stored can then be transmitted one-to-one to multiple storage nodes in the distributed storage system, eliminating the forwarding process handled by the primary storage node. This reduces data transmission volume and improves storage efficiency and user experience.

[0043] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a data storage method provided by the present invention. This data storage method is applied to a client of a distributed storage system and includes:

[0045] S101: Perform erasure calculations on the data to be stored;

[0046] Specifically, considering the technical problems mentioned above, and taking into account that if erasure calculation is performed on the storage node, all data to be stored must first be sent to the primary storage node, and after the primary storage node completes the erasure calculation, it also needs to distribute the erasure-calculated data to be stored to other storage nodes, resulting in a large data transmission volume, this application aims to transfer the erasure calculation work to the client of the distributed storage system to reduce the data forwarding volume. Therefore, the data storage method in this embodiment of the invention is applied to the client of the distributed storage system. In this step, the erasure calculation is first performed on the data to be stored, that is, the erasure calculation is completed in the client.

[0047] The erasure calculation is a necessary calculation process before data is stored in a distributed storage system. The specific process of erasure calculation will not be described in detail in this embodiment of the invention.

[0048] S102: Divide the data to be stored after erasure calculation into a preset number of parts;

[0049] Specifically, the inherent characteristic of a distributed storage system is that the data to be stored is stored on multiple storage nodes. In this embodiment of the invention, it is intended to directly transmit each copy of the data to be stored from the client to the corresponding storage node. Therefore, in this step, the data to be stored after erasure calculation can be divided into a preset number of copies.

[0050] The preset number of copies can be set independently, and its quantity can be the number of storage nodes in the distributed storage system. This embodiment of the invention does not limit this quantity.

[0051] S103: Transmit the divided data to be stored to multiple storage nodes of the distributed storage system one by one.

[0052] Specifically, after the data is split, each of the split data to be stored can be transmitted one by one to multiple storage nodes of the distributed storage system. For example, the first data to be stored can be transmitted directly to the first storage node, the second data to be stored can be transmitted directly to the second storage node, and so on.

[0053] Because the erasure calculation and segmentation actions are performed on the client side in this embodiment of the invention, each copy of the data to be stored can be directly sent to each storage node, saving a lot of data forwarding, thereby improving data storage efficiency and user experience.

[0054] In this embodiment of the invention, the client can be of various types, such as a kernel-mode client, etc., and this embodiment of the invention does not limit it.

[0055] This invention provides a data storage method. Considering that if erasure calculation is performed on the storage node, all data to be stored must first be sent to the main storage node. After the main storage node completes the erasure calculation, it also needs to distribute the erasure-processed data to other storage nodes, resulting in a large data transmission volume. This application transfers the erasure calculation work to the client of the distributed storage system. Subsequently, each piece of data to be stored can be transmitted one-to-one to multiple storage nodes of the distributed storage system, eliminating the forwarding process by the main storage node, thereby reducing the data transmission volume and improving storage efficiency and user experience.

[0056] Based on the above embodiments:

[0057] As a preferred embodiment, the erasure calculation for the data to be stored specifically involves:

[0058] Request RDMA (Remote Direct Memory Access) memory;

[0059] Copy the data to be stored to the requested RDMA memory;

[0060] Perform erasure calculations on the data to be stored in RDMA memory;

[0061] The specific steps involved are as follows: Each segment of the data to be stored is transmitted one-to-one to multiple storage nodes in the distributed storage system.

[0062] The storage addresses of each piece of data to be stored in RDMA memory are sent to the distributed storage system so that the distributed storage system can control its multiple storage nodes to read each piece of data to be stored one by one.

[0063] Specifically, considering that if the client actively sends each piece of data to be stored to multiple storage nodes of the distributed storage system, the client needs to provide resources to continuously send data and monitor the data transmission results, which increases the consumption of client computing resources. Therefore, in this embodiment of the invention, remote direct data access (RDMA) memory can be requested first, the data to be stored can be copied to the requested RDMA memory, and then erasure calculations can be performed on the data to be stored in the RDMA memory. At this step, the storage addresses of each piece of data to be stored in the RDMA memory can be sent to the distributed storage system. The distributed storage system can then notify its multiple storage nodes to retrieve the data from the storage address of the corresponding piece of data to be stored based on the received storage address. The client no longer needs to provide computing resources to continuously send data and monitor the data transmission results, thus reducing the consumption of client computing resources.

[0064] As a preferred embodiment, copying the data to be stored to the requested RDMA memory specifically involves:

[0065] Multiple copy threads are used to copy the data to be stored to the allocated RDMA memory.

[0066] Specifically, in order to further improve data storage efficiency, in this embodiment of the invention, multiple copy threads can be used to copy the data to be stored to the allocated RDMA memory simultaneously.

[0067] The specific number of copy threads can be set independently, and this embodiment of the invention does not limit it.

[0068] As a preferred embodiment, the erasure calculation for the data to be stored located in RDMA memory specifically involves:

[0069] Multiple computation threads, distinct from the copy thread, are used to perform erasure calculations on the data to be stored in RDMA memory.

[0070] Specifically, in order to further improve data storage efficiency, in this embodiment of the invention, multiple computing threads can be used to perform erasure calculations on the data to be stored in RDMA memory at the same time.

[0071] The specific number of computing threads can be set independently, and this embodiment of the invention does not limit it.

[0072] As a preferred embodiment, the specific method of using multiple computation threads, distinct from the copy thread, to perform erasure calculations on the data to be stored in RDMA memory is as follows:

[0073] When using multiple copy threads to copy the data to be stored to the allocated RDMA memory, determine whether the RDMA memory has received new unit data from the data to be stored.

[0074] If received, the idle calculation thread among the multiple calculation threads that are different from the copy thread is used to perform erasure calculations on the latest received unit data in the RDMA memory;

[0075] Determine whether all unit data in the data to be stored has undergone deletion calculation;

[0076] If all steps have been completed, then the erasure calculation for the data to be stored is considered complete.

[0077] Specifically, in order to further improve data storage efficiency, the copy thread and the calculation thread in this embodiment of the invention can work synchronously. That is, it is not necessary to wait until the copy thread has copied all the data to be stored before controlling the calculation thread to work. Whenever a new unit of data is received in the RDMA memory, the idle calculation thread can immediately start working.

[0078] As a preferred embodiment, the data storage method further includes:

[0079] When it is unloaded, it releases all copy threads and computation threads.

[0080] Specifically, considering that if the copying and computing threads still exist when the client is uninstalled, then computing resources are wasted. In order to ensure the reuse of computing resources after the client is uninstalled, the client in this embodiment of the invention can release all copying and computing threads when it is uninstalled.

[0081] In a preferred embodiment, the number of computation threads is greater than the number of copy threads.

[0082] Specifically, considering that the execution speed of copying is usually faster than that of erasure calculation, in order to reduce the waste of threads, the number of calculation threads in this embodiment of the invention can be greater than the number of copying threads, thereby minimizing the occupation of threads while ensuring the efficient execution of data storage.

[0083] Of course, besides this type of quantitative relationship, the quantitative relationship between computing threads and copying threads can also be of many other types, which are not limited in this embodiment of the invention.

[0084] Please refer to Figure 2 , Figure 2 This is a schematic diagram of a data storage device provided by the present invention. The data storage device is applied to a client side of a distributed storage system and includes:

[0085] Calculation module 21 is used to perform erasure calculations on the data to be stored;

[0086] The segmentation module 22 is used to segment the data to be stored after erasure calculation into a preset number of parts;

[0087] The transmission module 23 is used to transmit the divided data to be stored one by one to multiple nodes of the distributed storage system.

[0088] Specifically, considering the technical problems mentioned above, and taking into account that if erasure calculation is performed on the storage node, all data to be stored must first be sent to the primary storage node, and after the primary storage node completes the erasure calculation, it also needs to distribute the erasure-calculated data to be stored to other storage nodes, resulting in a large data transmission volume, this application aims to transfer the erasure calculation work to the client of the distributed storage system to reduce the data forwarding volume. Therefore, the data storage method in this embodiment of the invention is applied to the client of the distributed storage system. In this step, the erasure calculation is first performed on the data to be stored, that is, the erasure calculation is completed in the client.

[0089] The erasure calculation is a necessary calculation process before data is stored in a distributed storage system. The specific process of erasure calculation will not be described in detail in this embodiment of the invention.

[0090] Specifically, the inherent characteristic of a distributed storage system is that the data to be stored is stored on multiple storage nodes. In this embodiment of the invention, it is intended to directly transmit each copy of the data to be stored from the client to the corresponding storage node. Therefore, in this step, the data to be stored after erasure calculation can be divided into a preset number of copies.

[0091] The preset number of copies can be set independently, and its quantity can be the number of storage nodes in the distributed storage system. This embodiment of the invention does not limit this quantity.

[0092] Specifically, after the data is split, each of the split data to be stored can be transmitted one by one to multiple storage nodes of the distributed storage system. For example, the first data to be stored can be transmitted directly to the first storage node, the second data to be stored can be transmitted directly to the second storage node, and so on.

[0093] Because the erasure calculation and segmentation actions are performed on the client side in this embodiment of the invention, each copy of the data to be stored can be directly sent to each storage node, saving a lot of data forwarding, thereby improving data storage efficiency and user experience.

[0094] In this embodiment of the invention, the client can be of various types, such as a kernel-mode client, etc., and this embodiment of the invention does not limit it.

[0095] This invention provides a data storage method. Considering that if erasure calculation is performed on the storage node, all data to be stored must first be sent to the main storage node. After the main storage node completes the erasure calculation, it also needs to distribute the erasure-processed data to other storage nodes, resulting in a large data transmission volume. This application transfers the erasure calculation work to the client of the distributed storage system. Subsequently, each piece of data to be stored can be transmitted one-to-one to multiple storage nodes of the distributed storage system, eliminating the forwarding process by the main storage node, thereby reducing the data transmission volume and improving storage efficiency and user experience.

[0096] In a preferred embodiment, the computing module 21 includes:

[0097] The application module is used to apply for Remote Direct Data Access (RDMA) memory.

[0098] The copy module is used to copy the data to be stored to the requested RDMA memory;

[0099] The calculation submodule is used to perform erasure calculations on the data to be stored in RDMA memory;

[0100] The transmission module is specifically used for:

[0101] The storage addresses of each piece of data to be stored in RDMA memory are sent to the distributed storage system so that the distributed storage system can control its multiple storage nodes to read each piece of data to be stored one by one.

[0102] Specifically, considering that if the client actively sends each piece of data to be stored to multiple storage nodes of the distributed storage system, the client needs to provide resources to continuously send data and monitor the data transmission results, which increases the consumption of client computing resources. Therefore, in this embodiment of the invention, remote direct data access (RDMA) memory can be requested first, the data to be stored can be copied to the requested RDMA memory, and then erasure calculations can be performed on the data to be stored in the RDMA memory. At this step, the storage addresses of each piece of data to be stored in the RDMA memory can be sent to the distributed storage system. The distributed storage system can then notify its multiple storage nodes to retrieve the data from the storage address of the corresponding piece of data to be stored based on the received storage address. The client no longer needs to provide computing resources to continuously send data and monitor the data transmission results, thus reducing the consumption of client computing resources.

[0103] In a preferred embodiment, the copy module is specifically used for:

[0104] Multiple copy threads are used to copy the data to be stored to the allocated RDMA memory.

[0105] Specifically, in order to further improve data storage efficiency, in this embodiment of the invention, multiple copy threads can be used to copy the data to be stored to the allocated RDMA memory simultaneously.

[0106] The specific number of copy threads can be set independently, and this embodiment of the invention does not limit it.

[0107] In a preferred embodiment, the calculation submodule is specifically used for:

[0108] Multiple computation threads, distinct from the copy thread, are used to perform erasure calculations on the data to be stored in RDMA memory.

[0109] Specifically, in order to further improve data storage efficiency, in this embodiment of the invention, multiple computing threads can be used to perform erasure calculations on the data to be stored in RDMA memory at the same time.

[0110] The specific number of computing threads can be set independently, and this embodiment of the invention does not limit it.

[0111] In a preferred embodiment, the computing submodule includes:

[0112] The first judgment module is used to determine whether new unit data has been received in the RDMA memory when multiple copy threads are used to copy the data to be stored to the allocated RDMA memory. If it is received, the first execution module is triggered.

[0113] The first execution module is used to perform erasure calculations on the most recently received unit data in RDMA memory using an idle calculation thread among multiple calculation threads that are different from the copy thread;

[0114] The second judgment module is used to determine whether all unit data in the data to be stored has completed the deletion calculation. If all have been completed, the judgment module is triggered.

[0115] Specifically, in order to further improve data storage efficiency, the copy thread and the calculation thread in this embodiment of the invention can work synchronously. That is, it is not necessary to wait until the copy thread has copied all the data to be stored before controlling the calculation thread to work. Whenever a new unit of data is received in the RDMA memory, the idle calculation thread can immediately start working.

[0116] The determination module is used to determine whether the erasure calculation of the data to be stored has been completed.

[0117] As a preferred embodiment, the data storage manufacturing process further includes:

[0118] The release module is used to release all copy threads and computation threads when it is unloaded.

[0119] Specifically, considering that if the copying and computing threads still exist when the client is uninstalled, then computing resources are wasted. In order to ensure the reuse of computing resources after the client is uninstalled, the client in this embodiment of the invention can release all copying and computing threads when it is uninstalled.

[0120] In a preferred embodiment, the number of computation threads is greater than the number of copy threads.

[0121] Specifically, considering that the execution speed of copying is usually faster than that of erasure calculation, in order to reduce the waste of threads, the number of calculation threads in this embodiment of the invention can be greater than the number of copying threads, thereby minimizing the occupation of threads while ensuring the efficient execution of data storage.

[0122] Of course, besides this type of quantitative relationship, the quantitative relationship between computing threads and copying threads can also be of many other types, which are not limited in this embodiment of the invention.

[0123] Please refer to Figure 3 , Figure 3 This invention provides a schematic diagram of a data storage device, which includes:

[0124] Memory 31 is used to store computer programs;

[0125] The processor 32 is used to implement the steps of the data storage method as described in the foregoing embodiments when executing a computer program.

[0126] Specifically, the memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer-readable instructions, and the internal memory provides an environment for the operation of the operating system and computer-readable instructions in the non-volatile storage medium. When the processor executes the computer program stored in the memory, it can perform the following steps: perform erasure calculations on the data to be stored; divide the data to be stored after erasure calculations into a preset number of parts; and transfer each part of the data to be stored to multiple storage nodes of the distributed storage system in a one-to-one correspondence.

[0127] Specifically, considering the technical problems mentioned above, and taking into account that if erasure calculation is performed on the storage node, all data to be stored must first be sent to the primary storage node, and after the primary storage node completes the erasure calculation, it also needs to distribute the erasure-calculated data to be stored to other storage nodes, resulting in a large data transmission volume, this application aims to transfer the erasure calculation work to the client of the distributed storage system to reduce the data forwarding volume. Therefore, the data storage method in this embodiment of the invention is applied to the client of the distributed storage system. In this step, the erasure calculation is first performed on the data to be stored, that is, the erasure calculation is completed in the client.

[0128] The erasure calculation is a necessary calculation process before data is stored in a distributed storage system. The specific process of erasure calculation will not be described in detail in this embodiment of the invention.

[0129] Specifically, the inherent characteristic of a distributed storage system is that the data to be stored is stored on multiple storage nodes. In this embodiment of the invention, it is intended to directly transmit each copy of the data to be stored from the client to the corresponding storage node. Therefore, in this step, the data to be stored after erasure calculation can be divided into a preset number of copies.

[0130] The preset number of copies can be set independently, and its quantity can be the number of storage nodes in the distributed storage system. This embodiment of the invention does not limit this quantity.

[0131] Specifically, after the data is split, each of the split data to be stored can be transmitted one by one to multiple storage nodes of the distributed storage system. For example, the first data to be stored can be transmitted directly to the first storage node, the second data to be stored can be transmitted directly to the second storage node, and so on.

[0132] Because the erasure calculation and segmentation actions are performed on the client side in this embodiment of the invention, each copy of the data to be stored can be directly sent to each storage node, saving a lot of data forwarding, thereby improving data storage efficiency and user experience.

[0133] In this embodiment of the invention, the client can be of various types, such as a kernel-mode client, etc., and this embodiment of the invention does not limit it.

[0134] This invention provides a data storage method. Considering that if erasure calculation is performed on the storage node, all data to be stored must first be sent to the main storage node. After the main storage node completes the erasure calculation, it also needs to distribute the erasure-processed data to other storage nodes, resulting in a large data transmission volume. This application transfers the erasure calculation work to the client of the distributed storage system. Subsequently, each piece of data to be stored can be transmitted one-to-one to multiple storage nodes of the distributed storage system, eliminating the forwarding process by the main storage node, thereby reducing the data transmission volume and improving storage efficiency and user experience.

[0135] In a preferred embodiment, when the processor executes a computer subroutine stored in memory, it can perform the following steps:

[0136] Request remote direct data access (RDMA) memory;

[0137] Copy the data to be stored to the requested RDMA memory;

[0138] Perform erasure calculations on the data to be stored in RDMA memory;

[0139] The specific steps involved are as follows: Each segment of the data to be stored is transmitted one-to-one to multiple storage nodes in the distributed storage system.

[0140] The storage addresses of each piece of data to be stored in RDMA memory are sent to the distributed storage system so that the distributed storage system can control its multiple storage nodes to read each piece of data to be stored one by one.

[0141] In a preferred embodiment, when the processor executes a computer subroutine stored in memory, it can perform the following steps:

[0142] Multiple copy threads are used to copy the data to be stored to the allocated RDMA memory.

[0143] In a preferred embodiment, when the processor executes a computer subroutine stored in memory, it can perform the following steps:

[0144] Multiple computation threads, distinct from the copy thread, are used to perform erasure calculations on the data to be stored in RDMA memory.

[0145] In a preferred embodiment, when the processor executes a computer subroutine stored in memory, it can perform the following steps:

[0146] When using multiple copy threads to copy the data to be stored to the allocated RDMA memory, determine whether the RDMA memory has received new unit data from the data to be stored.

[0147] If received, the idle calculation thread among the multiple calculation threads that are different from the copy thread is used to perform erasure calculations on the latest received unit data in the RDMA memory;

[0148] Determine whether all unit data in the data to be stored has undergone deletion calculation;

[0149] If all steps have been completed, then the erasure calculation for the data to be stored is considered complete.

[0150] In a preferred embodiment, when the processor executes a computer subroutine stored in memory, it can perform the following steps:

[0151] When it is unloaded, it releases all copy threads and computation threads.

[0152] In a preferred embodiment, the number of computation threads is greater than the number of copy threads.

[0153] For a description of the data storage device provided in the embodiments of the present invention, please refer to the foregoing embodiments of the data storage method; the embodiments of the present invention will not be repeated here.

[0154] The present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the data storage method as described in the foregoing embodiments.

[0155] Specifically, the readable storage medium can include various media capable of storing program code, such as USB flash drives, external hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. The storage medium stores a computer program, which, when executed by a processor, performs the following steps: performing erasure calculations on the data to be stored; dividing the erasure-calculated data into a preset number of parts; and transmitting each part of the data to be stored, one-to-one, to multiple storage nodes in the distributed storage system.

[0156] This invention provides a data storage method. Considering that if erasure calculation is performed on the storage node, all data to be stored must first be sent to the main storage node. After the main storage node completes the erasure calculation, it also needs to distribute the erasure-processed data to other storage nodes, resulting in a large data transmission volume. This application transfers the erasure calculation work to the client of the distributed storage system. Subsequently, each piece of data to be stored can be transmitted one-to-one to multiple storage nodes of the distributed storage system, eliminating the forwarding process by the main storage node, thereby reducing the data transmission volume and improving storage efficiency and user experience.

[0157] In a preferred embodiment, when the processor executes a computer subroutine stored in memory, it can perform the following steps:

[0158] Request remote direct data access (RDMA) memory;

[0159] Copy the data to be stored to the requested RDMA memory;

[0160] Perform erasure calculations on the data to be stored in RDMA memory;

[0161] The specific steps involved are as follows: Each segment of the data to be stored is transmitted one-to-one to multiple storage nodes in the distributed storage system.

[0162] The storage addresses of each piece of data to be stored in RDMA memory are sent to the distributed storage system so that the distributed storage system can control its multiple storage nodes to read each piece of data to be stored one by one.

[0163] In a preferred embodiment, when the processor executes a computer subroutine stored in memory, it can perform the following steps:

[0164] Multiple copy threads are used to copy the data to be stored to the allocated RDMA memory.

[0165] In a preferred embodiment, when the processor executes a computer subroutine stored in memory, it can perform the following steps:

[0166] Multiple computation threads, distinct from the copy thread, are used to perform erasure calculations on the data to be stored in RDMA memory.

[0167] In a preferred embodiment, when the processor executes a computer subroutine stored in memory, it can perform the following steps:

[0168] When using multiple copy threads to copy the data to be stored to the allocated RDMA memory, determine whether the RDMA memory has received new unit data from the data to be stored.

[0169] If received, the idle calculation thread among the multiple calculation threads that are different from the copy thread is used to perform erasure calculations on the latest received unit data in the RDMA memory;

[0170] Determine whether all unit data in the data to be stored has undergone deletion calculation;

[0171] If all steps have been completed, then the erasure calculation for the data to be stored is considered complete.

[0172] In a preferred embodiment, when the processor executes a computer subroutine stored in memory, it can perform the following steps:

[0173] When it is unloaded, it releases all copy threads and computation threads.

[0174] In a preferred embodiment, the number of computation threads is greater than the number of copy threads.

[0175] For a description of the computer-readable storage medium provided in the embodiments of the present invention, please refer to the foregoing embodiments of the data storage method; the embodiments of the present invention will not be repeated here.

[0176] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should also be noted that in this specification, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0177] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data storage method, characterized in that, Clients used in distributed storage systems include: Perform erasure calculations on the data to be stored; The data to be stored, after correction calculation, is divided into a preset number of parts; The divided data to be stored is transmitted one by one to multiple storage nodes of the distributed storage system; The specific steps for performing erasure calculations on the data to be stored are as follows: Request remote direct data access (RDMA) memory; Copy the data to be stored to the requested RDMA memory; Perform erasure calculations on the data to be stored located in the RDMA memory; The specific steps of transmitting the divided portions of the data to be stored to multiple storage nodes of the distributed storage system in a one-to-one correspondence are as follows: The storage addresses of each copy of the data to be stored in the RDMA memory are sent to the distributed storage system, so that the distributed storage system can control its multiple storage nodes to read each copy of the data to be stored one by one.

2. The data storage method according to claim 1, characterized in that, The specific steps of copying the data to be stored to the requested RDMA memory are as follows: The data to be stored is copied to the allocated RDMA memory using multiple copy threads.

3. The data storage method according to claim 2, characterized in that, The specific steps of performing erasure calculations on the data to be stored located in the RDMA memory are as follows: Multiple computation threads, distinct from the copy thread, are used to perform erasure calculations on the data to be stored in the RDMA memory.

4. The data storage method according to claim 3, characterized in that, The specific steps of using multiple computation threads, distinct from the copy thread, to perform erasure calculations on the data to be stored in the RDMA memory are as follows: When using multiple copy threads to copy the data to be stored to the allocated RDMA memory, it is determined whether the RDMA memory has received new unit data from the data to be stored. If received, the idle computing thread among the multiple computing threads that are different from the copy thread is used to perform erasure calculations on the latest received unit data in the RDMA memory; Determine whether all unit data in the data to be stored has completed the deletion calculation; If all steps have been completed, then the erasure calculation of the data to be stored is considered complete.

5. The data storage method according to claim 3, characterized in that, This data storage method also includes: When it is unloaded, all the copy threads and computation threads are released.

6. The data storage method according to any one of claims 3 to 5, characterized in that, The number of computation threads is greater than the number of copy threads.

7. A data storage device, characterized in that, Clients used in distributed storage systems include: The calculation module is used to perform erasure calculations on the data to be stored; The segmentation module is used to divide the data to be stored, after erasure calculation, into a preset number of parts; The transmission module is used to transmit the divided portions of the data to be stored one by one to multiple nodes of the distributed storage system. The calculation module includes: The application module is used to apply for Remote Direct Data Access (RDMA) memory. The copy module is used to copy the data to be stored to the requested RDMA memory; The calculation submodule is used to perform erasure calculations on the data to be stored in RDMA memory; The transmission module is specifically used for: The storage addresses of each piece of data to be stored in RDMA memory are sent to the distributed storage system so that the distributed storage system can control its multiple storage nodes to read each piece of data to be stored one by one.

8. A data storage device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the data storage method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the data storage method as described in any one of claims 1 to 6.

Images