Data storage method and device, equipment, storage medium and program product

Abstract

The invention discloses a data storage method and device, equipment, a storage medium and a program product, which are applied to electronic equipment, and the specific technical scheme comprises the following steps: receiving to-be-stored data; under the condition that the sum value of the data length of the to-be-stored data and the data length of the historical residual data is greater than or equal to a preset data length threshold value, splicing the to-be-stored data and the historical residual data to obtain first to-be-fragmented data; splitting the first to-be-fragmented data into a first data fragment and first residual data according to a preset data length threshold; updating the historical verification fragment by using the first data fragment to obtain an updated verification fragment, and writing the first data fragment into a preset storage node; and when the number of different data fragments in the preset storage node is equal to a preset threshold value, writing the update verification fragment into the preset storage node. In this way, after the data fragments with the preset data length threshold value are filled, the data fragments are directly written into the preset storage node, and waste of storage space is avoided.

Description

Technical Field

[0001] This application belongs to the technical field of data storage, and in particular relates to a method, device, equipment, storage medium and program product for data storage. Background Art

[0002] To ensure the reliability and security of data storage, erasure code (EC) or multiple replicas can be used to write the received input / output (I / O) data into the storage system.

[0003] When storing data by means of erasure code, usually after receiving the I / O data, a EC stripe is directly filled with zeros, and then after encoding and calculating the EC stripe, the I / O data and the parity check code obtained by the encoding calculation are written into the storage system. In this way, due to the need to fill the received I / O data with zeros, a large amount of storage space will be wasted.

[0004] When storing data by means of multiple replicas, usually after receiving the I / O data, the I / O data is written into the storage nodes in the form of multiple replicas. When the electronic device detects that the stored I / O data can be assembled into a full EC stripe, the stored I / O data is read from the storage nodes, assembled into an EC stripe, encoded and calculated to obtain a parity check code, and the I / O data and the parity check code obtained by the encoding calculation are written into the storage system. In this way, since the I / O data is stored in the form of multiple replicas, there is also a large amount of storage space waste. Summary of the Invention

[0005] Embodiments of this application provide a method, device, equipment, storage medium and program product for data storage, which can save storage space.

[0006] In a first aspect, embodiments of this application provide a method for data storage, which is applied to an electronic device. The method includes:

[0007] Receiving data to be stored;

[0008] When the sum of the data length of the data to be stored and the data length of the historical remaining data is greater than or equal to a preset data length threshold, splicing the data to be stored and the historical remaining data to obtain first data to be sliced, where the historical remaining data is obtained by splitting the previous historical stored data of the data to be stored;

[0009] Splitting the first data to be sliced into a first data slice and a first remaining data according to the preset data length threshold;

[0010] Update the historical verification shard using the first data shard to obtain an updated verification shard, where the historical verification shard is calculated using the previous historical stored data of the data to be stored;

[0011] Write the first data shard to a preset storage node;

[0012] When the number of different data shards in the preset storage node is equal to a preset threshold, write the updated verification shard to the preset storage node.

[0013] In a possible implementation, it further includes:

[0014] When the data length of the data to be stored and the data length of the historical remaining data are less than a preset data length threshold, determine whether the sum of the data length of the unprocessed data received within a preset duration, the data length of the data to be stored, and the data length of the historical remaining data is greater than or equal to the preset data length threshold according to the unprocessed data;

[0015] When the sum of the data length of the unprocessed data, the data length of the data to be stored, and the data length of the historical remaining data is less than the preset data length threshold, perform zero-padding processing on the unprocessed data, the data to be stored, and the historical remaining data to obtain data shards with a data length equal to the preset data length threshold;

[0016] Write the data shards to a preset storage node;

[0017] Update the historical verification shard using the data shards to obtain the updated verification shard;

[0018] When the sum of the data lengths of different data shards in the preset storage node is equal to a preset threshold, write the updated verification shard and the data shards to the preset storage node respectively.

[0019] In a possible implementation, the step of writing the first data shard to a preset storage node includes:

[0020] Copy the first data shard to obtain a first preset number of first data shards;

[0021] Write the first preset number of first data shards to different preset storage nodes respectively;

[0022] In a possible implementation, after writing the updated verification shard to the preset storage node, the method further includes:

[0023] For each type of data shard, obtain the preset storage node corresponding to the data slice;

[0024] Randomly select a second preset number of target preset storage nodes from the preset storage nodes corresponding to the data slice, where the second preset number is equal to the number of the preset storage nodes minus one;

[0025] For the second preset number of target preset storage nodes, delete the data slice in the target preset storage nodes.

[0026] In a possible implementation manner, before updating the historical verification slice with the first data shard to obtain an updated verification slice, the method further includes:

[0027] Obtain a data storage configuration instruction, where the data storage configuration instruction carries the preset data length threshold;

[0028] Receive the first data to be stored;

[0029] When the data length of the first data to be stored is greater than or equal to the preset data length threshold, split the first data to be stored into a second data shard and a second remaining data according to the preset data length threshold;

[0030] Write the second data shard into the preset storage node;

[0031] Receive the second data to be stored;

[0032] When the sum of the data length of the second data to be stored and the data length of the second remaining data is greater than or equal to the preset data length threshold, splice the second data to be stored and the second remaining data to obtain second data to be sharded;

[0033] Split the second data to be sharded according to the preset data length threshold to obtain a third data shard and a third remaining data;

[0034] Write the third data shard into the preset storage node;

[0035] Calculate the historical verification slice by using the second data shard and the third data shard.

[0036] In a possible implementation manner, it further includes:

[0037] When the number of different data shards in the preset storage node is less than a preset threshold, store the updated verification slice in a temporary cache space.

[0038] In a second aspect, an embodiment of the present application provides a data storage device, which is applied to an electronic device, and the device includes:

[0039] A receiving module, configured to receive data to be stored;

[0040] A splicing module, configured to splice the data to be stored and the historical remaining data to obtain first data to be fragmented when the sum of the data length of the data to be stored and the data length of the historical remaining data is greater than or equal to a preset data length threshold, where the historical remaining data is obtained by splitting the previous historical stored data of the data to be stored;

[0041] A splitting module, configured to split the first data to be fragmented into a first data fragment and a first remaining data according to the preset data length threshold;

[0042] A calculation module, configured to update a historical check fragment with the first data fragment to obtain an updated check fragment, where the historical check fragment is calculated using the previous historical stored data of the data to be stored;

[0043] A storage module, configured to write the first data fragment into a preset storage node;

[0044] The storage module is configured to write the updated check fragment into the preset storage node when the number of different data fragments in the preset storage node is equal to a preset threshold.

[0045] In a third aspect, an embodiment of the present application provides an electronic device, including: a processor and a memory storing computer program instructions;

[0046] When the processor executes the computer program instructions, the method for data storage as described in the first aspect is implemented.

[0047] In a fourth aspect, an embodiment of the present application provides a computer storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the method for data storage as described in the first aspect is implemented.

[0048] In a fifth aspect, an embodiment of the present application provides a computer program product, and when the instructions in the computer program product are executed by a processor of an electronic device, the electronic device is caused to execute the method for data storage as described in the first aspect.

[0049] A method, apparatus, device, storage medium and program product for data storage according to an embodiment of the present application. After receiving the data to be stored, the electronic device splits the data block obtained by splicing the data to be stored and the historical remaining data to obtain a first data shard and a first remaining data. Then, the first data shard is written into a preset storage node. In this way, after the data shards are obtained by splitting, the data shards can be directly written into the preset storage node without waiting to fill a data stripe, thereby reducing the occupied storage space. Moreover, subsequently, the data to be stored received dynamically is further utilized to realize the dynamic update of the historical verification shards. When it is detected that the sum of the data lengths of different data shards in the preset storage node is equal to a preset threshold, it indicates that different data shards have filled a data stripe. At this time, the updated verification shard is stored in the preset storage node, thereby completing the dynamic storage of the data stripe. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required to be used in the embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0051] Figure 1 is an exemplary diagram of a data storage provided in the prior art;

[0052] Figure 2 is another exemplary diagram of a data storage provided in the prior art;

[0053] Figure 3 is a flowchart of a method for data storage provided by an embodiment of the present application;

[0054] Figure 4 is a flowchart of another method for data storage provided by an embodiment of the present application;

[0055] Figure 5 is a flowchart of a method for data storage applying a timeout padding zero strategy provided by an embodiment of the present application;

[0056] Figure 6 is an exemplary diagram of a method for data storage provided by an embodiment of the present application;

[0057] Figure 7 is an exemplary diagram of a method for redundant copy deletion provided by an embodiment of the present application;

[0058] Figure 8 is a structural diagram of a data storage device provided by an embodiment of the present application;

[0059] Figure 9It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0060] The features of various aspects of the present application and exemplary embodiments will be described in detail below. To make the objectives, technical solutions, and advantages of the present application clearer, the present application will be further described in detail below in combination with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by showing examples of the present application.

[0061] It should be noted that, in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, the elements defined by the statement "including..." do not exclude the existence of additional identical elements in the process, method, article or device including the elements.

[0062] Currently, in order to ensure the security of data storage, erasure codes are usually used for data storage or a method combining erasure codes and multiple replicas is used for data storage.

[0063] The following introduces erasure codes and multiple replicas:

[0064] First, a method of using erasure codes for data storage is introduced. By splitting the received data into multiple data blocks according to a preset data length, and then encoding the multiple data blocks, check data blocks are obtained. Finally, the multiple data blocks obtained by splitting and the check data blocks are stored in different positions respectively.

[0065] Specifically, as Figure 1 shown, after the electronic device receives the original data block, the original data block is split into k data shards, namely d1 to dk. Among them, the d1 to dk data shards form an EC stripe. By performing encoding calculations on this EC stripe, m check data blocks, namely c1 to cm, are obtained. Finally, the k data shards and the m check data blocks are written into different storage nodes respectively. The above erasure code scheme can be expressed as RS(k,m).

[0066] In this way, when a storage node fails, the electronic device can restore the original data block from the data blocks in other storage nodes, thus avoiding data loss caused by storage node failures.

[0067] Currently, in order to avoid data loss problems that occur during the waiting process of the original data block, after receiving an I / O data, the electronic device directly pads zeros to make the I / O data full to an EC stripe, and then performs encoding calculations on the EC stripe. After that, the EC stripe data and the corresponding parity data shards are stored in the storage nodes in a fragmented manner.

[0068] However, when storing data through the above method, since zeros are padded to each I / O data received, a large amount of meaningless data exists in the filled EC stripe. Subsequently, when storing in the storage nodes, the meaningless data will occupy a large amount of storage space, resulting in a waste of storage space.

[0069] The following introduces a method of storing data using multiple replicas.

[0070] After receiving the original data block, the electronic device copies the original data block multiple times and stores them on different storage nodes respectively. When the electronic device determines that the received original data blocks are full to an EC stripe, it reads different original data blocks from different storage nodes, and then performs encoding calculations on the different original data blocks to obtain the corresponding parity data blocks. The different original data blocks and parity data blocks are stored in the storage nodes, and thus the data storage is completed.

[0071] Currently, in order to avoid the problem of wasted storage space caused by zero-padding operations. Data can be stored by combining multiple replicas and erasure codes.

[0072] Specifically, as Figure 2 shown, after receiving the I / O data, the electronic device stores the I / O data in storage nodes 1 to 3 in a 3-replica manner. Then, when the electronic device detects that the received I / O data can be full to an EC stripe, the electronic device reads the stored I / O data from the storage nodes, and then splits and encodes the received I / O data to obtain data shards d1 to d4 and parity data shards c1 and c2. Finally, the electronic device stores the data shards d1 to d4 and the parity data shards c1 and c2 in storage nodes 1 to 6 in sequence.

[0073] However, since the electronic device stores each I / O data in multiple storage nodes in the form of multiple copies after receiving it, this will occupy a large amount of storage space. Moreover, when the electronic device determines that an EC shard is full, it needs to read the I / O data from multiple storage nodes, which will bring a large network overhead.

[0074] To solve the problems of the prior art, the embodiments of the present application provide a data storage method, apparatus, device, storage medium and program product. First, the data storage method provided by the embodiments of the present application will be described below.

[0075] As Figure 3 shown, this method is applied to an electronic device, and the method includes:

[0076] S301. Receive the data to be stored.

[0077] Among them, the data to be stored is the above-mentioned I / O data. I / O data refers to the data transfer between a computer and any external device. Common external devices include keyboards, printers and screens, etc. In the embodiments of the present application, the data to be stored may be small I / O data. Small I / O data refers to the data volume involved in one I / O operation is relatively small, usually less than or equal to 16KB.

[0078] S302. When the sum of the data length of the data to be stored and the data length of the historical remaining data is greater than or equal to the preset data length threshold, splice the data to be stored and the historical remaining data to obtain the first data to be sharded.

[0079] Among them, the historical remaining data is obtained by splitting the previous historical stored data of the data to be stored. The preset data length threshold is set by the user in advance according to actual business requirements.

[0080] When splicing the data to be stored and the historical remaining data, the data to be stored can be spliced after the historical remaining data to obtain the first data to be sharded.

[0081] S303. Split the first data to be sharded into a first data shard and a first remaining data according to the preset data length threshold.

[0082] S304. Update the historical check shard with the first data shard to obtain an updated check shard.

[0083] Among them, the historical check shard is calculated using the previous historical stored data of the data to be stored.

[0084] It should be noted that the number of verification shards is set by the user in advance according to actual business requirements. After the electronic device calculates a specified number of verification shards using the received I / O data, the electronic device receives a new data to be stored. The electronic device updates the specified number of verification shards using the data to be stored. After receiving the next data to be stored, the updated verification shards obtained this time are used as the above-mentioned historical verification shards, and the historical verification shards are updated using the next data to be stored. That is, after the electronic device receives each data to be stored, the above steps are repeatedly executed.

[0085] S305. Write the first data shard into a preset storage node.

[0086] S306. When the number of different data shards in the preset storage node is equal to the preset threshold, write the updated verification shard into the preset storage node.

[0087] Wherein, the preset threshold is set according to the data length of the EC stripe and the data shard. By calculating the ratio of the data length of the EC stripe to the data length of the preset data shard, the above-mentioned preset threshold can be obtained.

[0088] It can be understood that after the above-mentioned electronic device writes the first data shard into the preset storage node, the same data shards will be stored in the preset storage node. Therefore, the electronic device monitors the types of data shards stored in the storage node, obtains different data shards. When the number of different data shards is equal to the preset threshold, it means that the I / O data received by the electronic device has gathered an EC stripe. Therefore, after calculating the latest updated verification shard, storing the updated verification shard in the preset storage node can complete the writing of the data to be stored in the EC stripe.

[0089] Using the above method, after the electronic device receives the data to be stored, the data block obtained by splicing the data to be stored and the historical remaining data is split to obtain the first data shard and the first remaining data. Then the first data shard is written into the preset storage node. In this way, after the data shard is split, the data shard can be directly written into the preset node without having to fill up the data stripe to write into the preset storage node, thereby reducing the occupied storage space. And subsequently, further utilize the dynamically received data to be stored to realize the dynamic update of the historical verification shard. When it is detected that the sum value of the data lengths of different data shards in the preset storage node is equal to the preset threshold, it means that different data shards have filled up a data stripe. At this time, the updated verification shard is stored in the preset storage node, thereby completing the dynamic storage of the data stripe.

[0090] Regarding the above S305. Write the first data shard into the preset storage node, the following steps can be specifically adopted to write the first data shard into the preset storage node. Specifically:

[0091] The electronic device copies the first data shard to obtain a first preset number of first data shards, and writes the first preset number of first data shards into different preset storage nodes respectively. In this way, by copying the first data shard and writing each first data shard into a different preset storage node in turn, each data shard can be backed up in the form of multiple replicas. When the number of data shards written into the storage node does not meet the preset threshold, in the case of data loss caused by a storage node failure, data backup in the form of multiple replicas can improve data reliability.

[0092] Correspondingly, in order to reduce the waste of storage space in the preset storage node, after the electronic device writes the update check shard into the preset storage node, the electronic device can also delete the redundant data shards in the preset storage node, thus avoiding space waste. Based on this, after writing the update check shard into the preset storage node as described above, the method further includes:

[0093] Step 1: For each type of data shard, obtain the preset storage node corresponding to the data slice.

[0094] According to the method provided in the above embodiment, after the electronic device obtains the data shard, it can be written into different preset storage nodes respectively. At the same time, the electronic device can record the preset storage node corresponding to each data shard. In this way, according to the storage record, the electronic device can obtain the preset storage node corresponding to each type of data shard.

[0095] Step 2: Randomly select a second preset number of target preset storage nodes from the preset storage nodes corresponding to the data slice.

[0096] Wherein, the second preset number is equal to the number of preset storage nodes minus one.

[0097] Step 3: For the second preset number of target preset storage nodes, delete the data slice in the target preset storage node.

[0098] In an example, assume that there are a total of four data shards, two check shards, and six preset storage nodes. Specifically, data shard d1 is stored in preset storage node 1, data shard d1 and data shard d2 are stored in preset storage node 2, data shard d1, data shard d2, and data shard d3 are stored in preset storage node 3, data shard d2, data shard d3, and data shard d4 are stored in preset storage node 4, check shard c1 and data shard d3 are stored in preset storage node 5, and check shard c2 is stored in preset storage node 6.

[0099] The electronic device randomly selects two of the preset storage nodes 1, preset storage node 2, and preset storage node 3 corresponding to the data shard d1, and deletes the data shard d1 therein; randomly selects two of the preset storage nodes 2, preset storage node 3, and preset storage node 4 corresponding to the data shard d2, and deletes the data shard d2 therein; randomly selects two of the preset storage nodes 3, preset storage node 4, and preset storage node 5 corresponding to the data shard d3, and deletes the data shard d3 therein.

[0100] By using the method provided in the embodiment of the present application, when the electronic device determines that the number of different data shards stored in the preset storage node reaches the preset threshold, it indicates that an EC stripe is full. To reduce the waste of storage space in the preset storage node, the data shards previously stored in a multi-copy manner can be deleted to delete redundant copies, thereby improving the storage space utilization rate and avoiding waste of storage space.

[0101] It should be noted that for the updated parity shards calculated above, when the electronic device determines that the number of different data shards in the preset storage node does not reach the preset threshold, it indicates that the I / O data received by the current electronic device is for filling an EC stripe. Therefore, the updated parity shards can be stored in the temporary cache space. In this way, frequent read and write operations between the electronic device and the preset storage node can be avoided, thereby reducing the network overhead.

[0102] In some embodiments of the present application, after the user modifies the data storage configuration, since the historical parity shards are stored in the temporary cache space of the electronic device, the following will be combined with Figure 4 introduce the processing methods for the electronic device to receive the first I / O data and the second I / O data. As Figure 4 shown, the method includes:

[0103] S401. Obtain a data storage configuration instruction.

[0104] Among them, the data storage configuration instruction carries a preset data length threshold.

[0105] S402. Receive the first data to be stored.

[0106] Among them, the first data to be stored is the first I / O data received by the electronic device after receiving the data storage configuration instruction.

[0107] S403. When the data length of the first data to be stored is greater than or equal to the preset data length threshold, split the first data to be stored into a second data shard and a second remaining data according to the preset data length threshold.

[0108] S404. Write the second data shard to the preset storage node.

[0109] S405. Receive the second data to be stored.

[0110] Among them, the second data to be stored is the second I / O data received by the electronic device after receiving the data storage configuration instruction.

[0111] S406. When the sum of the data length of the second data to be stored and the data length of the second remaining data is greater than or equal to the preset data length threshold, splice the second data to be stored and the second remaining data to obtain the second data to be sharded.

[0112] Specifically, the splicing method refers to the relevant description in the above embodiments and will not be elaborated here.

[0113] S407. Split the second data to be sharded according to the preset data length threshold to obtain a third data shard and a third remaining data.

[0114] S408. Write the third data shard to the preset storage node.

[0115] S409. Calculate the historical check shard using the second data shard and the third data shard.

[0116] It should be noted that the embodiments of the present application do not specifically limit the method for calculating the check shard. For example, the historical check shard can be calculated by calculating the exclusive OR value of the second data shard and the third data shard, or by performing weighted summation on the second data shard and the third data shard.

[0117] Using the method provided in the embodiments of the present application, after the user modifies the data storage configuration requirements, the electronic device obtains the data storage configuration instruction and performs data storage based on this instruction. Among them, for the first I / O data and the second I / O data, since there is no corresponding historical check shard in the temporary cache space of the electronic device, the historical check shard can be calculated by calculating the second data shard and the third data shard. Subsequently, the electronic device uses the method provided in the Figure 3 illustrated embodiment for data storage. In this way, the normal encoding of the subsequent data to be stored is ensured.

[0118] In some embodiments of the present application, if the electronic device determines that the sum of the data length of the data to be stored and the data length of the historical remaining data does not reach the preset data length threshold, the electronic device continues to wait for the next device to be stored. If the data length of the unprocessed data received within the combined preset time duration still cannot reach the preset data length threshold, padding with zeros is used for processing. Specifically, such as Figure 5As shown, the method includes:

[0119] S501. When the data length of the data to be stored and the data length of the historical remaining data are less than a preset data length threshold, determine whether the sum value of the data length of the unprocessed data, the data length of the data to be stored, and the data length of the historical remaining data is greater than or equal to the preset data length threshold according to the unprocessed data received within a preset duration.

[0120] Among them, the preset duration is preset according to actual business requirements.

[0121] S502. When the sum value of the data length of the unprocessed data, the data length of the data to be stored, and the data length of the historical remaining data is less than the preset data length threshold, perform zero-padding processing on the unprocessed data, the data to be stored, and the historical remaining data to obtain a data shard with a data length equal to the preset data length threshold.

[0122] Among them, the unprocessed data is the data to be stored received by the electronic device within the preset duration.

[0123] Specifically, the electronic device first splices the historical remaining data, the data to be stored, and the unprocessed data in sequence, and then pads zeros after the spliced data until a data shard is completed.

[0124] S503. Write the data shard into a preset storage node.

[0125] S504. Update the historical check shard with the data shard to obtain an updated check shard.

[0126] S505. When the sum value of the data lengths of different data shards in the preset storage node is equal to the preset threshold, write the updated check shard and the data shard into the preset storage node respectively.

[0127] By using the method provided in the embodiment of the present application, when the electronic device detects that the sum value of the data lengths of the received data to be stored and the historical remaining data does not reach the preset data length threshold, it means that the data temporarily cached locally by the electronic device cannot fill a data shard. Therefore, the electronic device monitors the unprocessed data received within the preset duration. When it detects that the sum value of the data length of the unprocessed data, the data length of the data to be stored, and the data length of the historical remaining data is still less than the preset data length threshold, in order to avoid waiting for too long, zero-padding processing is performed on the unprocessed data, the data to be stored, and the historical remaining data to fill a data shard, thereby realizing the storage of the unprocessed data, the data to be stored, and the historical remaining data. In this way, after the electronic device waits for the preset duration, zero-padding operation is performed on the unprocessed data, the data to be stored, and the historical remaining data, avoiding zero-padding for each data to be stored, thereby reducing the waste of storage space caused by excessive zero-padding.

[0128] The following combines Figure 6 to introduce the complete process of data storage provided by the embodiments of this application. As Figure 6 shown, the electronic device receives a data write stream. After receiving the first I / O data, it splits the first I / O data into a data shard d1 and remaining data r1, caches the data shard d1 and the remaining data r1 locally, and writes the data shard d1 to a preset storage node in a triple-copy form.

[0129] The electronic device receives the second I / O data, concatenates the second I / O data and the remaining data r1, determines whether the data length of the concatenated data meets a preset data length threshold. If it meets the threshold, the concatenated data is split to obtain a data shard d2 and remaining data r2. The electronic device performs encoding calculations on the data shard d1 and the data shard d2 to obtain temporary check shards tc1 and tc2, caches the remaining data r2, the temporary check shards tc1 and tc2 locally, writes the data shard d2 to the preset storage node in a triple-copy form, and deletes the data shards d1 and d2 in the local cache.

[0130] The electronic device receives the third I / O data, concatenates the third I / O data and the remaining data r2, determines whether the data length of the concatenated data meets the preset data length threshold. If it does not meet the threshold, it continues to wait for the next I / O data to be written.

[0131] The electronic device receives the fourth I / O data, concatenates the remaining data r2, the third I / O data, and the fourth I / O data, determines whether the data length of the concatenated data meets the preset data length threshold. If it meets the threshold, the concatenated data is split to obtain a data shard d3 and remaining data r3, and the data shard d3 is used to update the temporary check shards tc1 and tc2 to obtain updated temporary check shards tc1 and tc2. The remaining data r3 and the updated temporary check shards tc1 and tc2 are stored in the local cache, the data shard d3 is written to the preset storage node in a triple-copy form, and the data shard d3 in the local cache is deleted.

[0132] The electronic device receives the fifth I / O data, concatenates the remaining data r3 and the fifth I / O data, determines whether the data length of the concatenated data meets the preset data length threshold. If it meets the threshold, the concatenated data is split to obtain a data shard d4 and remaining data r4, and the data shard d4 is used to update the temporary check shards tc1 and tc2 to obtain updated temporary check shards tc1 and tc2.

[0133] After the electronic device splits to obtain the data shard d4, the number of different data shards stored in the preset storage node has reached the preset threshold, that is, the data shards d1, d2, d3, and d4 have filled an EC stripe. Therefore, the electronic device uses the updated temporary check shards tc1 and tc2 as the final check shards c1 and c2, and stores the data shard d4 and the final check shards c1 and c2 in the preset storage node.

[0134] The electronic device deletes the data shards cached locally, and caches the remaining data r4 locally for the electronic device to splice the subsequent received I / O data using the remaining data r4.

[0135] Among them, after the electronic device stores the data shard d4 and the final check shards c1 and c2 in the preset storage node, to avoid wasting the storage space in the preset storage node, the redundant copies in the preset storage node are deleted. Specifically, as Figure 7 shown, the data shard d1 is stored in the preset storage node 1, the data shards d1 and d2 are stored in the preset storage node 2, the data shards d1, d2, and d3 are stored in the preset storage node 3, the data shards d2, d3, and d4 are stored in the preset storage node 4, the data shard d3 and the check shard c1 are stored in the preset storage node 5, and the check shard c2 is stored in the preset storage node 6.

[0136] The electronic device randomly selects two preset storage nodes from the preset storage nodes 1, 2, and 3 corresponding to the data shard d1, and deletes the data shard d1 therein; randomly selects two preset storage nodes from the preset storage nodes 2, 3, and 4 corresponding to the data shard d2, and deletes the data shard d2 therein; randomly selects two preset storage nodes from the preset storage nodes 3, 4, and 5 corresponding to the data shard d3, and deletes the data shard d3 therein.

[0137] Based on the same concept, an embodiment of the present application provides a data storage device, which is applied to an electronic device. As Figure 8 shown, the device includes:

[0138] A receiving module 801, configured to receive data to be stored;

[0139] A splicing module 802, configured to splice the data to be stored and the historical remaining data to obtain first data to be sharded when the sum of the data length of the data to be stored and the data length of the historical remaining data is greater than or equal to a preset data length threshold, where the historical remaining data is obtained by splitting the previous historical stored data of the data to be stored.

[0140] A splitting module 803, configured to split the first data to be sliced into a first data slice and a first remaining data according to the preset data length threshold;

[0141] A calculation module 804, configured to update the historical check slice by using the first data slice to obtain an updated check slice, where the historical check slice is calculated by using the previous historical stored data of the data to be stored;

[0142] A storage module 805, configured to write the first data slice into a preset storage node;

[0143] The storage module 805 is configured to write the updated check slice into the preset storage node when the number of different data slices in the preset storage node is equal to the preset threshold.

[0144] In a possible implementation manner, the apparatus further includes:

[0145] A judgment module, configured to, when the data length of the data to be stored and the data length of the historical remaining data are less than the preset data length threshold, judge whether the sum of the data length of the unprocessed data, the data length of the data to be stored, and the data length of the historical remaining data is greater than or equal to the preset data length threshold according to the unprocessed data received within a preset duration;

[0146] A zero-padding module, configured to, when the sum of the data length of the unprocessed data, the data length of the data to be stored, and the data length of the historical remaining data is less than the preset data length threshold, perform zero-padding processing on the unprocessed data, the data to be stored, and the historical remaining data to obtain a data slice with a data length equal to the preset data length threshold;

[0147] The storage module 805 is further configured to write the data slice into a preset storage node;

[0148] The calculation module 804 is further configured to update the historical check slice by using the data slice to obtain the updated check slice;

[0149] The storage module 805 is further configured to write the updated check slice and the data slice into the preset storage node respectively when the sum of the data lengths of different data slices in the preset storage node is equal to the preset threshold.

[0150] In a possible implementation manner, the storage module 805 is specifically configured to:

[0151] Copy the first data slice to obtain a first preset number of first data slices;

[0152] Write the first preset number of first data shards into different preset storage nodes respectively;

[0153] In a possible implementation, the apparatus further includes:

[0154] An acquisition module, configured to acquire, for each type of data shard, the preset storage node corresponding to the data slice;

[0155] A selection module, configured to randomly select a second preset number of target preset storage nodes from the preset storage nodes corresponding to the data slice, where the second preset number is equal to the number of the preset storage nodes minus one;

[0156] A deletion module, configured to delete the data slice in the target preset storage node for the second preset number of target preset storage nodes.

[0157] In a possible implementation, the apparatus further includes:

[0158] The acquisition module is further configured to acquire a data storage configuration instruction, where the data storage configuration instruction carries the preset data length threshold;

[0159] A receiving module 801, configured to receive first data to be stored;

[0160] A splitting module 803 is further configured to, when the data length of the first data to be stored is greater than or equal to the preset data length threshold, split the first data to be stored into a second data shard and second remaining data according to the preset data length threshold;

[0161] A storage module 805 is further configured to write the second data shard into the preset storage node;

[0162] A receiving module 801, configured to receive second data to be stored;

[0163] An assembling module 802 is further configured to, when the sum of the data length of the second data to be stored and the data length of the second remaining data is greater than or equal to the preset data length threshold, assemble the second data to be stored and the second remaining data to obtain second data to be sharded;

[0164] A splitting module 803 is further configured to split the second data to be sharded according to the preset data length threshold to obtain a third data shard and third remaining data;

[0165] A storage module 805 is further configured to write the third data shard into the preset storage node;

[0166] The calculation module 804 is further configured to calculate the historical check shard by using the second data shard and the third data shard.

[0167] In a possible implementation, the storage module 805 is further configured to:

[0168] When the number of different data shards in the preset storage node is less than a preset threshold, store the updated check shard in a temporary cache space.

[0169] It should be noted that the apparatus for data storage corresponds to the method for data storage described above. All implementation manners in the above method embodiments are applicable to the embodiments of this apparatus and can achieve the same technical effects.

[0170] Figure 9 FIG. shows a schematic hardware structure diagram of an electronic device provided in an embodiment of the present application.

[0171] The electronic device may include a processor 901 and a memory 902 storing computer program instructions.

[0172] Specifically, the above-mentioned processor 901 may include a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0173] The memory 902 may include a mass storage for data or instructions. By way of example and not limitation, the memory 902 may include a Hard Disk Drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a Universal Serial Bus (USB) drive or a combination of two or more of these. In a suitable case, the memory 902 may include a removable or non-removable (or fixed) medium. In a suitable case, the memory 902 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 902 is a non-volatile solid-state memory.

[0174] In a particular embodiment, the memory 902 may include a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk storage media device, an optical storage media device, a flash memory device, an electrical, optical, or other physical / tangible memory storage device. Thus, generally, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of the present disclosure.

[0175] The processor 901 reads and executes the computer program instructions stored in the memory 902 to implement any one of the data storage methods in the above embodiments.

[0176] In one example, the electronic device may further include a communication interface 903 and a bus 904. Among them, as Figure 9 shown, the processor 901, the memory 902, and the communication interface 903 are connected through the bus 904 to complete communication with each other.

[0177] The communication interface 903 is mainly used to implement communication between the various modules, devices, units, and / or devices in the embodiments of the present application.

[0178] The bus 904 includes hardware, software, or both, and couples components of the electronic device to each other. By way of example and not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Super Transmission (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low PinCount (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, the bus 904 may include one or more buses. Although the embodiments of the present application describe and illustrate specific buses, the present application contemplates any suitable bus or interconnect.

[0179] In addition, in combination with the data storage method in the above embodiments, an embodiment of the present application may be implemented by providing a computer storage medium. Computer program instructions are stored on the computer storage medium; when the computer program instructions are executed by a processor, any one of the data storage methods in the above embodiments is implemented.

[0180] An embodiment of the present application further provides a computer program product, including a computer program, which when executed by a processor implements any one of the data storage methods in the above embodiments.

[0181] It should be clear that the present application is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and illustrated as examples. However, the method process of the present application is not limited to the specific steps described and illustrated, and those skilled in the art can make various changes, modifications, and additions, or change the order between steps after understanding the spirit of the present application.

[0182] The functional blocks shown in the above structural block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a functional card, and so on. When implemented in software, the elements of the present application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium or transmitted via a data signal carried in a carrier wave on a transmission medium or a communication link. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable read-only memory (EROM), floppy disks, compact disc read-only memory (CD-ROM), optical discs, hard disks, fiber optic media, radio frequency (RF) links, and so on. The code segment can be downloaded via a computer network such as the Internet, an intranet, and so on.

[0183] It should also be noted that the exemplary embodiments mentioned in the present application describe some methods or systems based on a series of steps or devices. However, the present application is not limited to the order of the above steps, that is, the steps can be executed in the order mentioned in the embodiments, or different from the order in the embodiments, or several steps can be executed simultaneously.

[0184] Aspects of the present disclosure have been described above with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block in the flowcharts and / or block diagrams, and the combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the functions / actions specified in one or more blocks of the flowcharts and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It can also be understood that each block in the block diagrams and / or flowcharts, and the combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware that performs the specified functions or actions, or by a combination of dedicated hardware and computer instructions.

[0185] The above are only specific embodiments of the present application. Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, modules, and units described above can refer to the corresponding processes in the foregoing method embodiments, and will not be elaborated herein. It should be understood that the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of various equivalent modifications or substitutions, and these modifications or substitutions should be covered within the protection scope of the present application.

Claims

1. A method for data storage, characterized in that, Applied to an electronic device, the method includes: Receiving data to be stored; When the sum of the data length of the data to be stored and the data length of the historical remaining data is greater than or equal to a preset data length threshold, splicing the data to be stored and the historical remaining data to obtain first data to be sliced, where the historical remaining data is obtained by splitting the previous historical stored data of the data to be stored; Splitting the first data to be sliced into a first data slice and a first remaining data according to the preset data length threshold; Updating the historical check slice with the first data slice to obtain an updated check slice, where the historical check slice is calculated using the previous historical stored data of the data to be stored; Writing the first data slice into a preset storage node; When the number of different data slices in the preset storage node is equal to a preset threshold, writing the updated check slice into the preset storage node.

2. The method according to claim 1, wherein It further includes: When the sum of the data length of the data to be stored and the data length of the historical remaining data is less than the preset data length threshold, judging whether the sum of the data length of the unprocessed data received within a preset duration, the data length of the data to be stored, and the data length of the historical remaining data is greater than or equal to the preset data length threshold according to the unprocessed data; When the sum of the data length of the unprocessed data, the data length of the data to be stored, and the data length of the historical remaining data is less than the preset data length threshold, performing zero-padding processing on the unprocessed data, the unprocessed data, and the historical remaining data to obtain a data slice with a data length equal to the preset data length threshold; Writing the data slice into the preset storage node; Updating the historical check slice with the data slice to obtain the updated check slice; When the sum of the data lengths of different data slices in the preset storage node is equal to a preset threshold, writing the updated check slice and the data slice into the preset storage node respectively.

3. The method according to claim 1, wherein The writing the first data slice into a preset storage node includes: Copying the first data slice to obtain a first preset number of first data slices; Writing the first preset number of first data slices into different preset storage nodes respectively.

4. The method according to claim 3, wherein After writing the updated check slice into the preset storage node, the method further includes: For each type of data slice, obtaining a plurality of preset storage nodes corresponding to the data slice; Randomly selecting a second preset number of target preset storage nodes from the plurality of preset storage nodes corresponding to the data slice, where the second preset number is equal to the number of the plurality of preset storage nodes minus one; For the second preset number of target preset storage nodes, deleting the data slice in the target preset storage node.

5. The method according to claim 1, characterized in that, Before updating the historical check slice with the first data slice to obtain an updated check slice, the method further includes: Obtaining a data storage configuration instruction, where the data storage configuration instruction carries the preset data length threshold; Receiving first data to be stored; When the data length of the first data to be stored is greater than or equal to the preset data length threshold, split the first data to be stored into a second data shard and second remaining data according to the preset data length threshold; Write the second data shard into the preset storage node; Receive second data to be stored; When the sum of the data length of the second data to be stored and the data length of the second remaining data is greater than or equal to the preset data length threshold, splice the second data to be stored and the second remaining data to obtain second data to be sharded; Split the second data to be sharded according to the preset data length threshold to obtain a third data shard and third remaining data; Write the third data shard into the preset storage node; Calculate the historical check shard by using the second data shard and the third data shard.

6. The method according to claim 1, characterized in that, Further include: When the number of different data shards in the preset storage node is less than the preset threshold, store the updated check shard in the temporary cache space.

7. A data storage device, characterized in that, Applied to an electronic device, the device includes: A receiving module, configured to receive data to be stored; A splicing module, configured to splice the data to be stored and historical remaining data to obtain first data to be sharded when the sum of the data length of the data to be stored and the data length of the historical remaining data is greater than or equal to the preset data length threshold, and the historical remaining data is obtained by splitting the previous historical stored data of the data to be stored; A splitting module, configured to split the first data to be sharded into a first data shard and first remaining data according to the preset data length threshold; A calculation module, configured to update the historical check shard by using the first data shard to obtain an updated check shard, and the historical check shard is calculated by using the previous historical stored data of the data to be stored; A storage module, configured to write the first data shard into the preset storage node; The storage module is configured to write the updated check shard into the preset storage node when the number of different data shards in the preset storage node is equal to the preset threshold.

8. An electronic device, characterized in that, The device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the method for data storage as described in any one of claims 1-6 is implemented.

9. A computer-readable storage medium, characterized in that, Computer program instructions are stored on the computer-readable storage medium, and when the computer program instructions are executed by the processor, the method for data storage as described in any one of claims 1-6 is implemented.

10. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device is caused to execute the method for data storage as described in any one of claims 1-6.

Images