Data storage method and device, medium, electronic equipment and program product

Abstract

The invention discloses a data storage method and device, a medium, electronic equipment and a program product, belongs to the technical field of computers, and can reduce the number of disks or files needing to be accessed during reading and reduce reading overhead. The data storage method comprises the steps that first-level striping processing is carried out on data only added with files according to a first preset data length to obtain a first part and a second part, each of the first part and the second part comprises a plurality of data blocks, and the length of each data block is the first preset data length; performing second-level striping processing on the first part and the second part according to a second preset data length to obtain K data corresponding to the first part and the second part; performing redundant coding on the K data corresponding to the first part and the K data corresponding to the second part to obtain redundant data; and writing the K data corresponding to the first part into the first machine room, writing the K data corresponding to the second part into the second machine room, and writing the redundant data into the corresponding machine room for storing the redundant data.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technologies, and in particular, to a data storage method, apparatus, medium, electronic device, and program product. Background Art

[0002] Multi-data center redundant storage usually requires tolerating the failure of a single data center (i.e., the complete loss of one data center copy) plus the failure of some copies in the surviving data centers. Therefore, the number of redundant shards is relatively large, and the proportion of the original data shards, which is equal to the storage utilization rate, cannot be too low. Therefore, the stripe width (i.e., the number of units into which each stripe is split, and each unit of the same stripe will be written into a different disk or file) is usually very large. In this way, when the stripe length is small (such as 24 KB), even when reading a small amount of data (such as 24 KB), multiple disks need to be accessed to retrieve the data, resulting in excessive read overhead. Summary of the Invention

[0003] This Summary of the Invention section is provided to introduce concepts in a brief form, which will be described in detail in the subsequent Detailed Description section. This Summary of the Invention section is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] In a first aspect, the present disclosure provides a data storage method, including:

[0005] Performing first-level striping on the data of the append-only file according to a first preset data length to obtain a first part and a second part, where the first part and the second part each include a plurality of data blocks, and the length of each data block is the first preset data length;

[0006] Performing second-level striping on the first part according to a second preset data length to obtain K pieces of data corresponding to the first part, and performing the second-level striping on the second part according to the second preset data length in parallel to obtain K pieces of data corresponding to the second part; where the second preset data length is an integer multiple greater than 1 of the first preset data length, and K is an integer greater than 1;

[0007] Performing redundant encoding on the K pieces of data corresponding to the first part and the K pieces of data corresponding to the second part to obtain redundant data;

[0008] Writing the K pieces of data corresponding to the first part into a first data center, writing the K pieces of data corresponding to the second part into a second data center, and writing the redundant data into the corresponding data center for storing the redundant data.

[0009] In a second aspect, the present disclosure provides a data storage apparatus, including:

[0010] A striping processing module is configured to perform first-level striping processing on the data of the append-only file according to a first preset data length to obtain a first part and a second part. The first part and the second part respectively include a plurality of data blocks, and the length of each data block is the first preset data length. Second-level striping processing is performed on the first part according to a second preset data length to obtain K pieces of data corresponding to the first part, and second-level striping processing is performed on the second part according to the second preset data length in parallel to obtain K pieces of data corresponding to the second part. The second preset data length is an integer multiple greater than 1 of the first preset data length, and K is an integer greater than 1.

[0011] A redundant encoding module is configured to perform redundant encoding on the K pieces of data corresponding to the first part and the K pieces of data corresponding to the second part to obtain redundant data.

[0012] A writing module is configured to write the K pieces of data corresponding to the first part into a first computer room, write the K pieces of data corresponding to the second part into a second computer room, and write the redundant data into the corresponding computer room for storing the redundant data.

[0013] In a third aspect, the present disclosure provides a computer-readable medium, on which a computer program is stored. When the computer program is executed by a processing device, the steps of the method described in any one of the first aspects are implemented.

[0014] In a fourth aspect, the present disclosure provides an electronic device, including:

[0015] A storage device, on which a computer program is stored;

[0016] A processing device is configured to execute the computer program in the storage device to implement the steps of the method described in any one of the first aspects.

[0017] In a fifth aspect, the present disclosure provides a computer program product, including a computer program. When the computer program is executed by a processor, the steps of the method described in any one of the first aspects are implemented.

[0018] By adopting the above technical solution, it is possible to perform first-level striping on the data of only the appended file according to the first preset data length to obtain a first part and a second part, where the first part and the second part each include a plurality of data blocks, and the length of each data block is the first preset data length; perform second-level striping on the first part according to the second preset data length to obtain K pieces of data corresponding to the first part, and perform the second-level striping on the second part according to the second preset data length in parallel to obtain K pieces of data corresponding to the second part; where the second preset data length is an integer multiple greater than 1 of the first preset data length; perform redundant encoding on the K pieces of data corresponding to the first part and the K pieces of data corresponding to the second part to obtain redundant data; write the K pieces of data corresponding to the first part into the first computer room, write the K pieces of data corresponding to the second part into the second computer room, and write the redundant data into the corresponding computer room for storing the redundant data. Therefore, through the above two-level striping process, a larger strip length can be used, thereby reducing the number of disks or files that need to be accessed during reading and reducing the reading overhead.

[0019] Other features and advantages of the present disclosure will be described in detail in the subsequent specific implementation part. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In combination with the accompanying drawings and with reference to the following specific implementation manners, the above and other features, advantages and aspects of the various embodiments of the present disclosure will become more obvious. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic and the original components and elements are not necessarily drawn to scale. In the drawings:

[0021] Figure 1 is a flowchart of a data storage method according to an embodiment of the present disclosure.

[0022] Figure 2 shows a schematic diagram of two-level striping according to an embodiment of the present disclosure.

[0023] Figure 3 is according to an embodiment of the present disclosure Figure 2 The schematic diagram of the writing result taking the original data as an example.

[0024] Figure 4 is a schematic diagram of redundant encoding for an incomplete strip according to an embodiment of the present disclosure.

[0025] Figure 5 is a schematic diagram of redundant encoding for an incomplete strip according to an embodiment of the present disclosure.

[0026] Figure 6Shows a schematic diagram of redundant encoding for incomplete stripes according to an embodiment of the present disclosure.

[0027] Figure 7 Shows another schematic diagram of redundant encoding for incomplete stripes according to an embodiment of the present disclosure.

[0028] Figure 8 Is a schematic block diagram of a data storage device according to an embodiment of the present disclosure.

[0029] Figure 9 Shows a schematic structural diagram of an electronic device suitable for implementing the embodiments of the present disclosure. Detailed implementation manners

[0030] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not used to limit the protection scope of the present disclosure.

[0031] It should be understood that the various steps recited in the method embodiments of the present disclosure can be executed in a different order and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this regard.

[0032] As used herein, the term "including" and its variants are open-ended, that is, "including but not limited to". The term "based on" is "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0033] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0034] It should be noted that the modifications of "one" and "plural" mentioned in the present disclosure are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly stated in the context, it should be understood as "one or more".

[0035] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only for illustrative purposes and are not used to limit the scope of these messages or information.

[0036] It is understood that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, usage scopes, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to users in an appropriate manner and the authorization of users should be obtained in accordance with relevant laws and regulations.

[0037] For example, when receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested by the user will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, an application program, a server, or a storage medium that executes the operation of the technical solution of the present disclosure according to the prompt message.

[0038] As an optional but non-limiting implementation manner, the manner of sending a prompt message to the user in response to receiving an active request from the user may be, for example, in the form of a pop-up window, and the prompt message may be presented in text in the pop-up window. In addition, the pop-up window may also carry a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0039] It is understood that the above process of notifying and obtaining user authorization is only illustrative and does not limit the implementation manner of the present disclosure, and other manners that meet relevant laws and regulations can also be applied to the implementation manner of the present disclosure.

[0040] Meanwhile, it is understood that the data involved in the present technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of corresponding laws and regulations and related regulations.

[0041] Figure 1 is a flowchart of a data storage method according to an embodiment of the present disclosure. This data storage method can be applied to various scenarios for redundant storage of data. Such as Figure 1 shown, this data storage method may include the following steps S11 to S14.

[0042] In step S11, the data of the append-only file is subjected to a first-level striping process according to a first preset data length to obtain a first part and a second part, where the first part and the second part each include a plurality of data blocks, and the length of each data block is the first preset data length.

[0043] The append-only file is a commonly used file abstraction in storage, which allows writing data to the end of the file and randomly reading data, but the written data cannot be modified.

[0044] The first preset data length is configurable. For example, the first preset data length can be configured to 1 byte or other sizes.

[0045] The first-level striping process may first divide the data of the append-only file (i.e., the original data) into multiple data blocks according to a first preset data length, and then alternately divide these multiple data blocks into a first part and a second part. For example, after dividing the data of the append-only file into multiple data blocks, these multiple data blocks can be numbered, and the data blocks with odd numbers can be divided into the first part, and the data blocks with even numbers can be divided into the second part.

[0046] In step S12, a second-level striping process is performed on the first part according to a second preset data length to obtain K pieces of data corresponding to the first part, and a second-level striping process is performed on the second part according to the second preset data length in parallel to obtain K pieces of data corresponding to the second part; wherein, the second preset data length is an integer multiple greater than 1 of the first preset data length, and K is an integer greater than 1.

[0047] The first preset data length can be much smaller than the second preset data length.

[0048] Figure 2 A schematic diagram of the two-level striping process according to an embodiment of the present disclosure is shown. Figure 2 It is illustrated by taking the example of cutting the data of the append-only file into 72 data blocks according to the first preset data length, dividing these 72 data blocks into two parts, part 1 and part 2, striping part 1 into 3 pieces of data according to the second preset data length, and striping part 2 into 3 pieces of data according to the second preset data length. As Figure 2As shown, the data of the append - only file is sliced into data blocks numbered from 0 to 71. Among these data blocks, the data blocks with even numbers are assigned to part 1, and the data blocks with odd numbers are assigned to part 2. Part 1 is striped into 3 pieces of data, namely A0, A1, and A2, and part 2 is striped into 3 pieces of data, namely B0, B1, and B2. After the striping process, 2 stripes are obtained, namely stripe 1 and stripe 2. Stripe 1 includes 6 stripe units, namely the stripe units formed by data blocks 0, 2, 4, 6, 8, 10, the stripe units formed by data blocks 12, 14, 16, 18, 20, 22, the stripe units formed by data blocks 24, 26, 28, 30, 32, 34, the stripe units formed by data blocks 1, 3, 5, 7, 9, 11, the stripe units formed by data blocks 13, 15, 17, 19, 21, 23, and the stripe units formed by data blocks 25, 27, 29, 31, 33, 35. Stripe 2 includes 6 stripe units, namely the stripe units formed by data blocks 36, 38, 40, 42, 44, 46, the stripe units formed by data blocks 48, 50, 52, 54, 56, 58, the stripe units formed by data blocks 60, 62, 64, 66, 68, 70, the stripe units formed by data blocks 37, 39, 41, 43, 45, 47, the stripe units formed by data blocks 49, 51, 53, 55, 57, 59, and the stripe units formed by 61, 63, 65, 67, 69, 71. Figure 2 Among them, data blocks 0, 2, 4, 6, 8, 10 form a stripe unit, data blocks 12, 14, 16, 18, 20, 22 form a stripe unit, and other stripe units are similar. The length of the stripe unit is the second preset data length. In this example, the second preset data length is the length of 6 data blocks, and the stripe length of each stripe is the length of 36 data blocks. In addition, Figure 2 Among them, when striping part 1 into 3 pieces of data, the data blocks are alternately divided into 3 pieces of data with the second preset data length, that is, the stripe unit length as the unit. That is, the stripe unit formed by data blocks 0, 2, 4, 6, 8, 10 is divided into A0, the stripe unit formed by data blocks 12, 14, 16, 18, 20, 22 is divided into A1, the stripe unit formed by data blocks 24, 26, 28, 30, 32, 34 is divided into A2, and then alternately cycle. The stripe unit formed by data blocks 36, 38, 40, 42, 44, 46 is divided into A0, the stripe unit formed by data blocks 48, 50, 52, 54, 56, 58 is divided into A1, and the stripe unit formed by data blocks 60, 62, 64, 66, 68, 70 is divided into A2. The method of striping part 2 into 3 pieces of data is similar to the striping method of part 1.

[0049] In step S13, redundant encoding is performed on the K data pieces corresponding to the first part and the K data pieces corresponding to the second part to obtain redundant data.

[0050] Exemplarily, the first redundant encoding can be performed on the K data pieces corresponding to the first part to obtain the first redundant data corresponding to the first part, and the first redundant encoding can be performed on the K data pieces corresponding to the second part to obtain the first redundant data corresponding to the second part. Among them, the first redundant encoding is used to perform redundant encoding on the data blocks at the corresponding positions of each data piece in the K data pieces by using the first redundant encoding technique; by using the second redundant encoding technique, the data blocks participating in the first redundant encoding of the first part and the data blocks participating in the first redundant encoding of the second part are subjected to the second redundant encoding according to the position corresponding relationship to obtain the second redundant data; and the first redundant encoding technique is used to perform redundant encoding on the second redundant data to obtain the third redundant data. The first redundant data, the second redundant data, and the third redundant data are used as the redundant data.

[0051] Take Figure 3 as an example. The first redundant encoding can perform redundant encoding on the data block numbered 0 in the A0th data piece of part 1, the data block numbered 12 in the A1th data piece of part 1, and the data block numbered 24 in the A2th data piece of part 1 to obtain the first redundant data AE1 of part 1. The first redundant encoding can perform redundant encoding on the data block numbered 2 in the A0th data piece of part 1, the data block numbered 14 in the A1th data piece of part 1, and the data block numbered 26 in the A2th data piece of part 1 to obtain the first redundant data AE2 of part 1, and so on, to complete the first redundant encoding of the data blocks in part 1. For part 2, the first redundant encoding of the data blocks in part 2 is completed in a similar manner to part 1 to obtain the first redundant data BE1, BE2, etc. of part 2.

[0052] The second redundant encoding can perform redundant encoding on the data block numbered 0 in the A0th data piece of part 1 and the data block numbered 1 in the B0th data piece of part 2 to obtain the second redundant data X01. The second redundant encoding can perform redundant encoding on the data block numbered 12 in the A1th data piece of part 1 and the data block numbered 13 in the B1th data piece of part 2 to obtain the second redundant data X11. The second redundant encoding can perform redundant encoding on the data block numbered 24 in the A2th data piece of part 1 and the data block numbered 25 in the B2th data piece of part 2 to obtain the second redundant data X21. And so on, to complete the second redundant encoding of the data blocks participating in the first redundant encoding of the first part and the data blocks participating in the first redundant encoding of the second part according to the position corresponding relationship.

[0053] In addition, redundant encoding (such as the first redundant encoding) can also be performed on the second redundant data obtained by the second redundant encoding to obtain the third redundant data. Still take Figure 3For example, the second redundant data X01, X11, and X21 can be subjected to the first redundant encoding to obtain the third redundant data XE1, the second redundant data X02, X12, and X22 can be subjected to the first redundant encoding to obtain the third redundant data XE2, and so on.

[0054] Through the above various redundant encodings, redundant encoding for the complete stripe can be achieved. A complete stripe refers to a stripe whose length is equal to the preset stripe length. An incomplete stripe refers to a stripe whose length is less than the preset stripe length. An incomplete stripe is usually the last stripe in which the data of the append-only file is striped, because after the data of the append-only file is striped in integer multiples of the preset stripe length, the remaining data length in the append-only file may be less than the preset stripe length, thus generating an incomplete stripe.

[0055] The first redundant encoding can adopt erasure coding. Erasure coding is a coding method that can encode n data blocks into m coding blocks (m > n), and a certain number of coding blocks can be reconstructed back to the original data. Figure 3 For example, erasure coding encodes 3 original data blocks to obtain a redundant data, so a total of 4 copies of the original data are obtained.

[0056] The second redundant encoding can adopt, for example, exclusive or (XOR) encoding.

[0057] In step S14, the K data corresponding to the first part are written into the first computer room, the K data corresponding to the second part are written into the second computer room, and the redundant data are written into the corresponding computer room for storing redundant data.

[0058] When writing, different parts can be stored in different computer rooms. For each copy of the data in the same part, it can be written into different storage slices in the computer room corresponding to this part. For the first redundant data of each part, the first redundant data can be written into a storage slice in the computer room corresponding to this part that is different from the storage slices for storing each copy of the data. For the second redundant data, it can be written into a computer room different from the computer rooms storing each part. For the third redundant data, it can be written into a storage slice in the computer room storing the second redundant data that is different from the storage slice for storing the second redundant data.

[0059] For Figure 3For example, part 1 can be written into the first computer room DC0. That is, the A0th portion of the data of part 1 can be written into the storage shard C0 of the first computer room DC0, the A1th portion of the data of part 1 can be written into the storage shard C1 of the first computer room DC0, the A2th portion of the data of part 1 can be written into the storage shard C2 of the first computer room DC0, and the first redundant data of part 1 can be written into the storage shard C3 of the first computer room DC0. Part 2 can be written into the second computer room DC1. That is, the B0th portion of the data of part 2 can be written into the storage shard D0 of the second computer room DC1, the B1th portion of the data of part 2 can be written into the storage shard D1 of the second computer room DC1, the B2th portion of the data of part 2 can be written into the storage shard D2 of the second computer room DC1, and the first redundant data of part 2 can be written into the storage shard D3 of the second computer room DC1. The second redundant data can be written into the third computer room DC2. That is, the second redundant data between the A0th portion of the data of part 1 and the B0th portion of the data of part 2 can be written into the storage shard E0 of the third computer room DC2, the second redundant data between the A1th portion of the data of part 1 and the B1th portion of the data of part 2 can be written into the storage shard E1 of the third computer room DC2, and the second redundant data between the A2th portion of the data of part 1 and the B2th portion of the data of part 2 can be written into the storage shard E2 of the third computer room DC2. Additionally, for the third redundant data, for example, the third redundant data XE1 obtained by performing the first redundant encoding on the second redundant data X01, X11, and X21, can be written into the storage shard E3 of the third computer room DC2. Through this writing method, multi-copy redundant storage of the original data (that is, the data of the append-only file) can be achieved. Taking Figure 2 the write result of the original data as an example is shown in Figure 3 the figure.

[0060] By adopting the above technical solution, it is possible to perform the first-level striping process on the data of the append-only file according to the first preset data length to obtain the first part and the second part. Among them, the first part and the second part respectively include multiple data blocks, and the length of each data block is the first preset data length; perform the second-level striping process on the first part according to the second preset data length to obtain K portions of data corresponding to the first part, and perform the second-level striping process on the second part according to the second preset data length in parallel to obtain K portions of data corresponding to the second part; where the second preset data length is an integer multiple greater than 1 of the first preset data length; perform redundant encoding on the K portions of data corresponding to the first part and the K portions of data corresponding to the second part to obtain redundant data; write the K portions of data corresponding to the first part into the first computer room, write the K portions of data corresponding to the second part into the second computer room, and write the redundant data into the corresponding computer room for storing the redundant data. Therefore, through the above two-level striping process, a larger strip length can be used, thereby reducing the number of disks or files that need to be accessed during reading and reducing the reading overhead.

[0061] In the related art, if the strip length is relatively large, due to the limitation of strip alignment (i.e., the length of each write must be an integer multiple of the strip length), too much invalid data will be added, thus occupying storage space and reducing storage efficiency. For example, for an incomplete strip, the related art requires adding data to the incomplete strip to make it a complete strip before redundant storage can be performed. However, the data storage method according to the embodiments of the present disclosure does not require that the length of each write must be an integer multiple of the strip length, which can be achieved through the following redundant encoding.

[0062] In some embodiments, for an incomplete strip, the redundant encoding of the K pieces of data corresponding to the first part and the K pieces of data corresponding to the second part in step S13 to obtain redundant data can be achieved in the following manner.

[0063] Perform the first redundant encoding on the K pieces of data corresponding to the first part and the K pieces of data corresponding to the second part respectively, and perform the second redundant encoding on the data blocks for participating in the first redundant encoding of the first part and the data blocks for participating in the first redundant encoding of the second part according to the position correspondence relationship. If not all the data blocks involved in the first redundant encoding are obtained but all the data blocks involved in the second redundant encoding are obtained, then use the second redundant encoding technique to perform the second redundant encoding on the obtained data blocks involved in the second redundant encoding to obtain the second redundant data, and determine the restore log record of the obtained data blocks involved in the second redundant encoding; use the second redundant data and the restore log record as the redundant data; wherein, the first redundant encoding is used to perform redundant encoding on the data blocks at the corresponding positions of each piece of data in the K pieces of data by using the first redundant encoding technique.

[0064] Figure 4 FIG. shows a schematic diagram of redundant encoding of an incomplete strip according to an embodiment of the present disclosure. Figure 4Among them, the part filled with right slashes in the storage shard is the data block and its redundant data written during the previous write operation, and the part filled with left slashes in the storage shard is the data block and its redundant data written during this write operation. The other blank parts of the storage shard have not been written with data blocks and their redundant data. Taking the data block numbered 8 written this time as an example, since only data block 8 exists among the data blocks involved in the first redundant encoding, the first redundant encoding cannot be performed on data block 8. The same is true for other data blocks written this time, and the first redundant encoding cannot be performed either. Taking data block 8 and data block 9 as an example, since both data block 8 and data block 9 exist among the data blocks involved in the second redundant encoding, the second redundant encoding can be performed on data block 8 and data block 9. Similarly, the second redundant encoding can be achieved for other data blocks written this time. The encoding results X04, X05, and X06 of the second redundant encoding are correspondingly written into the storage shard E0 of the third computer room DC2, and the encoding results X11 and X12 of the second redundant encoding are correspondingly written into the storage shard E1 of the third computer room DC2.

[0065] In addition, since Figure 4 the data blocks written this time in it can only achieve the second redundant encoding but not the first redundant encoding, in order to ensure the redundancy, the recovery log records of the data blocks involved in the second redundant encoding can be determined, that is, the recovery log records of the data blocks 7 - 15 written this time and the data block 6 written last time. The reason for determining the recovery log record of the data block 6 written last time is that the data block 6 written last time is used when performing the second redundant encoding on the data block 7 written this time.

[0066] If the first redundant encoding is erasure code encoding and the second redundant encoding is exclusive - or encoding, the above - mentioned redundant encoding for the incomplete stripe can also be expressed as: for the multiple part that satisfies "2 * the first preset data length" in the incomplete stripe, perform the second redundant encoding and determine its recovery log record. In addition, the following restriction can also be imposed on the write, that is, it is required that the data size written at one time is equal to the multiple part of "2 * the first preset data length". In this way, the redundancy can be limited to the configured redundancy ratio, such as the preset erasure code redundancy, exclusive - or encoding redundancy, etc.

[0067] In addition, the determination of the recovery log records of the data blocks involved in the second redundant encoding can be achieved through the following method. That is, based on the Cauchy matrix, redundant encoding is performed on the data blocks involved in the second redundant encoding to obtain the recovery log records. For example, the data blocks in the first computer room DC0 that need to determine the recovery log records are uniformly represented as Data0, and the data blocks in the second computer room DC1 that need to determine the recovery log records are uniformly represented as Data1. Assuming that the second redundant encoding is exclusive - or encoding, the following encoding matrix can be used to achieve redundant encoding:

[0068]

[0069] In the above encoding matrix, XOR T is the result of XOR encoding (i.e., the second redundant encoding), is the restored log record obtained by performing erasure code encoding based on the Cauchy matrix. These three restored log records can be respectively written into the RA storage shards of the first computer room DC0 to obtain the restored log record R0, the RB storage shards of the second computer room DC1 to obtain the restored log record R1, and the RX storage shards of the third computer room DC2 to obtain the restored log record R2. In this way, a 2:4 redundant encoding is formed by using the 2x2 matrix composed of any row of the 2-column Cauchy matrix and (1,1) being invertible in GF(2^8) and jointly using XOR encoding and erasure code encoding.

[0070] Obviously, taking any 2 copies from Data and XOR, or any 2 copies from Data and R (i.e., the restored log record), can restore the complete data. For the case of XOR and one copy of R, its encoding matrix must be invertible in GF(2^8), and the complete data can also be restored because:

[0071]

[0072] By adopting the above technical solution, redundant encoding can be performed on incomplete stripes, without the requirement in the related art that the single write length must be an integer multiple of the stripe length. Therefore, the storage space is saved, the storage efficiency is improved, and a stripe length much larger than the single write length can be allowed, reducing the IO splitting for reading (the reading length is usually smaller than or equivalent to the write length). In addition, for data blocks that cannot achieve the first redundant encoding, redundant data is generated by means of the restored log record, thereby meeting the redundancy requirement. Taking the example of Figure 4 it can achieve a redundancy of 2:4 for 3 computer rooms in EC ratio, that is, any 2 copies out of a total of 6 copies in 3 computer rooms (2 copies in each computer room) can restore the complete data.

[0073] In some embodiments, for an incomplete stripe, the redundant encoding of the K pieces of data corresponding to the first part and the K pieces of data corresponding to the second part in step S13 to obtain redundant data can be implemented in the following manner. That is, perform the first redundant encoding on the K pieces of data corresponding to the first part and the K pieces of data corresponding to the second part respectively, and perform the second redundant encoding on the data blocks used for participating in the first redundant encoding of the first part and the data blocks used for participating in the first redundant encoding of the second part according to the position correspondence relationship. If not all the data blocks involved in the second redundant encoding are obtained, copy the obtained data blocks involved in the second redundant encoding to obtain a restoration log record; use the restoration log record as the redundant data; wherein, the first redundant encoding is used to perform redundant encoding on the data blocks at the corresponding positions of each piece of data in the K pieces of data by using the first redundant encoding technology.

[0074] Figure 5 FIG. shows a schematic diagram of redundant encoding for an incomplete stripe according to an embodiment of the present disclosure. Figure 5 In, the part filled with right slashes in the storage shard is the data block and its redundant data written during the previous write operation, and the part filled with left slashes in the storage shard is the data block and its redundant data written during the current write operation. For data block 16, since only data block 16 exists in the data blocks involved in performing the first redundant encoding and the second redundant encoding, the first redundant encoding and the second redundant encoding cannot be performed on data block 16. Therefore, in this case, data block 16 can be copied to obtain a restoration log record, and the restoration log record is written into the storage shard RA of the first computer room DC0, the storage shard RB of the second computer room DC1, and the storage shard RX of the third computer room DC2 respectively to obtain restoration log records R0, R1, and R2 respectively, thus realizing the redundant storage of data block 16.

[0075] Figure 6 FIG. shows another schematic diagram of redundant encoding for an incomplete stripe according to an embodiment of the present disclosure. Figure 6 In, the part filled with right slashes in the storage shard is the data block and its redundant data written during the previous write operation, and the part filled with left slashes in the storage shard is the data block and its redundant data written during the current write operation. For data block 28, since only data block 28 exists in the data blocks involved in performing the second redundant encoding, the second redundant encoding cannot be performed on data block 28. Therefore, in this case, data block 28 can be copied to obtain a restoration log record, and the restoration log record is written into the storage shard RA of the first computer room DC0, the storage shard RB of the second computer room DC1, and the storage shard RX of the third computer room DC2 respectively to obtain restoration log records R0, R1, and R2 respectively, thus realizing the redundant storage of data block 28.

[0076] Figure 7 Shows another schematic diagram of redundant encoding for incomplete stripes according to an embodiment of the present disclosure. Figure 7 In it, the part filled with right slashes in the storage shard is the data block and its redundant data written during the previous write operation, and the part filled with left slashes in the storage shard is the data block and its redundant data written during the current write operation. For data block 28, since only data block 28 exists among the data blocks involved in performing the second redundant encoding, the second redundant encoding cannot be performed on data block 28. Therefore, in this case, data block 28 can be copied to obtain its restore log record, and the restore log record is written into the storage shards RA of the first computer room DC0, the storage shard RB of the second computer room DC1, and the storage shard RX of the third computer room DC2 respectively, thus realizing redundant storage for data block 28.

[0077] In addition, if the first redundant encoding is erasure code encoding and the second redundant encoding is exclusive OR encoding, the above redundant encoding for incomplete stripes can also be expressed as: directly copying the remainder part of "2 * the first preset data length" in the incomplete stripe to obtain the restore log record.

[0078] By adopting the above technical solutions, redundant encoding can be performed on incomplete stripes, without requiring that the single write length must be an integer multiple of the stripe length as in the related art. Therefore, storage space is saved, storage efficiency is improved, write amplification is reduced, and a stripe length much larger than the single write length can be allowed, reducing the IO splitting of reading (the reading length is usually smaller or equivalent to the write length). In addition, for data blocks where the second redundant encoding cannot be implemented, the restore log record is generated by direct copying to obtain its redundant data, meeting the redundancy requirements.

[0079] In addition, for Figure 7 In the shown example, for the data blocks 24 - 27 written this time, since the data blocks involved in performing the first redundant encoding and the second redundant encoding all exist, the first redundant encoding and the second redundant encoding can be respectively performed on these data blocks. For example, the first redundant encoding is performed on data blocks 0, 12, and 24 to obtain the first redundant data AE1 of part 1, and the second redundant encoding is performed on data blocks 24 and 25 to obtain the second redundant data X21, and so on. That is, if the corresponding data blocks involved in the first redundant encoding all exist and the corresponding data blocks involved in the second redundant encoding also all exist, the first redundant encoding is performed on the corresponding data blocks involved in the first redundant encoding to obtain the first redundant data, and the second redundant encoding is performed on the corresponding data blocks involved in the second redundant encoding to obtain the second redundant data.

[0080] And for Figure 7In the illustrated example, the currently written data blocks 16 - 23 filled with left slashes cannot achieve the first redundant encoding but can achieve the second redundant encoding for these data blocks. Therefore, the method of generating a restoration log record combined with Figure 4 described can be used to generate redundant data.

[0081] In some embodiments, the restoration log record can be written to all computer rooms. Taking the 3 computer rooms mentioned above as an example, the restoration log record can be written to all 3 computer rooms. Alternatively, the restoration log record can also be written to some of all the computer rooms, and one of the computer rooms in the part of the computer rooms performs the operation of writing the restoration log record. Taking Figure 7 as an example, the restoration log record of data block 28 can be written to the second computer room DC1 and the third computer room DC2, and one of the second computer room DC1 and the third computer room DC2 performs the writing. In this way, the cross - computer - room write traffic can be reduced. For example, if DC0 writes the restoration log record to DC1 and DC2, both of these writes are cross - computer - room writes. However, if DC1 writes to DC1 and DC2, the write from DC1 to DC1 is an in - computer - room write, and the write from DC1 to DC2 is a cross - computer - room write. Therefore, the cross - computer - room write traffic is reduced.

[0082] In some embodiments, after all the data blocks involved in the first redundant encoding and the second redundant encoding can be fully obtained, the obtained restoration log record can be deleted. Taking Figure 7 as an example, if subsequent data block 29 exists, enabling the second redundant encoding to be performed on data blocks 28 and 29, the first redundant encoding to be performed on data blocks 4, 16, 28, and the first redundant encoding to be performed on data blocks 5, 17, 29, then in this case, the previously copied restoration log record for data block 28 can be deleted, thereby releasing storage space.

[0083] In some embodiments, the number of copies of the restoration log record may not change with the change of the redundancy ratio. For example, taking a total of 3 computer rooms and storing 3 - copy restoration log records as an example, if the erasure code ratio within the computer room is changed from 3 + 1 to 6 + 2, the number of copies of the restoration log record still only needs to be 3, thus saving storage space.

[0084] Figure 8 is a schematic block diagram of a data storage device according to an embodiment of the present disclosure. This data storage device can be applied to various scenarios for redundant storage of data. As Figure 8As shown, the data storage device 80 may include: a striping processing module 81, configured to perform a first-level striping process on the data of the append-only file according to a first preset data length to obtain a first part and a second part, where the first part and the second part each include a plurality of data blocks, and the length of each data block is the first preset data length; perform a second-level striping process on the first part according to a second preset data length to obtain K pieces of data corresponding to the first part, and perform a second-level striping process on the second part according to the second preset data length in parallel to obtain K pieces of data corresponding to the second part; where the second preset data length is an integer multiple greater than 1 of the first preset data length, and K is an integer greater than 1; a redundancy encoding module 82, configured to perform redundancy encoding on the K pieces of data corresponding to the first part and the K pieces of data corresponding to the second part to obtain redundant data; a writing module 83, configured to write the K pieces of data corresponding to the first part into a first computer room, write the K pieces of data corresponding to the second part into a second computer room, and write the redundant data into the corresponding computer room for storing the redundant data.

[0085] By adopting the above technical solution, it is possible to perform a first-level striping process on the data of the append-only file according to a first preset data length to obtain a first part and a second part, where the first part and the second part each include a plurality of data blocks, and the length of each data block is the first preset data length; perform a second-level striping process on the first part according to a second preset data length to obtain K pieces of data corresponding to the first part, and perform a second-level striping process on the second part according to the second preset data length in parallel to obtain K pieces of data corresponding to the second part; where the second preset data length is an integer multiple greater than 1 of the first preset data length; perform redundancy encoding on the K pieces of data corresponding to the first part and the K pieces of data corresponding to the second part to obtain redundant data; write the K pieces of data corresponding to the first part into a first computer room, write the K pieces of data corresponding to the second part into a second computer room, and write the redundant data into the corresponding computer room for storing the redundant data. Therefore, through the above two-level striping process, a larger strip length can be used, thereby reducing the number of disks or files that need to be accessed during reading and reducing the reading overhead.

[0086] Optionally, performing redundancy encoding on the K pieces of data corresponding to the first part and the K pieces of data corresponding to the second part to obtain redundant data includes:

[0087] Perform a first redundancy encoding on the K pieces of data corresponding to the first part to obtain the first redundant data corresponding to the first part, and perform the first redundancy encoding on the K pieces of data corresponding to the second part to obtain the first redundant data corresponding to the second part, where the first redundancy encoding is used to perform redundancy encoding on the data blocks at the corresponding positions of each piece of data in the K pieces of data by using a first redundancy encoding technique;

[0088] Adopt a second redundant encoding technique, and perform second redundant encoding on the data blocks participating in the first redundant encoding in the first part and the data blocks participating in the first redundant encoding in the second part according to the position correspondence relationship to obtain second redundant data;

[0089] Adopt the first redundant encoding technique to perform redundant encoding on the second redundant data to obtain third redundant data.

[0090] Optionally, performing redundant encoding on K data pieces corresponding to the first part and K data pieces corresponding to the second part to obtain redundant data includes:

[0091] Perform first redundant encoding on the K data pieces corresponding to the first part and the K data pieces corresponding to the second part respectively, and perform second redundant encoding on the data blocks participating in the first redundant encoding in the first part and the data blocks participating in the first redundant encoding in the second part according to the position correspondence relationship. If not all data blocks involved in the first redundant encoding are obtained but all data blocks involved in the second redundant encoding are obtained, then adopt the second redundant encoding technique to perform the second redundant encoding on the obtained data blocks involved in the second redundant encoding to obtain second redundant data, and determine the restoration log record of the obtained data blocks involved in the second redundant encoding; use the second redundant data and the restoration log record as the redundant data;

[0092] Among them, the first redundant encoding is used to perform redundant encoding on the data blocks at the corresponding positions of each data piece in the K data pieces by using the first redundant encoding technique.

[0093] Optionally, determining the restoration log record of the obtained data blocks involved in the second redundant encoding includes:

[0094] Perform redundant encoding on the obtained data blocks involved in the second redundant encoding based on a Cauchy matrix to obtain a restoration log record.

[0095] Optionally, performing redundant encoding on K data pieces corresponding to the first part and K data pieces corresponding to the second part to obtain redundant data includes:

[0096] Perform first redundant encoding on the K data pieces corresponding to the first part and the K data pieces corresponding to the second part respectively, and perform second redundant encoding on the data blocks participating in the first redundant encoding in the first part and the data blocks participating in the first redundant encoding in the second part according to the position correspondence relationship. If not all data blocks involved in the second redundant encoding are obtained, then copy the obtained data blocks involved in the second redundant encoding to obtain a restoration log record; use the restoration log record as the redundant data;

[0097] Among them, the first redundant encoding is used to perform redundant encoding on the data blocks at the corresponding positions of each piece of data in the K pieces of data by using the first redundant encoding technology.

[0098] Optionally, the data storage device 80 further includes a deletion module, configured to delete the obtained restoration log record after all the data blocks involved in the first redundant encoding and the second redundant encoding can be fully obtained.

[0099] Optionally, the writing module 83 is further configured to:

[0100] Write the restoration log record into all computer rooms; or

[0101] Write the restoration log record into some of all the computer rooms, and the operation of writing the restoration log record is performed by one of the some computer rooms.

[0102] The specific implementation manners of the operations performed by each module in the data storage device according to the embodiments of the present disclosure have been described in detail in the related methods, and will not be elaborated herein.

[0103] The present disclosure also provides a computer-readable medium, on which a computer program is stored, and when the computer program is executed by a processing device, the steps of any one of the methods in the present disclosure are implemented.

[0104] The present disclosure also provides an electronic device, including:

[0105] A storage device, on which a computer program is stored;

[0106] A processing device, configured to execute the computer program in the storage device to implement the steps of any one of the methods in the present disclosure.

[0107] The present disclosure also provides a computer program product, including a computer program, and when the computer program is executed by a processor, the steps of any one of the methods in the present disclosure are implemented.

[0108] Next, refer to Figure 9 , which shows a schematic structural diagram of an electronic device 600 suitable for implementing the embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet Computers), PMPs (Portable Multimedia Players), in-vehicle terminals (such as in-vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 9 The electronic device shown is only an example, and should not impose any limitation on the functions and usage scopes of the embodiments of the present disclosure.

[0109] As Figure 9As shown, the electronic device 600 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 601, which may perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0110] Generally, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or wiredly to exchange data. Although Figure 9 the electronic device 600 with various devices is shown, it should be understood that it is not required to implement or have all the shown devices. Instead, more or fewer devices may be implemented or had.

[0111] Specifically, according to an embodiment of the present disclosure, the process described above with reference to the flowchart may be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes program codes for performing the method shown in the flowchart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above functions defined in the method of the embodiment of the present disclosure are executed.

[0112] It should be noted that the above-mentioned computer-readable medium in the present disclosure can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, apparatus, or device. In the present disclosure, the computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, in which computer-readable program code is carried. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, and this computer-readable signal medium can send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted by any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0113] In some embodiments, the client and the server can communicate using any currently known or future-developed network protocol such as HTTP (HyperText Transfer Protocol), and can be interconnected with digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), the Internet (e.g., the Internet), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0114] The above-mentioned computer-readable medium can be included in the above-mentioned electronic device; it can also exist separately and not be assembled into the electronic device.

[0115] The above computer-readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: perform first-level striping on the data of the append-only file according to a first preset data length to obtain a first part and a second part, wherein the first part and the second part respectively include a plurality of data blocks, and the length of each data block is the first preset data length; perform second-level striping on the first part according to a second preset data length to obtain K pieces of data corresponding to the first part, and perform the second-level striping on the second part according to the second preset data length in parallel to obtain K pieces of data corresponding to the second part; wherein the second preset data length is an integer multiple greater than 1 of the first preset data length, and K is an integer greater than 1; perform redundant encoding on the K pieces of data corresponding to the first part and the K pieces of data corresponding to the second part to obtain redundant data; write the K pieces of data corresponding to the first part into a first computer room, write the K pieces of data corresponding to the second part into a second computer room, and write the redundant data into the corresponding computer room for storing the redundant data.

[0116] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages or combinations thereof. The programming languages include, but are not limited to, object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0117] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a portion of code that contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks may occur in a different order than noted in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or by a combination of dedicated hardware and computer instructions.

[0118] The modules described in the embodiments of the present disclosure can be implemented in software or in hardware. Among them, the name of the module does not constitute a limitation on the module itself in some cases. For example, the redundant coding module can also be described as "the module that performs redundant coding on the K copies of data corresponding to the first part and the K copies of data corresponding to the second part to obtain redundant data".

[0119] The functions described above herein can be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that can be used include: Field Programmable Gate Arrays (FPGA), Application Specific Integrated Circuits (ASIC), Application Specific Standard Products (ASSP), System on a Chip (SOC), Complex Programmable Logic Devices (CPLD), and so on.

[0120] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a Read-Only Memory (ROM), an Erasable Programmable Read-Only Memory (EPROM or Flash Memory), an optical fiber, a portable Compact Disc Read-Only Memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0121] According to one or more embodiments of the present disclosure, Example 1 provides a data storage method, including:

[0122] Performing first-level striping processing on the data of the append-only file according to a first preset data length to obtain a first part and a second part, wherein the first part and the second part respectively include a plurality of data blocks, and the length of each data block is the first preset data length;

[0123] Performing second-level striping processing on the first part according to a second preset data length to obtain K pieces of data corresponding to the first part, and parallelly performing the second-level striping processing on the second part according to the second preset data length to obtain K pieces of data corresponding to the second part; wherein the second preset data length is an integer multiple greater than 1 of the first preset data length, and K is an integer greater than 1;

[0124] Performing redundant encoding on the K pieces of data corresponding to the first part and the K pieces of data corresponding to the second part to obtain redundant data;

[0125] Writing the K pieces of data corresponding to the first part into a first computer room, writing the K pieces of data corresponding to the second part into a second computer room, and writing the redundant data into the corresponding computer room for storing the redundant data.

[0126] According to one or more embodiments of the present disclosure, Example 2 provides the method of Example 1, wherein performing redundant encoding on the K pieces of data corresponding to the first part and the K pieces of data corresponding to the second part to obtain redundant data includes:

[0127] Performing first redundant encoding on the K pieces of data corresponding to the first part to obtain first redundant data corresponding to the first part, and performing the first redundant encoding on the K pieces of data corresponding to the second part to obtain first redundant data corresponding to the second part, wherein the first redundant encoding is used to perform redundant encoding on the data blocks at the corresponding positions of each piece of data in the K pieces of data by using a first redundant encoding technique;

[0128] Using a second redundant encoding technique to perform second redundant encoding on the data blocks participating in the first redundant encoding of the first part and the data blocks participating in the first redundant encoding of the second part according to the position correspondence relationship to obtain second redundant data;

[0129] Performing redundant encoding on the second redundant data by using the first redundant encoding technique to obtain third redundant data.

[0130] According to one or more embodiments of the present disclosure, Example 3 provides the method of Example 1, wherein performing redundant encoding on the K pieces of data corresponding to the first part and the K pieces of data corresponding to the second part to obtain redundant data includes:

[0131] Perform first redundant encoding on the K pieces of data corresponding to the first part and the K pieces of data corresponding to the second part respectively, and perform second redundant encoding on the data blocks used in the first part for participating in the first redundant encoding and the data blocks used in the second part for participating in the first redundant encoding according to the position correspondence relationship. If not all the data blocks involved in the first redundant encoding are obtained but all the data blocks involved in the second redundant encoding are obtained, then use the second redundant encoding technique to perform the second redundant encoding on the obtained data blocks involved in the second redundant encoding to obtain second redundant data, and determine the restoration log record of the obtained data blocks involved in the second redundant encoding; use the second redundant data and the restoration log record as the redundant data;

[0132] Wherein, the first redundant encoding is used to perform redundant encoding on the data blocks at the corresponding positions of each piece of data in the K pieces of data by using the first redundant encoding technique.

[0133] According to one or more embodiments of the present disclosure, Example 4 provides the method of Example 3, wherein determining the restoration log record of the obtained data blocks involved in the second redundant encoding includes:

[0134] Perform redundant encoding on the obtained data blocks involved in the second redundant encoding based on the Cauchy matrix to obtain the restoration log record.

[0135] According to one or more embodiments of the present disclosure, Example 5 provides the method of Example 1, wherein performing redundant encoding on the K pieces of data corresponding to the first part and the K pieces of data corresponding to the second part to obtain redundant data includes:

[0136] Perform first redundant encoding on the K pieces of data corresponding to the first part and the K pieces of data corresponding to the second part respectively, and perform second redundant encoding on the data blocks used in the first part for participating in the first redundant encoding and the data blocks used in the second part for participating in the first redundant encoding according to the position correspondence relationship. If not all the data blocks involved in the second redundant encoding are obtained, then copy the obtained data blocks involved in the second redundant encoding to obtain the restoration log record; use the restoration log record as the redundant data;

[0137] Wherein, the first redundant encoding is used to perform redundant encoding on the data blocks at the corresponding positions of each piece of data in the K pieces of data by using the first redundant encoding technique.

[0138] According to one or more embodiments of the present disclosure, Example 6 provides the method of any one of Examples 3 to 5, wherein the method further includes:

[0139] After all the data blocks involved in the first redundant encoding and the second redundant encoding can be fully obtained, delete the obtained restoration log records.

[0140] According to one or more embodiments of the present disclosure, Example 7 provides the method of any one of Examples 3 to 5, wherein the method further includes:

[0141] Write the restoration log records to all computer rooms; or

[0142] Write the restoration log records to some of all computer rooms, and the operation of writing the restoration log records is performed by one of the some computer rooms.

[0143] According to one or more embodiments of the present disclosure, Example 8 provides a data storage device, including:

[0144] A striping processing module, configured to perform a first-level striping process on the data of the append-only file according to a first preset data length to obtain a first part and a second part, wherein the first part and the second part respectively include a plurality of data blocks, and the length of each data block is the first preset data length; perform a second-level striping process on the first part according to a second preset data length to obtain K copies of data corresponding to the first part, and perform a second-level striping process on the second part according to the second preset data length in parallel to obtain K copies of data corresponding to the second part; wherein the second preset data length is an integer multiple greater than 1 of the first preset data length, and K is an integer greater than 1;

[0145] A redundant encoding module, configured to perform redundant encoding on the K copies of data corresponding to the first part and the K copies of data corresponding to the second part to obtain redundant data;

[0146] A writing module, configured to write the K copies of data corresponding to the first part to a first computer room, write the K copies of data corresponding to the second part to a second computer room, and write the redundant data to the corresponding computer room for storing the redundant data.

[0147] According to one or more embodiments of the present disclosure, Example 9 provides a computer-readable medium, on which a computer program is stored, and when the computer program is executed by a processing device, the steps of the method of any one of Examples 1-7 are implemented.

[0148] According to one or more embodiments of the present disclosure, Example 10 provides an electronic device, including:

[0149] A storage device, on which a computer program is stored;

[0150] A processing device, configured to execute the computer program in the storage device to implement the steps of the method of any one of Examples 1-7.

[0151] According to one or more embodiments of the present disclosure, Example 11 provides a computer program product including a computer program which, when executed by a processor, implements the steps of the method according to any one of Examples 1-7.

[0152] The above description is only a preferred embodiment of the present disclosure and an explanation of the applied technical principles. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosure concept. For example, the technical solutions formed by mutually replacing the above features with the technical features (but not limited to) having similar functions disclosed in the present disclosure.

[0153] In addition, although the operations are depicted in a particular order, this should not be construed as requiring that the operations be performed in the particular order shown or in sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the present disclosure. Certain features described in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented separately or in any suitable sub-combination in multiple embodiments.

[0154] Although the subject matter has been described in language specific to structural features and / or methodological acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. On the contrary, the specific features and acts described above are merely example forms for implementing the claims. Regarding the apparatus in the above embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method, and will not be elaborated here.

Claims

1. A data storage method, characterized in that, Including: Performing first-level striping on the data of the append-only file according to a first preset data length to obtain a first part and a second part, wherein the first part and the second part respectively include a plurality of data blocks, and the length of each data block is the first preset data length; Performing second-level striping on the first part according to a second preset data length to obtain K pieces of data corresponding to the first part, and parallelly performing the second-level striping on the second part according to the second preset data length to obtain K pieces of data corresponding to the second part; wherein, the second preset data length is an integer multiple greater than 1 of the first preset data length, and K is an integer greater than 1; Performing redundant encoding on the K pieces of data corresponding to the first part and the K pieces of data corresponding to the second part to obtain redundant data; Writing the K pieces of data corresponding to the first part into a first computer room, writing the K pieces of data corresponding to the second part into a second computer room, and writing the redundant data into the corresponding computer room for storing the redundant data.

2. The method according to claim 1, wherein The performing redundant encoding on the K pieces of data corresponding to the first part and the K pieces of data corresponding to the second part to obtain redundant data includes: Performing first redundant encoding on the K pieces of data corresponding to the first part to obtain first redundant data corresponding to the first part, and performing the first redundant encoding on the K pieces of data corresponding to the second part to obtain first redundant data corresponding to the second part, wherein the first redundant encoding is used to perform redundant encoding on the data blocks at the corresponding positions of each piece of data in the K pieces of data by using a first redundant encoding technique; Using a second redundant encoding technique to perform second redundant encoding on the data blocks of the first part participating in the first redundant encoding and the data blocks of the second part participating in the first redundant encoding according to the position correspondence relationship to obtain second redundant data; Performing redundant encoding on the second redundant data by using the first redundant encoding technique to obtain third redundant data.

3. The method according to claim 1, characterized in that, The performing redundant encoding on the K pieces of data corresponding to the first part and the K pieces of data corresponding to the second part to obtain redundant data includes: Performing first redundant encoding on the K pieces of data corresponding to the first part and the K pieces of data corresponding to the second part respectively, and performing second redundant encoding on the data blocks of the first part participating in the first redundant encoding and the data blocks of the second part participating in the first redundant encoding according to the position correspondence relationship. If not all the data blocks involved in the first redundant encoding are obtained but all the data blocks involved in the second redundant encoding are obtained, then using the second redundant encoding technique to perform the second redundant encoding on the obtained data blocks involved in the second redundant encoding to obtain second redundant data, and determining the restoration log record of the obtained data blocks involved in the second redundant encoding; taking the second redundant data and the restoration log record as the redundant data; Wherein, the first redundant encoding is used to perform redundant encoding on the data blocks at the corresponding positions of each piece of data in the K pieces of data by using a first redundant encoding technique.

4. The method according to claim 3, characterized in that, Determining the restoration log record of the data blocks involved in the obtained second redundant encoding includes: Performing redundant encoding on the data blocks involved in the obtained second redundant encoding based on a Cauchy matrix to obtain the restoration log record.

5. The method according to claim 1, wherein The performing redundant encoding on the K pieces of data corresponding to the first part and the K pieces of data corresponding to the second part to obtain redundant data includes: Performing first redundant encoding on the K pieces of data corresponding to the first part and the K pieces of data corresponding to the second part respectively, and performing second redundant encoding on the data blocks used for participating in the first redundant encoding in the first part and the data blocks used for participating in the first redundant encoding in the second part according to the position correspondence relationship. If all the data blocks involved in the second redundant encoding cannot be obtained, copying the obtained data blocks involved in the second redundant encoding to obtain the restoration log record; using the restoration log record as the redundant data. Wherein, the first redundant encoding is used to perform redundant encoding on the data blocks at the corresponding positions of each piece of data in the K pieces of data by using a first redundant encoding technique.

6. The method according to any one of claims 3 to 5, characterized in that, The method further includes: After all the data blocks involved in the first redundant encoding and the second redundant encoding can be obtained, deleting the obtained restoration log record.

7. The method according to any one of claims 3 to 5, characterized in that, The method further includes: Writing the restoration log record into all the computer rooms; or Writing the restoration log record into some of all the computer rooms, and one of the some computer rooms performs the operation of writing the restoration log record.

8. A data storage device, characterized in that, It includes: A striping processing module, configured to perform first-level striping processing on the data of the append-only file according to a first preset data length to obtain a first part and a second part, wherein the first part and the second part respectively include a plurality of data blocks, and the length of each data block is the first preset data length; performing second-level striping processing on the first part according to a second preset data length to obtain K pieces of data corresponding to the first part, and concurrently performing the second-level striping processing on the second part according to the second preset data length to obtain K pieces of data corresponding to the second part; wherein, the second preset data length is an integer multiple greater than 1 of the first preset data length, and K is an integer greater than 1. A redundant encoding module, configured to perform redundant encoding on the K pieces of data corresponding to the first part and the K pieces of data corresponding to the second part to obtain redundant data. A writing module, configured to write the K pieces of data corresponding to the first part into a first computer room, write the K pieces of data corresponding to the second part into a second computer room, and write the redundant data into the corresponding computer room for storing the redundant data.

9. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processing device, it implements the steps of the method according to any one of claims 1-7.

10. An electronic device, characterized in that, It includes: A storage device, on which a computer program is stored; A processing device, configured to execute the computer program in the storage device to implement the steps of the method according to any one of claims 1-7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-7.

Images