A method, apparatus and device for data asynchronous redundancy coding

Abstract

The application provides a method, device and equipment for data asynchronous redundancy coding, applied to a distributed storage system. The method comprises the following steps: obtaining a data write request; determining corresponding sub-objects and their copies, redundant sub-objects based on the write request and predetermined parameters; determining the range of a covered striped data redundancy group and corresponding sub-objects and their copies, redundant sub-objects in the range based on the predetermined parameters and the sub-objects; cutting the data and writing the data into corresponding sub-object copies one by one, creating a metadata file of each sub-object copy, recording the write offset and data size of the corresponding sub-object; performing asynchronous redundancy coding on the data according to the metadata file of the sub-object copy based on an asynchronous redundancy coding service, and deleting the sub-object copy and the metadata file of the sub-object copy which have completed the asynchronous redundancy coding. Through the method, the data availability can be improved by using copy redundancy protection before the newly written data completes the redundancy protection of erasure coding.

Description

Technical Field

[0001] This application relates to the field of computer data processing technology, and in particular to a technique for asynchronous redundancy coding of data applied to distributed storage systems. Background Technology

[0002] In distributed storage systems, high data availability is crucial. To achieve this, distributed storage systems typically employ data redundancy strategies to protect data. This ensures that even when data anomalies occur within a certain tolerance range (such as disk failure causing data to become inaccessible), the redundancy strategy guarantees normal data access.

[0003] Erasure coding is a commonly used data redundancy strategy. First, user data to be written to a distributed storage system is divided into several original data fragments of the same size. Then, every n original data fragments are grouped together and erasure coding is used to calculate m redundant data fragments of the same size. These n original data fragments and m redundant data fragments are combined to form a striped data redundancy group. Typically, different data fragments within a striped data redundancy group are stored in different sub-objects on different storage nodes (or different disks on the same storage node). Data fragments of the same order in different striped data redundancy groups are stored in the same sub-object. When any number of no more than m original data fragments in a striped data redundancy group are abnormal, it can be recovered by decoding the other n data fragments. Therefore, if an abnormal data fragment occurs within a striped data redundancy group due to unforeseen circumstances, as long as the number of abnormal data fragments is less than m, it can be recovered using any of the other n normal data fragments, thus achieving high data availability.

[0004] Because erasure coding redundancy strategies work by fragmenting user data into original data fragments and then using erasure coding to calculate redundant data fragments, and storing the original data fragments and corresponding redundant data fragments in a striped redundancy group in the distributed storage system, rather than simply copying the user data, for newly written user data, in order to complete the calculation and / or update of redundant data, it is sometimes necessary not only to read the updated original data fragments in each relevant striped redundancy group, but also to read other unupdated original data fragments in the corresponding striped redundancy group, and then perform erasure coding to calculate the new redundant data fragments, thus completing the redundant data update. If the update of both the original data fragments and the redundant data fragments were performed synchronously every time user data was written, it would impose a significant read / write overhead on the distributed storage system.

[0005] In existing technologies, an asynchronous approach can be used to update the original data fragments and redundant data fragments when writing user data. That is, each time user data is written, as long as the updated original data fragments are written to the corresponding sub-objects, the writing of user data is considered complete. Then, the update of redundant data fragments is completed asynchronously through the background asynchronous service program.

[0006] However, before the redundant data shards are updated asynchronously through the background asynchronous service program, the original data shards that have been updated are in a state without redundancy protection. If any abnormality occurs in these original data shards that have been updated, or even if other unupdated original data shards experience some degree of abnormality, the corresponding user data will be unable to be read or written, resulting in user data unavailability. This is unacceptable for a distributed storage system with frequent data read and write operations. Summary of the Invention

[0007] The purpose of this application is to provide a method, apparatus, and device for asynchronous data redundancy coding, applicable to distributed storage systems, to at least partially solve the technical problem of lack of redundancy protection for user data in distributed storage systems where asynchronous data redundancy coding has not been completed.

[0008] According to one aspect of this application, a method for asynchronous redundancy coding of data is provided, applied to a distributed storage system, wherein the method includes:

[0009] Obtain a first data write request, wherein the data write request includes identification information, write offset, data size, and first data to be written;

[0010] Based on the number of original data fragments and redundant data fragments, the number of replicas, and the identification information determined by the distributed storage system for the striped data redundancy group, a sub-object of the corresponding storage node is determined. The sub-object includes an original data sub-object and corresponding original data sub-object replicas and redundant data sub-objects, which are used to store the original data fragments, original data fragment replicas, and redundant data fragments, respectively.

[0011] Based on the size of the data shard, the write offset, the data size, and the original data sub-object, the range of the striped data redundancy group covered by the first data and the corresponding original data sub-object and its copy, and redundant data sub-object of each striped data redundancy group within the range are determined.

[0012] Based on the size of the data shards, the first data is divided into several raw data shards, and each is written to a corresponding raw data sub-object copy of the corresponding striped data redundancy group. Metadata files are also created for each raw data sub-object copy, recording the write offset and data size of the corresponding raw data sub-object.

[0013] Based on the asynchronous redundancy coding service running on the storage node, the first data is asynchronously redundant coded according to the metadata file of each original data sub-object copy, and the original data sub-object copy and its metadata file that have completed asynchronous redundancy coding are deleted.

[0014] Optionally, the step of asynchronously redundantly encoding the first data based on the metadata file of each original data sub-object copy includes:

[0015] Read the metadata file of each original data sub-object copy to determine the striped data redundancy group corresponding to the first data overlay for each original data sub-object;

[0016] For each striped data redundancy group covered by the first data, the corresponding original data fragment is obtained from the original data sub-object copy, and other corresponding original data fragments are obtained from the original data sub-object, and merged into the original data fragment of the striped data redundancy group.

[0017] Asynchronous redundancy coding is performed on the original data fragments of the merged striped data redundancy group to obtain new redundant data fragments of the striped data redundancy group. The original data fragments in the corresponding original data sub-object copy and the generated new redundant data fragments are copied to the corresponding original data sub-object and the corresponding redundant data sub-object.

[0018] Traverse each striped data redundancy group covered by the first data to complete the asynchronous redundancy coding of the first data.

[0019] Optionally, if, when retrieving other corresponding original data fragments from the original data sub-object, there are original data fragments that cannot be successfully retrieved, the method further includes:

[0020] Based on the data fragments successfully obtained from the original data sub-objects and redundant data sub-objects corresponding to the striped data redundancy group, data recovery is performed on the original data fragments that could not be successfully obtained.

[0021] Obtain the original data fragments after data recovery.

[0022] Optionally, if a second data write request including the identification information is obtained before asynchronous redundancy encoding of the first data, the method further includes:

[0023] Based on the size of the data shards determined by the distributed storage system, the write offset and data size in the second data write request, and the sub-objects of the corresponding storage nodes, the range of the striped data redundancy group covered by the second data to be written in the second data write request and the corresponding original data sub-objects and their copies and redundant data sub-objects of each striped data redundancy group within the range are determined.

[0024] Based on the size of the data shards, the second data is divided into several original data shards, and each shard is written to the corresponding original data sub-object copy of the corresponding striped data redundancy group.

[0025] If there is no metadata file for the original data sub-object copy corresponding to the striped data redundancy group, then create the metadata file for the original data sub-object copy and record the write offset and data size of the corresponding original data sub-object. If there is a metadata file for the original data sub-object copy corresponding to the striped data redundancy group, then update the records in the metadata file. If the write offset and data size of this time are adjacent to or overlap with the write offset and data size recorded in the metadata file in terms of range, then merge them and record the write offset and data size of the corresponding original data sub-object after merging.

[0026] The step of asynchronously redundantly encoding the first data based on the metadata file of each original data sub-object copy includes:

[0027] Asynchronous redundant encoding is performed on the first data and the second data based on the metadata file of each original data sub-object copy.

[0028] Optionally, if a third data read request including the identification information is obtained before asynchronous redundancy encoding of the first data, the method further includes:

[0029] Based on the size of the data shards determined by the distributed storage system, the read offset and data size in the third data read request, and the sub-objects of the corresponding storage nodes, the range of the striped data redundancy group covered by the third data to be read in the third data read request and the corresponding original data sub-objects and their copies and redundant data sub-objects of each striped data redundancy group within the range are determined.

[0030] The read offset and data size in the third data read request are compared one by one with the write offset and data size of the corresponding original data sub-object recorded in the metadata file of the original data sub-object copy of each striped data redundancy group within the range. If they are included, the corresponding original data fragment is read from the original data sub-object copy. If they are not included, the corresponding original data fragment is determined from the original data sub-object and / or redundant data sub-object of the striped data redundancy group to complete the reading of the third data.

[0031] Optionally, the method for asynchronous data redundancy coding further includes:

[0032] Obtain a fourth data read request including the identification information;

[0033] Based on the size of the data shards determined by the distributed storage system, the read offset and data size in the fourth data read request, and the sub-objects of the corresponding storage nodes, the range of the striped data redundancy group covered by the fourth data to be read in the fourth data read request and the corresponding original data sub-objects and redundant data sub-objects of each striped data redundancy group within the range are determined.

[0034] The original data fragments are determined one by one from the corresponding original data sub-objects and / or redundant data sub-objects of each striped data redundancy group within the range to complete the reading of the fourth data.

[0035] According to another aspect of this application, an apparatus for asynchronous redundancy coding of data is provided, deployed in a distributed storage system, wherein the apparatus includes:

[0036] The first module is used to obtain a first data write request, wherein the data write request includes identification information, write offset, data size and first data to be written;

[0037] The second module is used to: determine the sub-object of the corresponding storage node based on the number of original data fragments and the number of redundant data fragments, the number of replicas, and the identification information of the striped data redundancy group determined by the distributed storage system. The sub-object includes an original data sub-object and a corresponding original data sub-object replica and a redundant data sub-object, which are used to store the original data fragments, the original data fragment replicas, and the redundant data fragments, respectively.

[0038] The third module is used to: determine the range of the striped data redundancy group covered by the first data and the corresponding original data sub-object and its copy and redundant data sub-object for each striped data redundancy group within the range, based on the size of the data shard determined by the distributed storage system, the write offset and the data size, and the original data sub-object;

[0039] The fourth module is used to: divide the first data into several raw data fragments based on the size of the data fragments, write them one by one to the corresponding raw data sub-object copies of the corresponding striped data redundancy group, and create a metadata file for each raw data sub-object copy, recording the write offset and data size of the corresponding raw data sub-object;

[0040] The fifth module is used to: perform asynchronous redundancy coding on the first data based on the asynchronous redundancy coding service running on the storage node, according to the metadata file of each original data sub-object copy, and delete the original data sub-object copy and its metadata file that have completed asynchronous redundancy coding.

[0041] Optionally, if the first module receives a second data write request including the identification information before asynchronous redundancy encoding of the first data, the third module is further configured to:

[0042] Based on the size of the data shards determined by the distributed storage system, the write offset and data size in the second data write request, and the sub-objects of the corresponding storage nodes, the range of the striped data redundancy group covered by the second data to be written in the second data write request and the corresponding original data sub-objects and their copies and redundant data sub-objects of each striped data redundancy group within the range are determined.

[0043] The fourth module is further configured to: divide the second data into several original data fragments based on the size of the data fragments, and write them one by one to the corresponding original data sub-object replicas of the corresponding striped data redundancy group; if there is no metadata file for the corresponding original data sub-object replicas of the corresponding striped data redundancy group, then create a metadata file for the original data sub-object replicas and record the write offset and data size of the corresponding original data sub-objects; if there is a metadata file for the corresponding original data sub-object replicas of the corresponding striped data redundancy group, then update the write offset and data size of the corresponding original data sub-objects recorded in the metadata file; if the write offset and data size of the current data sub-objects are adjacent to or overlap with the write offset and data size recorded in the metadata file, then merge them and record the write offset and data size of the corresponding original data sub-objects after merging.

[0044] The fifth module is used to: perform asynchronous redundancy coding on the first data and the second data based on the asynchronous redundancy coding service running on the storage node, according to the metadata file of each original data sub-object or its copy, and delete the original data sub-object copy and its metadata file that have completed asynchronous redundancy coding.

[0045] Optionally, if the first module receives a third data read request including the identification information before asynchronous redundancy encoding of the first data, the third module is further configured to:

[0046] Based on the size of the data shards determined by the distributed storage system, the read offset and data size in the third data read request, and the sub-objects of the corresponding storage nodes, the range of the striped data redundancy group covered by the third data to be read in the third data read request and the corresponding original data sub-objects and their copies and redundant data sub-objects of each striped data redundancy group within the range are determined.

[0047] The fourth module is further configured to: compare the read offset and data size in the third data read request with the write offset and data size of the corresponding original data sub-object recorded in the metadata file of the original data sub-object copy of each striped data redundancy group within the range; if they are included, read the corresponding original data fragment from the original data sub-object copy; if they are not included, determine the corresponding original data fragment from the original data sub-object and / or redundant data sub-object of the striped data redundancy group to complete the reading of the third data.

[0048] Optionally, if after asynchronous redundancy encoding of the first data, the first module obtains a fourth data read request including the identification information, the third module is further configured to:

[0049] Based on the size of the data shards determined by the distributed storage system, the read offset and data size in the fourth data read request, and the sub-objects of the corresponding storage nodes, the range of the striped data redundancy group covered by the fourth data to be read in the fourth data read request and the corresponding original data sub-objects and redundant data sub-objects of each striped data redundancy group within the range are determined.

[0050] The fourth module is further configured to: determine the original data fragments one by one from the original data sub-objects and / or redundant data sub-objects corresponding to each striped data redundancy group within the range, so as to complete the reading of the fourth data.

[0051] Compared with existing technologies, this application provides a method, apparatus, and device for asynchronous data redundancy coding, applied to a distributed storage system. The method includes: acquiring a first data write request, wherein the data write request includes identification information, a write offset, a data size, and first data to be written; determining a sub-object of a corresponding storage node based on the number of original data fragments and the number of redundant data fragments, the number of replicas, and the identification information of a striped data redundancy group determined by the distributed storage system, wherein the sub-object includes an original data sub-object and corresponding original data sub-object replicas and redundant data sub-objects, respectively used to store the original data fragments, original data fragment replicas, and redundant data fragments; and determining the first data write request based on the size of the data fragments determined by the distributed storage system, the write offset, the data size, and the original data sub-object. The first data covers the range of striped data redundancy groups and the corresponding original data sub-objects and their copies, and redundant data sub-objects for each striped data redundancy group within the range. Based on the size of the data shards, the first data is divided into several original data shards, and each is written to the corresponding original data sub-object copy of the striped data redundancy group, and a metadata file is created for each original data sub-object copy, recording the write offset and data size of the corresponding original data sub-object. Based on the asynchronous redundancy coding service running on the storage node, the first data is asynchronously redundant coded according to the metadata file of each original data sub-object copy, and the original data sub-object copies and their metadata files that have completed asynchronous redundancy coding are deleted. Through this method, in a distributed storage system that uses erasure coding asynchronous coding to provide data redundancy protection, for newly written user data, before asynchronous redundancy coding is performed on the user data and erasure coding redundancy protection is completed, replica redundancy protection can be applied to the user data to avoid the newly written user data being in a state without redundancy protection, affecting its high availability. Attached Figure Description

[0052] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0053] Figure 1 This diagram illustrates a data storage organization method for user data after redundancy encoding using erasure coding.

[0054] Figure 2 This invention provides a schematic flowchart of a method for asynchronous redundancy coding of data, representing one aspect of this application.

[0055] Figure 3 This diagram illustrates a data storage organization for asynchronous redundancy coding, representing an alternative embodiment of one aspect of this application.

[0056] Figure 4 This diagram illustrates the data storage organization after first asynchronous redundancy encoding, according to an optional embodiment of one aspect of this application.

[0057] Figure 5 This diagram illustrates the data storage organization of asynchronously redundantly encoded first and second data according to an optional embodiment of one aspect of this application.

[0058] Figure 6 A schematic diagram of an apparatus for asynchronous redundancy coding of data is shown, representing another aspect of this application.

[0059] The same or similar reference numerals in the accompanying drawings represent the same or similar parts. Detailed Implementation

[0060] The present invention will now be described in further detail with reference to the accompanying drawings.

[0061] In a typical configuration of various embodiments of this application, each trusted party of the device, system and / or each module of the apparatus may include one or more processors (CPU), input / output interfaces, network interfaces and memory.

[0062] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0063] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include non-transitory computer-readable media, such as modulated data signals and carrier waves.

[0064] Erasure coding is a commonly used data redundancy strategy in distributed storage systems. During the deployment of a distributed storage system, the hardware and software configuration, coding efficiency, and storage space utilization are determined based on the application scenario. The relevant parameters (n, m) of the erasure coding are also determined, along with the chunk size (theoretically unlimited, but considering coding efficiency and storage space utilization, it is usually a multiple of 2KB, such as 2KB, 4KB, 16KB, 64KB, 128KB, etc. Once determined for the same distributed storage system, this parameter is generally not changed). Then, the system plans to store the original data sub-objects of the original data chunks divided according to the chunk size, and the corresponding redundant data sub-objects of the redundant data chunks. Data chunks with the same user identification information and stored in the same order constitute a striped data redundancy group. Each striped data redundancy group includes the number of original data chunks (n) and the number of redundant data chunks (m). Generally speaking, parameters such as n, m, and chunk size are related to the efficiency of redundancy coding. The larger the value of these parameters, the longer the calculation time and the lower the coding efficiency.

[0065] Common (n, m) combinations for erasure coding include (4, 2), (5, 3), (6, 2), (8, 3), (10, 2), and (12, 4), which are typically set based on the number of data fragments allowed to be lost and the storage space utilization of the distributed storage system. Here, m represents the number of redundant data fragments, which is also the maximum number of data fragments allowed to be lost. That is, in a striped data redundancy group, i.e., (n+m) data fragments, the loss of any data fragment not exceeding m fragments can be recovered using any of the other n data fragments. The storage space utilization of the distributed storage system usually refers to the proportion of the n original data fragments in a striped data redundancy group, i.e., n / (n+m). If n=4 and m=2, then the storage space utilization is 4 / (4+2), approximately 66%.

[0066] The principle of erasure coding can be briefly described using n=4 and m=2 as an example:

[0067] User data is divided into several original data fragments based on chunk size. A striped data redundancy group includes 4 original data fragments, let's say a, b, c, d. To obtain the corresponding 2 redundant data fragments, let's say x, y, we can construct the following system of equations:

[0068] x=α1×a+α2×b+α3×c+α4×d

[0069] y=β1×a+β2×b+β3×c+β4×d

[0070] Two unknowns can be solved, meaning that if any two of the six data slices a, b, c, d, x, and y are lost, they can be recovered.

[0071] Among them, α1, α2, α3, α4 and β1, β2, β3, β4 are preset coefficient matrices for calculating redundant data slices x and y of size chunk size. Commonly used coefficient matrices include Cauchy matrix and Vandermonde matrix.

[0072] Typically, different data fragments within a striped data redundancy group are stored in different sub-objects on different storage nodes in a distributed storage system. Data fragments with the same priority in different striped data redundancy groups are stored in the same sub-object. If the number of storage nodes is less than (n+m), some data fragments may be stored in different sub-objects on the same storage node. However, it should be ensured that each sub-object is allocated a different hard drive (usually each storage node includes multiple hard drives, and the total number of hard drives on each storage node in a distributed storage system should be greater than (n+m)) to minimize the risk of simultaneous occurrence of abnormal data fragments.

[0073] For ease of understanding, an example is provided: a data storage organization form for user data after erasure coding redundancy using a (4,2) combined erasure coding strategy is as follows. Figure 1 As shown, user data is divided into several original data fragments of the same size. Every four original data fragments are encoded using erasure coding to obtain two corresponding redundant data fragments. These four original data fragments and the corresponding two redundant data fragments constitute a striped data redundancy group. The original data fragments are sequentially written to the corresponding sub-objects 0, 1, 2, 3, and the redundant data fragments are sequentially written to the corresponding sub-objects 4, 5. Regularly, data fragments of the same order belonging to different striped data redundancy groups are written to the same sub-object. For example, when user data is divided into several original data fragments, original data fragments 0, 4, 8... belonging to different striped data redundancy groups are sequentially written to sub-object 0; original data fragments 1, 5, 9... are sequentially written to sub-object 1; original data fragments 2, 6... are sequentially written to sub-object 2; original data fragments 3, 7... are sequentially written to sub-object 2; similarly, the two redundant data fragments belonging to different striped data redundancy groups are sequentially written to sub-objects 4 and 5.

[0074] In this distributed storage system, the storage nodes can be various computer storage devices, including but not limited to storage servers and storage server clusters. The computer storage devices mentioned here are merely examples; other existing or future devices and / or resource sharing platforms applicable to this application should also be included within the scope of protection of this application, and are hereby incorporated by reference.

[0075] To further illustrate the technical means adopted and the effects achieved in this application, the technical solution of this application will be clearly and completely described below in conjunction with the accompanying drawings and preferred embodiments.

[0076] Figure 2 This diagram illustrates a method flowchart for asynchronous redundancy coding of data according to one aspect of this application, applied to a distributed storage system, wherein one embodiment of the method includes:

[0077] S101 Obtain a first data write request, wherein the data write request includes identification information, write offset, data size, and first data to be written;

[0078] S102 Based on the number of original data fragments and redundant data fragments, the number of replicas, and the identification information of the striped data redundancy group determined by the distributed storage system, S102 determines the sub-object of the corresponding storage node, wherein the sub-object includes an original data sub-object and a corresponding original data sub-object replica and a redundant data sub-object, which are used to store the original data fragments, original data fragment replicas, and redundant data fragments, respectively.

[0079] S103 determines the range of the striped data redundancy group covered by the first data and the corresponding original data sub-object and its copy and redundant data sub-object for each striped data redundancy group within the range, based on the size of the data fragment, the write offset and the data size, and the original data sub-object.

[0080] S104 Based on the size of the data fragment, the first data is divided into several raw data fragments, and each fragment is written to the corresponding raw data sub-object copy of the corresponding striped data redundancy group, and a metadata file is created for each raw data sub-object copy to record the write offset and data size of the corresponding raw data sub-object.

[0081] S105 uses an asynchronous redundancy coding service running on the storage node to perform asynchronous redundancy coding on the first data according to the metadata file of each original data sub-object copy, and deletes the original data sub-object copy and its metadata file that have completed asynchronous redundancy coding.

[0082] In this embodiment, in step S101, the distributed storage system or its client receives a first data write request sent by the user through the business system, wherein the data write request includes the identification information of the first data, the write offset, the data size, and the first data to be written.

[0083] In this system, the distributed storage system or its client can allocate corresponding sub-objects of storage nodes for data related to the user based on the data's identification information. The write offset refers to the offset of the first piece of data relative to the starting storage location of the data related to the user.

[0084] Continuing in this embodiment, in step S102, based on the number of original data fragments and the number of redundant data fragments, the number of replicas, and the identification information determined by the distributed storage system, a sub-object of the corresponding storage node is determined. The sub-object includes an original data sub-object and corresponding original data sub-object replicas and redundant data sub-objects, which are used to store the original data fragments, original data fragment replicas, and redundant data fragments, respectively.

[0085] Specifically, based on the erasure coding combination (n, m) determined during the deployment of the distributed storage system, the number of original data fragments n and the number of redundant data fragments m constituting each striped data redundancy group can be predetermined. Furthermore, based on the storage space usage plan determined during the deployment of the distributed storage system, the number of replicas to be copied for the written data can be predetermined, with a minimum of one replica (if the number of replicas is greater than one, one primary replica and several secondary replicas can be determined; if accessing the primary replica is abnormal, secondary replicas can be accessed according to a preset order to improve data availability). Based on these parameters and the identification information of the first data, the sub-objects of the storage node corresponding to the first data can be determined, including: an original data sub-object for storing the first data divided into original data fragments, a replica of the original data sub-object storing the corresponding original data fragment copy, and a replica storing the corresponding redundant data fragment.

[0086] The original data sub-objects and their copies, as well as redundant data sub-objects, should be distributed across different storage nodes, or at least across different hard drives on the same storage node, in order to minimize the risk of simultaneous occurrence of abnormal data fragments.

[0087] For example, in a distributed storage system employing a (4,2) combined erasure coding data redundancy strategy, the written user data is divided into original data fragments based on the preset chunk size and the four original data sub-objects 0, 1, 2, and 3 corresponding to the user's identifier information. These fragments are then allocated to these four original data sub-objects 0, 1, 2, and 3 for storage. The redundant data fragments calculated using erasure coding asynchronous redundancy are allocated to two redundant data sub-objects 4 and 5 for storage. If the predetermined number of replicas is 1, the user data divided into original data fragments is written to the corresponding sub-object replicas of the original data sub-objects 0, 1, 2, and 3 to achieve replica redundancy protection for newly written user data. After asynchronous erasure coding redundancy is completed, erasure coding redundancy protection is achieved for all newly written user data. A data storage organization form for asynchronous redundancy coding that includes two replicas can be as follows: Figure 3 As shown. If the distributed storage system includes 6 storage nodes, corresponding to the numbers 0, 1, 2, 3, 4, and 5, and if the hash calculation of the identifier information in the first data write request yields a result of 14, and the modulo operation of 14 with the number of storage nodes 6 yields a result of 2, then the distribution of the 4 original data sub-objects 0, 1, 2, and 3 and their copies, and the 2 redundant data sub-objects 4 and 5 corresponding to the data to be stored based on the aforementioned identifier information can be determined as follows: Original data sub-object 0 and its copy are located on storage nodes 2 and 3 respectively; original data sub-object 1 and its copy are located on storage nodes 3 and 4 respectively; original data sub-object 2 and its copy are located on storage nodes 4 and 5 respectively; original data sub-object 3 and its copy are located on storage nodes 5 and 0 respectively; redundant data sub-object 4 is located on storage node 0; and redundant data sub-object 5 is located on storage node 1. If the predetermined number of replicas is 2, including 1 primary replica and 1 secondary replica, then the primary replicas of the 4 original data sub-objects 0, 1, 2, and 3 should be located on storage nodes 3, 4, 5, and 0, respectively, while the secondary replicas of the 4 original data sub-objects 0, 1, 2, and 3 should be located on storage nodes 4, 5, 0, and 1, respectively.

[0088] Continuing in this embodiment, in step S103, based on the size of the data shards determined by the distributed storage system, the write offset and the data size, and the original data sub-object, the range of the striped data redundancy group covered by the first data and the corresponding original data sub-object and its copy, and redundant data sub-object of each striped data redundancy group within the range are determined.

[0089] Specifically, based on the chunksize of the data shards predetermined during the deployment phase of the distributed storage system, the offset of the first data to be written, the data size, and the information of each sub-object determined according to the above-mentioned identification information, the range of the striped data redundancy group covered by the first data writing and the corresponding original data sub-object and its copy, and redundant data sub-object of each striped data redundancy group within the range can be determined.

[0090] For example, if the chunk size of a distributed storage system using an erasure coding data redundancy strategy with a combination of (4,2) is predetermined to be 64KB, then in a striped data redundancy group, the original data length of the four original data chunks is 256KB. Adding the data length of the two redundant data chunks, the total data length in a striped data redundancy group is 384KB. Assuming the write offset of the first data to be written is 0KB (typically when user data related to the same identification information is written for the first time), and the data length is 352KB, the range of the striped data redundancy group covered by the first data write can be calculated based on the offset and data size. For example, the start stripe can be determined by rounding down the offset / stripe data size, which is 0 here. The end stripe can be determined by rounding down the (offset + data size) / stripe data size, which is 1 here. Therefore, the first 352KB of data to be written will overwrite the two striped data redundancy groups, stripe0 and stripe1.

[0091] Each striped data redundancy group includes four raw data fragments, with a maximum of 256KB of data. Based on the available storage area blocks (each block stores one data fragment) of the raw data sub-objects corresponding to the striped data redundancy groups stripe0 and stripe1, it can be further determined which storage area blocks corresponding to the sub-objects within the striped data redundancy groups stripe0 and stripe1 should be covered. Specifically, the first data is the first write of the aforementioned identification information-related data (if not the first write, the offset in the write request is usually not 0KB). It can be determined that the write offset 0 for stripe0 is 0KB, and the storage areas blocks 00 to 03 corresponding to the raw data sub-objects 0, 1, 2, and 3 of stripe0 will be covered. The write offset 1 for stripe1 is 256KB, and the storage areas blocks 10 to 11 corresponding to the raw data sub-objects 0 and 1 of stripe1 will be covered.

[0092] Continuing in this embodiment, in step S104, based on the size of the data fragment, the first data is divided into several original data fragments, and each fragment is written to a corresponding original data sub-object copy of the corresponding striped data redundancy group, and a metadata file is created for each original data sub-object's metadata file copy, recording the write offset and data size of the corresponding original data sub-object, so as to complete the writing of the first data.

[0093] Specifically, based on the chunk size of the data shards predetermined during the deployment of the distributed storage system, the first data in the aforementioned write data request is divided into several raw data shards, and each is written to a corresponding raw data sub-object copy of the corresponding striped data redundancy group. Additionally, a metadata file IOMETFile is created for each raw data sub-object copy. In the metadata file IOMETFile, the write offset and data size of the corresponding raw data sub-object are recorded.

[0094] Continuing the example above, the 352KB first data can be divided into six 64KB raw data fragments. The last raw data fragment includes 32KB of valid data and 32KB of data 0 (when dividing the raw data fragments, if the user data is not an integer multiple of the chunk size, zeros can be padded to the end of the data). The first four raw data fragments can be written to the storage areas block00 to block03 of the raw data sub-objects 0, 1, 2, and 3. The last two raw data fragments can be written to the storage areas block10 and block11 of the raw data sub-objects 0, 1, 2, and 3. A metadata file IOMETaFile is created for each of the raw data sub-objects 0, 1, 2, and 3, recording the write offset and the actual size of the data written. Specifically, firstly, the data write requests for storage areas blocks00 to 03 of the original data sub-objects 0, 1, 2, and 3 are sent to the storage nodes where the original data sub-objects 0, 1, 2, and 3 reside. Each storage node receives a data write request containing the corresponding sub-object's write offset, data size (all write offsets are 0KB, and all data sizes are 64KB), and original data fragments. Upon receiving the data write request, each storage node writes the corresponding original data fragments to the corresponding sub-object replica's block according to the data write request and creates a metadata file IOMEtaFile, which records the corresponding original data sub-object's write offset of 0KB and the actual written data size of 64KB. Then, the data write requests for storage areas blocks10 to 11 of the original data sub-objects 0 and 1 are sent to the storage nodes where the original data sub-objects 0 and 1 reside. Each data write request includes the corresponding write offset, data size (all write offsets are 64KB, the data size corresponding to block10 is 64KB, and the data size corresponding to block10 is 32KB), and original data fragments. The IOMETFile file of each of the original data sub-objects 0 and 1 is updated with a new record. For example, the IOMETFile file of the original data sub-object 0 is updated with a record of the write offset of 64KB and the actual data size written of 64KB. The IOMETFile file of the original data sub-object 1 is updated with a record of the write offset of 64KB and the actual data size written of 32KB (zero padding is not required, therefore, the actual data size written to block 11 is 32KB). Since the writing of the first data does not involve blocks 12 to 13 of the original data sub-objects 2 and 3, the records in the IOMETFile files of the original data sub-objects 2 and 3 remain unchanged.

[0095] Continuing in this embodiment, in step S105, based on the asynchronous redundancy coding service running on the storage node, the first data is asynchronously redundant coded according to the metadata file of each original data sub-object copy, and the original data sub-object copy and its metadata file that have completed asynchronous redundancy coding are deleted.

[0096] In this system, the distributed storage system or its clients, through the asynchronous redundancy coding service running in the background of each storage node, monitor the metadata file IOMETAFile of each original data sub-object replica related to the aforementioned identification information. If there are multiple replicas, the IOMETAFile of the primary replica of the original data sub-object is monitored first. If this fails, the IOMETAFile of the secondary replica is then monitored. Based on the records obtained from parsing the IOMETAFile, asynchronous redundancy coding is performed on the relevant write data. After the asynchronous redundancy coding of the relevant write data is completed, the relevant write data is already under erasure coding redundancy protection. There is no need to retain the relevant first data replicas. The original data sub-object replicas and their metadata files IOMETAFiles related to the first data on the relevant storage nodes can be deleted. This avoids the asynchronous redundancy coding service running in the background of the relevant storage nodes from repeatedly performing asynchronous redundancy coding operations, and it also releases storage resources, improving storage resource utilization.

[0097] Optionally, in step S105, the asynchronous redundancy encoding of the first data based on the metadata file of each original data sub-object copy includes:

[0098] Read the metadata file of each original data sub-object copy to determine the striped data redundancy group corresponding to the first data overlay for each original data sub-object;

[0099] For each striped data redundancy group covered by the first data, the corresponding original data fragment is obtained from the original data sub-object copy, and other corresponding original data fragments are obtained from the original data sub-object, and merged into the original data fragment of the striped data redundancy group.

[0100] Asynchronous redundancy coding is performed on the original data fragments of the merged striped data redundancy group to obtain new redundant data fragments of the striped data redundancy group. The original data fragments in the corresponding original data sub-object copy and the generated new redundant data fragments are copied to the corresponding original data sub-object and the corresponding redundant data sub-object.

[0101] Traverse each striped data redundancy group covered by the first data to complete the asynchronous redundancy coding of the first data.

[0102] Specifically, the distributed storage system or its client, through an asynchronous redundancy coding service running in the background on the storage node associated with the identification information corresponding to the first data, monitors the metadata files (IOMetaFile) of one or more original data sub-object replicas related to the identification information. By parsing the records in each metadata file (IOMetaFile), it obtains the updated but not yet asynchronously redundancy coded original data fragments corresponding to the original data sub-objects associated with the first data, and the range of the striped data redundancy groups they cover. Then, for each striped data redundancy group covered by the first data, it retrieves the corresponding original data fragments from the corresponding original data sub-object replicas. If older original data fragments still exist in the striped data redundancy group, the distributed storage system or its client also retrieves the corresponding older original data fragments from the relevant storage nodes and merges them into all the original data fragments of the striped data redundancy group. Then, erasure coding asynchronous redundancy encoding is performed on all merged original data fragments to obtain new redundant data fragments for the striped data redundancy group. Write requests for the original data fragments and redundant data fragments are sent to the relevant storage nodes storing the original data sub-objects and redundant data sub-objects, respectively. Upon receiving the write request, the relevant storage nodes execute the write operation, writing the original data fragments in the original data sub-object copy and the obtained new redundant data fragments to the corresponding sub-objects, and returning the write result and synchronizing it to the relevant storage nodes (if the relevant storage nodes receive a write result indicating a failed write operation, they will perform asynchronous redundancy encoding again when the background asynchronous redundancy encoding service subsequently detects the metadata file IOMETAFile). This process iterates through each striped data redundancy group covered by the first data until all involved striped data redundancy groups have been processed, thus completing the erasure coding redundancy protection for the first data and the original data.

[0103] Continuing the example above, after the first data is written to the corresponding sub-object replica, the distributed storage system or its client monitors the metadata file IOMETAFile of each original data sub-object replica through the asynchronous encoding service running in the background of the storage node where the original data sub-object replicas 0, 1, 2, and 3 corresponding to the relevant identification information are located. After parsing the file, based on the records in it, it can be found that the original data in the corresponding striped data redundancy group in each original data sub-object replica (the data stored in blocks 00 to 03 of stripe0 and blocks 10 to 11 of stripe1) has not yet completed asynchronous redundancy encoding. Therefore, erasure coding asynchronous redundancy encoding calculation is performed for each striped data redundancy group stripe0 and stripe1 to obtain the corresponding new redundant data. Specifically, for stripe0, the original data is obtained from blocks 00 to 03 of the original data sub-objects 0, 1, 2, and 3 replicas. After erasure coding calculation, new redundant data fragments are obtained. Write requests are sent to the relevant storage nodes storing the original data sub-objects 0, 1, 2, and 3, as well as the storage nodes storing the redundant data sub-objects 4 and 5. After receiving the write requests, the relevant storage nodes perform the write operation, writing the relevant original data from the original data sub-objects 0, 1, 2, and 3 replicas and the obtained new redundant data fragments to the corresponding blocks of the original data sub-objects 0, 1, 2, and 3 and the redundant data sub-objects (corresponding to blocks 00 to 05 of stripe0), and returning the write results and synchronizing them with the relevant storage nodes.For stripe1, the original data (blocks 10-11) is retrieved from the replicas of the original data sub-objects 0, 1, 2, and 3 (block 10 contains 64KB of data, and block 11 contains 32KB of data). If old original data is stored in blocks 11, 12, and 13 of the original data sub-objects 1, 2, and 3, then the old original data unrelated to the first data (block 11 contains 32KB of data, and blocks 12 and 13 contain 64KB of data) is retrieved from blocks 11, 12, and 13 of the storage nodes of the original data sub-objects 1, 2, and 3. After padding with zeros, the merged data yields the complete original data fragment of stripe1. Erasure coding redundancy calculations are then performed to obtain a new redundant data fragment. Write requests are sent to the relevant storage nodes storing the original data sub-objects 0 and 1, and the redundant data sub-objects 4 and 5. Upon receiving the write request, the relevant storage nodes execute the write operation, writing the relevant original data from the copies of the original data sub-objects 0 and 1, as well as the obtained new redundant data fragment, to the corresponding blocks of the redundant data sub-objects (corresponding to blocks 10, 11, 14-15 of stripe1), and returning the write results synchronously to the relevant storage nodes. The data organization form after the first data is written and asynchronously redundantly encoded, and is in the erasure coding redundancy protection state, is as follows. Figure 4 As shown.

[0104] Once the asynchronous redundancy coding of all striped data redundancy groups is successfully executed, the original data fragments related to the first data and the original original data fragments written to the relevant storage nodes of the distributed storage system are already under erasure coding redundancy protection. There is no need to retain copies of the original data sub-objects 0, 1, 2, and 3 related to the first data. The copies of the original data sub-objects and their metadata files IOMET Files on the relevant storage nodes can be deleted.

[0105] Optionally, if, during the process of retrieving other corresponding original data fragments from the original data sub-object, there are original data fragments that cannot be successfully retrieved, the method further includes:

[0106] Based on the data fragments successfully obtained from the original data sub-objects and redundant data sub-objects corresponding to the striped data redundancy group, data recovery is performed on the original data fragments that could not be successfully obtained.

[0107] Obtain the original data fragments after data recovery.

[0108] If the striped data redundancy group still contains old original data fragments, the distributed storage system or its client will retrieve the corresponding old original data fragments from the relevant storage nodes and merge them into all the original data fragments of the striped data redundancy group. If the corresponding old original data fragments in the original data sub-objects of the relevant storage nodes cannot be retrieved successfully, and the number of abnormal old original data fragments does not exceed m, then n fragments can be retrieved from the corresponding normal old data fragments (old original data fragments and / or old redundant data fragments) of other original data sub-objects and / or redundant data sub-objects in the striped data redundancy group. These old original data fragments can then be recovered through erasure coding. After the recovered old original data fragments are retrieved, they are merged with the newly written original data fragments of the striped data redundancy group.

[0109] In existing distributed storage systems, asynchronous redundant updates of redundant data shards can take minutes, which is quite time-consuming. Since data updates in distributed storage systems are frequent, performing asynchronous redundant updates on every update would significantly increase the system's read / write overhead and reduce service performance. Therefore, distributed storage systems do not need to perform asynchronous redundant encoding on the updated original data shards with every write operation. Instead, they can use a polling method or a pre-set periodic interval, performing asynchronous redundant encoding only on the corresponding written original data shards when the polling period expires. For example, assuming user data with the same identifier information is updated 10 times within a polling or timed period (the data storage areas written each time may not overlap or may partially overlap), the distributed storage system does not need to perform asynchronous redundant encoding 10 times. It only needs to perform erasure coding asynchronous redundant encoding on the cumulative written user data after the 10th update when the polling period expires.

[0110] If a second data write request with the same identification information is received before the asynchronous redundancy encoding of the previously written first data has been completed, optionally, if the second data write request including the identification information is obtained before the asynchronous redundancy encoding of the first data is performed, the method further includes:

[0111] Based on the size of the data shards determined by the distributed storage system, the write offset and data size in the second data write request, and the sub-objects of the corresponding storage nodes, the range of the striped data redundancy group covered by the second data to be written in the second data write request and the corresponding original data sub-objects and their copies and redundant data sub-objects of each striped data redundancy group within the range are determined.

[0112] Based on the size of the data shards, the second data is divided into several original data shards, and each shard is written to the corresponding original data sub-object copy of the corresponding striped data redundancy group.

[0113] If there is no metadata file for the original data sub-object copy corresponding to the striped data redundancy group, then create the metadata file for the original data sub-object copy and record the write offset and data size of the corresponding original data sub-object. If there is a metadata file for the original data sub-object copy corresponding to the striped data redundancy group, then update the write offset and data size of the corresponding original data sub-object recorded in the metadata file. If the write offset and data size of this time overlap with the write offset and data size recorded in the metadata file in a range, then merge them and record the write offset and data size of the corresponding original data sub-object after merging.

[0114] In step S105, the asynchronous redundancy encoding of the first data based on the metadata file of each original data sub-object copy includes:

[0115] Asynchronous redundant encoding is performed on the first data and the second data based on the metadata file of each original data sub-object copy.

[0116] Specifically, based on the chunk size of the data block predetermined during the deployment phase of the distributed storage system, the write offset of the second data to be written in the received second data write request, the data size, and the information of each sub-object determined according to the above-mentioned identification information, the range of the striped data redundancy group covered by the second data write and the corresponding original data sub-object and its copy, and redundant data sub-object of each striped data redundancy group within the range can be determined.

[0117] Continuing with the example above, assuming the write offset of the second data to be written is 416KB and the data size is 256KB, the range of the striped data redundancy groups covered by the second data can be calculated based on the offset and data size. According to the rounding operation result, it can be determined that the start striped data redundancy group is 1 and the end striped data redundancy group is 2. Therefore, the 256KB of second data to be written will cover the two striped data redundancy groups stripe1 and stripe2.

[0118] Specifically, based on the chunk size of the data chunks, the second data is divided into several original data chunks, and each chunk is written to the corresponding original data sub-object and its copy in the corresponding striped data redundancy group.

[0119] Continuing the example above, the write offset for the second data is 416KB. Each striped data redundancy group includes four original data fragments, each with a chunk size of 64KB. Based on the available storage area blocks (each block stores one data fragment) of the original data sub-objects corresponding to the striped data redundancy groups stripe1 and stripe2, it can be further determined which storage area blocks corresponding to the sub-objects within the striped data redundancy groups stripe1 and stripe2 should be overwritten. Specifically, it can be determined that the write offset1 for stripe1 is 160KB, and the storage areas blocks 12 to 13 corresponding to the original data sub-objects 0, 1, 2, and 3 of stripe1 will be overwritten, allowing 96KB of data to be written. The remaining 160KB of the second data will then be written to stripe2. The write offset2 for stripe2 is 512KB, and it can be determined that the storage areas blocks 20 to 22 corresponding to the original data sub-objects 0, 1, and 2 of stripe2 will be overwritten.

[0120] The 256KB second data is divided into original data fragments and written to the storage areas of the original data sub-objects 0, 1, 2, and 3 (blocks 12-13 and 20-22). Blocks 12 and 22 store 32KB of data, while the other blocks store 64KB of data, in order to achieve redundant protection for the newly written second data.

[0121] If there is no metadata file for the original data sub-object copy corresponding to the striped data redundancy group, then a metadata file for the original data sub-object copy is created and the write offset and data size of the corresponding original data sub-object are recorded. If there is a metadata file for the original data sub-object copy corresponding to the striped data redundancy group, then the write offset and data size of the corresponding original data sub-object recorded in the metadata file are updated. If the current write offset and data size are adjacent to or overlap with the write offset and data size recorded in the metadata file, then they are merged, and the write offset and data size of the corresponding original data sub-object after merging are recorded.

[0122] Continuing with the example above, after the first data is written to the original data sub-objects 0, 1, 2, and 3, since the asynchronous erasure coding redundancy encoding has not yet been completed, the metadata files IOMETAFile of the original data sub-objects 0, 1, 2, and 3 already exist and do not need to be created. Instead, relevant records are added to the respective metadata files IOMETAFile of the original data sub-objects 0, 1, 2, and 3.

[0123] The first set of raw data fragments is written to the original data sub-object replicas, involving the original data sub-objects 0, 1, 2, and 3 replicas on the relevant storage nodes. The metadata file IOMetaFile corresponding to original data sub-object 0 (i.e., the replica of original data sub-object 0) contains two records:

[0124] 1. (Corresponding to striped data redundancy group stripe0) The write offset of the original data sub-object 0 is 0KB, and the data size is 64KB;

[0125] 2. (For the striped data redundancy group stripe1) The write offset of the original data sub-object 0 is 64KB, and the data size is 64KB.

[0126] Because these two records are adjacent, they can be merged and simplified into a single record, reducing storage space usage.

[0127] 1. The write offset of the original data sub-object 0 is 0KB, and the data size is 128KB (it can also be merged if there is overlap).

[0128] The metadata file IOMetaFile corresponding to the original data sub-object 1 (i.e., the copy of the original data sub-object 1) contains two records:

[0129] 1. (Corresponding to striped data redundancy group stripe0) The write offset of the original data sub-object 1 is 0KB, and the data size is 64KB.

[0130] 2. (For the striped data redundancy group stripe1) The write offset of the original data sub-object 1 is 64KB, and the data size is 32KB.

[0131] These two records are adjacent to each other and can be merged into one record:

[0132] 1. The write offset of the original data sub-object 1 is 0KB, and the data size is 96KB.

[0133] There is a record in the metadata file IOMETFile corresponding to the original data sub-object 2 (that is, the copy of the original data sub-object 2):

[0134] 1. (Corresponding to striped data redundancy group stripe0) The write offset of the original data sub-object 2 is 0KB, and the data size is 64KB.

[0135] There is a record in the metadata file IOMETFile corresponding to the original data sub-object 3 (that is, the copy of the original data sub-object 3):

[0136] 1. (Corresponding to striped data redundancy group stripe0) The write offset of the original data sub-object 3 is 0KB, and the data size is 64KB.

[0137] The relevant raw data fragments of the second data are written to the original data sub-object replicas, involving the original data sub-objects 0, 1, 2, and 3 replicas of the relevant storage nodes. Relevant records are added to the metadata files IOMETAFile of the original data sub-objects 0, 1, 2, and 3 replicas.

[0138] Add a new record to the IOMETFile metadata file corresponding to the original data sub-object 0 (i.e., the copy of the original data sub-object 0):

[0139] 2. (Corresponding to striped data redundancy group stripe2) The write offset of the original data sub-object 0 is 128KB, and the data size is 64KB.

[0140] Adjacent to existing records, they can be merged and simplified into a single record:

[0141] 1. The write offset of the original data sub-object 0 is 0KB, and the data size is 192KB.

[0142] Add a new record to the metadata file IOMETFile corresponding to the original data sub-object 1 (i.e., the copy of the original data sub-object 1):

[0143] 2. (Corresponding to striped data redundancy group stripe2) The write offset of the original data sub-object 1 is 128KB, and the data size is 64KB.

[0144] If the content of the original record does not meet the merging criteria, then the metadata file IOMETAFile will contain two records:

[0145] 1. The write offset of the original data sub-object 1 is 0KB, and the data size is 96KB;

[0146] 2. The write offset of the original data sub-object 1 is 128KB, and the data size is 64KB.

[0147] Add two records to the metadata file IOMETaFile corresponding to the original data sub-object 2 (i.e., the copy of the original data sub-object 2):

[0148] 2. (Corresponding to striped data redundancy group stripe1) The write offset of the original data sub-object 2 is 96KB, and the data size is 32KB;

[0149] 3. (Corresponding to striped data redundancy group stripe2) The write offset of the original data sub-object 2 is 128KB, and the data size is 32KB.

[0150] These two records are adjacent and can be merged into one record. Therefore, the metadata file IOMetaFile contains two records:

[0151] 1. The write offset of the original data sub-object 2 is 0KB, and the data size is 64KB;

[0152] 2. The write offset of the original data sub-object 2 is 96KB, and the data size is 64KB.

[0153] Add a record to the metadata file IOMETaFile corresponding to the original data sub-object 3 (that is, the copy of the original data sub-object 3):

[0154] 2. (Corresponding to the striped data redundancy group stripe1) The write offset of the original data sub-object 3 is 64KB, and the data size is 64KB.

[0155] Adjacent to existing records, they can be merged and simplified into a single record:

[0156] 1. The write offset of the original data sub-object 3 is 0KB, and the data size is 128KB.

[0157] If the asynchronous redundancy encoding service running in the background of the relevant storage node is polled or expires, then asynchronous redundancy encoding is performed on the written first and second data. In step S105, the asynchronous redundancy encoding of the first data based on the metadata file of each original data sub-object copy includes:

[0158] Asynchronous redundant encoding is performed on the first data and the second data based on the metadata file of each original data sub-object copy.

[0159] In this system, the distributed storage system or its clients, through the asynchronous redundancy coding service running in the background of each storage node, poll or periodically monitor the metadata file IOMETFile of each original data sub-object copy related to the first and second data associated with the aforementioned identification information. Based on the records obtained by parsing the metadata file IOMETFile, asynchronous redundancy coding is performed on the relevant write data. After the asynchronous redundancy coding of the relevant write data is completed, the data related to the aforementioned identification information is already under erasure coding redundancy protection. There is no need to retain the relevant first and second data copies. The original data sub-object copies corresponding to the first and second data and their metadata files IOMETFiles on the relevant storage nodes can be deleted. This avoids the asynchronous redundancy coding service running in the background of the relevant storage nodes from repeatedly performing asynchronous redundancy coding operations, and it can release storage resources and improve storage resource utilization.

[0160] Continuing the example above, after the first and second data are written to their corresponding sub-object replicas, the distributed storage system or its client monitors the metadata file IOMETAFile of each original data sub-object replica through the asynchronous encoding service running in the background of the storage node where the original data sub-object replicas 0, 1, 2, and 3 with the relevant identification information are located. After parsing the file, based on the records in it, it can be seen that the original data fragments (the data stored in blocks 00 to 03 of stripe0, blocks 10 to 11 of stripe1, blocks 12 to 13 of stripe1, and blocks 20 to 22 of stripe2) in the striped data redundancy group corresponding to each original data sub-object replica have not yet completed asynchronous redundancy encoding. Therefore, for each striped data redundancy group stripe0, stripe1, and stripe2, erasure coding asynchronous redundancy encoding calculation is performed to obtain the corresponding new redundant data fragments. Specifically, for stripe0, the original data is obtained from blocks 00 to 03 of the original data sub-objects 0, 1, 2, and 3 replicas. After erasure coding calculation, new redundant data fragments are obtained. Write requests are sent to the relevant storage nodes storing the original data sub-objects 0, 1, 2, and 3, as well as the storage nodes storing the redundant data sub-objects 4 and 5. After receiving the write requests, the relevant storage nodes perform the write operation, writing the relevant original data from the original data sub-objects 0, 1, 2, and 3 replicas and the obtained new redundant data fragments to the corresponding blocks of the original data sub-objects 0, 1, 2, and 3 and the redundant data sub-objects (corresponding to blocks 00 to 05 of stripe0), and returning the write results and synchronizing them with the relevant storage nodes.For stripe1, the original data will be retrieved from blocks 10 to 13 of the original data sub-objects 0, 1, 2, and 3 (where blocks 10 and 13 contain 64KB of data, and blocks 11 and 12 contain 32KB of data). If old original data is stored in blocks 11 and 12 of the original data sub-objects 1 and 2, then old original data unrelated to the first and second data will also be retrieved from blocks 11 and 12 of the original data sub-objects 1 and 2 of the relevant storage nodes (where blocks 11 and 12 each contain 32KB of data, and if no valid data is available). According to the data (filling in zeros), after merging, the complete original data fragment of stripe1 is obtained. After performing erasure coding redundancy calculation, a new redundant data fragment is obtained. Write requests are sent to the relevant storage nodes storing the original data sub-objects 0, 1, 2, 3 and the redundant data sub-objects 4, 5 respectively. After receiving the write request, the relevant storage nodes perform the write operation, writing the relevant original data in the copies of the original data sub-objects 0, 1, 2, 3 and the obtained new redundant data fragments to the corresponding blocks of the corresponding redundant data sub-objects (the entire area of ​​blocks 10 and 13 of stripe1, part of blocks 11 and 12, and blocks 14 to 15), and returning the write result and synchronizing it to the relevant storage nodes. For stripe2, the original data (blocks 20-22) is retrieved from the replicas of the original data sub-objects 0, 1, and 2 (blocks 20 and 21 contain 64KB of data, and block 22 contains 32KB of data). If old original data is stored in blocks 22 and 23 of the original data sub-objects 2 and 3, then old original data unrelated to the first and second data (block 22 contains 32KB of data, and block 23 contains 64KB of data, and if no valid data is found) is also retrieved from blocks 22 and 23 of the original data sub-objects 2 and 3 of the relevant storage nodes. After padding with zeros, the data is merged to obtain the complete original data fragment of strip2. After erasure coding calculation, a new redundant data fragment is obtained. Write requests are sent to the relevant storage nodes storing the original data sub-objects 0, 1, 2 and the redundant data sub-objects 4, 5 respectively. After receiving the write request, the relevant storage nodes perform the write operation, writing the relevant original data in the replicas of the original data sub-objects 0, 1, 2 and the obtained new redundant data fragment to the corresponding blocks of the original data sub-objects 0, 1, 2 and the redundant data sub-objects respectively (the entire area of ​​block 20 and block 21 of strip2, part of the area of ​​block 22, and blocks 24 to 25), and returning the write result and synchronizing it to the relevant storage nodes.The data organization form after the first and second data are written and asynchronously redundantly encoded, and then in the erasure code redundancy protection state is as follows. Figure 5 As shown.

[0161] If, before the first data undergoes asynchronous redundancy encoding, the distributed storage system or its client receives a data read request, and if a portion of the data to be read is under erasure coding redundancy protection, but another portion belongs to the area covered by the first data, then, in order to accurately read the latest data, optionally, if a third data read request including the identification information is received before the first data undergoes asynchronous redundancy encoding, the method further includes:

[0162] Based on the size of the data shards determined by the distributed storage system, the read offset and data size in the third data read request, and the sub-objects of the corresponding storage nodes, the range of the striped data redundancy group covered by the third data to be read in the third data read request and the corresponding original data sub-objects and their copies and redundant data sub-objects of each striped data redundancy group within the range are determined.

[0163] The read offset and data size in the third data read request are compared one by one with the write offset and data size of the corresponding original data sub-object recorded in the metadata file of the original data sub-object copy of each striped data redundancy group within the range. If they are included, the corresponding original data fragment is read from the original data sub-object copy. If they are not included, the corresponding original data fragment is determined from the original data sub-object and / or redundant data sub-object of the striped data redundancy group to complete the reading of the third data.

[0164] Based on the identification information in the acquired read request, it can be determined which storage nodes' original data sub-objects and their copies, as well as redundant data sub-objects, are covered by the third data to be read. Further, based on the read offset and data size in the acquired read request, the range of the striped data redundancy group covered by the third data to be read, and the corresponding original data sub-objects and their copies, and redundant data sub-objects for each striped data redundancy group within that range, can be determined. Then, the read offset and data size in the acquired read request are compared one by one with the write offset and data size of the corresponding original data sub-object recorded in the metadata file IOMetaFile of the original data sub-object copy for each striped data redundancy group within that range. If they are included, it indicates that the relevant original data fragment has not yet undergone asynchronous redundancy coding, and the corresponding original data fragment is read from the original data sub-object copy. If they are not included, the corresponding original data fragment is determined from the corresponding original data sub-objects and / or redundant data sub-objects of that striped data redundancy group to complete the reading of the third data.

[0165] Continuing the example above, before asynchronous redundancy coding is performed after the first data is written, it is in a replica redundancy protection state. The identification information in the read request of the third data is the same as that of the first data (i.e., it involves the original data sub-objects 0, 1, 2, 3 and their replicas, as well as the redundant data sub-objects 4, 5). Assuming the read offset is 0KB and the data size is 512KB, the striped data redundancy group covered by the third data to be read can be determined according to the size of the data fragments determined by the distributed storage system. This includes the original data fragments in blocks 00 to 03 of striped data redundancy group stripe0 and blocks 10 to 13 of stripe1. It can be seen that the covered data includes not only the first data, but also the part stored in block 12 of striped data redundancy group stripe1 and the old original data stored in block 13 (the old original data has usually completed asynchronous redundancy coding and is in an erasure coding redundancy protection state). After comparing each record in the metadata file of the original data sub-objects 0, 1, 2, and 3, if all the original data stored in blocks 00-03 of stripe0 and blocks 10-11 of stripe1, as well as the first 32KB of block 12, are covered, then the corresponding original data is read from the original data sub-object copy. If the original data stored in the last 32KB of block 12 of stripe1 and the original data stored in block 13 are not covered, then the original data is read from the corresponding original data sub-object's metadata file in that stripe data redundancy group. If individual raw data fragments cannot be read successfully due to an anomaly, the corresponding raw data fragments can be determined by combining the raw data fragments that can be read normally in the corresponding raw data sub-objects 0, 1, 2, and 3 of the striped data redundancy group and / or the redundant data fragments in the redundant data sub-objects 4 and 5 (for example, if raw data reading in block13 is abnormal, it can be determined by any 4 of the relevant raw data sub-objects block10, block11, block12, and redundant data sub-objects block14 and block15), thereby completing the reading of 512KB of third data.

[0166] After the first data completes asynchronous redundancy coding and before new user data with the same identification information is written, all data related to that identification information is in an erasure coding redundancy protection state. Optionally, this method for asynchronous data redundancy coding further includes:

[0167] Obtain a fourth data read request including the identification information;

[0168] Based on the size of the data shards determined by the distributed storage system, the read offset and data size in the fourth data read request, and the sub-objects of the corresponding storage nodes, the range of the striped data redundancy group covered by the fourth data to be read in the fourth data read request and the corresponding original data sub-objects and redundant data sub-objects of each striped data redundancy group within the range are determined.

[0169] The original data fragments are determined one by one from the corresponding original data sub-objects and / or redundant data sub-objects of each striped data redundancy group within the range to complete the reading of the fourth data.

[0170] Based on the identification information in the acquired fourth data read request, it can be determined which storage nodes' original data sub-objects and their copies, as well as redundant data sub-objects, are covered by the fourth data to be read. Further, based on the read offset and data size in the acquired read request, the range of the striped data redundancy group covered by the fourth data to be read, and the corresponding original data sub-objects and their copies, as well as redundant data sub-objects, for each striped data redundancy group within the range, can be determined. For each striped data redundancy group within the range, the asynchronous redundancy coding service running in the background of the storage node corresponding to the original data sub-object copy of that striped data redundancy group does not detect the metadata file IOMEtaFile of the original data sub-object copy. This indicates that the data related to the aforementioned identification information is under erasure coding redundancy protection. The corresponding original data fragments can be determined one by one from the original data sub-objects and / or redundant data sub-objects of that striped data redundancy group to complete the reading of the fourth data.

[0171] Continuing the example above, after the newly written first and second data complete asynchronous redundancy coding, they are in an erasure coding redundancy protection state. Because the identification information in the read request for the fourth data is the same as that of the first and second data, involving original data sub-objects 0, 1, 2, and 3 and their copies, as well as redundant data sub-objects 4 and 5, assuming the read offset of the fourth data is 0KB and the data size is 640KB, based on the data fragment size determined by the distributed storage system, the striped data redundancy group covered by the fourth data to be read includes all the original data in stripe0 blocks 00–03, stripe1 blocks 10–13, and stripe2 blocks 20–21. Through the asynchronous redundancy coding service running in the background of the corresponding storage nodes of the original data sub-objects 0, 1, 2, and 3 copies, no metadata file (IOMetaFile) of the original data sub-object copies was detected, indicating that the fourth data to be read is in an erasure coding redundancy protection state. The data can be read one by one from striped data redundancy groups. The raw data fragments are read from blocks 00 to 03 of stripe 0, blocks 10 to 13 of stripe 1, and blocks 20 to 21 of stripe 2. If an individual raw data fragment fails to be read, the corresponding raw data fragment can be determined by combining the raw data fragments that can be read normally in the corresponding raw data sub-objects 0, 1, 2, and 3 of the striped data redundancy group and / or the redundant data fragments in the redundant data sub-objects 4 and 5 (for example, if the raw data reading in block 13 fails, it can be determined by any 4 of the relevant raw data sub-objects blocks 10, 11, and 12 and the redundant data sub-objects blocks 14 and 15), thereby completing the reading of the 640KB fourth data.

[0172] In the above embodiments and / or optional embodiments of this application, a replica of the original data sub-object is first used to redundantly protect the newly written user data. A metadata file IOMETaFile is used to record the corresponding write offset and data size of the newly written user data. The existence of the metadata file IOMETaFile of the original data sub-object replica can be used to determine whether there is original data in the distributed storage system that has not been erasure coding asynchronous redundancy encoding. If the metadata file IOMETaFile does not exist, it means that no newly written user data needs to be erasure coding asynchronous redundancy encoding. If it exists, it means that newly written user data needs to be erasure coding asynchronous redundancy encoding. Then, the metadata file IOMETaFile is parsed, and the coverage area of ​​the newly written user data that has not yet undergone asynchronous redundancy encoding is determined according to the records in it. Asynchronous redundancy encoding is then performed through the asynchronous redundancy encoding service running in the background of the storage node. Accordingly, the newly written user data is in a replica redundancy protection state before asynchronous redundancy encoding is completed, and in an erasure coding redundancy protection state after asynchronous redundancy encoding is completed, which can ensure its high availability.

[0173] Figure 6 The diagram illustrates an apparatus for asynchronous redundancy coding of data according to another aspect of this application, deployed in a distributed storage system, wherein, in one embodiment, the apparatus includes:

[0174] The first module 210 is used to obtain a first data write request, wherein the data write request includes identification information, write offset, data size and first data to be written;

[0175] The second module 220 is used to: determine the sub-object of the corresponding storage node based on the number of original data fragments and the number of redundant data fragments, the number of replicas, and the identification information of the striped data redundancy group determined by the distributed storage system. The sub-object includes an original data sub-object and a corresponding original data sub-object replica and a redundant data sub-object, which are used to store the original data fragments, the original data fragment replicas, and the redundant data fragments, respectively.

[0176] The third module 230 is used to: determine the range of the striped data redundancy group covered by the first data and the corresponding original data sub-object and its copy and redundant data sub-object for each striped data redundancy group within the range, based on the size of the data shard determined by the distributed storage system, the write offset and the data size, and the original data sub-object;

[0177] The fourth module 240 is used to: divide the first data into several original data fragments based on the size of the data fragments, write them one by one to the corresponding original data sub-object copies of the corresponding striped data redundancy group, and create a metadata file for each original data sub-object copy, recording the write offset and data size of the corresponding original data sub-object;

[0178] The fifth module 250 is used to: perform asynchronous redundancy coding on the first data based on the asynchronous redundancy coding service running on the storage node, according to the metadata file of each original data sub-object copy, and delete the original data sub-object copy and its metadata file that have completed asynchronous redundancy coding.

[0179] In this embodiment, the device is integrated into a distributed storage system 100. The distributed storage system 100 has the same hardware and software environment as the distributed storage systems in the foregoing method embodiments and / or optional embodiments.

[0180] In this embodiment, the distributed storage system 100 or its client receives a first data write request sent by the user through the business system via the first module 210 of the device. The data write request includes the identification information of the first data, the write offset, the data size, and the first data to be written.

[0181] Continuing in this embodiment, through the second module 220 of the device, based on the erasure coding combination (n, m) determined during the deployment of the distributed storage system 100, the number of original data fragments n and the number of redundant data fragments m constituting each striped data redundancy group can be predetermined. Furthermore, based on the storage space usage plan determined during the deployment of the distributed storage system 100, the number of replicas to be copied for the written data can be predetermined, with a minimum of one replica (if the number of replicas is greater than one, one primary replica and several secondary replicas can be determined; if accessing the primary replica is abnormal, secondary replicas can be accessed according to a preset order to improve data availability). Based on these parameters and the identification information of the first data, the sub-objects of the storage node corresponding to the first data can be determined, including: an original data sub-object for storing the first data divided into original data fragments, a replica of the original data sub-object storing the corresponding original data fragment copy, and a replica storing the corresponding redundant data fragment.

[0182] Continuing in this embodiment, through the third module 230 of the device, based on the chunk size of the data fragments predetermined by the distributed storage system 100 during the deployment phase, the offset of the first data to be written, the data size, and the information of each sub-object determined according to the above-mentioned identification information, the range of the striped data redundancy group covered by the first data writing and the corresponding original data sub-object and its copy, and redundant data sub-object of each striped data redundancy group within the range can be determined.

[0183] Continuing in this embodiment, the fourth module 240 of the device divides the first data in the above-mentioned write data request into several original data fragments according to the chunk size of the data fragments predetermined during the deployment of the distributed storage system 100, and writes them one by one to the corresponding original data sub-object copies of the striped data redundancy group, and creates a metadata file IOMETFile for each original data sub-object copy. In the metadata file IOMETFile, the write offset and data size of the corresponding original data sub-object are recorded.

[0184] Continuing in this embodiment, the fifth module 250 of the device coordinates the asynchronous redundancy coding service running in the background of each storage node. For the first data related to the aforementioned identification information, the metadata file IOMETFile of each corresponding original data sub-object replica is monitored. If there are multiple replicas, the metadata file IOMETFile of the primary replica of the original data sub-object is monitored first. If this fails, the metadata file IOMETFile of the secondary replica of the original data sub-object is then monitored. Based on the records obtained by parsing the metadata file IOMETFile, asynchronous redundancy coding is performed on the relevant write data. After the asynchronous redundancy coding of the relevant write data is completed, the relevant write data is already under erasure coding redundancy protection. There is no need to retain the relevant first data replicas. The original data sub-object replicas and their metadata files IOMETFiles related to the first data on the relevant storage nodes can be deleted. This avoids the asynchronous redundancy coding service running in the background of the relevant storage nodes from repeatedly performing asynchronous redundancy coding operations, and it can release storage resources and improve storage resource utilization.

[0185] If a second data write request with the same identification information is received before the asynchronous redundancy encoding of the previously written first data has been completed, optionally, if the first module 210 obtains the second data write request including the identification information before the asynchronous redundancy encoding of the first data is performed, the third module 230 is further configured to:

[0186] Based on the size of the data shards determined by the distributed storage system 100, the write offset and data size in the second data write request, and the sub-objects of the corresponding storage nodes, the range of the striped data redundancy group covered by the second data to be written in the second data write request and the corresponding original data sub-objects and their copies and redundant data sub-objects of each striped data redundancy group within the range are determined.

[0187] The fourth module 240 is further configured to: divide the second data into several original data fragments based on the size of the data fragments, and write them one by one to the corresponding original data sub-object replicas of the corresponding striped data redundancy group; if there is no metadata file for the corresponding original data sub-object replicas of the corresponding striped data redundancy group, then create a metadata file for the original data sub-object replicas and record the write offset and data size of the corresponding original data sub-objects; if there is a metadata file for the corresponding original data sub-object replicas of the corresponding striped data redundancy group, then update the write offset and data size of the corresponding original data sub-objects recorded in the metadata file; if the write offset and data size of the current data sub-objects are adjacent to or overlap with the write offset and data size recorded in the metadata file, then merge them and record the write offset and data size of the corresponding original data sub-objects after merging.

[0188] The fifth module 250 is used to: perform asynchronous redundancy coding on the first data and the second data based on the metadata file of each original data sub-object copy, and delete the original data sub-object copy and its metadata file that have completed asynchronous redundancy coding, based on the asynchronous redundancy coding service running on the storage node.

[0189] If, before the first data undergoes asynchronous redundancy encoding, the distributed storage system 100 or its client receives a data read request, and if a portion of the data to be read is under erasure coding redundancy protection, but another portion belongs to the area covered by the first data, then, in order to accurately read the latest data, optionally, if, before the first module receives a third data read request including the identification information, the third module 230 is further configured to:

[0190] Based on the size of the data shards determined by the distributed storage system, the read offset and data size in the third data read request, and the sub-objects of the corresponding storage nodes, the range of the striped data redundancy group covered by the third data to be read in the third data read request and the corresponding original data sub-objects and their copies and redundant data sub-objects of each striped data redundancy group within the range are determined.

[0191] The fourth module 240 is further configured to: compare the read offset and data size in the third data read request with the write offset and data size of the corresponding original data sub-object recorded in the metadata file of the original data sub-object copy of each striped data redundancy group within the range; if they are included, read the corresponding original data fragment from the original data sub-object copy; if they are not included, determine the corresponding original data fragment from the original data sub-object and / or redundant data sub-object of the striped data redundancy group to complete the reading of the third data.

[0192] After the first data completes asynchronous redundancy encoding and before new user data with the same identification information is written, all data related to that identification information is in an erasure coding redundancy protection state. If the distributed storage system 100 receives a new data read request, optionally, the third module 230 is further configured to:

[0193] Based on the size of the data shards determined by the distributed storage system, the read offset and data size in the fourth data read request, and the sub-objects of the corresponding storage nodes, the range of the striped data redundancy group covered by the fourth data to be read in the fourth data read request and the corresponding original data sub-objects and redundant data sub-objects of each striped data redundancy group within the range are determined.

[0194] The fourth module 240 is further configured to: determine the original data fragments one by one from the corresponding original data sub-objects and / or redundant data sub-objects of each striped data redundancy group within the range, so as to complete the reading of the fourth data.

[0195] In the various embodiments and / or optional embodiments of the above-described device, the parts of the method steps performed by each module of the device that are not mentioned in the above-described related method embodiments are the same, and will not be repeated here.

[0196] According to another aspect of this application, a computer-readable medium is also provided, the computer-readable medium storing computer-readable instructions that can be executed by a processor to implement the foregoing method embodiments.

[0197] It should be noted that the method embodiments and / or optional embodiments in this application do not strictly limit the order of execution of each step, as long as each method embodiment can solve the defects existing in the prior art, achieve the inventive purpose of this application, and obtain beneficial effects. The method embodiments in this application can be implemented in software and / or a combination of software and hardware. The software program involved in this application can be executed by a processor to implement the steps or functions of the above embodiments. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium.

[0198] Furthermore, part or all of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. The program instructions invoking the methods of this application may be stored in a fixed or removable recording medium, and / or transmitted via data streams in broadcast or other signal carrying media, and / or stored in the working memory of a computer device operating according to the program instructions.

[0199] According to another aspect of this application, an apparatus for asynchronous redundancy coding of data is also provided. The apparatus includes: a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the apparatus is triggered to run the methods and / or technical solutions of the foregoing embodiments.

[0200] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in the apparatus claims may also be implemented by a single unit or device through software and / or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any particular order.

Claims

1. A method for asynchronous redundancy coding of data, applied to a distributed storage system, characterized in that, The method includes: Obtain a first data write request, wherein the data write request includes identification information, write offset, data size, and first data to be written; Based on the number of original data fragments and redundant data fragments, the number of replicas, and the identification information determined by the distributed storage system for the striped data redundancy group, a sub-object of the corresponding storage node is determined. The sub-object includes an original data sub-object and corresponding original data sub-object replicas and redundant data sub-objects, which are used to store the original data fragments, original data fragment replicas, and redundant data fragments, respectively. Based on the size of the data shards determined by the distributed storage system, the write offset and the data size, and the original data sub-object, the range of the striped data redundancy group covered by the first data and the corresponding original data sub-object and its copy, and redundant data sub-object of each striped data redundancy group within the range are determined; Based on the size of the data shards, the first data is divided into several raw data shards, and each is written to a corresponding raw data sub-object copy of the corresponding striped data redundancy group. Metadata files are also created for each raw data sub-object copy, recording the write offset and data size of the corresponding raw data sub-object. Based on the asynchronous redundancy coding service running on the storage node, the first data is asynchronously redundant coded according to the metadata file of each original data sub-object replica, and the original data sub-object replicas and their metadata files that have completed asynchronous redundancy coding are deleted. The asynchronous redundancy coding of the first data according to the metadata file of each original data sub-object replica includes: reading the metadata file of each original data sub-object replica to determine the striped data redundancy group covered by the first data for each original data sub-object; for each striped data redundancy group covered by the first data, obtaining the corresponding original data fragment from the original data sub-object replica and other corresponding original data fragments from the original data sub-object, merging them into the original data fragments of the striped data redundancy group; performing asynchronous redundancy coding on the merged original data fragments of the striped data redundancy group to obtain new redundant data fragments of the striped data redundancy group; copying the original data fragments in the corresponding original data sub-object replica and the generated new redundant data fragments to the corresponding original data sub-object and the corresponding redundant data sub-object; traversing each striped data redundancy group covered by the first data to complete the asynchronous redundancy coding of the first data.

2. The method according to claim 1, characterized in that, If, during the process of retrieving other corresponding original data fragments from the original data sub-object, there are original data fragments that cannot be successfully retrieved, the method further includes: Based on the data fragments successfully obtained from the original data sub-objects and redundant data sub-objects corresponding to the striped data redundancy group, data recovery is performed on the original data fragments that could not be successfully obtained. Obtain the original data fragments after data recovery.

3. The method according to claim 1, characterized in that, If a second data write request including the identification information is obtained before asynchronous redundancy encoding of the first data, the method further includes: Based on the size of the data shards determined by the distributed storage system, the write offset and data size in the second data write request, and the sub-objects of the corresponding storage nodes, the range of the striped data redundancy group covered by the second data to be written in the second data write request and the corresponding original data sub-objects and their copies and redundant data sub-objects of each striped data redundancy group within the range are determined. Based on the size of the data shards, the second data is divided into several original data shards, and each shard is written to the corresponding original data sub-object copy of the corresponding striped data redundancy group. If there is no metadata file for the original data sub-object copy corresponding to the striped data redundancy group, then create the metadata file for the original data sub-object copy and record the write offset and data size of the corresponding original data sub-object. If there is a metadata file for the original data sub-object copy corresponding to the striped data redundancy group, then update the write offset and data size of the corresponding original data sub-object recorded in the metadata file. If the write offset and data size of this time are adjacent to or overlap with the write offset and data size recorded in the metadata file in terms of range, then merge them and record the write offset and data size of the corresponding original data sub-object after merging. The step of asynchronously redundantly encoding the first data based on the metadata file of each original data sub-object copy includes: Asynchronous redundant encoding is performed on the first data and the second data based on the metadata file of each original data sub-object copy.

4. The method according to claim 1, characterized in that, If a third data read request, including the identification information, is obtained before the first data is asynchronously redundantly encoded, the method further includes: Based on the size of the data shards determined by the distributed storage system, the read offset and data size in the third data read request, and the sub-objects of the corresponding storage nodes, the range of the striped data redundancy group covered by the third data to be read in the third data read request and the corresponding original data sub-objects and their copies and redundant data sub-objects of each striped data redundancy group within the range are determined. The read offset and data size in the third data read request are compared one by one with the write offset and data size of the corresponding original data sub-object recorded in the metadata file of the original data sub-object copy of each striped data redundancy group within the range. If they are included, the corresponding original data fragment is read from the original data sub-object copy. If they are not included, the corresponding original data fragment is determined from the original data sub-object and / or redundant data sub-object of the striped data redundancy group to complete the reading of the third data.

5. The method according to claim 1, characterized in that, The method further includes: Obtain a fourth data read request including the identification information; Based on the size of the data shards determined by the distributed storage system, the read offset and data size in the fourth data read request, and the sub-objects of the corresponding storage nodes, the range of the striped data redundancy group covered by the fourth data to be read in the fourth data read request and the corresponding original data sub-objects and redundant data sub-objects of each striped data redundancy group within the range are determined. The original data fragments are determined one by one from the corresponding original data sub-objects and / or redundant data sub-objects of each striped data redundancy group within the range to complete the reading of the fourth data.

6. An apparatus for asynchronous redundancy coding of data, deployed in a distributed storage system, characterized in that, The device includes: The first module is used to obtain a first data write request, wherein the data write request includes identification information, write offset, data size and first data to be written; The second module is used to: determine the sub-object of the corresponding storage node based on the number of original data fragments and the number of redundant data fragments, the number of replicas, and the identification information of the striped data redundancy group determined by the distributed storage system. The sub-object includes an original data sub-object and a corresponding original data sub-object replica and a redundant data sub-object, which are used to store the original data fragments, the original data fragment replicas, and the redundant data fragments, respectively. The third module is used to: determine the range of the striped data redundancy group covered by the first data and the corresponding original data sub-object and its copy and redundant data sub-object for each striped data redundancy group within the range, based on the size of the data shard determined by the distributed storage system, the write offset and the data size, and the original data sub-object; The fourth module is used to: divide the first data into several raw data fragments based on the size of the data fragments, write them one by one to the corresponding raw data sub-object copies of the corresponding striped data redundancy group, and create a metadata file for each raw data sub-object copy, recording the write offset and data size of the corresponding raw data sub-object; The fifth module is used for: performing asynchronous redundancy encoding on the first data based on the asynchronous redundancy encoding service running on the storage node, according to the metadata file of each original data sub-object replica, and deleting the original data sub-object replicas and their metadata files that have completed asynchronous redundancy encoding. The asynchronous redundancy encoding of the first data based on the metadata file of each original data sub-object replica includes: reading the metadata file of each original data sub-object replica to determine the striped data redundancy group covered by the first data for each original data sub-object; for each striped data redundancy group covered by the first data, obtaining the corresponding original data fragment from the original data sub-object replica and other corresponding original data fragments from the original data sub-object, merging them into the original data fragments of the striped data redundancy group; performing asynchronous redundancy encoding on the merged original data fragments of the striped data redundancy group to obtain new redundant data fragments of the striped data redundancy group; copying the original data fragments in the corresponding original data sub-object replica and the generated new redundant data fragments to the corresponding original data sub-object and the corresponding redundant data sub-object; and traversing each striped data redundancy group covered by the first data to complete the asynchronous redundancy encoding of the first data.

7. The apparatus according to claim 6, characterized in that, If, before asynchronous redundant encoding of the first data, the first module obtains a second data write request including the identification information, the third module is further configured to: Based on the size of the data shards determined by the distributed storage system, the write offset and data size in the second data write request, and the sub-objects of the corresponding storage nodes, the range of the striped data redundancy group covered by the second data to be written in the second data write request and the corresponding original data sub-objects and their copies and redundant data sub-objects of each striped data redundancy group within the range are determined. The fourth module is further configured to: divide the second data into several original data fragments based on the size of the data fragments, and write them one by one to the corresponding original data sub-object replicas of the corresponding striped data redundancy group; if there is no metadata file for the corresponding original data sub-object replicas of the corresponding striped data redundancy group, then create a metadata file for the original data sub-object replicas and record the write offset and data size of the corresponding original data sub-objects; if there is a metadata file for the corresponding original data sub-object replicas of the corresponding striped data redundancy group, then update the write offset and data size of the corresponding original data sub-objects recorded in the metadata file; if the write offset and data size of the current data sub-objects are adjacent to or overlap with the write offset and data size recorded in the metadata file, then merge them and record the write offset and data size of the corresponding original data sub-objects after merging. The fifth module is used to: perform asynchronous redundancy coding on the first data and the second data based on the metadata file of each original data sub-object copy, and delete the original data sub-object copy and its metadata file that have completed asynchronous redundancy coding, based on the asynchronous redundancy coding service running on the storage node.

8. The apparatus according to claim 6, characterized in that, If, before performing asynchronous redundancy encoding on the first data, the first module obtains a third data read request including the identification information, the third module is further configured to: Based on the size of the data shards determined by the distributed storage system, the read offset and data size in the third data read request, and the sub-objects of the corresponding storage nodes, the range of the striped data redundancy group covered by the third data to be read in the third data read request and the corresponding original data sub-objects and their copies and redundant data sub-objects of each striped data redundancy group within the range are determined. The fourth module is further configured to: compare the read offset and data size in the third data read request with the write offset and data size of the corresponding original data sub-object recorded in the metadata file of the original data sub-object copy of each striped data redundancy group within the range; if they are included, read the corresponding original data fragment from the original data sub-object copy; if they are not included, determine the corresponding original data fragment from the original data sub-object and / or redundant data sub-object of the striped data redundancy group to complete the reading of the third data.

9. The apparatus according to claim 6, characterized in that, If, after asynchronous redundancy encoding of the first data, the first module obtains a fourth data read request including the identification information, the third module is further configured to: Based on the size of the data shards determined by the distributed storage system, the read offset and data size in the fourth data read request, and the sub-objects of the corresponding storage nodes, the range of the striped data redundancy group covered by the fourth data to be read in the fourth data read request and the corresponding original data sub-objects and redundant data sub-objects of each striped data redundancy group within the range are determined. The fourth module is further configured to: determine the original data fragments one by one from the original data sub-objects and / or redundant data sub-objects corresponding to each striped data redundancy group within the range, so as to complete the reading of the fourth data.

10. A computer-readable medium, characterized in that, It stores computer program instructions that are executed by a processor to implement the method as described in any one of claims 1 to 5.

11. A device for asynchronous redundancy coding of data, characterized in that, The device includes: One or more processors; and A memory storing computer program instructions, which, when executed, cause the processor to perform the operations of the method as described in any one of claims 1 to 5.

Images