Method and control system for writing data to a striped data redundancy group

Abstract

The application relates to a data writing method and a control system of a striped data redundancy group. User data is acquired through a storage node; the number of stripes of the user data coverage is calculated according to the offset and the data size of the user data coverage, and the number u of shares of the original data to be updated in the stripe is determined, 0 < u <= n, wherein n is the number of shares of the original data of the striped data redundancy group before the user data is written; the stripes are processed in sequence, the striped data redundancy group is constructed according to a preset construction mode, and the corresponding storage device is written, when u <= (n-m+1) / 2, the striped data redundancy group is constructed by adopting partial update processing; when u > (n-m+1), the striped data redundancy group is constructed by adopting whole stripe processing. In the case of partial original data update, the striped redundancy group can be constructed by selecting a partial update mode according to the actual number of shares of the original data to be updated, and the number of IO interactions under the striped data writing is effectively reduced.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a method and control system for writing data into a striped data redundancy group. Background Technology

[0002] In distributed storage systems, the cluster size and the amount of data it carries are both large, making hardware failures unavoidable during system operation. To ensure high data availability—that is, to prevent data loss in the event of hardware failure within a certain tolerance range—data redundancy strategies are needed for data redundancy protection when designing distributed storage systems. Currently, the two main data redundancy strategies employed are multiple replicas and erasure coding.

[0003] For a single set of user data, a multi-replica strategy involves copying the user data multiple times and storing it on different storage devices to achieve data protection. However, this strategy incurs significant storage overhead, and the storage utilization rate of the commonly used three-replica strategy in production systems is only 33%. In contrast, the erasure coding strategy divides the user data into multiple chunks of original data. Each set of n chunks of original data is then grouped together, and erasure coding is used to generate m redundant chunks of data, each also of chunk size. The n chunks of original data and the m chunks of redundant data form a striped data redundancy group. Even if any data is lost, the original data can be reconstructed using decoding. Compared to the multi-replica strategy, the erasure coding strategy can achieve higher storage utilization while maintaining the same fault tolerance.

[0004] Data consistency is required between the original data and redundant data within the same banded data redundancy group. For erasure coding strategies, data consistency means that the m redundant data obtained after erasure coding of n original data are identical to the m actually stored redundant data. If a data write to the same banded data redundancy group fails, data consistency cannot be guaranteed.

[0005] Currently, during data writing, a sequence number is typically generated for each striped data redundancy group as a unified identifier. A two-phase commit approach is used: first, the data and sequence number are written to the log; then, a commit request moves the log to the data available for user reading. Each write operation requires updating every piece of data within the striped data redundancy group, even if the written data only involves a portion of the original data. This increases the number of I / O interactions in the storage system, which significantly degrades overall system performance. Summary of the Invention

[0006] In view of this, this application proposes a data writing method and control system for striped data redundancy groups to solve the above problems.

[0007] This application proposes a method for writing data to a striped data redundancy group, comprising the following steps:

[0008] User data is obtained through storage nodes;

[0009] The number of stripes covered by the user data is calculated based on the offset and data size of the user data coverage, and the number of copies u of the original data to be updated in the stripe is determined, 0 < u ≤ n, where n is the number of copies of the original data in the striped data redundancy group before the user data is written.

[0010] The stripes are processed sequentially, and striped data redundancy groups are constructed and written to the corresponding storage devices according to the preset construction method.

[0011] When u≤(n-m+1) / 2, the striped data redundancy group is constructed using partial update processing;

[0012] When u > (n-m+1) / 2, the striped data redundancy group is constructed by processing the entire strip;

[0013] In the formula, m represents the m redundant data in the striped data redundancy group before the user data is written.

[0014] As an optional implementation of this application, when determining the number u of the original data to be updated, the number of original data to be updated can be obtained by calculating the data range of the user data currently covered by the stripe.

[0015] As an optional implementation of this application, the m copies of the redundant data may be obtained by encoding the n copies of the original data using erasure coding.

[0016] As an optional implementation of this application, when u≤(n-m+1) / 2, the striped data redundancy group is constructed using partial update processing, including:

[0017] The storage node reads u copies of the original data to be updated and m copies of the redundant data, and performs a consistency check on the original data to be updated and the redundant data.

[0018] If they match, then a preset calculation rule is used to calculate u copies of the original data to be updated and m copies of the redundant data according to the calculation rule, so as to obtain u copies of new original data and m copies of new redundant data.

[0019] Pre-generated data labels are concatenated with u new original data and m new redundant data to obtain striped data redundancy groups;

[0020] The striped data redundancy group is written to the corresponding storage device according to the preset writing rules;

[0021] If there is a discrepancy, the striped data redundancy group is constructed by processing the entire strip.

[0022] As an optional implementation of this application, optionally, a calculation rule is preset, and calculations are performed on u copies of the original data to be updated and m copies of the redundant data according to the calculation rule to obtain u copies of new original data and m copies of new redundant data, including:

[0023] The user data is overwritten by the storage node to u copies of the original data to be updated, resulting in u copies of new original data;

[0024] By subtracting u copies of the original data to be updated from m copies of the redundant data using the storage node, and then adding u copies of the new original data, m copies of new redundant data are obtained.

[0025] As an optional implementation of this application, optionally, when u > (n-m+1) / 2, the striped data redundancy group is constructed using full-strip processing, including:

[0026] The storage node reads the original data within the current stripe that does not need to be updated, as well as the original data whose update range is not aligned at the beginning and end.

[0027] The original data that does not need to be updated and the original data whose update range is not aligned with the beginning and end are merged with u copies of the data to be updated to obtain new n copies of original data;

[0028] The n original data are used to generate m new redundant data through a data redundancy strategy;

[0029] Pre-generated data labels are concatenated with n new copies of the original data and m new copies of the redundant data to construct striped redundant data groups;

[0030] The striped data redundancy group is written to the corresponding storage device according to the preset writing rules.

[0031] As an optional implementation of this application, the striped data redundancy group may be written to the corresponding storage device according to a preset writing rule, including:

[0032] The storage node sends write requests to m storage devices to write m copies of new redundant data.

[0033] After receiving the write request, the m storage devices update their local data according to the write request and send the write result to the storage node;

[0034] After receiving the write result from the storage device, the storage node determines whether m new redundant data have been successfully written.

[0035] When the write result is that m new redundant data have been successfully written, a write request to write u new original data is sent to the storage device, and it is determined whether the u new original data have been successfully written.

[0036] If the write result is either m new redundant data or u new original data and the write fails, a consistency check and repair process is performed on the striped data redundancy group.

[0037] As an optional implementation of this application, optionally, reading the original data within the current stripe that does not need to be updated, and the original data whose update range is not aligned, through the storage node, further includes:

[0038] Determine whether the original data that does not need to be updated, and the original data whose update range is not aligned, need to be read through redundant data repair.

[0039] As an optional implementation of this application, optionally, when the write result is that m new redundant data have been successfully written, a write request to write u new original data is sent to the storage device, and it is determined whether the u new original data have been successfully written, including:

[0040] When the write result is that m new redundant data have been successfully written, a write request to write u new original data is sent to the storage device.

[0041] If the original data that does not need to be updated, and the original data whose update range is not aligned, need to be read through redundant data repair, then it is written to the storage device along with u copies of new original data;

[0042] If the original data that does not need to be updated, and the original data whose update range is not aligned, do not need to be read, and do not need to be completed through redundant data repair, then only u copies of the new original data are written to the storage device.

[0043] Whether the write was successful is determined based on the write result sent by the storage device;

[0044] Yes, then the writing of the striped data redundancy group is completed;

[0045] If not, then a consistency check and repair process will be performed on the striped data redundancy group.

[0046] In another aspect, this application provides a control system, comprising:

[0047] processor;

[0048] Memory used to store processor-executable instructions;

[0049] The processor is configured to implement the data writing method for the striped data redundancy group described above when executing the executable instructions.

[0050] Technical effects of the present invention:

[0051] This application addresses the issue of writing data to striped data redundancy groups. When only a portion of the original data is updated, it eliminates the need to reconstruct every single data entry within the striped data redundancy group. Instead, it allows for partial update of the striped data redundancy group based on the actual number of original data entries to be updated, effectively reducing the number of I / O interactions during striped data writing and maintaining overall system performance. Specifically, when the number of original data entries to be updated is less than or equal to (n-m+1) / 2, a partial update is used to construct the striped data redundancy group, where n is the number of original data entries in the striped data redundancy group before data writing. In other words, during user data writing, it is unnecessary to update every single data entry within the striped data redundancy group. When the number of original data entries u to be updated is less than or equal to (n-m+1) / 2, only the original data and redundant data covered by the data being written need to be partially updated to construct the striped data redundancy group, effectively reducing the number of I / O interactions during striped data writing.

[0052] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0053] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0054] Figure 1 The diagram shown is a schematic diagram of the data writing method for striped data redundancy groups according to the present invention;

[0055] Figure 2 This diagram illustrates the process of generating and writing striped data redundancy groups for user data according to the present invention.

[0056] Figure 3 This diagram illustrates the construction of striped data redundancy groups as part of the update method of the present invention.

[0057] Figure 4 The diagram illustrates the construction of striped data redundancy groups using the whole-strip method of the present invention.

[0058] Figure 5 This diagram illustrates the stripe range covered by the user data written according to the present invention.

[0059] Figure 6 This diagram illustrates the user data writing process of the present invention.

[0060] Figure 7 A flowchart illustrating part of the update method of the present invention for constructing and writing striped data redundancy groups;

[0061] Figure 8 The flowchart illustrates the construction and writing of striped data redundancy groups using the whole-strip method of the present invention. Detailed Implementation

[0062] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0063] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0064] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0065] Example 1

[0066] like Figure 1 , Figure 2 and Figure 6 As shown, this application proposes a method for writing data into a striped data redundancy group, comprising the following steps:

[0067] User data is obtained through storage nodes;

[0068] The number of stripes covered by the user data is calculated based on the offset and data size of the user data coverage, and the number of copies u of the original data to be updated in the stripe is determined, 0 < u ≤ n, where n is the number of copies of the original data in the striped data redundancy group before the user data is written.

[0069] The stripes are processed sequentially, and striped data redundancy groups are constructed and written to the corresponding storage devices according to the preset construction method.

[0070] When u≤(n-m+1) / 2, the striped data redundancy group is constructed using partial update processing;

[0071] When u > (n-m+1) / 2, the striped data redundancy group is constructed by processing the entire strip;

[0072] In the formula, m represents the m redundant data in the striped data redundancy group before the user data is written.

[0073] In this embodiment, for writing data to a striped data redundancy group, when only part of the original data is updated, it is not necessary to rebuild every piece of data within the striped data redundancy group each time. The striped redundancy group can be constructed by selecting a partial update method based on the actual number of original data to be updated, effectively reducing the number of IO interactions during striped data writing and maintaining the overall system performance. Specifically, the data writing method for the striped data redundancy group includes: step S100, obtaining user data through a storage node. Here, it should be noted that the cluster client receives user data with a unique identifier and selects a storage node from the cluster as the processing and distribution node for the user data based on the unique identifier of the user data. Specifically, the unique identifier of the user data is such as the user data name (data name), and a specific storage node is selected as the processing and distribution node based on the data name.

[0074] After acquiring user data, step S200 calculates the number of stripes covered by the user data based on the offset and data size, and determines the number u copies of original data to be updated in each stripe, where 0 < u ≤ n, and n is the number of original data copies in the striped data redundancy group before the user data is written. Further, n copies of original data are encoded using erasure coding to calculate m copies of redundant data. Specifically, the original data is grouped and copied or encoded to generate redundant data according to the data redundancy strategy. Each copy of original data and redundant data occupies the same storage capacity, denoted as chunk size in bytes. For example, if n = 4, m = 2, and chunk size = 64KB, then the total length of all original data in a stripe is stripe datasize = 256KB, and the total length of the original data and redundant data is stripe total size = 320KB. The cluster client sends the offset and data size of the user data coverage to the processing and distribution node, which calculates the number of stripes covered by the user data based on the offset and data size. For example, if the offset of the user data being written is 224KB and the data size (i.e., data length) is 512KB, the number of stripes covered by the user data being written is calculated as follows: the start stripe id = offset / stripe data size, rounded down to 0; the end stripe id = (offset + data size) / stripe data size, rounded down to 2. The user data being written this time covers stripes 0, 1, and 2, denoted as stripe0, stripe1, and stripe2.

[0075] After obtaining the number of stripes covered by user data, the original data to be updated in each stripe is determined one by one. For example, the offset of the data range covered by stripe0 is offset0 = 224KB, and data size0 = 96KB. Based on offset0 and data size0, the sequence number of the original data covered by user data in stripe0 can be calculated as 3. That is, the user data written this time needs to update the 3rd original data in stripe0, denoted as data03. In other words, the total number of original data copies to be updated in stripe0 is 1.

[0076] After determining the number of original data copies to be updated, step S300 sequentially processes the stripes, constructing striped data redundancy groups according to a preset construction method and writing them to the corresponding storage devices. Specifically, when u ≤ (n-m+1) / 2, partial update processing is used to construct the striped data redundancy groups; when u > (n-m+1) / 2, full stripe processing is used to construct the striped data redundancy groups. It is important to note that the construction method of the striped data redundancy groups is determined by comparing the number of original data copies to be updated with the number of original data copies in the striped data redundancy groups before the user data is written. This includes partial update processing and full stripe processing. In other words, during the user data writing process, it is not necessary to update every single data copy within the striped data redundancy groups each time. When the number of original data copies to be updated is less than or equal to (n-m+1) / 2, the striped data redundancy groups can be constructed using partial update processing, and the constructed striped data redundancy groups are then written to the corresponding storage devices. After processing one stripe, repeat steps 200 and 300 until all stripes have been processed and written. This effectively reduces the number of I / O interactions during striped data writing, maintaining overall system performance.

[0077] As an optional implementation of this application, when determining the number u of the original data to be updated, the number of original data to be updated can be obtained by calculating the data range of the user data currently covered by the stripe.

[0078] As an optional implementation of this application, the striped data redundancy group may, before the user data is written, include m copies of redundant data, wherein the m copies of redundant data are calculated by erasure coding of the n copies of the original data.

[0079] like Figure 3 , Figure 5 and Figure 7 As shown, as an optional implementation of this application, when u≤(n-m+1) / 2, the striped data redundancy group is constructed using partial update processing, including:

[0080] The storage node reads u copies of the original data to be updated and m copies of the redundant data, and performs a consistency check on the original data to be updated and the redundant data.

[0081] If they match, then a preset calculation rule is used to calculate u copies of the original data to be updated and m copies of the redundant data according to the calculation rule, so as to obtain u copies of new original data and m copies of new redundant data.

[0082] Pre-generated data labels are concatenated with u new original data and m new redundant data to obtain striped data redundancy groups;

[0083] The striped data redundancy group is written to the corresponding storage device according to the preset writing rules;

[0084] If there is a discrepancy, the striped data redundancy group is constructed by processing the entire strip.

[0085] Furthermore, as an optional implementation of this application, optionally, a calculation rule is preset, and calculations are performed on u copies of the original data to be updated and m copies of the redundant data according to the calculation rule to obtain u copies of new original data and m copies of new redundant data, including:

[0086] The user data is overwritten by the storage node to u copies of the original data to be updated, resulting in u copies of new original data;

[0087] By subtracting u copies of the original data to be updated from m copies of the redundant data using the storage node, and then adding u copies of the new original data, m copies of new redundant data are obtained.

[0088] In this embodiment, when u ≤ (n-m+1) / 2, a partial update process is used to construct the striped data redundancy group. Specifically, the processing and distribution node reads u copies of the original data to be updated and m copies of redundant data, and performs a consistency check on them. If the original data to be updated and the redundant data are consistent, the old data information about the u copies of the original data to be updated is subtracted from the m copies of redundant data. Further, the processing and distribution node overwrites the corresponding data range of the u copies of the data to be updated with user data, constructing u new copies of original data. Further still, the u new copies of original data are added to the m copies of redundant data after subtracting the old data, obtaining new m copies of redundant data. A data tag is generated by the processing and distribution node. It should be noted that in the case of partial original data updates, it is not necessary to update the data tags of the n+m copies of data; only the data tags of the currently written and overwritten original data and redundant data need to be updated. The data tag is concatenated with the u new copies of original data and the m new copies of redundant data to form the data that needs to be distributed to each storage node, thus constructing a new striped data redundancy group. The appropriate storage device is selected based on the unique identifier of the user data. The specific storage devices include storage nodes and hard drives.

[0089] like Figure 4 , Figure 5 and Figure 8 As shown, as an optional embodiment of this application, when u > (n-m+1) / 2, the striped data redundancy group is constructed using full-strip processing, including:

[0090] The storage node reads the original data within the current stripe that does not need to be updated, as well as the original data whose update range is not aligned at the beginning and end.

[0091] The original data that does not need to be updated and the original data whose update range is not aligned with the beginning and end are merged with u copies of the data to be updated to obtain new n copies of original data;

[0092] The n original data are used to generate m new redundant data through a data redundancy strategy;

[0093] Pre-generated data labels are concatenated with n new copies of the original data and m new copies of the redundant data to construct striped redundant data groups;

[0094] The striped data redundancy group is written to the corresponding storage device according to the preset writing rules.

[0095] In this embodiment, when u > (n-m+1) / 2, the striped data redundancy group is constructed using whole-strip processing. Specifically, the processing and distribution node reads the original data within the current strip that does not need updating and the original data with the update range aligned to the beginning and end. These two are then merged with u copies of the original data to be updated into a complete stripe, constructing n complete copies of original data. Further, the processing and distribution node generates m new redundant data copies from the n original data copies according to the data redundancy strategy. Even further, the processing and distribution node generates a data tag, which is then concatenated with the n new original data copies and the m new redundant data copies to form the final data to be stored, thus constructing a new striped data redundancy group. The corresponding storage device is selected for writing based on the unique identifier of the user data. It should be noted that the generated data tag is stored together with the data, effectively ensuring strong consistency between the data tag and the data.

[0096] As an optional implementation of this application, the striped data redundancy group may be written to the corresponding storage device according to a preset writing rule, including:

[0097] The storage node sends write requests to m storage devices to write m copies of new redundant data.

[0098] After receiving the write request, the m storage devices update their local data according to the write request and send the write result to the storage node;

[0099] After receiving the write result from the storage device, the storage node determines whether m new redundant data have been successfully written.

[0100] When the write result is that m new redundant data have been successfully written, a write request to write u new original data is sent to the storage device, and it is determined whether the u new original data have been successfully written.

[0101] If the write result is either m new redundant data or u new original data and the write fails, a consistency check and repair process is performed on the striped data redundancy group.

[0102] In this embodiment, the constructed striped data redundancy group is written to the storage devices according to a preset write rule. Specifically, the processing and distribution node sends data write requests to the m storage devices corresponding to the m copies of redundant data and waits for responses from the m corresponding storage devices. After receiving the new redundant data, the m storage devices directly write it to their local storage media to complete the data update and send the data write result to the processing and distribution node. After receiving the write results from the m storage devices, the processing and distribution node performs a judgment process on the write result. If the write result fails, the data write fails, and the consistency of the striped data redundancy group written this time needs to be checked and repaired. If the write result of the m new redundant data is successful, a write request to write u copies of new original data is sent. When both the m new redundant data and the u new original data are successfully written, the data write is successful. For data in the same striped data redundancy group, the distribution access and response result processing are performed in groups of m copies, ensuring that the data within the striped data redundancy group is always repairable. If the number of original data copies overwritten by the written user data in a striped data redundancy group is less than or equal to m, then the absolute repairability of the striped data redundancy group can be guaranteed.

[0103] As an optional implementation of this application, optionally, reading the original data within the current stripe that does not need to be updated, and the original data whose update range is not aligned, through the storage node, further includes:

[0104] Determine whether the original data that does not need to be updated, and the original data whose update range is not aligned, need to be read through redundant data repair.

[0105] In this embodiment, when constructing a striped data redundancy group during overall strip processing, if the original data that does not need to be updated and the original data whose update range is not aligned at the beginning and end need to be read, the redundancy data repair needs to be completed. In this case, the original data read and the data written this time are written together again.

[0106] As an optional implementation of this application, optionally, when the write result is that m new redundant data have been successfully written, a write request to write u new original data is sent to the storage device, and it is determined whether the u new original data have been successfully written, including:

[0107] When the write result is that m new redundant data have been successfully written, a write request to write u new original data is sent to the storage device.

[0108] If the original data that does not need to be updated, and the original data whose update range is not aligned, need to be read through redundant data repair, then it is written to the storage device along with u copies of new original data;

[0109] If the original data that does not need to be updated, and the original data whose update range is not aligned, do not need to be read, and do not need to be completed through redundant data repair, then only u copies of the new original data are written to the storage device.

[0110] Whether the write was successful is determined based on the write result sent by the storage device;

[0111] Yes, then the writing of the striped data redundancy group is completed;

[0112] If not, then a consistency check and repair process will be performed on the striped data redundancy group.

[0113] In this embodiment, after m new redundant data copies are successfully written, it is necessary to further determine whether n new original data copies have been successfully written. If n new original data copies are also successfully written, the user data write is successful. Specifically, the processing distribution node sends a write request for u new original data copies to the storage device. It should be noted that during the process of constructing the striped data redundancy group, if original data that does not need updating and original data whose update range is not aligned need to be repaired using redundant data, these two original data copies, along with the original data to be updated, are combined into n new original data copies and sent to the corresponding storage device for writing. If m new original data copies are also successfully written, the write of this striped data redundancy group is successful. If the write of m new original data copies fails, a consistency check and repair process needs to be performed on the striped data redundancy group.

[0114] In summary, the method of this application for writing striped data redundancy groups eliminates the need to update every single piece of data in the striped data redundancy group when updating part of the original data. It can select a partial update method to construct the striped redundancy group based on the actual number of original data to be updated, effectively reducing the number of IO interactions during striped data writing and maintaining the performance of the entire system.

[0115] The specific implementation process of each step will be described below.

[0116] 1. The cluster client receives a user data write request. The user data write request includes: the name of the user data, the write offset (offset = 224KB), the data size (data size = 512KB), and the actual data, with the data range to be written as follows: Figure 5As shown; where the parameters of the striped data redundancy group before the user data is written are: n=4, m=2, chunk size=64KB, then the length of all original data in a stripe is stripe data size=256KB, and the total length of the original data and redundant data is stripe total size=320KB;

[0117] 2. Select storage node 2 as the processing and distribution node for the user data based on the data name, and send the user data write request to storage node 2;

[0118] 3. After receiving a user data write request, storage node 2 calculates which stripes the written user data covers:

[0119] The start stripe ID is set to offset / stripe data size, rounded down to 0.

[0120] End stripe ID = (offset + data size) / stripe data size, rounded down to 2;

[0121] The user data written this time covers stripes 0, 1, and 2, denoted as stripe0, stripe1, and stripe2, respectively. Figure 5 As shown;

[0122] 4. Process stripe0, stripe1, and stripe2 one by one:

[0123] The data range covered by stripe0 has an offset of offset0 = 224KB and a data size of 0 = 32KB.

[0124] The data range covered by stripe1 has an offset of offset1 = 256KB and a data size1 = 256KB.

[0125] The data range covered by stripe2 has an offset of offset2 = 512KB and a data size2 = 160KB.

[0126] 5. Write to stripe0:

[0127] (1) Based on offset0 and data size0, the sequence number of the original data covered by the user data in stripe0 can be calculated to be 3. That is, the user data written this time needs to update the 3rd original data in stripe0, denoted as data03. There is a total of 1 copy, which means u is less than (n-m+1) / 2. Therefore, the partial update method is used to construct the striped redundant data group and write it.

[0128] (2) Storage node 2 reads the old data of data03, as well as the old data of two redundant data, denoted as parity04 and parity05;

[0129] (3) Check the data tags of data03, parity04 and parity05 to determine if the data is inconsistent. If it is inconsistent, construct a striped data redundancy group in the whole strip method and write it.

[0130] (4) Calculate according to the calculation rules:

[0131] First, delete the information about data03 from parity04 and parity05 respectively;

[0132] The written user data is overwritten onto data03 to form new original data data03';

[0133] Add the data from data03' to parity04 and parity05 respectively to form new redundant data parity04' and parity05';

[0134] (5) Generate a data tag dataTag0 and attach it to the headers of data03', parity04' and parity05' respectively to form the final stored data, i.e., the striped data redundancy group;

[0135] (6) Storage node 2 first sends write requests for parity04' and parity05' to the corresponding storage nodes;

[0136] (7) After receiving the result, the corresponding storage node writes it to the local storage medium and returns the writing result to storage node 2.

[0137] (8) Storage node 2 receives write responses from parity04' and parity05'. If any write fails, the write operation fails and the consistency check and repair task of stripe0 is executed. If all writes are successful, the write request for data03' is sent to the corresponding storage node.

[0138] (9) After receiving the result, the corresponding storage node writes it to the local storage medium and returns the writing result to storage node 2.

[0139] (10) If storage node 2 receives the write response of data03', the write operation fails and the consistency check and repair task of stripe0 is executed; if all writes are successful, the stripe0 write is successful.

[0140] 6. Write to stripe1:

[0141] (1) Based on offset1 and data size1, it can be calculated that the user data written this time needs to update the 0th, 1st, 2nd and 3rd original data in stripe1, denoted as data10~13, a total of 4 data. Since u is greater than (n-m+1) / 2, the striped redundant data group is constructed and written using the whole strip method.

[0142] (2) The user data written this time covers the entire stripe1, so there is no need to read any data. The redundant data parity14 and parity15 are directly generated from the original data data10~13.

[0143] (3) Generate a data tag dataTag1 and attach it to the header of data10-13 and parity14 and parity15 respectively to form the final stored data, i.e., the striped data redundancy group;

[0144] (4) Storage node 2 first sends write requests for parity14 and parity15 to the corresponding storage nodes;

[0145] (5) After receiving the result, the corresponding storage node writes it to the local storage medium and returns the writing result to storage node 2.

[0146] (6) If storage node 2 receives write responses from parity14 and parity15, and one of them fails, the write operation fails and the consistency check and repair task of stripe1 is executed; if all writes are successful, write requests for data10 and data13 are sent to the corresponding storage nodes.

[0147] (7) If data10 and data13 fail to write, the write operation fails and the consistency check and repair task of stripe1 is executed; if all writes are successful, the write operation of stripe1 is successful.

[0148] 7. Write to stripe2:

[0149] (1) Based on offset2 and data size2, it can be calculated that the user data written this time needs to update the 0th, 1st and 2nd original data in stripe2, denoted as data20~22, a total of 3 copies. u is greater than (n-m+1) / 2, so the striped redundant data group is constructed and written in the whole strip method. Among them, data22 has no chunk size alignment and the 3rd original data data23 is not updated.

[0150] (2) Storage node 2 first reads the old data of data23 and data22, merges it with the new data of data20 to data22, and generates 4 new original data data20' to 23';

[0151] (3) If the previous step of reading data23 and data22 was completed by repair, then data23 and data22 also need to be sent and written together with the original data of this update;

[0152] (4) Generate redundant data parity24' and parity25' from the original data data20'~22';

[0153] (5) Generate a data tag dataTag2 and attach it to the header of data20'~22' and parity24' and parity25' respectively to form the final stored data, i.e., the striped data redundancy group;

[0154] (6) Storage node 2 first sends write requests for parity24' and parity25' to the corresponding storage nodes;

[0155] (7) After receiving the result, the corresponding storage node writes it to the local storage medium and returns the writing result to storage node 2.

[0156] (8) If storage node 2 receives write responses from parity24' and parity25', and one of them fails, the write operation fails and the consistency check and repair task of stripe2 is executed; if all writes are successful, write requests from data20' to data22' are sent to the corresponding storage nodes.

[0157] (9) If step (2) reads data23 and data22 by repair, then data23' needs to be sent to the corresponding storage node for writing along with data20' to data22'; otherwise, there is no need to send a write request for data23'.

[0158] (10) If data20' and data22' fail to write, the write operation fails and the consistency check and repair task of stripe2 is executed; if all writes are successful, the stripe2 write is successful.

[0159] 9. The write process is now complete.

[0160] It should be noted that although the above description is provided as an example, those skilled in the art will understand that this disclosure is not limited thereto. In fact, users can flexibly configure the settings according to actual application scenarios, as long as the technical functions of this application can be achieved by following the above technical methods.

[0161] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the control methods described above. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium can also include combinations of the above types of memory.

[0162] Example 2

[0163] Furthermore, in another aspect, this application provides a control system, comprising:

[0164] processor;

[0165] Memory used to store processor-executable instructions;

[0166] The processor is configured to implement the data writing method for the striped data redundancy group described above when executing the executable instructions.

[0167] This disclosure discloses an embodiment of a system including a processor and a memory for storing processor-executable instructions. The processor is configured to implement, when executing the executable instructions, a data writing method for a striped data redundancy group as described above.

[0168] It should be noted here that the number of processors can be one or more. Furthermore, the control system in this embodiment may also include input devices and output devices. The processors, memory, input devices, and output devices can be connected via a bus or other means, without specific limitations herein.

[0169] As a computer-readable storage medium, the memory can be used to store software programs, computer-executable programs, and various modules, such as the program or module corresponding to the data writing method for a striped data redundancy group according to an embodiment of this disclosure. The processor executes various functional applications and data processing of the control system by running the software programs or modules stored in the memory.

[0170] Input devices can be used to receive input digital numbers or signals. These signals can be key signals related to user settings and function control of the device / terminal / server. Output devices can include display devices such as screens.

[0171] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for writing data to a striped data redundancy group, characterized in that, Includes the following steps: User data is obtained through storage nodes; The number of stripes covered by the user data is calculated based on the offset and data size of the user data coverage, and the number of copies u of the original data to be updated in the stripe is determined, 0 < u ≤ n, where n is the number of copies of the original data in the striped data redundancy group before the user data is written. The stripes are processed sequentially, and striped data redundancy groups are constructed and written to the corresponding storage devices according to the preset construction method. Specifically, when u ≤ (n-m+1) / 2, a partial update process is used to construct the striped data redundancy group; u copies of the original data to be updated and m copies of redundant data are read through the storage node, and a consistency check is performed on the original data to be updated and the redundant data; if they are consistent, a preset calculation rule is used to calculate u copies of the original data to be updated and m copies of the redundant data according to the calculation rule to obtain u copies of new original data and m copies of new redundant data; data tags are pre-generated and concatenated with u copies of new original data and m copies of new redundant data to obtain the striped data redundancy group; the striped data redundancy group is written to the corresponding storage device according to a preset writing rule; if they are inconsistent, a full stripe processing process is used to construct the striped data redundancy group. When u > (n-m+1) / 2, the striped data redundancy group is constructed using full-strip processing; the original data that does not need to be updated within the current stripe and the original data whose update range is not aligned are read through the storage node; the original data that does not need to be updated and the original data whose update range is not aligned are merged with u copies of the original data to be updated to obtain new n copies of original data; the new n copies of original data generate m copies of new redundant data through a data redundancy strategy; data tags are pre-generated and concatenated with the new n copies of original data and the m copies of new redundant data to construct the striped redundant data group; the striped data redundancy group is written to the corresponding storage device according to a preset writing rule. In the formula, m represents the m redundant data in the striped data redundancy group before the user data is written; the m redundant data are calculated from the n original data through erasure coding.

2. The data writing method for striped data redundancy groups according to claim 1, characterized in that, When determining the number u of the original data to be updated, the number of original data to be updated is obtained by calculating the data range of the user data covered by the current stripe.

3. The data writing method for striped data redundancy groups according to claim 1, characterized in that, A preset calculation rule is established, and calculations are performed on u copies of the original data to be updated and m copies of the redundant data according to the calculation rule to obtain u copies of new original data and m copies of new redundant data, including: The user data is overwritten by the storage node to u copies of the original data to be updated, resulting in u copies of new original data; By subtracting u copies of the original data to be updated from m copies of the redundant data using the storage node, and then adding u copies of the new original data, m copies of new redundant data are obtained.

4. The data writing method for striped data redundancy groups according to claim 1, characterized in that, Writing the striped data redundancy group to the corresponding storage device according to a preset writing rule includes: The storage node sends write requests to m storage devices to write m copies of new redundant data. After receiving the write request, the m storage devices update their local data according to the write request and send the write result to the storage node; After receiving the write result from the storage device, the storage node determines whether m new redundant data have been successfully written. When the write result is that m new redundant data have been successfully written, a write request to write u new original data is sent to the storage device, and it is determined whether the u new original data have been successfully written. If the write result is either m new redundant data or u new original data, and the write fails, a consistency check and repair process is performed on the striped data redundancy group.

5. The data writing method for striped data redundancy groups according to claim 4, characterized in that, Reading the original data within the current stripe that does not need updating, and the original data whose update range is not aligned, through the storage node, also includes: Determine whether the original data that does not need to be updated, and the original data whose update range is not aligned, need to be read through redundant data repair.

6. The data writing method for striped data redundancy groups according to claim 5, characterized in that, When the write result indicates that m new redundant data have been successfully written, a write request to write u new original data is sent to the storage device, and it is determined whether the u new original data have been successfully written, including: When the write result is that m new redundant data have been successfully written, a write request to write u new original data is sent to the storage device. If the original data that does not need to be updated, and the original data whose update range is not aligned, need to be read through redundant data repair, then it is written to the storage device along with u copies of new original data; If the original data that does not need to be updated, and the original data whose update range is not aligned, do not need to be read, and do not need to be completed through redundant data repair, then only u copies of the new original data are written to the storage device. Whether the write was successful is determined based on the write result sent by the storage device; Yes, then the writing of the striped data redundancy group is completed; If not, then a consistency check and repair process will be performed on the striped data redundancy group.

7. A control system, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the data writing method of any one of claims 1 to 6 when executing the executable instructions.

Images