A method and system for reducing the overhead of erasure coding data update storage and repair.

Abstract

This invention discloses a method and system for reducing the storage and repair overhead of erasure coding data updates, relating to the technical field of distributed storage. The method involves obtaining file information of stored files from an erasure coding storage system, extracting features, selecting the optimal classification features and assigning weights to them, and then training a file type classification model. The trained model is used to classify newly written files into read-only and non-read-only files. For non-read-only files, storage space is reserved in the original parity block generated by erasure coding to store update increments; for read-only files, no storage space is reserved. When any storage node fails, the minimum parity update increment required for repair is calculated, and its shareability is utilized to read in parallel on the remaining non-faulty storage nodes to repair the faulty node. This invention reduces the storage overhead after erasure coding data updates and the repair overhead after failures, improving the update and repair performance of the erasure coding storage system.

Description

Technical Field

[0001] This invention relates to the technical field of distributed storage, and more specifically, to a method and system for reducing the overhead of erasure coding data update storage and repair. Background Technology

[0002] As storage systems continue to expand, component failures have become commonplace. Erasure coding divides stored files into multiple data blocks, encodes these blocks using a specific encoding method, generates new parity blocks, and places them on different storage nodes. When a data block fails, the storage system can read the intact data and parity blocks, decode them, and recover the faulty block. Compared to replication techniques, erasure coding offers lower storage overhead and higher reliability with the same redundancy. However, when applications update data, erasure coding systems need to maintain the consistency of data blocks and parity blocks, leading to significant I / O overhead. Furthermore, the recovery process after a data update requires randomly reading old parity blocks and update increments scattered throughout the logs, resulting in slow data recovery. Therefore, finding a more efficient erasure coding method for data recovery is crucial.

[0003] Existing data update methods, such as Full-logging (FL), reduce the overhead of reading parity blocks from disk to maintain up-to-date data by storing all data and parity update increments in a log cache. However, when reading data sequentially, a large number of lookups must be performed in the log. Parity-logging (PL) directly overwrites the latest data block when updating the original data block, and the parity increment is still stored in the log of the parity node. This saves the disk read overhead of parity blocks and avoids the large amount of log read overhead that exists in FL when reading data blocks. However, storing and reading parity increments still introduces additional disk seek overhead and generates disk fragmentation. The latest proposed Parity-logging with reserved space (PLR) uses reserved space to replace the log cache in PL. This method still updates data blocks directly, but reserves an additional storage space after each parity block to store the parity increment, thereby ensuring that parity blocks and parity increments can be retrieved sequentially. This method effectively reduces disk fragmentation and avoids the disk overhead of PL needing to access increments in the log. However, this method reserves storage space after the parity block of all types of files, resulting in a huge storage overhead. It does not take into account the existence of a large number of read-only files that do not need to be updated, nor does it take into account the shareability of increments during the repair process.

[0004] Existing technology discloses a data storage method and system, relating to the field of data storage, including the following steps: dividing a memory merge cache area into multiple erasure code stripes consisting of N data blocks and M parity blocks; matching the characteristics of the data to be stored with the data block locations; storing the matched data into the corresponding data block locations; calculating M parity blocks after the N data blocks are full; and storing the N data blocks and M parity blocks into a storage node. This application writes data to the cache area, and when reading data sequentially, a large number of lookups must be performed on the cache log, resulting in significant storage and repair overhead. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies in terms of high storage and repair costs during erasure coding data updates and repairs, this invention provides a method and system for reducing the storage and repair costs of erasure coding data updates. It takes into account the impact of file type on updates and the shareability of data, reduces the storage costs after erasure coding data updates and the repair costs after failures, and improves the update and repair performance of erasure coding storage systems.

[0006] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0007] This invention provides a method for reducing the overhead of erasure coding data update storage and repair, including:

[0008] S1: Retrieve file information of stored files from the erasure coding storage system;

[0009] S2: Perform feature extraction on the file information of the stored file to obtain the optimal classification features;

[0010] S3: Assign different weights to the optimal classification features, input the optimal classification features and their corresponding weights into the existing file type classification model for training, and obtain a trained file type classification model.

[0011] S4: Using a trained file type classification model, newly written files to the erasure coding storage system are classified into read-only files and non-read-only files;

[0012] S5: During data storage, each file is divided into several raw data blocks and raw parity blocks, and stored on several storage nodes; for non-read-only files, storage space is reserved after each parity block; for read-only files, no storage space is reserved after each parity block.

[0013] S6: During the data update process, non-read-only files generate new data blocks that overwrite the original data blocks; the parity update increment is calculated based on the new data blocks and stored in the reserved storage space after each original parity block; read-only files are not updated.

[0014] S7: During the data repair process, when any storage node fails, calculate the minimum parity update increment required for repair, read in parallel on the remaining non-faulty storage nodes, and repair the faulty node.

[0015] Preferably, the file information of the stored file includes IP address, timestamp, file size, user ID, user group, file type, file access time, and file modification time.

[0016] Preferably, the specific method of step S2 is as follows:

[0017] S2.1: Extract features from the file information of the stored files to obtain the corresponding classification features f. l The initial feature set F = {f1, f2, ..., f} is formed. l , ..., f n}, l = 1, ..., n;

[0018] S2.2: Use the information gain method to filter the initial feature set, and select the classification feature f with the largest information gain in the initial feature set F. j Construct the first round of optimized feature set F1 = {f j};

[0019] S2.3: In the initial feature set F, remove the classification feature f with the largest information gain. j ; Select the classification feature f with the highest information gain from the remaining classification features. k Add the features to the first round of optimization feature set to obtain the second round of optimization feature set F2 = {f j f k};

[0020] S2.4: Compare the first round of optimized feature set F1 与 The second round of feature set optimization F 2 The information gain effect, if the first round of optimization of feature set F 1 Superior to the second round of feature set optimization F 2 Then the first round of optimized feature set F 1 If the set of classification features is the optimal set, then the classification features in it are the optimal classification features; otherwise, proceed to step S2.5.

[0021] S2.5: Repeat steps S2.3-S2.4 until the feature set of the next round of optimization is no longer better than the feature set of the previous round of optimization. The previous round of optimization feature set is then used as the optimal classification feature set, and the classification features in it are the optimal classification features.

[0022] Preferably, the specific process of step S5 is as follows:

[0023] During data storage, the EC-(k,m) encoding method is used to divide each file into k original data blocks and m original parity blocks, which are stored on n storage nodes, n = k + m. For non-read-only files, storage space is reserved after each parity block on the corresponding storage node; for read-only files, no storage space is reserved after each parity block on the corresponding storage node.

[0024] Preferably, the specific process of step S6 is as follows:

[0025] During the data update process, a new data block d′ is generated in the non-read-only file. i Using the new data block d′ i Corresponding to the original data block d i , i = 0, 1, ..., k;

[0026] Calculate the data update increment:

[0027] Δd i =d′ i -d i

[0028] Calculate the parity update increment:

[0029] Δp j =α i，j ×Δd i

[0030] Where, Δp j α represents the parity update increment of the j-th parity block. i，j Δd represents the erasure coding coefficients of the j-th parity block for the i-th new data block, j = 0, 1, ..., m; i d′ represents the data update increment of the i-th new data block. i Let d represent the i-th new data block. i This represents the i-th original data block;

[0031] The parity update increment is stored in the reserved storage space after each parity block;

[0032] Read-only files do not undergo data updates.

[0033] Preferably, the specific process of step S7 is as follows:

[0034] During data repair, when w storage nodes fail, w ≤ m;

[0035] For non-read-only files, calculate the data update increment of the faulty node to obtain the corresponding parity update increment; merge the parity update increment with the original parity block to obtain a new parity block:

[0036] p′ j =Δpj+pj

[0037] In the formula, p′ j p represents the j-th new parity check block. j This represents the j-th parity check block;

[0038] When w < m, all new data blocks and the parity update increment of any new parity block on the non-faulty storage nodes are read in parallel, erasure coding decoding is performed, and the faulty nodes are repaired.

[0039] When w = m, all new data blocks and the parity update increment of each new parity block are read in parallel from the non-faulty storage nodes. Perform erasure coding decoding operations to repair faulty nodes;

[0040] For read-only files, read all raw data blocks and raw parity blocks from non-faulty storage nodes and repair the faulty nodes.

[0041] This invention also provides a system for reducing the overhead of erasure coding data update storage and repair, comprising:

[0042] The information acquisition module is used to obtain file information of stored files from the erasure coding storage system;

[0043] The feature optimization module is used to perform feature extraction operations on the file information of the stored files to obtain the optimal classification features;

[0044] The model training module is used to assign different weights to the optimal classification features, input the optimal classification features and their corresponding weights into the existing file type classification model for training, and obtain the trained file type classification model.

[0045] The file classification module uses a trained file type classification model to classify newly written files into the erasure coding storage system into read-only files and non-read-only files.

[0046] The data storage module is used to divide each file into several raw data blocks and raw parity blocks during the data storage process, and store them on several storage nodes; for non-read-only files, storage space is reserved after each parity block; for read-only files, no storage space is reserved after each parity block.

[0047] The data update module is used during the data update process. For non-read-only files, new data blocks are generated to overwrite the original data blocks. The parity update increment is calculated based on the new data blocks and stored in the reserved storage space after each original parity block. Read-only files are not updated.

[0048] The data repair module is used during the data repair process. When any storage node fails, it calculates the minimum parity update increment required for repair, reads it in parallel on the remaining non-faulty storage nodes, and repairs the faulty node.

[0049] Preferably, the data storage module operates as follows:

[0050] During data storage, the EC-(k,m) encoding method is used to divide each file into k raw data blocks and m raw parity blocks, which are stored on n storage nodes, where n = k + m. For non-read-only files, storage space is reserved after each parity block on the corresponding storage node; for read-only files, no storage space is reserved after each parity block on the corresponding storage node.

[0051] Preferably, the specific operation of the data update module is as follows:

[0052] During the data update process, a new data block d′ is generated in the non-read-only file. i Using the new data block d′ i This corresponds to covering the original data block di, i = 0, 1, ..., k;

[0053] Calculate the data update increment:

[0054] Δd i =d′ i -d i

[0055] Calculate the parity update increment:

[0056] Δp j =α i，j ×Δd i

[0057] Where, Δp j α represents the parity update increment of the j-th parity block. i，j Δd represents the erasure coding coefficients of the j-th parity block for the i-th new data block, j = 0, 1, ..., m; i d′ represents the data update increment of the i-th new data block. i Let d represent the i-th new data block. i This represents the i-th original data block;

[0058] The parity update increment is stored in the reserved storage space after each parity block;

[0059] Read-only files do not undergo data updates.

[0060] Preferably, the data repair module operates as follows:

[0061] During data repair, when w storage nodes fail, w ≤ m;

[0062] For non-read-only files, calculate the minimum parity update increment required for repair;

[0063] When w < m, read kw new data blocks, w original parity blocks and the parity update increment of any remaining original parity block from the non-faulty storage nodes in parallel, perform erasure coding decoding operation, and repair the faulty nodes.

[0064] When w = m, kw new data blocks, w original parity blocks, and the parity update increment of each original parity block are read in parallel from the non-faulty storage nodes. Perform erasure coding decoding operations to repair faulty nodes;

[0065] For read-only files, read all raw data blocks and raw parity blocks from non-faulty storage nodes and repair the faulty nodes.

[0066] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0067] This invention acquires file information of files already stored in an erasure coding storage system, extracts features from them, obtains and weights the optimal classification features, and trains a file type classification model to obtain a trained file type classification model. Then, the trained file type classification model is used to classify newly written files into read-only files and non-read-only files. During data storage, each file is divided into several original data blocks and original parity blocks for storage. However, for non-read-only files, storage space is reserved after each parity block; for read-only files, no storage space is reserved after each parity block. During data updates, non-read-only files are updated to generate new data blocks, overwriting the original data blocks, and the parity update increment is calculated based on the new data blocks and stored in the reserved storage space after each original parity block. Different update methods are used for newly written files depending on whether they are read-only or not. Storage space is reserved only after each parity block of non-read-only files, saving storage overhead after data updates and improving update speed. Finally, the minimum parity update increment required for repair is calculated, and parallel reading is performed on the remaining non-faulty storage nodes to repair faulty nodes. By considering the shareability of data and reading data in parallel, the overhead of repairing faulty data is reduced, and the repair speed is improved. This invention effectively reduces the storage overhead after erasure coding data updates and the repair overhead after failures, thus improving the update and repair performance of the erasure coding storage system. Attached Figure Description

[0068] Figure 1This is a flowchart of a method for reducing the overhead of erasure coding data update storage and repair as described in Example 1. Figure 2 This is a schematic diagram comparing the method provided in this embodiment as described in Example 2 with the traditional FO, FL, PL, and PLR methods in terms of data update and storage.

[0069] Figure 3 This is a schematic diagram illustrating the method provided in Example 2 for repairing any two faulty nodes.

[0070] Figure 4 This is a schematic diagram illustrating how the method provided in this embodiment, as described in Example 2, repairs any three faulty nodes.

[0071] Figure 5 This is a schematic diagram of the system for reducing the overhead of erasure coding data update storage and repair as described in Example 3. Detailed Implementation

[0072] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent.

[0073] To better illustrate this embodiment, some parts in the accompanying drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions;

[0074] It will be understood by those skilled in the art that certain well-known structures and their descriptions may be omitted in the accompanying drawings.

[0075] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0076] Example 1

[0077] This embodiment provides a method for reducing the overhead of erasure coding data update storage and repair, such as Figure 1 Shown, including:

[0078] S1: Retrieve file information of stored files from the erasure coding storage system;

[0079] S2: Perform feature extraction on the file information of the stored file to obtain the optimal classification features;

[0080] S3: Assign different weights to the optimal classification features, input the optimal classification features and their corresponding weights into the existing file type classification model for training, and obtain a trained file type classification model.

[0081] S4: Using a trained file type classification model, newly written files to the erasure coding storage system are classified into read-only files and non-read-only files;

[0082] S5: During data storage, each file is divided into several raw data blocks and raw parity blocks, and stored on several storage nodes; for non-read-only files, storage space is reserved after each parity block; for read-only files, no storage space is reserved after each parity block.

[0083] S6: During the data update process, non-read-only files generate new data blocks that overwrite the original data blocks; the parity update increment is calculated based on the new data blocks and stored in the reserved storage space after each original parity block; read-only files are not updated.

[0084] S7: During the data repair process, when any storage node fails, calculate the minimum parity update increment required for repair, read in parallel on the remaining non-faulty storage nodes, and repair the faulty node.

[0085] In the specific implementation process, this embodiment obtains the file information of the files already stored in the erasure coding storage system, extracts their features, obtains the optimal classification features and assigns weights, trains the file type classification model, and obtains a trained file type classification model. Then, the trained file type classification model is used to classify newly written files into read-only files and non-read-only files. During data storage, each file is divided into several original data blocks and original parity blocks for storage. However, for non-read-only files, storage space is reserved after each parity block; for read-only files, no storage space is reserved after each parity block. When data is updated, non-read-only files are updated to generate new data blocks, overwriting the original data blocks, and the parity update increment is calculated based on the new data blocks and stored in the reserved storage space after each original parity block. Different update methods are used for newly written files depending on whether they are read-only or not. Storage space is reserved only after each parity block of non-read-only files, saving storage overhead after data updates and improving update speed. Finally, the minimum parity update increment required for repair is calculated, and the data is read in parallel on the remaining non-faulty storage nodes to repair the faulty nodes. By considering the shareability of data and reading data in parallel, the overhead of repairing faulty data is reduced, and the repair speed is improved. This embodiment effectively reduces the storage overhead after erasure coding data updates and the repair overhead after failures, thus improving the update and repair performance of the erasure coding storage system.

[0086] Example 2

[0087] This embodiment provides a method for reducing the overhead of erasure coding data update storage and repair, including:

[0088] S1: Retrieve file information of stored files from the erasure coding storage system;

[0089] The file information of the stored file includes IP address, timestamp, file size, user ID, user group, file type, file access time, and file modification time;

[0090] S2: Perform feature extraction on the file information of the stored file to obtain the optimal classification features; the specific method is as follows:

[0091] S2.1: Extract features from the file information of the stored files to obtain the corresponding classification features f. l The initial feature set F = {f1, f2, ..., f} is formed. l ,…,f n}, l=1,…,n;

[0092] S2.2: Use the information gain method to filter the initial feature set, and select the classification feature f with the largest information gain in the initial feature set F. j Construct the first round of optimized feature set F 1 ={f j};

[0093] S2.3: In the initial feature set F, remove the classification feature f with the largest information gain. j ; Select the classification feature f with the highest information gain from the remaining classification features. k Add the features to the first round of optimization feature set to obtain the second round of optimization feature set F. 2 =f j ,fk};

[0094] S2.4: Compare the first round of optimized feature set F 1 With the second round of feature set optimization F 2 The information gain effect, if the first round of optimization of feature set F 1 Superior to the second round of feature set optimization F 2 Then the first round of optimized feature set F 1 If the set of classification features is the optimal set, then the classification features in it are the optimal classification features; otherwise, proceed to step S2.5.

[0095] S2.5: Repeat steps S2.3-S2.4 until the feature set of the next round of optimization is no longer better than the feature set of the previous round of optimization. The previous round of optimization feature set is then used as the optimal classification feature set, and the classification features in it are the optimal classification features.

[0096] S3: Assign different weights to the optimal classification features, input the optimal classification features and their corresponding weights into the existing file type classification model for training, and obtain a trained file type classification model.

[0097] S4: Using a trained file type classification model, newly written files to the erasure coding storage system are classified into read-only files and non-read-only files;

[0098] S5: During data storage, the EC-(k,m) encoding method is used to divide each file into k original data blocks and m original parity blocks, which are stored on n storage nodes, n = k + m; for non-read-only files, storage space is reserved after each parity block on the corresponding storage node; for read-only files, no storage space is reserved after each parity block on the corresponding storage node.

[0099] S6: During the data update process, non-read-only files generate new data blocks, overwriting the original data blocks; the parity update increment is calculated based on the new data blocks and stored in the reserved storage space after each original parity block; read-only files are not updated; the specific process is as follows:

[0100] During the data update process, a new data block d′ is generated in the non-read-only file. i Using the new data block d′ i This corresponds to covering the original data block di, i = 0, 1, ..., k;

[0101] Calculate the data update increment:

[0102] Δd i =d′ i -d i

[0103] Calculate the parity update increment:

[0104] Δpj=αi,j×Δdi

[0105] Where Δpj represents the parity update increment of the j-th parity block, α i，j Δd represents the erasure coding coefficients of the j-th parity block for the i-th new data block, j = 0, 1, ..., m; i Let d'i represent the data update increment of the i-th new data block, and d'i represent the i-th new data block. i This represents the i-th original data block;

[0106] The parity update increment is stored in the reserved storage space after each parity block;

[0107] Read-only files do not undergo data updates.

[0108] S7: During the data repair process, when any storage node fails, calculate the minimum parity update increment required for repair, read in parallel on the remaining non-faulty storage nodes, and repair the faulty node.

[0109] The specific process is as follows:

[0110] During data repair, when w storage nodes fail, w ≤ m;

[0111] For non-read-only files, calculate the minimum parity update increment required for repair;

[0112] When w < m, read kw new data blocks, w original parity blocks and the parity update increment of any remaining original parity block from the non-faulty storage nodes in parallel, perform erasure coding decoding operation, and repair the faulty nodes.

[0113] When w = m, kw new data blocks, w original parity blocks, and the parity update increment of each original parity block are read in parallel from the non-faulty storage nodes. Perform erasure coding decoding operations to repair faulty nodes;

[0114] For read-only files, read all raw data blocks and raw parity blocks from non-faulty storage nodes and repair the faulty nodes.

[0115] In the specific implementation process, taking newly written files a and b as an example, a pre-trained file type classification model is used to classify files a and b, where a is a non-read-only file and b is a read-only file; each file is divided into 2 original data blocks and 2 original parity blocks, stored on 4 storage nodes; for file a, storage space is reserved after the 2 original parity blocks, and for file b, no storage space is reserved after the 2 original parity blocks; when file a is updated, the updated parts are a′1 and a′2; as... Figure 2 As shown, the traditional FO strategy involves all data blocks a1, a2, and parity block p. ia p ja Updating to the latest version incurs huge storage overhead; the traditional FL strategy places the updated data parts a′1 and a′2 in the log cache of the data storage node, and updates all parity update increments Δp. ia , Δp ib All data is placed in the log cache of the parity storage node. When reading file 'a', the required update information needs to be read from the log cache, resulting in significant disk search overhead. Traditional PL strategies update the original data blocks directly, while parity update increments are still placed in the log cache of the parity storage node. During data repair, reading the randomly stored parity update increments from the log cache still incurs significant disk search overhead. Traditional PLR strategies update the original parity block p... ia p ja The next step reserves one storage space to hold the parity update increment Δp for file a. ia , Δpja However, for read-only file b, it is also in the original parity block p. ib p jb The subsequent sequential reservation of a storage space leads to the classification of storage spaces; however, the method provided in this embodiment distinguishes between read-only files and non-read-only files. For non-read-only file a, the original data block is directly updated, and the original parity block p is updated. ia p ja The next step is to reserve one storage space to store the parity update increment Δp for file a. ia , Δp ja For read-only files, the original parity block p ib p jb No storage space is reserved afterward; the method provided in this embodiment selects reserved storage space, which reduces storage overhead and improves update performance.

[0116] Taking the EC-(6,3) encoding method as an example, the file is divided into 6 raw data blocks and 3 raw parity blocks, stored on 9 storage nodes, i.e., k=6, m=3, n=9; let s represent the size of a block, |d l | represents the total amount of data updated in the l-th iteration, and t represents the number of updates;

[0117] like Figure 3 As shown, taking the number of faulty nodes w=2 as an example, where w<m; if data blocks d1 and d2 fail, the method provided in this embodiment reads the parity update increments on d3, d4, d5, d6, p1, p2, and p3 in parallel to repair the faulty data blocks. The total amount of parity data to be read is... Traditional methods require reading parity update increments from d3, d4, d5, d6, p1, p2, and p1, p2 to repair faulty data blocks. The total amount of parity data to be read is... The method provided in this embodiment reduces the time required for data repair compared to traditional methods. The reduced data read volume accelerated the repair speed and reduced repair costs;

[0118] like Figure 4 As shown, taking the number of faulty nodes w=3 as an example, w=m; if data blocks d1, d2, and d3 fail, the method provided in this embodiment reads d4, d5, d6, p1, p2, p3, and 1 / 3 of the parity update increment of each parity block in parallel to repair the faulty data blocks. The total amount of parity data that needs to be read is... Traditional methods require reading d4, d5, d6, p1, p2, p3, and all parity update increments for each parity block to repair the faulty data block. The total amount of parity data to be read is... The method provided in this embodiment reduces the time required for data repair compared to traditional methods. The reduced data read volume accelerated the repair process and reduced repair costs.

[0119] Example 3

[0120] This embodiment provides a system for reducing the overhead of erasure coding data update storage and repair, such as... Figure 5 Shown, including:

[0121] The information acquisition module is used to obtain file information of stored files from the erasure coding storage system;

[0122] The feature optimization module is used to perform feature extraction operations on the file information of the stored files to obtain the optimal classification features;

[0123] The model training module is used to assign different weights to the optimal classification features, input the optimal classification features and their corresponding weights into the existing file type classification model for training, and obtain the trained file type classification model.

[0124] The file classification module uses a trained file type classification model to classify newly written files into the erasure coding storage system into read-only files and non-read-only files.

[0125] The data storage module is used to divide each file into several raw data blocks and raw parity blocks during the data storage process, and store them on several storage nodes; for non-read-only files, storage space is reserved after each parity block; for read-only files, no storage space is reserved after each parity block.

[0126] The data update module is used during the data update process. For non-read-only files, new data blocks are generated to overwrite the original data blocks. The parity update increment is calculated based on the new data blocks and stored in the reserved storage space after each original parity block. Read-only files are not updated.

[0127] The data repair module is used during the data repair process. When any storage node fails, it calculates the minimum parity update increment required for repair, reads it in parallel on the remaining non-faulty storage nodes, and repairs the faulty node.

[0128] The specific operation of the data storage module is as follows:

[0129] During data storage, the EC-(k,m) encoding method is used to divide each file into k original data blocks and m original parity blocks, which are stored on n storage nodes, where n = k + m. For non-read-only files, storage space is reserved after each parity block on the corresponding storage node; for read-only files, no storage space is reserved after each parity block on the corresponding storage node.

[0130] The specific operation of the data update module is as follows:

[0131] During the data update process, a new data block d′ is generated in the non-read-only file. i Using the new data block d′ i Corresponding to the original data block d i , i = 0, 1, ..., k;

[0132] Calculate the data update increment:

[0133] Δd i =d′ i -d i

[0134] Calculate the parity update increment:

[0135] Δp j =α i，j ×Δd i

[0136] Where, Δp j α represents the parity update increment of the j-th parity block. i，j Δd represents the erasure coding coefficients of the j-th parity block for the i-th new data block, j = 0, 1, ..., m; i d′ represents the data update increment of the i-th new data block. i Let d represent the i-th new data block. i This represents the i-th original data block;

[0137] The parity update increment is stored in the reserved storage space after each parity block;

[0138] Read-only files do not undergo data updates.

[0139] The specific operation of the data repair module is as follows:

[0140] During data repair, when w storage nodes fail, w ≤ m;

[0141] For non-read-only files, calculate the minimum parity update increment required for repair;

[0142] When w < m, read kw new data blocks, w original parity blocks and the parity update increment of any remaining original parity block from the non-faulty storage nodes in parallel, perform erasure coding decoding operation, and repair the faulty nodes.

[0143] When w = m, kw new data blocks, w original parity blocks, and the parity update increment of each original parity block are read in parallel from the non-faulty storage nodes. Perform erasure coding decoding operations to repair faulty nodes;

[0144] For read-only files, read all raw data blocks and raw parity blocks from non-faulty storage nodes and repair the faulty nodes.

[0145] The same or similar labels correspond to the same or similar parts;

[0146] The terms used to describe positional relationships in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent.

[0147] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for reducing the overhead of erasure coding data update storage and repair, characterized in that, Including: S1: Obtain the file information of the stored file from the erasure code storage system; S2: Perform a feature extraction operation on the file information of the stored file to obtain the optimal classification features; S3: Assign different weights to the optimal classification features, and input the optimal classification features and their corresponding weights into an existing file type classification model for training to obtain a trained file type classification model; S4: Use the trained file type classification model to classify the files newly written into the erasure code storage system into read-only files and non-read-only files; S5: During the data storage process, divide each file into several original data blocks and original parity blocks, and store them on several storage nodes; for non-read-only files, reserve storage space after each parity block; for read-only files, do not reserve storage space after each parity block; S6: During the data update process, non-read-only files are updated to generate new data blocks to overwrite the original data blocks; calculate the parity update increment according to the new data blocks, and store it in the storage space reserved after each original parity block; read-only files do not perform data updates; the specific process is as follows: During the data update process, a new data block d' is generated in the non-read-only file. i Utilizing the new data block d' i Corresponding to the original data block d i , i = 0, 1, ..., k, where k represents the number of original data blocks; Calculate the data update increment: Δd i =d′ i -d i Calculate the parity update increment: Δp j =a i,j ×Δd i Where, Δp j α represents the parity update increment of the j-th parity block. i,j Δd represents the erasure coding coefficients of the j-th parity block for the i-th new data block, j = 0, 1, ..., m, where m represents the number of original parity blocks; i d' represents the data update increment of the i-th new data block. i Let d represent the i-th new data block. i This represents the i-th original data block; Store the parity update increment in the storage space reserved after each parity block; Read-only files do not perform data updates; S7: During the data repair process, when any storage node fails, calculate the minimum parity update increment required for repair, read it in parallel on the remaining non-failed storage nodes, and repair the failed node; the specific process is as follows: During the data repair process, when w storage nodes fail, w ≤ m; For non-read-only files, calculate the minimum parity update increment required for repair; When w < m, read k - w new data blocks, w original parity blocks, and the parity update increment of any remaining one original parity block on the non-failed storage nodes in parallel, perform erasure code decoding operations, and repair the failed node; When w = m, kw new data blocks, w original parity blocks, and the parity update increment of each original parity block are read in parallel from the non-faulty storage nodes. Perform erasure coding decoding operations to repair faulty nodes; For read-only files, read all the original data blocks and original parity blocks on the non-failed storage nodes to repair the failed node.

2. The method for reducing the overhead of erasure coding data update storage and repair according to claim 1, characterized in that, The file information of the stored file includes IP address, timestamp, file size, user ID, user group, file type, file access time, and file modification time.

3. The method for reducing the overhead of erasure coding data update storage and repair according to claim 1, characterized in that, The specific method of step S2 is as follows: S2.1: Extract features from the file information of the stored files to obtain the corresponding classification features f. l The initial feature set F = {f1, f2, ..., f} is formed. l ,…,f n }, l=1,…,n; S2.2: Use the information gain method to filter the initial feature set, and select the classification feature f with the largest information gain in the initial feature set F. j Construct the first round of optimized feature set F 1 ={f j }; S2.3: In the initial feature set F, remove the classification feature f with the largest information gain. j ; Select the classification feature f with the highest information gain from the remaining classification features. k Add the features to the first round of optimization feature set to obtain the second round of optimization feature set F. 2 ={f j ,f k }; S2.4: Compare the first round of optimized feature set F 1 With the second round of feature set optimization F 2 The information gain effect, if the first round of optimization of feature set F 1 Superior to the second round of feature set optimization F 2 Then the first round of optimized feature set F 1 If the set of classification features is the optimal set, then the classification features in it are the optimal classification features; otherwise, proceed to step S2.

5. S2.5: Repeat steps S2.3 - S2.4 until the optimized feature set in the later round is no longer better than the optimized feature set in the previous round, and the corresponding optimized feature set in the previous round is used as the optimal classification feature set, and the classification features therein are the optimal classification features.

4. The method for reducing the overhead of erasure coding data update storage and repair according to claim 1, characterized in that, The specific process of step S5 is as follows: During the data storage process, use the EC-(k,m) encoding method to divide each file into k original data blocks and m original parity blocks, and store them on n storage nodes, n = k + m; for non-read-only files, on the corresponding storage nodes, reserve storage space after each parity block; for read-only files, on the corresponding storage nodes, do not reserve storage space after each parity block.

5. A system for reducing the overhead of erasure coding data update storage and repair, characterized in that, Including: An information acquisition module for obtaining the file information of the stored file from the erasure code storage system; A feature optimization module for performing feature extraction operations on the file information of the stored file to obtain optimal classification features; A model training module for assigning different weights to the optimal classification features, inputting the optimal classification features and their corresponding weights into an existing file type classification model for training, and obtaining a trained file type classification model; A file classification module for classifying the files newly written into the erasure code storage system into read-only files and non-read-only files by using the trained file type classification model; A data storage module for dividing each file into several original data blocks and original parity check blocks during the data storage process and storing them on several storage nodes; for non-read-only files, reserved storage spaces are provided after each parity check block; for read-only files, no reserved storage spaces are provided after each parity check block; A data update module for, during the data update process, non-read-only files are updated to generate new data blocks to overwrite the original data blocks; calculating the parity check update increment according to the new data blocks and storing it in the reserved storage space after each original parity check block; read-only files are not updated with data; the specific operations are as follows: During the data update process, the non-read-only file generates a new data block d. ' i Using the new data block d ' i Corresponding to the original data block d i , i = 0, 1, ..., k, where k represents the number of original data blocks; Calculating the data update increment: Δd i =d' i -d i Calculating the parity check update increment: Δp j =a i,j ×Δd i Where, Δp j α represents the parity update increment of the j-th parity block. i,j Δd represents the erasure coding coefficients of the j-th parity block for the i-th new data block, j = 0, 1, ..., m, where m represents the number of original parity blocks; i d' represents the data update increment of the i-th new data block. i Let d represent the i-th new data block. i This represents the i-th original data block; Storing the parity check update increment in the reserved storage space after each parity check block; Read-only files are not updated with data; A data repair module for, during the data repair process, when any storage node fails, calculating the minimum parity check update increment required for repair, reading in parallel on the remaining non-failed storage nodes, and repairing the failed node; the specific operations are as follows: During the data repair process, when w storage nodes fail, w ≤ m; For non-read-only files, calculating the minimum parity check update increment required for repair; When w < m, reading in parallel k - w new data blocks, w original parity check blocks, and the parity check update increment of any remaining one original parity check block on the non-failed storage nodes, performing erasure code decoding operations, and repairing the failed node; When w = m, kw new data blocks, w original parity blocks, and the parity update increment of each original parity block are read in parallel from the non-faulty storage nodes. Perform erasure coding decoding operations to repair faulty nodes; For read-only files, reading all the original data blocks and original parity check blocks on the non-failed storage nodes and repairing the failed node.

6. The system for reducing the overhead of erasure coding data update storage and repair according to claim 5, characterized in that, The specific operations of the data storage module are as follows: During the data storage process, each file is divided into k original data blocks and m original parity check blocks by using the EC-(k,m) encoding method and stored on n storage nodes, n = k + m; for non-read-only files, on the corresponding storage nodes, reserved storage spaces are provided after each parity check block; for read-only files, on the corresponding storage nodes, no reserved storage spaces are provided after each parity check block.

Images