Data storage method, device and equipment of extended capacity erasure code storage system and medium

Abstract

This invention discloses a data storage method, apparatus, device, and medium for an expanded erasure coding storage system. The method includes: splitting a file to be stored into a first number of candidate fragment files based on a preset erasure coding structure; wherein each candidate fragment file includes a second number of data fragment files and a third number of verification fragment files; determining a fourth number of target fragment files from the candidate fragment files; wherein the fourth number is determined based on the third number, and the fourth number is greater than or equal to 1; storing the target fragment files in the expanded storage node, and storing the remaining fragment files in the original storage node. By adopting the technical solution of this application, after expanding the erasure coding storage system with new nodes, data balancing among the nodes can be quickly performed without compromising data integrity and erasure redundancy, without consuming additional system resources, and without affecting real-time storage services.

Description

Technical Field

[0001] This invention relates to the field of data storage technology for expanded erasure coding storage systems, and more particularly to a data storage method, apparatus, device, and medium for expanded erasure coding storage systems. Background Technology

[0002] As the video surveillance industry continues to develop, the amount of data collected by video surveillance is also increasing, putting increasing pressure on video surveillance cloud storage systems. The industry often addresses this storage pressure by adding new storage nodes; however, this often impacts the data balance of erasure coding storage systems.

[0003] In traditional methods, when an erasure coding storage system expands by adding a new node, it calculates the capacity of the new node and the data capacity already stored in the old nodes, and rewrites a portion of the data from each old node into the new node to achieve data balance among the nodes.

[0004] However, when writing to a new node, if all the fragments of the same file on different nodes need to be migrated to the same node, the redundancy of the erasure coding storage system will be lost. In addition, migrating the stored data to the new node is not part of the real-time storage service and may affect the normal real-time storage service. Summary of the Invention

[0005] This invention provides a data storage method, apparatus, device, and medium for expanding an erasure coding storage system, in order to solve the problem of quickly balancing data after the expansion of the erasure coding storage system without destroying the redundant distribution of stored files.

[0006] According to one aspect of the present invention, a data storage method for an extended erasure coding storage system is provided, the method comprising:

[0007] Based on the preset erasure coding structure, the file to be stored is split into a first number of candidate fragment files; among them, the candidate fragment files include a second number of data fragment files and a third number of verification fragment files, and the first number is the number of original storage nodes in the erasure coding storage system before expansion;

[0008] A fourth number of target fragment files are determined from the candidate fragment files; wherein the fourth number is determined based on the third number, and the fourth number is greater than or equal to 1;

[0009] The target shard file is stored in the expanded storage node, and the remaining shard files are stored in the original storage node; wherein the number of shard files stored in each original storage node is less than or equal to 1.

[0010] According to another aspect of the present invention, a data storage device for an extended erasure coding storage system is provided, the device comprising:

[0011] The file splitting module is used to split the file to be stored into a first number of candidate fragment files based on a preset erasure coding structure;

[0012] Among them, the candidate shard file includes a second number of data shard files and a third number of verification shard files, and the first number is the number of original storage nodes in the erasure coding storage system before expansion;

[0013] The file determination module is used to determine a fourth number of target fragment files from the candidate fragment files; wherein the fourth number is determined based on the third number, and the fourth number is greater than or equal to 1;

[0014] The file storage module is used to store the target fragment file in the expanded storage node and store the remaining fragment files in the original storage node;

[0015] In this case, the number of shard files stored in each original storage node is less than or equal to 1.

[0016] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0017] At least one processor; and

[0018] A memory that is communicatively connected to at least one processor; wherein,

[0019] The memory stores a computer program that can be executed by at least one processor, such that the at least one processor is able to execute the data storage method of the extended erasure coding storage system according to any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores computer instructions for causing a processor to execute and implement the data storage method of the extended erasure coding storage system of any embodiment of the present invention.

[0021] By pre-setting an erasure coding structure, the file to be stored is split into a first number of candidate fragments. This ensures that storing the file does not place a high operational burden on the storage system, thereby improving the stability of the storage system. By determining a fourth number of target fragments from the candidate fragments, the erasure coding redundancy of the storage system is guaranteed during storage. By storing the target fragments in the expanded storage nodes and storing the remaining fragments in the original storage nodes, the error correction redundancy of the storage system is maintained even after expansion. Through these steps, after expanding the erasure coding storage system with new nodes, data balancing can be quickly performed across nodes without compromising data integrity and erasure redundancy, without consuming additional system resources or affecting real-time storage services.

[0022] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of a data storage method for an expanded erasure coding storage system according to Embodiment 1 of the present invention;

[0025] Figure 2 This is a schematic diagram of a file node distribution structure applicable to an embodiment of the present invention;

[0026] Figure 3 This is a schematic diagram of the structure of file node distribution after expansion and erasure according to an embodiment of the present invention;

[0027] Figure 4 This is a flowchart of another data storage method for an expanded erasure coding storage system according to Embodiment 2 of the present invention;

[0028] Figure 5 This is a schematic diagram of the data storage device of an expanded erasure coding storage system according to Embodiment 3 of the present invention;

[0029] Figure 6 This is a schematic diagram of the structure of an electronic device that implements the data storage method of the expanded erasure coding storage system according to an embodiment of the present invention. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "candidate," "target," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] Example 1

[0033] Figure 1 This is a flowchart illustrating a data storage method for expanding an erasure coding storage system, as provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where, during the expansion of an erasure coding storage system, data across nodes is rebalanced without additional system resources being consumed or affecting normal storage services. This method can be executed by the data storage device of the expanded erasure coding storage system. This data storage device can be implemented in hardware and / or software and can be configured in an electronic device with data processing capabilities. Figure 1 As shown, the method includes:

[0034] S110. Based on the preset erasure coding structure, the file to be stored is split into a first number of candidate fragment files.

[0035] The candidate shard file includes a second number of data shard files and a third number of verification shard files, with the first number being the number of original storage nodes in the erasure coding storage system before expansion.

[0036] Erasure coding can be achieved by splitting a file into multiple data fragments and verification fragments, and storing these fragments in different storage nodes to ensure file redundancy and integrity. The file to be stored is the file to be stored in a storage system using erasure coding. The first quantity can be the number of candidate fragments from which the file to be stored will be split, which is equal to the number of original storage nodes in the erasure coding storage system; this invention does not impose a limitation on this number.

[0037] Figure 2 This is a schematic diagram of a file node distribution structure applicable to an embodiment of the present invention. See also: Figure 2 The erasure coding structure in the diagram contains six old nodes: storage node 1, storage node 2, storage node 3, storage node 4, storage node 5, and storage node 6, as well as one expanded storage node 7. Figure 1 On the left, File1, File2, File3, File4, and File5 represent file 1, file 2, file 3, file 4, and file 5, respectively.

[0038] When uploading a file to be stored to a storage system using erasure coding, the data processing module splits the received file into a first set of candidate fragments based on the number of erasure coding nodes. Due to the characteristics of the erasure coding storage system, the candidate fragments after splitting the file are divided into two categories: data fragments and checksum fragments. Data fragments can be used to store partial data of the file to be stored, while checksum fragments are used to ensure the integrity of the split file. If a data fragment is lost, the data is recovered based on the checksum fragments to ensure the integrity of the file content.

[0039] The first quantity equals the number of erasure coding structure nodes before expansion. The first quantity of candidate shard files contains the second quantity of data shard files and the third quantity of check shard files. The first quantity equals the sum of the second and third quantities. Taking File1 as an example, since the original storage system contains six nodes, File1 will be divided into four files with the same amount of data. These four files will then be checked to obtain two File1 check shard files. The data shard files are File1-3, File1-4, File1-5, and File1-6, and the check shard files are File1-1 and File1-2.

[0040] By pre-setting an erasure coding structure, the file to be stored is split into a first number of candidate fragment files, so that storing the file to be stored will not put high operating pressure on the storage system, thereby improving the stability of the storage system.

[0041] S120. Determine a fourth number of target fragment files from the candidate fragment files.

[0042] The fourth quantity is determined based on the third quantity, and the fourth quantity is greater than or equal to 1.

[0043] The target fragment file can be a file that will be stored on the expanded storage nodes of the erasure coding storage system. The fourth quantity can be the number of files that will be stored on the expanded storage nodes of the erasure coding storage system after the same file to be stored is split into candidate fragment files.

[0044] After the file to be stored is split into multiple candidate fragment files and a second number of data fragment files and a third number of verification fragment files are obtained, the second number of data fragment files and the third number of verification fragment files will be selected to finally determine the fourth number of target fragment files.

[0045] In order not to destroy the erasure redundancy of the erasure coding storage system, when selecting target shard files, the fourth quantity will be determined based on the third quantity, selecting at least one verification shard file, and the fourth quantity will be greater than or equal to 1. This ensures that when storing candidate shard files corresponding to the same file to be stored, the number of shard files stored on a single original node is less than or equal to the number of shard files stored on a single expansion node, thus speeding up data balancing.

[0046] By determining a fourth number of target fragment files from the candidate fragment files, the redundancy of the erasure coding storage system is not compromised during storage.

[0047] Optionally, a fourth number of target fragment files are determined from the candidate fragment files, including:

[0048] The target fragment file is formed by determining at most one data fragment file and at least one verification fragment file from the candidate fragment files.

[0049] Because erasure coding storage systems have erasure redundancy, a single storage node cannot contain more than one data fragment file derived from the same file to be stored. If more than one data fragment file is stored, a node failure will affect the data integrity of the corresponding file. However, storing multiple parity fragment files of the same file to be stored on the same storage node does not affect the erasure redundancy of the erasure coding storage system. This is because even if the node fails, causing all parity fragment files to be lost, the data integrity of the corresponding file will not be affected.

[0050] Therefore, after the file to be stored is split into candidate fragments, at most one data fragment and at least one parity fragment are selected as target fragments and stored on the expanded storage nodes of the erasure coding storage system. This ensures that the erasure coding redundancy of the erasure coding storage system is not compromised during storage. For example, when the erasure coding structure is 4+1, a parity fragment is selected as the target fragment from the candidate fragments. Since the second quantity is 4, if the target fragment is one data fragment and one parity fragment, the number of data fragments stored on the original storage node will be 3. If the expanded node fails, the file to be stored cannot be recovered using the remaining three data fragments. Therefore, the number of target fragments needs to be determined based on the third quantity to ensure that if the fragments stored on the expanded node fail, the file can still be recovered using the remaining fragments of the file to be stored.

[0051] Optional, the fourth quantity is less than or equal to the third quantity.

[0052] Because erasure coding storage systems have erasure redundancy, after a file to be stored is split into candidate fragments, at most one data fragment and at least one parity fragment are selected as the target fragment. Therefore, the size of the fourth quantity will be less than or equal to the size of the third quantity, meaning the size of the fourth quantity must be less than the total number of data fragments. This avoids storing all parity fragments and one data fragment on a single node, which would make file recovery difficult if that node fails. For example, in an erasure coding structure of 4+2 (i.e., the second quantity is 4, the third quantity is 2, and the fourth quantity is 2), the target fragment can be one data fragment plus one parity fragment, or two parity fragments.

[0053] S130. Store the target fragment file in the expanded storage node, and store the remaining fragment files in the original storage node.

[0054] In this system, each original storage node stores one or more fragment files. The remaining fragment files can be the candidate fragment files from the file to be stored, excluding the target fragment file. The original storage nodes can be the storage nodes that the erasure coding storage system had before expansion.

[0055] After the target fragment file is determined from the candidate fragment files, it is stored on the expanded storage nodes of the erasure coding storage system. The remaining fragment files are stored on the original storage nodes, ensuring that at most one candidate fragment file is stored on each original storage node. This guarantees that the storage system still has error correction redundancy even after the erasure coding storage system is expanded.

[0056] Figure 3This is a schematic diagram illustrating the structure of file node distribution after expansion and erasure according to an embodiment of the present invention. See also... Figure 3 In the diagram, File7 is the file to be stored. File7 is split into File7-1, File7-2, File7-3, and File7-4, where File7-1, File7-2, File7-3, and File7-4 are data fragments of File7. The expanded storage node in the diagram stores File7-5 and File7-6, where File7-5 and File7-6 are checksum fragments of File7.

[0057] File7 is split and verified to obtain File7-1, File7-2, File7-3, File7-4, File7-5, and File7-6. After determining that the target split files are File7-5 and File7-6, File7-5 and File7-6 are stored in the expanded storage node 7. The remaining File7-1, File7-2, File7-3, and File7-4 will be stored in a separate storage node, so that the number of candidate split files of File7 stored in the original storage node is less than or equal to 1.

[0058] If there are two expansion storage nodes, then two target shard files are selected according to the fourth quantity requirement, and the two target shard files are stored on the two expansion storage nodes respectively. For example, based on the above example, the target shard file is one data shard file and one parity shard file, that is, one data shard file and one parity shard file are stored on each expansion storage node respectively.

[0059] In one alternative approach, steps A1-A2 are included before storing the remaining fragment files in the original storage node:

[0060] Step A1: Determine the historical parity fragment files in the original storage node and sort them in ascending order according to storage time.

[0061] Step A2: Delete the fifth number of historical verification fragment files that are sorted first.

[0062] The fifth quantity is determined based on the fourth quantity.

[0063] Historical checksum files can be checksum files that were already stored in the original storage nodes before the expansion. The fifth quantity can be the number of historical checksum files to be deleted, where the value of the fifth quantity is less than or equal to the fourth quantity.

[0064] For early files stored using erasure coding storage systems, deleting the corresponding checksum files for these early files does not significantly impact data integrity. Furthermore, the checksum files for newly stored files in the erasure coding storage system ensure redundancy after node expansion. "Early files" refers to files stored in the erasure coding storage system at an earlier time. In some areas using erasure coding storage systems, early stored files are periodically deleted to prevent the amount of stored data from exceeding the system's capacity. For example, the retention period for video files in surveillance cloud storage is typically 3 months. Files with longer storage times are prioritized for full-overwrite deletion, while shorter files are retained for a longer period. To minimize the impact of deleting historical checksum files during the balancing process, before balancing begins, user directories are sorted by retention period in days. The oldest 30% (90-63 days ago) of historical checksum files are prioritized for deletion, followed by 30-60% (62-36 days ago), and the last 35 days are deleted.

[0065] Therefore, before storing the remaining fragment files in the original storage node, the historical verification fragment files are sorted in ascending order according to their storage time, so that the historical verification fragment files with earlier storage times are placed in the earlier positions.

[0066] Before storing the remaining fragment files to the original storage node, historical parity fragment files will be deleted in sorted order. To ensure redundancy in the erasure coding storage system after node expansion, the number of historical parity fragment files deleted will be equal to or less than the fourth number.

[0067] By ensuring the integrity of old data and the redundancy of new data, the system accelerates the data growth rate on expanded storage nodes while slowing down the data growth rate on original storage nodes, thus achieving faster data balancing. Furthermore, by incorporating tiered processing based on video file retention periods, the risk of data loss due to the deletion of historical verification files is reduced.

[0068] By pre-setting an erasure coding structure, the file to be stored is split into a first set of candidate fragments. This ensures that storing the file does not place a high operational burden on the storage system, thereby improving the stability of the storage system. By determining a fourth set of target fragments from the candidate fragments, the erasure coding redundancy of the storage system is guaranteed during storage. By storing the target fragments in the expanded storage nodes and storing the remaining fragments in the original storage nodes, the error correction redundancy of the storage system is maintained even after expansion. Through these steps, after expanding the erasure coding storage system with new nodes, data balancing across nodes can be quickly achieved without compromising data integrity and erasure redundancy, without consuming additional system resources or affecting real-time business operations.

[0069] Example 2

[0070] Figure 4 This is a flowchart of another data storage method for an expanded erasure coding storage system provided in Embodiment 2 of the present invention. This embodiment further optimizes the process of storing the remaining fragment files in the original storage node, based on the aforementioned embodiments. For example... Figure 4 As shown, the method includes:

[0071] S210. Based on the preset erasure coding structure, the file to be stored is split into a first number of candidate fragment files.

[0072] The candidate shard file includes a second number of data shard files and a third number of verification shard files, with the first number being the number of original storage nodes in the erasure coding storage system before expansion.

[0073] S220. Determine a fourth number of target fragment files from the candidate fragment files.

[0074] The fourth quantity is determined based on the third quantity, and the fourth quantity is greater than 1.

[0075] S230. Store the target fragment file in the expanded storage node, determine the number of fragment files already stored in the original storage node, and sort the original storage node in ascending order according to the number of fragment files already stored.

[0076] Existing fragment files can be fragment files that are already stored on the original storage node.

[0077] After determining the fourth number of target fragment files, these fourth number of target fragment files will be stored in the expansion storage nodes of the erasure coding storage system. The original storage nodes in the erasure coding storage system will then be sorted in ascending order based on the number of fragment files already stored, thereby identifying the original storage nodes with the fewest stored files in the current erasure coding storage system and reducing uneven growth rates in the usage capacity of the original storage nodes.

[0078] S240. Select the sixth number of target original storage nodes in the sorted list, and store the remaining fragment files in the target original storage nodes respectively.

[0079] The sixth quantity is equal to the difference between the first and fourth quantities.

[0080] To ensure that the redundancy and integrity of the pre-stored files are not compromised after the remaining fragment files are stored, the remaining fragment files need to be stored in different original storage nodes. In order to minimize the impact on the data balance of the erasure coding storage system and to maintain the even distribution of newly added data in the original storage nodes, the original storage nodes should be selected from those with a smaller number of files stored in the current erasure coding storage system.

[0081] Therefore, after identifying the original storage nodes with the fewest files in the current erasure storage system, the remaining fragment files will be stored on these original storage nodes. The sixth number of the original storage nodes is the data of the remaining fragment files, which is the number of candidate fragment files minus the number of target fragment files.

[0082] In one alternative approach, after storing the target fragment file in the expanded storage node and storing the remaining fragment files in the original storage node, steps B1-B2 are further included:

[0083] Step B1: Determine whether the storage cluster consisting of the original storage node and the expanded storage node has balanced data.

[0084] Step B2: If the data is balanced, the candidate shard files obtained from splitting the files to be stored will be evenly stored on all storage nodes.

[0085] Since the storage method when the storage quantity of each node in the erasure coding storage system reaches a balance is different from that when it is balanced, it is necessary to determine whether the original storage node and the expanded storage node have reached a balance after storing the file to be stored once.

[0086] After each storage of a file to be stored, the data processing module counts the number of files stored on each node in the current erasure coding storage system and records it as an array [n1, n2, n3, ..., ni], where ni represents the number of files on the i-th storage node in the erasure coding storage system. After multiple storages of files to be stored, the data in the recorded array will tend to be of similar size. When the difference between the largest and smallest data in the array is less than a preset value, it is considered that the storage cluster composed of the original storage node and the expanded storage node has achieved data balance.

[0087] Once data balance is achieved, the system is switched back to normal storage mode, where each storage node stores one shard file.

[0088] Optionally, determine whether the storage cluster consisting of the original storage nodes and the expanded storage nodes has data balance, including:

[0089] Based on whether the current balancing round has reached the balancing round threshold, it is determined whether the storage cluster composed of the original storage nodes and the expanded storage nodes is data balanced; wherein, the balancing round threshold is determined based on the node capacity, first number, second number and third number of a single original storage node before expansion.

[0090] In addition to step B1 above, the method for determining whether the storage cluster consisting of the original storage node and the expanded storage node has achieved data balance also includes:

[0091] The current balance round can be the number of files currently stored and waiting to be stored after expansion. The balance round threshold can be calculated to be the minimum balance round required for erasure coding storage nodes to achieve data balance after expansion, or it can be determined by empirical values.

[0092] When an erasure coding storage system expands to include new nodes but has not yet stored any new files to be stored, the number of rounds in which the erasure coding storage system achieves data balance is calculated based on the number of files already stored in the original storage nodes and the number of new nodes added. Here, the round can be the number of files that need to be stored, denoted by T.

[0093] Let the number of original storage nodes be S, the number of files stored in the original storage nodes be F, the number of data shards to be split into is N, and the number of verification shards is M. In order to speed up the data balancing process, the expansion node writes M shards each time, and the original storage node stores N shards respectively. Then the formula for calculating the number of storage node balancing rounds is: M*T=int(F+(((MN)*T) / S)), where int means rounding up.

[0094] For example, see Figure 2If the number of data fragments N to be split into is 4, and the number of parity fragments M is 2, then MN = 2. The number of original storage nodes S is 6, and the number of fragments stored on each original storage node F is 5. Therefore, according to the calculation formula, T = 3. For example... Figure 2 As shown, when the balancing round reaches 3, the storage cluster consisting of the original storage node and the expanded storage node achieves data balancing.

[0095] By sorting the original storage nodes in ascending order based on the number of existing fragment files, the storage node selection process becomes clearer and more organized. By selecting the target original storage node with the fewest existing fragment files, and then storing the remaining fragment files in that target original storage node, the data growth rate of the original storage nodes is kept relatively stable.

[0096] Example 3

[0097] Figure 5 This is a schematic diagram of the data storage device of an expanded erasure coding storage system according to Embodiment 3 of the present invention. This embodiment is applicable to situations where, during the expansion of an erasure coding storage system, data across nodes is rebalanced without additional system resources being consumed or affecting normal storage services. The data storage device of this expanded erasure coding storage system can be implemented in hardware and / or software, and can be configured in an electronic device with data processing capabilities. For example... Figure 5 As shown, the device includes: a file splitting module 310, a file determination module 320, and a file storage module 330. Wherein:

[0098] The file splitting module 310 is used to split the file to be stored into a first number of candidate fragment files based on a preset erasure coding structure;

[0099] Among them, the candidate shard file includes a second number of data shard files and a third number of verification shard files, and the first number is the number of original storage nodes in the erasure coding storage system before expansion;

[0100] The file determination module 320 is used to determine a fourth number of target fragment files from the candidate fragment files; wherein the fourth number is determined based on the third number, and the fourth number is greater than 1;

[0101] File storage module 330 is used to store the target fragment file in the expanded storage node and store the remaining fragment files in the original storage node;

[0102] In this case, the number of shard files stored in each original storage node is less than or equal to 1.

[0103] Based on the above embodiments, optionally, the file determination module 320 is specifically used for:

[0104] The target fragment file is formed by determining at most one data fragment file and at least one verification fragment file from the candidate fragment files.

[0105] Optional, the fourth quantity is less than or equal to the third quantity.

[0106] Based on the above embodiments, optionally, before the file storage module 330, the following further includes:

[0107] The ascending sorting module is used to determine the historical parity fragment files in the original storage node and sort the historical parity fragment files in ascending order according to their storage time.

[0108] The configuration file verification module is used to delete the fifth number of historical verification fragment files that are sorted first; the fifth number is determined based on the fourth number.

[0109] Based on the above embodiments, optionally, the file storage module 330 includes:

[0110] The ascending sorting unit is used to determine the number of existing fragment files in the original storage node and sort the original storage node in ascending order based on the number of existing fragment files.

[0111] The file storage unit is used to select the sixth number of target original storage nodes in the sorted order and store the remaining fragment files in the target original storage nodes respectively; wherein, the sixth number is equal to the difference between the first number and the fourth number.

[0112] Based on the above embodiments, optionally, after the file storage module 330, the following component is also included:

[0113] The data balance judgment module is used to determine whether the storage cluster consisting of the original storage nodes and the expanded storage nodes has data balance;

[0114] The file uniform storage module is used to evenly store the candidate shard files obtained by splitting the subsequent files to be stored across all storage nodes if the data is balanced.

[0115] Based on the above embodiments, optionally, the data balance judgment module is specifically used for:

[0116] Based on whether the current balancing round has reached the balancing round threshold, it is determined whether the storage cluster composed of the original storage nodes and the expanded storage nodes is data balanced; wherein, the balancing round threshold is determined based on the node capacity, first number, second number and third number of a single original storage node before expansion.

[0117] The data storage device of the expanded erasure coding storage system provided in the embodiments of the present invention can execute the data storage method of the expanded erasure coding storage system provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0118] The acquisition, storage, use, and processing of data in this application comply with relevant national laws and regulations and do not violate public order and good morals.

[0119] Example 4

[0120] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0121] Figure 6 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0122] like Figure 6 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0123] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0124] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as data storage in an erasure coding memory system.

[0125] In some embodiments, the data storage of the method-expanded erasure coding storage system may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the data storage of the method-expanded erasure coding storage system described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the data storage of the method-expanded erasure coding storage system by any other suitable means (e.g., by means of firmware).

[0126] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0127] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0128] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0129] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0130] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0131] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0132] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0133] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data storage method for an expanded erasure coding storage system, characterized in that, include: Based on a preset erasure coding structure, the file to be stored is split into a first number of candidate fragment files; wherein, the candidate fragment files include a second number of data fragment files and a third number of verification fragment files, and the first number is the number of original storage nodes in the erasure coding storage system before expansion; A fourth number of target fragment files are determined from the candidate fragment files; wherein the fourth number is determined based on the third number, and the fourth number is greater than or equal to 1; The target fragment file is stored in the expanded storage node, and the remaining fragment files are stored in the original storage node; wherein the number of fragment files stored in each original storage node is less than or equal to 1. Among them, determining a fourth number of target fragment files from the candidate fragment files includes: The target fragment file is formed by determining at most one data fragment file and at least one verification fragment file from the candidate fragment files.

2. The method according to claim 1, characterized in that, The fourth quantity is less than or equal to the third quantity.

3. The method according to claim 1, characterized in that, Before storing the remaining fragment files in the original storage node, the method further includes: Identify the historical verification fragment files in the original storage node, and sort the historical verification fragment files in ascending order according to their storage time; Delete the fifth number of historical verification fragment files that are ranked first; wherein the fifth number is determined based on the fourth number.

4. The method according to claim 3, characterized in that, Store the remaining fragment files in the original storage node, including: Determine the number of existing fragmented files in the original storage node, and sort the original storage node in ascending order based on the number of existing fragmented files; Select the sixth number of target original storage nodes in the sorted order, and store the remaining fragment files in the target original storage nodes respectively; wherein the sixth number is equal to the difference between the first number and the fourth number.

5. The method according to claim 1, characterized in that, After storing the target fragment file in the expanded storage node and storing the remaining fragment files in the original storage node, the method further includes: Determine whether the storage cluster composed of the original storage node and the expanded storage node is data-balanced; If the data is balanced, the candidate shard files obtained from splitting the files to be stored will be evenly stored across all storage nodes.

6. The method according to claim 5, characterized in that, Determining whether the storage cluster composed of the original storage node and the expanded storage node is data-balanced includes: Based on whether the current balancing round has reached the balancing round threshold, it is determined whether the storage cluster composed of the original storage node and the expanded storage node is data balanced; wherein, the balancing round threshold is determined based on the node capacity of a single original storage node before expansion, the first number, the second number, and the third number.

7. A data storage device for an expanded erasure coding storage system, characterized in that, include: The file splitting module is used to split the file to be stored into a first number of candidate fragment files based on a preset erasure coding structure; The candidate fragment file includes a second number of data fragment files and a third number of verification fragment files, where the first number is the original number of storage nodes in the erasure coding storage system before expansion. The file determination module is used to determine a fourth number of target fragment files from the candidate fragment files; wherein the fourth number is determined based on the third number, and the fourth number is greater than or equal to 1; The file storage module is used to store the target fragment file in the expanded storage node and store the remaining fragment files in the original storage node; In this case, the number of fragment files stored in each original storage node is less than or equal to 1; The file determination module is specifically used for: The target fragment file is formed by determining at most one data fragment file and at least one verification fragment file from the candidate fragment files.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the data storage method of the extended erasure coding storage system according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the data storage method of the extended erasure coding storage system as described in any one of claims 1-6.

Images