Data storage method, device, equipment and storage medium
By dynamically adjusting the EC encoding method and determining the storage unit to be used, the problem of taking into account both data security and space utilization when the storage nodes in the cloud storage cluster are offline is solved, and higher space utilization and data dispersion are achieved.
Patent Information
- Application Number
- CN202410868266.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-01
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2044-07-01
AI Technical Summary
In a cloud storage cluster, when the storage node is offline, if the data security and space utilization are to be ensured, the existing technology is difficult to take into account both, resulting in a reduction in the space utilization of the storage node.
By detecting the number of online storage nodes in the target storage cluster, when the number is less than the preset threshold, the erasure code (EC) encoding method is dynamically adjusted, the appropriate storage units to be used are determined, and the current EC encoding is determined according to the predetermined security level. Method to ensure data storage is distributed.
While ensuring that the predetermined security level remains unchanged, the space utilization of data storage is improved, and the security and dispersion of data are taken into account, reducing the cost of data recovery.
Smart Images

Figure CN118409714B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of cloud storage technology, and in particular to a data storage method, apparatus, device and storage medium. Background Art
[0002] With the development of Internet technology, cloud storage services are being applied in more and more fields. For example, cloud storage services can be applied in the field of video surveillance, where the captured video data is uploaded to storage nodes (cloud servers). Video data is generally divided into multiple data blocks and stored in multiple storage nodes in a cloud storage cluster.
[0003] In the related art, in order to ensure the security of data storage, EC (Erasure coding) coding is used to protect data blocks, so that when some data blocks are lost, the lost data blocks can be restored. EC is a data coding fault-tolerant technology, which can divide the original data into N data blocks and then encode them. The obtained encoded data includes N+M data blocks, where N and M are both positive integers. The N+M data blocks are dispersedly stored in N+M storage nodes, and each storage node is used to store a data block. The N data blocks are original data blocks, and the M data blocks are check blocks. The number of check blocks M determines the security level of data protection. That is, when data blocks are lost in the N+M data blocks, and the lost data blocks are no more than M, the remaining data blocks in the N+M data blocks can be used to restore the lost data blocks.
[0004] However, when a storage node in the cluster is offline and the number of online storage nodes is less than the number of nodes required by EC encoding, if the security level is guaranteed to remain unchanged, that is, the number of check blocks remains unchanged, the space utilization of the storage node will be reduced, that is, the value of N / (N+M) will be reduced.
[0005] Therefore, how to balance data security and space utilization has become a technical problem that needs to be solved urgently. Summary of the invention
[0006] The purpose of the embodiments of the present application is to provide a data storage method, device, equipment and storage medium to balance data security and space utilization. The specific technical solution is as follows:
[0007] In a first aspect, an embodiment of the present application provides a data storage method, the method comprising:
[0008] When a change is detected in the online storage nodes in the target storage cluster, the number of online storage nodes is obtained;
[0009] If the number of nodes obtained is less than a first preset threshold, then based on the storage resources in the online storage nodes, determine no less than the first preset threshold number of storage units to be used; wherein the first preset threshold is: the minimum number of storage units required to meet a predetermined space utilization rate and a predetermined security level when storing data in the storage unit based on the erasure code EC encoding method;
[0010] Determine a current EC encoding method based on the obtained number of storage units to be used and a predetermined security level;
[0011] The current data to be stored is dispersedly stored in the obtained storage units to be used according to the current EC encoding method.
[0012] In a second aspect, an embodiment of the present application provides a data storage device, the device comprising:
[0013] A detection module, used for obtaining the number of online storage nodes when detecting a change in online storage nodes in a target storage cluster;
[0014] A first determination module is configured to determine, if the number of nodes obtained is less than a first preset threshold, no less than the first preset threshold number of storage units to be used based on storage resources in online storage nodes; wherein the first preset threshold is: the minimum number of storage units required to meet a predetermined space utilization rate and a predetermined security level when data is stored in the storage unit based on the EC encoding method;
[0015] A second determination module is used to determine the current EC encoding method based on the obtained number of storage units to be used and the predetermined security level;
[0016] The storage module is used to disperse and store the current data to be stored into the obtained storage units to be used according to the current EC encoding method.
[0017] In a third aspect, an embodiment of the present application provides an electronic device, including:
[0018] Memory, used to store computer programs;
[0019] The processor is used to implement any of the data storage methods described above when executing the program stored in the memory.
[0020] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the data storage method described in any one of the above is implemented.
[0021] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes executable instructions. When the executable instructions are executed on a computer, the computer executes any one of the data storage methods described above.
[0022] Beneficial effects of the embodiments of the present application:
[0023] The solution provided by the embodiment of the present application is that if the number of nodes obtained is less than the first preset threshold, then based on the storage resources in the online storage nodes, no less than the first preset threshold storage units to be used are determined. Then, based on the obtained number of storage units to be used and the predetermined security level, the current EC encoding method is determined, and the current data to be stored is dispersedly stored in the obtained storage units to be used according to the current EC encoding method. Since the number of storage units to be used determined is not less than the first preset threshold, the first preset threshold is the minimum number of storage units required to meet the predetermined space utilization and the predetermined security level when data is stored in the storage unit based on the EC encoding method, and the current EC encoding method is determined based on the number of storage units to be used and the predetermined security level, therefore, when the number of check blocks required by the predetermined security level remains unchanged, the number of original data blocks supported by the EC encoding method becomes larger, so that the data to be stored is dispersedly stored in the obtained storage units to be used according to the current EC encoding method, which can take into account both data security and space utilization.
[0024] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.
[0026] Figure 1 A flowchart of a data storage method provided in an embodiment of the present application;
[0027] Figure 2 A flowchart for implementing step S102 in the data storage method provided in an embodiment of the present application;
[0028] Figure 3 A flowchart of a specific example of a data storage method provided in an embodiment of the present application;
[0029] Figure 4A flowchart of step S301 in a specific example of a data storage method provided in an embodiment of the present application;
[0030] Figure 5 A schematic diagram of the structure of a data storage device provided in an embodiment of the present application;
[0031] Figure 6 A block diagram of an electronic device for implementing the data storage method provided in an embodiment of the present application. DETAILED DESCRIPTION
[0032] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.
[0033] First, the professional terms involved in the embodiments of the present application are introduced.
[0034] EC: is a data protection and error correction technology, mainly used to improve the reliability of data storage and transmission. It divides the original data into N data blocks and then encodes them to generate N+M data blocks, and stores the N+M data blocks in different storage nodes. Among them, N represents the number of original data blocks, M represents the number of check blocks, and the N+M data blocks constitute the stripe data. In the embodiment of the present application, the EC encoding method represents the number of original data blocks and the number of check blocks obtained by EC encoding. For example, if the EC encoding method is 4+2, it means that the EC encoding obtains 4 original data blocks and 2 check blocks.
[0035] Node-level EC: Stripe data is stored in multiple nodes within the cluster, providing node-level security.
[0036] Disk-level EC: Stripe data is stored on multiple disks in a node to avoid cross-node traffic and provide disk-level security.
[0037] Cluster: A cluster is a group of independent computers that are interconnected by a high-speed network. They form a group and are managed as a single system.
[0038] Domain: A domain is a collection of computers with a security boundary. Computers in the same domain have established a trust relationship with each other. To access other machines within the domain, the permission of the accessed machine is no longer required. Each storage node is added to a different domain, and a corresponding bucket is created for each domain. The data written by the user to the bucket will be stored in the storage node in the corresponding domain, so that data can be isolated by domain. When a domain corresponds to a bucket, this bucket can be referred to as the resource pool of the domain.
[0039] Next, a data storage method provided in an embodiment of the present application is introduced.
[0040] A data storage method provided in an embodiment of the present application is applied to a management node, which can communicate with each storage node in a target storage cluster. The management node can be various electronic devices, such as personal computers, servers, and other devices with data processing capabilities. In addition, it is understood that the data storage method provided in an embodiment of the present application can be implemented by software, hardware, or a combination of software and hardware.
[0041] Among them, a data storage method provided in an embodiment of the present application may include:
[0042] When a change is detected in the online storage nodes in the target storage cluster, the number of online storage nodes is obtained;
[0043] If the number of nodes obtained is less than a first preset threshold, then based on the storage resources in the online storage nodes, determine no less than the first preset threshold number of storage units to be used; the first preset threshold is: the minimum number of storage units required to meet a predetermined space utilization rate and a predetermined security level when storing data in the storage unit based on the EC encoding method;
[0044] Determine a current EC encoding method based on the obtained number of storage units to be used and a predetermined security level;
[0045] The current data to be stored is dispersedly stored in the obtained storage units to be used according to the current EC encoding method.
[0046] The solution provided by the embodiment of the present application is that if the number of nodes obtained is less than the first preset threshold, then based on the storage resources in the online storage nodes, no less than the first preset threshold storage units to be used are determined. Then, based on the obtained number of storage units to be used and the predetermined security level, the current EC encoding method is determined, and the current data to be stored is dispersedly stored in the obtained storage units to be used according to the current EC encoding method. Since the number of storage units to be used determined is not less than the first preset threshold, the first preset threshold is the minimum number of storage units required to meet the predetermined space utilization and the predetermined security level when data is stored in the storage unit based on the EC encoding method, and the current EC encoding method is determined based on the number of storage units to be used and the predetermined security level, therefore, when the number of check blocks required by the predetermined security level remains unchanged, the number of original data blocks supported by the EC encoding method becomes larger, so that the data to be stored is dispersedly stored in the obtained storage units to be used according to the current EC encoding method, which can take into account both data security and space utilization.
[0047] The data storage method provided in the embodiments of the present application is introduced below in conjunction with the accompanying drawings.
[0048] like Figure 1 The data storage method provided in the embodiment of the present application includes steps S101-S104:
[0049] S101, when it is detected that the online storage nodes in the target storage cluster change, the number of online storage nodes is obtained;
[0050] In this embodiment, the target storage cluster may be a cluster composed of storage nodes in any domain, and data in different domains are isolated from each other.
[0051] It is understandable that in actual applications, changes such as nodes going offline or new storage nodes being online may occur in the target storage cluster, causing changes in the online storage nodes in the target storage cluster. Then, there is a scenario where the number of online storage nodes is less than the number of nodes required by EC coding, resulting in the inability to provide corresponding security level protection according to the pre-configured EC coding method when storing data, and the inability to meet the space utilization requirements. Therefore, in this embodiment, when a change in the online storage nodes in the target storage cluster is detected, the number of online storage nodes is first obtained to dynamically adjust the EC coding method according to the obtained number of nodes, thereby taking into account both data security and space utilization.
[0052] Exemplarily, the management node may periodically send heartbeat detection packets to each storage node in the target storage cluster in a broadcast manner, and receive feedback signals from the storage nodes in response to the received heartbeat detection packets, and determine whether the online node has changed by comparing the machine identifier (such as the machine's MAC address, etc.) carried in the currently received feedback signal with the machine identifier carried in the last received feedback signal. For example, if the management node currently receives feedback signal 1 carrying the identifier of node 1 and feedback signal 2 carrying the identifier of node 2, and the last time it received feedback signal 1 carrying the identifier of node 1, feedback signal 2 carrying the identifier of node 2, and feedback signal 3 carrying the identifier of node 3, then node 3 in the target storage cluster is offline, and it is determined that the node has changed.
[0053] In addition, when a node change is detected, the number of feedback signals received for the heartbeat detection packet sent at the same time can be determined as the number of online storage nodes. It should be noted that the embodiment of the present application does not limit the method for obtaining the number of online storage nodes when a change is detected in the online storage nodes in the target storage cluster.
[0054] S102, if the number of nodes obtained is less than a first preset threshold, determining no less than a first preset threshold number of storage units to be used based on storage resources in online storage nodes; wherein the first preset threshold is: the minimum number of storage units required to meet a predetermined space utilization rate and a predetermined security level when storing data in the storage unit based on the EC encoding method;
[0055] It can be understood that, when the number of lost data blocks among the N+M data blocks obtained by the EC encoding method is not more than M, the lost data blocks can be restored by using the remaining data blocks. Therefore, the predetermined security level is related to the number of check blocks M. The larger M is, the more data blocks can be allowed to be lost, and the higher the security level is. The space utilization is the ratio of the number of real data blocks N to the total number of data blocks N+M obtained by EC encoding. Exemplarily, if the predetermined security level is a security level indicating that a maximum of 2 data blocks can be lost, and the predetermined space utilization is 66%, then M in the EC encoding method is 2, and N is not less than 4 at this time. When the EC encoding method is 4+2, that is, N is 4 and M is 2, the number of storage units required is the least, and the first preset threshold is 6 at this time. If the number of nodes obtained is less than 6, if the security level is to be kept unchanged, that is, M is unchanged, then N will decrease, resulting in the space utilization not reaching the predetermined space utilization. For example, if the number of nodes is 5, M is 2 and N is 3, then the space utilization is 60%, which does not reach the predetermined space utilization.
[0056] Exemplarily, it is possible to determine no less than a first preset threshold number of storage units to be used based on the storage resources in all online storage nodes, or to determine no less than a first preset threshold number of storage units to be used based on the storage resources in some online storage nodes, which is reasonable. Among them, the method of determining the storage units to be used for an online storage node may be: virtualizing the storage node to obtain multiple virtual nodes, and using each virtual node as a storage unit to be used, or using each disk in the storage node as a storage unit to be used, which is reasonable. It should be noted that in order to make the layout of the solution clear, the method of determining no less than a first preset threshold number of storage units to be used based on the storage resources in the online storage nodes is described in the following embodiments, and will not be repeated here.
[0057] S103, determining a current EC (erasure code) encoding method based on the obtained number of storage units to be used and a predetermined security level;
[0058] Optionally, in one implementation, based on the obtained number of storage units to be used and a predetermined security level, determining the current EC encoding method includes:
[0059] The minimum number of check blocks required for a predetermined security level is determined as the number of check blocks supported by the current EC encoding method, and the difference between the obtained number of storage units to be used and the minimum number of check blocks required for the predetermined security level is determined as the number of original data blocks supported by the current EC encoding method.
[0060] Exemplarily, if the predetermined security level is a security level indicating that a maximum of 2 data blocks are allowed to be lost, the minimum number of check blocks required by the security level is 2, and M is 2, that is, the number of check blocks supported by the current EC encoding method is 2. If the number of storage units to be used is 10, the difference between the number of storage units to be used and the minimum number of check blocks required by the predetermined security level is 8, and 8 is used to determine the value of N, that is, the number of original data blocks supported by the EC encoding method is 8. At this time, it can be determined that the current encoding method is 8+2.
[0061] It can be understood that the difference between the number of storage units to be used and the minimum number of check blocks required by the predetermined security level is determined as the number of original data blocks supported by the current EC encoding method, so that when subsequent data blocks are stored in each storage unit to be used, they can be stored in a more dispersed manner on each storage unit to be used while improving space utilization, thereby meeting the dispersion requirements for data storage.
[0062] It should be noted that the embodiment of the present application does not limit the method of determining the current EC encoding method based on the obtained number of storage units to be used and the predetermined security level. For example, in another implementation, the minimum number of check blocks required by the predetermined security level can be determined as the number of check blocks supported by the current EC encoding method, and any integer within the numerical range formed by the difference between the first preset threshold and the minimum number of check blocks required by the predetermined security level, and the difference between the number of storage units to be used and the minimum number of check blocks required by the predetermined security level is determined as the number of original data blocks supported by the current EC encoding method. Exemplarily, if the minimum number of check blocks required by the predetermined security level is 2, the first preset threshold is 6, and the number of storage units to be used is 10, then N can be 4, 5, 6, 7 or 8. At this time, the current encoding method can be 4+2, 5+2, 6+2, 7+2, or 8+2, which are all reasonable.
[0063] S104, dispersively storing the current data to be stored in the obtained storage units to be used according to the current EC encoding method.
[0064] In this embodiment, the data to be stored may be video data in security monitoring or other types of data. After the current EC encoding method is determined in step S103, the current data to be stored may be dispersedly stored in the obtained storage units to be used according to the current EC encoding method, that is, the current data to be stored may be EC-encoded according to the current EC encoding method and then dispersedly stored in the obtained storage units to be used.
[0065] Exemplarily, if the current EC encoding mode is 4+2, EC encoding will divide the current data to be stored into 4 original data blocks, and then use the pre-constructed 4+2 row and 4 column encoding matrix to perform a product operation with the 4 original data blocks obtained by the division to encode the current data to be stored, and the obtained encoded data includes 4+2 data blocks. Since the matrix composed of the elements of the first 4 rows in the encoding matrix is a unit matrix, among the 4+2 data blocks included in the encoded data, the 4 data blocks are original data blocks and the 2 data blocks are check blocks.
[0066] For example, if the number of storage units to be used is 6 and the current EC encoding method is 4+2, the data blocks obtained by the EC encoding are dispersedly stored in the 6 storage units to be used, and each storage unit to be used stores a data block. If the number of storage units to be used is 10 and the current EC encoding method is 4+2, 6 storage units to be used can be selected from all storage units to be used, and the data blocks obtained by the EC encoding are dispersedly stored in the 6 storage units to be used.
[0067] It can be understood that, since the number of storage units to be used obtained in step S102 is not less than the first preset threshold value, the first preset threshold value is the minimum number of storage units required to meet the predetermined space utilization and the predetermined security level when storing data in the storage unit based on the EC encoding method, the number of storage units to be used obtained can meet the minimum number required by EC encoding. Since the current EC encoding method is determined based on the number of storage units to be used obtained and the predetermined security level, the number of original data blocks supported by the EC encoding method becomes larger when the number of check blocks required by the predetermined security level remains unchanged. Therefore, according to the current EC encoding method, the data to be stored is dispersedly stored in the obtained storage units to be used, which can improve the space utilization while taking into account the security of the data.
[0068] The solution provided by the embodiment of the present application is that if the number of nodes obtained is less than the first preset threshold, then based on the storage resources in the online storage nodes, no less than the first preset threshold storage units to be used are determined. Then, based on the obtained number of storage units to be used and the predetermined security level, the current EC encoding method is determined, and the current data to be stored is dispersedly stored in the obtained storage units to be used according to the current EC encoding method. Since the number of storage units to be used determined is not less than the first preset threshold, the first preset threshold is the minimum number of storage units required to meet the predetermined space utilization and the predetermined security level when data is stored in the storage unit based on the EC encoding method, and the current EC encoding method is determined based on the number of storage units to be used and the predetermined security level, therefore, when the number of check blocks required by the predetermined security level remains unchanged, the number of original data blocks supported by the EC encoding method becomes larger, so that the data to be stored is dispersedly stored in the obtained storage units to be used according to the current EC encoding method, which can take into account both data security and space utilization.
[0069] Optionally, in another embodiment of the present application, before determining the current EC encoding mode based on the obtained number of storage units to be used and the predetermined security level in the above step S103, the method further includes:
[0070] If the number of nodes obtained is not less than a first preset threshold, each online storage node is taken as a storage unit to be used, and the number of nodes, storage units to be used, is obtained.
[0071] In this embodiment, when the number of online storage nodes is not less than the first preset threshold, each online storage node is used as a storage unit to be used. In this way, when the number of online storage nodes increases, the number of storage units to be used increases. Since the EC encoding method is determined based on the number of storage units to be used and the predetermined security level, the EC encoding method can be dynamically adjusted when the number of storage units to be used increases. For example, if the number of online storage nodes increases from 6 to 8, the EC encoding method can be adjusted from 4+2 to 6+2. In this way, the data blocks obtained by EC encoding are stored in the storage units to be used in a distributed manner, and each storage unit to be used stores a data block. While taking into account the security of the data and the space utilization rate, the decentralization of data storage can be improved. In addition, by distributing the data on each storage node, node-level EC can be achieved.
[0072] It is understandable that the N value in the EC encoding method affects the data recovery cost. The smaller the N value, the smaller the dispersion of data storage, the larger the impact of the failure, and the greater the data recovery cost. Therefore, improving the dispersion of data storage can reduce the data recovery cost.
[0073] It can be seen that through this solution, the decentralization of data storage can be improved on the basis of taking into account data security and space utilization, thereby reducing the cost of data recovery.
[0074] Optionally, in another embodiment of the present application, if the number of nodes acquired in the above step S102 is less than a first preset threshold, then based on the storage resources in the online storage nodes, determining no less than the first preset threshold number of storage units to be used includes:
[0075] If the number of nodes obtained is less than the first preset threshold and not less than the second preset threshold, then based on the storage resources in the online storage nodes, each online storage node is virtualized into a preset number of virtual nodes, and no less than the first preset threshold number of virtual nodes are obtained, which are used as storage units to be used; wherein the second preset threshold represents: the ratio of the first preset threshold to the preset number.
[0076] In this embodiment, the number of virtual nodes that can be virtualized for each storage node can be preset. For example, if the preset number is 2, each online storage node can be virtualized into 2 virtual nodes. Exemplarily, based on the storage resources in the online storage node, each online storage node can be virtualized into a preset number of virtual nodes by: dividing all storage space of each online storage node into a preset number of parts to obtain a preset number of virtual nodes. At this time, the virtualized virtual nodes evenly divide the storage resources of the physical node; or, dividing all disks in each online storage node into a preset number of disk groups, and dividing the storage space in the same disk into the same group to obtain a preset number of virtual nodes. It should be noted that the embodiment of the present application does not limit the virtualization method. For example, all storage space of the storage node can also be divided into a preset number of parts according to different sizes.
[0077] The second preset threshold is the ratio of the first preset threshold to the preset number. For example, if the preset number is 2 and the first preset threshold is 6, the second preset threshold is 3. It should be noted that in actual applications, the preset number can be set by relevant staff based on experience, and the embodiments of the present application are not limited to this.
[0078] It is understandable that if the preset number is 2 and the first preset threshold is 6, when the number of nodes obtained is less than 3, each online storage node is virtualized according to the preset number, and the number of virtual nodes obtained still cannot reach the minimum number of storage units required to meet the predetermined space utilization and the predetermined security level. Therefore, in this embodiment, when the number of nodes obtained is less than the first preset threshold and not less than the second preset threshold, each online storage node is virtualized into a preset number of virtual nodes to obtain a plurality of storage units to be used. For example, if the preset number is 2 and the first preset threshold is 6, when the number of nodes is 5, each online storage node is virtualized into 2 virtual nodes to obtain 10 virtual nodes, which are respectively used as storage units to be used.
[0079] In addition, it should be noted that, in other embodiments, each online storage node can also be virtualized into different numbers of virtual nodes, and each virtual node is used as a storage unit to be used, so as to obtain multiple storage units to be used. Among them, the number of virtual nodes that can be virtualized by each online storage node is positively correlated with the total disk storage space in the node. For example, if there are 12 disks in node A and 24 disks in node B, and the disk sizes of the disks in node A and node B are both 2TB, then node A can be virtualized into 2 virtual nodes and node B can be virtualized into 4 virtual nodes.
[0080] Accordingly, in this embodiment, if Figure 2As shown, the method for implementing the above step S102 also includes steps S201-S202:
[0081] S201, if the number of nodes obtained is less than a second preset threshold, selecting a node to be used from the online storage nodes;
[0082] S202, selecting disks no less than a first preset threshold from the storage resources of the node to be used as storage units to be used.
[0083] In this embodiment, if the number of nodes obtained is less than the second preset threshold, a node to be used can be selected from the online storage nodes. Exemplarily, any node can be selected from the online storage nodes as the node to be used, or the node where the current data to be stored is located can be selected as the node to be used, which is reasonable.
[0084] After selecting the node to be used, multiple disks not less than the first preset threshold are selected from the storage resources of the node to be used as storage units to be used. Exemplarily, if the first preset threshold is 6, the second preset threshold is 3, and the number of nodes obtained is 2, any storage node can be selected from the two storage nodes as the node to be used, and then 8 disks are selected from all the disks of the node to be used as storage units to be used. In this way, when the number of nodes obtained is less than the second preset threshold, the number of storage units required to meet the predetermined space utilization and the predetermined security level can be obtained, so that the current EC encoding method can be adjusted according to the number of storage units to be used, so that when the storage data is stored according to the current EC encoding method, the security of the data and the space utilization can be taken into account. For example, if the number of disks selected from the storage resources of the node to be used is 8, based on the number of storage units to be used and the predetermined security level, the current EC encoding method is determined to be 6+2, and the 8 data blocks obtained by EC encoding are distributed and stored on the 8 disks selected from the node to be used, and each selected disk stores a data block. It should be noted that the embodiment of the present application does not limit the method of selecting no less than the first preset threshold number of disks from all the disks of the node.
[0085] It can be seen that through this solution, the storage resources in the storage node can be divided to obtain the number of unused storage units that meet the predetermined space utilization and the predetermined security level, so that after the EC encoding method is dynamically adjusted according to the number of obtained storage units to be used, the data to be stored is dispersedly stored in the obtained storage units to be used according to the adjusted EC encoding method, which can take into account both data security and space utilization.
[0086] In order to better understand the present solution, the data storage method provided in the embodiment of the present application is described below with reference to a specific example.
[0087] Cloud storage clusters need to support various scenarios such as node offline and capacity expansion. In scenarios such as node offline and online, there are scenarios where the number of online nodes is less than the number of nodes required by EC coding, resulting in the inability to provide protection at the corresponding security level according to the pre-configured EC coding method, and at the same time, it is impossible to provide a high level of space utilization. For example, in a scenario where there are many offline nodes, if the security level is guaranteed to remain unchanged, the number of check blocks will remain unchanged, and the number of original data blocks will be greatly reduced, resulting in a significant reduction in space utilization.
[0088] This example proposes to dynamically adjust the EC encoding method according to the number of online nodes to support both security and space utilization requirements. Figure 3 As shown, the process of implementing this example includes steps S301-S303:
[0089] S301, when the number of online nodes is not less than a first preset threshold, dynamically adjust the EC (erasure code) encoding mode;
[0090] For example, Figure 4 As shown, the process of implementing step S301 includes steps S3011-S3012:
[0091] S3011, count the number of online nodes;
[0092] S3012, dynamically adjust the EC (erasure code) encoding method according to the number of online nodes.
[0093] That is, according to the number of online nodes in the domain, the EC encoding method is dynamically adjusted to ensure that the predetermined security level is achieved while ensuring that the space utilization is not lower than the configured threshold.
[0094] For example, if the minimum number of check blocks required by the predetermined security level is 2, and the threshold of space utilization (corresponding to the predetermined space utilization mentioned above) is 66%, then the first preset threshold is 6. If the number of online nodes is 8 and the EC encoding method is 6+2, when the number of online nodes increases to 10, the EC encoding method is automatically adjusted to 8+2; when the number of online nodes decreases to 6, the EC encoding method is automatically adjusted to 4+2.
[0095] When the number of online nodes is not less than the first preset threshold, combined with the number of online nodes, node-level EC can be achieved on the basis of taking into account data security and space utilization, that is, the data obtained by EC encoding is distributed to different nodes to provide node-level security.
[0096] S302, when the number of online nodes is less than a first preset threshold, switching to virtual node level EC;
[0097] If the number of online nodes is less than the first preset threshold, it will automatically switch to virtual node-level EC. For example, if the space utilization rate is required to be no less than 66% and the EC encoding method is N+2, the first preset threshold is 6. When the number of online nodes is less than 6, it will automatically switch to virtual node-level EC.
[0098] Specifically, each online node is virtualized into multiple virtual nodes (for example, 2), and the virtual nodes evenly share the disks of the storage nodes. For example, in a 4+2 scenario, if there are less than 6 storage nodes online, the EC encoding method cannot be adjusted to 3+2 (the space utilization rate is 60%, which is lower than the threshold configured for space utilization). At this time, each online node is virtualized into 2 virtual nodes (corresponding to the disk group mentioned above). If there are 5 online nodes, 10 virtual nodes can be obtained, and the stripe data obtained by EC encoding is stored in these 10 virtual nodes, which meets both the dispersion requirements and the space utilization.
[0099] In addition, if the number of online nodes increases subsequently, the already stored stripe data will no longer be distributed to multiple nodes for storage.
[0100] S303: When the number of online nodes is less than a second preset threshold, switch to disk-level EC.
[0101] In this example, the second preset threshold is the ratio of the first preset threshold to the number of virtual nodes that can be divided from each online node. If the number of online nodes is less than the second preset threshold, after each online node is divided into a preset number of virtual nodes, the security and space utilization requirements cannot be met, and the disk-level EC is switched at this time. For example, if the preset number is 2, the second preset threshold is 3. When there are only 2 online nodes, it is automatically adjusted to disk-level EC, that is, the stripe data obtained by EC encoding is stored in multiple disks in one node, fully ensuring space utilization. At this time, select an online node (corresponding to the node to be used above). If the number of disks in the online node is 12, the EC encoding method is determined to be 10+2, and the 12 encoded data blocks are stored in a dispersed manner on the 12 disks in the selected online node. In addition, it is worth mentioning that if the security requirements are not high, the EC encoding method can also be determined to be 11+1.
[0102] It can be seen that this solution dynamically adjusts the EC encoding method according to the number of online nodes, and cooperates with node-level EC, virtual node-level EC, disk-level EC and other strategies to maximize data security while ensuring that the space utilization is not lower than the predetermined threshold, thereby achieving a balance between data security and space utilization.
[0103] Corresponding to the above method embodiment, the present application embodiment also provides a data storage device, such as Figure 5 As shown, including:
[0104] The detection module 510 is used to obtain the number of online storage nodes when detecting a change in the online storage nodes in the target storage cluster;
[0105] The first determination module 520 is configured to determine, if the number of nodes obtained is less than a first preset threshold, no less than the first preset threshold number of storage units to be used based on storage resources in online storage nodes; wherein the first preset threshold is the minimum number of storage units required to meet a predetermined space utilization rate and a predetermined security level when storing data in the storage unit based on the EC encoding method;
[0106] A second determination module 530 is used to determine a current EC encoding method based on the obtained number of storage units to be used and a predetermined security level;
[0107] The storage module 540 is used to disperse and store the current data to be stored into the obtained storage units to be used according to the current EC encoding method.
[0108] Optionally, the device further comprises:
[0109] The third determination module is used to execute, before the determination module 530 determines the current EC encoding method based on the obtained number of storage units to be used and the predetermined security level, if the obtained number of nodes is not less than the first preset threshold, each online storage node is regarded as a storage unit to be used, and the number of storage units to be used is obtained.
[0110] Optionally, the first determining module 520 is specifically configured to:
[0111] If the number of nodes obtained is less than the first preset threshold and not less than the second preset threshold, then based on the storage resources in the online storage nodes, each online storage node is virtualized into a preset number of virtual nodes to obtain no less than the first preset threshold number of virtual nodes, which are used as storage units to be used; wherein the second preset threshold represents: the ratio of the first preset threshold to the preset number.
[0112] Optionally, the device further comprises:
[0113] A first selection module, configured to select a node to be used from online storage nodes if the number of nodes obtained is less than the second preset threshold;
[0114] The second selection module is used to select no less than the first preset threshold number of disks from the storage resources of the node to be used as storage units to be used.
[0115] Optionally, the second determining module 530 is specifically configured to:
[0116] The minimum number of check blocks required for a predetermined security level is determined as the number of check blocks supported by the current EC encoding method, and the difference between the obtained number of storage units to be used and the minimum number of check blocks required for the predetermined security level is determined as the number of original data blocks supported by the current EC encoding method.
[0117] In the technical solution of this application, the operations involved in data acquisition, storage, use, processing, transmission, provision and disclosure are all carried out with the user's authorization.
[0118] The present application also provides an electronic device, such as Figure 6 As shown, including:
[0119] Memory 601, used for storing computer programs;
[0120] The processor 602 is used to implement the steps of any of the above data storage methods when executing the program stored in the memory 601.
[0121] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 602, the communication interface, and the memory 601 communicate with each other via the communication bus.
[0122] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0123] The communication interface is used for communication between the above electronic device and other devices.
[0124] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0125] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0126] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above data storage methods are implemented.
[0127] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any of the data storage methods in the above embodiments.
[0128] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.
[0129] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0130] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0131] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.
Claims
1. A data storage method, characterized in that: The method comprises: When a change is detected in the online storage nodes in the target storage cluster, the number of online storage nodes is obtained; If the number of nodes obtained is less than a first preset threshold, then based on the storage resources in the online storage nodes, determine no less than the first preset threshold storage units to be used; wherein the first preset threshold is: when data is stored in the storage unit based on the erasure code EC encoding method, the minimum number of storage units required to meet the predetermined space utilization and the predetermined security level; the space utilization is calculated as follows: N / (N+M); N is the number of original data blocks in the erasure code EC encoding method, and M is the number of check blocks in the erasure code EC encoding method; the security level is the minimum value of the number of check blocks; the first preset threshold is the sum of the number of check blocks corresponding to the predetermined security level and the minimum number of original data blocks that meet the predetermined space utilization; Determine a current EC encoding method based on the obtained number of storage units to be used and a predetermined security level; According to the current EC encoding method, the current data to be stored is dispersedly stored in the obtained storage units to be used; The determining of the current EC encoding method based on the obtained number of storage units to be used and the predetermined security level includes: The minimum number of check blocks required for a predetermined security level is determined as the number of check blocks supported by the current EC encoding method, and any integer within a numerical range formed by a difference between the first preset threshold and the minimum number of check blocks required for the predetermined security level, and a difference between the obtained number of storage units to be used and the minimum number of check blocks required for the predetermined security level is determined as the number of original data blocks supported by the current EC encoding method.
2. The method according to claim 1, characterized in that Before determining the current EC encoding mode based on the obtained number of storage units to be used and the predetermined security level, the method further includes: If the number of nodes obtained is not less than the first preset threshold, each online storage node is used as a storage unit to be used, and the number of storage units to be used is obtained.
3. The method according to claim 1, characterized in that If the number of nodes obtained is less than a first preset threshold, determining no less than the first preset threshold number of storage units to be used based on storage resources in online storage nodes, including: If the number of nodes obtained is less than the first preset threshold and not less than the second preset threshold, then based on the storage resources in the online storage nodes, each online storage node is virtualized into a preset number of virtual nodes to obtain no less than the first preset threshold number of virtual nodes, which are used as storage units to be used; wherein the second preset threshold represents: the ratio of the first preset threshold to the preset number.
4. The method according to claim 3, characterized in that The method further comprises: If the number of nodes obtained is less than the second preset threshold, selecting a node to be used from the online storage nodes; No less than the first preset threshold number of disks are selected from the storage resources of the node to be used as storage units to be used.
5. A data storage device, characterized in that: The device comprises: A detection module, used for obtaining the number of online storage nodes when detecting a change in online storage nodes in a target storage cluster; A first determination module is configured to determine, if the number of nodes obtained is less than a first preset threshold, no less than the first preset threshold number of storage units to be used based on storage resources in online storage nodes; wherein the first preset threshold is: the minimum number of storage units required to meet a predetermined space utilization rate and a predetermined security level when data is stored in a storage unit based on an EC encoding method; the space utilization rate is calculated as follows: N / (N+M); N is the number of original data blocks in the EC encoding method, and M is the number of check blocks in the EC encoding method; the security level is the minimum value of the number of check blocks; the first preset threshold is the sum of the number of check blocks corresponding to the predetermined security level and the minimum number of original data blocks that meet the predetermined space utilization rate; A second determination module is used to determine the current EC encoding method based on the obtained number of storage units to be used and the predetermined security level; A storage module, used for dispersing and storing the current data to be stored into the obtained storage units to be used according to the current EC encoding method; The second determining module is specifically used to: The minimum number of check blocks required for a predetermined security level is determined as the number of check blocks supported by the current EC encoding method, and any integer within a numerical range formed by a difference between the first preset threshold and the minimum number of check blocks required for the predetermined security level, and a difference between the obtained number of storage units to be used and the minimum number of check blocks required for the predetermined security level is determined as the number of original data blocks supported by the current EC encoding method.
6. The device according to claim 5, characterized in that The device also includes: A third determining module is configured to, before the determining module determines the current EC encoding method based on the obtained number of storage units to be used and the predetermined security level, execute if the obtained number of nodes is not less than the first preset threshold, taking each online storage node as a storage unit to be used, to obtain the number of storage units to be used; and / or, The first determining module is specifically configured to: If the number of nodes obtained is less than the first preset threshold and not less than the second preset threshold, then based on the storage resources in the online storage nodes, each online storage node is virtualized into a preset number of virtual nodes, and no less than the first preset threshold number of virtual nodes are obtained, which are respectively used as storage units to be used; wherein the second preset threshold represents: the ratio of the first preset threshold to the preset number; and / or, The device also includes: A first selection module, configured to select a node to be used from online storage nodes if the number of nodes obtained is less than the second preset threshold; The second selection module is used to select no less than the first preset threshold number of disks from the storage resources of the node to be used as storage units to be used.
7. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-4 when executing a program stored in a memory.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.
9. A computer program product, characterized in that The computer program product comprises executable instructions, and when the executable instructions are executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 4.
Citation Information
Patent Citations
Cloud storage system and data deployment method thereof
CN102035884A
Data storage method and storage device
CN110874185A
Data protection storage method and device and equipment
CN113590042A
User data stable storage device based on cloud computing
CN114924694A
Data processing method and device, equipment and storage medium
CN115514775A
