Method and system for implementing foldable fault isolation domain in distributed storage system

Abstract

The present invention provides a method and system for implementing a foldable fault isolation domain in a distributed storage system, comprising the following steps: obtaining metadata of each storage unit, generating an original weight according to the metadata; generating a new weight for each storage unit based on the original weight and allocation strategy, and generating a view according to the metadata and the new weight, wherein the view includes a top layer, a second top layer, and a bottom layer; judging whether the storage space of the top layer of the view satisfies the maximum redundancy of the data cluster; if so, assigning the data cluster to the top layer; otherwise, using a folding mechanism to fold the data cluster; if the redundancy of the data cluster after folding is less than its minimum redundancy, the allocation fails, and the redundancy judgment is performed again by switching to the second top layer of the view until the judgment result is satisfied. The beneficial effects of the present invention: while ensuring data security, it can reduce unnecessary storage overhead, and ensure data availability and accessibility even if some storage nodes fail.

Description

Technical Field

[0001] The present invention belongs to the field of computer systems, and in particular to a method and system for implementing a foldable fault isolation domain in a distributed storage system. Background Art

[0002] In distributed storage systems, fault isolation domain (FaultDomain) is a key concept. This concept has become a key indicator for measuring the reliability and fault tolerance of distributed storage systems. Its basic feature is a logical area divided in the system, in which the impact of faults or errors is limited to the area to avoid causing wider damage to the entire system. By dividing the system into multiple isolation domains, the propagation of faults can be limited. For example, if a server in a domain goes down, it will not affect the operation of other domains. There is usually a data redundancy strategy within each fault isolation domain or between domains to ensure that data is still available even if some nodes in the domain fail. Fault isolation domains need to limit resource allocation, try to avoid excessive consumption of resources in a certain domain, and when it is unavoidable that some domain resources are exhausted first, the resources of other domains are still available.

[0003] In distributed storage, fault isolation domains, in addition to meeting the above basic requirements, need to consider data randomness, uniformity, security, resource priority, and resource balance when laying out data. Randomness means that different data clusters should be randomly distributed in different fault domains. More fault domains can balance greater access pressure and avoid uneven pressure caused by access similar to boot storms; data uniformity means that more fault domains should be used as much as possible to distribute data between different redundant copies of the same data cluster; security means that the same or related redundancy in the same data cluster should not be distributed in the same fault domain to avoid exceeding the redundancy of the same data cluster when fault domains are isolated; resource priority requires that storage resources that meet the priority level be allocated data first, so that excellent resources are fully utilized and the upper threshold of storage system performance is improved; resource balance means using storage resources as comprehensively as possible to increase the capacity threshold of the storage system. Resource priority and resource balance are two conflicting characteristics, which should be configured and used according to the characteristics of storage resources in the storage system and the needs of the business system.

[0004] In a storage system that supports fault isolation domains, when some domains are isolated and the remaining domains distribute data, if the maximum redundancy of the data cluster cannot be met, but the minimum redundancy can be met, whether it should be used as a temporary compromise solution to destroy the fault isolation domain security allocation principle to meet the normal operation of the business system is a new challenge to the implementation of fault isolation domains in distributed storage systems. Summary of the invention

[0005] In view of this, the present invention aims to propose a method and system for implementing a foldable fault isolation domain in a distributed storage system, in order to solve at least one of the above-mentioned technical problems.

[0006] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0007] A first aspect of the present invention provides a method for implementing a foldable fault isolation domain in a distributed storage system, comprising the following steps:

[0008] Obtain metadata of each storage unit, and generate an original weight according to the metadata;

[0009] Generate a new weight for each storage unit based on the original weight and the allocation strategy, and generate a view according to the metadata and the new weight, wherein the view includes a top layer, a second top layer, and a bottom layer;

[0010] Determine whether the storage space at the top level of the view meets the maximum redundancy of the data cluster; if so, allocate the data cluster to the top level; otherwise, use the folding mechanism to fold the data cluster;

[0011] If the redundancy of the data cluster after folding is less than its minimum redundancy, the allocation fails and the view is switched to the second top layer to perform redundancy judgment again until the judgment result is satisfied.

[0012] Furthermore, the process of generating the original weight according to the metadata includes:

[0013] A weight vector W is generated according to the available capacity C and the performance value P of the storage unit, and the original weight is calculated: J(W)=P(C / 1TB); wherein the performance value is a value assigned to different types of storage devices, including:

[0014] nvme_ssd=2, sata_ssd=1, sata_hdd=0.3, sas_hdd=0.7;

[0015] The metadata is obtained by sending a heartbeat request to the storage unit.

[0016] Furthermore, the process of generating a new weight for each storage unit based on the original weight and the allocation strategy includes:

[0017] The allocation strategy is to constrain the sum of the weights of all storage units ΣW i =1;

[0018] Establish the Lagrangian function L(W,λ)=J(W)-λ(ΣW i -1), where λ is the Lagrange multiplier;

[0019] Set the Lagrangian function of each storage unit with respect to its weight Wi The partial derivative of ;

[0020] According to the allocation strategy, Lagrangian function, and partial derivatives, W is calculated. i value, and use W i Replace the original weights.

[0021] Furthermore, the maximum redundancy of the data cluster is specifically the maximum amount of redundant data lost in the data cluster when the read and write functions are normal;

[0022] The method for obtaining the maximum redundancy of the data cluster includes a copy method, and the copy method includes a double copy mode and a triple copy mode;

[0023] In the dual-copy mode, the maximum redundancy of the data cluster is 1, and the data cluster cannot be folded; in the triple-copy mode, the maximum redundancy of the data cluster is 2.

[0024] Furthermore, the method for obtaining the maximum redundancy of the data cluster also includes an erasure code method, and the erasure code method includes three types of allocation methods;

[0025] Under the first type of allocation method, the maximum redundancy of the data cluster is 1; under the second type of allocation method, the maximum redundancy of the data cluster is 2, and the two redundant data are located in the same fault domain; under the third type of allocation method, the maximum redundancy of the data cluster is 2, and the two redundant data are located in different fault domains, and the data cluster cannot be folded.

[0026] Furthermore, in the process of using the folding mechanism, the total number of allocable data clusters doubles each time the folding is performed; in the process of allocating data clusters, each element of the current level is allocated according to the number of folding times until the allocation satisfies the total number of data cluster copies.

[0027] A second aspect of the present invention provides a system for implementing a foldable fault isolation domain in a distributed storage system, comprising:

[0028] A metadata manager, which records the metadata and original weights of the storage unit and sends the metadata to the view generator after receiving a heartbeat request;

[0029] A weight generator, wherein the weight generator generates original weights using a weight calculation strategy, wherein generation parameters of the original weights include metadata and calculation factors;

[0030] A view generator, wherein the view generator sends a heartbeat request to the metadata manager and receives metadata and original weights returned by the metadata manager; generates new weights based on the original weights and allocation strategy, combined with the capacity, load, and performance of the device, and generates a view of the distributed storage system based on the new weights and metadata;

[0031] a data distributor, which distributes data clusters according to the views generated by the view generator, processes the data clusters using a folding mechanism when data cluster allocation is blocked, and completely distributes the processed data clusters to the storage unit;

[0032] A controller stores weight calculation strategies and calculation factors, and sends the stored contents to the weight generator upon receiving a request from the weight generator; and displays the current weights and views upon receiving an external query request.

[0033] Compared with the prior art, the method and system for implementing foldable fault isolation domains in a distributed storage system according to the present invention have the following beneficial effects:

[0034] It can reduce unnecessary storage overhead while ensuring data security, and ensure data availability and accessibility even when some storage nodes fail.

[0035] Avoiding some nodes being overloaded while others are idle reduces the need for manual intervention by administrators and simplifies storage management;

[0036] Provide dynamic resource adjustment for distributed storage systems, improve adaptability to different workloads, allocate data and tasks to storage units with the best performance, reduce unnecessary resource usage, reduce energy consumption and hardware investment, and thus reduce operating costs;

[0037] Able to adapt to different types of data and diverse data access requirements. Distributed storage systems can flexibly expand to support more data and higher loads while maintaining high performance and high reliability to meet growing business needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0039] Figure 1 A schematic flow chart of a method for implementing a foldable fault isolation domain in a distributed storage system according to an embodiment of the present invention; DETAILED DESCRIPTION

[0040] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0041] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", and the like may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0042] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood by specific circumstances.

[0043] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0044] A method for implementing a foldable fault isolation domain in a distributed storage system comprises the following steps:

[0045] S1. Obtain metadata of each storage unit and generate original weights based on the metadata.

[0046] The process of generating the original weight according to the metadata in step S1 includes:

[0047] A weight vector W is generated according to the available capacity C and the performance value P of the storage unit, and the original weight is calculated: J(W)=P(C / 1TB); wherein the performance value is a value assigned to different types of storage devices.

[0048] The above J(W) is used as an objective function to calculate a performance index or cost index according to the value of the weight vector W. In the above formula, C / 1TB can be used as the capacity coefficient and P can be used as the calculation method, that is, the original weight can be understood as minimizing the weighted capacity. Therefore, the calculation of the original weight can be done without the participation of the weight vector W.

[0049] During system operation, the calculation of J(W) can be based on different algorithms, such as maximizing weighted performance There is also a load balancing mode .

[0050] In some embodiments, the performance values ​​include:

[0051] nvme_ssd=2, sata_ssd=1, sata_hdd=0.3, sas_hdd=0.7; metadata is obtained by sending heartbeat requests to the storage unit.

[0052] In other embodiments, the above-mentioned different types of storage devices are assigned different performance values ​​according to business needs, for example: HDD=1.6, SATASSD=10, NVMeSSD=60, SSHD=6, NAS=20, SAN=100, cloud storage=10; at the same time, metadata can also be implemented by adding a corresponding request code in the message header when sending a request to the storage unit.

[0053] S2. Generate a new weight for each storage unit based on the original weight and allocation strategy, and generate a view based on the metadata and the new weight. The view includes the top layer, the second top layer, and the bottom layer.

[0054] The process of generating a new weight for each storage unit based on the original weight and the allocation strategy in step S2 includes:

[0055] The allocation strategy is to constrain the sum of the weights of all storage units ΣW i =1;

[0056] Establish the Lagrangian function L(W,λ)=J(W)-λ(ΣW i -1), where λ is the Lagrangian multiplier; where the Lagrangian function is used to generate the constraint ΣW i =1, the Lagrange multiplier λ is used to ensure that the weight vector W satisfies the constraints;

[0057] Set the Lagrangian function of each storage unit with respect to its weight W i The partial derivative of ;

[0058] According to the allocation strategy, Lagrangian function, and partial derivatives, W is calculated. i value, and use W i Replace the original weights.

[0059] S3. Determine whether the storage space at the top level of the view satisfies the maximum redundancy of the data cluster; if so, allocate the data cluster to the top level; otherwise, use a folding mechanism to fold the data cluster; wherein the data cluster is a storage unit for logical data, and is composed of data and redundant data arranged according to the offset sequence in the logical volume.

[0060] In the above content, the specific meaning of "folding" is:

[0061] The number of data slices that can be placed in a fault domain. Data slices refer to the real data and their redundant data in the data cluster. For example, there are two copies. There is only one copy of the real data, but two identical copies are stored. These two copies are data slices. The 4+2 data cluster has 4 copies of real data and 2 copies of verification data, totaling 6 data slices. If one data slice is placed in each fault domain to satisfy the data cluster, then the folding degree is 1. When the data cluster allows folding, each fault domain stores two or more data slices. At this time, the folding degree is 2 or more.

[0062] The maximum redundancy of the data cluster in step S3 is specifically the maximum amount of redundant data lost in the data cluster when the read and write functions are normal;

[0063] In some embodiments, a method for obtaining the maximum redundancy of a data cluster includes a copy method, and the copy method includes a dual copy mode and a triple copy mode; in the dual copy mode, the maximum redundancy of the data cluster is 1, and the data cluster cannot be folded; in the triple copy mode, the maximum redundancy of the data cluster is 2.

[0064] The copy mode can also be a multi-copy mode. In the multi-copy mode, the maximum redundancy of the data cluster changes according to the number of copies.

[0065] In some other embodiments, the method for obtaining the maximum redundancy of the data cluster in step S3 includes an erasure code method, and the erasure code method includes three types of allocation methods;

[0066] Under the first type of allocation method, the maximum redundancy of the data cluster is 1; under the second type of allocation method, the maximum redundancy of the data cluster is 2, and the two redundant data are located in the same fault domain; under the third type of allocation method, the maximum redundancy of the data cluster is 2, and the two redundant data are located in different fault domains, and the data cluster cannot be folded.

[0067] In some embodiments, common ratios in erasure codes are 2+1, 4+2:1, 4+2:2, 8+2:1, and 8+2:2, among which 2+1 belongs to the first allocation method, 4+2:1 and 8+2:1 belong to the second allocation method, and 4+2:2 and 8+2:2 belong to the third allocation method.

[0068] In other embodiments, the erasure code ratio is a non-standard ratio. In this case, the erasure code ratio is adjusted accordingly according to business needs, and is divided into different types of allocation methods based on the adjusted results, and different allocation methods have different maximum redundancies.

[0069] It is worth noting that all data clusters belonging to the same logical volume have the same redundancy mode.

[0070] In step S3, during the use of the folding mechanism, the total number of allocable data clusters doubles each time the fold is performed; during the allocation of data clusters, each element of the current level is allocated according to the number of folds until the allocation satisfies the total number of data cluster copies.

[0071] The specific process of the above allocation according to the number of folding times is as follows:

[0072] Assuming that the data cluster is 4+2:1, and only 5 fault domains are available, the fault domain folding degree needs to be 2. Therefore, each fault domain needs to first take out 1 storage unit for allocation, and the remaining 1 needs to be filtered again from the 5 storage units selected for the second time from the 5 fault domains to retain one, and finally the storage of six data slices is gathered.

[0073] S4. If the redundancy of the data cluster after folding is less than its minimum redundancy, the allocation fails, and the view is switched to the second top layer to perform redundancy judgment again until the judgment result is satisfied.

[0074] In some embodiments, the specific execution process of step S3-4 is as follows: the default folding number of the top layer of the view is 1. When all the top layers of the view are allocated according to the folding number and still cannot meet the total number of data clusters to be allocated (data + redundancy), if the data cluster supports folding, then allocating again in all top-level domains according to the folding number 2;

[0075] If it still cannot be met at this time, continue to fold within the minimum redundancy range. When the allocation exceeds the total number required, the appropriate domain in the fault domain selected after folding is used. For example, when the data cluster adopts 4+2:1 and there are only 5 top-level fault domains, 5 fault domains are selected in the first folding and 5 fault domains are selected in the second folding. At this time, 10 domains need to be selected from the domains selected in the second folding to form 6 domains to meet the total number of data clusters.

[0076] A system for implementing a foldable fault isolation domain in a distributed storage system, comprising:

[0077] The metadata manager records the metadata and original weight of the storage unit and sends the metadata to the view generator after receiving the heartbeat request;

[0078] Weight generator, which uses weight calculation strategy to generate original weights. The generation parameters of original weights include metadata and calculation factors.

[0079] View generator: The view generator sends a heartbeat request to the metadata manager and receives the metadata and original weights returned by the metadata manager. It generates new weights based on the original weights and allocation strategy, combined with the capacity, load, and performance of the device, and generates a view of the distributed storage system based on the new weights and metadata.

[0080] Data distributor, the data distributor distributes data clusters according to the views generated by the view generator, uses the folding mechanism to process the data clusters when the data cluster allocation is blocked, and distributes the processed data clusters completely to the storage unit;

[0081] The controller stores the weight calculation strategy and calculation factors, and sends the stored content to the weight generator when receiving a request from the weight generator; when receiving an external query request, it displays the current weight and view.

[0082] In some embodiments, a system for implementing collapsible fault isolation domains in a distributed storage system includes five parts;

[0083] 1) Device metadata manager, attached to the smallest storage unit (e.g. hard disk), records the location of the storage unit, statistics on I / O latency, original weight coefficients, and other data;

[0084] 2) The fault domain controller exposes the management interface to the outside world, including query, policy selection, weight system modification, etc.

[0085] 3) Weight generator, which generates storage device weights based on the real-time data and original setting data of the storage device;

[0086] 4) The view generator is responsible for sending heartbeats to all storage devices to determine whether to isolate the abnormal domain. Through the heartbeat reply, it collects device metadata and generates a domain view consisting of all available storage devices based on the device weights returned by the weight generator.

[0087] 5) Data distributor, on the same domain view, determines whether folding is required based on the maximum or minimum redundancy of the data cluster, and finally distributes the data to the storage device.

[0088] Device Metadata Manager: Responsible for recording detailed information of each storage unit, such as location, I / O latency and original weight coefficient. Location information mainly records the node and cabinet where the storage device is located, which is important information for marking the domain to which the storage device belongs. I / O latency measures the actual performance of the storage device by collecting actual performance data during statistical access, and determines whether the storage device is abnormal and the priority of the storage device in the data allocation process. The original weight coefficient is the basis for the weight generator to calculate the storage device. This coefficient can be modified by the controller, and it is also the basis for the implementation of the capacity strategy. The basic data provided by the metadata manager is used by the weight generator to generate weights, which reflect the performance and reliability of the storage device. When receiving the heartbeat request from the view generator, the device metadata is packaged and sent to the view generator through the heartbeat reply process.

[0089] View Generator: Responsible for sending heartbeat requests, collecting and processing metadata from the device metadata manager, and generating views of the current storage system. The view generator periodically sends heartbeat signals to all online storage devices to check their health status and availability. After receiving the reply from the device metadata manager, the view generator processes this information, including key indicators such as the device's I / O latency, storage capacity, and load. The view generator uses the permission calculator based on the original weight coefficient and allocation strategy to bridge factors such as the device's capacity, load, and performance, and generates a new weight for each device. This weight reflects the current applicability and performance level of the device. The view generator then generates a comprehensive current view based on the collected metadata and calculated weights. The unit storage device is the bottom layer of the view, the node is the second top layer of the view, and the cabinet description is the top layer, which together constructs the view. This view includes all devices in the online domain and their weights. The generated view serves as the basis for the data distributor, which determines the best location for data storage based on this view.

[0090] Weight generator: configures multiple weight calculation models in a plug-in manner, such as weighted average, regression analysis, decision tree or neural network models; can use the weight calculation strategy configured by the controller, comprehensively consider (storage device) parameters, including but not limited to: performance indicators, load information, capacity parameters and other information recorded by the storage device metadata, as well as weight decay factors, quality of service (QoS) requirements, priority parameters, data access modes and other calculation factors recorded by the controller that can be manually intervened as parameters for weight calculation, dynamically adapt to changes in the storage system, set the optimal weight configuration for the storage unit, and send it back to the view generator to generate a unified view.

[0091] Data distributor: accepts data cluster description data, among which the most important description data is the maximum and minimum redundancy. The data distributor needs to try to distribute in the unified view according to the above two data. The specific distribution process is: according to the total number of data cluster data and data cluster redundant data, traverse from the top level of the view layer by layer, and the starting point of the traversal is the end point of the previous distribution plus 1. If the top view cannot meet the maximum redundancy, it is necessary to consider folding one by one. Each time the folding is performed, the total number of data cluster copies that can be allocated is 2*2. At this time, the entire view level needs to be fully equipped, that is, each element of this level is allocated according to the number of foldings, and so on until the allocation meets the total number of data cluster copies. At this time, if the folding is greater than the minimum redundancy, the allocation fails; starting from the second top level, the folding number is used as the total number of allocations, and the above steps are repeated until the physical storage device is allocated. The selection of physical devices adopts smooth weight selection. At this point, a list of physical unit devices is obtained. Following the principle of "folding requires full configuration", the number of devices in this list must be greater than or equal to the total number of data cluster copies. It is necessary to select the storage device with the total number of data cluster copies after comparing the weight sorting to complete this allocation process.

[0092] Controller: As a manual intervention interface, it stores the default original data, including the strategy of the dynamic weight model, basic weights, and storage device indicator priorities. When modified through the API, the controller packages all parameters and sends them to the weight generator, and persists these changes in this module. The controller provides a query interface that can display the dynamic weight of the current storage device, the latest view, etc. When performance indicators are used as the priority, it will cause an imbalance in the use of storage capacity of storage devices in the domain. At this time, the controller can modify the storage device indicator priority through the controller to alleviate this situation.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

[0094] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for implementing a foldable fault isolation domain in a distributed storage system, characterized in that: The steps include: Obtain metadata of each storage unit, and generate an original weight according to the metadata; Generate a new weight for each storage unit based on the original weight and the allocation strategy, and generate a view based on the metadata and the new weight, the view including the top layer, the second top layer, and the bottom layer; wherein the allocation strategy is to constrain the sum of the weights of all storage units ΣW i =1; Determine whether the storage space at the top level of the view meets the maximum redundancy of the data cluster; if so, allocate the data cluster to the top level; otherwise, use the folding mechanism to fold the data cluster; If the redundancy of the data cluster after folding is less than its minimum redundancy, the allocation fails, and the view is switched to the second top layer to perform redundancy judgment again until the judgment result is satisfied; The process of generating a new weight for each storage unit based on the original weight and the allocation strategy includes: establishing a Lagrangian function L(W,λ)=J(W)-λ(ΣW i -1), where λ is the Lagrangian multiplier; set the Lagrangian function of each storage unit with respect to its weight W i The partial derivative ∂L / ∂W i =0; According to the allocation strategy, Lagrangian function, and partial derivatives, W is calculated i value, and use W i Replace the original weights.

2. The method for implementing a foldable fault isolation domain in a distributed storage system according to claim 1, characterized in that: The process of generating raw weights from metadata includes: A weight vector W is generated according to the available capacity C and the performance value P of the storage unit, and the original weight is calculated: J(W)=P(C / 1TB); wherein the performance value is a value assigned to different types of storage devices; The metadata is obtained by sending a heartbeat request to the storage unit.

3. The method for implementing a foldable fault isolation domain in a distributed storage system according to claim 1, characterized in that: The maximum redundancy of the data cluster is specifically the maximum amount of redundant data lost in the data cluster when the read and write functions are normal.

4. The method for implementing a foldable fault isolation domain in a distributed storage system according to claim 1, characterized in that: The method for obtaining the maximum redundancy of the data cluster includes a copy method; The copy mode includes a dual copy mode and a triple copy mode: in the dual copy mode, the maximum redundancy of the data cluster is 1, and the data cluster cannot be folded; in the triple copy mode, the maximum redundancy of the data cluster is 2.

5. The method for implementing a foldable fault isolation domain in a distributed storage system according to claim 1, characterized in that: The method for obtaining the maximum redundancy of the data cluster includes an erasure code method; The erasure code method includes three types of allocation methods: under the first type of allocation method, the maximum redundancy of the data cluster is 1; under the second type of allocation method, the maximum redundancy of the data cluster is 2, and the two redundant data are located in the same fault domain; under the third type of allocation method, the maximum redundancy of the data cluster is 2, and the two redundant data are located in different fault domains, and the data cluster cannot be folded.

6. The method for implementing a foldable fault isolation domain in a distributed storage system according to claim 1, characterized in that: During the use of the folding mechanism, the total number of allocatable data clusters doubles each time the folding is performed.

7. The method for implementing a foldable fault isolation domain in a distributed storage system according to claim 1, characterized in that: In the process of allocating data clusters, each element of this level is allocated according to the number of folding times until the allocation meets the total number of data cluster copies.

8. A system for implementing foldable fault isolation domains in a distributed storage system, characterized in that: include: A metadata manager, which records the metadata and original weights of the storage unit and sends the metadata to the view generator after receiving a heartbeat request; A weight generator, wherein the weight generator generates original weights using a weight calculation strategy, wherein generation parameters of the original weights include metadata and calculation factors; A view generator, wherein the view generator sends a heartbeat request to the metadata manager and receives metadata and original weights returned by the metadata manager; Generate new weights based on the original weights and allocation strategy, combined with the capacity, load, and performance of the device, and generate a view of the distributed storage system based on the new weights and metadata; The allocation strategy is to constrain the sum of the weights of all storage units ΣW i =1; The process of generating the new weight includes: Establish the Lagrangian function L(W,λ)=J(W)-λ(ΣW i -1), where λ is the Lagrangian multiplier; set the Lagrangian function of each storage unit with respect to its weight W i The partial derivative ∂L / ∂W i =0; According to the allocation strategy, Lagrangian function, and partial derivatives, W is calculated i value, and use W i Replace the original weights; a data distributor, which distributes data clusters according to the views generated by the view generator, processes the data clusters using a folding mechanism when data cluster allocation is blocked, and completely distributes the processed data clusters to the storage unit; A controller stores weight calculation strategies and calculation factors, and sends the stored contents to the weight generator upon receiving a request from the weight generator; and displays the current weights and views upon receiving an external query request.

Images