A method and apparatus for labeling distributed cluster super-redundant objects

Abstract

The application discloses a marking method and equipment of a distributed cluster super redundancy object, comprising: judging whether super redundancy occurs according to a minimum redundancy strategy of a cluster in the case of storage medium state updating of the distributed cluster; in the case of super redundancy, updating a cluster pool map while publishing a fault map, persisting based on the fault map, and the fault map being cleared only in the case of exiting super redundancy; and calling an object-to-storage medium routing algorithm based on the fault map to determine the object of which super redundancy occurs. The embodiment of the application realizes accurate identification of super redundancy by snapshotting the cluster storage medium state, and effectively reduces the demand for memory resources.

Description

Technical Field

[0001] This application relates to the field of distributed storage technology, and in particular to a method and device for marking super-redundant objects in a distributed cluster. Background Technology

[0002] Ensuring data consistency is a fundamental requirement of storage systems. In storage cluster systems, if it is not possible to accurately identify overly redundant state objects, it will lead to clients obtaining unreliable data and the inability to guarantee data consistency between redundant replicas, ultimately resulting in data inconsistency issues.

[0003] like Figure 1 As shown, traditional distributed storage systems place excessively redundant information on placement groups (PGs). User data objects are assigned to a PG based on a hash algorithm, and the PG is then mapped to specific storage media according to a routing algorithm and a predefined redundancy strategy. The storage service also manages cluster objects at the PG level.

[0004] The PG layer records the historical changes of the mapped storage media. When there is a change in the state of a server / storage media in the cluster, all PGs trigger a check of the historical change records to confirm whether any records indicate a complete failure of the mapped media, meaning the PG is in an over-redundant state. PGs in an over-redundant state will update their state accordingly, while PGs in a non-over-redundant state may have a new change record added for subsequent checks.

[0005] In existing solutions, each PG generates an instance on each storage medium it maps to, and all PG information resides in memory and is persisted. This primarily results in significant memory consumption. To reduce memory and network bandwidth consumption and facilitate PG information maintenance, PG information is typically avoided from being synchronized to the management plane or clients. The drawback is that clients cannot directly perceive whether the data object they need to operate on belongs to a PG in an overly redundant state—they only obtain this information after the server returns a corresponding error code, resulting in a delay in the client's perception of overly redundant objects. Summary of the Invention

[0006] This application provides a method and device for marking super-redundant objects in a distributed cluster, avoiding the introduction of PGs and overcoming the technical problems caused by their large resource requirements. It takes snapshots of the cluster storage medium status to accurately identify super-redundancy and effectively reduces the demand for memory resources.

[0007] This application provides a method for marking super-redundant objects in a distributed cluster, including:

[0008] In the event of a storage medium status update in the distributed cluster, it is determined whether excessive redundancy has occurred based on the cluster's minimum redundancy strategy.

[0009] In the event of excessive redundancy, a fault map is published while updating the cluster pool map, and the fault map is persisted only when the excessive redundancy is removed.

[0010] Based on the fault map, a routing algorithm from the object to the storage medium is invoked to identify the objects that have become excessively redundant.

[0011] Optionally, based on the fault map, an object-to-storage medium routing algorithm is invoked to determine the objects that have become excessively redundant, including:

[0012] Based on the fault map, the routing algorithm from object to storage medium is invoked. If all storage media mapped to any object are in a fault state, then it is determined that any object is over-redundant.

[0013] After determining that any of the objects has excessive redundancy, mark the excessive redundancy state of any of the objects.

[0014] Optionally, after marking the super-redundant state of any of the objects, operations on any of the objects are prohibited.

[0015] Optionally, the fault map may also be cleared if the status of the faulty storage medium associated with it in the fault map returns to normal.

[0016] This application also proposes a computer device, including a processor configured as follows:

[0017] In the event of a storage medium status update in the distributed cluster, it is determined whether excessive redundancy has occurred based on the cluster's minimum redundancy strategy.

[0018] In the event of excessive redundancy, a fault map is published while updating the cluster pool map, and the fault map is persisted only when the excessive redundancy is removed.

[0019] Based on the fault map, a routing algorithm from the object to the storage medium is invoked to identify the objects that have become excessively redundant.

[0020] Optionally, the processor is configured to:

[0021] Based on the fault map, the routing algorithm from object to storage medium is invoked. If all storage media mapped to any object are in a fault state, then it is determined that any object is over-redundant.

[0022] After determining that any of the objects has excessive redundancy, mark the excessive redundancy state of any of the objects.

[0023] Optionally, the processor is configured to:

[0024] After marking the super-redundant state of any of the objects, operations on any of the objects are prohibited.

[0025] Optionally, the faultmap is cleared if the status of the faulty storage medium associated with it returns to normal.

[0026] This application also proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the aforementioned method for marking super-redundant objects in a distributed cluster.

[0027] To avoid introducing PGs and overcome the technical problems caused by their large resource requirements, this application embodiment achieves accurate identification of super-redundancy by taking snapshots of the cluster storage media status, effectively reducing the demand for memory resources.

[0028] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0029] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0030] Figure 1 This is an example of excessively redundant information labeling in a traditional distributed storage system.

[0031] Figure 2 This is an example of the Faultmap generation logic flow in an embodiment of this application;

[0032] Figure 3 This is an example of the logic flow for marking redundant objects in an embodiment of this application. Detailed Implementation

[0033] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0034] This application provides a method for marking super-redundant objects in a distributed cluster, such as... Figure 2 As shown, it includes:

[0035] In step S201, when the storage medium status of the distributed cluster is updated, it is determined whether excessive redundancy has occurred based on the cluster's minimum redundancy strategy. In specific implementations, the minimum redundancy strategy can be set according to commonly used redundancy strategies. The minimum redundancy strategy of the distributed cluster can be used to preliminarily determine whether excessive redundancy has occurred in the cluster's storage medium. In this example, the storage medium can be a disk, hard disk, CD-ROM, or other media used to record information.

[0036] In step S202, in the event of excessive redundancy, while updating the latest pool map of the cluster, a fault map is published for persistence. This fault map is only cleared upon exiting excessive redundancy. The pool map refers to metadata describing the cluster system topology. A simple example of a pool map is a cluster-rack-server node-storage medium hierarchy. The pool map records information about which racks are included in the cluster, which server nodes are under each rack, and which storage media are mounted under each node. The fault map is used to identify pool maps where excessive redundancy may occur in the cluster; that is, it is a snapshot of the current pool map taken when excessive redundancy occurs.

[0037] In step S203, based on the fault map, a routing algorithm from the object to the storage medium is invoked to identify objects that have become excessively redundant. In practice, the client can use the fault map to invoke the routing algorithm from the object to the storage medium to accurately determine whether an object has become excessively redundant. The specific routing algorithm can be set according to actual needs.

[0038] To avoid introducing PGs and overcome the technical problems caused by their large resource requirements, this application embodiment achieves accurate identification of super-redundancy by taking snapshots of the cluster storage media status, effectively reducing the demand for memory resources.

[0039] In some embodiments, based on the fault map, invoking the object-to-storage medium routing algorithm to determine objects with excessive redundancy includes: based on the fault map, invoking the object-to-storage medium routing algorithm; if all storage media mapped to any object are in a fault state, then the object is determined to have excessive redundancy; after determining that any object has excessive redundancy, the excessive redundancy state of the object is marked. In some embodiments, after marking the excessive redundancy state of any object, operations on the object are prohibited. That is, in this example, the object-to-storage medium routing algorithm can be used to further determine the excessive redundancy state of any object. If all storage media mapped to any object are in a fault state, then the object is in an excessive redundancy state, thereby allowing the object's state to be determined in real time and directly avoiding operations on that object.

[0040] In some embodiments, the fault map is cleared when the state of the faulty storage medium associated with it in the fault map returns to normal. After clearing, it means that each object is no longer in an over-redundant state.

[0041] By directly removing the PG (Programme Point) intermediate key, or by having a PG but not recording its historical changes, the method described in this application can solve the problem of accurately identifying excessive redundancy, effectively reducing the demand for memory resources. By identifying excessively redundant objects in the cluster through on-demand computation, clients can directly perceive whether objects are excessively redundant, reducing the impact of excessive redundancy on business operations.

[0042] This application also proposes a computer device, including a processor configured as follows:

[0043] In the event of a storage medium status update in the distributed cluster, it is determined whether excessive redundancy has occurred based on the cluster's minimum redundancy strategy.

[0044] In the event of excessive redundancy, a fault map is published while updating the latest pool map of the cluster, and the fault map is persisted only when the excessive redundancy is exited.

[0045] Based on the fault map, a routing algorithm from the object to the storage medium is invoked to identify the objects that have become excessively redundant.

[0046] In some embodiments, the processor is configured to:

[0047] Based on the fault map, the routing algorithm from object to storage medium is invoked. If all storage media mapped to any object are in a fault state, then it is determined that any object is over-redundant.

[0048] After determining that any of the objects has excessive redundancy, mark the excessive redundancy state of any of the objects.

[0049] In some embodiments, the processor is configured to:

[0050] After marking the super-redundant state of any of the objects, operations on any of the objects are prohibited.

[0051] In some embodiments, the fault map is cleared when the state of the faulty storage medium associated with the fault map returns to normal.

[0052] This application also proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the aforementioned method for marking super-redundant objects in a distributed cluster.

[0053] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0054] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0055] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0056] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims. All of these forms are within the protection scope of this application.

Claims

1. A method for marking super-redundant objects in a distributed cluster, characterized in that, include: When the storage media status of the distributed cluster is updated, the minimum redundancy strategy of the distributed cluster is used to preliminarily determine whether the storage media of the cluster has become over-redundant. In the event of excessive redundancy, while updating the cluster pool map, a fault map is published for persistence. The fault map is only cleared when the excessive redundancy is removed. The pool map refers to metadata describing the cluster system topology. The fault map is used to identify when the cluster may have excessive redundancy, that is, to take a snapshot of the current pool map when the cluster has excessive redundancy. Based on the fault map, the routing algorithm from object to storage medium is invoked. If all storage media mapped to any object are in a fault state, then it is determined that any object is over-redundant. After determining that any of the objects has excessive redundancy, mark the excessive redundancy state of any of the objects; After marking the super-redundant state of any of the objects, operations on any of the objects are prohibited.

2. The method for marking super-redundant objects in a distributed cluster as described in claim 1, characterized in that, It also includes clearing the fault map if the status of the faulty storage medium associated with it returns to normal.

3. A computer device, characterized in that, The processor is configured as follows: When the storage media status of a distributed cluster is updated, the minimum redundancy strategy of the distributed cluster is used to make a preliminary judgment on whether the storage media of the cluster has become over-redundant. In the event of excessive redundancy, while updating the cluster pool map, a fault map is published for persistence. The fault map is only cleared when the excessive redundancy is removed. The pool map refers to metadata describing the cluster system topology. The fault map is used to identify when the cluster may have excessive redundancy, that is, to take a snapshot of the current pool map when the cluster has excessive redundancy. Based on the fault map, a routing algorithm from the object to the storage medium is invoked to identify the objects that have become excessively redundant. The processor is configured to: Based on the fault map, the routing algorithm from object to storage medium is invoked. If all storage media mapped to any object are in a fault state, then it is determined that any object is over-redundant. After determining that any of the objects has excessive redundancy, mark the excessive redundancy state of any of the objects; The processor is configured to: After marking the super-redundant state of any of the objects, operations on any of the objects are prohibited.

4. The computer device as described in claim 3, characterized in that, If the faulty storage medium associated with the fault map returns to normal, the fault map is cleared.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method for marking super-redundant objects in a distributed cluster as described in claim 1 or 2.

Images