Cluster management method and device

Abstract

The invention relates to a cluster management method and device, and relates to the technical field of distributed systems. The cluster comprises a logic cluster composed of a plurality of physical clusters, and the management method comprises the steps that according to a first preset frequency, the resource utilization rate of each physical cluster in the logic cluster is acquired; under the condition that the resource utilization rate of each physical cluster in the logic cluster meets a self-balance triggering condition, generating a resource balance strategy based on the resource utilization rate; and migrating data in the logic cluster based on the resource balance strategy. The resource self-balancing capability of a plurality of physical clusters can be realized, the metadata pressure of a single physical cluster is reduced, and the storage efficiency is improved.

Description

Technical Field

[0001] This disclosure relates to the field of distributed systems technology, and more particularly to a cluster management method and apparatus. Background Technology

[0002] A distributed system is a loosely coupled system consisting of multiple processors interconnected via communication lines. It utilizes the storage and computing resources of each machine through a network, distributing the storage and computing load across multiple servers. For example, Elasticsearch is a distributed, highly scalable, and real-time search and data analysis engine that can be used to search various documents.

[0003] However, with the continuous enrichment of business scenarios and the sustained growth of data scale, the continuous expansion of a single-cluster architecture may face many problems. Furthermore, the research scope of resource balancing techniques for distributed systems across multiple clusters is currently relatively limited in the industry. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides a cluster management method and apparatus.

[0005] According to a first aspect of the present disclosure, a cluster management method is provided, the cluster comprising a logical cluster composed of multiple physical clusters, the management method comprising:

[0006] According to the first preset frequency, obtain the resource utilization rate of each physical cluster within the logical cluster;

[0007] If the resource utilization of each physical cluster within the logical cluster meets the self-balancing triggering condition, a resource balancing strategy is generated based on the resource utilization.

[0008] Based on the resource balancing strategy, data is migrated within the logical cluster.

[0009] In some embodiments, obtaining the resource utilization rate of each physical cluster within the logical cluster includes:

[0010] For each physical cluster within the logical cluster, obtain operating system-level monitoring metrics data and distributed system software-level monitoring metrics data for each physical cluster; wherein, the operating system-level monitoring metrics data are resource usage metrics related to the operating system, and the distributed system software-level monitoring metrics data are resource usage metrics related to the distributed system software.

[0011] Based on the operating system-level monitoring metrics data and the distributed system software-level monitoring metrics data, a weighted resource utilization factor is calculated for each physical cluster; wherein, the weighted resource utilization factor is used to calculate the resource utilization rate of the physical cluster.

[0012] Based on the weighted resource utilization factor and the maximum resource utilization factor of each physical cluster, the resource utilization rate of each physical cluster is obtained; wherein, the maximum resource utilization factor is the upper limit of resource utilization of the physical cluster.

[0013] In some embodiments, generating a resource balancing strategy based on the resource utilization rate includes:

[0014] Based on the resource utilization rate of each physical cluster within the logical cluster, the preset migration-out threshold, and the preset migration-in threshold, the migration-out cluster list and the migration-in cluster list are determined.

[0015] For each outgoing cluster in the list of outgoing clusters, a list of outgoing index groups for each outgoing cluster is determined; wherein, the list of outgoing index groups includes multiple index groups to be migrated out.

[0016] The current target cluster to be migrated into is determined from the list of clusters to be migrated into. For each cluster to be migrated out of the list of clusters to be migrated out, the resource utilization rate of the current cluster to be migrated out of the current index group in the list of the current index group to the current target cluster to be migrated into is pre-calculated.

[0017] If the resource utilization rate of the current target migration cluster meets the first preset condition, a migration sub-scheme is generated, and the resource utilization rates of the current migration cluster and the current target migration cluster are updated based on the pre-calculated resource utilization rates of the current migration-out cluster and the current target migration-in cluster. The current target migration-in cluster is then re-determined from the migration cluster list based on the resource utilization rate of the current target migration-in cluster. This continues until the resource utilization rate of the current migration-out cluster meets the second preset condition, or multiple index groups in the migration index group list of the current migration-out cluster have completed pre-calculation.

[0018] Based on the migration sub-scheme, a resource balancing strategy is generated.

[0019] In some embodiments,

[0020] The process of determining the list of clusters to be migrated out and the list of clusters to be migrated in based on the resource utilization rate of each physical cluster within the logical cluster, a preset migration-out threshold, and a preset migration-in threshold includes:

[0021] Based on the resource utilization rate of each physical cluster within the logical cluster, the physical clusters are sorted.

[0022] The physical clusters with resource utilization rates greater than the preset migration threshold are sorted in descending order to obtain a list of migration clusters.

[0023] Physical clusters with resource utilization rates below the preset migration threshold are sorted in ascending order to obtain a list of migration clusters.

[0024] In some embodiments, determining the current target cluster to migrate from the list of destination clusters includes:

[0025] From the list of incoming clusters, the incoming cluster with the lowest resource utilization rate is determined as the current target incoming cluster.

[0026] In some embodiments, determining the list of migration index groups for each migration cluster includes:

[0027] From the operating system-level monitoring metrics data and distributed system software-level monitoring metrics data of the current migrated cluster, determine the resource utilization maximum impact factor of the current migrated cluster; wherein, the resource utilization maximum impact factor is the resource type that has the greatest impact on resource utilization among the various resources of the physical cluster;

[0028] Based on the maximum impact factor of resource utilization, the index groups of the current migrated cluster are sorted to obtain a list of migrated index groups.

[0029] In some embodiments, the logical cluster includes a first layer of metadata, which is used to manage cluster information of multiple physical clusters within the logical cluster and the correspondence between each index group and the multiple physical clusters within the logical cluster; the step of performing data balancing migration within the logical cluster based on the resource balancing strategy includes:

[0030] Based on the resource balancing strategy, the index metadata and corresponding master data are migrated within the logical cluster, and the first layer metadata is updated.

[0031] In some embodiments, the method further includes: sending cluster expansion alarm information when the resource balancing policy is empty.

[0032] In some embodiments, the method further includes: sending a cluster expansion alarm message when it is determined that the resource utilization rate of each physical cluster within the logical cluster has reached a bottleneck threshold.

[0033] In some embodiments, the method further includes: acquiring multiple real-time alarm indicator data according to a second preset frequency;

[0034] Based on the multiple real-time alarm indicator data and their respective alarm thresholds, alarm information is issued.

[0035] In some embodiments, the logical cluster includes a first layer of metadata, which is used to manage cluster information of multiple physical clusters within the logical cluster and the correspondence between each index group and the multiple physical clusters within the logical cluster; the method further includes:

[0036] In response to the received registration application of the first index group, the physical cluster with the lowest resource utilization rate in the logical cluster is determined as the first target physical cluster;

[0037] The first index group and its corresponding data are distributed to the first target physical cluster; and the first layer metadata is updated.

[0038] In some embodiments, the method further includes:

[0039] In response to a user's read / write request for the first index, the link information of the second target physical cluster corresponding to the first index is obtained from the first layer metadata.

[0040] Verify the connectivity of the second target physical cluster;

[0041] After successful verification, the link information of the second target physical cluster will be returned to the user.

[0042] According to a second aspect of the embodiments of this disclosure, a cluster management device is provided.

[0043] The cluster comprises a logical cluster consisting of multiple physical clusters, and the management device includes:

[0044] The data acquisition module is used to acquire the resource utilization rate of each physical cluster within the logical cluster according to a first preset frequency.

[0045] The scheduling strategy generation module is used to generate a resource balancing strategy based on the resource utilization rate when the resource utilization rate of each physical cluster in the logical cluster meets the self-balancing triggering condition.

[0046] The scheduling strategy execution module is used to migrate data within the logical cluster based on the resource balancing strategy.

[0047] In some embodiments, when the data acquisition module acquires the resource utilization rate of each physical cluster within the logical cluster, it is specifically used for:

[0048] For each physical cluster within the logical cluster, obtain operating system-level monitoring metrics data and distributed system software-level monitoring metrics data for each physical cluster; wherein, the operating system-level monitoring metrics data are resource usage metrics related to the operating system, and the distributed system software-level monitoring metrics data are resource usage metrics related to the distributed system software.

[0049] Based on the operating system-level monitoring metrics data and the distributed system software-level monitoring metrics data, a weighted resource utilization factor is calculated for each physical cluster; wherein, the weighted resource utilization factor is used to calculate the resource utilization rate of the physical cluster.

[0050] Based on the weighted resource utilization factor and the maximum resource utilization factor of each physical cluster, the resource utilization rate of each physical cluster is obtained; wherein, the maximum resource utilization factor is the upper limit of resource utilization of the physical cluster.

[0051] In some embodiments, when generating a resource balancing strategy based on the resource utilization rate, the scheduling strategy generation module is used to:

[0052] Based on the resource utilization rate of each physical cluster within the logical cluster, the preset migration-out threshold, and the preset migration-in threshold, the migration-out cluster list and the migration-in cluster list are determined.

[0053] For each outgoing cluster in the list of outgoing clusters, a list of outgoing index groups for each outgoing cluster is determined; wherein, the list of outgoing index groups includes multiple index groups to be migrated out.

[0054] The current target cluster to be migrated into is determined from the list of clusters to be migrated into. For each cluster to be migrated out of the list of clusters to be migrated out, the resource utilization rate of the current cluster to be migrated out of the current index group in the list of the current index group to the current target cluster to be migrated into is pre-calculated.

[0055] If the resource utilization rate of the current target migration cluster meets the first preset condition, a migration sub-scheme is generated, and the resource utilization rates of the current migration cluster and the current target migration cluster are updated based on the pre-calculated resource utilization rates of the current migration-out cluster and the current target migration-in cluster. The current target migration-in cluster is then re-determined from the migration cluster list based on the resource utilization rate of the current target migration-in cluster. This continues until the resource utilization rate of the current migration-out cluster meets the second preset condition, or multiple index groups in the migration index group list of the current migration-out cluster have completed pre-calculation.

[0056] Based on the migration sub-scheme, a resource balancing strategy is generated.

[0057] In some embodiments, when the scheduling policy generation module determines the list of clusters to be migrated out and the list of clusters to be migrated in based on the resource utilization rate of each physical cluster within the logical cluster, a preset migration-out threshold, and a preset migration-in threshold, it is used to:

[0058] Based on the resource utilization rate of each physical cluster within the logical cluster, the physical clusters are sorted.

[0059] The physical clusters with resource utilization rates greater than the preset migration threshold are sorted in descending order to obtain a list of migration clusters.

[0060] Physical clusters with resource utilization rates below the preset migration threshold are sorted in ascending order to obtain a list of migration clusters.

[0061] In some embodiments, when the scheduling policy generation module determines the current target cluster to migrate to from the list of destination clusters, it is used to:

[0062] From the list of incoming clusters, the incoming cluster with the lowest resource utilization rate is determined as the current target incoming cluster.

[0063] In some embodiments, when determining the list of migration index groups for each migration cluster, the scheduling policy generation module is used to:

[0064] From the operating system-level monitoring metrics data and distributed system software-level monitoring metrics data of the current migrated cluster, determine the resource utilization maximum impact factor of the current migrated cluster; wherein, the resource utilization maximum impact factor is the resource type that has the greatest impact on resource utilization among the various resources of the physical cluster;

[0065] Based on the maximum impact factor of resource utilization, the index groups of the current migrated cluster are sorted to obtain a list of migrated index groups.

[0066] In some embodiments, the logical cluster includes a first layer of metadata, which is used to manage cluster information of multiple physical clusters within the logical cluster and the correspondence between each index group and the multiple physical clusters within the logical cluster; the scheduling policy execution module is specifically used for:

[0067] Based on the resource balancing strategy, the index metadata and corresponding master data are migrated within the logical cluster, and the first layer metadata is updated.

[0068] In some embodiments, the scheduling policy execution module is further configured to: send cluster expansion alarm information when the resource balancing policy is empty.

[0069] In some embodiments, the scheduling policy execution module is further configured to: send cluster expansion alarm information when it is determined that the resource utilization rate of each physical cluster within the logical cluster has reached a bottleneck threshold.

[0070] In some embodiments, the scheduling strategy execution module is further configured to: acquire multiple real-time alarm indicator data according to a second preset frequency;

[0071] Based on the multiple real-time alarm indicator data and their respective alarm thresholds, alarm information is issued.

[0072] In some embodiments, the logical cluster includes a first layer of metadata, which is used to manage cluster information of multiple physical clusters within the logical cluster and the correspondence between each index group and the multiple physical clusters within the logical cluster; the apparatus further includes a metadata management module, used for:

[0073] In response to the received registration application of the first index group, the physical cluster with the lowest resource utilization rate in the logical cluster is determined as the first target physical cluster;

[0074] The first index group and its corresponding data are distributed to the first target physical cluster; and the first layer metadata is updated.

[0075] In some embodiments, the apparatus further includes an index read / write control module, configured to:

[0076] In response to a user's read / write request for the first index, the link information of the second target physical cluster corresponding to the first index is obtained from the first layer metadata.

[0077] Verify the connectivity of the second target physical cluster;

[0078] After successful verification, the link information of the second target physical cluster will be returned to the user.

[0079] According to a third aspect of the present disclosure, a server is provided, comprising:

[0080] At least one processor; and

[0081] A memory communicatively connected to the at least one processor; wherein,

[0082] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the cluster management method described in the first aspect above.

[0083] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, comprising:

[0084] The storage medium stores a computer program, which, when executed by a processor, implements the cluster management method described in the first aspect above.

[0085] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0086] By implementing the embodiments of this disclosure, cluster management can be performed on a logical cluster composed of multiple physical clusters as a whole, enabling unified automated management of multiple physical clusters; reducing operational complexity and improving operational automation capabilities. Resource balancing strategies are automatically generated based on the resource utilization of each physical cluster. These strategies achieve resource balancing among multiple physical clusters within the logical cluster, thereby realizing the self-balancing capability of multiple physical clusters, reducing metadata pressure on a single physical cluster, and improving storage efficiency.

[0087] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0088] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0089] Figure 1 This is an architecture diagram of a single cluster according to an exemplary embodiment.

[0090] Figure 2 This is a flowchart illustrating a cluster management method according to an exemplary embodiment.

[0091] Figure 3 This is an architecture diagram of a logical cluster illustrated according to an exemplary embodiment.

[0092] Figure 4 This is a flowchart illustrating a cluster management method according to another exemplary embodiment.

[0093] Figure 5 This is a comparison diagram of a logical cluster before and after resource balancing, according to an exemplary embodiment.

[0094] Figure 6 This is a block diagram illustrating a service architecture according to an exemplary embodiment.

[0095] Figure 7 This is a flowchart illustrating metadata registration according to an exemplary embodiment.

[0096] Figure 8 This is a flowchart illustrating user read / write index according to an exemplary embodiment.

[0097] Figure 9 This is a block diagram illustrating a cluster management device according to an exemplary embodiment.

[0098] Figure 10 This is a block diagram illustrating a server according to an exemplary embodiment. Detailed Implementation

[0099] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0100] A distributed system is a loosely coupled system composed of multiple processors interconnected via communication lines. It features distribution, autonomy, parallelism, and globality. By utilizing the storage and computing resources of each machine within an enterprise through a network, these distributed storage and computing resources are combined into a virtual device. Data is distributed across multiple independent devices within the enterprise, with multiple storage servers sharing the storage load and the computing resources provided by multiple servers distributing the computational pressure. This not only improves the system's reliability, availability, and storage and computing efficiency but also facilitates scalability.

[0101] For example, Elasticsearch is a distributed, highly scalable, and real-time search and data analysis engine. It easily enables the searching, analysis, and exploration of large amounts of data. Fully leveraging Elasticsearch's horizontal scalability can make data more valuable in production environments. The implementation principle of Elasticsearch mainly involves the following steps: First, the user submits data to the Elasticsearch database. Then, the tokenizer segments the corresponding text into words, storing the weights and segmentation results in the database. When a user searches for data, the results are ranked and scored according to their weights, and then the results are presented to the user.

[0102] Elasticsearch can be used to search a wide variety of documents. It provides scalable search with near real-time performance and supports multi-tenancy. Elasticsearch is distributed, meaning that indexes can be split into shards, each with zero or more replicas. Each node hosts one or more shards and acts as a coordinator, delegating operations to the correct shard. Rebalancing and routing are handled automatically. Related data is typically stored in the same index, which consists of one or more primary shards and zero or more replica shards. Once an index is created, the number of primary shards cannot be changed.

[0103] Elasticsearch uses Lucene and attempts to provide all its features through JSON and a Java API (Application Programming Interface). It supports faceting and percolating, which is very useful for notifications if a new document matches a registered query. Another feature, called a "gateway," handles long-term index persistence; for example, in the event of a server crash, indexes can be recovered from the gateway. Elasticsearch supports real-time GET requests, making it suitable as a data store for NoSQL databases, but it lacks distributed transactions.

[0104] Research on resource self-balancing techniques across multiple clusters in distributed systems is currently limited. A common approach is to use automated load balancers and container orchestration tools, such as Kubernetes, to coordinate and manage resource allocation across multiple clusters. These tools can automatically identify and manage imbalances in resource utilization, reallocating loads to more idle nodes and ensuring balanced processing of applications across multiple clusters. Another common approach is to achieve automatic resource balancing through distributed computing frameworks such as Apache Spark and Apache Flink. These frameworks provide built-in automatic load balancers that can automatically allocate and manage computing tasks across multiple clusters to ensure optimal resource utilization. These frameworks also offer rich visualization tools to help users understand resource utilization and performance parameters across different clusters, enabling better optimization and adjustments.

[0105] Furthermore, with the development of blockchain technology and smart contracts, several distributed resource self-balancing solutions based on blockchain and smart contracts have emerged. These solutions utilize the decentralized nature of blockchain and the programmability of smart contracts to create a trusted, fair, and automated resource allocation system that can automatically balance resource allocation across different clusters, thereby achieving the goal of multi-cluster resource self-balancing.

[0106] However, there is relatively little research on technologies for collecting, analyzing, scheduling, and self-balancing resource utilization data across clusters in any distributed system under the big data field.

[0107] A cluster is a group of computers that work together as a whole to provide a set of network resources to users. Individual computer systems within the cluster are called nodes. An Elasticsearch (ES) cluster consists of one or more nodes. An index group is viewed as a table, and an index is viewed as a table partition. Each table partition is distributed and can contain multiple shards, each storing a portion of the data from one table partition.

[0108] With the continuous development of the ElasticSearch system, and against the backdrop of increasingly diverse business scenarios and continuously growing data scale, single-cluster architectures (such as...) are becoming increasingly common. Figure 1 (As shown) Continued expansion may face the following problems:

[0109] Operation and maintenance issues: As the scale of a single cluster server continues to grow, the excessive number of physical servers in a single cluster can significantly increase the probability of communication anomalies and hardware damage, resulting in greater operation and maintenance risks and difficulties.

[0110] Data management issues: These involve physical data management and metadata management. As the cluster scales up, resource skew and uneven resource utilization can easily occur on hot data nodes. Furthermore, data redistribution computation takes a long time, and data synchronization cycles are lengthy. Simultaneously, excessively large metadata volumes can lead to high metadata storage consumption and long cluster metadata synchronization cycles, potentially causing a split-brain scenario in the cluster.

[0111] Data security issues: In sensitive data scenarios, tenant isolation mechanisms based on physical isolation are required. However, access control of excessively large clusters is difficult and can easily lead to data security problems.

[0112] Based on the above issues, when a single ElasticSearch system cluster is too large, multiple physical clusters are needed to support the data. However, having too many physical clusters in the same region can also lead to problems such as high cluster management costs, difficulty in access control, and uneven resource utilization.

[0113] Therefore, this disclosure aims to provide a cluster management method and system, a method and apparatus for managing multiple physical clusters and self-balancing resources among physical clusters in a distributed system based on the concept of logical clusters, providing unified logical cluster management capabilities among multiple physical clusters in a single area, reducing cluster metadata pressure, reducing the difficulty of data management and cluster management, reducing access control difficulty, and improving data security.

[0114] Figure 2This is a flowchart illustrating a cluster management method according to an exemplary embodiment. It should be noted that the cluster management method of this disclosure can be applied to a cluster management device of this disclosure, which can be configured on a server. Figure 2 As shown, the management method of this cluster may include the following steps.

[0115] In this disclosure, such as Figure 3 As shown, this cluster is a logical cluster composed of multiple physical clusters.

[0116] A logical cluster can be viewed as a service provided by a single cluster; the user doesn't need to know which physical cluster they are accessing. The logical cluster gateway is a crucial component of a logical cluster. User requests to a logical cluster are first sent to the logical cluster gateway, which then accesses the logical cluster management service to route the request and forward it to the specific physical cluster.

[0117] In step S101, the resource utilization rate of each physical cluster within the logical cluster is obtained according to the first preset frequency.

[0118] In one implementation, the resource utilization rate of each physical cluster within the logical cluster is obtained, including:

[0119] For each physical cluster within the logical cluster, obtain operating system-level monitoring metrics and distributed system software-level monitoring metrics for each physical cluster. The operating system-level monitoring metrics are resource usage metrics related to the operating system, while the distributed system software-level monitoring metrics are resource usage metrics related to the distributed system software. Based on these metrics, calculate the weighted resource utilization factor for each physical cluster. This factor is used to calculate the resource utilization rate of the physical cluster. Based on the weighted resource utilization factor and the maximum resource utilization factor for each physical cluster, obtain the resource utilization rate for each physical cluster. The maximum resource utilization factor represents the upper limit of resource usage for the physical cluster.

[0120] Operating system-level monitoring metrics may include, but are not limited to, CPU (Central Processing Unit) load, memory utilization, disk utilization, and network input / output. Distributed system software-level monitoring metrics may include, but are not limited to, the index-level status and index shard-level status of a single physical cluster.

[0121] In this embodiment, an index group is considered a table, and an index is considered a table partition. Each table partition is distributed and can contain multiple shards, with each shard storing a portion of the data within a table partition. Index-level status may include the number of documents per index, the storage size of a single index, the number of queries accessing a single index, the memory usage of a single index, the total number of shards per index, and the number of primary and secondary shards per index; index shard-level status includes the number of documents per index shard, the storage size of a single index shard, and the memory usage of a single index shard.

[0122] The operating system-level monitoring metrics data and the distributed system software-level monitoring metrics data will be referred to as multi-dimensional monitoring metrics data.

[0123] In one implementation, acquiring multi-dimensional monitoring metrics data for the physical cluster can be achieved through periodic reporting. The first preset frequency is the period for acquiring resource utilization data; as an example, the period could be one day.

[0124] As one implementation method, a weighted resource utilization factor for each physical cluster is calculated based on multi-dimensional monitoring indicator data and the corresponding weight of each monitoring indicator.

[0125] In this embodiment, the multi-dimensional monitoring indicator data includes operating system-level monitoring indicator data and distributed system software-level monitoring indicator data.

[0126] As an example, the weighted resource usage factor (usage_factor) for each physical cluster can be calculated using the following formula:

[0127] asage_factor=∑(Voi(Metric i ) / ToTal(Metric i ))*100*weight i

[0128] Where, weight i Val(Metric) represents the weight of the i-th metric item. i ) represents the value of the i-th metric in the current physical cluster, ToTal(Metric) i ) represents the maximum capacity of the i-th metric for the current physical cluster, where metric represents a metric item. As an example, the metrics involved in calculating the weighted resource utilization factor for each physical cluster are shown in the table below:

[0129]

[0130] Taking the table above as an example, i is the index, metric1 represents the metric item CPU_AVG_LOAD_15min, Val(metric1) represents the current value of the metric item CPU_AVG_LOAD_15min for this physical cluster, ToTal(metric1) represents the maximum value of the metric item CPU_AVG_LOAD_15min for this physical cluster, and weight1 represents the weight of the metric item CPU_AVG_LOAD_15min in the calculation, which is 0.5.

[0131] One implementation method for obtaining the resource utilization rate of a physical cluster includes: calculating the quotient of the weighted resource utilization factor of the physical cluster and the maximum resource utilization factor of the physical cluster, and using the quotient as the resource utilization rate of the physical cluster. That is:

[0132] Resource utilization rate = (Weighted resource utilization factor / Cluster maximum resource utilization factor)

[0133] It should be noted that the maximum resource utilization factor of a physical cluster refers to the maximum available resource data of the physical cluster, which can be understood as the maximum capacity of a container.

[0134] In step S102, if the resource utilization of each physical cluster within the logical cluster meets the self-balancing triggering condition, a resource balancing strategy is generated based on the resource utilization.

[0135] In one implementation, the self-balancing trigger condition includes at least one physical cluster having a resource utilization rate greater than a first threshold, and the physical cluster with the lowest resource utilization rate having a resource utilization rate less than a second threshold. For example, the first threshold can be 70%, and the second threshold can be 60%. Self-balancing is triggered when the resource utilization rate of at least one physical cluster is greater than 70%, and the resource utilization rate of the physical cluster with the lowest resource utilization rate is less than 60%.

[0136] After the self-balancing trigger condition is met, i.e., self-balancing is triggered, it is necessary to determine the outgoing and incoming clusters and generate a resource balancing strategy.

[0137] In one implementation, physical clusters with resource utilization rates greater than a preset migration-out threshold and those with resource utilization rates less than a preset migration-in threshold are respectively identified as migration-out clusters requiring data migration and migration-in clusters capable of handling data migration. For each migration-out cluster, a pre-calculation is performed to determine whether migrating out and migration-in conditions are met after migrating a portion of the index groups from the migration-out cluster to the target migration-in cluster. The migration-out condition is the lower limit of the resource utilization rate of the migration-out cluster for migrating out data (e.g., 65%), and the migration-in condition is the upper limit of the resource utilization rate of the migration-in cluster capable of handling data migration (e.g., 70%). If the pre-calculation results meet both the migration-out and migration-in conditions, migration is allowed; if the pre-calculation results do not meet the migration-out and migration-in conditions, migration is not allowed. Finally, a resource balancing strategy is generated.

[0138] As an example, a logical cluster includes three physical clusters: physical cluster A, physical cluster B, and physical cluster C. The resource utilization rates of the three physical clusters are 71%, 62%, and 50%, respectively. If physical cluster A is determined to be the outgoing cluster and physical cluster C is the incoming cluster, calculate the resource utilization rates of physical cluster A and physical cluster C after migrating a portion of the index groups from physical cluster A to physical cluster C. If the resource utilization rates of physical cluster A and physical cluster C after migration meet the outgoing and incoming conditions, respectively, then migration is possible, and a resource balancing strategy is generated.

[0139] In step S103, data is migrated within the logical cluster based on a resource balancing strategy.

[0140] It should be noted that the generated resource balancing strategy includes at least two possibilities. One possibility is that it includes migration target data, that is, it includes migration targets for specific index groups from the migration-out cluster to the migration-in cluster. The other possibility is that the resource balancing strategy is empty, that is, no migration target data that meets the migration conditions is found, meaning that no migration-in cluster can handle the migration of any data from the migration-out cluster, and there is no migration target data.

[0141] In this embodiment, the logical cluster includes a first layer of metadata, which is used to manage the cluster information of multiple physical clusters within the logical cluster and the correspondence between each index group and multiple physical clusters within the logical cluster.

[0142] In one implementation of a resource balancing strategy that includes migration target data, the method for performing data balancing migration within a logical cluster includes: migrating index metadata and corresponding master data within the logical cluster based on the resource balancing strategy, and updating the first-level metadata.

[0143] In other words, the index metadata of all index groups that meet the migration conditions involved in the resource balancing strategy, along with the master data corresponding to each index group, are migrated between physical clusters according to the migration-out cluster and the target migration-in cluster determined in the resource balancing strategy. Here, index metadata is configuration data, representing the distribution of indexes within a logical cluster, such as which physical cluster a particular index is located on. Master data represents the actual business data stored in the indexes.

[0144] When the generated resource balancing policy is empty, a cluster expansion alarm is sent. This can be understood as follows: If self-balancing is triggered, it means that the resource utilization of some physical clusters within the logical cluster has reached a point where some data needs to be migrated out. Since the resource balancing policy is empty, it means that other physical clusters cannot support the migration of any index groups. In other words, the resource utilization of all physical clusters within the logical cluster is generally high, and the logical cluster needs to be expanded.

[0145] By implementing the embodiments of this disclosure, cluster management can be performed on a logical cluster composed of multiple physical clusters as a whole, achieving unified automated management of multiple physical clusters; reducing operational complexity and improving operational automation capabilities. Resource balancing strategies are automatically generated based on the resource utilization of each physical cluster. According to these strategies, resource balancing among multiple physical clusters within the logical cluster is achieved, thereby realizing the self-balancing capability of multiple physical clusters, reducing metadata pressure on a single physical cluster, and improving storage efficiency. Furthermore, multi-dimensional indicator collection and monitoring are possible, providing fine-grained cluster resource management and monitoring alerting capabilities, with dimensional analysis of both cluster hardware and software indicators.

[0146] The following section provides a detailed explanation of the generation of a resource balancing strategy based on resource utilization in step S12.

[0147] Figure 4 This is a flowchart illustrating a method for generating a resource balancing strategy based on resource utilization, according to an exemplary embodiment. Figure 4 As shown, the method includes the following steps.

[0148] In step S201, the list of clusters to be migrated out and the list of clusters to be migrated in are determined based on the resource utilization rate of each physical cluster within the logical cluster, the preset migration-out threshold, and the preset migration-in threshold.

[0149] In one implementation, a list of clusters to be migrated out and a list of clusters to be migrated in are determined based on the resource utilization rate of each physical cluster within the logical cluster, a preset migration-out threshold, and a preset migration-in threshold. This includes: sorting each physical cluster based on its resource utilization rate; sorting physical clusters with resource utilization rates greater than the preset migration-out threshold in descending order to obtain a list of clusters to be migrated out; and sorting physical clusters with resource utilization rates less than the preset migration-in threshold in ascending order to obtain a list of clusters to be migrated in.

[0150] It is understandable that the outgoing clusters migrate out in descending order, while the incoming clusters migrate in in ascending order.

[0151] As an example, the preset out-of-region threshold is 70%, and the preset in-region threshold is 60%. Physical clusters with resource utilization greater than 70% are found and sorted in descending order of resource utilization to form an out-of-region cluster list; physical clusters with resource utilization no greater than 60% are found and sorted in ascending order of resource utilization to form an in-region cluster list.

[0152] In step S202, for each outgoing cluster in the outgoing cluster list, a list of outgoing index groups for each outgoing cluster is determined; wherein, the list of outgoing index groups includes multiple index groups to be outgoing.

[0153] In one implementation, the method for determining the migration index group list for each migration cluster includes: determining the maximum impact factor on resource utilization of the current migration cluster from the operating system-level monitoring indicator data and the distributed system software-level monitoring indicator data of the current migration cluster; and sorting the index groups of the current migration cluster based on the maximum impact factor on resource utilization to obtain the migration index group list. The maximum impact factor on resource utilization is the resource type that has the greatest impact on resource utilization among the various resources of the physical cluster.

[0154] For example, the maximum impact factor on resource utilization for each physical cluster is calculated, specifically from monitoring metrics such as CPU, shards, and disks. If it's CPU, the index groups of the physical cluster are sorted according to CPU utilization. For instance, they are sorted from largest to smallest, meaning the resource utilization of the index group with the greatest impact is pre-calculated after migration. Then, if the largest index group doesn't have enough data to be migrated into the cluster, the resource utilization of the next index group is pre-calculated sequentially.

[0155] In step S203, the current target migration cluster is determined from the migration cluster list. For each migration cluster in the migration cluster list, the resource utilization rate of the current migration cluster and the current target migration cluster is pre-calculated if the current index group in the migration index group list of the current migration cluster migrates to the current target migration cluster.

[0156] In one implementation, determining the current target migration cluster from the migration cluster list includes: selecting the migration cluster with the lowest resource utilization from the migration cluster list as the current target migration cluster. In other words, if the migration cluster with the lowest resource utilization cannot handle the migration of the current index group, then none of the other migration clusters can handle the migration of the current index group.

[0157] In step S204, if the resource utilization rate of the current target migration cluster meets the first preset condition, a migration sub-scheme is generated, and the resource utilization rates of the current migration cluster and the current target migration cluster are updated based on the pre-calculated resource utilization rates of the current migration cluster and the current target migration cluster. The current target migration cluster is then re-determined from the migration cluster list based on the resource utilization rate of the current target migration cluster. This continues until the resource utilization rate of the current migration cluster meets the second preset condition, or multiple index groups in the migration index group list of the current migration cluster have completed pre-calculation.

[0158] For example, the second preset condition can be 65%, and the first preset condition can be 70%. For each migration cluster, the resource utilization rate of the current migration cluster and the current target migration cluster is pre-calculated if the current index group in the migration index group list of the current migration cluster is migrated to the current target migration cluster. It is then determined whether the resource utilization rate of the current migration cluster is less than 65% and the resource utilization rate of the current target migration cluster is not greater than 70%.

[0159] If the resource utilization rate of the current target migration cluster is not greater than 70%, it indicates that the current target migration cluster can withstand the migration of the index group, and the migration sub-scheme is determined. The resource utilization rates of the current migration cluster and the current target migration cluster are updated based on the pre-calculated resource utilization rates of the current migration cluster and the current target migration cluster. The resource utilization rate of the migration cluster list is recalculated and sorted according to the indicators after migration. That is, the migration cluster list is re-sorted in ascending order of resource utilization rate in order to quickly determine the new current target migration cluster.

[0160] In one scenario, if the resource utilization of the current outgoing cluster is still greater than 65% after all indexes of the index group have been migrated out, the next index group in the current outgoing index group list is scanned sequentially, and the above logic continues until the resource utilization of the current outgoing cluster is no greater than 65%. In another scenario, if the current target incoming cluster cannot handle the migration of the index group, the next index group in the current outgoing index group list is scanned sequentially, and the above logic continues.

[0161] If the resource utilization of the current outgoing cluster is less than 65% after the outgoing index group, then the process will move to the next outgoing cluster for pre-computation. If some outgoing clusters in the outgoing cluster list do not meet the conditions for outgoing from any index group, then those outgoing clusters will be skipped. This continues until the resource balancing scheme for all outgoing clusters is completed.

[0162] In step S205, a resource balancing strategy is generated based on the migration sub-scheme.

[0163] Once all resource balancing schemes for migrating out of the cluster are completed, the resulting migration sub-schemes are merged to generate a resource balancing strategy. Based on the resource balancing strategy, the index group metadata and master data are migrated, and the first-level metadata of the logical cluster is updated.

[0164] like Figure 5 As shown, the resource utilization of each physical cluster within the logical cluster can be intuitively compared before and after resource balancing. Obviously, after resource balancing, the resource utilization of each physical cluster within the logical cluster is more balanced.

[0165] By implementing the embodiments of this disclosure, a list of outgoing and incoming clusters is determined by preset thresholds. For each outgoing cluster, multiple index groups are pre-calculated for migration. The pre-calculation evaluates whether the resource utilization of the index group after migration to the target incoming cluster meets the migration conditions. If the conditions are met, a migration plan is generated. Data migration is performed according to the migration plan, thereby achieving resource self-balancing capability of multiple physical clusters, improving the resource utilization of logical clusters, reducing metadata pressure on a single physical cluster, increasing the upper limit of index storage and the upper limit of the number of shards of a logical cluster in a single region, and improving storage efficiency.

[0166] It should also be noted that in this embodiment, a bottleneck threshold is set for the logical cluster. When the bottleneck threshold is reached, there is no need to perform self-balancing to generate a resource balancing strategy; instead, an expansion alarm is triggered directly.

[0167] In some embodiments, when it is determined that the resource utilization rate of each physical cluster within the logical cluster has reached a bottleneck threshold, a cluster expansion alarm message is sent.

[0168] As an example, when the average resource utilization of multiple physical clusters exceeds 70%, and the minimum resource utilization among these clusters exceeds 60%, a bottleneck threshold is reached, and a cluster expansion alarm is sent directly. This can be understood as the entire logical cluster reaching its capacity limit; at this point, self-balancing is no longer necessary, and cluster expansion is required directly.

[0169] By implementing the embodiments of this disclosure, when the resource utilization of each physical cluster within the logical cluster reaches the bottleneck threshold, a cluster expansion alarm message is directly sent so that cluster administrators can respond in a timely manner to prevent abnormal situations from occurring.

[0170] In addition to achieving resource self-balancing capabilities by monitoring resource utilization, the logical cluster of this disclosure embodiment also has the ability to issue real-time alarms for various abnormal situations.

[0171] In some embodiments, multiple real-time alarm indicator data are acquired according to a second preset frequency; alarm information is issued based on the multiple real-time alarm indicator data and their respective corresponding alarm thresholds.

[0172] As an example, according to the real-time alarm rules in the table below, an alarm message is issued when the corresponding alarm conditions are met.

[0173]

[0174]

[0175] By implementing the embodiments of this disclosure, real-time alarms are generated by monitoring the overload of specific monitoring indicators, enabling managers to respond promptly and prevent abnormal situations from occurring.

[0176] In the overall implementation of the logical cluster, since the logical cluster is built on multiple physical clusters, a layer of metadata is needed to specifically manage the cluster information of all physical clusters under the logical cluster and the correspondence between each index and the physical cluster within the logical cluster, so as to facilitate user requests and access and the construction of the resource self-balancing mechanism within the logical cluster.

[0177] In some embodiments, the logical cluster includes a first layer of metadata, which is used to manage cluster information of multiple physical clusters within the logical cluster and the correspondence between each index group and the multiple physical clusters.

[0178] By implementing the embodiments of this disclosure, the cluster information of multiple physical clusters and the correspondence between each index group and multiple physical clusters are managed through the first layer of metadata. Metadata layering is achieved with the metadata of each physical cluster. By introducing the concept of logical clusters, a layered architecture for table and index metadata is realized, thereby reducing the pressure problem caused by the continuous expansion of metadata of a single physical cluster.

[0179] Based on the above embodiments, in some embodiments, in response to the received registration application of the first index group, the physical cluster with the lowest resource utilization in the logical cluster is determined as the first target physical cluster; the first index group and the corresponding data are distributed to the first target physical cluster; and the first layer metadata is updated.

[0180] When registering a new index group, selecting the physical cluster with the lowest resource utilization for storage is beneficial for the balanced utilization of resources across multiple physical clusters.

[0181] By implementing the embodiments of this disclosure, the registration of index group metadata under a logical cluster is achieved.

[0182] In some embodiments, in response to a user's read / write request for the first index, the link information of the second target physical cluster corresponding to the first index is obtained from the first layer metadata; the connectivity of the second target physical cluster is verified; and after the verification is passed, the link information of the second target physical cluster is returned to the user.

[0183] An error message is returned if the verification fails.

[0184] After the client receives a normal request, it directly establishes a direct connection to the second target physical cluster and performs index read and write operations.

[0185] By implementing the embodiments of this disclosure, client-side read / write index logic under a logical cluster can be realized.

[0186] As an example, Figure 6 This is a block diagram of a service architecture. Taking an ElasticSearch logical cluster as an example, this service architecture can implement the cluster management methods of any of the above embodiments. Figure 6 As shown, this service architecture includes logical clusters and a logical cluster management system for managing the logical clusters.

[0187] A logical cluster consists of one or more ElasticSearch physical clusters, all of which share the same read / write separation deployment architecture and are composed of the same machine types. However, it can support multiple physical clusters of heterogeneous scale to form a logical cluster, meaning that logical clusters can be built using the same number of hosts or different numbers of hosts.

[0188] The logical cluster management system mainly includes three services: operational data service, metadata service, and resource scheduling service. The system is primarily responsible for building operational metrics collection, metrics data warehouse construction, daily cluster inspection, unified monitoring and alerting, unified metadata management, and resource scheduling management capabilities for the entire logical cluster.

[0189] Operational data service: mainly provides monitoring indicator data at the logical cluster level. Each physical cluster in the same logical cluster reports multi-dimensional monitoring indicator data at a set period, for example, the default set period is one minute.

[0190] As an example, operational data services may include, but are not limited to, a metrics data warehouse, a service inspection module, a monitoring dashboard module, and a dimension management module.

[0191] In this embodiment, the multi-dimensional monitoring metrics data includes operating system-level monitoring metrics data and distributed system software-level monitoring metrics data. In this example, the distributed system software-level monitoring metrics data is ElasticSearch system metrics data.

[0192] As a specific example, Table 1 below shows the indicator items and descriptions of multi-dimensional monitoring indicator data.

[0193]

[0194] As one approach, an indicator data warehouse is built based on the collected indicators to achieve persistent storage of indicator data, and the indicator data of the most recent year is stored as a backtracking indicator so that data analysis and other tasks can be carried out through historical data.

[0195] The service inspection module, based on periodically reported indicators, enables the overall logical cluster to perform anomaly inspection and alarm notifications and daily inspection reports. It can assess the overall health status of the current logical cluster, push real-time alarms, cluster scaling notifications, daily anomaly information summaries, and cluster SLA (Service Level Agreement) assessments.

[0196] As one implementation method, different inspection rules can be set to achieve timed reporting, corresponding alarms, and self-balancing triggering based on the inspection rules. See Table 2 for a specific example.

[0197]

[0198] The service inspection module enables real-time monitoring of overload conditions for specific metrics, providing real-time alerts and allowing administrators to respond promptly to prevent anomalies. The resource self-balancing strategy of the logical cluster is triggered by the service inspection module within the daily inspection rules.

[0199] Metadata services: Primarily provides metadata services related to logical clusters, including registration, collection, and storage of cluster-level metadata (cluster metadata), index-level metadata (index metadata), and user-level metadata (user metadata).

[0200] It should be noted that the overall implementation of the logical cluster is based on the construction of multiple physical clusters. Therefore, a layer of metadata is needed to manage the cluster information of all physical clusters under the logical cluster, as well as the correspondence between each index in the logical cluster and the physical cluster, so as to facilitate user requests and access and the construction of the resource self-balancing mechanism within the logical cluster.

[0201] like Figure 7As shown, during the metadata registration process, the user submits a registration request for an index group. This request is sent to the metadata service, which then requests the intelligent resource analysis module of the resource scheduling service. Based on the calculation, the module returns the physical cluster with the lowest resource utilization (i.e., the lowest weighted resource utilization factor) in the logical cluster to handle the data distribution of the newly registered index group. Finally, the data is returned to the metadata service for metadata registration updates and metadata storage, completing the entire process.

[0202] The following uses an Elasticsearch (ES) system cluster as an example to illustrate the client-side read / write index logic. Figure 8 As shown, when a client makes an Elasticsearch read / write request, it first sends the index operation request to the metadata service module of the logical cluster to obtain the physical cluster link information corresponding to the index. When the metadata service module receives the user request, it queries the physical cluster information corresponding to the index and verifies that the connectivity of the physical cluster is normal. If the connectivity is normal, it returns the link information of the corresponding physical cluster to the client; otherwise, it returns an exception message. When the client receives a normal request, it directly establishes a direct connection to the physical cluster to perform index read / write operations.

[0203] Resource scheduling service: Provides resource utilization analysis and evaluation for each physical cluster within the logical cluster, and can generate resource balancing strategies based on resource utilization. At the same time, it can complete migration operations such as ES index data balancing migration and metadata update within each physical cluster.

[0204] The resource scheduling service includes a resource intelligent analysis module, a rule management module, a data migration module, and a metadata update module. The resource intelligent analysis module is mainly used to perform resource utilization analysis and evaluation for each physical cluster within the logical cluster. The rule management module is mainly used to manage rules such as self-balancing trigger conditions. The data migration module is mainly used to implement data migration corresponding to resource balancing strategies. The metadata update module is mainly used to provide metadata update services after data migration.

[0205] By implementing this embodiment, all the functions of the above-mentioned cluster management method can be realized. It enables unified automated management of multiple physical clusters, reducing the number of physical clusters, lowering operational complexity, and improving operational automation capabilities. It allows for multi-dimensional indicator collection and monitoring, providing fine-grained resource management and monitoring alerts for the cluster, with dimensional analysis of both cluster hardware and software indicators. It enables resource self-balancing capabilities across multiple physical clusters, improving the resource utilization of logical clusters; after application, resource utilization increased from 26% to 60%. It reduces the metadata pressure on a single physical cluster, increases the upper limit of index storage and shard count for logical clusters within a single region, improves storage efficiency, and allows for the management of over 10,000 indexes and nearly 100,000 shards in a single logical cluster within a single region.

[0206] Figure 9This is a block diagram illustrating a cluster management device according to an exemplary embodiment. (Refer to...) Figure 9 The device includes a data acquisition module 301, a scheduling strategy generation module 302, and a scheduling strategy execution module 303.

[0207] The cluster is a logical cluster composed of multiple physical clusters, and the management device includes:

[0208] The data acquisition module 301 is used to acquire the resource utilization rate of each physical cluster within the logical cluster according to a first preset frequency.

[0209] The scheduling strategy generation module 302 is used to generate a resource balancing strategy based on the resource utilization rate when the resource utilization rate of each physical cluster in the logical cluster meets the self-balancing triggering condition.

[0210] The scheduling strategy execution module 303 is used to migrate data within the logical cluster based on the resource balancing strategy.

[0211] In some embodiments, when the data acquisition module 301 acquires the resource utilization rate of each physical cluster within the logical cluster, it is specifically used for:

[0212] For each physical cluster within the logical cluster, obtain operating system-level monitoring metrics and distributed system software-level monitoring metrics for each physical cluster; among them, the operating system-level monitoring metrics are resource usage metrics related to the operating system, and the distributed system software-level monitoring metrics are resource usage metrics related to the distributed system software.

[0213] Based on operating system-level monitoring metrics and distributed system software-level monitoring metrics, a weighted resource utilization factor is calculated for each physical cluster; the weighted resource utilization factor is used to calculate the resource utilization rate of the physical cluster.

[0214] The resource utilization rate of each physical cluster is obtained based on the weighted resource utilization factor and the maximum resource utilization factor of each physical cluster; where the maximum resource utilization factor is the upper limit of resource utilization of the physical cluster.

[0215] In some embodiments, when generating a resource balancing strategy based on resource utilization, the scheduling strategy generation module 302 is specifically used for:

[0216] Based on the resource utilization rate of each physical cluster within the logical cluster, the preset migration-out threshold, and the preset migration-in threshold, determine the migration-out cluster list and the migration-in cluster list.

[0217] For each outgoing cluster in the outgoing cluster list, determine the outgoing index group list for each outgoing cluster; wherein, the outgoing index group list includes multiple index groups to be migrated out.

[0218] Determine the current target cluster to migrate from the list of incoming clusters. For each outgoing cluster in the list of outgoing clusters, pre-calculate the resource utilization of the current outgoing cluster and the current target incoming cluster after migrating out the current index group in the list of outgoing index groups of the current outgoing cluster to the current target incoming cluster.

[0219] If the resource utilization rate of the current target migration cluster meets the first preset condition, a migration sub-scheme is generated, and the resource utilization rates of the current migration-out cluster and the current target migration-in cluster are updated based on the pre-calculated resource utilization rates of the current migration-out cluster and the current target migration-in cluster. The current target migration-in cluster is then re-determined from the migration-in cluster list based on the resource utilization rate of the current target migration-in cluster. This continues until the resource utilization rate of the current migration-out cluster meets the second preset condition, or multiple index groups in the migration-out index group list of the current migration-out cluster have completed pre-calculation.

[0220] Based on the migration sub-scheme, a resource balancing strategy is generated.

[0221] In some embodiments, when the scheduling policy generation module 302 determines the list of clusters to be migrated out and the list of clusters to be migrated in based on the resource utilization rate of each physical cluster within the logical cluster, a preset migration-out threshold, and a preset migration-in threshold, it is used to:

[0222] Based on the resource utilization of each physical cluster within the logical cluster, the physical clusters are sorted.

[0223] The physical clusters with resource utilization rates greater than the preset migration threshold are sorted in descending order to obtain a list of migration clusters.

[0224] Physical clusters with resource utilization rates below the preset migration threshold are sorted in ascending order to obtain a list of migration clusters.

[0225] In some embodiments, when the scheduling policy generation module 302 determines the current target cluster to migrate to from the list of destination clusters, it is used to:

[0226] From the list of incoming clusters, the cluster with the lowest resource utilization is selected as the current target cluster for migration.

[0227] In some embodiments, the scheduling policy generation module 302 in

[0228] When determining the list of migration index groups for each migration cluster, it is used for:

[0229] From the operating system-level monitoring metrics data and distributed system software-level monitoring metrics data of the current migrated cluster, determine the resource utilization impact factor of the current migrated cluster; among them, the resource type with the greatest impact on resource utilization among the various resources of the physical cluster.

[0230] Based on the factor with the greatest impact on resource utilization, the index groups of the current migrated cluster are sorted to obtain a list of migrated index groups.

[0231] In some embodiments, the logical cluster includes a first-layer metadata, which is used to manage cluster information of multiple physical clusters within the logical cluster and the correspondence between each index group and the multiple physical clusters within the logical cluster; the scheduling policy execution module 303 is specifically used for:

[0232] Based on the resource balancing strategy, the index metadata and corresponding master data are migrated within the logical cluster, and the first-level metadata is updated.

[0233] In some embodiments, the scheduling policy execution module 303 is further configured to: send cluster expansion alarm information when the resource balancing policy is empty.

[0234] In some embodiments, the scheduling policy execution module 303 is further configured to: send cluster expansion alarm information when it is determined that the resource utilization rate of each physical cluster within the logical cluster has reached the bottleneck threshold.

[0235] In some embodiments, the scheduling strategy execution module 303 is further configured to: acquire multiple real-time alarm indicator data according to a second preset frequency;

[0236] Alarm information is issued based on multiple real-time alarm metrics and their corresponding alarm thresholds.

[0237] In some embodiments, the logical cluster includes a first layer of metadata, which is used to manage cluster information of multiple physical clusters within the logical cluster and the correspondence between each index group and the multiple physical clusters within the logical cluster; the apparatus further includes a metadata management module 304, used for:

[0238] In response to the registration application received from the first index group, the physical cluster with the lowest resource utilization within the logical cluster is determined as the first target physical cluster;

[0239] Distribute the first index group and its corresponding data to the first target physical cluster; and update the first layer metadata.

[0240] In some embodiments, the apparatus further includes an index read / write control module 305, configured to:

[0241] In response to a user's read / write request for the first index, the link information of the second target physical cluster corresponding to the first index is obtained from the first-level metadata.

[0242] Verify the connectivity of the second target physical cluster;

[0243] After successful verification, the link information of the second target physical cluster will be returned to the user.

[0244] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0245] By implementing the embodiments of this disclosure, a logical cluster composed of multiple physical clusters is managed as a whole. A resource balancing strategy is automatically generated based on the resource utilization of each physical cluster. The resource balancing strategy is used to achieve resource balancing among multiple physical clusters within the logical cluster, thereby realizing the resource self-balancing capability of multiple physical clusters, reducing the metadata pressure of a single physical cluster, and improving storage efficiency.

[0246] Figure 10 This is a block diagram illustrating an apparatus 1900 for cluster management according to an exemplary embodiment. For example, apparatus 1900 may be provided as a server. (Refer to...) Figure 10 The apparatus 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the management methods of the aforementioned method cluster.

[0247] Device 1900 may also include a power supply component 1926 configured to perform power management of device 1900, a wired or wireless network interface 1950 configured to connect device 1900 to a network, and an input / output (I / O) interface 1958. Device 1900 can operate on an operating system stored in memory 1932, such as Windows Server™, MacOS X™, Unix™, Linux™, FreeBSD™, or similar.

[0248] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.

[0249] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A cluster management method, characterized in that: The cluster includes a logical cluster composed of multiple physical clusters, and the management method includes: Obtaining resource utilization of each physical cluster in the logical cluster according to a first preset frequency; When the resource utilization of each physical cluster in the logical cluster meets the self-balancing trigger condition, generating a resource balancing strategy based on the resource utilization; Based on the resource balancing policy, data is migrated within the logical cluster.

2. The method according to claim 1, characterized in that The obtaining of resource utilization of each physical cluster in the logical cluster includes: For each physical cluster in the logical cluster, obtain operating system level monitoring indicator data and distributed system software level monitoring indicator data of each physical cluster; wherein the operating system level monitoring indicator data is a resource occupancy indicator related to the operating system, and the distributed system software level monitoring indicator data is a resource occupancy indicator related to the distributed system software; Based on the operating system level monitoring indicator data and the distributed system software level monitoring indicator data, a weighted resource usage factor of each physical cluster is calculated; wherein the weighted resource usage factor is used to calculate the resource utilization of the physical cluster; The resource utilization rate of each physical cluster is obtained based on the weighted resource utilization factor of each physical cluster and the cluster maximum resource utilization factor of each physical cluster; wherein the cluster maximum resource utilization factor is the upper limit value of resource utilization of the physical cluster.

3. The method according to claim 1, characterized in that Generating a resource balancing strategy based on the resource utilization rate includes: Determine a migration-out cluster list and a migration-in cluster list based on resource utilization of each physical cluster in the logical cluster, a preset migration-out threshold, and a preset migration-in threshold; For each outgoing cluster in the outgoing cluster list, determining an outgoing index group list of each outgoing cluster; wherein the outgoing index group list includes a plurality of pre-migrated index groups; Determine the current target incoming cluster from the incoming cluster list, and for each outgoing cluster in the outgoing cluster list, pre-calculate the resource utilization of the current outgoing cluster and the current target incoming cluster if the current index group in the outgoing index group list of the current outgoing cluster is migrated to the current target incoming cluster; If the resource utilization of the current target incoming cluster meets the first preset condition, a migration sub-plan is generated, and the resource utilization of the current outgoing cluster and the current target incoming cluster is updated based on the pre-calculated resource utilization of the current outgoing cluster and the current target incoming cluster, and the current target incoming cluster is re-determined from the incoming cluster list based on the resource utilization of the current target incoming cluster; until the resource utilization of the current outgoing cluster meets the second preset condition, or the multiple index groups in the outgoing index group list of the current outgoing cluster are all pre-calculated; Based on the migration sub-scheme, a resource balancing strategy is generated.

4. The method according to claim 3, characterized in that The determining of the migration-out cluster list and the migration-in cluster list based on the resource utilization rate of each physical cluster in the logical cluster, the preset migration-out threshold, and the preset migration-in threshold includes: sorting the physical clusters based on resource utilization of the physical clusters within the logical cluster; Sort the physical clusters whose resource utilization is greater than the preset migration threshold in descending order to obtain a migration-out cluster list; The physical clusters whose resource utilization is less than the preset migration threshold are sorted in ascending order to obtain a migration cluster list.

5. The method according to claim 4, characterized in that The step of determining the current target migration cluster from the migration cluster list includes: From the migration-in cluster list, a migration-in cluster with the lowest resource utilization is determined as the current target migration-in cluster.

6. The method according to claim 3, characterized in that The determining of the migration index group list of each migration cluster includes: Determine the maximum impact factor of resource utilization of the current outgoing cluster from the operating system level monitoring indicator data and the distributed system software level monitoring indicator data of the current outgoing cluster; wherein the maximum impact factor of resource utilization is the resource type that has the greatest impact on resource utilization among the multiple resources of the physical cluster; Based on the maximum resource utilization impact factor, the index groups currently migrated out of the cluster are sorted to obtain a migration out index group list.

7. The method according to claim 1, characterized in that The logical cluster includes a first layer of metadata, where the first layer of metadata is used to manage cluster information of multiple physical clusters within the logical cluster and a correspondence between each index group within the logical cluster and the multiple physical clusters; The performing data balance migration in the logical cluster based on the resource balancing strategy includes: Based on the resource balancing strategy, index metadata and corresponding master data are migrated within the logical cluster, and the first-layer metadata is updated.

8. The method according to claim 1, characterized in that The method further comprises: When the resource balancing strategy is empty, a cluster expansion alarm message is sent.

9. The method according to claim 1, characterized in that: The method further comprises: When it is determined that the resource utilization of each physical cluster in the logical cluster reaches a bottleneck threshold, cluster expansion alarm information is sent.

10. The method according to claim 1, characterized in that The method further comprises: According to a second preset frequency, obtaining multiple real-time alarm indicator data; Based on the multiple real-time alarm indicator data and their corresponding alarm thresholds, an alarm message is issued.

11. The method according to claim 1, characterized in that: The logical cluster includes a first layer of metadata, where the first layer of metadata is used to manage cluster information of multiple physical clusters within the logical cluster and a correspondence between each index group within the logical cluster and the multiple physical clusters; The method further comprises: In response to the received registration application of the first index group, determining a physical cluster with the lowest resource utilization in the logical cluster as a first target physical cluster; Distribute the first index group and corresponding data to the first target physical cluster; and update the first layer metadata.

12. The method according to claim 11, characterized in that The method further comprises: In response to a read / write request of a user terminal for a first index, acquiring link information of a second target physical cluster corresponding to the first index from the first layer metadata; Verifying connectivity of the second target physical cluster; After the verification is passed, the link information of the second target physical cluster is returned to the user end.

13. A cluster management device, characterized in that: The cluster includes a logical cluster composed of multiple physical clusters, and the management device includes: A data acquisition module, used to acquire the resource utilization rate of each physical cluster in the logical cluster according to a first preset frequency; A scheduling strategy generation module, configured to generate a resource balancing strategy based on the resource utilization when the resource utilization of each physical cluster in the logical cluster meets the self-balancing trigger condition; The scheduling strategy execution module is used to migrate data within the logical cluster based on the resource balancing strategy.

14. A server, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: include: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images