Control method and device of multi-cluster system, computer equipment and storage medium

Abstract

The embodiment of the invention relates to a control method and device for a multi-cluster system, computer equipment and a storage medium, and the method comprises the steps: obtaining the resource state information of each cluster from a target database when a task is received; determining a first cluster from the plurality of clusters according to the resource state information of the plurality of clusters, so as to allocate the task to the first cluster, and to allocate the task to the first cluster; when it is detected that the first cluster breaks down, resource state information of each cluster is obtained again, and a second cluster is determined from the multiple clusters except the first cluster according to the resource state information obtained again; and distributing the tasks which are not executed completely in the first cluster to the second cluster for continuous execution. Therefore, cluster resources can be reasonably allocated when the multi-cluster system processes tasks, so that system-level fault tolerance is realized and service continuity is guaranteed through multi-cluster redundancy and automatic failover when a single cluster fails.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of distributed computing technology, and in particular to a control method, apparatus, computer equipment and storage medium for a multi-cluster system. Background Technology

[0002] In current distributed computing systems, Ray, as a high-performance distributed execution framework, is widely used in scenarios such as machine learning, graph computing, and large-scale task scheduling. Existing Ray frameworks primarily employ fault tolerance mechanisms within a single cluster to ensure high task availability. Specifically, they monitor node status through a Global Control Store (GCS) and combine this with mechanisms such as Actor restart, task retries, and object replica storage for error recovery. When a node in the cluster fails, GCS can reschedule the task to other healthy nodes. However, this approach lacks effective mechanisms to cope with cluster-level failures (such as GCS service crashes, network isolation, and resource exhaustion), thus limiting the overall system availability and making it difficult to exceed the limits of a single cluster in terms of resource utilization.

[0003] In summary, the traditional Ray architecture relies on a single GCS (Global Cluster Controller), and if the GCS fails, the entire cluster becomes unavailable. Therefore, in the process of processing tasks in a distributed multi-cluster system, how to improve the global fault tolerance of the system by reasonably allocating cluster resources to achieve the effect of a cluster failure has become an urgent problem to be solved. Summary of the Invention

[0004] In view of this, in order to solve the above-mentioned technical problems or some of the technical problems, the present invention provides a control method, apparatus, computer equipment and storage medium for a multi-cluster system.

[0005] In a first aspect, embodiments of the present invention provide a control method for a multi-cluster system, comprising:

[0006] When a task is received, the resource status information of each cluster is retrieved from the target database;

[0007] A first cluster is determined from the multiple clusters based on the resource status information of the multiple clusters, so as to allocate the task to the first cluster;

[0008] When a failure is detected in the first cluster, the resource status information of each cluster is reacquired, and a second cluster is determined from multiple clusters other than the first cluster based on the reacquired resource status information.

[0009] Tasks that have not been completed in the first cluster are reassigned to the second cluster for continued execution.

[0010] In one possible implementation, determining the first cluster from the plurality of clusters based on the resource status information of the plurality of clusters includes:

[0011] Obtain the weight information corresponding to each resource status information, wherein the resource status information includes at least one of the following: CPU utilization, memory usage, task queue length, resource fragmentation rate, cluster geographical location, and network latency with the user terminal;

[0012] For each cluster, a cluster score is determined based on the resource status information and the corresponding weight information.

[0013] The cluster with the highest cluster score among the multiple clusters is selected as the first cluster.

[0014] In one possible implementation, before retrieving the resource status information for each cluster from the target database, the method further includes:

[0015] In the target database, a corresponding namespace is set for each cluster;

[0016] At each first time interval, control each cluster to send resource status information to the target database;

[0017] The target database is controlled to store the resource status information of each cluster into the namespace corresponding to the cluster.

[0018] In one possible implementation, detecting a failure in the first cluster includes:

[0019] Every second time interval, a heartbeat signal sent by the first cluster is received, the heartbeat signal indicating whether the first cluster is operating normally;

[0020] The health score of the first cluster is determined based on the heartbeat signal and the current resource status information of the first cluster.

[0021] When the health score is lower than the score threshold, it is determined that the first cluster has failed, new tasks are stopped being assigned to the first cluster, and the first cluster is marked as faulty.

[0022] In one possible implementation, the step of allocating unfinished tasks from the first cluster to the second cluster for continued execution includes:

[0023] Obtain a state snapshot of the unfinished task, wherein the state snapshot represents the task state obtained at every third time interval during the historical processing of the task;

[0024] The unfinished tasks are assigned to the second cluster;

[0025] The second cluster is controlled to continue executing the unfinished tasks based on the state snapshot.

[0026] In one possible implementation, the method further includes:

[0027] Load information is periodically obtained for each of the aforementioned clusters;

[0028] The cluster category corresponding to each cluster in each cycle is determined based on the load information;

[0029] Within each cycle, each cluster is either expanded or reduced in size according to its cluster category.

[0030] Secondly, embodiments of the present invention provide a multi-cluster system for implementing the method described in the first aspect above, the system comprising: multiple clusters, a target database, and a control unit;

[0031] The cluster is used to send resource status information to the target database;

[0032] The target database is used to store the status information of each resource in the namespace corresponding to each cluster;

[0033] The control unit is configured to, when a task is received, determine a first cluster from the multiple clusters based on the resource status information of the multiple clusters, and allocate the task to the first cluster;

[0034] The control unit is further configured to, when a failure is detected in the first cluster, determine a second cluster from multiple clusters other than the first cluster based on the resource status information of each of the clusters that are currently being reacquired, and control the first cluster to allocate the tasks to be completed to the second cluster for continued execution.

[0035] Thirdly, embodiments of the present invention provide a control device for a multi-cluster system, comprising:

[0036] The acquisition module is used to retrieve resource status information for each cluster from the target database when a task is received.

[0037] A determining module is configured to determine a first cluster from the multiple clusters based on the resource status information of the multiple clusters, so as to allocate the task to the first cluster;

[0038] The determining module is further configured to, when a failure is detected in the first cluster, reacquire the resource status information of each cluster, and determine a second cluster from multiple clusters other than the first cluster based on the reacquired resource status information.

[0039] The allocation module is used to allocate unfinished tasks in the first cluster to the second cluster for continued execution.

[0040] Fourthly, embodiments of the present invention provide a computer device, including: a processor and a memory, wherein the processor is configured to execute a control program for a multi-cluster system stored in the memory, so as to implement the control method for the multi-cluster system described in any one of the first aspects.

[0041] Fifthly, embodiments of the present invention provide a storage medium storing one or more programs, which can be executed by one or more processors to implement the control method of the multi-cluster system described in any one of the first aspects.

[0042] The control scheme for a multi-cluster system provided in this invention involves the following steps: When a task is received, resource status information of each cluster is obtained from a target database; a first cluster is determined from the multiple clusters based on the resource status information of the multiple clusters, and the task is assigned to the first cluster; when a failure is detected in the first cluster, the resource status information of each cluster is re-obtained, and a second cluster is determined from the multiple clusters other than the first cluster based on the re-obtained resource status information; and unfinished tasks in the first cluster are assigned to the second cluster for continued execution. Therefore, cluster resources can be rationally allocated when the multi-cluster system processes tasks, so that when a single cluster fails, system-level fault tolerance is achieved through multi-cluster redundancy and automatic failover, ensuring service continuity. Attached Figure Description

[0043] Figure 1 A flowchart illustrating a control method for a multi-cluster system provided in an embodiment of the present invention;

[0044] Figure 2 A flowchart illustrating another control method for a multi-cluster system provided in an embodiment of the present invention;

[0045] Figure 3 A flowchart illustrating another control method for a multi-cluster system provided in an embodiment of the present invention;

[0046] Figure 4 This is a schematic diagram of the structure of a multi-cluster system provided in an embodiment of the present invention;

[0047] Figure 5 This is a schematic diagram of the structure of a control device for a multi-cluster system provided in an embodiment of the present invention;

[0048] Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] To facilitate understanding of the embodiments of the present invention, further explanations and descriptions will be provided below with reference to the accompanying drawings and specific embodiments. These embodiments do not constitute a limitation on the embodiments of the present invention.

[0051] Figure 1 This is a flowchart illustrating a control method for a multi-cluster system provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the method specifically includes:

[0052] S11. When a task is received, retrieve the resource status information of each cluster from the target database.

[0053] The control method for a multi-cluster system provided in this invention is applied to computer equipment and may include: control units, servers, desktop computers, etc. of a distributed multi-cluster system. Specifically, it can be applied to scenarios such as large-scale data processing, artificial intelligence training, and real-time data analysis. Enterprises typically need to deploy multiple Ray distributed computing clusters to meet the business requirements of high concurrency, low latency, and high reliability, such as: financial risk control model training scenarios, multi-tenant data cleaning platforms, cloud-native computing service platforms, etc.

[0054] In this embodiment, the execution entity can be a controller, scheduler, or other control unit of a multi-cluster system. The system's front-end service (such as a RESTful API or message queue) receives a task to be executed, which contains necessary running parameters (such as required CPU / GPU, memory, task type, priority, etc.).

[0055] The control unit can control the extended Ray SDK (Ray Software Development Kit) to query the current resource status information of each registered Ray cluster from the target database (e.g., Redis database) which serves as the global metadata center. This information may include, but is not limited to, the current available resources of the cluster (such as the number of CPUs, GPUs, and remaining memory), the number of tasks being executed and the length of the pending queue, the current load rate, the resource fragmentation rate, the cluster health score (based on heartbeat signal evaluation), network latency, and other network topology information.

[0056] S12. Determine the first cluster from the multiple clusters based on the resource status information of multiple clusters, so as to allocate tasks to the first cluster.

[0057] In this embodiment, the system can evaluate the resource status of multiple available clusters, then dynamically select the optimal cluster as the first cluster, and assign the task to that cluster for execution.

[0058] Specifically, after obtaining the resource status information of each cluster, unavailable clusters (such as those that are down, downgraded, or under maintenance) are filtered out first; clusters with acceptable health are retained as candidate clusters. The system uses a scheduling scoring algorithm to score and rank the candidate clusters. Scoring factors may include: remaining resources (the more the better), current number of tasks (the fewer the better), geographical location and user proximity (the lower the latency the better), historical stability (the lower the failure rate the better), and resource utilization (the less fragmentation the better). A weighted model or machine learning model (such as a reinforcement learning-based scheduling strategy) can be used to comprehensively evaluate multiple factors to obtain an evaluation score for each cluster. The cluster with the highest score is selected as the first cluster for recording the allocation results for subsequent tracking and monitoring.

[0059] Furthermore, the system sends the task to the first cluster via Ray SDK or API, and the first cluster's GCS and node manager handle resource allocation and task execution.

[0060] As an example, the resource status information of cluster A includes: CPU utilization of 80% and 10 pending tasks; the resource status information of cluster B includes: CPU utilization of 40% and 2 pending tasks; and the resource status information of cluster C includes: CPU utilization of 60% and 5 pending tasks. The intelligent scheduling algorithm will consider these factors and select cluster B as the first cluster because it has a lower load and more available resources.

[0061] S13. When a failure is detected in the first cluster, the resource status information of each cluster is reacquired, and the second cluster is determined from multiple clusters other than the first cluster based on the reacquired resource status information.

[0062] In this embodiment, each cluster periodically sends heartbeat signals to the target database or control unit, which include node status, resource usage, etc. A fault is determined based on the heartbeat signals; for example, a fault is determined when a heartbeat times out (e.g., no heartbeat received for N consecutive times), the cluster health index falls below a set threshold, or the task execution failure rate spikes. Once the first cluster is found to be unavailable, the control unit immediately accesses the target database to obtain the latest resource status information of the other clusters besides the first cluster. Any clusters that are unavailable or under maintenance are removed. A second cluster is determined based on the resource status information of the remaining available clusters. This second cluster is used to receive and process tasks that have not been completed in the first cluster. The method for determining the second cluster is the same as the method for determining the first cluster in step S12.

[0063] S14. Distribute the unfinished tasks in the first cluster to the second cluster for continued execution.

[0064] In this embodiment, unfinished tasks may include, but are not limited to: tasks that are currently running but have been interrupted, tasks that have been scheduled but not yet started, and tasks waiting for results to be returned. All unfinished tasks in the first cluster are scheduled to the second cluster so that the second cluster can continue to execute the scheduled tasks.

[0065] Specifically, for stateful tasks already executed in the first cluster, execution continues in the second cluster following the previously executed steps (restoring the context and data environment); for stateless tasks not yet executed in the first cluster, they can be executed directly in the second cluster. If the task requires data source access, the system will automatically configure the network or credentials so that the second cluster can access the original data.

[0066] Optionally, in the current multi-cluster system, if any cluster fails, the method of this embodiment can be used to determine the second cluster based on the resource status information of the non-failed cluster, and schedule the unfinished tasks in the failed cluster to the second cluster.

[0067] The control method for a multi-cluster system provided in this invention involves the following steps: When a task is received, resource status information of each cluster is obtained from a target database; a first cluster is determined from multiple clusters based on the resource status information of multiple clusters for task allocation; when a failure is detected in the first cluster, resource status information of each cluster is re-obtained, and a second cluster is determined from multiple clusters other than the first cluster based on the re-obtained resource status information; unfinished tasks in the first cluster are allocated to the second cluster for continued execution. Therefore, cluster resources can be rationally allocated when the multi-cluster system processes tasks, so that when a single cluster fails, system-level fault tolerance is achieved through multi-cluster redundancy and automatic failover, ensuring service continuity.

[0068] Figure 2 A flowchart illustrating another control method for a multi-cluster system provided in an embodiment of the present invention is shown below. Figure 2 As shown, the method specifically includes:

[0069] S21. Set up a corresponding namespace for each cluster in the target database; control each cluster to send resource status information to the target database at the first time interval; control the target database to store the resource status information of each cluster in the namespace corresponding to the cluster.

[0070] In this embodiment, a namespace is a logical isolation mechanism used to distinguish data from different clusters within the same database. It can be implemented through Redis key prefixes or MongoDB collection naming. Taking Redis as an example: Cluster A uses the namespace cluster:A, and Cluster B uses the namespace cluster:B. Each namespace can store the resource status information of the cluster, for example: cluster:A:cpu=8, cluster:A:memory=32GB, cluster:A:task_queue=15.

[0071] The monitoring service or GCS in each cluster performs periodic heartbeat reporting, with a period of a first duration (e.g., 5 seconds, 10 seconds). The reported content is resource status information, which may include, but is not limited to, at least one of the following: CPU utilization (CPU / GPU utilization or the number of currently available CPUs / GPUs), memory usage (free memory), task queue length (PendingJob queue length, representing the number of running tasks and queued tasks), resource fragmentation rate (fragmentation can be reduced through a remaining resource distribution model (e.g., a Bayesian-optimized resource matching algorithm), cluster geographical location (distance from the user's geographical location), network latency to the user's end, node health status, etc.

[0072] After each cluster reports its resource status information, the target database automatically stores it according to the namespace rules. The namespace acts as a logical index, which facilitates querying the resource status of a specified cluster and achieving isolation and concurrent access to multiple cluster statuses. For example, the system can directly query the resource status information of a cluster from Redis, which is convenient for subsequent intelligent scheduling, disaster recovery switching, and elastic scaling control.

[0073] S22. When a task is received, retrieve the resource status information of each cluster from the target database.

[0074] In this embodiment, the procedure is similar to step S11, and can be referred to for details. Figure 1 For the sake of brevity, the relevant content will not be elaborated upon here.

[0075] S23. Obtain the weight information corresponding to each resource status information; for each cluster, determine the cluster score based on each resource status information and the corresponding weight information; determine the cluster with the highest cluster score from multiple clusters as the first cluster.

[0076] In this embodiment, the weight information can be preset by the system or dynamically adjusted to express the importance of each type of resource in the current task scenario. The greater the impact of resource status information on the cluster's ability to process tasks, the higher the weight. For example, CPU utilization weight = 0.4, memory weight = 0.2, task queue length weight = 0.3, and network latency weight = 0.1. The weights can be stored in the target database or configuration center and support dynamic configuration.

[0077] After the resource status information reported by each cluster is collected, its comprehensive cluster score can be calculated according to the following formula: Score = w1×f1 + w2×f2 + ... + wn×fn, where: w1 to wn are the weights of various types of resource status information, and f1 to fn are the standardized values ​​of each resource dimension (such as CPU remaining ratio, inverse of memory utilization, inverse ratio of task queue load, etc.).

[0078] The cluster scores are compared, and the cluster with the highest score is selected as the first cluster. A threshold or sliding window mechanism can be set to prevent frequent switching of the first cluster.

[0079] S24. Every second time interval, receive the heartbeat signal sent by the first cluster; determine the health score of the first cluster based on the heartbeat signal and the current resource status information of the first cluster; when the health score is lower than the score threshold, determine that the first cluster has failed, stop assigning new tasks to the first cluster, and perform fault marking processing on the first cluster.

[0080] In this embodiment, the system sets a second duration (e.g., every 10 seconds) as the heartbeat detection cycle. Each cluster sends a heartbeat signal (heartbeat packet) to the target database through a background daemon thread or monitoring module, which may include basic operating indicators such as cluster identifier ID, timestamp, number of nodes, and activity rate.

[0081] The control unit receives the heartbeat signal and records the time and content. If no heartbeat signal is received for N consecutive cycles, it is considered a heartbeat interruption. A weighted score can be calculated based on the heartbeat signal and resource status information. For example, a normal heartbeat results in a higher score, CPU saturation results in a lower score, and queue backlog exceeding a set threshold results in a lower score. The weights can be dynamically configured or set based on historical training models. The system derives a comprehensive health score, such as 0-100, based on the scoring rules.

[0082] The system presets a scoring threshold (e.g., 40 points). When the health score of the first cluster is lower than the scoring threshold, it is determined that the first cluster has failed. New tasks are stopped being assigned to the first cluster, and the first cluster is removed from the scheduling candidate pool. The cluster status is marked as Degraded or Failed in the target database, and the failure time, indicator details, etc. are recorded. The system can also link with the operation and maintenance alarm system to send notifications, preparing for subsequent task migration and failure recovery (e.g., snapshot recovery or cluster replacement).

[0083] S25. When a failure is detected in the first cluster, the resource status information of each cluster is reacquired, and the second cluster is determined from multiple clusters other than the first cluster based on the reacquired resource status information.

[0084] In this embodiment, the procedure is similar to step S13, and can be referred to for details. Figure 1 For the sake of brevity, the relevant content will not be elaborated upon here.

[0085] S26. Obtain a state snapshot of the unfinished tasks. The state snapshot represents the task status obtained at every third time interval during the historical task processing. Assign the unfinished tasks to the second cluster. Control the second cluster to continue executing the unfinished tasks according to the state snapshot.

[0086] In this embodiment, a state snapshot is an intermediate execution state periodically saved during task execution. It may include: task processing progress information, completed subtasks, memory context, data cache, dependency graph state, etc. During task execution, at preset third intervals (e.g., every 60 seconds), the task running node submits the current task state to the snapshot service. The checkpoint library or Ray's built-in actor state persistence function can be used. State snapshots are uniformly stored in a shared storage system (such as HDFS, S3, Ceph) or a metadata database for easy subsequent querying and migration.

[0087] After the system detects a failure in the first cluster, it first retrieves its list of incomplete tasks, obtains the latest status snapshot for each task, and selects the second cluster with the highest score as the migration target. The control scheduling module then redistributes these incomplete tasks (with status snapshots) to the second cluster. This can be achieved using the task rescheduling interface provided by the extended Ray SDK.

[0088] After receiving the task request and its state snapshot, the second cluster initializes the task execution environment through the recovery API; it reconstructs the task context based on the snapshot content, restoring the computation graph, variable states, cache dependencies, etc.; it continues the execution logic of the original task, processing subsequent data from the interruption point; it supports mechanisms such as breakpoint continuation and idempotent execution to ensure the correctness of the final processing result.

[0089] In one possible implementation, load information is periodically acquired for each cluster; the cluster category corresponding to each cluster in each period is determined based on the load information; and in each period, expansion or contraction operations are performed on each cluster according to the cluster category.

[0090] In this embodiment, load information may include, but is not limited to: CPU utilization, GPU utilization, memory usage, number of currently running tasks, length of the pending task queue, network throughput, I / O pressure, etc. At preset time intervals, or for different time periods each day, the cluster reports load metrics through the monitoring module; the data can be written to a namespace; or it can be collected by connecting to metrics services such as Kubernetes.

[0091] Clusters can be categorized based on load information as follows: High Load Cluster: CPU or memory utilization is higher than the threshold (e.g., >80%), and the task queue continues to accumulate; Low Load Cluster: Resource utilization is consistently lower than the threshold (e.g., <20%), and the proportion of idle tasks is large; Normal Cluster: Resource utilization is within a reasonable range, and task processing is stable.

[0092] When classifying clusters, a cluster classification service can be used to read the load information within the current time period. Based on preset rules and weight models (such as Bayesian classification, threshold judgment, and moving average), each cluster category is determined to obtain the cluster category corresponding to each cluster in different time periods (for example, cluster A is a hot cluster from 10:00 to 12:00, and cluster A is a cold cluster from 12:00 to 2:00).

[0093] If the cluster is hot, the Kubernetes API or cloud service API is called to perform node scaling. Compute nodes, Pod replicas, and the number of Ray workers can be dynamically added (e.g., for a hot cluster, when the load is >70%, worker nodes are automatically scaled up via Kubernetes HPA (HorizontalPod Autoscaler)). If the cluster is cold, low-utilization nodes can be released and the number of worker processes reduced. A graceful shutdown mechanism can be used to ensure that tasks on scaled-down nodes are migrated before resources are released (e.g., for a cold cluster, when the load is <30% for 10 minutes, it is marked as sleepable, and non-persistent resources (such as Spot instances) are released). Scaling-down strategies can be embedded in the Ray SDK or deployed as a resource elasticity control service, supporting more intelligent periodic adjustments based on predictive models (such as ARIMA and LSTM).

[0094] As an example Figure 3 A flowchart illustrating another control method for a multi-cluster system provided in an embodiment of the present invention is shown below. Figure 3As shown, the method specifically includes:

[0095] Multiple clusters operate normally, each continuously executing tasks and periodically sending heartbeat signals to a central database (such as Redis) to indicate its operational status. The control unit continuously monitors cluster heartbeat signals. If a cluster fails to report a heartbeat for an extended period (timeout), it determines that the cluster has failed or is abnormal. To prevent new tasks from being assigned to the failed cluster, the system immediately performs a de-tapping operation: terminating the dispatch of new tasks to that cluster; and marking the cluster as Degraded in the metadata center (Redis). The control unit begins migrating tasks that have not yet been completed in that cluster. The specific process includes: Checkpointing / incremental logging: obtaining a snapshot of the task's status or execution trajectory from shared storage or log systems such as Kafka; restoring the state from shared storage: restoring the task to the latest breakpoint, ensuring no duplication or loss of execution content. The task is redirected to a new cluster for execution. The system reselects the target cluster (second cluster) with the best health from the clusters other than the failed cluster and reschedules the tasks to be restored to that cluster for execution. After a successful migration, the system will update the task status and cluster allocation information in Redis; overall service capabilities will be restored, user tasks will continue to execute normally, and the disaster recovery process will end.

[0096] The control method for a multi-cluster system provided in this invention introduces a Redis-based global metadata center and health monitoring mechanism, achieving real-time status synchronization and fault detection across clusters. Compared to the traditional Ray architecture relying on a single GCS, this solution can automatically complete task migration and recovery in the event of a complete cluster failure or large-scale node disconnection, thereby significantly improving system stability and business continuity. The system dynamically calculates the score of each cluster based on multi-dimensional indicators such as cluster resource status, task load, and network topology, and optimizes scheduling by combining geographical location priority and resource fragmentation rate. Compared to static strategies or polling mechanisms, this method achieves higher task scheduling efficiency and resource matching, improving task throughput and processing latency. The system has the ability to identify hot and cold clusters and periodically scale up and down, automatically adjusting resource scale according to changes in cluster load. It can automatically release computing resources during idle periods and rapidly scale up during peak periods, avoiding prolonged resource idleness or congestion, thereby reducing overall operation and maintenance costs and improving resource utilization. By using a task status snapshot mechanism, the processing progress of unfinished tasks can be quickly located after a failure occurs, and breakpoint continuation can be implemented in other clusters, which greatly reduces task retry overhead and recovery latency, meeting the needs of business scenarios with high requirements for real-time performance and consistency.

[0097] Figure 4 This is a schematic diagram of the structure of a multi-cluster system provided in an embodiment of the present invention, used to achieve, for example... Figure 1-2The system shown includes: multiple clusters, a target database, and a control unit;

[0098] Clusters are used to send resource status information to the target database;

[0099] The target database is used to store the status information of each resource in the namespace corresponding to each cluster;

[0100] Control unit: When a task is received, it determines the first cluster from multiple clusters based on the resource status information of multiple clusters, and assigns the task to the first cluster;

[0101] The control unit is also used to determine the second cluster from multiple clusters other than the first cluster based on the resource status information of each cluster that has been reacquired, when a failure is detected in the first cluster, and to control the first cluster to allocate the tasks to be completed to the second cluster for continued execution.

[0102] In this embodiment, the system constructs a distributed computing platform with intelligent scheduling and fault tolerance capabilities for a multi-Ray cluster environment. Its core comprises three main components: multiple clusters, a target database (sharing a Redis metadata center), and a control unit (extending the Ray SDK and scheduling module). The specific structure and workflow are as follows:

[0103] Figure 4 The bottom of the screen displays multiple clusters (Cluster 1, Cluster 2, Cluster N). Each cluster deploys a Ray system instance and includes its own GCS module for local task scheduling and status management. Each cluster periodically sends resource status information to the target database as the foundational data for global system scheduling. The target database (shared Redis) serves as the system's target database and metadata center, maintaining an independent namespace for each cluster; storing corresponding resource status information; and providing real-time data support for task scheduling to the control unit. Each cluster registers its corresponding namespace in Redis upon registration, and all subsequent resource status information is written to this namespace. The control unit (extended Ray SDK + load balancer) Figure 4 The process of a user submitting a job is demonstrated. The extended Ray SDK initiates a task scheduling process, which includes: the control unit queries the resource status information of all clusters from Redis; calculates the score of each cluster based on indicators such as task queue length, resource load rate, and network latency, and selects the first cluster with the highest score from multiple candidate clusters; and assigns the task to the first cluster for execution.

[0104] When the control unit periodically fails to receive a heartbeat signal from a certain cluster (such as the first cluster) or its health score is lower than the threshold, it is considered to have failed: The control unit retrieves the resource status from Redis again; a new second cluster is selected from other clusters besides the first cluster; and through the task snapshot (Checkpoint) mechanism, tasks that have not yet been completed in the first cluster are migrated to the second cluster to continue execution, ensuring that the business is not interrupted.

[0105] This system achieves centralized management of resource status across clusters through Redis and implements cross-cluster task scheduling and fault tolerance mechanisms by extending the Ray SDK. It demonstrates the entire process from user task submission to task scheduling, fault detection, and automatic migration, effectively solving problems such as weak fault tolerance, low resource utilization, and slow fault recovery inherent in traditional single-cluster architectures.

[0106] Figure 5 This is a schematic diagram of the structure of a control device for a multi-cluster system provided in an embodiment of the present invention, as shown below. Figure 5 As shown, the device specifically includes:

[0107] The acquisition module 51 is used to obtain resource status information of each cluster from the target database when a task is received;

[0108] The determining module 52 is used to determine a first cluster from the multiple clusters based on the resource status information of the multiple clusters, so as to allocate the task to the first cluster;

[0109] The determining module is further configured to, when a failure is detected in the first cluster, reacquire the resource status information of each cluster, and determine a second cluster from multiple clusters other than the first cluster based on the reacquired resource status information.

[0110] The allocation module 53 is used to allocate unfinished tasks in the first cluster to the second cluster for continued execution.

[0111] In one possible implementation, the determining module is specifically used to obtain weight information corresponding to each resource status information, wherein the resource status information includes at least one of the following: CPU utilization, memory usage, task queue length, resource fragmentation rate, cluster geographical location, and network latency with the user terminal.

[0112] For each cluster, a cluster score is determined based on the resource status information and the corresponding weight information.

[0113] The cluster with the highest cluster score among the multiple clusters is selected as the first cluster.

[0114] In one possible implementation, the control module 54 is configured to set a corresponding namespace for each cluster in the target database;

[0115] At each first time interval, control each cluster to send resource status information to the target database;

[0116] The target database is controlled to store the resource status information of each cluster into the namespace corresponding to the cluster.

[0117] In one possible implementation, the determining module is further configured to receive a heartbeat signal sent by the first cluster at every second time interval, the heartbeat signal indicating whether the first cluster is operating normally;

[0118] The health score of the first cluster is determined based on the heartbeat signal and the current resource status information of the first cluster.

[0119] When the health score is lower than the score threshold, it is determined that the first cluster has failed, new tasks are stopped being assigned to the first cluster, and the first cluster is marked as faulty.

[0120] In one possible implementation, the allocation module is specifically used to obtain a state snapshot of the unfinished task, the state snapshot representing the task state obtained every third time interval during the historical task processing process;

[0121] The unfinished tasks are assigned to the second cluster;

[0122] The second cluster is controlled to continue executing the unfinished tasks based on the state snapshot.

[0123] In one possible implementation, the acquisition module is further configured to periodically acquire load information for each of the clusters;

[0124] The determining module is further configured to determine the cluster category corresponding to each cluster in each period based on the load information;

[0125] The processing module 55 is used to perform expansion or reduction operations on each cluster according to the cluster category in each cycle.

[0126] The control device for the multi-cluster system provided in this embodiment can be as follows: Figure 5 The apparatus shown can perform, for example Figure 1-2 All steps of the control method for a multi-cluster system, thereby achieving Figure 1-2 For details on the technical effects of the control method for the multi-cluster system shown, please refer to [link / reference]. Figure 1-2 The relevant descriptions are presented concisely and will not be elaborated upon here.

[0127] Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Figure 6 The computer device 600 shown includes at least one processor 601, a memory 602, at least one network interface 604, and other user interfaces 603. The various components in the computer device 600 are coupled together via a bus system 605. It is understood that the bus system 605 is used to implement communication between these components. In addition to a data bus, the bus system 605 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 6 The general designated all buses as Bus System 605.

[0128] The user interface 603 may include a display, keyboard, or clicking device (e.g., mouse, trackball, touchpad, or touchscreen).

[0129] It is understood that the memory 602 in this embodiment of the invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 602 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0130] In some implementations, memory 602 stores elements, executable units or data structures, or subsets thereof, or extended sets thereof: operating system 6021 and application program 6022.

[0131] The operating system 6021 includes various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic business functions and handle hardware-based tasks. The application program 6022 includes various applications, such as a media player and a browser, used to implement various application functions. The program implementing the method of this embodiment can be included in the application program 6022.

[0132] In this embodiment of the invention, by calling the program or instructions stored in memory 602, specifically the program or instructions stored in application program 6022, processor 601 executes the method steps provided in each method embodiment, including, for example:

[0133] When a task is received, the resource status information of each cluster is retrieved from the target database;

[0134] A first cluster is determined from the multiple clusters based on the resource status information of the multiple clusters, so as to allocate the task to the first cluster;

[0135] When a failure is detected in the first cluster, the resource status information of each cluster is reacquired, and a second cluster is determined from multiple clusters other than the first cluster based on the reacquired resource status information.

[0136] Tasks that have not been completed in the first cluster are reassigned to the second cluster for continued execution.

[0137] In one possible implementation, weight information corresponding to each resource status information is obtained, wherein the resource status information includes at least one of the following: CPU utilization, memory usage, task queue length, resource fragmentation rate, cluster geographical location, and network latency with the user terminal.

[0138] For each cluster, a cluster score is determined based on the resource status information and the corresponding weight information.

[0139] The cluster with the highest cluster score among the multiple clusters is selected as the first cluster.

[0140] In one possible implementation, a corresponding namespace is set up in the target database for each cluster;

[0141] At each first time interval, control each cluster to send resource status information to the target database;

[0142] The target database is controlled to store the resource status information of each cluster into the namespace corresponding to the cluster.

[0143] In one possible implementation, at every second time interval, a heartbeat signal sent by the first cluster is received, the heartbeat signal indicating whether the first cluster is operating normally;

[0144] The health score of the first cluster is determined based on the heartbeat signal and the current resource status information of the first cluster.

[0145] When the health score is lower than the score threshold, it is determined that the first cluster has failed, new tasks are stopped being assigned to the first cluster, and the first cluster is marked as faulty.

[0146] In one possible implementation, a state snapshot of the unfinished task is obtained, the state snapshot representing the task state obtained at every third time interval during the historical processing of the task.

[0147] The unfinished tasks are assigned to the second cluster;

[0148] The second cluster is controlled to continue executing the unfinished tasks based on the state snapshot.

[0149] In one possible implementation, load information is periodically acquired for each of the clusters;

[0150] The cluster category corresponding to each cluster in each cycle is determined based on the load information;

[0151] Within each cycle, each cluster is either expanded or reduced in size according to its cluster category.

[0152] The methods disclosed in the above embodiments of the present invention can be applied to or implemented by processor 601. Processor 601 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware or by instructions in the form of software in processor 601. The processor 601 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present invention can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software units in the decoding processor. The software units may be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 602. Processor 601 reads the information in memory 602 and, in conjunction with its hardware, completes the steps of the above method.

[0153] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0154] For software implementation, the techniques described herein can be implemented by units that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0155] The computer device provided in this embodiment may be as follows: Figure 6 The device shown can perform, for example Figure 1-2 All steps of the control method for a multi-cluster system, thereby achieving Figure 1-2 For details on the technical effects of the control method for the multi-cluster system shown, please refer to [link / reference]. Figure 1-2 The relevant descriptions are presented concisely and will not be elaborated upon here.

[0156] This invention also provides a storage medium (computer-readable storage medium). This storage medium stores one or more programs. The storage medium may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as read-only memory, flash memory, hard disk, or solid-state drive; the memory may also include combinations of the above types of memory.

[0157] When one or more programs in the storage medium can be executed by one or more processors, the control method of the multi-cluster system executed on the device side described above can be implemented.

[0158] The processor is used to execute the control program for the multi-cluster system stored in the memory to implement the following steps of the control method for the multi-cluster system executed on the device side:

[0159] When a task is received, the resource status information of each cluster is retrieved from the target database;

[0160] A first cluster is determined from the multiple clusters based on the resource status information of the multiple clusters, so as to allocate the task to the first cluster;

[0161] When a failure is detected in the first cluster, the resource status information of each cluster is reacquired, and a second cluster is determined from multiple clusters other than the first cluster based on the reacquired resource status information.

[0162] Tasks that have not been completed in the first cluster are reassigned to the second cluster for continued execution.

[0163] In one possible implementation, weight information corresponding to each resource status information is obtained, wherein the resource status information includes at least one of the following: CPU utilization, memory usage, task queue length, resource fragmentation rate, cluster geographical location, and network latency with the user terminal.

[0164] For each cluster, a cluster score is determined based on the resource status information and the corresponding weight information.

[0165] The cluster with the highest cluster score among the multiple clusters is selected as the first cluster.

[0166] In one possible implementation, a corresponding namespace is set up in the target database for each cluster;

[0167] At each first time interval, control each cluster to send resource status information to the target database;

[0168] The target database is controlled to store the resource status information of each cluster into the namespace corresponding to the cluster.

[0169] In one possible implementation, at every second time interval, a heartbeat signal sent by the first cluster is received, the heartbeat signal indicating whether the first cluster is operating normally;

[0170] The health score of the first cluster is determined based on the heartbeat signal and the current resource status information of the first cluster.

[0171] When the health score is lower than the score threshold, it is determined that the first cluster has failed, new tasks are stopped being assigned to the first cluster, and the first cluster is marked as faulty.

[0172] In one possible implementation, a state snapshot of the unfinished task is obtained, the state snapshot representing the task state obtained at every third time interval during the historical processing of the task.

[0173] The unfinished tasks are assigned to the second cluster;

[0174] The second cluster is controlled to continue executing the unfinished tasks based on the state snapshot.

[0175] In one possible implementation, load information is periodically acquired for each of the clusters;

[0176] The cluster category corresponding to each cluster in each cycle is determined based on the load information;

[0177] Within each cycle, each cluster is either expanded or reduced in size according to its cluster category.

[0178] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0179] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0180] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A control method for a multi-cluster system, characterized in that, include: When a task is received, the resource status information of each cluster is retrieved from the target database; A first cluster is determined from the multiple clusters based on the resource status information of the multiple clusters, so as to allocate the task to the first cluster; When a failure is detected in the first cluster, the resource status information of each cluster is reacquired, and a second cluster is determined from multiple clusters other than the first cluster based on the reacquired resource status information. Tasks that have not been completed in the first cluster are reassigned to the second cluster for continued execution.

2. The method according to claim 1, characterized in that, The step of determining the first cluster from the multiple clusters based on the resource status information of the multiple clusters includes: Obtain the weight information corresponding to each resource status information, wherein the resource status information includes at least one of the following: CPU utilization, memory usage, task queue length, resource fragmentation rate, cluster geographical location, and network latency with the user terminal. For each cluster, a cluster score is determined based on the resource status information and the corresponding weight information. The cluster with the highest cluster score among the multiple clusters is selected as the first cluster.

3. The method according to claim 1, characterized in that, Before retrieving the resource status information of each cluster from the target database, the method further includes: In the target database, a corresponding namespace is set for each cluster; At each first time interval, control each cluster to send resource status information to the target database; The target database is controlled to store the resource status information of each cluster into the namespace corresponding to the cluster.

4. The method according to claim 1, characterized in that, The detection of a fault in the first cluster includes: Every second time interval, a heartbeat signal sent by the first cluster is received, the heartbeat signal indicating whether the first cluster is operating normally; The health score of the first cluster is determined based on the heartbeat signal and the current resource status information of the first cluster. When the health score is lower than the score threshold, it is determined that the first cluster has failed, new tasks are stopped being assigned to the first cluster, and the first cluster is marked as faulty.

5. The method according to claim 1, characterized in that, The step of assigning unfinished tasks from the first cluster to the second cluster for continued execution includes: Obtain a state snapshot of the unfinished task, wherein the state snapshot represents the task state obtained at every third time interval during the historical processing of the task; The unfinished tasks are assigned to the second cluster; The second cluster is controlled to continue executing the unfinished tasks based on the state snapshot.

6. The method according to claim 1, characterized in that, The method further includes: Load information is periodically obtained for each of the aforementioned clusters; The cluster category corresponding to each cluster in each cycle is determined based on the load information; Within each cycle, each cluster is either expanded or reduced in size according to its cluster category.

7. A multi-cluster system, characterized in that, The system for implementing the method of claim 1 includes: multiple clusters, a target database, and a control unit; The cluster is used to send resource status information to the target database; The target database is used to store the status information of each resource in the namespace corresponding to each cluster; The control unit is configured to, when a task is received, determine a first cluster from the multiple clusters based on the resource status information of the multiple clusters, and allocate the task to the first cluster; The control unit is further configured to, when a failure is detected in the first cluster, determine a second cluster from multiple clusters other than the first cluster based on the resource status information of each of the clusters that are currently being reacquired, and control the first cluster to allocate the tasks to be completed to the second cluster for continued execution.

8. A control device for a multi-cluster system, characterized in that, include: The acquisition module is used to retrieve resource status information for each cluster from the target database when a task is received. A determining module is configured to determine a first cluster from the multiple clusters based on the resource status information of the multiple clusters, so as to allocate the task to the first cluster; The determining module is further configured to, when a failure is detected in the first cluster, reacquire the resource status information of each cluster, and determine a second cluster from multiple clusters other than the first cluster based on the reacquired resource status information. The allocation module is used to allocate unfinished tasks in the first cluster to the second cluster for continued execution.

9. A computer device, characterized in that, include: A processor and a memory, the processor being configured to execute a control program for a multi-cluster system stored in the memory to implement the control method for the multi-cluster system according to any one of claims 1 to 6.

10. A storage medium, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the control method of the multi-cluster system according to any one of claims 1 to 6.

Images