Big data cluster resource manager scheduling method and system

Abstract

The application provides a big data cluster resource manager scheduling method, which can be applied to the technical field of big data. The method comprises the following steps: in response to a batch job request, collecting state information of multiple resource manager scheduling instances in a current cluster in the case that the batch job request does not specify a resource manager scheduling instance; inputting the state information into a hierarchical decision model, determining a target resource manager scheduling instance from the multiple resource manager scheduling instances by executing a first decision of the hierarchical decision model; executing a second decision of the hierarchical decision model to determine a resource allocation strategy when the target resource manager scheduling instance schedules the batch job; wherein the resource allocation strategy is used to indicate an amount of computing resources allocated for the batch job; and routing the batch job request to the target resource manager scheduling instance, so that the target resource manager scheduling instance schedules and executes the batch job according to the resource allocation strategy.

Description

Technical Field

[0001] This application relates to the field of big data technology, specifically to the field of cluster resource scheduling technology, and more specifically to a big data cluster resource manager scheduling method and system. Background Technology

[0002] With the development of big data technology, more and more enterprises are building data lake systems around big data clusters. In big data clusters, the execution of batch jobs relies on the scheduling of the resource manager component. In related technologies, the resource manager component typically adopts a high-availability deployment architecture of one primary and one standby instance, where one primary instance is responsible for scheduling, and a standby instance takes over the service when the primary instance fails. However, as enterprise businesses continue to expand, the number of batch jobs running in the big data cluster continues to grow, and the scheduling requests that the resource manager component needs to handle increase daily, leading to a continuous increase in its load pressure. Under the one-primary-one-standby architecture, the processing capacity of a single primary resource manager instance has an upper limit. When the load reaches this limit, scheduling performance degrades, the submission and scheduling response time of batch jobs increases, and consequently, the data processing efficiency of the entire cluster is affected. Summary of the Invention

[0003] In view of the above problems, embodiments of this application provide a big data cluster resource manager scheduling method, system, electronic device, storage medium, and program product.

[0004] According to a first aspect of this application, a big data cluster resource manager scheduling method is provided, comprising: in response to a batch job request, if the batch job request does not specify a resource manager scheduling instance, collecting status information of multiple resource manager scheduling instances in the current cluster; inputting the status information into a hierarchical decision model, executing a first decision through the hierarchical decision model to determine a target resource manager scheduling instance from the multiple resource manager scheduling instances; executing a second decision through the hierarchical decision model to determine a resource allocation strategy for the target resource manager scheduling instance when scheduling batch jobs; wherein the resource allocation strategy is used to indicate the amount of computing resources allocated to the batch jobs; and routing the batch job request to the target resource manager scheduling instance, so that the target resource manager scheduling instance schedules and executes the batch jobs according to the resource allocation strategy.

[0005] According to an embodiment of this application, the step of performing a first decision through the hierarchical decision model to determine a target resource manager scheduling instance from the plurality of resource manager scheduling instances includes: calculating a scheduling priority score of the resource manager scheduling instance based on the status information of the plurality of resource manager scheduling instances; and determining the target resource manager scheduling instance based on the scheduling priority score.

[0006] According to an embodiment of this application, calculating the scheduling priority score of a resource manager scheduling instance based on the status information of the plurality of resource manager scheduling instances includes: extracting the CPU utilization, memory utilization, and container queue length of each resource manager scheduling instance from the status information; calculating the scheduling priority score of each resource manager scheduling instance based on the CPU utilization, the memory utilization, the container queue length, and a preset weight coefficient; wherein the preset weight coefficient is preset according to the objective of minimizing the sum of the container queue lengths of each resource manager scheduling instance.

[0007] According to an embodiment of this application, the step of executing the second decision through the hierarchical decision model to determine the resource allocation strategy when the target resource manager scheduling instance schedules batch jobs includes: generating multiple candidate resource allocation schemes based on the resource usage status of the node where the target resource manager scheduling instance is located, each candidate resource allocation scheme including the number of CPU cores, memory size, and number of graphics processors; calculating the execution efficiency prediction value of each candidate resource allocation scheme; and determining the resource allocation strategy based on the execution efficiency prediction value.

[0008] According to an embodiment of this application, calculating the predicted execution efficiency of each candidate resource allocation scheme includes: obtaining the CPU utilization, memory utilization, GPU utilization, and data locality score of the node where the target resource manager scheduling instance is located, wherein the data locality score is used to characterize the degree of matching between the storage location of the data blocks required by the batch job and the node where the resource manager scheduling instance is located; and for each candidate resource allocation scheme, calculating the predicted execution efficiency of the candidate resource allocation scheme based on the CPU utilization, memory utilization, GPU utilization, and data locality score.

[0009] According to an embodiment of this application, the method further includes: if the batch job request specifies a resource manager scheduling instance, using the specified resource manager scheduling instance as the target resource manager scheduling instance.

[0010] According to an embodiment of this application, the plurality of resource manager scheduling instances are deployed in a parallel running state.

[0011] According to a second aspect of this application, a big data cluster resource manager scheduling system is provided, comprising: a collection module, configured to, in response to a batch job request, collect status information of multiple resource manager scheduling instances in the current cluster when the batch job request does not specify a resource manager scheduling instance; a first determination module, configured to input the status information into a hierarchical decision model, execute a first decision through the hierarchical decision model, and determine a target resource manager scheduling instance from the multiple resource manager scheduling instances; a second determination module, configured to execute a second decision through the hierarchical decision model, and determine a resource allocation strategy for the target resource manager scheduling instance when scheduling batch jobs; wherein the resource allocation strategy is used to indicate the amount of computing resources allocated to the batch jobs; and a routing module, configured to route the batch job request to the target resource manager scheduling instance, so that the target resource manager scheduling instance schedules and executes the batch jobs according to the resource allocation strategy.

[0012] According to an embodiment of this application, the first determining module includes a first calculation submodule and a first determining submodule.

[0013] The calculation submodule is used to calculate the scheduling priority score of the resource manager scheduling instance based on the status information of the plurality of resource manager scheduling instances; the first determination submodule is used to determine the target resource manager scheduling instance based on the scheduling priority score.

[0014] According to an embodiment of this application, the first calculation submodule is further configured to extract the CPU utilization, memory utilization, and container queue length of each resource manager scheduling instance from the status information; calculate the scheduling priority score of each resource manager scheduling instance based on the CPU utilization, the memory utilization, the container queue length, and a preset weight coefficient; wherein the preset weight coefficient is preset according to the goal of minimizing the sum of the container queue lengths of each resource manager scheduling instance.

[0015] According to an embodiment of this application, the second determining module includes a generation submodule, a second calculation submodule, and a second determining submodule.

[0016] A generation submodule is used to generate multiple candidate resource allocation schemes based on the resource usage status of the node where the target resource manager scheduling instance is located. Each candidate resource allocation scheme includes the number of CPU cores, memory size, and number of graphics processors. A second calculation submodule is used to calculate the execution efficiency prediction value of each candidate resource allocation scheme. A second determination submodule is used to determine the resource allocation strategy based on the execution efficiency prediction value.

[0017] According to an embodiment of this application, the second calculation submodule is further configured to obtain the CPU utilization, memory utilization, GPU utilization, and data locality score of the node where the target resource manager scheduling instance is located. The data locality score is used to characterize the degree of matching between the storage location of the data blocks required by the batch job and the node where the resource manager scheduling instance is located. For each candidate resource allocation scheme, the execution efficiency prediction value corresponding to the candidate resource allocation scheme is calculated based on the CPU utilization, memory utilization, GPU utilization, and data locality score.

[0018] According to a third aspect of this application, an electronic device is provided, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0019] According to a fourth aspect of this application, a computer-readable storage medium is also provided, on which a computer program or instructions are stored, wherein the computer program or instructions, when executed by a processor, implement the steps of the above-described method.

[0020] According to a fifth aspect of this application, a computer program product is also provided, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method. Attached Figure Description

[0021] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0022] Figure 1 The diagram illustrates an application scenario of the big data cluster resource manager scheduling method according to an embodiment of this application.

[0023] Figure 2 The diagram illustrates the system architecture of a big data cluster resource manager scheduling system according to an embodiment of this application.

[0024] Figure 3 A flowchart illustrating a big data cluster resource manager scheduling method according to an embodiment of this application is shown.

[0025] Figure 4 A schematic diagram of the structure of a big data cluster resource manager scheduling system according to an embodiment of this application is shown.

[0026] Figure 5 A block diagram schematically illustrates an electronic device suitable for implementing a big data cluster resource manager scheduling method according to an embodiment of this application. Detailed Implementation

[0027] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0028] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0029] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0030] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0031] To address the aforementioned technical issues, this application provides a method for scheduling resource managers in a big data cluster, comprising: responding to a batch job request, and when the batch job request does not specify a resource manager scheduling instance, collecting status information of multiple resource manager scheduling instances in the current cluster; inputting the status information into a hierarchical decision model, executing a first decision through the hierarchical decision model to determine a target resource manager scheduling instance from the multiple resource manager scheduling instances; executing a second decision through the hierarchical decision model to determine a resource allocation strategy for the target resource manager scheduling instance when scheduling batch jobs; wherein the resource allocation strategy is used to indicate the amount of computing resources allocated to the batch jobs; and routing the batch job request to the target resource manager scheduling instance so that the target resource manager scheduling instance schedules and executes the batch jobs according to the resource allocation strategy.

[0032] Figure 1 The diagram illustrates an application scenario of a big data cluster resource manager scheduling method according to an embodiment of this application. For example... Figure 1 As shown, application scenario 100 according to an embodiment of this application may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables. For example, a user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send information, etc.

[0033] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be electronic devices such as smartphones, wearable devices, personal computers, intelligent voice interaction devices, smart home appliances, intelligent vehicles, in-vehicle terminals, aircraft, unmanned vending terminals, and extended reality devices. Extended reality devices can include virtual reality devices, augmented reality devices, and mixed reality devices. A client application for the target application can be installed and run on the terminal devices. This target application can include, but is not limited to, financial transaction applications, payment applications, shopping applications, web browser applications, search applications, instant messaging tools, email clients, and social media platform software (these are just examples). Furthermore, this application embodiment does not limit the form of the target application, and it can include, but is not limited to, applications, mini-programs, etc., installed on the terminal devices, and can also be in the form of web pages.

[0034] Server 105 can be a server providing various services, such as a backend management server supporting websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process received user requests and other data, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services such as cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and basic cloud computing services such as big data. The server can be the backend server of the aforementioned target application, used to provide backend services to the clients of the target application.

[0035] It should be noted that the big data cluster resource manager scheduling method provided in this application embodiment can generally be executed by server 105 and / or terminal devices 101-103. Accordingly, the big data cluster resource manager scheduling system provided in this application embodiment can generally be set up in server 105 and / or terminal devices 101-103.

[0036] It should be noted that the big data cluster resource manager scheduling method provided in this application embodiment can be used in the field of big data technology, as well as in the field of fintech, and can also be used in any field other than fintech. The application field of the big data cluster resource manager scheduling method provided in this application embodiment is not limited.

[0037] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0038] Figure 2 A schematic diagram illustrating the system architecture of a big data cluster resource manager scheduling system according to an embodiment of this application is provided. Figure 2 As shown in the figure, the big data cluster resource manager scheduling system of this application embodiment includes a decision-maker, a routing module, and multiple sets of resource manager scheduling components. This system deploys multiple sets of resource manager scheduling components, each set containing one primary resource manager instance and one backup resource manager instance, forming a primary-backup high-availability pair. Each set operates independently and in parallel, jointly bearing the batch job scheduling load of the entire cluster. Each set of scheduling components can independently receive and process batch job requests. When the primary instance in a set fails, the backup instance in the same set can quickly take over the service, ensuring the scheduling availability of that set.

[0039] In one example, the decision-maker is the core decision-making unit of the system, responsible for dynamically deciding which group of resource manager scheduling components should be assigned to handle the batch job request based on its characteristics and the current running status of the cluster. Internally, the decision-maker maintains real-time status information for each scheduling component, including metrics such as CPU utilization, memory utilization, container queue length, and network latency for each group of master instances. When a batch job request is received, the decision-maker executes an intelligent selection algorithm based on this status information and outputs the identifier of the target scheduling component.

[0040] In one example, the routing module connects the user submission end with multiple resource manager scheduling components, responsible for receiving batch job requests and forwarding them to the correct target components. The routing module works in conjunction with the decision-maker: when a user submits a batch job, the routing module first calls the decision-maker to obtain the decision result of the target scheduling component; then, based on the decision result, it routes the request to the corresponding scheduling component.

[0041] In one example, each resource manager scheduling component group consists of a primary resource manager and a standby resource manager, forming a primary-standby high-availability pair. The primary resource manager instance is responsible for receiving batch job requests forwarded by the routing module and executing job scheduling operations, including resource allocation, task creation, and status monitoring. The standby resource manager instance is in hot standby mode, synchronizing its scheduling status with the primary instance in real time. It automatically takes over when the primary instance fails, ensuring uninterrupted scheduling service for the group.

[0042] The complete processing flow for batch jobs is as follows: The user submits a batch job request to the system, and the request reaches the routing module. The routing module calls the decision-maker, which, based on the current load status of each scheduling component, decides which group of scheduling components the job should be assigned to. The decision-maker returns the identifier of the target scheduling component (e.g., Group 1). The routing module forwards the batch job request to the primary resource manager instance of Group 1. Upon receiving the request, the primary resource manager instance of Group 1 performs scheduling operations, allocates computing resources to the job, and starts execution. If the primary resource manager instance of Group 1 fails, the backup resource manager instance in the same group automatically takes over and continues to process the jobs being scheduled and already scheduled in that group, ensuring service continuity.

[0043] Figure 3 A flowchart illustrating a big data cluster resource manager scheduling method according to an embodiment of this application is shown. The big data cluster resource manager scheduling method provided in this embodiment is applied to a distributed scheduling system. Unlike the traditional master-slave architecture, multiple resource manager scheduling instances in this distributed scheduling system are deployed in parallel operation, with all instances simultaneously active, collectively bearing the batch job scheduling load from users.

[0044] like Figure 3 As shown, it includes steps S310 to S340.

[0045] In step S310, in response to a batch job request, if the batch job request does not specify a resource manager scheduling instance, the status information of multiple resource manager scheduling instances in the current cluster is collected.

[0046] In one example, the scheduling entry component receives a batch job request submitted by a user. This request includes the job identifier, estimated resource requirements, and data dependency information. The scheduling entry component parses the request and checks if a resource manager scheduling instance is specified. If specified, it is processed using the specified instance; otherwise, subsequent steps are executed.

[0047] In step S320, the status information is input into the hierarchical decision model, and the first decision is executed through the hierarchical decision model to determine the target resource manager scheduling instance from the plurality of resource manager scheduling instances.

[0048] In one example, the scheduling entry component sends a status query command to all deployed resource manager scheduling instances in the cluster. Upon receiving the command, each instance collects its own and its host node's running status, including metrics such as CPU utilization, memory utilization, container queue length, network latency, and data locality, and returns this information to the scheduling entry component. The scheduling entry component aggregates the status information of all instances. It then organizes this collected status information into structured data and inputs it into a pre-built hierarchical decision model. This model contains two decision levels, used for instance selection and resource allocation, respectively. Based on the input status information, the hierarchical decision model executes the first decision. The model analyzes the current load status of each resource manager scheduling instance, evaluates a comprehensive priority for each instance, and selects one instance from multiple instances as the target resource manager scheduling instance based on the priority. This process is entirely automated by the model without manual intervention.

[0049] In step S330, a second decision is made through the hierarchical decision model to determine the resource allocation strategy when the target resource manager scheduling instance schedules batch jobs; wherein, the resource allocation strategy is used to indicate the amount of computing resources allocated to the batch jobs;

[0050] In one example, the hierarchical decision model then executes a second decision. The model obtains resource usage details of the node where the target resource manager scheduling instance resides, and, combined with the resource requirements of the batch job, determines a resource allocation strategy. This strategy indicates the amount of computing resources allocated to the batch job, such as the number of CPU cores and memory size.

[0051] In step S340, the batch job request is routed to the target resource manager scheduling instance, so that the target resource manager scheduling instance schedules and executes the batch job according to the resource allocation policy.

[0052] In one example, the hierarchical decision model then executes a second decision. The model obtains resource usage details of the node where the target resource manager scheduling instance resides, and, combined with the resource requirements of the batch jobs, determines a resource allocation strategy. This strategy indicates the amount of computing resources allocated to the batch jobs, such as the number of CPU cores and memory size. Upon receiving the request and strategy, the target resource manager scheduling instance allocates the corresponding computing resources to the jobs according to the resource allocation strategy and schedules the batch jobs for execution. During execution, the instance monitors the job status until the jobs are completed.

[0053] The method provided in this application achieves global awareness of all scheduling instances in the cluster by collecting status information from multiple resource manager scheduling instances and inputting it into a hierarchical decision model. The first decision dynamically selects the optimal target instance from multiple sets of instances, breaking the performance bottleneck of a single active instance in the traditional master-slave architecture. This allows the scheduling load to be distributed across multiple instances, thereby linearly scaling the cluster's scheduling processing capacity. The second decision determines a resource allocation strategy for the selected instance, ensuring that resource allocation adapts to the actual load of the current node and avoids over- or under-allocation of resources. Separating routing and scheduling execution allows batch job requests to be flexibly distributed to different scheduling instances, achieving distributed deployment of the scheduling component. Overall, this method significantly improves the scheduling throughput and response speed of big data clusters in high-concurrency batch job scenarios.

[0054] According to an embodiment of this application, the step of performing a first decision through the hierarchical decision model to determine a target resource manager scheduling instance from the plurality of resource manager scheduling instances includes: calculating a scheduling priority score of the resource manager scheduling instance based on the status information of the plurality of resource manager scheduling instances; and determining the target resource manager scheduling instance based on the scheduling priority score.

[0055] In one example, when the hierarchical decision-making model executes its first decision, it first extracts key load metrics from the collected status information of each resource manager scheduling instance, including CPU utilization, memory utilization, and container queue length. For each resource manager scheduling instance, the model calculates a scheduling priority score based on these metrics. The calculation can be performed using a weighted linear function: multiplying CPU utilization, memory utilization, and container queue length by their respective weight coefficients and then summing the results to obtain a comprehensive score. CPU utilization and memory utilization reflect the current idleness of the instance's computing resources; lower utilization indicates more available resources and a higher score contribution. Container queue length reflects the number of jobs currently waiting to be scheduled for the instance; shorter queues indicate more idle time for the instance and a higher score contribution. After calculating the scheduling priority scores for all instances, the model compares the scores of each instance and selects the instance with the highest score as the target resource manager scheduling instance.

[0056] By quantifying CPU utilization, memory utilization, and container queue length into scheduling priority scores, a quantitative assessment of the load status of each resource manager scheduling instance is achieved. Selecting target instances based on these scores ensures that newly arriving batch jobs are assigned to the most idle and highest-capacity scheduling instance, thus achieving dynamic load balancing across multiple instances. Compared to random selection or round-robin methods, this approach adaptively responds to real-time load changes in each instance, preventing some instances from becoming overloaded while others remain idle, thereby improving resource utilization and job response fairness across the entire distributed scheduling system.

[0057] According to an embodiment of this application, calculating the scheduling priority score of a resource manager scheduling instance based on the status information of the plurality of resource manager scheduling instances includes: extracting the CPU utilization, memory utilization, and container queue length of each resource manager scheduling instance from the status information; calculating the scheduling priority score of each resource manager scheduling instance based on the CPU utilization, the memory utilization, the container queue length, and a preset weight coefficient; wherein the preset weight coefficient is preset according to the objective of minimizing the sum of the container queue lengths of each resource manager scheduling instance.

[0058] In one example, the hierarchical decision model extracts the CPU utilization (CPU_i), memory utilization (MEM_i), and container queue length (QUEUE_i) of each resource manager scheduling instance from the collected state information. The model has three pre-defined weight coefficients, denoted as α, β, and γ, where α corresponds to the weight coefficient for CPU utilization, β to the weight coefficient for memory utilization, and γ to the weight coefficient for container queue length. These weight coefficients are pre-set according to the optimization objective of "minimizing the sum of container queue lengths of all resource manager scheduling instances." Specifically, historical scheduling data is collected, and a set of weight coefficients is found using reinforcement learning or parameter search methods to minimize the sum of container queue lengths for all instances during historical scheduling. When calculating the scheduling priority score, for the i-th instance, it is calculated according to the formula Score_i = γ * QUEUE_i - (α * CPU_i + β * MEM_i), meaning that the longer the container queue length, the higher the score; and the higher the CPU utilization and memory utilization, the lower the score. Alternatively, a weighted summation method can be used: Score_i = α*(1-CPU_i) + β*(1-MEM_i) + γ*(1- QUEUE_i / QUEUE_max), where QUEUE_max is the preset maximum queue length threshold. After calculation, the instance with the largest Score_i is selected as the target resource manager scheduling instance.

[0059] By introducing preset weighting coefficients and optimizing to minimize the sum of queue lengths for all instances, the calculation of scheduling priority scores has a clear optimization direction. These weighting coefficients can be tuned offline based on the actual operating characteristics of the cluster, adapting to different load patterns (such as compute-intensive, memory-intensive, or hybrid). A weighted calculation method is used to integrate multi-dimensional load metrics into a single scalar score, simplifying the decision-making comparison process. This constraint ensures the mathematical interpretability and optimizability of the first decision, achieving more accurate global load balancing compared to heuristic rules, effectively reducing the average queue length of all scheduled instances, and thus reducing the waiting time for batch jobs.

[0060] According to an embodiment of this application, the step of executing the second decision through the hierarchical decision model to determine the resource allocation strategy when the target resource manager scheduling instance schedules batch jobs includes: generating multiple candidate resource allocation schemes based on the resource usage status of the node where the target resource manager scheduling instance is located, each candidate resource allocation scheme including the number of CPU cores, memory size, and number of graphics processors; calculating the execution efficiency prediction value of each candidate resource allocation scheme; and determining the resource allocation strategy based on the execution efficiency prediction value.

[0061] In one example, when the hierarchical decision model executes the second decision, it first obtains the resource usage status of the node where the target resource manager scheduling instance resides, including the total number of CPU cores, the number of cores used, the total amount of memory, the amount of memory used, and the number and usage of GPUs. Based on this information, the model generates multiple candidate resource allocation schemes. Each candidate resource allocation scheme contains three parameters: the number of CPU cores allocated to the batch job, the memory size, and the number of GPUs. When generating candidate schemes, a grid search approach can be used: within the feasible range of CPU cores (e.g., from 1 core to the remaining cores), the feasible range of memory size (e.g., from 1GB to the remaining memory), and the feasible range of GPU numbers (e.g., from 0 to the remaining GPUs), all possible combinations are enumerated with a certain step size, or a representative set of combinations can be generated using search methods such as genetic algorithms. After generating candidate schemes, the model needs to calculate an execution efficiency prediction value for each scheme. This prediction value represents the expected time or resource utilization efficiency required for the batch job to complete if resources are allocated according to the scheme. The predicted values ​​can be calculated using machine learning regression models (such as neural networks or random forests). The inputs are the number of resources in the candidate solutions and the resource utilization rate of the current node, and the output is the predicted execution time or throughput. After obtaining the predicted execution efficiency values ​​of all candidate solutions, the model selects the solution with the highest predicted value as the final resource allocation strategy.

[0062] By generating multiple candidate resource allocation schemes and evaluating their predicted execution efficiency, this method achieves refined optimization of resource allocation strategies. Compared to fixed resource allocation or simple proportional allocation, this method comprehensively considers the resource requirements of jobs and the real-time available resources of nodes, tailoring the optimal resource quota for each job. The predictive evaluation mechanism avoids resource waste or performance deficiencies caused by blind allocation, making resource allocation decisions forward-looking. The finally selected resource allocation strategy maximizes the overall utilization of node resources while ensuring job execution efficiency, thereby improving the overall job throughput of the cluster.

[0063] According to an embodiment of this application, calculating the predicted execution efficiency of each candidate resource allocation scheme includes: obtaining the CPU utilization, memory utilization, GPU utilization, and data locality score of the node where the target resource manager scheduling instance is located, wherein the data locality score is used to characterize the degree of matching between the storage location of the data blocks required by the batch job and the node where the resource manager scheduling instance is located; and for each candidate resource allocation scheme, calculating the predicted execution efficiency of the candidate resource allocation scheme based on the CPU utilization, memory utilization, GPU utilization, and data locality score.

[0064] In one example, four key metrics are obtained for the node where the target resource manager scheduling instance resides: CPU utilization (cpu_util), memory utilization (mem_util), GPU utilization (gpu_util), and data locality score (loc_score). The data locality score characterizes the degree of matching between the storage locations of the data blocks required by the batch job and the current node: a score of 1.0 is given if all data blocks required by the job are stored on the current node; a score of 0.5 is given if the data blocks are stored on other nodes within the same rack; and a score of 0.0 is given if the data blocks are stored on different racks. For each candidate resource allocation scheme (including the allocated number of CPU cores cpu_alloc, memory size mem_alloc, and number of GPUs gpu_alloc), the model calculates the predicted execution efficiency value for that scheme based on the above four metrics. The specific calculation method is as follows: First, based on the allocated number of cores and memory size, estimate the theoretical time base T_base required for job execution, for example, T_base = f(cpu_alloc, mem_alloc, gpu_alloc), where f is an empirical formula or lookup table function. Then, adjust T_base using the resource utilization of the current node: T_pred = T_base * (1 + k1 * cpu_util + k2 * mem_util + k3 * gpu_util) * (2 - loc_score), where k1, k2, and k3 are positive adjustment coefficients. The adjusted T_pred is the predicted execution time; the smaller the T_pred, the higher the execution efficiency.

[0065] By introducing data locality score as a crucial factor in execution efficiency prediction, resource allocation strategies can become aware of the locational relationship between data and computation. A high data locality score allows jobs to avoid cross-node network transmission, significantly improving execution efficiency and consequently increasing predicted values. This guides the model to prioritize scheduling jobs to the nodes where the data resides and allocating resources accordingly. Simultaneously, by comprehensively considering the utilization of the CPU, memory, and GPU, the model avoids allocating excessive resources to resource-constrained nodes, thus preventing performance degradation. This calculation method integrates multi-dimensional real-time status information into comparable predicted values, providing a quantitative basis for subsequent optimal selection and effectively improving the accuracy of resource allocation decisions and job execution efficiency.

[0066] According to an embodiment of this application, the method further includes: if the batch job request specifies a resource manager scheduling instance, using the specified resource manager scheduling instance as the target resource manager scheduling instance.

[0067] In one example, if the field exists and its value is a valid resource manager scheduling instance identifier (e.g., the instance's IP address, hostname, or unique ID), then the batch job request is determined to specify a resource manager scheduling instance. In this case, the scheduling entry component no longer performs status collection and hierarchical decision model calculations, but directly uses the resource manager scheduling instance specified in the request as the target resource manager scheduling instance. Subsequently, the scheduling entry component routes the batch job request to the specified instance, which then executes the job according to the normal scheduling process. If no instance is specified in the request, a target instance is dynamically selected according to the method described in claim 1.

[0068] The offline pre-training process of the hierarchical decision model is described below with reference to an example.

[0069] According to an embodiment of this application, scheduling-related data is extracted from the historical operation logs of a big data cluster. The historical logs record information such as submission time, resource requirements, data distribution, scheduling decisions, execution duration, and resource usage of past batch jobs. This log data is cleaned and structured in chronological order to form a training sample set. A digital twin environment is constructed; this digital twin environment is a software simulation system capable of simulating the operational behavior of a real cluster. In this environment, the same resource manager scheduling instance count, node specifications, network topology, and data distribution as the real cluster are configured. Job requests from the historical scheduling logs are input into the digital twin environment in their original chronological order. The simulated scheduler in the environment makes decisions according to the hierarchical decision model to be trained, and the simulated nodes in the environment provide feedback on job execution results (such as execution time and resource consumption) based on historical resource usage patterns.

[0070] In one example, a hierarchical decision-making model is pre-trained offline in a digital twin environment. The training process employs a reinforcement learning framework: the model acts as an agent, outputting actions (which instance to select, how many resources to allocate) based on the current state (load of each instance, node resource utilization, etc.), and the digital twin environment provides a reward value. The reward function is designed as a composite function, comprehensively considering job execution efficiency, resource utilization, and load balancing. Through repeated training on a large number of historical jobs, the model gradually learns the optimal scheduling strategy. The model parameters after training convergence are saved as a pre-trained model for subsequent online deployment.

[0071] In a digital twin environment, various scheduling strategies can be safely explored, including those that might cause performance degradation in a real-world environment, without affecting the operation of the real cluster. Simultaneously, training with real-world load patterns from historical logs allows the model to adapt to the actual workload characteristics of the cluster.

[0072] Based on the above offline pre-training, the embodiments of this application further enhance the robustness of the model through training.

[0073] In a digital twin environment, in addition to training with normal workloads from historical logs, various simulated abnormal scenarios are proactively constructed. These include:

[0074] Node failure scenario: During operation in a digital twin environment, certain nodes are randomly or according to a pre-set plan to be marked as faulty, simulating hardware or software failures of nodes in a real cluster. At this time, containers originally running on that node need to be rescheduled, and the resource manager scheduling instance needs to detect the node's disconnection and make corresponding decisions.

[0075] Network partitioning scenario: This simulates a situation where network connectivity is interrupted between some nodes in a cluster. In this scenario, some resource manager scheduling instances cannot communicate with each other, requiring the model to be trained to make reasonable scheduling decisions even under network isolation conditions.

[0076] Traffic surge scenario: This involves injecting significantly higher-than-normal batch job requests into the digital twin environment within a short period, simulating the pressure surge from promotional activities or peak data periods. The model needs to learn how to prioritize high-priority jobs under resource constraints while avoiding system overload.

[0077] After constructing the aforementioned abnormal scenarios, the model is placed within these scenarios for further training. Each time an abnormal scenario is encountered, the model makes a scheduling decision, and the environment provides a reward value. Through repeated training, the model learns to identify abnormal states (such as a large number of instance heartbeat timeouts or a rapid increase in queue length) and adopt corresponding strategies (such as degraded scheduling or initiating a fast-fail mechanism). After training, the model's adaptability to abnormal and high-pressure situations is significantly improved.

[0078] This application describes the online update process of a hierarchical decision model after it has been deployed in a real cluster.

[0079] During real cluster operation, the scheduling entry component generates a scheduling record for each batch job request it processes. This record includes: the request's status information, the model's first and second decisions, and the actual execution results (job completion time, resource consumption, queuing time, etc.). These records are asynchronously written to the log storage system. The online incremental learning module periodically (e.g., hourly) reads the newly added scheduling records from the log storage system as incremental training data. Incremental training employs lightweight update algorithms, such as experience replay or policy gradient updates, avoiding a full retraining of the entire model. Instead, it fine-tunes the existing pre-trained model. The specific update steps are as follows:

[0080] A batch of scheduling records is sampled from incremental data. Each record contains a state, action, and actual reward (calculated based on job execution results). The state is input into the current model to obtain the model's predicted action and predicted value. The difference between the predicted action and the actual action, as well as the difference between the predicted value and the actual cumulative reward, is compared to calculate the loss function. The model parameters are updated in reverse according to the gradient of the loss function, with a small learning rate used for the update step size to avoid disrupting the stable strategy obtained from pre-training. Online incremental learning enables the model to adapt to dynamic changes in cluster load patterns. For example, when a new service is launched, causing changes in job characteristics, the model can gradually adjust the decision boundary through online learning without retraining offline. At the same time, the incremental learning module periodically (e.g., daily) exports the accumulated incremental data for sample expansion in the next offline pre-training, forming a closed-loop optimization process of "offline pre-training → online deployment → incremental collection → re-offline training".

[0081] The three embodiments above collectively describe a hybrid training strategy for hierarchical decision-making models: offline pre-training provides the basic strategy, training in abnormal scenarios enhances robustness, and online incremental learning enables adaptive updates. These training methods ensure the model's scheduling performance and stability in real-world, complex environments.

[0082] Based on the above-described big data cluster resource manager scheduling method, embodiments of this application also provide a big data cluster resource manager scheduling system. The following will combine... Figure 4 The device is described in detail.

[0083] Figure 4 A schematic diagram of the structure of a big data cluster resource manager scheduling system according to an embodiment of this application is shown.

[0084] like Figure 4 As shown, the big data cluster resource manager scheduling system 400 of this embodiment includes a collection module 410, a first determination module 420, a second determination module 430, and a routing module 440.

[0085] The acquisition module 410 is used to collect status information of multiple resource manager scheduling instances in the current cluster in response to a batch job request, provided that the batch job request does not specify a resource manager scheduling instance. In one embodiment, the acquisition module 410 can be used to execute step S310 described above, which will not be repeated here.

[0086] The first determining module 420 is used to input the status information into the hierarchical decision model, execute a first decision through the hierarchical decision model, and determine the target resource manager scheduling instance from the plurality of resource manager scheduling instances. In one embodiment, the first determining module 420 can be used to execute step S320 described above, which will not be repeated here.

[0087] The second determining module 430 is used to execute a second decision through the hierarchical decision model to determine the resource allocation strategy when the target resource manager scheduling instance schedules batch jobs; wherein, the resource allocation strategy is used to indicate the amount of computing resources allocated to the batch jobs. In one embodiment, the second determining module 430 may be used to execute step S330 described above, which will not be repeated here.

[0088] The routing module 440 is used to route the batch job requests to the target resource manager scheduling instance, so that the target resource manager scheduling instance schedules and executes the batch jobs according to the resource allocation policy. In one embodiment, the routing module 440 can be used to perform step S340 described above, which will not be repeated here.

[0089] According to an embodiment of this application, the first determining module includes a first calculation submodule and a first determining submodule.

[0090] The calculation submodule is used to calculate the scheduling priority score of the resource manager scheduling instance based on the status information of the plurality of resource manager scheduling instances; the first determination submodule is used to determine the target resource manager scheduling instance based on the scheduling priority score.

[0091] According to an embodiment of this application, the first calculation submodule is further configured to extract the CPU utilization, memory utilization, and container queue length of each resource manager scheduling instance from the status information; calculate the scheduling priority score of each resource manager scheduling instance based on the CPU utilization, the memory utilization, the container queue length, and a preset weight coefficient; wherein the preset weight coefficient is preset according to the goal of minimizing the sum of the container queue lengths of each resource manager scheduling instance.

[0092] According to an embodiment of this application, the second determining module includes a generation submodule, a second calculation submodule, and a second determining submodule.

[0093] A generation submodule is used to generate multiple candidate resource allocation schemes based on the resource usage status of the node where the target resource manager scheduling instance is located. Each candidate resource allocation scheme includes the number of CPU cores, memory size, and number of graphics processors. A second calculation submodule is used to calculate the execution efficiency prediction value of each candidate resource allocation scheme. A second determination submodule is used to determine the resource allocation strategy based on the execution efficiency prediction value.

[0094] According to an embodiment of this application, the second calculation submodule is further configured to obtain the CPU utilization, memory utilization, GPU utilization, and data locality score of the node where the target resource manager scheduling instance is located. The data locality score is used to characterize the degree of matching between the storage location of the data blocks required by the batch job and the node where the resource manager scheduling instance is located. For each candidate resource allocation scheme, the execution efficiency prediction value corresponding to the candidate resource allocation scheme is calculated based on the CPU utilization, memory utilization, GPU utilization, and data locality score.

[0095] According to embodiments of this application, any plurality of modules among the acquisition module 410, the first determination module 420, the second determination module 430, and the routing module 440 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of this application, at least one of the acquisition module 410, the first determination module 420, the second determination module 430, and the routing module 440 can be at least partially implemented as hardware circuits, such as field-programmable gate arrays, programmable logic arrays, systems-on-a-chip, systems-on-a-substrate, systems-on-package, application-specific integrated circuits, or other reasonable means of integrating or packaging circuits, or implemented in any one of software, hardware, and firmware methods, or in a suitable combination of any of these. Alternatively, at least one of the acquisition module 410, the first determination module 420, the second determination module 430, and the routing module 440 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0096] Figure 5 A block diagram schematically illustrates an electronic device suitable for implementing a big data cluster resource manager scheduling method according to an embodiment of this application.

[0097] like Figure 5 As shown, an electronic device 500 according to an embodiment of this application includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory 502 or a program loaded from a storage portion 508 into a random access memory 503. The processor 501 may include, for example, a general-purpose microprocessor, an instruction set processor and / or an associated chipset and / or a dedicated microprocessor. The processor 501 may also include onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for executing different steps of the method flow according to an embodiment of this application.

[0098] Random access memory 503 stores various programs and data required for the operation of electronic device 500. Processor 501, read-only memory 502, and random access memory 503 are interconnected via bus 504. Processor 501 executes various steps of the method flow according to embodiments of this application by executing programs stored in read-only memory 502 and / or random access memory 503. It should be noted that programs may also be stored in one or more memories other than read-only memory 502 and random access memory 503. Processor 501 may also execute various steps of the method flow according to embodiments of this application by executing programs stored in one or more memories.

[0099] According to embodiments of this application, the electronic device 500 may further include an input / output interface 505, which is also connected to a bus 504. The electronic device 500 may also include one or more of the following components connected to the input / output interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube, liquid crystal display, etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card, such as a local area network card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 510 as needed so that computer programs read from it can be installed into the storage section 508 as needed.

[0100] Embodiments of this application also provide a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0101] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof. In embodiments of this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include the read-only memory 502, and / or random access memory 503, and / or one or more memories other than read-only memory 502 and random access memory 503 described above.

[0102] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this application.

[0103] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 509, and / or installed from a removable medium 511. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0104] In embodiments of this application, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by processor 501, it performs the functions defined in the system of this application embodiment. According to embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0105] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0106] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0107] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

Claims

1. A scheduling method for a big data cluster resource manager, characterized in that, The method includes: In response to a batch job request, if the batch job request does not specify a resource manager scheduling instance, the status information of multiple resource manager scheduling instances in the current cluster is collected; The status information is input into the hierarchical decision model, and the first decision is executed through the hierarchical decision model to determine the target resource manager scheduling instance from the multiple resource manager scheduling instances; The second decision is executed through the hierarchical decision model to determine the resource allocation strategy when the target resource manager scheduling instance schedules batch jobs; wherein, the resource allocation strategy is used to indicate the amount of computing resources allocated to the batch jobs; The batch job request is routed to the target resource manager scheduling instance, so that the target resource manager scheduling instance schedules and executes the batch job according to the resource allocation policy.

2. The method according to claim 1, characterized in that, The step of executing the first decision through the hierarchical decision model, and determining the target resource manager scheduling instance from the plurality of resource manager scheduling instances, includes: Calculate the scheduling priority score of the resource manager scheduling instance based on the status information of the multiple resource manager scheduling instances; The target resource manager scheduling instance is determined based on the scheduling priority score.

3. The method according to claim 2, characterized in that, The step of calculating the scheduling priority score of the resource manager scheduling instance based on the status information of the plurality of resource manager scheduling instances includes: Extract the CPU utilization, memory utilization, and container queue length of each resource manager scheduling instance from the status information; The scheduling priority score of each resource manager scheduling instance is calculated based on the CPU utilization, memory utilization, container queue length, and a preset weight coefficient; wherein the preset weight coefficient is preset according to the goal of minimizing the sum of the container queue lengths of each resource manager scheduling instance.

4. The method according to claim 1, characterized in that, The step of executing the second decision through the hierarchical decision model to determine the resource allocation strategy when the target resource manager scheduling instance schedules batch jobs includes: Based on the resource usage status of the node where the target resource manager scheduling instance is located, multiple candidate resource allocation schemes are generated. Each candidate resource allocation scheme includes the number of CPU cores, memory size, and number of graphics processors. Calculate the predicted execution efficiency for each candidate resource allocation scheme; The resource allocation strategy is determined based on the predicted execution efficiency value.

5. The method according to claim 4, characterized in that, The calculation of the predicted execution efficiency for each candidate resource allocation scheme includes: The CPU utilization, memory utilization, GPU utilization, and data locality score of the node where the target resource manager scheduling instance is located are obtained. The data locality score is used to characterize the degree of matching between the storage location of the data blocks required by the batch job and the node where the resource manager scheduling instance is located. For each candidate resource allocation scheme, the predicted execution efficiency value corresponding to the candidate resource allocation scheme is calculated based on the CPU utilization, memory utilization, GPU utilization, and data locality score.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: If the batch job request specifies a resource manager scheduling instance, the specified resource manager scheduling instance will be used as the target resource manager scheduling instance.

7. The method according to claim 1, characterized in that, The multiple resource manager scheduling instances are deployed to run in parallel.

8. A big data cluster resource manager scheduling system, characterized in that, include: The data acquisition module is used to respond to a batch job request and, if the batch job request does not specify a resource manager scheduling instance, collect the status information of multiple resource manager scheduling instances in the current cluster. The first determining module is used to input the status information into the hierarchical decision model, execute the first decision through the hierarchical decision model, and determine the target resource manager scheduling instance from the plurality of resource manager scheduling instances; The second determining module is used to execute a second decision through the hierarchical decision model to determine the resource allocation strategy when the target resource manager scheduling instance schedules batch jobs; wherein, the resource allocation strategy is used to indicate the amount of computing resources allocated to the batch jobs; The routing module is used to route the batch job request to the target resource manager scheduling instance, so that the target resource manager scheduling instance schedules and executes the batch job according to the resource allocation strategy.

9. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 7.

Images