Resource scheduling method and device, equipment, storage medium and program product

Abstract

The invention discloses a resource scheduling method and device, equipment, a storage medium and a program product, and relates to the technical field of resource management. The method comprises the steps of obtaining unstructured text information of a to-be-executed task, and predicting a resource demand prediction result of the to-be-executed task according to the unstructured text information; determining a target node according to the unstructured text information, the resource demand prediction result and resource state information of each node in the cluster; and performing resource scheduling and task execution of the to-be-executed task on the target node according to the resource demand prediction result and a security marginal control mechanism. The method comprises the following steps: extracting deep features from unstructured text information to generate high-precision resource demand prediction; a prediction result, task semantics and a cluster real-time state are fused to carry out comprehensive scheduling decision making; a prospective safety control mechanism is introduced in a task execution level, and a complete closed loop from task understanding to accurate prediction and then to safety scheduling and elastic execution is formed.

Description

Technical Field

[0001] This invention relates to the field of resource management technology, and in particular to a resource scheduling method, apparatus, equipment, storage medium, and program product. Background Technology

[0002] In distributed computing environments such as cloud computing, high-performance computing, and big data processing, one of the core objectives of a task scheduling system is to accurately predict the computing resource requirements of a task.

[0003] Currently, common resource demand prediction methods in task scheduling systems are mainly divided into the following two categories: (1) Prediction methods based on manual user declaration. This method relies heavily on user experience and generally suffers from problems such as over-declaration leading to resource waste or under-declaration leading to task failure. (2) Machine learning prediction methods based on historical statistics. To compensate for the shortcomings of manual declaration, some improved solutions introduce machine learning models to predict the resource demand of new tasks by analyzing the structured characteristics of historical tasks. Then, the scheduler schedules tasks based on the prediction results and node resource status. However, such methods have significant bottlenecks: their prediction models usually do not make full use of the "unstructured text information" of tasks, and the accuracy of the prediction models is limited. More importantly, existing solutions usually regard "resource prediction" and "scheduling execution" as two separate links, and resource prediction fails to form a closed-loop feedback and collaborative optimization with the scheduling execution mechanism. Summary of the Invention

[0004] This invention provides a resource scheduling method, apparatus, device, storage medium, and program product to solve the problems of inaccurate resource prediction for unstructured text information and the failure of resource prediction to form a closed-loop feedback and collaborative optimization with the scheduling execution mechanism.

[0005] In a first aspect, embodiments of the present invention provide a resource scheduling method, including: Obtain unstructured text information of the task to be executed, and predict the resource requirement prediction result of the task to be executed based on the unstructured text information; The target node is determined based on the unstructured text information, the resource demand prediction results, and the resource status information of each node in the cluster. Based on the resource demand forecast results and the safety margin control mechanism, resource scheduling and task execution of the task to be executed are performed on the target node.

[0006] Secondly, embodiments of the present invention provide a resource scheduling apparatus, comprising: The resource requirement prediction module is used to obtain unstructured text information of the task to be executed, and predict the resource requirement prediction result of the task to be executed based on the unstructured text information. The target node determination module is used to determine the target node based on the unstructured text information, the resource demand prediction results, and the resource status information of each node in the cluster. The resource scheduling module is used to schedule resources and execute tasks on the target node based on the resource demand prediction results and the safety margin control mechanism.

[0007] Thirdly, embodiments of the present invention provide an electronic device, the electronic device comprising: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the resource scheduling method according to any embodiment of the present invention.

[0008] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing computer instructions that are used to cause a processor to execute the resource scheduling method described in any embodiment of the present invention.

[0009] Fifthly, embodiments of the present invention provide a computer program product including a computer program, which, when executed by a processor, implements the resource scheduling method described in any embodiment of the present invention.

[0010] The technical solution of this invention obtains unstructured text information of the task to be executed and predicts the resource demand of the task based on the unstructured text information; determines the target node based on the unstructured text information, the resource demand prediction result, and the resource status information of each node in the cluster; and performs resource scheduling and task execution on the target node based on the resource demand prediction result and a security margin control mechanism. By extracting deep features from the unstructured text information, high-precision resource demand prediction is generated; comprehensive scheduling decisions are made by integrating the prediction results, task semantics, and real-time cluster status; and a forward-looking security control mechanism is introduced at the task execution level. This forms a complete closed loop from task understanding to accurate prediction, and then to secure scheduling and flexible execution, solving the problems of inaccurate resource prediction based on unstructured text information and the failure of resource prediction to form a closed-loop feedback and collaborative optimization with the scheduling and execution mechanism. It has the beneficial effects of improving resource utilization efficiency and system reliability.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A flowchart of a resource scheduling method provided in Embodiment 1 of the present invention; Figure 2 This is a flowchart of a resource scheduling method provided in Embodiment 2 of the present invention; Figure 3 A flowchart of a resource scheduling method provided in Embodiment 3 of the present invention; Figure 4 This is a schematic diagram of the structure of a resource scheduling device provided in Embodiment 4 of the present invention; Figure 5 A schematic diagram of the structure of an electronic device for implementing the resource scheduling method of this embodiment of the invention. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0016] Example 1 Figure 1This is a flowchart of a resource scheduling method provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where resources required for a task to be executed are predicted and scheduled, and the task is executed. This method can be executed by a resource scheduling device, which can be implemented in hardware and / or software, and can be configured in an electronic device. Figure 1 As shown, the method includes: S110. Obtain the unstructured text information of the task to be executed, and predict the resource requirement prediction result of the task to be executed based on the unstructured text information.

[0017] Unstructured text information can be considered a type of weak text information containing rich semantics, such as command-line argument strings, script names, script file paths, and working directory paths. This type of unstructured text information is characterized by semantic sparsity and an implicit correlation with resource requirements. Therefore, traditional prediction models typically predict resource requirements based on structured task information that directly reflects resource needs.

[0018] In this embodiment, by analyzing the task behavior characteristics contained in the unstructured text information of the task to be performed, the correlation between task behavior characteristics and resource requirements is mined to obtain resource requirement prediction results. The resource requirement prediction results may include predicted values ​​for the demand for central processing unit (CPU) resources and / or memory resources; they may also include assessment values ​​of the uncertainty of the demand prediction values, such as confidence levels or quantiles.

[0019] As an optional embodiment, a resource demand prediction model can be constructed and trained to predict resource demand for unstructured text information, thereby obtaining resource demand prediction results. To improve the accuracy of resource demand prediction, multiple regression heads and self-attention mechanisms can be introduced into the resource demand prediction model to deeply explore the relationship between unstructured text information and resource demand.

[0020] S120. Determine the target node based on unstructured text information, resource demand prediction results, and resource status information of each node in the cluster.

[0021] In this context, the resource status information of a node can be considered as the information regarding the status of the resources possessed by the node. The target node can be considered as the node in the cluster used to execute the task to be executed.

[0022] In this embodiment, the unstructured text information and resource requirement prediction results of the task to be executed are matched with the resource status information of each node in the cluster, and the most suitable or most promising node is selected from the cluster as the target node for executing the task. To select the most suitable or most promising target node, the matching degree between the unstructured text information and resource requirement prediction results of the task to be executed and the resource status information of each node in the cluster can be quantified. This matching degree can be comprehensively calculated from multiple dimensions such as the node's performance in executing the task, stability, and whether the resource type matches.

[0023] S130. Based on the resource demand forecast results and the safety margin control mechanism, perform resource scheduling and task execution for the tasks to be executed at the target node.

[0024] Among them, the safety margin control mechanism is a buffer protection strategy designed to cope with uncertainty, prediction errors and system fluctuations in resource demand forecasting. It adds an extra safety margin to the forecast value to ensure that the system has sufficient resource redundancy to cope with various unexpected situations in actual operation.

[0025] In this embodiment, a safety margin control mechanism is introduced during the resource allocation phase based on resource demand forecasting results to strike a balance between task execution reliability and resource utilization efficiency. The safety margin control mechanism can make dynamic decisions based on the runtime context of the target node.

[0026] The technical solution of this invention obtains unstructured text information of the task to be executed and predicts the resource demand of the task based on the unstructured text information; determines the target node based on the unstructured text information, the resource demand prediction result, and the resource status information of each node in the cluster; and performs resource scheduling and task execution on the target node based on the resource demand prediction result and a security margin control mechanism. By extracting deep features from the unstructured text information, high-precision resource demand prediction is generated; comprehensive scheduling decisions are made by integrating the prediction results, task semantics, and real-time cluster status; and a forward-looking security control mechanism is introduced at the task execution level. This forms a complete closed loop from task understanding to accurate prediction, and then to secure scheduling and flexible execution, improving resource utilization efficiency and system reliability.

[0027] Example 2 Figure 2This is a flowchart of a resource scheduling method provided in Embodiment 2 of the present invention. Based on the above embodiments, this embodiment specifies the prediction of resource requirements for the task to be executed based on the unstructured text information as follows: The unstructured text information is input into the embedding layer of the resource requirement prediction model to perform word segmentation and embedding processing, resulting in an unstructured text vector; the unstructured text vector is input into an encoding layer based on a self-attention mechanism to encode the unstructured text vector, resulting in an encoded vector; the encoded vector is input into the output layer based on a multi-task regression head to perform shared feature extraction and multi-task regression prediction, resulting in a resource requirement prediction result; wherein the resource requirement prediction result includes an expected value of resource requirements and a corresponding uncertainty quantification result, and the expected value of resource requirements includes an expected value of CPU resource requirements and an expected value of memory resource requirements.

[0028] like Figure 2 As shown, the method includes: S210. Obtain the unstructured text information of the task to be executed.

[0029] In this embodiment, unstructured text information may include command-line parameter strings for the task to be executed, script file name, script file type, script file path, and working directory path, etc.

[0030] S220. Using the resource demand prediction model, input unstructured text information into the embedding layer of the resource demand prediction model, perform word segmentation and embedding processing on the unstructured text information, and obtain unstructured text vectors.

[0031] The resource demand prediction model can be considered as a model used to predict the resources required to perform a task based on unstructured text information. The resource demand prediction model can be trained using a training sample set consisting of unstructured text information and actual resource demands. Unstructured text vectors can be considered as vectorized representations of unstructured text information.

[0032] In this embodiment, the resource demand prediction model may include an embedding layer, an encoding layer, and an output layer. The embedding layer is used to preprocess the input unstructured text information by word segmentation, sub-path segmentation, and / or key-value pair decomposition to obtain a text sequence; then, the text sequence is converted into word segmentation vector tokens by a tokenizer. The word segmentation vector tokens are input into the embedding layer to obtain unstructured text vectors.

[0033] S230. Input the unstructured text vector into the encoding layer based on the self-attention mechanism to encode the unstructured text vector and obtain the encoded vector.

[0034] In this embodiment, the encoding layer of the resource demand prediction model includes a self-attention mechanism. The unstructured text vector output from the embedding layer is input into the encoding layer, such as a Transformer neural network or a variant thereof. The self-attention mechanism automatically assigns weights to different word segmentation vector tokens according to the context, thereby capturing the implicit correlation between word segmentation and the resource demand of the task, realizing deep semantic modeling of weak text features, and finally obtaining the encoded vector output by the encoding layer.

[0035] In this embodiment, through a self-attention mechanism, the resource demand prediction model can learn the influence patterns of unstructured text such as command-line argument strings, script names, script file types, script file paths, and working directory paths on resource demand trends, providing semantic support for subsequent resource demand prediction.

[0036] S240. Input the encoded vector into the output layer based on the multi-task regression head, perform shared feature extraction and multi-task regression prediction on the encoded vector, and obtain the resource demand prediction result; wherein, the resource demand prediction result includes the expected value of resource demand and the corresponding uncertainty quantification result, the expected value of resource demand includes the expected value of CPU resource demand and the expected value of memory resource demand.

[0037] In this embodiment, the output layer of the resource demand prediction model includes a multi-task regression head. The encoded vector output from the encoding layer is input into the output layer, and the multi-task regression head in the output layer predicts the expected CPU resource demand and expected memory resource demand of the task to be executed. Specifically, the multi-task regression head includes: a shared feature extraction layer for generating shared features of the task to be executed with a unified semantic representation; and two independent regression branches (heads). The first regression branch predicts CPU resource demand, and the second regression branch predicts memory resource demand. Each regression branch can be a one- or multi-layer fully connected neural network structure, which maps the shared features to the corresponding predicted values ​​through nonlinear transformations. The predicted values ​​can be expected resource demand values, i.e., the average predicted value of resource demand. Expected values ​​can include expected CPU resource demand and expected memory resource demand; the expected CPU resource demand characterizes the typical CPU demand of the task to be executed under normal operating conditions, and the expected memory resource demand characterizes the typical memory usage of the task to be executed under normal operating conditions.

[0038] During training, multi-task learning is achieved by minimizing the joint loss function of CPU and memory resource requirements. This allows the model to share information while taking into account the differences between tasks, thereby improving the overall prediction accuracy and generalization ability.

[0039] In this embodiment, to enhance the reliability of the prediction and the safety of the scheduling, the output of each regression head includes not only the expected prediction value but also an uncertainty quantification result. This result quantifies the uncertainty of the prediction result and provides an adjustable safety boundary. The uncertainty quantification result includes the uncertainty quantification result corresponding to the expected CPU resource requirement and the uncertainty quantification result corresponding to the expected memory resource requirement.

[0040] Uncertainty quantification results can include confidence intervals or multiple quantiles. Specifically, quantiles represent one or more key quantiles in the model's predicted distribution of resource demand, such as p50, p90, and p99; where: p50 (i.e., the median): indicates that there is a 50% probability that the task's resource demand will not exceed this value, reflecting a typical load level; p90 and p99 represent the upper bounds of resource demand under high load or extreme conditions, which can be used to mitigate the risk of insufficient resources. A confidence interval indicates that, at a preset confidence level (such as 90% or 95%), the probability that the actual resource consumption of the task to be executed falls within this interval is not lower than the corresponding confidence level.

[0041] S250. Determine the target node based on unstructured text information, resource demand prediction results, and resource status information of each node in the cluster.

[0042] S260. Based on the resource demand forecast results and the safety margin control mechanism, perform resource scheduling and task execution for the tasks to be executed at the target node.

[0043] As an optional embodiment of the present invention, the method further includes: during the execution of the task to be executed by the target node, collecting resource usage data and task execution status data of the target node; and updating the model parameters of the resource demand prediction model based on the resource usage data and the task execution status data.

[0044] Resource usage data can be the actual resources consumed by the target node in executing the task. For example, resource usage data can include CPU resource consumption and memory resource consumption. Task execution status data can be considered as data related to the status of the target node in executing the task. Task execution status data can include task execution results (failure or success) and resource utilization, etc.

[0045] In this embodiment, during the process of resource scheduling and task execution at the target node based on resource demand prediction results and a safety margin control mechanism, real-time data on resource usage and task execution status is collected. Based on the collected resource usage data, task execution status data, and sample data generated from the tasks to be executed, the model parameters of the resource demand prediction model are updated. This allows the resource demand prediction model to be updated online and its parameters to be fine-tuned, ensuring the accuracy of the resource demand prediction model.

[0046] The technical solution of this invention involves: acquiring unstructured text information of the task to be executed; inputting the unstructured text information into the embedding layer of a resource demand prediction model to perform word segmentation and embedding processing, resulting in an unstructured text vector; inputting the unstructured text vector into an encoding layer based on a self-attention mechanism to encode the unstructured text vector, resulting in an encoded vector; inputting the encoded vector into an output layer based on a multi-task regression head to perform shared feature extraction and multi-task regression prediction, resulting in a resource demand prediction result; wherein the resource demand prediction result includes expected resource demand values ​​and corresponding uncertainty quantification results, the expected resource demand values ​​including expected CPU resource demand values ​​and expected memory resource demand values; determining a target node based on the unstructured text information, the resource demand prediction result, and the resource status information of each node in the cluster; and performing resource scheduling and task execution on the target node based on the resource demand prediction result and a safety margin control mechanism. By using the attention mechanism and multi-task regression head of the resource demand prediction model, deep features are extracted from unstructured text information to generate high-precision resource demand predictions. The prediction results, task semantics and real-time cluster status are integrated to make comprehensive scheduling decisions. A forward-looking safety control mechanism is introduced at the task execution level, forming a complete closed loop from task understanding to accurate prediction, and then to safe scheduling and elastic execution, thereby improving resource utilization efficiency and system reliability.

[0047] Example 3 Figure 3 This is a flowchart of a resource scheduling method provided in Embodiment 3 of the present invention. Based on the above embodiments, this embodiment specifies the determination of the target node based on the unstructured text information, the resource demand prediction result, and the resource status information of each node in the cluster as follows: A candidate node set composed of all nodes in the cluster is determined, and the resource status information of each node in the candidate node set is collected in real time through a resource monitoring module; the expected resource demand value in the resource demand prediction result is compared with the resource status information of each node in the candidate node set, and candidate nodes whose resource status information cannot meet the expected resource demand value are removed from the candidate node set; the resource bottleneck type of the task to be executed is determined based on the expected resource demand value and / or the unstructured text information; the resource bottleneck type includes at least one of CPU-intensive, memory-intensive, and read / write-intensive; for each remaining candidate node in the candidate node set, the matching degree between the candidate node and the task to be executed is determined based on the resource status information of the candidate node, the resource bottleneck type of the task to be executed, and the unstructured text information; the candidate node with the highest matching degree is determined as the target node.

[0048] like Figure 3 As shown, the method includes: S310. Obtain the unstructured text information of the task to be executed, and predict the resource requirement prediction result of the task to be executed based on the unstructured text information.

[0049] S320. Determine the candidate node set composed of all nodes in the cluster, and collect the resource status information of each node in the candidate node set in real time through the resource monitoring module.

[0050] In this embodiment, each node in the cluster serves as a resource available for executing tasks. The scheduling device can collect real-time resource status information of all nodes in the cluster through the resource monitoring module. This resource status information may include: the number of available CPU cores, available physical memory capacity, current system load rate, I / O utilization, historical task submission success rate, and resource sustainability duration for each node.

[0051] S330. Compare the expected resource demand value in the resource demand forecast results with the resource status information of each node in the candidate node set, and remove the candidate nodes whose resource status information cannot meet the expected resource demand value from the candidate node set.

[0052] In this embodiment, target nodes are determined through a two-stage node filtering and matching process. In the first stage, the expected resource demand in the resource demand prediction results is compared with the resource status information of each node in the candidate node set. Candidate nodes that cannot meet the expected resource demand are removed from the candidate node set. For example, if a candidate node's available memory is less than the predicted memory expectation of the task to be executed, or if a candidate node's available CPU is less than the predicted CPU expectation of the task to be executed, then that candidate node is removed from the candidate node set.

[0053] S340. Determine the resource bottleneck type of the task to be executed based on the expected value of resource requirements and / or unstructured text information; the resource bottleneck type includes at least one of CPU-intensive, memory-intensive, and read-write-intensive.

[0054] In this embodiment, the resource bottleneck type of the task to be executed may include at least one of CPU-intensive, memory-intensive, and read / write-intensive. If the predicted expected value of CPU for the task to be executed is higher than the preset CPU percentage, the resource bottleneck type of the task to be executed is determined to be CPU-intensive; if the predicted expected value of memory for the task to be executed is higher than the preset memory percentage, the resource bottleneck type of the task to be executed is determined to be memory-intensive; if the number of data read / write features contained in the file access or path parameters in the unstructured text information of the task to be executed is greater than a threshold, the resource bottleneck type of the task to be executed is determined to be read / write (I / O)-intensive.

[0055] S350. For each remaining candidate node in the candidate node set, determine the matching degree between the candidate node and the task to be executed based on the resource status information of the candidate node and the resource bottleneck type and unstructured text information of the task to be executed; determine the candidate node with the highest matching degree as the target node.

[0056] In this embodiment, in the second stage, for each remaining candidate node after removing candidate nodes from the candidate node set that cannot meet the expected value of resource requirements, the resource configuration type in the resource status information of the candidate node and the resource bottleneck type of the task to be executed are compared, and the matching degree between the candidate node and the task to be executed is determined based on the resource status information of the candidate node and the unstructured text information of the task to be executed.

[0057] For example, it is determined whether the resource configuration type of the candidate node is consistent with the resource bottleneck type of the task to be executed, and a comprehensive score for the candidate node to execute the task is determined based on the resource status information of the candidate node and the unstructured text information of the task to be executed. This comprehensive score can be evaluated from aspects such as latency, task execution success rate and resource sustainability duration. The matching degree between the candidate node and the task to be executed is determined based on the consistency of resource type and comprehensive score.

[0058] As an optional implementation of this embodiment, determining the matching degree between the candidate node and the task to be executed based on the resource status information of the candidate node, the resource bottleneck type of the task to be executed, and the unstructured text information includes: A1. Determine the matching weight of the candidate node based on the resource bottleneck type of the task to be executed and the resource configuration type of the candidate node.

[0059] Among them, the resource configuration type of the candidate node can be considered as the type of resource advantages that the candidate node is equipped with. For example, it may include computing-optimized nodes with high-frequency CPUs or more cores, memory-optimized nodes equipped with large-capacity physical memory, and read-write-optimized nodes equipped with high-speed local storage or high-bandwidth network storage.

[0060] In this embodiment, the matching weight between the candidate node and the task to be executed is determined based on the number of candidate nodes whose resource bottleneck type matches the resource configuration type of the task to be executed. The principle for setting the matching weight is: the matching weight for nodes with the same type is significantly greater than the matching weight for nodes with different types; the more nodes with the same type, the greater the matching weight.

[0061] For example, for CPU-intensive tasks to be executed, the matching weight of computing-optimized nodes with high-frequency CPUs or more cores is significantly increased; for memory-intensive tasks to be executed, the matching weight of memory-optimized nodes with large-capacity physical memory is significantly increased; and for I / O-intensive tasks to be executed, the matching weight of read-write-optimized nodes with high-speed local storage or high-bandwidth network storage is significantly increased.

[0062] A2. Determine the data access time of the candidate node based on the task data storage location in the unstructured text information of the task to be executed and the location of the candidate node.

[0063] In this context, the task data storage location can be considered the location where the task data to be executed is stored, and the task data can be considered the data that needs to be read and used during task execution. The candidate node location can be considered the position of the candidate node within the cluster.

[0064] In this embodiment, during the execution of a task, a candidate node needs to access and read task data from the task data storage location and execute the task. Therefore, the network distance between the task data storage location and the candidate node's location directly affects the data access time. For example, the data transmission time between the task data storage location and the candidate node's location is determined as the data access time.

[0065] A3. Determine the comprehensive score for the candidate node to execute the task based on the current load rate, historical task submission success rate, resource sustainability duration, and data access time in the resource status information of the candidate node.

[0066] In this embodiment, the current load rate is used to reflect the overall load level of the candidate node's CPU, memory, and I / O; the historical task submission success rate is used to reflect the stability of the candidate node's successful task execution; the resource sustainability duration is used to reflect the sustainability of the candidate node's resource provision; and the data access time is used to reflect the candidate node's data access efficiency.

[0067] For example, a comprehensive scoring model is constructed that integrates the current load rate, historical task submission success rate, resource sustainability duration, data access time, and overall score. This comprehensive scoring model can be a linear scoring model obtained through data fitting or an intelligent scoring model based on machine learning; this embodiment of the invention does not impose any limitations on this. The current load rate, historical task submission success rate, resource sustainability duration, and data access time are input into the comprehensive scoring model to obtain the comprehensive score output by the model.

[0068] A4. Determine the matching degree between the candidate node and the task to be executed based on the matching weight and the comprehensive score.

[0069] In this embodiment, the comprehensive score of the candidate nodes is weighted according to the matching weight of the candidate nodes to obtain the matching degree between the candidate nodes and the task to be executed.

[0070] In this embodiment, by comprehensively evaluating the matching degree between candidate nodes and tasks to be executed from several dimensions such as resource type, load rate, historical task submission success rate, resource sustainability duration, and data access time, the most suitable candidate node can be matched for the task to be executed.

[0071] S360. Based on the resource demand forecast results and the safety margin control mechanism, perform resource scheduling and task execution for tasks to be executed at the target node.

[0072] As an optional implementation of this embodiment, the step of scheduling resources and executing the task to be executed on the target node based on the resource demand prediction results and the safety margin control mechanism includes: B1. Determine the forecast concentration based on the uncertainty quantification results in the resource demand forecast results.

[0073] In this embodiment, the uncertainty quantification result can be a confidence level or a quantile. For example, the difference between quantile p99 and quantile p50 can be determined as the prediction concentration; or the interval width corresponding to the expected value of resource demand prediction at a certain confidence level (such as 90%) can be determined as the prediction concentration.

[0074] In this embodiment, when the prediction concentration is high (low uncertainty), a lower quantile value is used as the resource application benchmark to prioritize improving resource efficiency; when the prediction dispersion is high (high uncertainty), a higher quantile value is used as the resource application benchmark to prioritize ensuring task reliability.

[0075] B2. Input the predicted concentration, the current load rate of the target node, and the environmental fault tolerance of the target node into the decision model to obtain the safety margin level output by the decision model.

[0076] The safety margin level can be considered as the relative extent or buffer space by which a system's actual capacity or available resources exceed its expected demand or critical load. The safety margin level is generally a ratio, percentage, or difference.

[0077] In this embodiment, there is a monotonic mapping relationship between the current load rate of the target node and the safety margin level. For example, when the current load rate is high, it means that the task is light and resources are sufficient, so a higher safety margin level can be applied (such as using p95) to exchange low cost for high reliability. When the current load rate is high, it means that the task is heavy and resources are tight, so the safety margin level is tightened (such as using p50) to prioritize the overall throughput and resource turnover of the cluster and avoid resource deadlock caused by excessive reservation.

[0078] Environmental fault tolerance can be considered as the ability of a target node to recover and restore itself quickly and at minimal cost when current resource predictions are inaccurate or when there is a sudden load. Environmental fault tolerance can be determined by the configuration and technology stack of the entire cluster, or it can be considered as the ability of the node's underlying infrastructure itself to provide fast read / write and elastic scaling. A node's environmental fault tolerance can be obtained by reading its attribute information. There is a negative correlation between environmental fault tolerance and the level of safety margin. In environments with strong fault tolerance (such as container platforms that support second-level vertical scaling), the system can adopt a lower safety margin, allowing tasks to start with resources close to actual needs, and then quickly scale up if insufficient. In environments with weak fault tolerance (such as where restarting a failed task is costly), a higher safety margin must be adopted, striving to request sufficient resources at once to avoid failure.

[0079] In this embodiment, a configurable decision model (such as a function or engine based on weighted scoring or rules) is constructed. The predicted concentration, the current load rate of the target node, and the environmental fault tolerance of the target node are input into the decision model, and the safety margin level is calculated through the decision model.

[0080] B3. Determine the resource scheduling value to be requested from the target node based on the security margin level and the expected value of resource demand.

[0081] The resource scheduling value can be the resource value requested by the node.

[0082] In this embodiment, the predicted expected value of resource demand and the safety margin level are multiplied or summed to obtain the resource scheduling value requested from the target node.

[0083] Optionally, the resource scheduling value can be used as the resource scheduling suggestion value, and the user's requested resource value can be obtained. If the difference between the resource scheduling suggestion value and the user's requested resource value is within a preset range, resources are requested from the target node according to the resource scheduling value. If the difference between the resource scheduling suggestion value and the user's requested resource value exceeds the preset range, automatic correction is performed according to the configuration, or a prompt is issued to the user requesting confirmation. All correction and confirmation actions are recorded in the audit log.

[0084] B4. Submit the resource scheduling value and the task to be executed to the target node, so that the target node can schedule resources and execute the task to be executed according to the resource scheduling value.

[0085] In this embodiment, the obtained resource scheduling value and the task to be executed are submitted to the target node, and the target node can schedule resources to execute the task according to the resource scheduling value.

[0086] The technical solution of this invention involves: acquiring unstructured text information of a task to be executed; predicting the resource requirements of the task based on the unstructured text information; determining a candidate node set composed of nodes in the cluster; collecting the resource status information of each node in the candidate node set in real time through a resource monitoring module; comparing the expected resource requirements in the resource requirement prediction results with the resource status information of each node in the candidate node set, and removing candidate nodes whose resource status information cannot meet the expected resource requirements from the candidate node set; determining the resource bottleneck type of the task to be executed based on the expected resource requirements and / or the unstructured text information; the resource bottleneck type includes at least one of CPU-intensive, memory-intensive, and read / write-intensive; for each remaining candidate node in the candidate node set, determining the matching degree between the candidate node and the task to be executed based on the resource status information of the candidate node, the resource bottleneck type of the task to be executed, and the unstructured text information; determining the candidate node with the highest matching degree as the target node; and performing resource scheduling and task execution on the target node based on the resource requirement prediction results and a security margin control mechanism. By extracting deep features from unstructured text information, high-precision resource demand predictions are generated; comprehensive scheduling decisions are made by integrating prediction results, task semantics, and real-time cluster status; and a forward-looking security control mechanism is introduced at the task execution level, forming a complete closed loop from task understanding to accurate prediction, and then to secure scheduling and elastic execution, thereby improving resource utilization efficiency and system reliability.

[0087] Example 4 Figure 4 This is a schematic diagram of a resource scheduling device provided in Embodiment 4 of the present invention. Figure 4 As shown, the device includes: a resource demand prediction module 410, a target node determination module 420, and a resource scheduling module 430; wherein: The resource demand prediction module 410 is used to acquire unstructured text information of the task to be executed, and predict the resource demand prediction result of the task to be executed based on the unstructured text information. The target node determination module 420 is used to determine the target node based on the unstructured text information, the resource demand prediction results, and the resource status information of each node in the cluster. The resource scheduling module 430 is used to perform resource scheduling and task execution of the task to be executed on the target node based on the resource demand prediction results and the safety margin control mechanism.

[0088] Optionally, the resource demand prediction module 410 is specifically used for: The unstructured text information is input into the embedding layer of the resource demand prediction model, and word segmentation and embedding are performed on the unstructured text information to obtain an unstructured text vector. The unstructured text vector is input into an encoding layer based on a self-attention mechanism to encode the unstructured text vector, thereby obtaining an encoded vector; The encoded vector is input into the output layer based on the multi-task regression head, and shared feature extraction and multi-task regression prediction are performed on the encoded vector to obtain the resource demand prediction result; wherein, the resource demand prediction result includes the expected value of resource demand and the corresponding uncertainty quantification result, and the expected value of resource demand includes the expected value of CPU resource demand and the expected value of memory resource demand.

[0089] Optionally, the device further includes: The data acquisition module is used to collect resource usage data and task execution status data of the target node during the execution of the task to be executed on the target node; The model update module is used to update the model parameters of the resource demand prediction model based on the resource usage data and the task execution status data.

[0090] Optionally, the target node determination module 420 includes: The resource status information acquisition unit is used to determine the candidate node set composed of each node in the cluster, and to collect the resource status information of each node in the candidate node set in real time through the resource monitoring module. A node filtering unit is used to compare the expected value of resource demand in the resource demand prediction result with the resource status information of each node in the candidate node set, and to remove candidate nodes whose resource status information cannot meet the expected value of resource demand from the candidate node set. A resource bottleneck type determination unit is used to determine the resource bottleneck type of the task to be executed based on the expected value of resource demand and / or the unstructured text information; the resource bottleneck type includes at least one of CPU-intensive, memory-intensive, and read-write-intensive. The matching degree determination unit is used to determine the matching degree between the candidate node and the task to be executed for each remaining candidate node in the candidate node set, based on the resource status information of the candidate node, the resource bottleneck type of the task to be executed, and the unstructured text information. The target node determination unit is used to determine the candidate node with the highest matching degree as the target node.

[0091] Optionally, the matching degree determination unit is specifically used for: The matching weight of the candidate node is determined based on the resource bottleneck type of the task to be executed and the resource configuration type of the candidate node; The data access time of the candidate node is determined based on the task data storage location in the unstructured text information of the task to be executed and the location of the candidate node; The comprehensive score for the candidate node to execute the task is determined based on the current load rate, historical task submission success rate, resource sustainability duration, and data access time in the resource status information of the candidate node. The matching degree between the candidate node and the task to be executed is determined based on the matching weight and the comprehensive score.

[0092] Optionally, the resource scheduling module 430 is specifically used for: The forecast concentration is determined based on the uncertainty quantification results in the resource demand forecast results. The predicted concentration, the current load rate of the target node, and the environmental fault tolerance of the target node are input into the decision model to obtain the safety margin level output by the decision model. The resource scheduling value requested from the target node is determined based on the safety margin level and the expected value of resource demand; The resource scheduling value and the task to be executed are submitted to the target node, so that the target node can schedule resources and execute the task to be executed according to the resource scheduling value.

[0093] The resource scheduling device provided in the embodiments of the present invention can execute the resource scheduling method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0094] Example 5 Figure 5A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0095] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0096] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0097] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as resource scheduling methods.

[0098] In some embodiments, the resource scheduling method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the resource scheduling method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the resource scheduling method by any other suitable means (e.g., by means of firmware).

[0099] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0100] In some embodiments, the resource scheduling method may be implemented as a computer program, which is implicitly included in a computer program product. When executed by a processor, the computer program implements the resource scheduling method of the present invention. The computer program product can be understood as a software product that primarily implements its solution through a computer program. The computer program used to implement the method of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer program causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a standalone software package, or entirely on a remote machine or server.

[0101] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0102] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0103] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0104] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0105] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and no limitation is imposed herein.

[0106] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A resource scheduling method, characterized in that, include: Obtain unstructured text information of the task to be executed, and predict the resource requirement prediction result of the task to be executed based on the unstructured text information; The target node is determined based on the unstructured text information, the resource demand prediction results, and the resource status information of each node in the cluster. Based on the resource demand forecast results and the safety margin control mechanism, resource scheduling and task execution of the task to be executed are performed on the target node.

2. The method according to claim 1, characterized in that, The prediction of resource requirements for the task to be executed based on the unstructured text information includes: The unstructured text information is input into the embedding layer of the resource demand prediction model, and word segmentation and embedding are performed on the unstructured text information to obtain an unstructured text vector. The unstructured text vector is input into an encoding layer based on a self-attention mechanism to encode the unstructured text vector, thereby obtaining an encoded vector; The encoded vector is input into the output layer based on the multi-task regression head, and shared feature extraction and multi-task regression prediction are performed on the encoded vector to obtain the resource demand prediction result; wherein, the resource demand prediction result includes the expected value of resource demand and the corresponding uncertainty quantification result, and the expected value of resource demand includes the expected value of CPU resource demand and the expected value of memory resource demand.

3. The method according to claim 2, characterized in that, Also includes: During the execution of the task to be executed on the target node, resource usage data and task execution status data of the target node are collected; The model parameters of the resource demand prediction model are updated based on the resource usage data and the task execution status data.

4. The method according to claim 1, characterized in that, The step of determining the target node based on the unstructured text information, the resource demand prediction results, and the resource status information of each node in the cluster includes: A candidate node set consisting of all nodes in the cluster is determined, and the resource status information of each node in the candidate node set is collected in real time through the resource monitoring module. Compare the expected resource demand value in the resource demand prediction result with the resource status information of each node in the candidate node set, and remove the candidate nodes whose resource status information cannot meet the expected resource demand value from the candidate node set; The resource bottleneck type of the task to be executed is determined based on the expected resource requirements and / or the unstructured text information; the resource bottleneck type includes at least one of CPU-intensive, memory-intensive, and read-write-intensive. For each remaining candidate node in the candidate node set, the matching degree between the candidate node and the task to be executed is determined based on the resource status information of the candidate node, the resource bottleneck type of the task to be executed, and the unstructured text information. The candidate node with the highest matching degree is determined as the target node.

5. The method according to claim 4, characterized in that, The step of determining the matching degree between the candidate node and the task to be executed based on the resource status information of the candidate node, the resource bottleneck type of the task to be executed, and the unstructured text information includes: The matching weight of the candidate node is determined based on the resource bottleneck type of the task to be executed and the resource configuration type of the candidate node; The data access time of the candidate node is determined based on the task data storage location in the unstructured text information of the task to be executed and the location of the candidate node; The comprehensive score for the candidate node to execute the task is determined based on the current load rate, historical task submission success rate, resource sustainability duration, and data access time in the resource status information of the candidate node. The matching degree between the candidate node and the task to be executed is determined based on the matching weight and the comprehensive score.

6. The method according to claim 1, characterized in that, The step of scheduling resources and executing tasks on the target node based on the resource demand forecast results and the safety margin control mechanism includes: The forecast concentration is determined based on the uncertainty quantification results in the resource demand forecast results. The predicted concentration, the current load rate of the target node, and the environmental fault tolerance of the target node are input into the decision model to obtain the safety margin level output by the decision model. The resource scheduling value requested from the target node is determined based on the safety margin level and the expected value of resource demand; The resource scheduling value and the task to be executed are submitted to the target node, so that the target node can schedule resources and execute the task to be executed according to the resource scheduling value.

7. A resource scheduling device, characterized in that, include: The resource requirement prediction module is used to obtain unstructured text information of the task to be executed, and predict the resource requirement prediction result of the task to be executed based on the unstructured text information. The target node determination module is used to determine the target node based on the unstructured text information, the resource demand prediction results, and the resource status information of each node in the cluster. The resource scheduling module is used to schedule resources and execute tasks on the target node based on the resource demand prediction results and the safety margin control mechanism.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the resource scheduling method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the resource scheduling method according to any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the resource scheduling method according to any one of claims 1-6.

Images