Task scheduling processing method and electronic equipment

Abstract

The present application discloses a task scheduling processing method and electronic equipment, which relate to the field of data processing technology, including: collecting operating data of each device; obtaining multiple candidate tasks to be processed and current resource status; inputting the operating data of each device into a trained load prediction model for processing to obtain the predicted load of each device; inputting each candidate task and the current resource status into a preset priority evaluation model for processing to obtain the priority index of each candidate task; inputting the predicted load of each device and the priority index of each candidate task into a preset scheduling model for processing to obtain the scheduling strategy of each candidate task; scheduling the corresponding candidate tasks according to each scheduling strategy, and determining the task scheduling strategy that meets the current business through the collaboration of the trained load prediction model, the preset priority evaluation model and the preset scheduling model, so that the task scheduling processing method can adapt to various task resource scheduling environments.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a task scheduling processing method and electronic equipment. Background Art

[0002] Task scheduling refers to the process by which a server's operating system schedules tasks at specific, agreed-upon times to automatically complete them. Once scheduled, the operating system can automatically execute them, freeing up manual effort and improving work efficiency. As a crucial component of operating systems, especially real-time operating systems, task scheduling directly impacts the system's real-time performance. Therefore, task resource scheduling in servers faces complex and ever-changing requirements and real-time challenges.

[0003] Current task scheduling methods in the related art primarily use a pre-defined fixed scheduling policy to schedule one or more tasks waiting on a server in real time until each task is completed. However, this method of scheduling one or more tasks on a server in real time using a fixed scheduling policy is only suitable for a single task resource scheduling environment and cannot address the task scheduling and real-time requirements caused by dynamic business changes. Summary of the Invention

[0004] The present application provides a task scheduling processing method and electronic device to at least solve the problem that the method of real-time scheduling of one or more tasks in a server by using a fixed scheduling strategy in the related art is only suitable for a single task resource scheduling environment and cannot cope with the task scheduling needs and real-time requirements caused by dynamic changes in business.

[0005] This application provides a task scheduling method, including:

[0006] Collect operating data of each device;

[0007] Get the pending task list and current resource status, where the task list contains multiple candidate tasks;

[0008] Input the operating data of each device into the trained load prediction model for processing to obtain the predicted load of each device;

[0009] Input each candidate task and the current resource status into the preset priority evaluation model for processing to obtain the priority index of each candidate task;

[0010] The predicted load of each device and the priority index of each candidate task are input into the preset scheduling model for processing to obtain the scheduling strategy of each candidate task;

[0011] Schedule the corresponding candidate tasks according to each scheduling strategy.

[0012] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned task scheduling processing methods when executing the computer program.

[0013] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned task scheduling processing methods are implemented.

[0014] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned task scheduling processing methods when executed by a processor.

[0015] The task scheduling processing method and electronic device provided in the embodiments of the present application collect the operating data of each device; obtain a list of tasks to be processed and the current resource status, wherein the task list contains multiple candidate tasks; input the operating data of each device into a trained load prediction model for processing to obtain the predicted load of each device; input each candidate task and the current resource status into a preset priority evaluation model for processing to obtain the priority index of each candidate task; input the predicted load of each device and the priority index of each candidate task into a preset scheduling model for processing to obtain a scheduling strategy for each candidate task; schedule the corresponding candidate tasks according to each scheduling strategy, and through the collaboration of the trained load prediction model, the preset priority evaluation model and the preset scheduling model, perform real-time processing on the operating data of each device, the current resource status and each candidate task to determine the task scheduling strategy that meets the current business, so that it can adapt to a variety of task resource scheduling environments and meet the scheduling requirements and real-time requirements of each candidate task. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0017] Figure 1 A schematic diagram of an application scenario of the task scheduling processing method provided in an embodiment of the present application;

[0018] Figure 2 Schematic diagram of the task scheduling process provided in this embodiment of the application Figure 1 ;

[0019] Figure 3 Schematic diagram of the task scheduling process provided in this embodiment of the application Figure 2 ;

[0020] Figure 4 Schematic diagram of the task scheduling process provided in this embodiment of the application Figure 3 ;

[0021] Figure 5 A schematic diagram of the structure of a task scheduling processing device provided in an embodiment of the present application;

[0022] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0024] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0025] Task scheduling refers to the process by which the server's operating system schedules tasks at a specific agreed time in order to automatically complete the corresponding tasks, where task scheduling can be multi-task scheduling, distributed scheduling, and parallel scheduling, etc. After scheduling the tasks, the operating system can automatically execute the corresponding tasks, thereby relieving manual pressure and improving work efficiency. At the same time, task scheduling, as an important part of the operating system, especially in real-time operating systems, directly affects the real-time performance of the system. Therefore, the task resource scheduling in the server faces complex and changeable demands and real-time challenges. In the related art, the current task scheduling processing method mainly uses a pre-set fixed scheduling strategy to schedule one or more tasks waiting in the server in real time until the scheduling of each task is completed. However, in the related art, the method of using a fixed scheduling strategy to schedule one or more tasks in the server in real time is only suitable for a single task resource scheduling environment, and cannot cope with the task scheduling needs and real-time requirements caused by dynamic changes in business.

[0026] In order to solve the above technical problems, the embodiments of the present application propose the following technical concepts: the inventor takes into account the operating data of each device, multiple candidate tasks and the current resource status, and processes the operating data of each device based on the trained load prediction model to obtain the predicted load of each device, processes each candidate task and the current resource status based on the preset priority evaluation model to obtain the priority index of each candidate task, uses the preset scheduling model to process the predicted load of each device and the priority index of each candidate task to obtain the scheduling strategy of each candidate task, schedules the corresponding candidate tasks according to each scheduling strategy, and through the collaboration of the trained load prediction model, the preset priority evaluation model and the preset scheduling model, processes the operating data of each device, the current resource status and each candidate task in real time to determine the task scheduling strategy that meets the current business, so that it can adapt to a variety of task resource scheduling environments and meet the scheduling requirements and real-time requirements of each candidate task.

[0027] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0028] In conjunction with the specific application environment architecture or specific hardware architecture on which the execution of the task scheduling processing method depends, the specific application environment architecture or specific hardware architecture is described here. Figure 1 , Figure 1 This is a schematic diagram of an application scenario of the task scheduling processing method.

[0029] like Figure 1 As shown, the application scenario of the task scheduling processing method includes: an electronic device 10.

[0030] The electronic device 10 may be a single server or a cluster consisting of multiple servers.

[0031] The electronic device 10 includes a plurality of devices 101, including at least a central processing unit, a network card, a disk, and a memory stick.

[0032] The electronic device 10 collects the operating data of each device 101; obtains a list of tasks to be processed and the current resource status, wherein the task list includes multiple candidate tasks; inputs the operating data of each device 101 into a trained load prediction model for processing to obtain the predicted load of each device; inputs each candidate task and the current resource status into a preset priority evaluation model for processing to obtain the priority index of each candidate task; inputs the predicted load of each device and the priority index of each candidate task into a preset scheduling model for processing to obtain the scheduling strategy of each candidate task; and schedules the corresponding candidate tasks according to each scheduling strategy.

[0033] Figure 2 Schematic diagram of the task scheduling process provided in this embodiment of the application Figure 1 ,like Figure 2 As shown, the embodiment of the present application provides a task scheduling processing method, which is described in detail as follows:

[0034] S201: Collecting operation data of each device.

[0035] Specifically, the preset collection tool is controlled to periodically access the monitoring port of each device through a preset request to collect the operating data of each device.

[0036] In this embodiment, the preset collection tool may be a Prometheus tool or other collection tools.

[0037] Among them, Prometheus is an open source monitoring solution. Its core functions include: multi-dimensional data model data classification and query function, PromQL language query function, data collection function and autonomous function.

[0038] In addition, Prometheus can also configure the crawling targets in each file node.

[0039] In this embodiment, the preset request may be an HTTP request or other requests.

[0040] In this embodiment, the period may be any one of 10 seconds, 15 seconds, or 30 seconds, or other periods.

[0041] In this embodiment, the monitoring port is a monitoring plug-in deployed by the server node through a service unit file configuration method or a port mapping configuration method.

[0042] In this embodiment, the devices include a central processing unit, a network card, a disk, a memory stick, and other devices.

[0043] In this embodiment, the operation data is time series data, wherein the operation data includes indicator name, collected data and description information.

[0044] For example, the CPU operation data is: the indicator name is node cpu seconds total, the collected data is CPU core and usage data, and the description information is CPU utilization and inter-core load distribution.

[0045] The network card's operating data is as follows: the indicator names are node network receive_bytes_total, node network transmit_bytes_total, and node network receive_drop_total. The collected data is the receive traffic, transmit traffic, and packet loss data. The description information is the monitoring of the ingress and egress bandwidth load pressure and network congestion or interruption.

[0046] Disk operation data includes: indicator names are node_disk_read_bytes_total, node diskwritten_bytes_total, disk io_time_seconds_total, and node disk queue_length. The collected data is the read and write speed per unit time, IO wait queue, and delay. The explanatory information is disk / IO throughput and disk abnormalities.

[0047] The operating data of the memory bar is as follows: the indicator names are node_memory_MemTotal_bytes, node memoryMemAvailable_bytes, node_memory_Cached, and node memory Buffers. The collected data includes available memory, total memory, and cache hit data. The description information includes memory pressure, insufficient memory warning, and storage strategy optimization.

[0048] In addition, data such as system load and task execution can also be obtained.

[0049] S202: Obtain a list of tasks to be processed and a current resource status, wherein the task list includes multiple candidate tasks.

[0050] In this embodiment, the current resource status is a real-time service guarantee status, an energy consumption optimization status, and a resource shortage status.

[0051] S203: Inputting the operating data of each device into the trained load prediction model for processing to obtain the predicted load of each device.

[0052] Specifically, the training process of the load prediction model in step S203 is as follows:

[0053] S2031: Extracting historical operation data of each device from a preset database using a preset extraction tool.

[0054] In this embodiment, the preset extraction tool may be the PromQL language or other extraction tools.

[0055] Among them, PromQL language is the built-in data query language of Prometheus, which is specifically used to process time series data.

[0056] In this embodiment, the preset database may be a local time series storage unit or another database.

[0057] In this embodiment, the historical operation data is time series data.

[0058] S2032: Perform feature selection processing on each historical operation data to obtain each selected historical operation data.

[0059] Specifically, each historical operation data is processed based on time dimension, system indication, node status label and prediction target selection to obtain each selected historical operation data.

[0060] In this embodiment, the time dimension can be set to hours, minutes, and holidays.

[0061] In this embodiment, the system indicators can be set to CPU usage, memory usage, disk I / O, network traffic, task running time, and other indicators.

[0062] In this embodiment, the node status label can be set to node type, number of running tasks, and load level.

[0063] In this embodiment, the prediction target can be set as the usage rate within the next 1 minute, 5 minutes, or 15 minutes.

[0064] S2033: Preprocessing each selected historical operation data to obtain each processed historical operation data.

[0065] Specifically, step S2033 includes:

[0066] S20331: Perform time series extraction processing on each selected historical operation data according to a preset sliding window to obtain each extracted historical operation data.

[0067] In this embodiment, the preset sliding window can be any time sliding window of 30 minutes, 1 hour or 3 hours, which is used to predict the value of a single future time point based on historical time series data.

[0068] S20332: Perform outlier adjustment processing on each extracted historical operation data to obtain each adjusted historical operation data.

[0069] In this embodiment, the outlier adjustment includes null value filling, outlier correction, and other outlier adjustments.

[0070] S20333: Standardize the adjusted historical operation data to obtain standardized historical operation data.

[0071] In this embodiment, the standardization process is a standard deviation standardization process or a normalization process.

[0072] S20334: Perform label construction processing on each standardized historical operation data to obtain each processed operation data.

[0073] In this embodiment, the label construction uses the data at time t+n as the prediction target of the current window.

[0074] S2034: Training a preset load prediction model according to each processed historical operation data to obtain a trained load prediction model.

[0075] Specifically, the processed historical operation data is exported into JSON format data through the PromQL language, and the preset load prediction model is trained according to the JSON format data to obtain a trained load prediction model.

[0076] In this embodiment, the load prediction model can be a long short-term memory network, a gradient boosting tree model, a random forest or other models, which can be selected according to needs.

[0077] Among them, the application advantages of long short-term memory networks are: they have the ability to remember time series and are suitable for capturing the trends and periodicity of load changes over time; their applicable scenarios are: precise prediction of short-term (seconds or minutes) operating changes of various devices.

[0078] Among them, the application advantages of the gradient boosting tree model are: strong multi-dimensional feature modeling capabilities, high robustness, fast training and strong interpretability; the applicable scenario is: multi-variable concurrent load modeling.

[0079] Among them, the application advantages of random forest are: it can be used as a comparative model to verify prediction stability and generalization ability; the applicable scenario is: it is used to establish a model accuracy baseline.

[0080] In addition, the training architecture of the load prediction model includes: training framework and model training platform.

[0081] Among them, the training framework is a preset programming language.

[0082] Among them, the model training platform can be a local training platform, a remote GPU node or a scheduling cluster training platform.

[0083] In addition, after the load prediction model is trained, it can be deployed as a Kubernetes microservice and called through an HTTP POST request.

[0084] Among them, Kubernetes is a portable and scalable open source platform for managing containerized workloads and services.

[0085] Among them, the HTTP POST request is a request method in the HTTP protocol.

[0086] In addition, after step S2034, steps a and b are also included:

[0087] Step a: Perform indicator evaluation on the trained load prediction model to obtain evaluation results.

[0088] In this embodiment, the indicators are mean absolute error, root mean square error, coefficient of determination, and model deviation analysis.

[0089] Among them, model deviation analysis is used to analyze holiday fluctuations and task peak response capabilities.

[0090] Step b: Update the trained load prediction model according to the evaluation results to obtain an updated load prediction model.

[0091] In this embodiment, the update method includes: re-labeling the data in the time period with higher prediction errors, scheduling retraining tasks every morning, and pushing the new model to the online service to retain the old model to support grayscale testing and rollback.

[0092] S204: Input each candidate task and the current resource status into a preset priority evaluation model for processing to obtain a priority index for each candidate task.

[0093] Specifically, step S204 includes:

[0094] S2041: Obtain multiple scheduling parameters for each candidate task.

[0095] In this embodiment, the plurality of scheduling parameters are an urgency value, a resource utilization benefit value, and a migration cost value.

[0096] S2042: Determine the weight of each scheduling parameter according to the current resource status.

[0097] For example, when the current resource state is the real-time service guarantee state, the weights of the urgency value, resource utilization benefit value, and migration cost value are 0.7, 0.2, and 0.1, respectively, to give priority to high-priority tasks and delay ordinary tasks.

[0098] When the current resource state is the energy consumption optimization state, the weights of the urgency value, resource utilization benefit value, and migration cost value are 0.2, 0.6, and 0.2, respectively, to merge tasks as much as possible and improve resource utilization.

[0099] When the current resource status is resource-constrained, the weights of the urgency value, resource utilization benefit value, and migration cost value are 0.3, 0.3, and 0.4, respectively, to avoid migrating tasks with high migration costs.

[0100] S2043: Calculate the priority index of the corresponding candidate task according to each scheduling parameter and each weight.

[0101] In this embodiment, each scheduling parameter includes an urgency value, a resource utilization benefit value, and a migration cost value; accordingly, the priority index of the corresponding candidate task is calculated according to each scheduling parameter and each weight, and the calculation formula includes:

[0102]

[0103] Where Score is the priority indicator of any candidate task; Pururgency is the urgency value, w1 is the corresponding weight; Refficiency is the resource utilization benefit value, w2 is the corresponding weight; Coverhead is the migration cost value, w3 is the corresponding weight.

[0104] In addition, the historical scheduling status can also be used as auxiliary calculation data for the priority indicator.

[0105] S205: Inputting the predicted load of each device and the priority index of each candidate task into a preset scheduling model for processing to obtain a scheduling strategy for each candidate task.

[0106] Specifically, step S205 includes:

[0107] S2051: Obtain resource idle ratio, current energy consumption data and historical scheduling data of each device.

[0108] In this embodiment, the historical scheduling data is the historical task completion rate and failure rate.

[0109] S2052: Input the predicted load of each device, the priority index of each candidate task, the resource idle ratio of each device, the current energy consumption data and the historical scheduling data into the preset scheduling model for processing to obtain the scheduling strategy corresponding to each candidate task.

[0110] In this embodiment, the predicted load is the load condition in 5 minutes, 10 minutes or 15 minutes.

[0111] Exemplarily, the scheduling strategy is: scheduling the i-th candidate task to the local node, delaying the scheduling of the i-th candidate task, migrating the i-th candidate task to other designated nodes, or starting or shutting down low-priority background tasks on the node.

[0112] In this embodiment, the preset scheduling model can be considered as a preset agent, that is, a task scheduler. Kafka can be used as the agent status reporting and experience data channel, while the model parameters are stored in a lightweight key-value store such as Redis / Etcd.

[0113] Among them, Kafka is a distributed data stream platform that enables applications to publish, subscribe, store and process message streams in real time; Redis is a high-performance in-memory database, and Etcd is a distributed strong consistency key-value storage.

[0114] The agent is part of a shared Q network, and the training process of the shared Q network includes steps c to e:

[0115] Step c: Collect experience samples from each device through the central learner.

[0116] In this embodiment, the experience samples of each device are resource utilization, power consumption per unit time, and asynchronous delay data of each device.

[0117] Step d: Optimize each experience sample through the activation function to obtain each optimized sample.

[0118] In this embodiment, the calculation formula of the excitation function includes:

[0119]

[0120] Where Reward is the sample optimization value of any device; ResourceUtilization is the resource utilization of the device; PowerConsumption is the power consumption per unit time; TaskDelay is the asynchronous delay data; α, β, and γ are adjustable parameters.

[0121] Step e: Train the shared Q network based on each optimized sample.

[0122] S206: Scheduling corresponding candidate tasks according to each scheduling strategy.

[0123] Specifically, the corresponding candidate tasks are scheduled according to each scheduling strategy and a preset scheduler.

[0124] The preset scheduler can be a K8s scheduler or other schedulers.

[0125] In addition, the preset priority evaluation model and the preset scheduling model can be linked together. The corresponding linkage points are:

[0126] Embed the scoring module into the default scheduler or resource scheduling platform of the custom cluster; support task pre-ordering and priority queue mechanism; support the following policy linkage: high-priority tasks preempt low-priority tasks, dynamic delay of tasks at the end of the queue, hot node task merging, and isolated deployment of intensive tasks.

[0127] In summary, the task scheduling processing method provided in this embodiment collects the operating data of each device; obtains a list of tasks to be processed and the current resource status, wherein the task list contains multiple candidate tasks; inputs the operating data of each device into a trained load prediction model for processing to obtain the predicted load of each device; inputs each candidate task and the current resource status into a preset priority evaluation model for processing to obtain the priority index of each candidate task; inputs the predicted load of each device and the priority index of each candidate task into a preset scheduling model for processing to obtain the scheduling strategy of each candidate task; schedules the corresponding candidate tasks according to each scheduling strategy, and through the collaboration of the trained load prediction model, the preset priority evaluation model and the preset scheduling model, performs real-time processing on the operating data of each device, the current resource status and each candidate task, and determines a task scheduling strategy that meets the current business, so that it can adapt to a variety of task resource scheduling environments and meet the scheduling requirements and real-time requirements of each candidate task.

[0128] In addition, the task scheduling processing method provided in this embodiment generates a predicted load through a trained load prediction model and evaluates the priority of each candidate task through a preset priority evaluation model, thereby making the generation of subsequent task scheduling strategies more accurate.

[0129] In addition, the task scheduling processing method provided in this embodiment determines the scheduling strategy of each candidate task through a preset scheduling model, thereby realizing adaptive task scheduling and improving the response speed of the server.

[0130] Figure 3 Schematic diagram of the task scheduling process provided in this embodiment of the application Figure 2 In the embodiment of the present application, Figure 2 Based on the provided embodiments, the specific implementation method of power consumption adjustment for each device is described in detail. Figure 3 As shown, the task scheduling processing method includes:

[0131] S301: Obtain the power consumption of each device per unit time.

[0132] In this embodiment, each device includes a central processing unit, a network card, a disk, and a memory stick; accordingly, step S301 specifically includes:

[0133] S3011: Integrate exporter plug-in through preset monitoring tools.

[0134] In this embodiment, the preset monitoring tool may be a Prometheus tool or other monitoring tools.

[0135] The discussion about Prometheus has been described in detail in step S201 and will not be repeated here.

[0136] S3012: Use the exporter plug-in to collect the power consumption of the central processing unit per unit time.

[0137] S3013: Use the preset indicator collection tool to obtain the power consumption of the network card, disk, and memory module per unit time.

[0138] In this embodiment, the preset indicator collection tool may be a NodeExporter tool or other collection tools.

[0139] Among them, NodeExporter is a core component in the Prometheus monitoring system, which is specifically used to collect hardware and indicator data of the operating system.

[0140] S302: Determine the ratio of the usage rate to the power consumption of each device.

[0141] S303: Generate a corresponding historical energy consumption trend graph according to the historical power consumption of each device.

[0142] S304: Determine the time ratio of each device in the active state to the idle state.

[0143] S305: Determine whether the corresponding device is in a low-load state according to the power consumption, the ratio, the historical energy consumption trend graph, and the time ratio.

[0144] S306: If any device is determined to be in a low-load state, the power consumption of the device is adjusted according to a preset energy-saving strategy.

[0145] Exemplarily, if it is determined that the CPU is in a low-load state, the power consumption of the CPU is adjusted according to a preset energy-saving strategy.

[0146] The preset energy-saving strategy may be one or more of a DVFS control strategy, a service freezing strategy, a task merging and node offline strategy, and a Pod dynamic scheduling optimization strategy.

[0147] Among them, the DVFS control strategy is a dynamic voltage and frequency adjustment strategy, which is used to enable dynamic voltage and frequency regulation, dynamically adjust the frequency of the central processing unit according to the load, and reduce energy consumption.

[0148] Among them, the service freeze policy is used to suspend or reduce the priority of non-critical services to release resources.

[0149] Among them, the task merging and node offline strategy is used to merge and migrate multiple low-load tasks to a minimum number of nodes, and then offline the nodes after releasing redundant nodes.

[0150] Among them, the Pod dynamic scheduling optimization strategy is used to cooperate with the K8s scheduler to readjust the affinity and anti-affinity rules of the Pod to promote resource centralization.

[0151] Among them, Pod is the smallest scheduling and management unit in the Kubernetes cluster. Its essence is a collection of containers that share network, storage and lifecycle resources.

[0152] In summary, the task scheduling processing method provided in this embodiment obtains the power consumption of each device per unit time; determines the ratio of the utilization rate of each device to the power consumption; generates a corresponding historical energy consumption trend chart based on the historical power consumption of each device; determines the time ratio of the active state and the idle state of each device; determines whether the corresponding device is in a low-load state based on each power consumption, each ratio, each historical energy consumption trend chart and each time ratio; if any device is determined to be in a low-load state, the power consumption of the device is adjusted according to the preset energy-saving strategy, so that the energy consumption of the server is reduced during the task scheduling processing.

[0153] Figure 4 Schematic diagram of the task scheduling process provided in this embodiment of the application Figure 3 In the embodiment of the present application, Figure 2 Based on the provided embodiments, the specific implementation method for retraining the trained load prediction model, the preset priority evaluation model and the preset scheduling model is described in detail. Figure 4 As shown, the task scheduling processing method includes:

[0154] S401: Collecting an optimization feedback data stream, wherein the optimization feedback data stream includes a plurality of task scheduling process data.

[0155] In this embodiment, the multiple task scheduling process data include one or more of the following data: task execution logs, historical task scheduling data, load prediction deviations, scheduling strategy execution results, task scheduling effects, node status changes, and other process data.

[0156] In addition, ELK assists in data inflow and builds visualization channels to optimize the feedback data flow into Kafka.

[0157] Among them, ELK is a complete set of centralized log processing solutions, including: open source distributed search engine, data collection engine and open source Web interface tools.

[0158] Among them, Kafka is a distributed data stream platform that enables applications to publish, subscribe, store and process message streams in real time.

[0159] S402: Determine whether the amount of each task scheduling process data is greater than a preset threshold, and / or determine whether each task scheduling process data triggers a preset strategy failure alarm.

[0160] In this embodiment, the preset threshold may be any value among 100, 200, or 300, or other values.

[0161] For example, the triggering of the preset strategy failure alarm may be as follows:

[0162] In the data of each task scheduling process, an alarm is triggered when the node resource idle rate is higher than 65% for three consecutive hours, where the normal threshold is <50%; an alarm is triggered when there is cumulative deviation in the time series data, such as when the prediction deviation exceeds ±15% for five consecutive cycles; an alarm is triggered when the prediction accuracy of the machine learning model drops to the set threshold; an alarm is triggered when the collaboration between subsystems is abnormal, such as when the scheduling instruction response delay exceeds the timeout threshold of 500ms.

[0163] S403: If it is determined that the number of task scheduling process data is greater than a preset threshold, and / or a preset strategy failure alarm is triggered, the trained load prediction model, preset priority evaluation model and preset scheduling model are retrained to obtain a new load prediction model, a new priority evaluation model and a new scheduling model.

[0164] Specifically, if it is determined that the amount of data in multiple task scheduling processes is greater than a preset threshold, and / or a preset strategy failure alarm is triggered, the training task is triggered through the Kafka message, and the trained load prediction model, preset priority evaluation model and preset scheduling model are retrained to obtain a new load prediction model, a new priority evaluation model and a new scheduling model.

[0165] The retraining process of the trained load prediction model is as follows: retraining the trained load prediction model using new periodic data to obtain a new load prediction model.

[0166] Among them, the retraining process of the preset priority evaluation model is: based on the feedback data of the actual scheduling effect, the weight parameters of the evaluation function in the preset priority evaluation model are readjusted to obtain a new priority evaluation model.

[0167] Among them, the retraining process of the preset scheduling model is: use the DQN policy network to input a new state-action-reward sequence, update the preset scheduling model, and obtain a new scheduling model.

[0168] In addition, after the training of each model is completed, each new model will be pushed to the model warehouse and automatically deployed to the online service, and a version rollback mechanism is supported.

[0169] In addition, each model can be hosted in a Git repository, and the management and update process through the GitOps mechanism is as follows: when each model changes, the CI Pipeline is triggered to automatically build and deploy the image, and cooperate with the K8s scheduler to perform hot updates of the image; introduce A / B Testing or Canary release to implement grayscale testing and version comparison for risk control.

[0170] Among them, the Git repository is a business working directory tracked and managed by Git. It contains the business history and version information, and facilitates code branch management and version control.

[0171] Among them, the GitOps mechanism uses the Git code repository to manage the deployment process of infrastructure and application code.

[0172] Among them, CI Pipeline is a continuous integration pipeline.

[0173] Among them, A / B Testing is an experimental method that uses data to drive decision-making by comparing the effects of different versions of web pages, functions or operation plans.

[0174] Among them, Canary release is a progressive software release strategy.

[0175] In summary, the task scheduling processing method provided in this embodiment collects optimized feedback data streams, wherein the optimized feedback data streams include multiple task scheduling process data; determines whether the number of each task scheduling process data is greater than a preset threshold, and / or determines whether each task scheduling process data triggers a preset strategy failure alarm; if it is determined that the number of multiple task scheduling process data is greater than the preset threshold, and / or triggers a preset strategy failure alarm, the trained load prediction model, preset priority evaluation model and preset scheduling model are retrained to obtain a new load prediction model, a new priority evaluation model and a new scheduling model, and by continuously optimizing the corresponding models, the operating efficiency of the server task call is improved.

[0176] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0177] Figure 5 This is a schematic diagram of the structure of the task scheduling processing device provided in the embodiment of the present application. Figure 5 As shown, an embodiment of the present application further provides a task scheduling processing device, including: a collection module 501, a first acquisition module 502, a first processing module 503, a second processing module 504, a third processing module 505 and a scheduling module 506.

[0178] The acquisition module 501 is used to collect the operating data of each device;

[0179] A first acquisition module 502 is used to acquire a list of tasks to be processed and a current resource status, wherein the task list includes multiple candidate tasks;

[0180] The first processing module 503 is used to input the operating data of each device into the trained load prediction model for processing to obtain the predicted load of each device;

[0181] The second processing module 504 is used to input each candidate task and the current resource status into a preset priority evaluation model for processing to obtain a priority index for each candidate task;

[0182] The third processing module 505 is used to input the predicted load of each device and the priority index of each candidate task into the preset scheduling model for processing to obtain the scheduling strategy of each candidate task;

[0183] The scheduling module 506 is used to schedule corresponding candidate tasks according to each scheduling strategy.

[0184] In a possible implementation, the apparatus further includes:

[0185] An extraction module is used to extract historical operation data of each device from a preset database using a preset extraction tool;

[0186] The fourth processing module is used to perform feature selection processing on each historical operation data to obtain each selected historical operation data;

[0187] a fifth processing module, configured to pre-process each selected historical operation data to obtain each processed historical operation data;

[0188] The first training module is used to train the preset load prediction model according to the processed historical operation data to obtain a trained load prediction model.

[0189] In a possible implementation, the fifth processing module specifically includes:

[0190] A first processing unit is configured to perform time series extraction processing on each selected historical operation data according to a preset sliding window to obtain each extracted historical operation data;

[0191] a second processing unit, configured to perform outlier adjustment processing on each extracted historical operation data to obtain each adjusted historical operation data;

[0192] a third processing unit, configured to perform standardization processing on each adjusted historical operation data to obtain each standardized historical operation data;

[0193] The fourth processing unit is used to perform label construction processing on each standardized historical operation data to obtain each processed operation data.

[0194] In a possible implementation, the fourth processing module is specifically used to perform time dimension, system indication, node status label and prediction target selection processing on each historical operation data to obtain each selected historical operation data.

[0195] In a possible implementation, the apparatus further includes:

[0196] Evaluation module, used to evaluate the indicators of the trained load prediction model and obtain the evaluation results;

[0197] The updating module is used to update the trained load prediction model according to the evaluation results to obtain an updated load prediction model.

[0198] In a possible implementation, the second processing module 504 specifically includes:

[0199] An acquisition unit, used to acquire multiple scheduling parameters of each candidate task;

[0200] A determination unit, configured to determine the weight of each scheduling parameter according to the current resource status;

[0201] The calculation unit is used to calculate the priority index of the corresponding candidate task according to various scheduling parameters and various weights.

[0202] In a possible implementation, each scheduling parameter includes an urgency value, a resource utilization benefit value, and a migration cost value; accordingly, a calculation formula for calculating the priority index of a corresponding candidate task based on each scheduling parameter and each weight includes:

[0203]

[0204] Where Score is the priority indicator of any candidate task; Pururgency is the urgency value, w1 is the corresponding weight; Refficiency is the resource utilization benefit value, w2 is the corresponding weight; Coverhead is the migration cost value, w3 is the corresponding weight.

[0205] In a possible implementation, the third processing module 505 specifically includes:

[0206] An acquisition unit is used to obtain resource idle ratio, current energy consumption data and historical scheduling data of each device;

[0207] The processing unit is used to input the predicted load of each device, the priority index of each candidate task, the resource idle ratio of each device, the current energy consumption data and the historical scheduling data into the preset scheduling model for processing to obtain the scheduling strategy corresponding to each candidate task.

[0208] In a possible implementation, the collection module 501 is specifically configured to control a preset collection tool to periodically access the monitoring port of each device through a preset request to collect the operating data of each device.

[0209] In a possible implementation, the apparatus further includes:

[0210] The second acquisition module is used to obtain the power consumption of each device per unit time;

[0211] A first determining module is used to determine the ratio of the usage rate to the power consumption of each device;

[0212] A generation module is used to generate a corresponding historical energy consumption trend graph based on the historical power consumption of each device;

[0213] The second determining module is used to determine the time ratio of the active state to the idle state of each device;

[0214] A first judgment module is used to judge whether the corresponding device is in a low-load state according to each power consumption, each ratio, each historical energy consumption trend graph and each time ratio;

[0215] If any device is determined to be in a low-load state, the power consumption of the device is adjusted according to the preset energy-saving strategy.

[0216] In a possible implementation, each device includes a central processing unit, a network card, a disk, and a memory stick; accordingly, the second acquisition module specifically includes:

[0217] Integration unit for integrating exporter plugins with pre-defined monitoring tools;

[0218] The collection unit is used to collect the power consumption of the central processing unit per unit time through the exporter plug-in;

[0219] The acquisition unit is used to obtain the power consumption of the network card, disk and memory bar per unit time through a preset indicator collection tool.

[0220] In a possible implementation, the apparatus further includes:

[0221] A collection module is used to collect optimization feedback data streams, wherein the optimization feedback data streams include multiple task scheduling process data;

[0222] A second judgment module is used to judge whether the amount of each task scheduling process data is greater than a preset threshold, and / or to judge whether each task scheduling process data triggers a preset strategy failure alarm;

[0223] The second training module is used to retrain the trained load prediction model, preset priority evaluation model and preset scheduling model to obtain new load prediction model, new priority evaluation model and new scheduling model if it is determined that the number of data in multiple task scheduling processes is greater than a preset threshold and / or triggers a preset strategy failure alarm.

[0224] For the description of the features in the embodiment corresponding to the task scheduling processing device, please refer to the relevant description of the embodiment corresponding to the task scheduling processing method, and no further details will be given here.

[0225] Figure 6 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 6 As shown, the electronic device provided by this embodiment includes: at least one processor 601 and a memory 602. Optionally, the electronic device further includes a communication component 603. The processor 601, the memory 602 and the communication component 603 are connected via a bus.

[0226] During the specific implementation process, at least one processor 601 executes the computer execution instructions stored in the memory 602, so that the at least one processor 601 executes the above-mentioned task scheduling processing method embodiment.

[0227] The specific implementation process of the processor 601 can be found in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here in this embodiment.

[0228] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the application may be directly executed by a hardware processor or by a combination of hardware and software modules within the processor.

[0229] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage.

[0230] A bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be categorized as address buses, data buses, and control buses. For ease of illustration, the buses in the drawings of this application are not limited to just one bus or just one type of bus.

[0231] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned task scheduling processing method embodiments when running.

[0232] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0233] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above-mentioned task scheduling processing method embodiments are implemented.

[0234] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps in any of the above-mentioned task scheduling processing method embodiments.

[0235] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0236] The above is a detailed introduction to a task scheduling processing method and electronic device provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A task scheduling processing method, characterized in that: include: Collect operating data of each device; Obtaining a list of pending tasks and a current resource status, wherein the task list includes multiple candidate tasks; the current resource status includes a real-time service guarantee status, an energy consumption optimization status, and a resource shortage status; Input the operating data of each device into the trained load prediction model for processing to obtain the predicted load of each device; Inputting each candidate task and the current resource status into a preset priority evaluation model for processing to obtain a priority index for each candidate task; Inputting the predicted load of each device and the priority index of each candidate task into a preset scheduling model for processing to obtain a scheduling strategy for each candidate task; Schedule the corresponding candidate tasks according to each scheduling strategy; The candidate tasks and current resource status are input into a preset priority evaluation model for processing to obtain a priority index for each candidate task, including: Obtain multiple scheduling parameters for each candidate task; wherein each scheduling parameter includes an urgency value, a resource utilization benefit value, and a migration cost value; Determining the weight of each scheduling parameter according to the current resource state; Calculate the priority index of the corresponding candidate task according to each scheduling parameter and each weight; The predicted load of each device and the priority index of each candidate task are input into a preset scheduling model for processing to obtain a scheduling strategy for each candidate task, including: Obtain resource idle ratio, current energy consumption data and historical scheduling data for each device; Input the predicted load of each device, the priority index of each candidate task, the resource idle ratio of each device, the current energy consumption data and the historical scheduling data into a preset scheduling model for processing to obtain a scheduling strategy corresponding to each candidate task; The method further comprises: Obtain the power consumption of each device per unit time; Determine the ratio of utilization to power consumption of each device; Generate a corresponding historical energy consumption trend chart based on the historical power consumption of each device; Determine the proportion of time each device spends in active versus idle states; Determine whether the corresponding device is in a low-load state based on each power consumption, each ratio, each historical energy consumption trend chart, and each time ratio; If any device is determined to be in a low-load state, the power consumption of the device is adjusted according to a preset energy-saving strategy.

2. The task scheduling processing method according to claim 1, characterized in that: The training process of the load prediction model includes: Extract historical operating data of each device from the preset database using the preset extraction tool; Perform feature selection processing on each historical operation data to obtain each selected historical operation data; Preprocessing each selected historical operation data to obtain each processed historical operation data; The preset load prediction model is trained according to the processed historical operation data to obtain the trained load prediction model.

3. The task scheduling processing method according to claim 2, characterized in that: The preprocessing of each selected historical operation data to obtain each processed historical operation data includes: According to the preset sliding window, time series extraction processing is performed on each selected historical operation data to obtain each extracted historical operation data; Performing outlier adjustment processing on each extracted historical operation data to obtain each adjusted historical operation data; Performing standardization processing on each adjusted historical operation data to obtain each standardized historical operation data; Label construction processing is performed on each standardized historical operation data to obtain each processed operation data.

4. The task scheduling processing method according to claim 2, characterized in that: The feature selection process is performed on each historical operation data to obtain each selected historical operation data, including: The historical operation data are processed by time dimension, system indication, node status label and prediction target selection to obtain the historical operation data after each selection.

5. The task scheduling processing method according to claim 2, characterized in that: After the preset load prediction model is trained according to the processed historical operation data to obtain a trained load prediction model, the method further includes: Performing an index evaluation on the trained load prediction model to obtain an evaluation result; The trained load prediction model is updated according to the evaluation result to obtain an updated load prediction model.

6. The task scheduling processing method according to claim 1, characterized in that: The calculation formula for calculating the priority index of the corresponding candidate task according to each scheduling parameter and each weight includes: Where Score is the priority indicator of any candidate task; Pururgency is the urgency value, w1 is the corresponding weight; Refficiency is the resource utilization benefit value, w2 is the corresponding weight; Coverhead is the migration cost value, w3 is the corresponding weight.

7. The task scheduling method according to claim 1, wherein: The collecting of the operation data of each device includes: Control the preset collection tool, periodically access the monitoring port of each device through preset requests, and collect the operating data of each device.

8. The task scheduling method according to claim 1, wherein: The devices include central processing unit, network card, disk and memory stick; Accordingly, the power consumption of each device per unit time is obtained, including: Integrate exporter plug-ins through preset monitoring tools; The power consumption of the central processing unit per unit time is collected through the exporter plug-in; The power consumption of the network card, the disk, and the memory stick per unit time is obtained through a preset indicator collection tool.

9. The task scheduling method according to claim 1, wherein: Also includes: Collecting an optimization feedback data stream, wherein the optimization feedback data stream includes a plurality of task scheduling process data; Determine whether the amount of data from each task scheduling process is greater than a preset threshold, and / or determine whether the data from each task scheduling process triggers a preset strategy failure alarm; If it is determined that the number of data of the multiple task scheduling processes is greater than the preset threshold, and / or the preset strategy failure alarm is triggered, the trained load prediction model, preset priority evaluation model and preset scheduling model are retrained to obtain a new load prediction model, a new priority evaluation model and a new scheduling model.

10. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the task scheduling method according to any one of claims 1 to 9 when executing the computer program.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the task scheduling processing method according to any one of claims 1 to 9.

12. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the task scheduling processing method according to any one of claims 1 to 9 are implemented.

Images