Task scheduling processing method and electronic equipment

Abstract

The invention discloses a task scheduling processing method and electronic equipment, and relates to the technical field of data processing, and the task scheduling processing method comprises the following steps: collecting operation data of each piece of equipment; obtaining a plurality of candidate tasks to be processed and a current resource state; inputting the operation data of each device into the trained load prediction model for processing to obtain a predicted load of each device; inputting each candidate task and the current resource state into a preset priority evaluation model for processing to obtain a priority index of each candidate task; inputting the predicted load of each device and the priority index of each candidate task into a preset scheduling model for processing to obtain a scheduling strategy of each candidate task; and scheduling the corresponding candidate tasks according to the scheduling strategies, and determining the task scheduling strategy conforming to the current service through cooperation of the trained load prediction model, the preset priority evaluation model and the preset scheduling model, so that the task scheduling processing method adapts to various task resource scheduling environments.

Description

Technical Field

[0001] The present application relates to the technical field of data processing, and particularly relates to a task scheduling processing method and an electronic device. Background Art

[0002] Task scheduling refers to the process in which the operating system of a server schedules tasks at a specific agreed-upon moment in order to automatically complete corresponding tasks. After task scheduling, the operating system can automatically execute the corresponding tasks, thereby relieving the manual pressure and improving work efficiency. At the same time, task scheduling, as an important part of the operating system, especially in a real-time operating system, directly affects the real-time performance of the system. Therefore, task resource scheduling in a server faces complex and ever-changing requirements and real-time challenges.

[0003] In related technologies, the current task scheduling processing method mainly performs real-time scheduling on one or more tasks waiting in the server through a preset fixed scheduling strategy until the scheduling of each task is completed. However, in related technologies, the method of performing real-time scheduling on one or more tasks in the server by using a fixed scheduling strategy is only suitable for a single task resource scheduling environment and cannot cope with the task scheduling requirements and real-time requirements caused by dynamic business changes. Summary of the Invention

[0004] The present application provides a task scheduling processing method and an electronic device, so as to at least solve the problem that in related technologies, the method of performing real-time scheduling on one or more tasks in the server by using a fixed scheduling strategy is only suitable for a single task resource scheduling environment and cannot cope with the task scheduling requirements and real-time requirements caused by dynamic business changes.

[0005] The present application provides a task scheduling processing method, including:

[0006] Collect the operation data of each device;

[0007] Obtain the task list to be processed and the current resource status, where the task list includes multiple candidate tasks;

[0008] Input the operation data of each device into a trained load prediction model for processing to obtain the predicted load of each device;

[0009] Input each candidate task and the current resource status into a preset priority evaluation model for processing to obtain the priority index of each candidate task;

[0010] Input the predicted load of each device and the priority index of each candidate task into a preset scheduling model for processing to obtain the scheduling strategy of each candidate task;

[0011] Schedule the corresponding candidate tasks according to each scheduling strategy.

[0012] The present application also provides an electronic device, including: a memory for storing a computer program; a processor for implementing the steps of any one of the above task scheduling processing methods when executing the computer program.

[0013] The present application also provides a computer-readable storage medium storing a computer program, wherein the computer program implements the steps of any one of the above task scheduling processing methods when executed by a processor.

[0014] The present application also provides a computer program product including a computer program, which implements the steps of any one of the above task scheduling processing methods when executed by a processor.

[0015] The task scheduling processing method and the electronic device provided by the embodiments of the present application collect the operation data of each device; obtain the task list to be processed and the current resource status, where the task list includes multiple candidate tasks; input the operation data of each device into the trained load prediction model for processing to obtain the predicted load of each device; input each candidate task and the current resource status into the preset priority evaluation model for processing to obtain the priority index of each candidate task; input the predicted load of each device and the priority index of each candidate task into the preset scheduling model for processing to obtain the scheduling strategy of each candidate task; schedule the corresponding candidate tasks according to each scheduling strategy. Through the cooperation of the trained load prediction model, the preset priority evaluation model, and the preset scheduling model, the operation data of each device, the current resource status, and each candidate task are processed in real time to determine the task scheduling strategy that meets the current business, so as to adapt to various task resource scheduling environments and meet the scheduling requirements and real-time requirements of each candidate task. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present application, the drawings required for the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0017] Figure 1 It is a schematic diagram of the application scenario of the task scheduling processing method provided by the embodiments of the present application;

[0018] Figure 2 It is a flowchart of the task scheduling processing method provided by the embodiments of the present application Figure 1 ;

[0019] Figure 3 It is a flowchart of the task scheduling processing method provided by the embodiments of the present application Figure 2 ;

[0020] Figure 4 Flow schematic of the task scheduling processing method provided by the embodiment of the present application Figure 3 ;

[0021] Figure 5 Structural schematic diagram of the task scheduling processing device provided by the embodiment of the present application;

[0022] Figure 6 Hardware structural schematic diagram of the electronic device provided by the embodiment of the present application. Specific embodiments

[0023] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the protection scope of the present application.

[0024] It should be noted that in the description of the present application, the terms "including", "comprising" or any other variant thereof are intended to cover a non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in the present application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0025] Task scheduling refers to the process in which the operating system of the server schedules tasks at a specific agreed moment in order to automatically complete corresponding tasks. Among them, task scheduling can be multi-task scheduling, distributed scheduling, parallel scheduling, etc. After scheduling the tasks, the operating system can automatically execute the corresponding tasks, thereby liberating the manual pressure and improving work efficiency. At the same time, task scheduling, as an important part of the operating system, especially in a real-time operating system, directly affects the real-time performance of the system. Therefore, the task resource scheduling in the server faces complex and changeable requirements and real-time challenges. In the related art, the current task scheduling processing method mainly performs real-time scheduling on one or more tasks waiting in the server through a preset fixed scheduling strategy until the scheduling of each task is completed. However, in the related art, the method of performing real-time scheduling on one or more tasks in the server by using a fixed scheduling strategy only adapts to a single task resource scheduling environment and cannot cope with the task scheduling requirements and real-time requirements caused by the dynamic change of the service.

[0026] To solve the above technical problems, the embodiments of the present application propose the following technical concepts: The inventor takes into account the operation data of each device, multiple candidate tasks, and the current resource status, processes the operation data of each device based on the trained load prediction model to obtain the predicted load of each device, processes each candidate task and the current resource status based on the preset priority evaluation model to obtain the priority index of each candidate task, uses the preset scheduling model to process the predicted load of each device and the priority index of each candidate task to obtain the scheduling strategy of each candidate task, schedules the corresponding candidate tasks according to each scheduling strategy, and through the cooperation of the trained load prediction model, the preset priority evaluation model, and the preset scheduling model, processes the operation data of each device, the current resource status, and each candidate task in real time, determines the task scheduling strategy that conforms to the current business, so as to adapt to various task resource scheduling environments and meet the scheduling requirements and real-time requirements of each candidate task.

[0027] To enable those skilled in the art of this technology to better understand the solution of the present application, the following further detailed description of the present application will be given in conjunction with the accompanying drawings and specific embodiments.

[0028] Combined with the specific application environment architecture or specific hardware architecture on which the execution of the task scheduling processing method depends, the specific application environment architecture or specific hardware architecture will be described here. Refer to Figure 1 , Figure 1 is a schematic diagram of the application scenario of the task scheduling processing method.

[0029] As Figure 1 shown, the application scenario of this task scheduling processing method includes: an electronic device 10.

[0030] The electronic device 10 can be a single server or a cluster composed of multiple servers.

[0031] The electronic device 10 includes multiple devices 101, at least a central processing unit, a network card, a disk, and a memory module.

[0032] The electronic device 10 collects the operation data of each device 101; obtains the task list to be processed and the current resource status, where the task list contains multiple candidate tasks; inputs the operation data of each device 101 into the trained load prediction model for processing to obtain the predicted load of each device; inputs each candidate task and the current resource status into the preset priority evaluation model for processing to obtain the priority index of each candidate task; inputs the predicted load of each device and the priority index of each candidate task into the preset scheduling model for processing to obtain the scheduling strategy of each candidate task; and schedules the corresponding candidate tasks according to each scheduling strategy.

[0033] Figure 2 is the flow schematic of the task scheduling processing method provided by the embodiments of the present applicationFigure 1 , as Figure 2 shown, an embodiment of the present application provides a task scheduling processing method, and a detailed description of the method is as follows:

[0034] S201: Collect the operation data of each device.

[0035] Specifically, control the preset collection tool to periodically access the monitoring ports of each device through a preset request to collect the operation data of each device.

[0036] In this embodiment, the preset collection tool can be the Prometheus tool or other collection tools.

[0037] Among them, Prometheus is an open-source monitoring solution, and its core functions include: multi-dimensional data model data classification and query function, PromQL language query function, data collection function and autonomous function.

[0038] In addition, Prometheus can also configure the scraping targets in each file node.

[0039] In this embodiment, the preset request can be an HTTP request or other requests.

[0040] In this embodiment, the period can be any period among 10 seconds, 15 seconds or 30 seconds, or other periods.

[0041] In this embodiment, the monitoring port is a monitoring plugin deployed by the server node through the service unit file configuration method or the port mapping configuration method.

[0042] In this embodiment, each device is a central processing unit, network card, disk, memory module and other devices.

[0043] In this embodiment, the operation data is time series data, where the operation data includes an index name, collected data and description information.

[0044] Exemplarily, the operation data of the central processing unit is: the index name is node cpu seconds total, the collected data is the central processing unit core and usage data, and the description information is the central processing unit utilization rate and inter-core load distribution.

[0045] The running data of the network card are: the metric names are node network receive_bytes_total, node network transmit_bytes_total, and node network receive_drop_total, the collected data are the received traffic, transmitted traffic, and packet loss data, and the description information is the monitoring of the bandwidth load pressure at the entrance and exit and network congestion or interruption.

[0046] The running data of the disk are: the metric names are node_disk_read_bytes_total, node disk written_bytes_total, disk io_time_seconds_total, and node disk queue_length, the collected data are the read and write speeds per unit time and the IO waiting queue and latency, and the description information is the disk / IO throughput and disk abnormal conditions.

[0047] The running data of the memory module are: the metric names are node_memory_MemTotal_bytes, node memory MemAvailable_bytes, node_memory_Cached, and node memory Buffers, the collected data are the available memory, total memory, and cache hit data, and the description information is the memory pressure, memory shortage warning, and storage policy optimization.

[0048] In addition, data such as system load and task execution can also be obtained.

[0049] S202: Obtain the list of tasks to be processed and the current resource status, where the task list contains multiple candidate tasks.

[0050] In this embodiment, the current resource status is the real-time service guarantee status, the energy consumption optimization status, and the resource tension status.

[0051] S203: Input the running data of each device into the trained load prediction model for processing to obtain the predicted load of each device.

[0052] Specifically, the training process of the load prediction model in step S203 is specifically as follows:

[0053] S2031: Extract the historical running data of each device from the preset database through a preset extraction tool.

[0054] In this embodiment, the preset extraction tool can be the PromQL language or other extraction tools.

[0055] Among them, the PromQL language is the built-in data query language of Prometheus, which is specifically used to process time series data.

[0056] In this embodiment, the preset database can be a local time series storage unit or other databases.

[0057] In this embodiment, the historical operation data is time series data.

[0058] S2032: Perform feature selection processing on each piece of historical operation data to obtain each piece of selected historical operation data.

[0059] Specifically, perform time dimension, system indication, node status label, and prediction target selection processing on each piece of historical operation data to obtain each piece of selected historical operation data.

[0060] In this embodiment, the time dimension can be set to hours, minutes, and holidays.

[0061] In this embodiment, the system metrics can be set to central processor usage rate, memory usage, disk I / O, network transmission traffic, task running time, and other metrics.

[0062] In this embodiment, the node status labels can be set to node type, number of running tasks, and load level.

[0063] In this embodiment, the prediction target can be set to the usage rate within any one of the next 1 minute, 5 minutes, or 15 minutes.

[0064] S2033: Perform preprocessing on each piece of selected historical operation data to obtain each piece of processed historical operation data.

[0065] Specifically, step S2033 specifically includes:

[0066] S20331: According to the preset sliding window, perform time series extraction processing on each piece of selected historical operation data to obtain each piece of extracted historical operation data.

[0067] In this embodiment, the preset sliding window can be any one of a 30-minute, 1-hour, or 3-hour time sliding window, which is used to predict the value of a single future time point through historical time series data.

[0068] S20332: Perform outlier adjustment processing on each piece of extracted historical operation data to obtain each piece of adjusted historical operation data.

[0069] In this embodiment, the outlier adjustment is null value filling, outlier correction, and other outlier adjustments.

[0070] S20333: Standardize each adjusted historical operation data to obtain each standardized historical operation data.

[0071] In this embodiment, the standardization process is standard deviation standardization or normalization.

[0072] S20334: Perform label construction processing on each standardized historical operation data to obtain each processed operation data.

[0073] In this embodiment, label construction uses the data at time t + n as the prediction target for the current window.

[0074] S2034: Train a preset load prediction model based on each processed historical operation data to obtain a trained load prediction model.

[0075] Specifically, export each processed historical operation data as JSON format data through PromQL language, and train a preset load prediction model based on the JSON format data to obtain a trained load prediction model.

[0076] In this embodiment, the load prediction model can be a long short-term memory network, a gradient boosting tree model, a random forest, or other models, which can be selected according to requirements.

[0077] Among them, the application advantages of the long short-term memory network are: having the ability to remember time series, being suitable for capturing the trends and periodicities of load changes over time; the applicable scenario is: fine prediction of the operation changes of each device in the short term (seconds or minutes).

[0078] Among them, the application advantages of the gradient boosting tree model are: strong multi-dimensional feature modeling ability, high robustness, fast training, and strong interpretability; the applicable scenario is: multi-variable concurrent load modeling.

[0079] Among them, the application advantages of the random forest are: it can be used as a comparison model to verify the prediction stability and generalization ability; the applicable scenario is: used to establish a model accuracy baseline.

[0080] In addition, the training architecture of the load prediction model includes: a training framework and a model training platform.

[0081] Among them, the training framework is a preset programming language.

[0082] Among them, the model training platform can be a local training platform, a remote GPU node, or a scheduling cluster training platform.

[0083] In addition, after training the load prediction model, the trained load prediction model can be deployed as a Kubernetes microservice and called through an HTTP POST request.

[0084] Among them, Kubernetes is a portable and extensible open-source platform for managing containerized workloads and services.

[0085] Among them, the HTTP POST request is a request method in the HTTP protocol.

[0086] In addition, after step S2034, steps a to b are further included:

[0087] Step a: Evaluate the metrics of the trained load prediction model to obtain an evaluation result.

[0088] In this embodiment, the metrics are mean absolute error, root mean square error, coefficient of determination, and model bias analysis.

[0089] Among them, the model bias analysis is used to analyze holiday fluctuations and task peak response capabilities, etc.

[0090] Step b: Update the trained load prediction model according to the evaluation result to obtain an updated load prediction model.

[0091] In this embodiment, the update method includes: relabeling the data in the time period with a higher prediction error, scheduling the retraining task every day at midnight, and pushing the new model to the online service to retain the old model to support gray-box testing and rollback.

[0092] S204: Input each candidate task and the current resource status into a preset priority evaluation model for processing to obtain the priority metrics of each candidate task.

[0093] Specifically, step S204 specifically includes:

[0094] S2041: Obtain multiple scheduling parameters of each candidate task.

[0095] In this embodiment, the multiple scheduling parameters are the urgency value, the resource utilization benefit value, and the migration cost value.

[0096] S2042: Determine the weights of each scheduling parameter according to the current resource status.

[0097] Exemplarily, when the current resource status is the real-time service guarantee status, the weights of the urgency value, the resource utilization benefit value, and the migration cost value are 0.7, 0.2, and 0.1 respectively, which are used to give priority to high-priority tasks and delay ordinary tasks.

[0098] When the current resource status is the energy consumption optimization status, the weights of the urgency value, the resource utilization benefit value, and the migration cost value are 0.2, 0.6, and 0.2 respectively, which are used to try to merge tasks and improve resource utilization.

[0099] When the current resource state is in a resource - strained state, the weights of the urgency value, the resource - utilization benefit value, and the migration - cost value are 0.3, 0.3, and 0.4 respectively, which are used to avoid the migration of tasks with high migration costs.

[0100] S2043: Calculate the priority index of the corresponding candidate task according to each scheduling parameter and each weight.

[0101] In this embodiment, each scheduling parameter includes an urgency value, a resource - utilization benefit value, and a migration - cost value; correspondingly, the calculation formula for calculating the priority index of the corresponding candidate task according to each scheduling parameter and each weight includes:

[0102]

[0103] In the formula, Score is the priority index of any candidate task; Purgency is the urgency value, w1 is the corresponding weight; Refficiency is the resource - utilization benefit value, w2 is the corresponding weight; Coverhead is the migration - cost value, w3 is the corresponding weight.

[0104] In addition, the historical scheduling state can also be used as auxiliary calculation data for the priority index.

[0105] S205: Input the predicted load of each device and the priority index of each candidate task into a preset scheduling model for processing to obtain the scheduling policy of each candidate task.

[0106] Specifically, step S205 specifically includes:

[0107] S2051: Obtain the resource idle ratio, current energy - consumption data, and historical scheduling data of each device.

[0108] In this embodiment, the historical scheduling data is the historical task - completion rate and failure rate.

[0109] S2052: Input the predicted load of each device, the priority index of each candidate task, the resource idle ratio of each device, the current energy - consumption data, and the historical scheduling data into a preset scheduling model for processing to obtain the scheduling policy corresponding to each candidate task.

[0110] In this embodiment, the predicted load is the load condition after 5 minutes, 10 minutes, or 15 minutes.

[0111] Exemplarily, the scheduling policy is: scheduling the i - th candidate task to the local node, delaying the scheduling of the i - th candidate task, migrating the i - th candidate task to other specified nodes, or starting or shutting down low - priority background tasks on the node.

[0112] In this embodiment, the preset scheduling model can be regarded as a preset Agent, that is, a task scheduler. Kafka can be used as the Agent status reporting and experience data channel, and the model parameters are saved in lightweight key-value stores such as Redis / Etcd.

[0113] Among them, Kafka is a distributed data stream platform that enables applications to publish, subscribe to, store, and process message streams in real time; Redis is a high-performance in-memory database, and Etcd is a distributed strongly consistent key-value store.

[0114] Among them, the Agent belongs to a part of the shared Q-network, and the training process of the shared Q-network includes steps c to e:

[0115] Step c: Collect the experience samples of each device through the central learner.

[0116] In this embodiment, the experience samples of each device are the resource utilization rate, power consumption per unit time, and asynchronous delay data of each device.

[0117] Step d: Optimize each experience sample through an incentive function to obtain each optimized sample.

[0118] In this embodiment, the calculation formula of the incentive function includes:

[0119]

[0120] In the formula, Reward is the sample optimization value of any device; ResourceUtilization is the resource utilization rate of the device; PowerConsumption is the power consumption per unit time; TaskDelay is the asynchronous delay data; α, β, and γ are adjustable parameters.

[0121] Step e: Train the shared Q-network according to each optimized sample.

[0122] S206: Schedule the corresponding candidate tasks according to each scheduling policy.

[0123] Specifically, schedule the corresponding candidate tasks according to each scheduling policy and the preset scheduler.

[0124] Among them, the preset scheduler can be a K8s scheduler or other schedulers.

[0125] In addition, the preset priority evaluation model and the preset scheduling model can be linked, and the corresponding linkage key points are:

[0126] Embed the scoring module into the default scheduler or resource scheduling platform of the custom cluster; support task pre-sorting and priority queue mechanism; support the following policy linkages: high-priority tasks preempt low-priority tasks, tasks at the end of the queue are dynamically delayed, tasks on hot nodes are merged, and intensive tasks are isolated and deployed.

[0127] In summary, the task scheduling and processing method provided in this embodiment collects the operation data of each device; obtains the task list to be processed and the current resource status, where the task list includes multiple candidate tasks; inputs the operation data of each device into the trained load prediction model for processing to obtain the predicted load of each device; inputs each candidate task and the current resource status into the preset priority evaluation model for processing to obtain the priority index of each candidate task; inputs the predicted load of each device and the priority index of each candidate task into the preset scheduling model for processing to obtain the scheduling strategy of each candidate task; schedules the corresponding candidate tasks according to each scheduling strategy. Through the cooperation of the trained load prediction model, the preset priority evaluation model, and the preset scheduling model, the operation data of each device, the current resource status, and each candidate task are processed in real time to determine the task scheduling strategy that meets the current business, so that it can adapt to various task resource scheduling environments and meet the scheduling requirements and real-time requirements of each candidate task.

[0128] In addition, the task scheduling and processing method provided in this embodiment uses the predicted load generated by the trained load prediction model and evaluates the priority of each candidate task through the preset priority evaluation model, making the generation of subsequent task scheduling strategies more accurate.

[0129] In addition, the task scheduling and processing method provided in this embodiment determines the scheduling strategy of each candidate task through the preset scheduling model, realizing adaptive task scheduling and improving the response speed of the server.

[0130] Figure 3 Schematic flow of the task scheduling and processing method provided in the embodiment of the present application Figure 2 In the embodiment of the present application, based on the embodiment provided Figure 2 a detailed description is given of the specific implementation method for power consumption adjustment corresponding to each device. As Figure 3 shown, the task scheduling and processing method includes:

[0131] S301: Obtain the power consumption of each device per unit time.

[0132] In this embodiment, each device includes a central processing unit, a network card, a disk, and a memory module; correspondingly, step S301 specifically includes:

[0133] S3011: Integrate and export the plug-in through a preset monitoring tool.

[0134] In this embodiment, the preset monitoring tool can be the Prometheus tool or other monitoring tools.

[0135] Among them, the discussion about Prometheus has been elaborated in step S201 and will not be repeated here.

[0136] S3012: Collect the power consumption of the central processing unit per unit time through the exporter plugin.

[0137] S3013: Obtain the power consumption of the network card, disk, and memory module per unit time through the preset metric collection tool.

[0138] In this embodiment, the preset metric collection tool can be the NodeExporter tool or other collection tools.

[0139] Among them, NodeExporter is a core component in the Prometheus monitoring system, specifically used to collect hardware and metric data of the operating system.

[0140] S302: Determine the ratio of the usage rate to the power consumption of each device.

[0141] S303: Generate a corresponding historical energy consumption trend chart based on the historical power consumption of each device.

[0142] S304: Determine the time ratio of the active state to the idle state of each device.

[0143] S305: Determine whether the corresponding device is in a low-load state based on each power consumption, each ratio, each historical energy consumption trend chart, and each time ratio.

[0144] S306: If it is determined that any device is in a low-load state, adjust the power consumption of the device according to the preset energy-saving strategy.

[0145] Exemplarily, if it is determined that the central processing unit is in a low-load state, adjust the power consumption of the central processing unit according to the preset energy-saving strategy.

[0146] Among them, the preset energy-saving strategy can be one or more of the DVFS regulation strategy, service freezing strategy, task merging and node offline strategy, and Pod dynamic scheduling optimization strategy.

[0147] Among them, the DVFS regulation strategy is the dynamic voltage and frequency adjustment strategy, which is used to enable dynamic voltage and frequency adjustment, dynamically adjust the frequency of the central processing unit according to the load, and reduce energy consumption.

[0148] Among them, the service freezing strategy is used to pause or reduce the priority of non-critical services and release resources.

[0149] Among them, the task merging and node offline strategy is used to merge and migrate multiple low-load tasks to at least a small number of nodes, and after releasing redundant nodes, node offline is performed.

[0150] Among them, the Pod dynamic scheduling optimization strategy is used to cooperate with the K8s scheduler to readjust the affinity and anti-affinity rules of Pods to promote resource centralization.

[0151] Among them, a Pod is the smallest scheduling and management unit in the Kubernetes cluster, and its essence is a set of containers that share network, storage, and lifecycle resources.

[0152] In summary, the task scheduling processing method provided in this embodiment obtains the power consumption of each device per unit time; determines the ratio of the usage rate to the power consumption of each device; generates a corresponding historical energy consumption trend chart based on the historical power consumption of each device; determines the time ratio of the active state to the idle state of each device; determines whether the corresponding device is in a low-load state according to each power consumption, each ratio, each historical energy consumption trend chart, and each time ratio; if it is determined that any device is in a low-load state, the power consumption of the device is adjusted according to a preset energy-saving strategy, so that during the task scheduling processing, the energy consumption of the server is reduced.

[0153] Figure 4 is a flowchart of the task scheduling processing method provided in an embodiment of the present application Figure 3 . In the embodiment of the present application, on the basis of the Figure 2 embodiment provided, a detailed description is given of the specific implementation method of retraining the trained load prediction model, preset priority evaluation model, and preset scheduling model. As Figure 4 shown, the task scheduling processing method includes:

[0154] S401: Collect the optimization feedback data stream, where the optimization feedback data stream contains multiple task scheduling process data.

[0155] In this embodiment, the multiple task scheduling process data includes one or more of the following data: task execution logs, historical task scheduling data, load prediction deviations, scheduling policy execution results, task scheduling effects, node status changes, and other process data.

[0156] In addition, the ELK is used to assist data inflow and build a visualization channel to flow the optimization feedback data stream into Kafka.

[0157] Among them, ELK is a complete set of log centralized processing solutions, including: an open-source distributed search engine, a data collection engine, and an open-source Web interface tool.

[0158] Among them, Kafka is a distributed data stream platform that enables applications to publish, subscribe to, store, and process message streams in real time.

[0159] S402: Determine whether the quantity of each task scheduling process data is greater than a preset threshold, and / or determine whether each task scheduling process data triggers a preset policy failure warning.

[0160] In this embodiment, the preset threshold can be any value among 100, 200, or 300, or it can be other values.

[0161] Exemplarily, the triggering of the preset policy failure warning can be in the following ways:

[0162] Among the data of each task scheduling process, when the node resource idle rate is continuously higher than 65% for 3 hours, a warning is triggered, where the normal threshold < 50%; when there is a cumulative deviation in the time series data, such as the prediction deviation continuously exceeds ±15% for 5 cycles, a warning is triggered; when the prediction accuracy of the machine learning model drops to the set threshold, a warning is triggered; when there is an abnormal collaboration between subsystems, such as the response delay of the scheduling instruction exceeds the timeout threshold of 500 ms, a warning is triggered.

[0163] S403: If it is determined that the quantity of multiple task scheduling process data is greater than the preset threshold, and / or a preset policy failure warning is triggered, then retrain the trained load prediction model, preset priority evaluation model, and preset scheduling model to obtain a new load prediction model, a new priority evaluation model, and a new scheduling model.

[0164] Specifically, if it is determined that the quantity of multiple task scheduling process data is greater than the preset threshold, and / or a preset policy failure warning is triggered, then trigger a training task through Kafka messages to retrain the trained load prediction model, preset priority evaluation model, and preset scheduling model to obtain a new load prediction model, a new priority evaluation model, and a new scheduling model.

[0165] Among them, the retraining process of the trained load prediction model is: use the new cycle data to retrain the trained load prediction model to obtain a new load prediction model.

[0166] Among them, the retraining process of the preset priority evaluation model is: based on the feedback data of the actual scheduling effect, readjust the weight parameters of the evaluation function in the preset priority evaluation model to obtain a new priority evaluation model.

[0167] Among them, the retraining process of the preset scheduling model is: use the DQN policy network to input a new state-action-reward sequence to update the preset scheduling model to obtain a new scheduling model.

[0168] In addition, after the training of each model is completed, push each new model to the model repository and automatically deploy each new model to the online service, and a version rollback mechanism is supported.

[0169] In addition, each model can be hosted in a Git repository. The management and update process through the GitOps mechanism is as follows: when each model changes, it triggers the CI Pipeline to automatically build and deploy the image, and cooperates with the K8s scheduler for hot image update; introduce A / B Test or Canary release to implement gray-box testing and version comparison for risk control.

[0170] Among them, the Git repository is a business working directory tracked and managed by Git. It contains the historical records and version information of the business, facilitating code branch management and version control.

[0171] Among them, the GitOps mechanism is to use the Git code repository to manage the deployment process of infrastructure and application code.

[0172] Among them, the CI Pipeline is a continuous integration pipeline.

[0173] Among them, A / B Test is an experimental method that makes data-driven decisions by comparing the effects of different versions of web pages, functions, or operation plans.

[0174] Among them, Canary release is a progressive software release strategy.

[0175] In summary, the task scheduling and processing method provided in this embodiment collects an optimized feedback data stream, where the optimized feedback data stream includes multiple task scheduling process data; determines whether the quantity of each task scheduling process data is greater than a preset threshold, and / or determines whether each task scheduling process data triggers a preset policy failure alarm; if it is determined that the quantity of multiple task scheduling process data is greater than the preset threshold, and / or triggers a preset policy failure alarm, then retrain the trained load prediction model, preset priority evaluation model, and preset scheduling model to obtain a new load prediction model, a new priority evaluation model, and a new scheduling model, and continuously optimize the corresponding models to improve the operation efficiency of server task retrieval.

[0176] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases, the former is a better implementation method.

[0177] Figure 5 This is the structural schematic diagram of the task scheduling and processing device provided in the embodiment of the present application. As Figure 5 shown, the embodiment of the present application also provides a task scheduling and processing device, including: a collection module 501, a first acquisition module 502, a first processing module 503, a second processing module 504, a third processing module 505, and a scheduling module 506.

[0178] The acquisition module 501 is used to acquire the operation data of each device;

[0179] The first obtaining module 502 is used to obtain the task list to be processed and the current resource status, where the task list includes multiple candidate tasks;

[0180] The first processing module 503 is used to input the operation data of each device into the trained load prediction model for processing to obtain the predicted load of each device;

[0181] The second processing module 504 is used to input each candidate task and the current resource status into the preset priority evaluation model for processing to obtain the priority index of each candidate task;

[0182] The third processing module 505 is used to input the predicted load of each device and the priority index of each candidate task into the preset scheduling model for processing to obtain the scheduling strategy of each candidate task;

[0183] The scheduling module 506 is used to schedule the corresponding candidate tasks according to each scheduling strategy.

[0184] In a possible implementation manner, the device further includes:

[0185] The extraction module is used to extract the historical operation data of each device from the preset database through a preset extraction tool;

[0186] The fourth processing module is used to perform feature selection processing on each historical operation data to obtain each selected historical operation data;

[0187] The fifth processing module is used to perform preprocessing on each selected historical operation data to obtain each processed historical operation data;

[0188] The first training module is used to train the preset load prediction model according to each processed historical operation data to obtain the trained load prediction model.

[0189] In a possible implementation manner, the fifth processing module specifically includes:

[0190] The first processing unit is used to perform time series extraction processing on each selected historical operation data according to a preset sliding window to obtain each extracted historical operation data;

[0191] The second processing unit is used to perform outlier adjustment processing on each extracted historical operation data to obtain each adjusted historical operation data;

[0192] The third processing unit is used to perform normalization processing on each adjusted historical operation data to obtain each normalized historical operation data;

[0193] The fourth processing unit is configured to perform label construction processing on each standardized historical operation data to obtain each processed operation data.

[0194] In a possible implementation manner, the fourth processing module is specifically configured to: perform time dimension, system indication, node status label, and prediction target selection processing on each historical operation data to obtain each selected historical operation data.

[0195] In a possible implementation manner, the apparatus further includes:

[0196] An evaluation module, configured to evaluate the trained load prediction model to obtain an evaluation result;

[0197] An update module, configured to update the trained load prediction model according to the evaluation result to obtain an updated load prediction model.

[0198] In a possible implementation manner, the second processing module 504 specifically includes:

[0199] An acquisition unit, configured to acquire multiple scheduling parameters of each candidate task;

[0200] A determination unit, configured to determine the weight of each scheduling parameter according to the current resource status;

[0201] A calculation unit, configured to calculate the priority index of the corresponding candidate task according to each scheduling parameter and each weight.

[0202] In a possible implementation manner, each scheduling parameter includes an urgency value, a resource utilization benefit value, and a migration cost value; correspondingly, the calculation formula for calculating the priority index of the corresponding candidate task according to each scheduling parameter and each weight includes:

[0203]

[0204] In the formula, Score is the priority index of any candidate task; Purgency is the urgency value, w1 is the corresponding weight; Refficiency is the resource utilization benefit value, w2 is the corresponding weight; Coverhead is the migration cost value, w3 is the corresponding weight.

[0205] In a possible implementation manner, the third processing module 505 specifically includes:

[0206] An acquisition unit, configured to acquire the resource idle ratio, the current energy consumption data, and the historical scheduling data of each device;

[0207] A processing unit, configured to input the predicted load of each device, the priority metrics of each candidate task, the resource idle ratio of each device, the current energy consumption data, and the historical scheduling data into a preset scheduling model for processing, so as to obtain the scheduling strategy corresponding to each candidate task.

[0208] In a possible implementation, the acquisition module 501 is specifically configured to: control a preset acquisition tool to periodically access the monitoring ports of each device through a preset request, and acquire the operation data of each device.

[0209] In a possible implementation, the device further includes:

[0210] A second acquisition module, configured to acquire the power consumption of each device per unit time;

[0211] A first determination module, configured to determine the ratio of the usage rate to the power consumption of each device;

[0212] A generation module, configured to generate a corresponding historical energy consumption trend chart according to the historical power consumption of each device;

[0213] A second determination module, configured to determine the time ratio of the active state to the idle state of each device;

[0214] A first judgment module, configured to judge whether the corresponding device is in a low-load state according to each power consumption, each ratio, each historical energy consumption trend chart, and each time ratio;

[0215] If it is determined that any device is in a low-load state, the power consumption of the device is adjusted according to a preset energy-saving strategy.

[0216] In a possible implementation, each device includes a central processing unit, a network card, a disk, and a memory module; correspondingly, the second acquisition module specifically includes:

[0217] An integration unit, configured to integrate and export the exporter plugin through a preset monitoring tool;

[0218] An acquisition unit, configured to acquire the power consumption of the central processing unit per unit time through the exporter plugin;

[0219] A obtaining unit, configured to obtain the power consumption of the network card, the disk, and the memory module per unit time through a preset metric acquisition tool.

[0220] In a possible implementation, the device further includes:

[0221] A collection module, configured to collect an optimization feedback data stream, where the optimization feedback data stream includes multiple task scheduling process data;

[0222] A second judgment module, configured to judge whether the quantity of each task scheduling process data is greater than a preset threshold, and / or judge whether each task scheduling process data triggers a preset policy failure warning;

[0223] A second training module, configured to retrain the trained load prediction model, preset priority evaluation model, and preset scheduling model to obtain a new load prediction model, a new priority evaluation model, and a new scheduling model if it is determined that the quantity of multiple task scheduling process data is greater than a preset threshold, and / or a preset policy failure warning is triggered.

[0224] For the descriptions of the features in the embodiments corresponding to the task scheduling processing device, reference may be made to the relevant descriptions of the embodiments corresponding to the task scheduling processing method, which will not be elaborated here one by one.

[0225] Figure 6 It is a schematic structural diagram of an electronic device provided by the present application. As Figure 6 shown, the electronic device provided in this embodiment includes: at least one processor 601 and a memory 602. Optionally, the electronic device further includes a communication component 603. Among them, the processor 601, the memory 602, and the communication component 603 are connected through a bus.

[0226] In a specific implementation process, at least one processor 601 executes computer execution instructions stored in the memory 602, so that at least one processor 601 executes the above-mentioned task scheduling processing method embodiment.

[0227] For the specific implementation process of the processor 601, reference may be made to the above method embodiment, and its implementation principle and technical effect are similar, which will not be elaborated here in this embodiment.

[0228] In the above embodiment, it should be understood that the processor may be a central processing unit (Central Processing Unit, abbreviated as: CPU), or other general-purpose processors, digital signal processors (Digital Signal Processor, abbreviated as: DSP), application specific integrated circuits (Application Specific Integrated Circuit, abbreviated as: ASIC), etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the application can be directly embodied as being executed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0229] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0230] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of easy representation, the buses in the drawings of this application are not limited to only one bus or one type of bus.

[0231] An embodiment of the present application also provides a computer-readable storage medium, in which a computer program is stored. Wherein, the computer program is set to execute the steps in any of the above-mentioned task scheduling processing method embodiments when running.

[0232] In an exemplary embodiment, the above-mentioned computer-readable storage medium may include, but is not limited to: various media such as a USB flash drive, a read-only memory (ROM for short), a random access memory (RAM for short), a mobile hard disk, a magnetic disk, or an optical disc, etc., which can store computer programs.

[0233] An embodiment of the present application also provides a computer program product. The above-mentioned computer program product includes a computer program, and when the computer program is executed by a processor, it implements the steps in any of the above-mentioned task scheduling processing method embodiments.

[0234] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium. The non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the steps in any of the above-mentioned task scheduling processing method embodiments.

[0235] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0236] The above has introduced in detail a task scheduling and processing method and an electronic device provided by the present application. Specific examples are used in this article to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A task scheduling and processing method, characterized in that, Including: Collecting the operation data of each device; Obtaining the task list to be processed and the current resource status, where the task list contains multiple candidate tasks; Inputting the operation data of each device into the trained load prediction model for processing to obtain the predicted load of each device; Inputting each candidate task and the current resource status into a preset priority evaluation model for processing to obtain the priority index of each candidate task; Inputting the predicted load of each device and the priority index of each candidate task into a preset scheduling model for processing to obtain the scheduling strategy of each candidate task; Scheduling the corresponding candidate tasks according to each scheduling strategy.

2. The task scheduling processing method according to claim 1, wherein The training process of the load prediction model includes: Extracting the historical operation data of each device from a preset database through a preset extraction tool; Performing feature selection processing on each historical operation data to obtain each selected historical operation data; Performing preprocessing on each selected historical operation data to obtain each preprocessed historical operation data; Training a preset load prediction model according to each preprocessed historical operation data to obtain the trained load prediction model.

3. The task scheduling processing method according to claim 2, wherein The performing preprocessing on each selected historical operation data to obtain each preprocessed historical operation data includes: Performing time series extraction processing on each selected historical operation data according to a preset sliding window to obtain each extracted historical operation data; Performing outlier adjustment processing on each extracted historical operation data to obtain each adjusted historical operation data; Performing normalization processing on each adjusted historical operation data to obtain each normalized historical operation data; Performing label construction processing on each normalized historical operation data to obtain each processed operation data.

4. The task scheduling processing method according to claim 2, wherein The performing feature selection processing on each historical operation data to obtain each selected historical operation data includes: Performing time dimension, system indication, node status label, and prediction target selection processing on each historical operation data to obtain each selected historical operation data.

5. The task scheduling and processing method according to claim 2, wherein After training the preset load prediction model according to each preprocessed historical operation data to obtain the trained load prediction model, it further includes: Evaluating the indicators of the trained load prediction model to obtain an evaluation result; Updating the trained load prediction model according to the evaluation result to obtain an updated load prediction model.

6. The task scheduling processing method according to claim 1, characterized in that The inputting each candidate task and the current resource status into a preset priority evaluation model for processing to obtain the priority index of each candidate task includes: Obtaining multiple scheduling parameters of each candidate task; Determining the weights of each scheduling parameter according to the current resource status; Calculating the priority index of the corresponding candidate task according to each scheduling parameter and each weight.

7. The task scheduling and processing method according to claim 6, characterized in that, Where each scheduling parameter includes an urgency value, a resource utilization benefit value, and a migration cost value; Correspondingly, the calculation formula for calculating the priority index of the corresponding candidate task according to each scheduling parameter and each weight includes: Wherein, Score is the priority index of any candidate task; Purgency is the urgency value, w1 is the corresponding weight; Refficiency is the resource utilization benefit value, w2 is the corresponding weight; Coverhead is the migration cost value, w3 is the corresponding weight.

8. The task scheduling processing method according to claim 1, characterized in that, The method of inputting the predicted load of each device and the priority index of each candidate task into a preset scheduling model for processing to obtain the scheduling strategy of each candidate task includes: Obtaining the resource idle ratio, current energy consumption data and historical scheduling data of each device; Inputting the predicted load of each device, the priority index of each candidate task, the resource idle ratio of each device, the current energy consumption data and the historical scheduling data into a preset scheduling model for processing to obtain the scheduling strategy corresponding to each candidate task.

9. The task scheduling processing method according to claim 1, wherein The method of collecting the operation data of each device includes: Controlling a preset collection tool to periodically access the monitoring ports of each device through a preset request to collect the operation data of each device.

10. The task scheduling processing method according to claim 1, wherein It further includes: Obtaining the power consumption of each device per unit time; Determining the ratio of the utilization rate to the power consumption of each device; Generating a corresponding historical energy consumption trend chart according to the historical power consumption of each device; Determining the time ratio of the active state to the idle state of each device; Judging whether the corresponding device is in a low-load state according to each power consumption, each ratio, each historical energy consumption trend chart and each time ratio; If it is determined that any device is in a low-load state, adjusting the power consumption of the device according to a preset energy-saving strategy.

11. The task scheduling processing method according to claim 10, wherein Wherein each device includes a central processing unit, a network card, a disk and a memory module; Correspondingly, obtaining the power consumption of each device per unit time includes: Integrating an exporter plugin through a preset monitoring tool; Collecting the power consumption of the central processing unit per unit time through the exporter plugin; Obtaining the power consumption of the network card, the disk and the memory module per unit time through a preset index collection tool.

12. The task scheduling processing method according to claim 1, wherein It further includes: Collecting an optimization feedback data stream, wherein the optimization feedback data stream contains a plurality of task scheduling process data; Judging whether the quantity of each task scheduling process data is greater than a preset threshold, and / or judging whether each task scheduling process data triggers a preset policy failure warning; If it is determined that the quantity of the plurality of task scheduling process data is greater than a preset threshold, and / or triggers a preset policy failure warning, re-training the trained load prediction model, preset priority evaluation model and preset scheduling model to obtain a new load prediction model, a new priority evaluation model and a new scheduling model.

13. An electronic device, characterized in that, It includes: A memory for storing a computer program; A processor for implementing the steps of the task scheduling processing method according to any one of claims 1 to 12 when executing the computer program.

14. A computer-readable storage medium, characterized in that, A computer program is stored in the computer-readable storage medium, wherein the computer program implements the steps of the task scheduling processing method according to any one of claims 1 to 12 when executed by a processor.

15. A computer program product, comprising a computer program, characterized in that, The computer program implements the steps of the task scheduling processing method according to any one of claims 1 to 12 when executed by a processor.

Images