Batch task scheduling method and device, electronic equipment and storage medium
By obtaining and analyzing the resource utilization rate of the target batch running system, performing system classification and task scheduling, the problem of task accumulation in batch task scheduling is solved, and processing efficiency and resource utilization are improved.
Patent Information
- Application Number
- CN202411573754.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-05
- Publication Date
- 2025-02-11
AI Technical Summary
Existing batch task scheduling methods can easily lead to tasks with average or lighter importance, affecting processing efficiency.
By obtaining the historical and current resource utilization rate of the target batch system, performing system classification, identifying system status, and obtaining task attributes in a busy state, performing task scheduling, and realizing task allocation to systems with low resource utilization.
The processing efficiency of batch tasks is improved, and the task allocation is dynamically adjusted, the system pressure is reduced and resource utilization is improved.
Smart Images
Figure CN119415236A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of network optimization and artificial intelligence technology, and in particular to a batch task scheduling method and device, an electronic device and a storage medium. Background Art
[0002] Batch tasks refer to tasks that automatically process large amounts of data on a regular basis in the background. For example, in the financial field, transactions in insurance, funds and other products will generate a large amount of financial data. By scheduling batch tasks for a large amount of financial data, the processing efficiency of financial data can be improved.
[0003] Currently, common batch task scheduling usually grades batch tasks according to pre-set task importance and gives priority to tasks with high task importance. However, this method easily leads to the accumulation of a large number of tasks with average or low task importance, thereby affecting the task processing efficiency. Therefore, how to improve the processing efficiency of batch tasks has become a technical problem that needs to be solved urgently. Summary of the invention
[0004] The main purpose of the embodiments of the present application is to propose a face mask detection method and device, an electronic device and a storage medium, aiming to improve the processing efficiency of batch tasks.
[0005] To achieve the above object, a first aspect of an embodiment of the present application proposes a batch task scheduling method, the method comprising:
[0006] Obtain the highest resource utilization rate of the target batch system in the historical period and the current resource utilization rate at the current moment, wherein the historical period is the day before the date to which the current moment belongs;
[0007] Based on the highest resource utilization and the current resource utilization, the target batch system is systematically classified to obtain a divertible system and a non-divertible system, wherein the divertible system represents a target batch system to which task allocation can be performed, and the non-divertible system represents a target batch system to which task allocation cannot be performed;
[0008] Based on the current resource utilization, identifying the system state of the non-dividable system, wherein the system state includes a busy state and a normal state;
[0009] If the system status indicates that the non-dividable system is in a busy state, then obtaining a task attribute of a target task, wherein the target task is a task running in the non-dividable system;
[0010] Based on the task attributes and the divertible system, the target task is scheduled.
[0011] In some embodiments, the step of scheduling the target task based on the task attribute and the divertible system includes:
[0012] Based on the task attributes, priority evaluation is performed on the target task to obtain target evaluation data;
[0013] Based on the current resource utilization rate and the target evaluation data, flow planning is performed on the target task to obtain task planning data;
[0014] Based on the task planning data and the divertible system, the target task is assigned to obtain task assignment data.
[0015] In some embodiments, the step of performing a priority evaluation on the target task based on the task attribute to obtain target evaluation data includes:
[0016] Based on a preset attribute evaluation mapping table and the task attributes, an attribute evaluation is performed on the target task to obtain attribute evaluation data, wherein the attribute evaluation mapping table is used to represent a mapping relationship between the task attributes and the evaluation data;
[0017] Based on the preset performance characteristic weights, the attribute evaluation data are merged to obtain task evaluation data;
[0018] The task evaluation data is normalized to obtain the target evaluation data.
[0019] In some embodiments, the flow planning of the target task based on the current resource utilization and the target evaluation data to obtain the task planning data includes:
[0020] Based on a preset resource utilization threshold and the current resource utilization, the state of the non-dividable system is subdivided to obtain a state subdivision result, wherein the state subdivision result is used to indicate that the non-dividable system is in a generally busy state or a severely busy state;
[0021] If the state segmentation result indicates that the non-divertable system is in a generally busy state, diverting the target task to obtain the task planning data;
[0022] If the state segmentation result indicates that the non-divertable system is in a severely busy state, the target task is classified based on the target evaluation data and a preset evaluation threshold to obtain a low evaluation data task and a high evaluation data task;
[0023] Performing task suspension preprocessing on the low-assessment data task to obtain first planning data;
[0024] Performing task operation preprocessing on the high-assessment data task to obtain second planning data;
[0025] The first planning data and the second planning data are merged to obtain the task planning data.
[0026] In some embodiments, after performing a priority evaluation on the target task based on the task attributes and obtaining target evaluation data, the method further includes:
[0027] Based on the task allocation data, monitoring resource utilization of the divertible system to obtain monitored resource utilization;
[0028] Based on the monitoring resource utilization, the divertible system is secondary classified to obtain a system category.
[0029] In some embodiments, if the system status indicates that the non-dividable system is in a busy state, obtaining the task attribute of the target task includes:
[0030] When the system status indicates that the non-dividable system is in a busy state, querying the target task to obtain basic task information;
[0031] Performing a data source query on the target task to obtain task data source information;
[0032] Performing a life cycle query on the target task to obtain the task life cycle;
[0033] Performing an execution information query on the target task to obtain task execution information;
[0034] Performing dependency query on the target task to obtain task dependency;
[0035] The task basic information, the task data source information, the task life cycle, the task execution information and the task dependency are merged to obtain the task attributes.
[0036] In some embodiments, the system classification of the target batch system based on the highest resource utilization and the current resource utilization to obtain a divertible system and a non-divertible system includes:
[0037] Calculating a resource utilization ratio of the target batch system based on the highest resource utilization and the current resource utilization;
[0038] If the resource utilization ratio is less than a preset ratio threshold, the target batch running system is used as the divertible batch running system;
[0039] If the resource utilization ratio is greater than or equal to the ratio threshold, the target batch running system is used as the non-divertible batch running system.
[0040] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a batch task scheduling device, the device comprising:
[0041] A data acquisition module, used to obtain the highest resource utilization rate of the target batch system in the historical period and the current resource utilization rate at the current moment, wherein the historical period is the day before the date to which the current moment belongs;
[0042] A system classification module, for classifying the target batch system based on the highest resource utilization and the current resource utilization to obtain a divertible system and a non-divertible system, wherein the divertible system represents a target batch system to which task allocation can be performed, and the non-divertible system represents a target batch system to which task allocation cannot be performed;
[0043] A state identification module, used for identifying the system state of the non-dividable system based on the current resource utilization, wherein the system state includes a busy state and a normal state;
[0044] an attribute acquisition module, based on if the system state indicates that the non-dividable system is in a busy state, acquiring a task attribute of a target task, wherein the target task is a task running in the non-dividable system;
[0045] A task scheduling module is used to schedule the target task based on the task attributes and the divertible system.
[0046] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.
[0047] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.
[0048] The batch task scheduling method and device, electronic device and storage medium proposed in the present application obtain the highest resource utilization rate of the target batch system in the historical period and the current resource utilization rate at the current moment, and classify the target batch system according to the highest resource utilization rate and the current resource utilization rate to obtain a divertible system and a non-divertible system. Furthermore, according to the current resource utilization rate, the system status of the non-divertible system is identified, and when the system status indicates that the non-divertible system is busy, the task attributes of the target task are obtained, and the target task is scheduled according to the task attributes and the divertible system. It is possible to allocate tasks in a system with higher resource utilization to a system with lower resource utilization, thereby speeding up the system's processing speed for batch tasks and improving the system's processing efficiency for batch tasks. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1 is a flow chart of a batch task scheduling method provided by an embodiment of the present application;
[0050] Figure 2 yes Figure 1 Flow chart of step S102 in FIG.
[0051] Figure 3 yes Figure 1 Flow chart of step S104 in FIG.
[0052] Figure 4 yes Figure 1 Flow chart of step S105 in FIG.
[0053] Figure 5 yes Figure 4 Flow chart of step S401 in FIG.
[0054] Figure 6 yes Figure 4 Flow chart of step S402 in FIG.
[0055] Figure 7 is a flow chart of a batch task scheduling method provided by another embodiment of the present application;
[0056] Figure 8 It is a structural diagram of a batch task scheduling device provided in an embodiment of the present application;
[0057] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION
[0058] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0059] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0061] First, some nouns involved in this application are analyzed:
[0062] Normalization: Normalization is a data preprocessing technique that aims to scale the data to a uniform range to eliminate the dimensionality effects between features and make them comparable on the same scale. The normalization method can improve the training efficiency and prediction accuracy of machine learning models, especially when using distance metrics and gradient descent optimization.
[0063] Network task scheduling: Network task scheduling refers to the process of optimizing performance and improving efficiency by rationally arranging and allocating the order of task execution and system resources in a computer network environment. It involves strategies such as task priority management, resource allocation, load balancing, and latency optimization, aiming to reduce response time, improve throughput, and ensure system stability. Effective network task scheduling is essential to support efficient data transmission and processing, improve user experience, and maximize resource utilization.
[0064] Batch tasks refer to tasks that automatically process large amounts of data on a regular basis in the background. For example, in the financial field, transactions in insurance, funds and other products will generate a large amount of financial data. By scheduling batch tasks for a large amount of financial data, the processing efficiency of financial data can be improved.
[0065] Currently, common batch task scheduling usually grades batch tasks according to pre-set task importance and gives priority to tasks with high task importance. However, this method easily leads to the accumulation of a large number of tasks with average or low task importance, thereby affecting the task processing efficiency. Therefore, how to improve the processing efficiency of batch tasks has become a technical problem that needs to be solved urgently.
[0066] Based on this, the embodiments of the present application provide a batch task scheduling method and device, an electronic device and a storage medium, aiming to improve the processing efficiency of batch tasks.
[0067] The batch task scheduling method and device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the batch task scheduling method in the embodiments of the present application is described.
[0068] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0069] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0070] The batch task scheduling method provided in the embodiment of the present application relates to the field of network optimization and artificial intelligence technology. The batch task scheduling method provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or it can be configured as a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the batch task scheduling method, etc., but is not limited to the above forms.
[0071] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.
[0072] Figure 1 is an optional flowchart of the batch task scheduling method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S105.
[0073] Step S101, obtaining the highest resource utilization rate of the target batch system in the historical period and the current resource utilization rate at the current moment, wherein the historical period is the day before the date to which the current moment belongs;
[0074] Step S102, based on the highest resource utilization and the current resource utilization, the target batch system is classified into a divertible system and a non-divertible system, wherein the divertible system represents the target batch system to which task allocation can be performed, and the non-divertible system represents the target batch system to which task allocation cannot be performed;
[0075] Step S103, based on the current resource utilization, identifying the system state of the non-dividable system, wherein the system state includes a busy state and a normal state;
[0076] Step S104, if the system state indicates that the non-dividable system is in a busy state, then obtaining the task attributes of the target task, wherein the target task is a task running in the non-dividable system;
[0077] Step S105, scheduling the target task based on the task attributes and the divertible system.
[0078] Steps S101 to S105 shown in the embodiment of the present application can obtain the highest resource utilization of the target batch system in the historical period and the current resource utilization at the current moment by querying the resource utilization of the target batch system, wherein the historical period is the day before the date to which the current moment belongs. Furthermore, the target batch system is classified according to the highest resource utilization and the current resource utilization to obtain a divertible system and a non-divertible system, wherein the divertible system represents a target batch system to which task allocation can be performed, and the non-divertible system represents a target batch system to which task allocation cannot be performed. Secondly, according to the current resource utilization, the system status of the non-divertible system is identified, wherein the system status includes a busy state and a normal state, and when the system status indicates that the non-divertible system is in a busy state, the task attributes of the target task are obtained, wherein the target task is a task running in the non-divertible system. Finally, according to the task attributes and the divertible system, the target task is scheduled. Therefore, the present application obtains the highest resource utilization rate of the target batch system in the historical period and the current resource utilization rate at the current moment, and classifies the target batch system according to the highest resource utilization rate and the current resource utilization rate to obtain a divertible system and a non-divertible system. Furthermore, according to the current resource utilization rate, the system status of the non-divertible system is identified, and when the system status indicates that the non-divertible system is busy, the task attributes of the target task are obtained, and the target task is scheduled according to the task attributes and the divertible system. It can realize the allocation of tasks in the system with higher resource utilization to the system with lower resource utilization, thereby speeding up the system's processing speed for batch tasks and improving the system's processing efficiency for batch tasks.
[0079] In step S101 of some embodiments, the target batch system refers to a system used to process the target task. For example, in the financial field, financial companies generate a large amount of financial data every day. The processing flow of financial data can be to use a data warehouse tool to process the collected financial data. In this processing flow, the data warehouse tool is the target batch system. Resource utilization can be data such as CPU utilization and memory utilization of the target batch system.
[0080] This application can query the resource utilization of the target batch system according to the data of the required time period, so as to obtain the historical resource utilization of the historical time period and the current resource utilization at the current moment. Furthermore, the data with the highest resource utilization can be filtered out from the historical resource utilization to obtain the highest resource utilization of the historical time period.
[0081] In step S102 of some embodiments, by comparing the current resource utilization of the target batch system at the current moment with the highest resource utilization of the target batch system in the historical period, and when the result of the data comparison meets the pre-set resource utilization condition, the target batch system is divided into a divertible system and a non-divertible system.
[0082] It is important to know that the divertible system is a target batch system with a currently low resource utilization rate. Therefore, the divertible system can also perform task allocation to reduce the workload of the non-divertible system. The non-divertible system is a target batch system with a currently high resource utilization rate. Therefore, in order to avoid excessive pressure on the non-divertible system, task allocation cannot be performed on the non-divertible system.
[0083] For details, see Figure 2 In some embodiments, step S102 may include but is not limited to steps S201 to S203:
[0084] Step S201, calculating the resource utilization ratio of the target batch system based on the highest resource utilization and the current resource utilization;
[0085] Step S202: if the resource utilization ratio is less than a preset ratio threshold, the target batch running system is used as a divertable batch running system;
[0086] Step S203: If the resource utilization ratio is greater than or equal to the ratio threshold, the target batch running system is regarded as a non-divertable batch running system.
[0087] In step S201 of some embodiments, the resource utilization ratio of the target batch system can be obtained by calculating the quotient between the maximum resource utilization and the current resource utilization. It should be noted that the resource utilization ratio can be used to indicate the task pressure of the target batch system at the current moment. Specifically, the larger the resource utilization ratio, the greater the task pressure of the target batch system at the current moment, and the smaller the resource utilization ratio, the smaller the task pressure of the target batch system at the current moment.
[0088] In step S202 and step S203 of some embodiments, the ratio threshold is a threshold used to distinguish the size of the target batch system pressure. For example, in a financial data processing scenario, the ratio threshold may be 0.5.
[0089] When the resource utilization ratio of the target batch system is less than the ratio threshold, it indicates that the current pressure of the target batch system is small. Therefore, the target batch system can be used as a divertible batch system. When the resource utilization ratio of the target batch system is greater than or equal to the ratio threshold, it indicates that the current pressure of the target batch system is large. Therefore, the target batch system can be used as a non-divertible batch system. For example, in the financial data processing scenario, when the resource utilization ratio between the current resource utilization and the maximum resource utilization is 0.34, it indicates that the data warehouse tool can be diverted. Therefore, the data warehouse tool is a divertible system. When the resource utilization ratio between the current resource utilization and the maximum resource utilization is 0.55, it indicates that the data warehouse tool cannot be diverted. Therefore, the data warehouse tool is a non-divertible system.
[0090] In steps S201 to S203 shown in the present embodiment, the resource utilization ratio of the target batch system is determined by calculating the ratio between the current resource utilization and the maximum resource utilization, and when the resource utilization ratio is less than the ratio threshold, the target batch system is treated as a divertible system, and when the resource utilization ratio is greater than or equal to the ratio threshold, the target batch system is treated as a non-divertible system. This method can simply, quickly and effectively screen out divertible systems that can be used for task allocation and non-divertible systems that cannot be assigned tasks in the target batch system, thereby improving the efficiency of system classification during batch task processing.
[0091] In step S103 of some embodiments, the current resource utilization of the non-dividable system may be compared with a preset utilization threshold to determine whether the system state of the non-dividable system is a busy state or a normal state.
[0092] It should be noted that the present application can screen out target batch systems with greater pressure from non-divertable systems based on current resource utilization, thereby achieving task diversion and priority processing for target batch systems with greater pressure.
[0093] In step S104 of some embodiments, when the system state of the non-divertable system is busy, it indicates that the non-divertable system is under great pressure and needs to divert tasks to reduce system pressure. Therefore, the present application can obtain the task attributes of the target task by performing an attribute query on the target task of the non-divertable system in the busy state, so as to facilitate the diversion processing of the target task of the non-divertable system in the busy state.
[0094] It is important to know that the task attributes of the target task generally include the basic information of the task, data source information, life cycle, execution information, dependencies, etc., among which the basic information can be the basic attributes of the task, such as task name, task label, task type, etc.; the data source information can be the data source information of the task, such as the task source library name, the task source table name, etc.; the life cycle can be the effective storage time of the task; the execution information can be the information of the task during the execution process, such as the task job number, task running status, task running time, etc.; the dependency can be the dependency between tasks, for example, task A can be the dependent task of task B, and tasks B and C are dependent on each other.
[0095] For details, see Figure 3 In some embodiments, step S104 may include but is not limited to steps S301 to S306:
[0096] Step S301, when the system status indicates that the non-divertable system is in a busy state, query the target task information to obtain basic task information;
[0097] Step S302, query the data source of the target task to obtain task data source information;
[0098] Step S303, query the life cycle of the target task to obtain the task life cycle;
[0099] Step S304, query the execution information of the target task to obtain the task execution information;
[0100] Step S305, query the dependency relationship of the target task to obtain the task dependency relationship;
[0101] Step S306, merging the basic information of the task, the information of the data source of the task, the life cycle of the task, the execution information of the task and the dependency of the task to obtain the task attributes.
[0102] In steps S301 to S305 of some embodiments, when the system state of the non-dividable system is busy, the basic information, data source, life cycle, execution information, dependencies and other information of the target task in the non-dividable system can be queried, and the basic information of the task, task data source information, task life cycle, task execution information, task dependencies and other data can be obtained.
[0103] In step S306 of some embodiments, the task attributes can be obtained by simply aggregating the above-mentioned task basic information, task data source information, task life cycle, task execution information, task dependency and other data.
[0104] In steps S301 to S306 illustrated in the present embodiment, when the system state of the non-divertable system is indicated as a busy state, information is queried on the target task to obtain basic task information, a data source query is performed on the target task to obtain task data source information, a life cycle query is performed on the target task to obtain task life cycle, an execution information query is performed on the target task to obtain task execution information, a dependency query is performed on the target task to obtain task dependency, and then data merging is performed on the basic task information, task data source information, task life cycle, task execution information and task dependency to obtain task attributes, so as to facilitate identification of the importance and operation status of the task, thereby performing task scheduling on the target task in a targeted manner and improving the scheduling efficiency of batch tasks.
[0105] In step S105 of some embodiments, after obtaining the task attributes of the target task in the non-dividable system in a busy state, the target task can be graded according to the priority based on the task attributes to obtain target evaluation data, and then the target task can be scheduled according to the current resource utilization of the non-dividable system and the target evaluation data of the target task to obtain task allocation data.
[0106] For details, see Figure 4 In some embodiments, step S105 may include but is not limited to steps S401 to S403:
[0107] Step S401, based on the task attributes, priority evaluation is performed on the target task to obtain target evaluation data;
[0108] Step S402, based on the current resource utilization and target evaluation data, flow planning is performed on the target task to obtain task planning data;
[0109] Step S403, based on the task planning data and the divertible system, the target task is assigned to obtain task assignment data.
[0110] In step S401 of some embodiments, attribute evaluation data of each data of the target task can be obtained by scoring each data in the task attribute, and then all attribute evaluation data of the target task can be merged to obtain target evaluation data of the target task.
[0111] For details, see Figure 5 In some embodiments, step S401 may include but is not limited to steps S501 to S503:
[0112] Step S501, based on a preset attribute evaluation mapping table and task attributes, an attribute evaluation is performed on the target task to obtain attribute evaluation data, wherein the attribute evaluation mapping table is used to represent a mapping relationship between task attributes and evaluation data;
[0113] Step S502, based on the preset performance feature weights, the attribute evaluation data is merged to obtain task evaluation data;
[0114] Step S503, normalize the task evaluation data to obtain target evaluation data.
[0115] In step S501 of some embodiments, the present application may query the evaluation data mapped to the task attributes of the target task, that is, the attribute evaluation data of each item of data in the task attributes, according to a pre-set attribute evaluation mapping table.
[0116] In step S502 of some embodiments, the performance feature weight may be the weight ratio of each data in the task attribute in the priority evaluation, and the performance feature weight may be pre-set according to the importance of the data.
[0117] The present application performs weighted summation of the above attribute evaluation data according to pre-set performance feature weights to obtain task evaluation data.
[0118] In step S503 of some embodiments, in order to facilitate comparison of the size of the task evaluation data of the target task, the present application may obtain the target evaluation data by normalizing the task evaluation data.
[0119] It should be noted that the present application can achieve normalization processing of task evaluation data through normalization formulas such as Min-Max normalization, standard deviation normalization, and logarithmic normalization.
[0120] In steps S501 to S503 shown in the present embodiment, attribute evaluation is performed on the target task according to the preset attribute evaluation mapping table and task attributes to obtain attribute evaluation data, and then the attribute evaluation data is merged according to the preset performance characteristic weights to obtain task evaluation data. Furthermore, the task evaluation data is normalized to obtain target evaluation data, which can indicate the priority level of the target task, thereby facilitating the target batch system to regulate the target task and improving the processing efficiency of batch tasks.
[0121] In step S402 of some embodiments, the state of the non-dividable system in a busy state can be secondary classified according to the current resource utilization rate, and the processing flow of the target task can be clarified based on the state classification result of the secondary classification to obtain the task planning data of the target task.
[0122] For details, see Figure 6 In some embodiments, step S402 may include but is not limited to steps S601 to S606:
[0123] Step S601, based on a preset resource utilization threshold and a current resource utilization, the state of the non-dividable system is subdivided to obtain a state subdivision result, wherein the state subdivision result is used to indicate that the non-dividable system is in a generally busy state or a severely busy state;
[0124] Step S602, if the state segmentation result indicates that the non-dividable system is in a generally busy state, then the target task is diverted to obtain task planning data;
[0125] Step S603: if the state segmentation result indicates that the non-divertable system is in a severely busy state, the target task is classified based on the target evaluation data and a preset evaluation threshold to obtain a low evaluation data task and a high evaluation data task;
[0126] Step S604, performing task suspension preprocessing on the low-assessment data task to obtain first planning data;
[0127] Step S605, performing task operation preprocessing on the high-assessment data task to obtain second planning data;
[0128] Step S606: merge the first planning data and the second planning data to obtain task planning data.
[0129] In step S601 of some embodiments, the resource utilization threshold may be threshold data for distinguishing a non-dividable system in a busy state.
[0130] The present application can identify the state interval of the non-dividable system by comparing the current resource utilization with the resource utilization threshold, thereby obtaining the state segmentation result of the non-dividable system. For example, if the resource utilization threshold indicates that the CPU utilization is 60% and the memory utilization is 60%, then the current utilization indicates that the CPU utilization is 55% and the memory utilization is 55%. The non-dividable system can be a state segmentation result of a generally busy state, and the current utilization indicates that the CPU utilization is 65% and the memory utilization is 65%. The non-dividable system can be a state segmentation result of a severely busy state.
[0131] In step S602 of some embodiments, when the state subdivision result indicates that the non-divertible system is in a generally busy state, the system pressure of the non-divertible system can be further reduced by diverting the target task in the non-divertible system, and task planning data can be generated based on the task diversion planning for the target task.
[0132] In steps S603 to S606 of some embodiments, when the state subdivision result indicates that the non-dividable system is in a severely busy state, it indicates that the non-dividable system is seriously overloaded. At this time, the system pressure of the non-dividable system can be reduced by suspending the processing of target tasks with lower priorities. In detail, when the state subdivision result indicates that the non-dividable system is in a severely busy state, the target tasks can be prioritized according to the target evaluation data of the target tasks and a pre-set evaluation threshold to obtain low-evaluation data tasks and high-evaluation data tasks, wherein the evaluation threshold can be data for distinguishing the priority of the target tasks. Furthermore, a pause process can be executed for low-evaluation data tasks evaluated as low priority, and a first planning data can be generated. A continue running process can be executed for low-evaluation data tasks evaluated as high priority, and a second planning data can be generated. Finally, the above-mentioned first planning data and the second planning data are merged to obtain the task planning data of the non-dividable system in a severely busy state.
[0133] In steps S601 to S606 shown in the present embodiment, the state of the non-divertable system is subdivided according to the resource utilization threshold and the current resource utilization to obtain the state subdivision result, and when the state subdivision result indicates that the non-divertable system is in a generally busy state, the target task is diverted to obtain task planning data, and when the state subdivision result indicates that the non-divertable system is in a severely busy state, the target task is classified based on the target evaluation data and the preset evaluation threshold to obtain low-evaluation data tasks and high-evaluation data tasks, the low-evaluation data task is pre-processed for task suspension to obtain the first planning data, the high-evaluation data task is pre-processed for task operation to obtain the second planning data, and the first planning data is merged with the second planning data to obtain task planning data, which can reduce the task accumulation of the target batch running system with greater system pressure, thereby improving the processing efficiency of batch tasks.
[0134] In step S403 of some embodiments, after obtaining the task planning data of the target task, the target task can be assigned according to the task planning data to obtain the task assignment data. It should be noted that when the task planning data indicates that the target task is to be diverted, the target task in the non-divertible system needs to be assigned to the divertible system. The assignment rule may be to assign tasks according to the current resource utilization of the divertible system, with the task being assigned to the divertible system with low current resource utilization as a priority.
[0135] In steps S401 to S403 illustrated in the present embodiment, priority evaluation is performed on the target task according to the task attributes to obtain target evaluation data, and then flow planning is performed on the target task according to the current resource utilization and the target evaluation data to obtain task planning data. Finally, task allocation is performed on the target task according to the task planning data and the divertible system to obtain task allocation data. This enables dynamic adjustment of batch tasks, so that target batch tasks with greater system pressure can give priority to important tasks, and target batch tasks with less system pressure can be allocated to more target tasks, thereby improving the processing efficiency of batch tasks.
[0136] It is important to know that the current resource utilization of the target batch system is not constant. Therefore, it is also necessary to monitor the current resource utilization of the target batch system to schedule batch tasks in real time.
[0137] For details, see Figure 7 In some embodiments, after step S105, the batch task scheduling method may further include but is not limited to steps S701 to S702:
[0138] Step S701, based on the task allocation data, monitor the resource utilization of the divertible system to obtain the monitored resource utilization;
[0139] Step S702: Based on the monitoring resource utilization, secondary classification is performed on the divertible system to obtain a system category.
[0140] In step S701 and step S702 of some embodiments, after the target task is scheduled according to the above-mentioned task allocation data, the resource utilization of the divertible system receiving the target task is monitored in real time to obtain the monitored resource utilization, and when the ratio between the monitored resource utilization of the divertible system and the highest resource utilization of the divertible system in the historical period is greater than or equal to the above-mentioned ratio threshold, the divertible system is classified as a non-divertible system to achieve real-time classification of the divertible system and obtain the system category of the divertible system.
[0141] The present application obtains the highest resource utilization rate of the target batch system in the historical period and the current resource utilization rate at the current moment, and classifies the target batch system according to the highest resource utilization rate and the current resource utilization rate to obtain a divertible system and a non-divertible system. Furthermore, according to the current resource utilization rate, the system status of the non-divertible system is identified, and when the system status indicates that the non-divertible system is busy, the task attributes of the target task are obtained, and the target task is scheduled according to the task attributes and the divertible system. It is possible to allocate tasks in a system with higher resource utilization to a system with lower resource utilization, thereby speeding up the system's processing speed for batch tasks and improving the system's processing efficiency for batch tasks.
[0142] See also Figure 8 The embodiment of the present application further provides a batch task scheduling device, which can implement the above batch task scheduling method, and the device includes:
[0143] The data acquisition module 801 is used to obtain the highest resource utilization rate of the target batch system in the historical period and the current resource utilization rate at the current moment, wherein the historical period is the day before the date to which the current moment belongs;
[0144] The system classification module 802 is used to classify the target batch system based on the highest resource utilization rate and the current resource utilization rate to obtain a divertible system and a non-divertible system, wherein the divertible system represents a target batch system to which task allocation can be performed, and the non-divertible system represents a target batch system to which task allocation cannot be performed;
[0145] A state identification module 803 is used to identify the system state of the non-dividable system based on the current resource utilization, wherein the system state includes a busy state and a normal state;
[0146] The attribute acquisition module 804 acquires the task attribute of the target task based on the fact that the system state indicates that the non-dividable system is in a busy state, wherein the target task is a task running in the non-dividable system;
[0147] The task scheduling module 805 is used to schedule the target task based on the task attributes and the divertible system.
[0148] The specific implementation of the batch task scheduling device is substantially the same as the specific implementation of the batch task scheduling method described above, and will not be described in detail herein.
[0149] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above batch task scheduling method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.
[0150] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:
[0151] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;
[0152] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the batch task scheduling method of the embodiment of this application;
[0153] Input / output interface 903, used to implement information input and output;
[0154] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);
[0155] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);
[0156] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .
[0157] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned batch task scheduling method is implemented.
[0158] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0159] The batch task scheduling method, batch task scheduling device, electronic device and storage medium provided in the embodiments of the present application can obtain the highest resource utilization rate of the target batch system in the historical period and the current resource utilization rate of the target batch system at the current moment by querying the resource utilization rate of the target batch system, wherein the historical period is the day before the date to which the current moment belongs. Furthermore, according to the highest resource utilization rate and the current resource utilization rate, the target batch system is systematically classified to obtain a divertible system and a non-divertible system, wherein the divertible system represents a target batch system to which task allocation can be performed, and the non-divertible system represents a target batch system to which task allocation cannot be performed. Secondly, according to the current resource utilization rate, the system status of the non-divertible system is identified, wherein the system status includes a busy status and a normal status, and when the system status indicates that the non-divertible system is in a busy status, the task attributes of the target task are obtained, wherein the target task is a task running in the non-divertible system. Finally, according to the task attributes and the divertible system, the target task is scheduled.
[0160] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0161] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.
[0162] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0163] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.
[0164] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0165] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0166] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0167] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0168] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0169] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.
[0170] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.
Claims
1. A batch task scheduling method, characterized in that: The method comprises: Obtain the highest resource utilization rate of the target batch system in the historical period and the current resource utilization rate at the current moment, wherein the historical period is the day before the date to which the current moment belongs; Based on the highest resource utilization and the current resource utilization, the target batch system is systematically classified to obtain a divertible system and a non-divertible system, wherein the divertible system represents a target batch system to which task allocation can be performed, and the non-divertible system represents a target batch system to which task allocation cannot be performed; Based on the current resource utilization, identifying the system state of the non-dividable system, wherein the system state includes a busy state and a normal state; If the system status indicates that the non-dividable system is in a busy state, then obtaining a task attribute of a target task, wherein the target task is a task running in the non-dividable system; Based on the task attributes and the divertible system, the target task is scheduled.
2. The method according to claim 1, characterized in that The step of scheduling the target task based on the task attribute and the divertible system includes: Based on the task attributes, priority evaluation is performed on the target task to obtain target evaluation data; Based on the current resource utilization rate and the target evaluation data, flow planning is performed on the target task to obtain task planning data; Based on the task planning data and the divertible system, the target task is assigned to obtain task assignment data.
3. The method according to claim 2, characterized in that The step of performing a priority evaluation on the target task based on the task attributes to obtain target evaluation data includes: Based on a preset attribute evaluation mapping table and the task attributes, an attribute evaluation is performed on the target task to obtain attribute evaluation data, wherein the attribute evaluation mapping table is used to represent a mapping relationship between the task attributes and the evaluation data; Based on the preset performance characteristic weights, the attribute evaluation data are merged to obtain task evaluation data; The task evaluation data is normalized to obtain the target evaluation data.
4. The method according to claim 2, characterized in that: The flow planning of the target task based on the current resource utilization rate and the target evaluation data to obtain the task planning data includes: Based on a preset resource utilization threshold and the current resource utilization, the state of the non-dividable system is subdivided to obtain a state subdivision result, wherein the state subdivision result is used to indicate that the non-dividable system is in a generally busy state or a severely busy state; If the state segmentation result indicates that the non-divertable system is in a generally busy state, diverting the target task to obtain the task planning data; If the state segmentation result indicates that the non-divertable system is in a severely busy state, the target task is classified based on the target evaluation data and a preset evaluation threshold to obtain a low evaluation data task and a high evaluation data task; Performing task suspension preprocessing on the low-assessment data task to obtain first planning data; Performing task operation preprocessing on the high-assessment data task to obtain second planning data; The first planning data and the second planning data are merged to obtain the task planning data.
5. The method according to claim 2, characterized in that: After performing priority evaluation on the target task based on the task attributes and obtaining target evaluation data, the method further includes: Based on the task allocation data, monitoring resource utilization of the divertible system to obtain monitored resource utilization; Based on the monitoring resource utilization, the divertible system is secondary classified to obtain a system category.
6. The method according to any one of claims 1 to 5, characterized in that: If the system state indicates that the non-dividable system is in a busy state, obtaining the task attribute of the target task includes: When the system status indicates that the non-dividable system is in a busy state, querying the target task to obtain basic task information; Performing a data source query on the target task to obtain task data source information; Performing a life cycle query on the target task to obtain the task life cycle; Performing an execution information query on the target task to obtain task execution information; Performing dependency query on the target task to obtain task dependency; The task basic information, the task data source information, the task life cycle, the task execution information and the task dependency are merged to obtain the task attributes.
7. The method according to any one of claims 1 to 5, characterized in that: The method of classifying the target batch running system based on the highest resource utilization rate and the current resource utilization rate to obtain a divertible system and a non-divertible system includes: Calculating a resource utilization ratio of the target batch system based on the highest resource utilization and the current resource utilization; If the resource utilization ratio is less than a preset ratio threshold, the target batch running system is used as the divertible batch running system; If the resource utilization ratio is greater than or equal to the ratio threshold, the target batch running system is used as the non-divertible batch running system.
8. A batch task scheduling device, characterized in that: The device comprises: A data acquisition module, used to obtain the highest resource utilization rate of the target batch system in the historical period and the current resource utilization rate at the current moment, wherein the historical period is the day before the date to which the current moment belongs; A system classification module, for classifying the target batch system based on the highest resource utilization and the current resource utilization to obtain a divertible system and a non-divertible system, wherein the divertible system represents a target batch system to which task allocation can be performed, and the non-divertible system represents a target batch system to which task allocation cannot be performed; A state identification module, used for identifying the system state of the non-dividable system based on the current resource utilization, wherein the system state includes a busy state and a normal state; an attribute acquisition module, based on if the system state indicates that the non-dividable system is in a busy state, acquiring a task attribute of a target task, wherein the target task is a task running in the non-dividable system; A task scheduling module is used to schedule the target task based on the task attributes and the divertible system.
9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the batch task scheduling method according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the batch task scheduling method according to any one of claims 1 to 7 is implemented.