Task scheduling balancing method and device and readable storage medium

Abstract

The invention provides a task scheduling balancing method and device and a readable storage medium. The method comprises the steps of obtaining a to-be-scheduled task; according to the time information of the to-be-scheduled tasks, performing time-sensitive task combination on the to-be-scheduled tasks, and distributing the to-be-scheduled tasks to corresponding time windows; task duplicate removal processing is carried out on the to-be-scheduled tasks with the distributed time windows, and optimized to-be-scheduled tasks are obtained; and determining a corresponding task scheduling balancing strategy according to the optimized to-be-scheduled task. According to the invention, for a large-scale task processing system, intelligent decision making is carried out through accurate analysis and utilization of time parameters, and intelligent task combination is carried out in the time dimension, so that stable use of system resources is maintained, and the resource utilization rate is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a task scheduling balancing method, device and readable storage medium. Background Art

[0002] As data scale continues to expand and computing resources are relatively limited, how to improve system resource utilization under limited computing resources and efficient task scheduling and deployment strategies have become key links.

[0003] In the existing technology, task scheduling usually simply relies on preset rules and fixed strategies, or optimizes strategies from the perspective of resource space load balancing, but does not adequately explore the sequence characteristics of the time dimension, making it difficult to adapt to dynamically changing workloads and environmental conditions.

[0004] Therefore, how to more scientifically balance task scheduling to improve resource utilization has become a problem that needs to be solved. Summary of the invention

[0005] The technical problem to be solved by the present application is to provide a task scheduling balancing method, device and readable storage medium to solve the problems existing in the prior art in view of the above-mentioned deficiencies in the prior art.

[0006] In a first aspect, the present application provides a task scheduling balancing method, the method comprising:

[0007] S1. Obtain the tasks to be scheduled;

[0008] S2. According to the time information of the tasks to be scheduled, the tasks to be scheduled are combined into time-sensitive tasks, and the tasks to be scheduled are allocated to corresponding time windows;

[0009] S3, performing task deduplication processing on the tasks to be scheduled that have been assigned time windows to obtain optimized tasks to be scheduled;

[0010] S4. Determine the corresponding task scheduling balancing strategy according to the optimized tasks to be scheduled.

[0011] In some embodiments, in S2, the time information of the task to be scheduled includes at least one of task submission time, expected running time, deadline, inter-task dependency, and historical execution time data of the task.

[0012] In some embodiments, S3 includes:

[0013] S32, performing internal logic analysis on the tasks to be scheduled, and determining tasks with the same internal logic as candidate tasks that can be deduplicated;

[0014] S32, performing task logic similarity comparison on candidate tasks that can be deduplicated, performing task deduplication processing based on the comparison results, and obtaining tasks to be scheduled after the first optimization.

[0015] In some embodiments, in S32, comparing the logical similarity of the candidate tasks that can be deduplicated includes:

[0016] Determine whether the task logic of the candidate task is consistent, wherein the task logic includes at least one of an operation type, an operation object, an operation frequency, an operation statement, and a shooting destination target;

[0017] If the task logic is inconsistent, the candidate tasks are determined to be inconsistent;

[0018] If the task logic is consistent, determine whether the task results of the candidate tasks are consistent;

[0019] If the task results are consistent, the candidate tasks are determined to be consistent;

[0020] If the task results are inconsistent, determine whether there is an inclusion relationship between the task results of the candidate tasks;

[0021] If there is an inclusion relationship between the task results, the candidate tasks are determined to be consistent;

[0022] If there is no inclusion relationship between the task results, it is determined that the candidate tasks are inconsistent.

[0023] In some embodiments, S3 further includes:

[0024] S33. Perform secondary deduplication processing on the tasks to be scheduled after the initial optimization through text parsing and hash algorithm to obtain the tasks to be scheduled after the secondary optimization.

[0025] In some embodiments, S33 includes:

[0026] The task description text of the task to be scheduled after the first optimization is parsed through natural language processing technology to extract semantic features and logical structure;

[0027] The similarity of semantic features and logical structures is evaluated by a text similarity model to obtain a first matching result;

[0028] Generate a hash signature of the task to be scheduled after the first optimization through a hash algorithm, and match the hash signatures of all tasks to obtain a second matching result;

[0029] A secondary deduplication process is performed based on the first matching result and the second matching result to obtain the tasks to be scheduled after secondary optimization.

[0030] In some embodiments, S4 includes:

[0031] According to the optimized tasks to be scheduled, the corresponding task scheduling balance strategy is determined by particle swarm optimization algorithm and ant colony algorithm.

[0032] In a second aspect, the present application provides a task scheduling and balancing device, characterized in that the device comprises:

[0033] A task acquisition module, which is configured to acquire tasks to be scheduled;

[0034] A task allocation module, which is configured to perform time-sensitive task combination on the tasks to be scheduled according to the time information of the tasks to be scheduled, and allocate the tasks to be scheduled to corresponding time windows;

[0035] A deduplication processing module is configured to perform task deduplication processing on the tasks to be scheduled that have been assigned time windows to obtain optimized tasks to be scheduled;

[0036] The strategy determination module is configured to determine the corresponding task scheduling balancing strategy according to the optimized tasks to be scheduled.

[0037] In a third aspect, the present application provides a task scheduling and balancing device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to implement the task scheduling and balancing method described in the first aspect.

[0038] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the task scheduling and balancing method described in the first aspect is implemented.

[0039] The task scheduling balancing method, device and readable storage medium provided by the present application include: obtaining the tasks to be scheduled; combining the tasks to be scheduled with time-sensitive tasks according to the time information of the tasks to be scheduled, and assigning the tasks to be scheduled to the corresponding time windows; performing task deduplication processing on the tasks to be scheduled with the assigned time windows to obtain optimized tasks to be scheduled; and determining the corresponding task scheduling balancing strategy according to the optimized tasks to be scheduled. The present application is aimed at large-scale task processing systems, and performs intelligent task combination in the time dimension by accurately analyzing and using time parameters for intelligent decision-making, so as to maintain the stability of system resource usage and improve resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0041] Figure 1 A flowchart of a task scheduling balancing method provided in an embodiment of the present application;

[0042] Figure 2 A schematic diagram of performing time-sensitive task combination provided in an embodiment of the present application;

[0043] Figure 3 A schematic diagram of performing task deduplication processing provided in an embodiment of the present application;

[0044] Figure 4 A schematic diagram of determining a corresponding task scheduling balancing strategy through a particle swarm optimization algorithm and an ant colony algorithm provided in an embodiment of the present application;

[0045] Figure 5 A 24-hour resource load curve trend chart before the use of a strategy of a certain instance provided in an embodiment of the present application;

[0046] Figure 6 A 24-hour resource load curve trend chart after the strategy of a certain instance provided in the embodiment of the present application is used;

[0047] Figure 7 A structural diagram of a task scheduling and balancing device provided in an embodiment of the present application;

[0048] Figure 8 A schematic diagram of the structure of another task scheduling and balancing device provided in an embodiment of the present application.

[0049] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0050] In order to enable those skilled in the art to better understand the technical solution of the present application, the implementation mode of the present application will be further described in detail below with reference to the accompanying drawings.

[0051] It should be understood that the specific embodiments and drawings described herein are only used to explain the present application, rather than to limit the present application.

[0052] It can be understood that, in the absence of conflict, the various embodiments in the present application and the various features in the embodiments can be combined with each other.

[0053] It can be understood that, for the convenience of description, the drawings of the present application only show the parts related to the present application, while the parts unrelated to the present application are not shown in the drawings.

[0054] It can be understood that each unit and module involved in the embodiments of the present application may correspond to only one physical structure, or may be composed of multiple physical structures, or multiple units and modules may be integrated into one physical structure.

[0055] It can be understood that the terms "first", "second", etc. in the embodiments of the present application are used to distinguish different objects, or to distinguish different processing of the same object, rather than to describe a specific order of objects.

[0056] It is understandable that, in the absence of conflict, the functions and steps marked in the flowcharts and block diagrams of the present application may occur in an order different from that marked in the drawings.

[0057] It is understood that the flowcharts and block diagrams of the present application illustrate the possible architectures, functions, and operations of the systems, devices, equipment, and methods according to the various embodiments of the present application. Among them, each box in the flowchart or block diagram may represent a unit, module, program segment, code, which contains executable instructions for implementing the specified functions. Moreover, each box or combination of boxes in the block diagram and flowchart may be implemented by a hardware-based system that implements the specified functions, or by a combination of hardware and computer instructions.

[0058] It can be understood that the units and modules involved in the embodiments of the present application can be implemented by software or hardware, for example, the units and modules can be located in a processor.

[0059] The existing technology generally has the following problems when performing task scheduling:

[0060] 1. Task running parameters need to be set manually, and computing resources cannot be automatically optimized and allocated in the time dimension, resulting in a large fluctuation range of resource load rate, that is, there is a long period of resource overload or no load, resulting in low resource utilization and uneven use;

[0061] 2. The internal logic of the task and the task results are not analyzed and compared, and there is a problem of repeated running of the same calculation logic, the same calculation results, or the included calculation tasks, resulting in ineffective and repeated consumption of resources.

[0062] In view of the above shortcomings, the present application proposes a task scheduling balancing method based on time feature driving to overcome the above technical problems existing in the existing related technologies. The main concepts of the present application include:

[0063] 1. By analyzing the time-related data of the scheduling tasks, establish an allocation strategy based on the time dimension to ensure that computing resources are used under a relatively stable curve to avoid long-term overload or no-load situations;

[0064] 2. By analyzing the processing logic within the task, the same or included computing tasks are screened out, so that the computing resources consumed once for the same task can be reused multiple times.

[0065] This application targets large-scale task processing systems, makes intelligent decisions through precise analysis and utilization of time parameters, and performs intelligent task combination in the time dimension to maintain stable use of system resources and improve resource utilization.

[0066] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0067] The present application provides a task scheduling and balancing method, the working process of which can be implemented by electronic devices, such as computers, handheld smart terminals, etc. For the convenience of explanation, the embodiments of the present application are described with the method execution subject being a computer.

[0068] Figure 1 A schematic diagram of a task scheduling balancing method provided in an embodiment of the present application, such as Figure 1 As shown, the present application provides a task scheduling balancing method, the method includes S1-S4, which are as follows:

[0069] S1. Obtain the tasks to be scheduled;

[0070] Among them, the tasks to be scheduled refer to multiple tasks that need to be scheduled and balanced.

[0071] S2. According to the time information of the tasks to be scheduled, the tasks to be scheduled are combined into time-sensitive tasks, and the tasks to be scheduled are allocated to corresponding time windows;

[0072] Specifically, according to the time attributes of the tasks, a dynamic priority sorting algorithm is constructed to predict and evaluate the impact of different task combinations on system resources when executed in the same time window, and to find a set of tasks that can smoothly consume system resources within a given time.

[0073] In some embodiments, in S2, the time information of the task to be scheduled includes at least one of task submission time, expected running time, deadline, inter-task dependency, and historical execution time data of the task.

[0074] Specifically, Figure 2 A schematic diagram of performing time-sensitive task combination provided in an embodiment of the present application, such as Figure 2 As shown, basic information of each task to be scheduled is collected, including but not limited to: task submission time, expected running time, deadline, dependencies between tasks, historical execution time data of tasks and other time-related attributes. The following table is a partial time parameter table of one of the preferred examples:

[0075] Configuration items type illustrate Tasktype enum Task Type Taskcost string Resource usage of tasks Tasktime timestamp Task execution time (range) Tasktimeduration time Task runtime Taskupstream integer Upstream dependencies of tasks Taskdownstream integer Downstream impact of tasks Taskpriority enum Task Priority Taskfailurecount integer Number of mission failures Taskfailureinterval time Task retry interval

[0076] Then, according to the changing rules of the system's available resources and the characteristics of the tasks, the time is divided into different time windows, and the tasks are assigned to their respective appropriate time windows according to their time attributes.

[0077] S3, performing task deduplication processing on the tasks to be scheduled that have been assigned time windows to obtain optimized tasks to be scheduled;

[0078] Specifically, the resource efficiency is further improved through the intelligent task deduplication function. Before the task enters the scheduling queue, logical parsing and hash comparison technology are used to identify task instances with the same logical essence and can be combined for execution, and they are merged to reduce unnecessary repeated calculations.

[0079] In some embodiments, S3 includes:

[0080] S32, performing internal logic analysis on the tasks to be scheduled, and determining tasks with the same internal logic as candidate tasks that can be deduplicated;

[0081] S32, performing task logic similarity comparison on candidate tasks that can be deduplicated, performing task deduplication processing based on the comparison results, and obtaining tasks to be scheduled after the first optimization.

[0082] Specifically, we first analyze the internal logic of the task, conduct an in-depth analysis of the scheduled task, and identify the internal logic of the task, such as SQL query statements, required data sources, and expected query results. Tasks with the same logic (such as the same SQL query, pointing to the same data source and expecting the same results) are considered as potential candidates for deduplication; then we perform task logic deduplication, and match and deduplicate tasks with similar logic through the task logic similarity comparison method. Once a duplicate task is found, only one instance is retained, and the remaining instances are marked as pending merging.

[0083] Figure 3 A schematic diagram of performing task deduplication processing provided in an embodiment of the present application, such as Figure 3 As shown, in some embodiments, in S32, the task logic similarity comparison is performed on the candidate tasks that can be deduplicated, including:

[0084] Determine whether the task logic of the candidate task is consistent, wherein the task logic includes at least one of an operation type, an operation object, an operation frequency, an operation statement, and a shooting destination target;

[0085] If the task logic is inconsistent, the candidate tasks are determined to be inconsistent;

[0086] If the task logic is consistent, determine whether the task results of the candidate tasks are consistent;

[0087] If the task results are consistent, the candidate tasks are determined to be consistent;

[0088] If the task results are inconsistent, determine whether there is an inclusion relationship between the task results of the candidate tasks;

[0089] If there is an inclusion relationship between the task results, the candidate tasks are determined to be consistent;

[0090] If there is no inclusion relationship between the task results, it is determined that the candidate tasks are inconsistent.

[0091] In this application, the consistency of tasks is determined by double verification, that is, both the process and the result must be the same to be considered consistent, including:

[0092] 1. Perform matching polling on tasks;

[0093] 2. Determine whether the internal logic is consistent. If yes, go to step 3; if not, go to step 6.

[0094] 3. Determine whether the result set is consistent. If yes, go to step 5; if not, go to step 4.

[0095] 4. Determine whether the result set has a containment relationship. If yes, go to step 5; if no, go to step 6.

[0096] 5. The judgment is consistent;

[0097] 6. Judged as inconsistent.

[0098] In some embodiments, S3 further includes:

[0099] S33. Perform secondary deduplication processing on the tasks to be scheduled after the initial optimization through text parsing and hash algorithm to obtain the tasks to be scheduled after the secondary optimization.

[0100] In some embodiments, S33 includes:

[0101] The task description text of the task to be scheduled after the first optimization is parsed through natural language processing technology to extract semantic features and logical structure;

[0102] The similarity of semantic features and logical structures is evaluated by a text similarity model to obtain a first matching result;

[0103] Generate a hash signature of the task to be scheduled after the first optimization through a hash algorithm, and match the hash signatures of all tasks to obtain a second matching result;

[0104] A secondary deduplication process is performed based on the first matching result and the second matching result to obtain the tasks to be scheduled after secondary optimization.

[0105] Specifically, before a task enters the scheduling queue, natural language processing technology is used to parse the task description text, extract semantic features and logical structure, and evaluate the similarity of task logic through a text similarity model combined with contextual information and user behavior analysis, and generate a preliminary task logic vector. The semantic similarity is calculated as follows:

[0106]

[0107] Among them, T1 and T2 are two task descriptions, V1 and V2 are their corresponding vector representations, and cosine similarity is calculated to evaluate the semantic similarity of task descriptions.

[0108] Then, a customized hash function is designed in the workflow, and a multi-level hash strategy is implemented to initially screen and verify the task logic vector generated in the previous step, and a hash signature is generated and compared with the hash signature of the existing task. If the hash signature matches, the logic parsing results are further used for in-depth comparison to confirm whether the task is truly repeated.

[0109] Among them, the local sensitive hashing is calculated as follows:

[0110]

[0111] Among them, x is the task description feature vector, r is the hash bucket size, b is the cardinality, and k is the number of hash buckets.

[0112] The Hamming distance of the hash signature is calculated as follows;

[0113]

[0114] Among them, H1 and H2 are the hash signatures of the two tasks, n is the length of the hash signature, and dH is their Hamming distance.

[0115] If both the task logic and the hash signature match, the task is marked as duplicate and merged or optimized as needed. If not, the new task is added to the scheduling queue, and the hash table and logic parsing model are updated to include the characteristics of the new task. At the same time, a feedback loop is established to continuously adjust the parameters of the hash function and logic parsing model based on the deduplication results to improve the accuracy and efficiency of subsequent task deduplication.

[0116] The task similarity threshold is calculated as follows:

[0117]

[0118] Here, α is a weight between 0 and 1, balancing the influence of semantic similarity and hash signature Hamming distance.

[0119] S4. Determine the corresponding task scheduling balancing strategy according to the optimized tasks to be scheduled.

[0120] Specifically, a dynamic task scheduling strategy is formulated for the deduplicated tasks according to the results of the task combination model, and then the task execution sequence is adjusted to ensure that the execution order of the logically deduplicated tasks is reasonably arranged while meeting the time requirements, so as to achieve the best system performance.

[0121] This step integrates the deduplicated task instances and incorporates them into the aforementioned theoretical framework of time-sensitive task combination according to their time attributes. It comprehensively considers the resource usage balance and task execution efficiency to generate an execution strategy that can both meet time constraints and minimize resource conflicts.

[0122] For example, when it is predicted that resources may be idle within a certain period of time, some non-urgent but moderately resource-consuming tasks are scheduled in advance; and when resources may reach peak periods, short-term and efficient urgent tasks are scheduled first to avoid resource accumulation.

[0123] In some embodiments, S4 includes:

[0124] According to the optimized tasks to be scheduled, the corresponding task scheduling balance strategy is determined by particle swarm optimization algorithm and ant colony algorithm.

[0125] Specifically, the task strategy generation of this application adopts the particle swarm optimization algorithm, which regards each particle as a task scheduling scheme. On this basis, the algorithm is customized and improved, and the ant colony algorithm is introduced for fusion to further improve the accuracy and efficiency of the scheduling strategy.

[0126] Figure 4 The schematic diagram of determining the corresponding task scheduling balance strategy through the particle swarm optimization algorithm and the ant colony algorithm provided in the embodiment of the present application is as follows: Figure 4 As shown, the process mainly includes the following steps:

[0127] 1. Collect and analyze historical data, identify peak hours for task submission, resource bottlenecks, and inefficient scheduling decisions through exploratory data analysis, and identify possible constraints, task patterns, and optimization space.

[0128] 2. Screen and test the identified constraints, such as task submission mode, resource usage, task duration, task dependencies, resource availability, deadlines, and priorities, and establish a customized mathematical model to enable the particle swarm algorithm to handle different types of tasks more intelligently and help identify task scheduling patterns and resource demand trends.

[0129] 3. Introduce the ant colony algorithm and particle swarm optimization algorithm for integration. Ant colony algorithm, with its advantages in path optimization problems, provides an effective local search mechanism for task scheduling strategy.

[0130] 4. Dynamically adjust the inertia weight. As the number of iterations increases, the inertia weight is gradually reduced to promote the algorithm to focus more on local search in the later stage. The specific calculation is as follows:

[0131]

[0132] Among them, Wmax and Wmin are the maximum and minimum values ​​of the inertia weight respectively, iter is the current iteration number, and itermax is the maximum iteration number.

[0133] 5. Initialization:

[0134] (1) Set the particle swarm size N, and the position of each particle is:

[0135] Xi∈R^N (indicates the start time of N tasks)

[0136] The speed of each particle is:

[0137] Vi∈R^N.

[0138] (2) Initialize the particle swarm and randomly assign each particle an initial position Xi and velocity Vi.

[0139] (3) Set the inertia weight w, acceleration constants c1 and c2, and the global and local optimal positions Pg and Pi.

[0140] (4) Initialize the ant colony and define the number of ants and pheromone concentration parameters.

[0141] 6. Perform iterations. For the number of iterations t = 1, 2, ..., max_iter:

[0142] (1) Update the speed of each particle:

[0143] Vi(t+1)=wVi(t)+c1rand()[Pi(t)-Xi(t)]+c2rand()*[Pg(t)-Xi(t)]

[0144] Among them, rand() returns a random number between [0,1] to introduce randomness; Pi(t) is the local optimal solution of particle i at time t; Pg(t) is the global optimal solution.

[0145] (2) Update the position of each particle while ensuring that the position is within the valid range (i.e., the task start time is reasonable):

[0146] Xi(t+1)=Xi(t)+Vi(t+1)

[0147] (3) According to the new position Xi(t + 1), calculate the variance of the resource utilization rate within 24 hours as the fitness function F(Xi), and update the pheromone concentration according to the fitness of the particles in the particle swarm algorithm:

[0148] F(Xi) = ∑((P_j(Xi) - μ)^2) / 24

[0149] Where, P_j(Xi) is the total resource utilization rate of the j-th hour calculated according to Xi, and μ is the average value of the resource utilization rates of all hours.

[0150] (5) Under the guidance of the global search of the particle swarm algorithm, the ants conduct a detailed search in the local area based on the pheromone concentration and heuristic factors (such as task urgency and resource availability) to find a better task scheduling scheme, and update the local optimal solution Pi(t + 1) of each particle:

[0151] If F(Xi(t + 1)) < F(Pi(t)), then Pi(t + 1) = Xi(t + 1).

[0152] (6) Combine the global search results of the particle swarm algorithm and the local search results of the ant colony algorithm, and dynamically adjust the task scheduling strategy to achieve the optimal allocation of the resource utilization rate, and update the global optimal solution Pg(t + 1):

[0153] If there exists a local optimal solution F(Pi(t + 1)) < F(Pg(t)) of a certain particle, then Pg(t + 1) = Pi(t + 1).

[0154] 7. Termination condition: When the preset maximum number of iterations max_iter is reached, or the global optimal solution no longer improves significantly, stop the iteration, and use the global optimal solution Pg as the final task scheduling scheme.

[0155] 8. Dynamically construct a task dependency graph. After using the scheduling strategy of the present application, the dependency and sequential relationships between tasks will be unfixed. Therefore, the present application dynamically updates the constructed task dependency graph to reflect the latest dependency relationships between tasks and feedback the dynamic changes of tasks.

[0156] For ease of understanding, the following is a specific example of the technical solution of the present application:

[0157] Suppose a series of batch tasks with different execution durations and deadlines enter the scheduling queue.

[0158] Using the method of the present application, first classify these tasks into corresponding time slots (such as time windows divided by hours or minutes) according to factors such as deadlines, priorities, and estimated execution durations.

[0159] Next, a machine learning algorithm is used to predict the system resource usage curve after different task combinations are executed in each time window, striving to make the curve as smooth as possible to maintain stable use of system resources.

[0160] Finally, the scheduler schedules tasks according to the optimized task combination order to ensure that tasks are completed on time while maximizing the use of system resources to avoid resource waste or bottlenecks.

[0161] The specific steps of the above example process are as follows:

[0162] 1. Time window division: Divide a day into 24 consecutive time periods

[0163] H={h_1,h_2,...,h_24}

[0164] Where h_j represents the jth hour.

[0165] 2. Task set modeling: Assume that the task set

[0166] T={Task_1,Task_2,...,Task_n}

[0167] Each task Task_i has attributes such as Tasktime_i (task execution time), Tasktimeduration_i (task running time), Taskcost_i (task resource utilization) and other parameters.

[0168] 3. Resource usage allocation: For each task, its resource usage in time period h_j is Taskcost_ij. If the task covers this time period during its operation, then

[0169] Taskcost_ij=Taskcost_i*(duration of the task in time period h_j / Tasktimeduration_i).

[0170] 4. Optimization goal of scheduling strategy: Set the objective function to reduce the variance of resource utilization within 24 hours:

[0171] min∑((P(h_j)-μ)^2) / 24

[0172] Where P(h_j) is the sum of resource utilization in the jth hour, and μ is the mean resource utilization over all hours.

[0173] 5. Constraints: The start and end time of each task must meet the following conditions:

[0174] Tasktime_i≤Tasktime_i+Tasktimeduration_i≤Tasktime_end_i

[0175] where Tasktime_end_i is the end time of task i.

[0176] Resource allocation constraints:

[0177]

[0178] Among them, C is the maximum total amount of resources that can be allocated per hour, and all tasks must be completed within 24 hours:

[0179] ∑(Tasktimeduration_i)≤24×3600seconds

[0180] 6. Generate scheduling strategy and execute it: Use the improved particle swarm algorithm to generate the optimal scheduling strategy, dynamically program the optimization of the objective function, recursively calculate the optimal allocation of all tasks until all tasks are allocated, and finally determine the strategy and execute it.

[0181] Through the particle swarm optimization algorithm, the speed and position update formulas are appropriately modified to adapt to the requirements of discrete space search. This application can continuously iterate and gradually optimize the start time of the task to achieve the goal of making resource utilization as balanced as possible within 24 hours.

[0182] Figure 5 A 24-hour resource load curve trend chart before the use of a strategy of an instance provided in an embodiment of the present application, Figure 6 The 24-hour resource load curve trend diagram after the strategy of a certain instance provided in the embodiment of the present application is used, such as Figure 5 as well as Figure 6 As shown, the present application provides a task scheduling balancing method for large-scale task processing systems. By accurately analyzing and utilizing time parameters to make intelligent decisions, intelligent task combinations are performed in the time dimension to maintain stable system resource usage and improve resource utilization.

[0183] For example, a company and its customers have a large amount of data, such as user behavior data, network traffic data, etc. By using the method of this application, it is possible to dynamically allocate and adjust computing resources, avoid resource waste, and improve hardware equipment utilization; speed up the processing of big data computing tasks, improve business responsiveness; ensure that key business data analysis tasks are processed first, guarantee service quality, and improve customer satisfaction. At the same time, it also helps to reduce data center operation and maintenance costs, including power consumption, equipment life maintenance, etc., which is conducive to more accurate cost budgeting and control for companies.

[0184] It should be understood that, although the various steps in the flowcharts in the above-described embodiments are sequentially displayed according to the indications of the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and they can be executed in other orders. Moreover, at least a portion of the steps in the figure may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily to be carried out sequentially, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0185] Figure 7 A schematic diagram of a task scheduling and balancing device provided in an embodiment of the present application, such as Figure 7 As shown, the present application provides a task scheduling balancing device, the device comprising:

[0186] A task acquisition module 11, which is configured to acquire tasks to be scheduled;

[0187] A task allocation module 12, which is configured to perform time-sensitive task combination on the tasks to be scheduled according to the time information of the tasks to be scheduled, and allocate the tasks to be scheduled to corresponding time windows;

[0188] A deduplication processing module 13, which is configured to perform task deduplication processing on the tasks to be scheduled that have been assigned time windows to obtain optimized tasks to be scheduled;

[0189] The strategy determination module 14 is configured to determine the corresponding task scheduling balancing strategy according to the optimized tasks to be scheduled.

[0190] Regarding the limitation of the task scheduling and balancing device, reference may be made to the limitation of the task scheduling and balancing method in the above-mentioned embodiments of the present application, which will not be elaborated in this embodiment.

[0191] Figure 8 Another schematic diagram of the task scheduling and balancing device provided in an embodiment of the present application is as follows Figure 8 As shown, in some embodiments, the present application provides a task scheduling and balancing device, including a memory 22 and a processor 21, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the task scheduling and balancing method in the above-mentioned embodiments of the present application.

[0192] The memory is connected to the processor, the memory may be a flash memory or a read-only memory or other memory, and the processor may be a central processing unit or a single-chip microcomputer.

[0193] In some embodiments, the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the task scheduling balancing method in the above-mentioned embodiments of the present application is implemented.

[0194] The computer-readable storage medium includes volatile or non-volatile, removable or non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, computer program modules or other data). Computer-readable storage media include, but are not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other memory technology, CD-ROM (Compact Disk Read-Only Memory), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer.

[0195] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present application, but the present application is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and substance of the present application, and these modifications and improvements are also considered to be within the scope of protection of the present application.

Claims

1. A task scheduling balancing method, characterized in that: The method comprises: S1. Obtain the tasks to be scheduled; S2. According to the time information of the tasks to be scheduled, the tasks to be scheduled are combined into time-sensitive tasks, and the tasks to be scheduled are allocated to corresponding time windows; S3, performing task deduplication processing on the tasks to be scheduled that have been assigned time windows to obtain optimized tasks to be scheduled; S4. Determine the corresponding task scheduling balancing strategy according to the optimized tasks to be scheduled.

2. The task scheduling balancing method according to claim 1, characterized in that: In S2, the time information of the task to be scheduled includes at least one of the task submission time, expected running time, deadline, dependency between tasks, and historical execution time data of the task.

3. The task scheduling balancing method according to claim 1, characterized in that: S3, including: S32, performing internal logic analysis on the tasks to be scheduled, and determining tasks with the same internal logic as candidate tasks that can be deduplicated; S32, performing task logic similarity comparison on candidate tasks that can be deduplicated, performing task deduplication processing based on the comparison results, and obtaining tasks to be scheduled after the first optimization.

4. The task scheduling balancing method according to claim 3, characterized in that: In S32, the task logic similarity comparison is performed on the candidate tasks that can be deduplicated, including: Determine whether the task logic of the candidate task is consistent, wherein the task logic includes at least one of an operation type, an operation object, an operation frequency, an operation statement, and a shooting destination target; If the task logic is inconsistent, the candidate tasks are determined to be inconsistent; If the task logic is consistent, determine whether the task results of the candidate tasks are consistent; If the task results are consistent, the candidate tasks are determined to be consistent; If the task results are inconsistent, determine whether there is an inclusion relationship between the task results of the candidate tasks; If there is an inclusion relationship between the task results, the candidate tasks are determined to be consistent; If there is no inclusion relationship between the task results, it is determined that the candidate tasks are inconsistent.

5. The task scheduling balancing method according to claim 3, characterized in that: S3, also includes: S33. Perform secondary deduplication processing on the tasks to be scheduled after the initial optimization through text parsing and hash algorithm to obtain the tasks to be scheduled after the secondary optimization.

6. The task scheduling balancing method according to claim 5, characterized in that: S33, including: The task description text of the task to be scheduled after the first optimization is parsed through natural language processing technology to extract semantic features and logical structure; The similarity of semantic features and logical structures is evaluated by a text similarity model to obtain a first matching result; Generate a hash signature of the task to be scheduled after the first optimization through a hash algorithm, and match the hash signatures of all tasks to obtain a second matching result; A secondary deduplication process is performed based on the first matching result and the second matching result to obtain the tasks to be scheduled after secondary optimization.

7. The task scheduling balancing method according to any one of claims 1 to 6, characterized in that: S4, including: According to the optimized tasks to be scheduled, the corresponding task scheduling balance strategy is determined by particle swarm optimization algorithm and ant colony algorithm.

8. A task scheduling balancing device, characterized in that: The device comprises: A task acquisition module, which is configured to acquire tasks to be scheduled; A task allocation module, which is configured to combine the tasks to be scheduled into time-sensitive tasks according to the time information of the tasks to be scheduled, and allocate the tasks to be scheduled to corresponding time windows; A deduplication processing module is configured to perform task deduplication processing on the tasks to be scheduled that have been assigned time windows to obtain optimized tasks to be scheduled; The strategy determination module is configured to determine the corresponding task scheduling balancing strategy according to the optimized tasks to be scheduled.

9. A task scheduling balancing device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to implement the task scheduling balancing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the task scheduling and balancing method according to any one of claims 1 to 7 is implemented.

Images