Pipeline scheduling method and apparatus, computer-readable storage medium

Abstract

This disclosure relates to a pipeline scheduling method and apparatus, and a computer-readable storage medium. The pipeline scheduling method includes: collecting actual task execution time data of tasks in a pipeline instance; determining the predicted execution time of each task in the pipeline instance based on the collected task execution time data; determining the scheduling priority of the pipeline instance based on the predicted execution time of each task and the weight of the pipeline instance; and performing pipeline task scheduling according to the scheduling priority of the pipeline instances. This disclosure can dynamically adjust the pipeline scheduling strategy by combining dynamic weight configuration with pipeline task execution time prediction.

Description

Technical Field

[0001] This disclosure relates to the field of IT and software development, and in particular to a pipeline scheduling method and apparatus, and a computer-readable storage medium. Background Technology

[0002] With the increasing popularity of agile development methodologies and the rise of DevOps (Development and Operations, a general term for development, operation, and maintenance), more and more enterprises and development teams are using CI (Continuous Integration) / CD (Continuous Delivery and / or Continuous Deployment) pipeline tools to optimize development process efficiency in their daily software development projects. Currently, the most widely used and popular pipeline tool is Jenkins, whose deployment scheme relies solely on Kubernetes (an open-source container cluster management system) and Docker (an open-source application container engine) for task scheduling and node load balancing.

[0003] As can be seen from the scheduling and load balancing functions of Kubernetes, when a CI / CD pipeline cluster is deployed in Kubernetes, job scheduling can only be allocated based on node resources, storage pressure, capacity, inherent weighting strategies, and load balancing strategies. This can easily lead to situations where some nodes have ample resources but a large backlog of jobs, while other nodes, although resource-constrained, have fewer jobs backlogging as resources can be released quickly. Consequently, this results in some nodes having their resources occupied for extended periods, while others have longer periods of inactivity, leading to unreasonable resource utilization and job scheduling allocation. Summary of the Invention

[0004] In view of at least one of the above technical problems, this disclosure provides a pipeline scheduling method and apparatus, and a computer-readable storage medium, which can dynamically adjust the pipeline scheduling strategy by combining dynamic weight configuration with pipeline task time prediction.

[0005] According to one aspect of this disclosure, a pipeline scheduling method is provided, comprising:

[0006] Collect actual task execution time data in pipeline instances;

[0007] Based on the collected task time data, determine the predicted time for each task in the pipeline instance;

[0008] The scheduling priority of the pipeline instance is determined based on the predicted time of each task in the pipeline instance and the weight of the pipeline instance.

[0009] Pipeline tasks are scheduled based on the scheduling priority of pipeline instances.

[0010] In some embodiments of this disclosure, the actual task execution time data in the collection pipeline instance includes:

[0011] By recording data through pipeline execution, task time data is collected from all pipeline instances. The task time data includes the number of times the task was executed correctly and the time it took, as well as the number of times the task was executed abnormally and the time it took.

[0012] In some embodiments of this disclosure, determining the predicted time for each task in the pipeline instance based on the collected task time data includes:

[0013] The prediction model is trained in advance using task time data;

[0014] Input the collected task time data into the trained prediction model, and determine the predicted time based on the collected task time data.

[0015] In some embodiments of this disclosure, determining the predicted time based on the collected task time data includes:

[0016] For each task in the pipeline instance, the predicted execution time is determined based on the number of times the task was executed correctly and the execution time, as well as the number of times the task was executed abnormally and the execution time.

[0017] In some embodiments of this disclosure, determining the predicted execution time of the task based on the number of correct executions and their duration, as well as the number of abnormal executions and their duration, includes:

[0018] Determine the average time for a task to be executed correctly based on the time taken for the task to be executed correctly and the number of times the task was executed correctly.

[0019] The average execution time of the task is determined based on the time taken by the task to be executed abnormally and the number of times the task was executed abnormally.

[0020] The normal execution rate and abnormal execution rate of a task are determined based on the number of times the task is executed correctly and the number of times the task is executed abnormally.

[0021] The predicted execution time of a task is determined based on the average time taken for the task to execute correctly, the success rate of task execution, the average time taken for the task to execute abnormally, and the failure rate of task execution.

[0022] In some embodiments of this disclosure, the pipeline scheduling method further includes:

[0023] The weights of each pipeline instance are dynamically adjusted based on the current state of each pipeline instance and the similarity of the time taken for multiple tasks in each pipeline instance.

[0024] In some embodiments of this disclosure, determining the scheduling priority of a pipeline instance based on the predicted time of each task and the weight of the pipeline instance includes:

[0025] Based on the predicted time of each task in the pipeline instance, determine the duration for which resources can be released for that pipeline instance.

[0026] The scheduling priority of a pipeline instance is determined based on the duration during which resources can be released and the weight of the pipeline instance.

[0027] In some embodiments of this disclosure, determining the duration for which resources can be released for a pipeline instance based on the predicted time consumption of each task in the pipeline instance includes:

[0028] A distributed linked list queue data structure is used to perform a complete queue mapping for tasks in the pipeline tool cluster;

[0029] For each task in the pipeline instance, determine the duration for which resources can be released based on the predicted time and the current time of the task.

[0030] The duration of releasable resources for a pipeline instance is determined based on the duration of releasable resources for each task in the pipeline instance.

[0031] In some embodiments of this disclosure, the pipeline task scheduling based on the scheduling priority of pipeline instances includes:

[0032] Based on the scheduling priority of multiple pipeline instances, newly produced pipeline tasks are assigned to the pipeline instance with the highest scheduling priority.

[0033] According to another aspect of this disclosure, a production line scheduling device is provided, comprising:

[0034] The time consumption acquisition module is configured to collect the actual task consumption data of tasks running in the pipeline instance;

[0035] The time consumption prediction module is configured to determine the predicted time consumption of each task in the pipeline instance based on the collected task time consumption data.

[0036] The scheduling module is configured to determine the scheduling priority of a pipeline instance based on the predicted time of each task and the weight of the pipeline instance; and to schedule pipeline tasks according to the scheduling priority of the pipeline instances.

[0037] In some embodiments of this disclosure, the time consumption acquisition module is configured to collect task time consumption data in all pipeline instances by recording pipeline execution data, wherein the task time consumption data includes the number of times the task was executed correctly and the time consumed, as well as the number of times the task was executed abnormally and the time consumed.

[0038] In some embodiments of this disclosure, the pipeline scheduling device further includes:

[0039] The artificial intelligence module is configured to train the prediction model in advance using task time data;

[0040] The time consumption prediction module is configured to input the collected task time consumption data into the trained prediction model and determine the predicted time consumption based on the collected task time consumption data.

[0041] In some embodiments of this disclosure, the time prediction module is configured to determine the predicted time for each task in the pipeline instance based on the number of times the task was executed correctly and the time it took, as well as the number of times the task was executed abnormally and the time it took.

[0042] In some embodiments of this disclosure, the time prediction module is configured to: determine the average time for correct task execution based on the time taken for correct task execution and the number of times the task is executed correctly; determine the average time for abnormal task execution based on the time taken for abnormal task execution and the number of times the task is executed abnormally; determine the normal rate and abnormal rate of task execution based on the number of times the task is executed correctly and the number of times the task is executed abnormally; and determine the predicted time for the task based on the average time taken for correct task execution, the normal rate of task execution, the average time taken for abnormal task execution, and the abnormal rate of task execution.

[0043] In some embodiments of this disclosure, the pipeline scheduling device further includes:

[0044] The weight configuration module is configured to dynamically adjust the weights of each pipeline instance based on the current state of each pipeline instance and the similarity of the time consumption of multiple tasks in each pipeline instance.

[0045] In some embodiments of this disclosure, the pipeline scheduling device further includes:

[0046] The shadow queue module is configured to determine the duration for which resources can be released for a pipeline instance based on the predicted time of each task in the pipeline instance.

[0047] The scheduling module is configured to determine the scheduling priority of a pipeline instance based on the duration during which resources can be released and the weight of the pipeline instance.

[0048] In some embodiments of this disclosure, the shadow queue module is configured to use a distributed linked list queue data structure to perform a complete queue mapping for tasks in the pipeline tool cluster; for each task in the pipeline instance, the duration of releasable resources for the task is determined based on the predicted time consumption and the current time consumption of the task; and the duration of releasable resources for the pipeline instance is determined based on the duration of releasable resources for each task in the pipeline instance.

[0049] In some embodiments of this disclosure, the scheduling module is configured to distribute newly produced pipeline tasks to the pipeline instance with the highest scheduling priority based on the scheduling priorities of multiple pipeline instances.

[0050] According to another aspect of this disclosure, a production line scheduling device is provided, comprising:

[0051] The memory is configured to store instructions;

[0052] The processor is configured to execute the instructions, causing the pipeline scheduling device to perform operations implementing the pipeline scheduling method as described in any of the above embodiments.

[0053] According to another aspect of this disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions that, when executed by a processor, implement the pipeline scheduling method as described in any of the above embodiments.

[0054] This disclosure allows for dynamic adjustment of pipeline scheduling strategies by combining dynamic weight configuration with pipeline task time prediction. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a schematic diagram of some embodiments of the pipeline scheduling method disclosed herein.

[0057] Figure 2 This is a schematic diagram of some embodiments of the assembly line scheduling device disclosed herein.

[0058] Figure 3 This is a schematic diagram illustrating the implementation principle of shadow queues and scheduling in some embodiments of this disclosure.

[0059] Figure 4 The diagram illustrates some other embodiments of the pipeline scheduling method disclosed herein.

[0060] Figure 5 The diagram shows the structure of some other embodiments of the assembly line scheduling device disclosed herein. Detailed Implementation

[0061] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0062] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0063] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0064] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0065] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0066] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0067] The inventors discovered through research that the cluster and scheduling strategies implemented based on Kubernetes are relatively simple, require manual intervention to define balancing rules, and cannot adapt scheduling strategies according to important factors such as task length, average task time, and average time of cluster nodes.

[0068] During peak business periods, some instances in the cluster experience task backlog, resulting in longer task waiting times. Conversely, some instances complete their tasks relatively quickly, leading to less task backlog and lower utilization. Ultimately, the long backlog of pipeline tasks will result in adverse effects such as prolonged waiting times or timeouts.

[0069] Related technologies allocate tasks randomly or according to fixed parameters to a pipeline cluster via message queues, and then select the execution node through the Kubernetes cluster's load balancing and scheduling strategies. However, this approach cannot solve the problem of dynamic imbalance in the resource load of the pipeline cluster.

[0070] In view of at least one of the above technical problems, this disclosure provides a pipeline scheduling method and apparatus, and a computer-readable storage medium. The present disclosure will be described below through specific embodiments.

[0071] Figure 1 This is a schematic diagram of some embodiments of the pipeline scheduling method of this disclosure. Preferably, this embodiment can be executed by the apparatus of the pipeline scheduling method of this disclosure. The method includes at least one step from steps 11-14, wherein:

[0072] Step 11: Collect the actual task execution time data of the tasks in the pipeline instance.

[0073] In some embodiments of this disclosure, step 11 may include: collecting task time data from all pipeline instances by executing pipeline execution record data, wherein the task time data includes the number of times and time of correct task execution, and the number of times and time of abnormal task execution, etc.

[0074] Step 12: Based on the collected task time data, determine the predicted time for each task in the pipeline instance.

[0075] In some embodiments of this disclosure, step 12 may include at least one of steps 121-123, wherein:

[0076] Step 121: Train the prediction model in advance using task time data.

[0077] Step 122: Input the collected task time data into the trained prediction model and determine the predicted time based on the collected task time data.

[0078] In some embodiments of this disclosure, step 122 may include: for each task in the pipeline instance, determining the predicted execution time of the task based on the number of times and the time taken for correct execution and the number of times and the time taken for abnormal execution of the task.

[0079] In some embodiments of this disclosure, step 122 may include: determining the average time taken for the task to execute correctly based on the time taken for the task to execute correctly and the number of times the task is executed correctly; determining the average time taken for the task to execute abnormally based on the time taken for the task to execute abnormally and the number of times the task is executed abnormally; determining the normal rate and the abnormal rate of the task execution based on the number of times the task is executed correctly and the number of times the task is executed abnormally; and determining the predicted time taken for the task based on the average time taken for the task to execute correctly, the normal rate of the task execution, the average time taken for the task to execute abnormally, and the abnormal rate of the task execution.

[0080] In some embodiments of this disclosure, step 122 may include: summing the time taken for correctly executed pipeline tasks based on the collected data and dividing by the number of times they were correctly executed to obtain the average time taken for correct execution; summing the time taken for abnormally executed pipeline tasks and dividing by the number of times they were abnormally executed to obtain the average time taken for abnormal execution; and finally, multiplying the average time taken for normal execution by the normality rate and adding the average time taken for abnormal execution by the abnormality rate to obtain the predicted time.

[0081] In some embodiments of this disclosure, step 122 may include: determining the predicted time T according to formula (1), where Ts is the time taken for the s-th task to be executed correctly, s is a natural number greater than or equal to 1 and less than or equal to n, and n is the number of times the task is executed correctly; Tf is the time taken for the f-th task to be executed abnormally, f is a natural number greater than or equal to 1 and less than or equal to m, and m is the number of times the task is executed abnormally.

[0082]

[0083] Step 13: Determine the scheduling priority of the pipeline instance based on the predicted time of each task in the pipeline instance and the weight of the pipeline instance.

[0084] In some embodiments of this disclosure, the pipeline scheduling method may further include: dynamically adjusting the weights of each pipeline instance based on the current state of each pipeline instance and the similarity of the time consumption of multiple tasks of each pipeline instance.

[0085] In some embodiments of this disclosure, step 13 may include at least one of steps 121-123, wherein:

[0086] Step 131: Determine the duration for which resources can be released for the pipeline instance based on the predicted time of each task in the pipeline instance.

[0087] In some embodiments of this disclosure, step 131 may include: using a distributed linked list queue data structure to perform a complete queue mapping on the tasks of the pipeline tool cluster; for each task in the pipeline instance, determining the duration of the releasable resources for the task based on the predicted time consumption and the current time consumption of the task; and determining the duration of the releasable resources for the pipeline instance based on the duration of the releasable resources for each task in the pipeline instance.

[0088] In some embodiments of this disclosure, step 131 may include: implementing a complete queue mapping for tasks in the pipeline tool cluster using a distributed linked list queue data structure of Redis (Remote DictionaryServer). For example, if the pipeline tool cluster connected to the system has 3 instances, then 3 linked list queues are generated accordingly. The elements in the queues are synchronized in real time with the status, time consumption, and results of tasks being executed and waiting to be executed in the pipeline cluster. Combined with the predicted time consumption statistics from the prediction module, the duration for which the instance can release resources can be calculated.

[0089] In some embodiments of this disclosure, step 131 may include: determining the duration S for which the instance can release resources according to formula (2), where T j For the predicted time T of the j-th task in a given pipeline instance j The value is determined according to formula (1), where j is a natural number greater than or equal to 1 and less than or equal to k, k is the number of pipeline instances included in the pipeline cluster, and t is the current time of the j-th task.

[0090]

[0091] Step 132: Determine the scheduling priority of the pipeline instance based on the duration during which the pipeline instance can release resources and the weight of the pipeline instance.

[0092] In some embodiments of this disclosure, step 131 may include: determining the scheduling priority Pi of the i-th pipeline instance according to formula (3), where Si is the waiting time required for the i-th instance to release resources, and Wi is the configuration weight of the i-th instance for the user, ranging from 1 to n. Therefore, the lower the calculated value of Pi, the higher the priority.

[0093] Pi = Si * Wi (3)

[0094] Step 14: Schedule pipeline tasks according to the scheduling priority of pipeline instances.

[0095] In some embodiments of this disclosure, step 14 may include: assigning newly produced pipeline tasks to the pipeline instance with the highest scheduling priority based on the scheduling priority of multiple pipeline instances.

[0096] The embodiments disclosed above can accurately predict the resource consumption time of pipeline tool instances and the waiting time for new tasks to go from the ready state to the execution state by combining the actual task execution time of the pipeline. Therefore, based on a strategy of minimizing waiting time and prioritizing resources, task waiting time is optimized, effectively avoiding long-term task backlogs.

[0097] The embodiments disclosed above can dynamically adjust the pipeline scheduling strategy by combining dynamic weight configuration with pipeline task time prediction, without manual intervention and taking effect in real time.

[0098] Figure 2 These are schematic diagrams illustrating some embodiments of the assembly line scheduling device disclosed herein. For example... Figure 2 As shown, the pipeline scheduling device 2 disclosed herein may include a time consumption acquisition module 21, a time consumption prediction module 22, and a scheduling module 23, wherein:

[0099] The time consumption acquisition module 21 is configured to collect the actual task time consumption data of the tasks running in the pipeline instance.

[0100] In some embodiments of this disclosure, the time consumption acquisition module 21 can be configured to collect task time consumption data in all pipeline instances by recording pipeline execution data, wherein the task time consumption data includes the number of times the task was executed correctly and the time consumed, as well as the number of times the task was executed abnormally and the time consumed.

[0101] The time consumption prediction module 22 is configured to determine the predicted time consumption of each task in the pipeline instance based on the collected task time consumption data.

[0102] The scheduling module 23 is configured to determine the scheduling priority of the pipeline instance based on the predicted time of each task in the pipeline instance and the weight of the pipeline instance; and to perform pipeline task scheduling based on the scheduling priority of the pipeline instance.

[0103] In some embodiments of this disclosure, such as Figure 2 As shown, the assembly line scheduling device 2 may further include an AI (Artificial Intelligence) module 24, wherein:

[0104] Artificial intelligence module 24 is configured to pre-train the prediction model using task time data.

[0105] In some embodiments of this disclosure, the artificial intelligence module 24 can be configured to provide an AI model and data training. It trains a deep learning model on data collected after the actual time consumption of the pipeline and outputs the training results.

[0106] In some embodiments of this disclosure, the time consumption prediction module 22 can be configured to input the collected task time consumption data into the trained prediction model and determine the predicted time consumption based on the collected task time consumption data.

[0107] In some embodiments of this disclosure, the time prediction module 22 can be configured to determine the predicted time for each task in the pipeline instance based on the number of times the task was executed correctly and the time it took, as well as the number of times the task was executed abnormally and the time it took.

[0108] In some embodiments of this disclosure, the time prediction module 22 can be configured to determine the average time for correct task execution based on the time taken for correct task execution and the number of times the task is executed correctly; determine the average time for abnormal task execution based on the time taken for abnormal task execution and the number of times the task is executed abnormally; determine the normal rate and abnormal rate of task execution based on the number of times the task is executed correctly and the number of times the task is executed abnormally; and determine the predicted time for the task based on the average time taken for correct task execution, the normal rate of task execution, the average time taken for abnormal task execution, and the abnormal rate of task execution.

[0109] In some embodiments of this disclosure, the time consumption prediction module 22 can be configured to send the predicted time consumption to the AI ​​module for training and store the training results; when the pipeline task starts to execute again, the time consumption prediction module provides the trained prediction value to the scheduling module as a scheduling basis.

[0110] In some embodiments of this disclosure, such as Figure 2 As shown, the pipeline scheduling device 2 may further include a weight configuration module 25, wherein:

[0111] The weight configuration module 25 is configured to dynamically adjust the weight of each pipeline instance based on the current state of each pipeline instance and the similarity of the time consumption of multiple tasks of each pipeline instance.

[0112] In some embodiments of this disclosure, the weight configuration module 25 can be configured to provide configuration of pipeline instance weights. When the pipeline enters the scheduling ready state, assuming that all instances are currently idle, or that the tasks in each instance require similar execution times, the higher the pipeline instance weight, the higher the priority of the pipeline task scheduling assigned to that instance for execution.

[0113] In some embodiments of this disclosure, such as Figure 2 As shown, the pipeline scheduling device 2 may further include a shadow queue module 26, wherein:

[0114] The shadow queue module 26 is configured to determine the duration for which resources can be released for a pipeline instance based on the predicted time of each task in the pipeline instance.

[0115] In some embodiments of this disclosure, the scheduling module 23 may be configured to determine the scheduling priority of a pipeline instance based on the duration during which the pipeline instance can release resources and the weight of the pipeline instance.

[0116] In some embodiments of this disclosure, the shadow queue module 26 can be configured to use a distributed linked list queue data structure to perform a complete queue mapping for tasks in the pipeline tool cluster; for each task in the pipeline instance, the duration of the releasable resources for the task is determined based on the predicted time consumption and the current time consumption of the task; and the duration of the releasable resources for the pipeline instance is determined based on the duration of the releasable resources for each task in the pipeline instance.

[0117] In some embodiments of this disclosure, the shadow queue module 26 can be configured as a distributed linked list queue data structure implemented through Redis to achieve complete queue mapping for tasks in the pipeline tool cluster. For example, if the pipeline tool cluster connected to the system has 3 instances, then 3 linked list queues will be generated accordingly. The elements in the queues are synchronized in real time with the status, time consumption, and results of tasks being executed and waiting to be executed in the pipeline cluster. Combined with the predicted time consumption statistics from the prediction module, the duration for which the instance can release resources can be calculated.

[0118] In some embodiments of this disclosure, the scheduling module 23 can be configured to distribute newly produced pipeline tasks to the pipeline instance with the highest scheduling priority based on the scheduling priorities of multiple pipeline instances.

[0119] In some embodiments of this disclosure, the scheduling module 23 can be configured to perform scheduling calculations on newly produced pipeline tasks based on the sum of the predicted task time in each queue of the shadow queue and the instance weight configuration data, and to match the optimal pipeline instance for task distribution.

[0120] Figure 3 This is a schematic diagram illustrating the implementation principle of shadow queues and scheduling in some embodiments of this disclosure. For example... Figure 3 As shown, the pipeline cluster consists of two pipeline instances: Pipeline Instance 1 and Pipeline Instance 2. Pipeline Instance 1 has three nodes: Node 1-1, Node 1-2, and Node 1-3. Tasks 1 and 2 are executed on Node 1-1, tasks 3 and 4 are executed on Node 1-2, and task 5 is executed on Node 1-3. Pipeline Instance 2 has three nodes: Node 2-1, Node 2-2, and Node 2-3. Tasks 6 and 7 are executed on Node 2-1, tasks 8 and 9 are executed on Node 2-2, and task 10 is executed on Node 2-3.

[0121] The shadow queue module 26 can be configured to implement a complete queue mapping for tasks in the pipeline cluster, forming a cluster shadow queue. Specifically, a task shadow queue 1 is formed for pipeline instance 1, which includes tasks 1 to 5. A task shadow queue 2 is formed for pipeline instance 2, which includes tasks 6 to 10.

[0122] The scheduling module 23 can be configured to perform task scheduling prediction and scheduling ready queue.

[0123] In some embodiments of this disclosure, the shadow queue and scheduling method may include: scheduling module 23 scheduling pipeline task data to shadow queue module 26; shadow queue module 26 notifying scheduling module 23 of tasks that have been in a waiting state for a long time; shadow queue module 26 updating and obtaining pipeline cluster, task and other status data in real time through event subscription or API polling; and shadow queue module 26 scheduling tasks or rescheduling timed-out tasks through API.

[0124] The technical solutions disclosed in the above embodiments of this disclosure are adaptable to various mainstream CI / CD pipeline tools (such as Jenkins). The implementation of the technical solutions in the above embodiments of this disclosure is highly decoupled from the business layer and pipeline tools. Therefore, the above embodiments of this disclosure can be widely applied to various types of DevOps implementation schemes, cloud-native development, automated production processes, etc., thereby improving the problem of a single scheduling method after pipeline tool deployment. Furthermore, the above embodiments of this disclosure can configure system scheduling principles according to business needs to achieve scheduling combinations for different resource environments.

[0125] The embodiments disclosed above can achieve time consumption prediction and weighted configuration: both predicting the time consumption of task queues in the cluster instance and selecting the optimal scheduling object according to weight priority, thereby realizing dynamic resource adaptation, weight priority, and adaptive task waiting. The embodiments disclosed above can optimize the user experience and cluster resource allocation.

[0126] The embodiments disclosed above can achieve time consumption prediction: the time consumption of task queues in the cluster instance is predicted, but the weight data configuration of the cluster instance is uniform, realizing the principle of prioritizing short-time tasks. When the resources of the cluster instances are highly consistent, the task waiting time can be optimized to a certain extent, and the cluster resources can be utilized evenly.

[0127] The embodiments described above enable weighted configuration: they allow for differentiated weighting based on cluster resource differences, without relying on time prediction. These embodiments can highly match user configurations for task scheduling. Furthermore, they allow for dynamic hot updates, resolving the issue of fixed and monotonous cluster configurations in traditional systems.

[0128] Figure 4 The diagram illustrates some other embodiments of the pipeline scheduling method disclosed herein. Preferably, this embodiment can be executed by the apparatus of the pipeline scheduling method disclosed herein. Figure 4 The embodiments also provide schematic diagrams of other embodiments of the assembly line scheduling device. For example... Figure 4 As shown, the pipeline scheduling method apparatus may include a time consumption acquisition module 21, a time consumption prediction module 22, a scheduling module 23, an AI module 24, a weight configuration module 25, and a shadow queue module 26, wherein:

[0129] The time consumption acquisition module 21 is configured to collect execution data of all pipeline tasks in the system.

[0130] The time prediction module 22 is configured to predict and calculate the average execution time of the pipeline based on the collected data.

[0131] AI module 24 is configured to provide AI models and data training.

[0132] The weight configuration module 25 is configured to provide users with configuration of pipeline instance weights.

[0133] The shadow queue module 26 is configured to implement a distributed linked list queue data structure through Redis to map the task queues of the pipeline tool cluster, thereby achieving dedifferentiation of pipeline tools and decoupling of task scheduling management from pipeline tools.

[0134] In some embodiments of this disclosure, such as Figure 4 As shown, the pipeline cluster includes two master nodes (Master1 and Master2). Master1 has two slave nodes (Slave1 and Slave2), and Master2 has two slave nodes (Slave3 and Slave4).

[0135] The scheduling module 23 is configured to match the optimal instance for task distribution based on the task time and weight of each pipeline instance.

[0136] Figure 4 The pipeline scheduling method of the embodiment includes at least one of steps 1-6, wherein:

[0137] Step 1: The code repository triggers the continuous integration pipeline via Webhook and submits code events, including the scheduling of newly generated tasks.

[0138] Step 2, the time consumption prediction module 22, outputs the scheduling prediction vector through the trained model based on the code event parameters.

[0139] Step 3: Scheduling module 23 performs scheduling based on the scheduling prediction vector.

[0140] In some embodiments of this disclosure, steps 2 and 3 may include those described in the embodiments of this disclosure above (e.g., Figure 1 The pipeline scheduling method in the embodiment.

[0141] Step 4: The shadow queue module 26 selects the pipeline cluster according to the scheduling vector and inputs the selection result and scheduling vector into the time consumption acquisition module 21.

[0142] Step 5, the time-consuming data collection module 21, forms a training set from the collected data and updates the deep learning model periodically.

[0143] Step 6, AI module 24, provides the updated AI model so that time-consuming prediction module 22 can output the scheduling prediction vector.

[0144] The embodiments disclosed above can decouple the pipeline scheduling device from the pipeline tools through the shadow queue method, while achieving consistent data projection, providing a data foundation and operational conditions for pipeline time prediction and automatic scheduling adaptation.

[0145] The above embodiments of this disclosure design an algorithm for scheduling calculation based on time consumption prediction, time consumption statistics, and weight configuration.

[0146] The embodiments disclosed above improve pipeline task waiting efficiency and cluster resource utilization.

[0147] The present disclosure provides an apparatus and method for optimizing pipeline scheduling.

[0148] Figure 5 These are schematic diagrams illustrating the structure of other embodiments of the assembly line scheduling device disclosed herein. For example... Figure 5 As shown, the pipeline scheduling device disclosed herein may include a memory 51 and a processor 52.

[0149] Memory 51 is used to store instructions, and processor 52 is coupled to memory 51. Processor 52 is configured to execute instructions stored in memory as described in any of the above embodiments (e.g., Figure 1 or Figure 4 The pipeline scheduling method described in the embodiment)

[0150] like Figure 5 As shown, the session management function network element also includes a communication interface 53 for exchanging information with other devices. Additionally, the session management function network element includes a bus 54, through which the processor 52, communication interface 53, and memory 51 communicate with each other.

[0151] The memory 51 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk drive. The memory 51 may also be a memory array. The memory 51 may also be divided into blocks, and the blocks may be combined into virtual volumes according to certain rules.

[0152] Furthermore, processor 52 may be a central processing unit (CPU), or a fourth integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present disclosure.

[0153] The embodiments described above can be applied to continuous integration and continuous deployment in DevOps processes.

[0154] The embodiments disclosed above can continuously collect and analyze the time consumed by the pipeline business, dynamically predict the pipeline time and the release time of cluster resources, and realize the full utilization of cluster resources.

[0155] The method proposed in the above embodiments of this disclosure can simultaneously support integrated scheduling schemes of related technologies, and can also hot modify scheduling weights according to user configuration, thereby realizing differentiated cluster resources and rich scheduling rules.

[0156] According to another aspect of this disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions that, when executed by a processor, implement any of the embodiments described above (e.g., Figure 1 or Figure 4 The pipeline scheduling method described in the embodiment)

[0157] In some embodiments of this disclosure, the computer-readable storage medium may be a non-transitory computer-readable storage medium.

[0158] The embodiments of this disclosure propose an apparatus and method for automatically optimizing pipeline scheduling based on existing pipeline tool clusters. These embodiments collect data on the actual execution time of pipeline tasks, including various time-consuming data such as successful execution, execution errors, and manual termination. This data is then fed back to an AI module, which uses a deep learning model to calculate predicted execution times. This prediction guides the pipeline to anticipate the resource consumption time of pipeline tool instances and the waiting time required for new task execution before the next execution. Therefore, these embodiments can optimize task scheduling based on strategies such as minimizing waiting time, prioritizing resources, and weighting, effectively avoiding long waiting times for task execution and the accumulation of pipeline tasks in a particular instance.

[0159] The apparatus and method of the above embodiments of this disclosure can dynamically adjust the pipeline scheduling strategy by combining dynamic weight configuration with pipeline task time prediction, without manual intervention and taking effect in real time.

[0160] Compared with related technologies, the main advantages of the above embodiments of this disclosure are:

[0161] 1. The embodiments of this disclosure can accurately predict the resource consumption time of pipeline tool instances and the waiting time required for new task execution by combining the actual task execution time of the pipeline. Therefore, based on the strategy of prioritizing resources for minimal waiting, task waiting time is optimized, effectively avoiding long-term task backlog.

[0162] 2. The above embodiments of this disclosure can make full use of cluster resources in high-concurrency pipeline scenarios, reduce pipeline waiting time, and improve user experience.

[0163] 3. The above embodiments of this disclosure can dynamically adjust the pipeline scheduling strategy by combining dynamic weight configuration with pipeline task time prediction, without manual intervention and taking effect in real time.

[0164] 4. The above embodiments of this disclosure can map pipeline clusters through shadow queues, thereby decoupling functions such as scheduling analysis, scheduling settings, and scheduling changes from pipeline tools and clusters.

[0165] 5. The above embodiments of this disclosure can increase the continuous collection, statistics and analysis of pipeline runtime, realize dynamic prediction of pipeline execution events, and accurately judge the consumption of pipeline cluster resources.

[0166] 7. The above embodiments of this disclosure can support the scheduling of pipeline tasks during continuous integration and optimize the full utilization of pipeline cluster resources.

[0167] 8. The above embodiments of this disclosure can ensure the stability of pipeline operations during high concurrency and improve the user experience in high concurrency scenarios.

[0168] 9. The above embodiments disclosed herein can be used as a general solution to adapt to various DevOps platforms or pipeline clusters, and can be reused in the future or formed into a general cloud-native product in the industry.

[0169] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, apparatus, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable non-transitory storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0170] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a fourth computer, an embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0171] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0172] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0173] The pipeline scheduling device, time acquisition module, time prediction module, scheduling module, artificial intelligence module, weight configuration module, and shadow queue module described above can be implemented as a general-purpose processor, programmable logic controller (PLC), digital signal processor (DSP), fourth integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, or any suitable combination thereof for performing the functions described in this application.

[0174] This concludes the detailed description of the present disclosure. To avoid obscuring the concept of the disclosure, some details known in the art have not been described. Those skilled in the art will fully understand how to implement the technical solutions disclosed herein based on the above description.

[0175] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware, or by a program instructing the relevant hardware to implement them. The program can be stored in a non-transitory computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0176] The description in this disclosure is provided for illustrative and descriptive purposes only and is not intended to be exhaustive or to limit the disclosure to its forms. Many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described in order to better illustrate the principles and practical application of this disclosure and to enable those skilled in the art to understand this disclosure and to design various embodiments with various modifications suitable for a particular purpose.

Claims

1. A pipeline scheduling method, comprising: Collect actual task execution time data in pipeline instances; Based on the collected task time data, the predicted time of each task in the pipeline instance is determined. The process of determining the predicted time of each task in the pipeline instance based on the collected task time data includes: for each task in the pipeline instance, the predicted time of the task is determined based on the number of times and the time of correct execution and the number of times and the time of abnormal execution. The scheduling priority of a pipeline instance is determined based on the predicted time of each task and the weight of the pipeline instance. This determination includes: determining the duration for which resources can be released by the pipeline instance based on the predicted time of each task; and determining the scheduling priority of the pipeline instance based on the duration for which resources can be released and the weight of the pipeline instance. Pipeline tasks are scheduled based on the scheduling priority of pipeline instances.

2. The pipeline scheduling method according to claim 1, wherein, The actual task execution time data in the data acquisition pipeline instance includes: By recording data through pipeline execution, task time data is collected from all pipeline instances. The task time data includes the number of times the task was executed correctly and the time it took, as well as the number of times the task was executed abnormally and the time it took.

3. The pipeline scheduling method according to claim 2, wherein, The step of determining the predicted time for each task in the pipeline instance based on the collected task time data includes: The prediction model is trained in advance using task time data; Input the collected task time data into the trained prediction model, and determine the predicted time based on the collected task time data.

4. The pipeline scheduling method according to claim 1, wherein, The process of determining the predicted execution time of a task based on the number of correct executions and their duration, as well as the number of abnormal executions and their duration, includes: Determine the average time for a task to be executed correctly based on the time taken for the task to be executed correctly and the number of times the task was executed correctly. The average execution time of the task is determined based on the time taken by the task to be executed abnormally and the number of times the task was executed abnormally. The normal execution rate and abnormal execution rate of a task are determined based on the number of times the task is executed correctly and the number of times the task is executed abnormally. The predicted execution time of a task is determined based on the average time taken for the task to execute correctly, the success rate of task execution, the average time taken for the task to execute abnormally, and the failure rate of task execution.

5. The pipeline scheduling method according to any one of claims 1-4, further comprising: The weights of each pipeline instance are dynamically adjusted based on the current state of each pipeline instance and the similarity of the time taken for multiple tasks in each pipeline instance.

6. The pipeline scheduling method according to any one of claims 1-4, wherein, The step of determining the duration for which resources can be released for a pipeline instance based on the predicted time consumption of each task in the pipeline instance includes: A distributed linked list queue data structure is used to perform a complete queue mapping for tasks in the pipeline tool cluster; For each task in the pipeline instance, determine the duration for which resources can be released based on the predicted time and the current time of the task. The duration of releasable resources for a pipeline instance is determined based on the duration of releasable resources for each task in the pipeline instance.

7. The pipeline scheduling method according to any one of claims 1-4, wherein, The process of scheduling pipeline tasks based on the scheduling priority of pipeline instances includes: Based on the scheduling priority of multiple pipeline instances, newly produced pipeline tasks are assigned to the pipeline instance with the highest scheduling priority.

8. A production line scheduling device, comprising: The time consumption acquisition module is configured to collect the actual task consumption data of tasks running in the pipeline instance; The time consumption prediction module is configured to determine the predicted time consumption of each task in the pipeline instance based on the collected task time consumption data. Specifically, the time consumption prediction module is configured to determine the predicted time consumption of each task in the pipeline instance based on the number of times the task was executed correctly and the time consumed, as well as the number of times the task was executed abnormally and the time consumed. The scheduling module is configured to determine the scheduling priority of a pipeline instance based on the predicted time of each task and the weight of the pipeline instance; and to schedule pipeline tasks according to the scheduling priority of the pipeline instances. The shadow queue module is configured to determine the duration for which resources can be released for a pipeline instance based on the predicted time of each task in the pipeline instance. The scheduling module is configured to determine the scheduling priority of a pipeline instance based on the duration during which resources can be released and the weight of the pipeline instance.

9. The assembly line scheduling device according to claim 8, wherein: The time consumption acquisition module is configured to collect task time consumption data from all pipeline instances by recording pipeline execution data. The task time consumption data includes the number of times the task was executed correctly and the time consumed, as well as the number of times the task was executed abnormally and the time consumed.

10. The assembly line scheduling device according to claim 9, wherein, Also includes: The artificial intelligence module is configured to train the prediction model in advance using task time data; The time consumption prediction module is configured to input the collected task time consumption data into the trained prediction model and determine the predicted time consumption based on the collected task time consumption data.

11. The assembly line scheduling device according to claim 8, wherein: The time-consuming prediction module is configured to determine the average time for a task to execute correctly based on the time taken for the task to execute correctly and the number of times the task was executed correctly; to determine the average time for a task to execute abnormally based on the time taken for the task to execute abnormally and the number of times the task was executed abnormally; to determine the normal rate and the abnormal rate of the task execution based on the number of times the task was executed correctly and the number of times the task was executed abnormally; and to determine the predicted time for the task based on the average time taken for the task to execute correctly, the normal rate of the task execution, the average time taken for the task to execute abnormally, and the abnormal rate of the task execution.

12. The assembly line scheduling device according to any one of claims 8-11, further comprising: The weight configuration module is configured to dynamically adjust the weights of each pipeline instance based on the current state of each pipeline instance and the similarity of the time consumption of multiple tasks in each pipeline instance.

13. The assembly line scheduling device according to any one of claims 8-11, wherein: The shadow queue module is configured to use a distributed linked list queue data structure to perform a complete queue mapping for tasks in the pipeline tool cluster; For each task in the pipeline instance, determine the duration of releasable resources for that task based on its predicted execution time and current execution time; and determine the duration of releasable resources for the pipeline instance based on the duration of releasable resources for each task in the pipeline instance.

14. The assembly line scheduling device according to any one of claims 8-11, wherein: The scheduling module is configured to distribute newly produced pipeline tasks to the pipeline instance with the highest scheduling priority based on the scheduling priority of multiple pipeline instances.

15. A production line scheduling device, comprising: The memory is configured to store instructions; The processor is configured to execute the instructions, causing the pipeline scheduling apparatus to perform operations implementing the pipeline scheduling method as described in any one of claims 1-7.

16. A computer-readable storage medium, wherein, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the pipeline scheduling method as described in any one of claims 1-7.

Images