A self-feedback task scheduling method for DevOps platform

Abstract

The present invention particularly relates to a DevOps platform self-feedback task scheduling method. The DevOps platform self-feedback task scheduling method constructs a task pool and a thread pool to buffer the impact of DevOps platform task peak requests on the DevOps platform backend performance, and plays a dynamic peak-cutting role; monitors the running status of the Jenkins instance, provides the best scheduling strategy for Jenkins task allocation, realizes front-end request task scheduling and feedback-based load elastic scaling, thereby improving the overall processing efficiency of Jenkins tasks and the stability of DevOps services. The DevOps platform self-feedback task scheduling method dynamically corrects the ability of the Jenkins instance to provide services to the outside world based on the number of tasks currently being executed by the Jenkins instance as a feedback factor, ensures the robustness of the DevOps platform when facing a task peak, uses the scheduling strategy with the minimum number of tasks, enables scientific scheduling of tasks, avoids task accumulation, and improves the overall service efficiency of the DevOps platform.

Description

Technical Field

[0001] The present invention relates to the technical field of software development, and in particular to a DevOps platform self-feedback task scheduling method. Background Art

[0002] In the process of providing services to the outside world based on the DevOps platform of Jenkins, the backend performs basic processing on the task request of the DevOps platform and directly hands it over to the Jenkins instance for processing. When a large number of centralized requests appear, a large number of tasks pile up, which increases the pressure on Jenkins and easily leads to deadlock due to insufficient resources. At the same time, the commonly used task scheduling strategies of the round-robin method, the hashing method and the least-link method cannot be changed and adjusted according to the real-time task status of the Jenkins instance, and the task scheduling made cannot reach the global optimal situation. In view of the above situation, the present invention proposes a self-feedback task scheduling method for a DevOps platform. Summary of the invention

[0003] In order to overcome the defects of the prior art, the present invention provides a simple and efficient DevOps platform self-feedback task scheduling method.

[0004] The present invention is achieved through the following technical solutions:

[0005] A DevOps platform self-feedback task scheduling method, characterized in that it includes the following steps:

[0006] The first step is to build a task pool and thread pool to buffer the impact of DevOps platform task peak requests on the DevOps platform backend performance and play a dynamic peak reduction role;

[0007] After receiving the request to execute the Jenkins task, the DevOps platform decides whether to insert the task into the task pool based on the number of existing tasks in the current task pool;

[0008] The second step is to monitor the running status of the Jenkins instance and provide the best scheduling strategy for Jenkins task allocation, so as to realize front-end request task scheduling and feedback-based load elastic scaling.

[0009] Use the running status of the current Jenkins instance as a basis to increase or decrease the number of Jenkins instances, expand or reduce the capacity of the task pool and thread pool, thereby improving the overall processing efficiency of Jenkins tasks and improving the stability of DevOps services.

[0010] In the first step, the construction process of the task pool and thread pool is as follows:

[0011] S1. Get the number M of backend Jenkins instances and the number N of concurrent tasks that each Jenkins instance can execute in parallel;

[0012] S2. Use the Go language to generate a specified number of resident threads. The number of threads T is M*N, and one thread corresponds to one concurrent task of the Jenkins instance.

[0013] S3. Create a task pool through the Map data structure. The number of tasks that can be accommodated in the task pool is S (M+1)*N. The number of tasks that can be accommodated in the task pool is N more than the number of threads.

[0014] S4. When the number of tasks S in the task pool reaches the maximum number of tasks currently supported, the Jenkins task request sent by the front end returns a busy prompt.

[0015] S5. Start the monitoring task. When a new task arrives in the task pool, call the idle thread from the thread pool to schedule the task.

[0016] In step S5, the thread pool monitors the actions of the task pool in real time, and when it is found that the front end has a request to execute a Jenkins task, it queries the capacity of the current task pool, and inserts the task into the task pool if the number of tasks S in the task pool after insertion does not exceed the upper limit of the task pool capacity.

[0017] In step S5, the threads in the thread pool monitor the task pool in real time, and when finding that there is task data in the task pool, randomly select an idle thread from the thread pool to execute the scheduled task of Jenkins.

[0018] In the second step, the DevOps platform backend is equipped with a Jenkins instance monitoring module to obtain the running status of the Jenkins instance from all current Jenkins instances, and use the obtained running status as an important influencing factor to react to the load elastic scaling and scheduling strategy;

[0019] The process of monitoring the running status of the Jenkins instance is as follows:

[0020] S1. Obtain information about the Jenkins instances in use in the environment, including the Jenkins running status and the number of currently running tasks, and store them in the MySQL database;

[0021] S2. Perform a scheduled login operation on the daemon thread of each Jenkins instance. If the login is successful, it means that the Jenkins instance is running well. If the login fails, it means that the corresponding Jenkins instance service is terminated. At the same time, modify the running status attributes of the corresponding Jenkins instance in the MySQL database;

[0022] S3. Get the number of tasks running on each running Jenkins instance and update the corresponding information into the MySQL database.

[0023] S4. When a new Jenkins instance is started, the information of the newly started Jenkins instance is synchronized to the MySQL database. In the initial state, the number of running tasks of the Jenkins instance is 0;

[0024] S5. When a Jenkins instance is deleted, the information of the Jenkins instance is deleted from the MySQL database.

[0025] In the second step, the front-end request task scheduling process is as follows:

[0026] S1. Store the front-end Jenkins tasks in the task pool, and the threads in the thread pool obtain tasks from the task pool for scheduling;

[0027] S2. Obtain the number of tasks in all current Jenkins instances from the MySQL database, and sort the Jenkins instances from low to high according to the number of tasks;

[0028] S3. If the task volume of the Jenkins instance with the lowest number of tasks is less than N, the task is scheduled to the corresponding Jenkins instance for execution;

[0029] S4. If the number of tasks of the Jenkins instance with the lowest number of tasks is equal to N, the Go coroutine is used to monitor the number of tasks in all Jenkins instances. If a Jenkins instance with a number of ongoing tasks less than N is found, it will be considered to be scheduled to the corresponding Jenkins instance for execution;

[0030] S5. Get the number of tasks accumulated in the task pool. When the number of tasks accumulated in the task pool is greater than or equal to N, reopen a Jenkins instance to provide external services.

[0031] S6. Get the number of tasks accumulated in the task pool. When the number of tasks running in all Jenkins instances is less than or equal to N, close a Jenkins instance to release resources.

[0032] In the second step, the number of backend Jenkins instances will increase and decrease as the load changes. This change in number in turn affects the capacity of the task pool and thread pool. The capacity of the task pool and thread pool is used to control the backend's ability to handle the load.

[0033] When the number of backend Jenkins instances increases by 1, the capacity of the task pool increases by N on the original basis, and the capacity of the thread pool increases by 1 on the original basis;

[0034] When the number of backend Jenkins instances decreases by 1, the capacity of the task pool decreases by N on the original basis, and the capacity of the thread pool decreases by 1 on the original basis.

[0035] More specifically, the load elastic scaling process based on feedback is as follows:

[0036] S1. Obtain the total number of tasks n in the current task pool. If n < (M + 1) * N, directly insert the task into the task pool;

[0037] S2. If n = (M + 1) * N, the DevOps platform creates a new Jenkins instance and stores the new Jenkins instance in the Mysql database;

[0038] S3. If n < N, the DevOps platform shuts down the redundant Jenkins instances and deletes the corresponding Jenkins instance information from the Mysql database;

[0039] S4. If the number of Jenkins instances increases by a, the capacity of the task pool expands by a * N, and the number of threads in the thread pool expands by a;

[0040] S5. If the number of Jenkins instances decreases by a, the capacity of the task pool decreases by a * N, and the number of threads in the thread pool decreases by a;

[0041] S6. The reduction of Jenkins instances is limited by the number of benchmark Jenkins instances and cannot be less than the limit of the minimum number of Jenkins instances.

[0042] The beneficial effects of the present invention are as follows: The self-feedback task scheduling method of the DevOps platform uses the number of tasks currently being executed by the Jenkins instance as a feedback factor to dynamically correct the service-providing ability of the Jenkins instance, ensuring the robustness of the DevOps platform in the face of task peaks. Using the scheduling strategy with the minimum number of tasks makes the task scheduling scientific, avoids task backlogs, and improves the overall service efficiency of the DevOps platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0044] Attached Figure 1 Schematic diagram of the self-feedback task scheduling method of the DevOps platform of the present invention.

[0045] Attached Figure 2 It is a schematic diagram of the Jenkins task scheduling strategy flow of the present invention.

[0046] Attached Figure 3 The figure is a schematic diagram of the load elastic scaling process based on feedback of the present invention. DETAILED DESCRIPTION

[0047] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0048] The task pool and thread pool can effectively buffer and reduce the peak of front-end task requests. At the same time, the real-time monitoring results of Jenkins status can be used in the task scheduling strategy to help obtain the global optimal solution for scheduling. Dynamically adjusting the task pool and thread pool capacity according to the monitoring results can help DevOps dynamically adjust the load capacity.

[0049] The DevOps platform self-feedback task scheduling method includes the following steps:

[0050] The first step is to build a task pool and thread pool to buffer the impact of DevOps platform task peak requests on the DevOps platform backend performance and play a dynamic peak reduction role;

[0051] After receiving the request to execute the Jenkins task, the DevOps platform decides whether to insert the task into the task pool based on the number of existing tasks in the current task pool;

[0052] The second step is to monitor the running status of the Jenkins instance and provide the best scheduling strategy for Jenkins task allocation, so as to realize front-end request task scheduling and feedback-based load elastic scaling.

[0053] Use the running status of the current Jenkins instance as a basis to increase or decrease the number of Jenkins instances, expand or reduce the capacity of the task pool and thread pool, thereby improving the overall processing efficiency of Jenkins tasks and improving the stability of DevOps services.

[0054] In the first step, the construction process of the task pool and thread pool is as follows:

[0055] S1. Get the number M of backend Jenkins instances and the number N of concurrent tasks that each Jenkins instance can execute in parallel;

[0056] S2. Use the Go language to generate a specified number of resident threads. The number of threads T is M*N, and one thread corresponds to one concurrent task of the Jenkins instance.

[0057] S3. Create a task pool through the Map data structure. The number of tasks that can be accommodated in the task pool is S (M+1)*N. The number of tasks that can be accommodated in the task pool is N more than the number of threads.

[0058] S4. When the number of tasks S in the task pool reaches the maximum number of tasks currently supported, the Jenkins task request sent by the front end returns a busy prompt.

[0059] S5. Start the monitoring task. When a new task arrives in the task pool, call the idle thread from the thread pool to schedule the task.

[0060] In step S5, the thread pool monitors the actions of the task pool in real time, and when it is found that the front end has a request to execute a Jenkins task, it queries the capacity of the current task pool, and inserts the task into the task pool if the number of tasks S in the task pool after insertion does not exceed the upper limit of the task pool capacity.

[0061] In step S5, the threads in the thread pool monitor the task pool in real time, and when finding that there is task data in the task pool, randomly select an idle thread from the thread pool to execute the scheduled task of Jenkins.

[0062] In the second step, the DevOps platform backend is equipped with a Jenkins instance monitoring module to obtain the running status of the Jenkins instance from all current Jenkins instances, and use the obtained running status as an important influencing factor to react to the load elastic scaling and scheduling strategy;

[0063] The process of monitoring the running status of the Jenkins instance is as follows:

[0064] S1. Obtain information about the Jenkins instances in use in the environment, including the Jenkins running status and the number of currently running tasks, and store them in the MySQL database;

[0065] S2. Perform a scheduled login operation on the daemon thread of each Jenkins instance. If the login is successful, it means that the Jenkins instance is running well. If the login fails, it means that the corresponding Jenkins instance service is terminated. At the same time, modify the running status attributes of the corresponding Jenkins instance in the MySQL database;

[0066] S3. Get the number of tasks running on each running Jenkins instance and update the corresponding information into the MySQL database.

[0067] S4. When a new Jenkins instance is started, the information of the newly started Jenkins instance is synchronized to the MySQL database. In the initial state, the number of running tasks of the Jenkins instance is 0;

[0068] S5. When a Jenkins instance is deleted, the information of the Jenkins instance is deleted from the MySQL database.

[0069] In the second step, the front-end request task scheduling process is as follows:

[0070] S1. Store the front-end Jenkins tasks in the task pool, and the threads in the thread pool obtain tasks from the task pool for scheduling;

[0071] S2. Obtain the number of tasks in all current Jenkins instances from the MySQL database, and sort the Jenkins instances from low to high according to the number of tasks;

[0072] S3. If the task volume of the Jenkins instance with the lowest number of tasks is less than N, the task is scheduled to the corresponding Jenkins instance for execution;

[0073] S4. If the number of tasks of the Jenkins instance with the lowest number of tasks is equal to N, the Go coroutine is used to monitor the number of tasks in all Jenkins instances. If a Jenkins instance with a number of ongoing tasks less than N is found, it will be considered to be scheduled to the corresponding Jenkins instance for execution;

[0074] S5. Get the number of tasks accumulated in the task pool. When the number of tasks accumulated in the task pool is greater than or equal to N, reopen a Jenkins instance to provide external services.

[0075] S6. Get the number of tasks accumulated in the task pool. When the number of tasks running in all Jenkins instances is less than or equal to N, close a Jenkins instance to release resources.

[0076] In the second step, the number of backend Jenkins instances will increase and decrease as the load changes. This change in number in turn affects the capacity of the task pool and thread pool. The capacity of the task pool and thread pool is used to control the backend's ability to handle the load.

[0077] When the number of backend Jenkins instances increases by 1, the capacity of the task pool increases by N on the original basis, and the capacity of the thread pool increases by 1 on the original basis;

[0078] When the number of backend Jenkins instances decreases by 1, the capacity of the task pool decreases by N on the original basis, and the capacity of the thread pool decreases by 1 on the original basis.

[0079] More specifically, the load elastic scaling process based on feedback is as follows:

[0080] S1. Obtain the total number of tasks n in the current task pool. If n < (M + 1) * N, directly insert the task into the task pool;

[0081] S2. If n = (M + 1) * N, the DevOps platform creates a new Jenkins instance and stores the new Jenkins instance in the Mysql database;

[0082] S3. If n < N, the DevOps platform shuts down the redundant Jenkins instances and deletes the corresponding Jenkins instance information from the Mysql database;

[0083] S4. If the number of Jenkins instances increases by a, the capacity of the task pool expands by a * N, and the number of threads in the thread pool expands by a;

[0084] S5. If the number of Jenkins instances decreases by a, the capacity of the task pool decreases by a * N, and the number of threads in the thread pool decreases by a;

[0085] S6. The reduction of Jenkins instances is limited by the number of benchmark Jenkins instances and cannot be less than the limit of the minimum number of Jenkins instances.

[0086] The above-described embodiments are only one of the specific implementation manners of the present invention. Any ordinary changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included in the protection scope of the present invention.

Claims

1. A DevOps platform self-feedback task scheduling method, characterized in that: The following steps are involved: The first step is to build a task pool and thread pool to buffer the impact of DevOps platform task peak requests on the DevOps platform backend performance and play a dynamic peak reduction role; After receiving the request to execute the Jenkins task, the DevOps platform decides whether to insert the task into the task pool based on the number of existing tasks in the current task pool; The second step is to monitor the running status of the Jenkins instance and provide the best scheduling strategy for Jenkins task allocation, so as to realize front-end request task scheduling and feedback-based load elastic scaling. Use the current running status of the Jenkins instance as a basis to increase or decrease the number of Jenkins instances, expand or reduce the capacity of the task pool and thread pool, thereby improving the overall processing efficiency of Jenkins tasks and improving the stability of DevOps services; the details are as follows: The number of backend Jenkins instances will increase or decrease as the load changes. This change in number in turn affects the capacity of the task pool and thread pool. The capacity of the task pool and thread pool is used to control the backend's ability to handle the load. When the number of backend Jenkins instances increases by 1, the capacity of the task pool increases by N, and the capacity of the thread pool increases by 1. When the number of backend Jenkins instances decreases by 1, the capacity of the task pool decreases by N, and the capacity of the thread pool decreases by 1.

2. The DevOps platform self-feedback task scheduling method according to claim 1, characterized in that: In the first step, the construction process of the task pool and thread pool is as follows: S101. Obtain the number M of backend Jenkins instances and the number N of concurrent tasks that each Jenkins instance can execute in parallel; S102. Use the Go language to generate a specified number of resident threads, where the number of threads T is M*N, and one thread corresponds to one concurrent task of the Jenkins instance; S103. Create a task pool through the Map data structure. The number of tasks that can be accommodated in the task pool is S (M+1)*N. The number of tasks that can be accommodated in the task pool is N more than the number of threads. S104. When the number of tasks S in the task pool reaches the maximum number of tasks currently supported, the Jenkins task request sent by the front end returns a busy prompt; S105. Start the monitoring task. When a new task arrives in the task pool, call the idle thread from the thread pool to schedule the task.

3. The DevOps platform self-feedback task scheduling method according to claim 2, characterized in that: In step S105, the thread pool monitors the actions of the task pool in real time, and when it is found that the front end has a request to execute a Jenkins task, it queries the capacity of the current task pool, and inserts the task into the task pool if the number of tasks S in the task pool after insertion does not exceed the upper limit of the task pool capacity.

4. The DevOps platform self-feedback task scheduling method according to claim 2, characterized in that: In step S105, the threads in the thread pool monitor the task pool in real time, and when finding that there is task data in the task pool, randomly select an idle thread from the thread pool to execute the scheduled task of Jenkins.

5. The DevOps platform self-feedback task scheduling method according to claim 1, characterized in that: In the second step, the DevOps platform backend is equipped with a Jenkins instance monitoring module to obtain the running status of the Jenkins instance from all current Jenkins instances, and use the obtained running status as an important influencing factor to react to the load elastic scaling and scheduling strategy; The process of monitoring the running status of the Jenkins instance is as follows: S201. Obtain the Jenkins instance information used in the environment, including the Jenkins running status and the number of currently running tasks, and store it in the MySQL database; S202. Perform a scheduled login operation on the daemon thread of each Jenkins instance. If the login is successful, it means that the Jenkins instance is running well. If the login fails, it means that the corresponding Jenkins instance service is terminated. At the same time, modify the running status attributes of the corresponding Jenkins instance in the MySQL database; S203. Obtain the number of tasks currently being run by each running Jenkins instance, and update the corresponding information into the MySQL database; S204. When a new Jenkins instance is started, the information of the newly started Jenkins instance is synchronized to the MySQL database. In the initial state, the number of running tasks of the Jenkins instance is 0; S205. When a Jenkins instance is deleted, the information of the Jenkins instance is deleted from the MySQL database.

6. The DevOps platform self-feedback task scheduling method according to claim 1 or 5, characterized in that: In the second step, the front-end request task scheduling process is as follows: S301. Store the front-end Jenkins tasks in the task pool, and the threads in the thread pool obtain tasks from the task pool for scheduling; S302. Obtain the number of tasks in all current Jenkins instances from the MySQL database, and sort the Jenkins instances from low to high according to the number of tasks; S303. If the task amount of the Jenkins instance with the lowest number of tasks is less than N, the task is scheduled to the corresponding Jenkins instance for execution; S304. If the number of tasks of the Jenkins instance with the lowest number of tasks is equal to N, use the Go coroutine to monitor the number of tasks in all Jenkins instances. If a Jenkins instance with a number of ongoing tasks less than N is found, it will be considered to be scheduled to the corresponding Jenkins instance for execution; S305. Get the number of tasks accumulated in the task pool. When the number of tasks accumulated in the task pool is greater than or equal to N, reopen a Jenkins instance to provide external services. S306. Obtain the number of tasks accumulated in the task pool, and when the number of tasks running in all Jenkins instances is less than or equal to N, close a Jenkins instance to release resources.

7. The DevOps platform self-feedback task scheduling method according to claim 1 or 5, characterized in that: In the second step, the feedback-based load elastic scaling process is as follows: S401. Get the total number of tasks n in the current task pool. If n<(M+1)*N, directly insert the task into the task pool. S402. If n = (M + 1) * N, the DevOps platform creates a new Jenkins instance and stores the new Jenkins instance in the Mysql database; S403. If n < N, the DevOps platform shuts down the redundant Jenkins instances and deletes the corresponding Jenkins instance information from the Mysql database; S404. If the number of Jenkins instances increases by a, the capacity of the task pool expands by a * N, and the number of threads in the thread pool expands by a; S405. If the number of Jenkins instances decreases by a, the capacity of the task pool decreases by a * N, and the number of threads in the thread pool decreases by a; S406. The reduction of Jenkins instances is limited by the number of benchmark Jenkins instances and cannot be less than the limit of the minimum number of Jenkins instances.

Images