A method, system, electronic device and storage medium for dynamically adjusting pipeline construction resources

Abstract

The present invention discloses a method, system, electronic device and storage medium for dynamically adjusting pipeline construction resources, including monitoring the pipeline to be run; judging whether there is historical construction record data for the pipeline; if not, constructing an environment according to the preset resources of the pipeline, and scheduling the pipeline to run; if so, calculating the resource value according to the historical construction record data and the remaining resources in the current resource pool; constructing an environment based on the resource value, and scheduling the pipeline to run; recording the resource application amount and construction time of the pipeline. The present invention dynamically adjusts the resource size of each pipeline to be constructed each time, improves the utilization rate of the construction resource pool, and improves the overall performance of pipeline construction as a whole.

Description

Technical Field

[0001] The present invention relates to the technical field of resource adjustment, and in particular to a method, system, electronic device and storage medium for dynamically adjusting pipeline construction resources. Background Art

[0002] Devops continuous integration generally uses pipelines to achieve build automation, reduce manual intervention, and achieve rapid delivery. Currently, the development of container technology has led to the pooling of build resources, which can be dynamically scaled. Each time the pipeline runs, it applies for a containerized build environment from the resource pool. The allocation of build resources is generally configured manually. Regardless of the current usage of the resource pool, the build resources of a pipeline are generally fixed. This may result in some pipelines not being able to achieve optimal build performance due to the small set of build resources when resources are sufficient. One of the core concepts of Devops is automation. How to automatically and reasonably plan the resource pool and make full use of the build resources to speed up the build efficiency is an urgent problem to be solved.

[0003] Traditional pipeline build resources are generally set manually by users. Due to the diversity of projects, the complexity of each project is different, and the required build resources are also different. It is difficult for users to evaluate the optimal build resources for each pipeline, and it may take multiple debugging to come up with a more appropriate amount of resources.

[0004] The resource values ​​of traditional pipeline build environments are generally fixed, and it is difficult for users to perceive the current resource volume of the build resource pool in real time, and it is impossible to adjust the build resources in a timely manner.

[0005] When multiple different pipelines are running at the same time, they generally apply for resources in sequence, which may cause some pipelines to wait for idle resources for a long time. Summary of the invention

[0006] In view of the shortcomings existing in the above problems, the present invention provides a method, system, electronic device and storage medium for dynamically adjusting pipeline construction resources.

[0007] To achieve the above object, the present invention provides a method for dynamically adjusting pipeline construction resources, comprising:

[0008] Monitor the pipeline to be run;

[0009] Determine whether there is any historical construction record data for the pipeline;

[0010] If it does not exist, build the environment according to the preset resources of the pipeline and schedule the pipeline to run;

[0011] If so, the resource value is calculated according to the historical construction record data and the remaining resources in the current resource pool;

[0012] Building the environment based on the resource value and scheduling the pipeline to run;

[0013] Record the resource request amount and build time of the pipeline.

[0014] Preferably, the preset resources are initial preset values ​​of the CPU and memory.

[0015] Preferably, the resource value is calculated based on the historical construction record data and the remaining resources in the current resource pool. When the construction time is negatively correlated with the resource amount, the calculation formula is:

[0016] e=p*(1+q*n)

[0017] In the formula: e is the resource value; p is the preset resource; q is the set ratio; n is the number of increments.

[0018] Preferably, when the build time is non-negatively correlated with the amount of resources, the calculation formula is:

[0019] e=p*(1-q*n).

[0020] Preferably, a relationship curve is drawn based on the resource amount and the construction time, and the minimum resource amount is obtained based on the inflection point of the relationship curve, and the formula is:

[0021] e=Δt / Δr

[0022] Where: t is the construction time; r is the resource amount.

[0023] Preferably, when multiple pipelines are running, resource requirements are met based on the efficiency ranking of the pipelines.

[0024] The present application also provides a pipeline construction resource dynamic adjustment method system, including:

[0025] A monitoring module is used to monitor the pipeline to be run;

[0026] A judgment module, used to judge whether there is any historical construction record data for the pipeline;

[0027] An initialization module, used to build an environment according to the preset resources of the pipeline if it does not exist, and schedule the pipeline to run;

[0028] A calculation module, configured to calculate the resource value according to the historical construction record data and the remaining resources in the current resource pool, if any;

[0029] An operation module, used for constructing the environment based on the resource value and scheduling the pipeline to run;

[0030] The recording module is used to record the resource application amount and construction time of the pipeline.

[0031] The present invention also provides an electronic device, comprising at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the program is executed by the processing unit, the processing unit executes the above method.

[0032] The present invention also provides a storage medium storing a computer program executable by an electronic device, and when the program runs on the electronic device, the electronic device executes the above method.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] The present invention dynamically adjusts the resource size of each pipeline construction each time, improves the utilization rate of the construction resource pool, and improves the overall performance of the pipeline construction as a whole. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a flow chart of the method for dynamically adjusting pipeline construction resources of the present invention;

[0036] Figure 2 It is the relationship curve between resources and construction time in the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] Reference Figure 1 The present invention provides a method for dynamically adjusting pipeline construction resources, comprising:

[0039] Monitor the pipeline to be run;

[0040] Determine whether there is historical build record data for the pipeline;

[0041] If it does not exist, build the environment according to the preset resources of the pipeline and schedule the pipeline to run;

[0042] If it exists, the resource value is calculated based on the historical construction record data and the remaining resources in the current resource pool;

[0043] Build the environment based on resource values ​​and schedule the pipeline to run;

[0044] Record the resource request and build time of the pipeline.

[0045] In this embodiment, when the pipeline is running, the calculation module calculates the expected value e of the current resource amount based on the data of the historical construction record.

[0046] Specifically, when there is no historical build record for the first build, the computing module directly uses the initial preset values ​​(p) of the CPU and memory as build resources; (build resources refer to the CPU and memory)

[0047] e=p

[0048] During the second build, the computing module first increases the amount of build resources by a certain ratio (q) of the preset value; when the time of each build is negatively correlated with the amount of resources (the larger the resources, the less time), the amount of resources will continue to increase during the next build, and n is the number of increments;

[0049] e=p*(1+q*n)

[0050] If increasing the amount of resources during the second build does not reduce the build time, the calculation module will reduce the amount of resources by a certain proportion (q) of the preset value. If the time of this build does not increase with the reduction of resources, the amount of resources will continue to be reduced during the next build, and n is the number of reductions.

[0051] e=p*(1-q*n)

[0052] After multiple builds, the calculation module can be based on the relationship curve between resource r and build time t, such as Figure 2 As shown in the figure, if there are two trends: the build time decreases significantly with the increase of resources, and the time tends to be stable, then the resource value at the inflection point can be considered as the minimum resource amount that can make the pipeline build time the shortest, that is, the minimum r value when e=Δt / Δr approaches 0;

[0053] When multiple pipelines need to be executed, the resource pool may not be able to allocate the best amount of resources to each pipeline. At this time, the computing module will also dynamically plan the reasonable resources for each pipeline based on the resource utilization of different pipelines. From the perspective of resource efficiency, the above steps can calculate the time efficiency Δt / Δr of the resource volume of each pipeline. For example, if adding one unit of resources can reduce the construction time of pipeline A more than pipeline B, then the resource efficiency of pipeline A is higher. When the total amount of resources cannot meet the optimal resource requirements of both pipelines at the same time, the resource requirements of pipeline A will be met first.

[0054] The control module applies for a build environment for the pipeline and schedules the pipeline to run.

[0055] The monitoring module stores data such as resource application amount and construction time of each pipeline in the database for subsequent analysis.

[0056] In the daily development process, developers define many pipelines for continuous integration and continuous deployment to release new features or fixed defects to various environments. For example, a user defines a pipeline for code compilation and release in the test environment. The code is iterated every day and the pipeline needs to be run several times to release the test environment.

[0057] When defining the pipeline, the user presets the initial resource amount of the pipeline. In general scenarios, the pipeline will always run with the preset value.

[0058] After the device is introduced, it will calculate the current resource value based on the historical build data. If there is no historical build record, the preset value will be used. If there is a build record but the optimal resource has not been calculated, the resource amount will be automatically increased or decreased according to a certain ratio to run the pipeline. The purpose is to find the minimum resource amount that minimizes the build time after multiple builds. When the user runs the pipeline again, it will run with the optimal resources.

[0059] During the project development process of a software development company, the team uses pipelines for continuous integration and continuous deployment, and frequently runs multiple pipelines every day to ensure code updates and function testing. Taking one of the pipelines A used for code compilation and release in the test environment as an example, the implementation process of the present invention is described in detail.

[0060] When the developer defined this pipeline, he preset the initial resource volume as 1C2G (i.e. 1 CPU core and 2GB memory). When the pipeline was first run, since there was no historical build record data, the system built the environment according to the preset resources and scheduled the pipeline to run. The recording module recorded that the resource application volume of the pipeline was 1C2G and the first build time was 250 seconds.

[0061] As the project progresses, the user runs the pipeline for the second time, and the calculation module starts to calculate the resource value based on the data recorded in the first build. Assuming that the ratio q is set to 0.1, since the build time and the amount of resources are usually negatively correlated (the larger the resources, the shorter the build time), according to the formula e = p*(1+q*n) (n = 1 at this time, p is the preset resource 1C2G), the resource value of this build is calculated to be 1.1C2.2G. The system builds the environment based on this resource value and schedules the pipeline to run. This time the build time is shortened to 215 seconds. After the build is completed, the record module records the resource application amount as 1.1C2.2G and the build time as 215 seconds.

[0062] During the third build, the calculation module continued to calculate the resource value according to the formula e = p*(1+q*n) (n = 2 at this time), and obtained 1.2C2.4G. The system used the resource to build the environment and run the pipeline, and the build time was further shortened to 204 seconds. The recording module recorded the corresponding resource application amount and build time.

[0063] In the subsequent construction process, as the amount of resources gradually increased, the construction time continued to decrease. When the amount of resources reached 1.5C3G, the construction time was 143 seconds; when the amount of resources increased to 1.6C3.2G, the construction time was 144 seconds, and the construction time did not decrease due to the increase in the amount of resources. When the resources continued to increase to 1.7C3.4G and 1.8C3.6G, the construction time stabilized at around 142 seconds. The calculation module can obtain the relationship curve based on the data of the construction time t and the amount of resources r (such as Figure 2 As shown in the figure). It can be concluded from the curve that when the resource amount is 1.5C3G, the build time tends to be stable. At this time, the resource value corresponding to the inflection point of the curve can be considered as the minimum resource amount that can make the pipeline build time reach the shortest.

[0064] Through the above implementation process, the pipeline found the optimal resource configuration after multiple builds, and the build time was shortened from the initial 250 seconds to 143 seconds.

[0065] In a scenario where multiple pipelines are running simultaneously, assume that there is another pipeline B for code deployment in the development environment. The total amount of resources in the resource pool is limited. If it is not possible to simultaneously satisfy the optimal resource value for all pipelines to run, the calculation module calculates the resource efficiency (Δt / Δr) of each pipeline based on the resource utilization of different pipelines. For example, reducing one unit of resources (0.1C0.2G) will increase the build time of pipeline A under the optimal resource configuration by 17 seconds, while reducing the same unit of resources will increase the build time of pipeline B by 30 seconds, so pipeline B has a higher resource efficiency. In this case, appropriately reduce the amount of resources of pipeline A, give priority to meeting the resource requirements of pipeline B, and ensure the improvement of overall build efficiency.

[0066] The present application also provides a pipeline construction resource dynamic adjustment method system, including:

[0067] A monitoring module is used to monitor the pipeline to be run;

[0068] A judgment module is used to judge whether there is historical construction record data for the pipeline;

[0069] The initialization module is used to build the environment according to the preset resources of the pipeline if it does not exist, and schedule the pipeline to run;

[0070] A calculation module, used to calculate the resource value according to the historical construction record data and the remaining resources in the current resource pool, if any;

[0071] The run module is used to build the environment based on resource values ​​and schedule the pipeline to run;

[0072] The recording module is used to record the resource application amount and construction time of the pipeline.

[0073] The present invention also provides an electronic device, comprising at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the program is executed by the processing unit, the processing unit executes the above method.

[0074] The present invention also provides a storage medium storing a computer program executable by an electronic device. When the program runs on the electronic device, the electronic device executes the above method.

[0075] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for dynamically adjusting pipeline construction resources, characterized in that: include: Monitor the pipeline to be run; Determine whether there is any historical construction record data for the pipeline; If it does not exist, build the environment according to the preset resources of the pipeline and schedule the pipeline to run, where the preset resources are the initial preset values ​​of the CPU and memory; If it exists, the resource value is calculated according to the historical construction record data and the remaining resources in the current resource pool. When the construction time is negatively correlated with the resource amount, the calculation formula is: ; Where: e is the resource value; p is the preset resource; q To set the ratio; n is the number of increments; when the build time and the amount of resources are non-negatively correlated, the calculation formula is: ; Building the environment based on the resource value and scheduling the pipeline to run; Record the resource application amount and construction time of the pipeline; Among them, based on the relationship curve drawn between the resource amount and the construction time, there are two trends: the construction time decreases significantly with the increase of resource amount, and the time tends to be stable. The resource value at the inflection point is the minimum resource amount that makes the pipeline construction time the shortest, and e=Δt / Δr is the minimum r value when it approaches 0; where: t is the construction time; r is the resource amount.

2. The method for dynamically adjusting pipeline construction resources according to claim 1, characterized in that: When multiple pipelines are running, resource requirements are met based on the efficiency of the pipelines.

3. A pipeline construction resource dynamic adjustment method system, characterized in that: include: A monitoring module is used to monitor the pipeline to be run; A judgment module, used to judge whether there is any historical construction record data for the pipeline; An initialization module, used to build an environment according to the preset resources of the pipeline if it does not exist, and schedule the pipeline to run; A calculation module is used to calculate the resource value according to the historical construction record data and the remaining resources in the current resource pool, if any, wherein when the construction time is negatively correlated with the resource amount, the calculation formula is: ; Where: e is the resource value; p is the preset resource; q To set the ratio; n is the number of increments; when the build time and the amount of resources are non-negatively correlated, the calculation formula is: ; An operation module, used for constructing the environment based on the resource value and scheduling the pipeline to run; A recording module, used to record the resource application amount and construction time of the pipeline; Among them, based on the relationship curve drawn between the resource amount and the construction time, there are two trends: the construction time decreases significantly with the increase of resource amount, and the time tends to be stable. The resource value at the inflection point is the minimum resource amount that makes the pipeline construction time the shortest, and e=Δt / Δr is the minimum r value when it approaches 0; where: t is the construction time; r is the resource amount.

4. An electronic device, characterized in that: The method comprises at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the program is executed by the processing unit, the processing unit executes the method according to any one of claims 1 to 2.

5. A storage medium, characterized in that: It stores a computer program executable by an electronic device. When the program runs on the electronic device, the electronic device executes the method described in any one of claims 1 to 2.

Images