Optimization method and system for continuous integration pipeline

Abstract

The present invention provides a method and system for optimizing a continuous integration pipeline. The embodiment of the present invention can also use a task importance rating algorithm to distinguish between general tasks and important tasks in the pipeline, intelligently monitor and adjust the task execution order of the pipeline, and when the pipeline is restarted, unnecessary tasks can be skipped and important tasks can be executed directly, which greatly reduces the time cost of the continuous integration pipeline execution while ensuring the successful delivery of the application, saves system resources, and improves the execution efficiency of the pipeline.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and system for optimizing a continuous integration pipeline. Background Art

[0002] The goal of modern application development is to have multiple developers working on different features of the same application at the same time. However, if an enterprise schedules all branches of source code to be merged together in one day (known as "merge day"), it can end up being a tedious, time-consuming, and manual task. This is because when a developer working independently makes changes to the application, there is a chance that they will conflict with changes made by other developers at the same time. This problem is exacerbated if each developer customizes their own local Integrated Development Environment (IDE) instead of having the team agree on a cloud-based IDE.

[0003] Continuous integration (CI) is a method of delivering applications to customers frequently by introducing automation during the application development phase. CI helps developers merge code changes into a shared branch or "trunk" more frequently. Once the changes made by the developer to the application are merged, the system will verify these changes by automatically building the application and running different levels of automated tests to ensure that these changes have not caused damage to the application. This means that the testing content covers the different modules that make up the entire application. If automated testing finds conflicts between new code and existing code, CI can make it easier to quickly fix these errors. Recently, CI technology has also been widely used in the field of machine learning.

[0004] A solution in the prior art for optimizing CI request queues using code review quality determines the reliability value of each request pulled from the CI queue based on a reliability factor and classifies them into high-priority integration queues and low-priority integration queues. The classifier sends pull requests in the trusted classification to the high-priority integration queue, and sends pull requests in the untrusted classification to the low-priority integration queue to ensure product delivery quality when multiple integration requests are made.

[0005] The above solution does not involve the internal aspects of CI. It simply sets priorities based on code quality to determine the execution order of the CI pipeline. However, it does not consider the impact of modifications to source code and configuration files on each task in the entire pipeline, nor does it consider more refined monitoring and deployment of pipeline tasks based on modifications to different files. Therefore, the execution efficiency of the CI pipeline is still low. Summary of the invention

[0006] The technical problem to be solved by the embodiments of the present invention is to provide a method and system for optimizing a continuous integration pipeline, thereby improving the execution efficiency of the continuous integration pipeline and saving system resources.

[0007] To solve the above technical problems, according to one aspect of the present invention, a method for optimizing a continuous integration pipeline is provided, comprising:

[0008] According to the pipeline configuration file of the target application, execute each task of the pipeline in sequence;

[0009] When the current task fails to execute, the pipeline state corresponding to the current task is saved, and the pipeline state includes the states of all tasks before the current task;

[0010] After the pipeline modification is completed, the importance score of each task is calculated according to the importance index and corresponding weight of each task;

[0011] Determine the remaining tasks that need to be continued to be executed in the pipeline according to the importance score, and update the pipeline configuration file;

[0012] Execute the remaining tasks in the pipeline according to the updated pipeline configuration file and the saved pipeline state.

[0013] Optionally, in the process of sequentially executing each remaining task in the pipeline, if any task fails to execute, the process returns to the step of saving the pipeline state corresponding to the current task when the current task fails to execute.

[0014] Optionally, the optimization method further includes:

[0015] After all tasks in the pipeline are successfully executed, the target application is released or deployed.

[0016] Optionally, the task importance index includes all of the following indexes:

[0017] The number of files modified in the source code files related to the task;

[0018] The number of modified parameters among the external parameters related to the task;

[0019] The number of containers that need to be started for the task;

[0020] whether the task interacts with the optimization platform of the continuous integration pipeline;

[0021] Whether the task uses graphics processor GPU resources.

[0022] Optionally, determining the remaining tasks that need to be continued to be executed in the pipeline according to the importance score includes:

[0023] Select the important tasks whose importance scores exceed a preset threshold value, and determine the first task with the highest execution order in the pipeline from the important tasks;

[0024] From the first task and the second task, a target task that is executed earlier in the pipeline is selected, and all tasks in the pipeline starting from the target task are used as remaining tasks that need to be continued to be executed in the pipeline, wherein the second task is a task that failed to execute before the pipeline was modified.

[0025] Optionally, when the current task fails to execute, saving the pipeline state corresponding to the current task includes:

[0026] Combine the name of the pipeline, the timestamp when the current task fails to execute, and the name of the current task to generate a pipeline state name corresponding to the current task, and obtain the states of all tasks before the current task;

[0027] The keyword key is the pipeline state name, and the value value is the key-value pair of the pipeline state corresponding to the current task, which is saved in the database of the state machine container.

[0028] Optionally, executing the remaining tasks in the pipeline according to the updated pipeline configuration file and the saved pipeline state includes:

[0029] According to the pipeline state name, obtaining the pipeline state corresponding to the pipeline state name from the database of the state machine container;

[0030] According to the obtained pipeline status, the remaining tasks in the pipeline are executed.

[0031] Optionally, when updating the pipeline configuration file, the pipeline state name is further used as the file name of the pipeline configuration file;

[0032] When obtaining the pipeline state corresponding to the pipeline state name from the database of the state machine container according to the pipeline state name, query the database of the state machine container according to the file name of the pipeline configuration file to obtain the pipeline state corresponding to the file name of the pipeline configuration file.

[0033] According to another aspect of the present invention, there is also provided an optimization system for a continuous integration pipeline, comprising: a pipeline operation module, for sequentially executing each task of the pipeline according to a pipeline configuration file of a target application;

[0034] A state management module, used to save the pipeline state corresponding to the current task when the current task fails to execute, and the pipeline state includes the states of all tasks before the current task;

[0035] An importance rating module, used to calculate the importance score of each task according to the importance index and corresponding weight of each task after the pipeline modification is completed; determine the remaining tasks that need to be continued in the pipeline according to the importance score, and update the pipeline configuration file;

[0036] The pipeline running module is also used to execute the remaining tasks in the pipeline according to the updated pipeline configuration file and the saved pipeline status.

[0037] Optionally, the state management module is also used to trigger the state management module to save the pipeline state corresponding to the current task if any task fails to execute during the process of sequentially executing each remaining task in the pipeline.

[0038] Optionally, the importance index of each task includes all of the following indicators:

[0039] The number of files modified in the source code files related to the task;

[0040] The number of modified parameters among the external parameters related to the task;

[0041] The number of containers that need to be started for the task;

[0042] whether the task interacts with the optimization platform of the continuous integration pipeline;

[0043] Whether the task uses graphics processor GPU resources.

[0044] Optionally, the importance rating module is also used to select important tasks whose importance scores exceed a preset threshold value, and determine the first task with the highest execution order in the pipeline from the important tasks; from the first and second tasks, select a target task with the highest execution order in the pipeline, and use all tasks in the pipeline starting from the target task as the remaining tasks that need to be continued in the pipeline, wherein the second task is the task that failed to execute before the pipeline was modified.

[0045] According to another aspect of the present invention, there is also provided a continuous integration pipeline optimization system, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the continuous integration pipeline optimization method as described above.

[0046] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the continuous integration pipeline optimization method described above are implemented.

[0047] Compared with the prior art, the continuous integration pipeline optimization method and system provided by the embodiment of the present invention have at least the following beneficial effects:

[0048] In the embodiment of the present invention, state preservation technology is used to ensure the correct acquisition of task input when the pipeline is restarted. In addition, the embodiment of the present invention can also use a task importance rating algorithm to distinguish between general tasks and important tasks in the pipeline, and intelligently monitor and allocate the task execution order of the pipeline. When the pipeline is restarted, unnecessary tasks can be skipped and important tasks can be executed directly. While ensuring the successful delivery of the application, the time cost of the continuous integration pipeline execution is greatly reduced, system resources are saved, and the execution efficiency of the pipeline is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0050] Figure 1 A schematic diagram of an application scenario of the continuous integration pipeline optimization method of an embodiment of the present invention;

[0051] Figure 2 A schematic diagram of a flow chart of a method for optimizing a continuous integration pipeline according to an embodiment of the present invention;

[0052] Figure 3 An example flow chart of a method for optimizing a continuous integration pipeline according to an embodiment of the present invention;

[0053] Figure 4 A schematic diagram of the structure of an optimization system for a continuous integration pipeline according to an embodiment of the present invention;

[0054] Figure 5 Another structural schematic diagram of the optimization system of the continuous integration pipeline according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, the following will be described in detail in conjunction with the accompanying drawings and specific embodiments. In the following description, specific details such as specific configurations and components are provided only to help fully understand the embodiments of the present invention. Therefore, it should be clear to those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. In addition, for clarity and brevity, the description of known functions and structures is omitted.

[0056] It should be understood that the references to "one embodiment" or "an embodiment" throughout the specification mean that the specific features, structures, or characteristics associated with the embodiment are included in at least one embodiment of the present invention. Therefore, the references to "in one embodiment" or "in an embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0057] In various embodiments of the present invention, it should be understood that the size of the serial numbers of the following processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0058] As described in the background technology, the continuous integration pipeline of the prior art has problems such as low execution efficiency during execution. In order to solve at least one of the above problems, an embodiment of the present invention provides an optimization method for a continuous integration pipeline. The method intelligently monitors and adjusts the task execution order of the pipeline. When the pipeline is restarted, unnecessary tasks can be skipped and important tasks can be executed directly, thereby improving the execution efficiency of the pipeline.

[0059] The embodiment of the present invention provides an optimization method for a continuous integration pipeline, which can solve the problems of task state recovery and efficient execution of the continuous integration pipeline in application development such as machine learning. The following will mainly take the Kubernetes platform as an example to illustrate the relevant technical solutions of the embodiment of the present invention. It should be noted that the embodiment of the present invention is not limited to the Kubernetes platform, and any platform that can build a continuous integration pipeline can be applied to the present invention.

[0060] Kubernetes is an open source container orchestration engine developed by Google that supports automated deployment, large-scale scalability, and containerized application management. The Kubernetes platform is used to manage containerized applications on multiple hosts in a cloud platform. The goal of Kubernetes is to make the deployment of containerized applications simple and efficient. Kubernetes provides a mechanism for application deployment, planning, updating, and maintenance. In Kubernetes, you can create multiple containers, each of which runs an application instance. Then, through the built-in load balancing strategy, you can manage, discover, and access this group of application instances. These details do not require the operation and maintenance personnel to perform complex manual configuration and processing.

[0061] The embodiment of the present invention implements a continuous integration pipeline task optimization technology based on the Kubernetes platform. The continuous integration framework intelligently classifies pipeline tasks according to the running status and quickly restores pipeline execution. Figure 1 A schematic diagram of an application scenario of the optimization method of the continuous integration pipeline described in an embodiment of the present invention is provided. Figure 1 In the Kubernetes platform, the Kubernetes platform is built on a hardware resource platform, which includes CPU, graphics processing unit (GPU), AI chip, memory, and network resources. A continuous integration / deployment pipeline is built on the Kubernetes platform for continuous integration and continuous deployment of applications.

[0062] like Figure 2 As shown, the optimization method of the continuous integration pipeline provided by the embodiment of the present invention includes:

[0063] Step 21, executing each task of the pipeline in sequence according to the pipeline configuration file of the target application.

[0064] Here, the target application may be an application such as machine learning, specifically artificial intelligence (AI), sentiment analysis (Sentimental Analysis), etc. The pipeline configuration file usually configures the tasks to be executed and the execution order of the tasks. The specific parameters may include: the name, version and URL of the image running the task, the name and version of the model corresponding to the target application, and other information.

[0065] Step 22, when the current task fails to execute, save the pipeline state corresponding to the current task, and the pipeline state includes the states of all tasks before the current task.

[0066] Here, each task is executed in sequence according to the pipeline configuration file until all tasks are completed. During the task execution process, if a task fails to execute, the execution of the pipeline is terminated. At this time, the corresponding pipeline state is saved, and the pipeline state includes the state of all tasks before the current task that failed to execute. For example, assuming that the current task that failed to execute is the third task, and the two tasks between the third task are successfully executed, at this time, the state of the previous two tasks needs to be saved. The state of the task can specifically include the state value of the completed task and its output. The state value is used to indicate whether the task is successfully executed, and the output can be characterized by the name of the output file after the task is successfully executed and the address of the output file. In this way, when a task fails, the embodiment of the present invention will save the state of all successfully executed tasks before the failed task.

[0067] Step 23, after the pipeline modification is completed, the importance score of each task is calculated according to the importance index and corresponding weight of each task.

[0068] After the task execution fails in the above step 22, wait for the developer to modify the pipeline. The specific modification can be the file and related external parameters of the task that failed to execute, or the file and related external parameters of other tasks. The source code files, parameter configurations and containers of one or some tasks may be modified during the modification. In order to improve the execution efficiency of the pipeline, after the above modification is completed, the embodiment of the present invention calculates the importance score of each task based on the importance index and corresponding weight of each task.

[0069] Here, the importance index of the task includes all of the following indexes:

[0070] 1) The number of modified files in the source code files related to the task;

[0071] 2) the number of modified parameters in the external parameters related to the task;

[0072] 3) The number of containers that need to be started for the task;

[0073] 4) Whether the task interacts with the optimization platform of the continuous integration pipeline (such as the Kubernetes platform);

[0074] 5) Whether the task uses GPU resources.

[0075] Generally, among the above five parameters, the importance of the first to third parameters is relatively high, and the importance of the fourth to fifth parameters is relatively low. In other words, the weights of the first to third parameters are usually higher than the weights of the fourth to fifth parameters. Optionally, the weights of the first to third parameters decrease in sequence.

[0076] When calculating the importance score of a task, the product of each importance indicator of the task and the corresponding weight can be calculated, and then the products are summed up, and the sum is used as the importance score of the task.

[0077] Step 24: Determine the remaining tasks that need to be continued in the pipeline according to the importance score, and update the pipeline configuration file.

[0078] Here, it is possible to determine whether the importance score exceeds a preset threshold value, select the important tasks whose importance score exceeds the preset threshold value, and then select the first task with the highest execution order according to the execution order of these important tasks in the pipeline. Then, from the first and second tasks, select the target task with the highest execution order in the pipeline, and use all tasks in the pipeline starting from the target task as the remaining tasks that need to be continued in the pipeline. Then, based on these remaining tasks, update the pipeline configuration file, and the updated pipeline configuration file only includes these remaining tasks. Among them, the second task is the task that failed to execute before the pipeline was modified.

[0079] As a special case, the first task and the second task may be the same task. In this case, any one of the two can be selected as the target task.

[0080] In this way, the embodiment of the present invention can use the task importance rating algorithm to distinguish between general tasks and important tasks in the pipeline, intelligently monitor and allocate the task execution order of the pipeline, and when the pipeline is restarted, unnecessary tasks can be skipped and important tasks can be executed directly.

[0081] For example, suppose the pipeline has tasks 1 to 6 that are executed in sequence, and suppose a failure occurs when executing task 3. After the pipeline is subsequently modified, the importance scores of each task are calculated, and it is found that tasks 2 and 4 are important tasks. At this time, the important tasks with the highest execution order are task 2 (i.e., the first task). Then, the target task with the highest order is selected from task 2 (i.e., the first task) and task 3 (i.e., the second task), and the target task is task 2. Therefore, tasks 2 to 6 starting from task 2 are used as the remaining tasks to update the pipeline configuration file. The updated pipeline configuration file includes the remaining tasks that need to be executed and the execution order of the remaining tasks. The specific parameters may include: the name, version and URL address of the image library of the remaining task running image, the name and version of the model corresponding to the target application, and other information.

[0082] For another example, suppose that the pipeline has tasks 1 to 6 that are executed in sequence, and suppose that a failure occurs when executing task 3. After the pipeline is subsequently modified, the importance scores of each task are calculated, and it is found that tasks 5 and 6 are important tasks. At this time, the important tasks with the highest execution order are task 5 (i.e., the first task). Then, the target task with the highest order is selected from task 5 (i.e., the first task) and task 3 (i.e., the second task), and the target task is task 3. Therefore, tasks 3 to 6 starting from task 3 are used as the remaining tasks to update the pipeline configuration file. It should be noted that the above example is only for a more rigorous explanation of how to determine the remaining tasks, and the possibility of the situation in this example occurring is very small, because when the pipeline is modified for task 3 that failed to execute, it usually affects task 3 and the tasks before it, resulting in the importance scores of these tasks being usually higher.

[0083] Step 25, executing the remaining tasks in the pipeline according to the updated pipeline configuration file and the saved pipeline status.

[0084] Here, when executing the first remaining task in the pipeline configuration file, it is necessary to obtain the output of the previous task of the first remaining task in the pipeline according to the saved pipeline state, and use the output as the input of the first remaining task to execute the first remaining task. For subsequent remaining tasks, the output of the previous task of the subsequent remaining task is used as the input of the subsequent remaining task, and each subsequent remaining task is executed in sequence.

[0085] Through the above steps, after completing the pipeline modification, the embodiment of the present invention can determine the remaining tasks that need to be executed according to the importance score of the tasks, and can avoid executing all tasks from the beginning in some cases. Since the tasks that need to be executed are reduced, unnecessary tasks can be skipped when the pipeline is restarted, and important tasks can be executed directly. While ensuring the successful delivery of the application, the time cost of the continuous integration pipeline execution is greatly reduced, system resources are saved, and the execution efficiency of the pipeline is improved. In addition, the state preservation technology is also used in the embodiment of the present invention to ensure the correct acquisition of task input when the pipeline is restarted.

[0086] As an implementation method, in the above step 22, when the current task fails to execute and the pipeline state corresponding to the current task is saved, an embodiment of the present invention can combine the name of the pipeline, the timestamp when the current task fails to execute, and the name of the current task to generate a pipeline state name corresponding to the current task, and obtain the state of all tasks before the current task; then, the key-value pair with the keyword (key) being the pipeline state name and the value (value) being the pipeline state corresponding to the current task is saved in the database of the state machine container.

[0087] In the above step 25, when executing the remaining tasks in the pipeline according to the updated pipeline configuration file and the saved pipeline state, the pipeline state corresponding to the pipeline state name can be obtained from the database of the state machine container according to the pipeline state name; according to the obtained pipeline state, the remaining tasks in the pipeline are executed, for example, from the obtained pipeline state, the output of the previous task in the pipeline of the first remaining task in the updated pipeline configuration file is obtained, and the output is used as the input of the first remaining task to execute the first remaining task.

[0088] Optionally, in the embodiment of the present invention, when the pipeline configuration file is updated in the above step 24, the pipeline state name may be used as the file name of the pipeline configuration file. In this way, when the pipeline state corresponding to the pipeline state name is obtained from the database of the state machine container according to the pipeline state name, the database of the state machine container is queried according to the file name of the pipeline configuration file to obtain the pipeline state corresponding to the file name of the pipeline configuration file.

[0089] When executing the remaining tasks, execution failure may still occur. In the process of sequentially executing each remaining task in the pipeline, if any task fails to execute, the embodiment of the present invention returns to the above step 22, thereby looping the above steps until all tasks are successfully executed.

[0090] Optionally, after all tasks in the pipeline are successfully executed, the embodiment of the present invention may also publish or deploy the target application.

[0091] To help better understand the above embodiments, the present invention is described below through a more specific example.

[0092] In this example, assume that Sentimental Analysis is an application that the user needs to deliver. This is an application involving machine learning, and a distributed Tensorflow container cluster needs to be deployed on the Kubernetes platform to process this application. In order to automatically deploy (deliver) the sentiment analysis application, a continuous integration framework is deployed on the Kubernetes platform and a pipeline is configured. This pipeline includes four tasks that are executed in sequence, namely: 1) code packaging and image generation tasks; 2) model training tasks; 3) model output tasks; 4) model deployment and prediction tasks.

[0093] For these four tasks, the code files are saved in four folders of the file version management server (for example: http: / / 192.168.200.138:31587 / NLP build / , train / , export / , serving / ), and the configuration parameters of each task are saved in the configuration files of each task. For example, the build task parameters include: code server URL address, mirror server URL address, mirror version; train task parameters: application model training execution file address, application model training mirror URL address, mirror version; output task parameters: model address, model version, model output mirror URL address, mirror version; application model service parameters: model service execution file address, model service mirror URL address, mirror version, model address, model name) Continuous integration pipeline optimization technology is used to dynamically and continuously monitor the running status of each task in the pipeline, and intelligently rate the pipeline tasks according to the running status and quickly restore the pipeline execution, which improves the pipeline execution performance while automating the delivery of applications.

[0094] After the code development of the sentiment analysis application is completed, configure the pipeline file to execute all the tasks of the pipeline. The optimization system will execute this file on the Kubernetes platform to start the pipeline corresponding to the application. Figure 3 , specifically explains the process of preserving the state of failed tasks on the machine during the execution of the pipeline.

[0095] S1, the user configures the sentiment analysis application pipeline task execution file and provides it to the task running module, including configuring four subtasks and their execution order. The parameters include: the name, version and URL of the task running image, the name of the model, the version of the model, etc.

[0096] S2, executes the sentiment analysis pipeline file on the Kubernetes platform to start the pipeline on the continuous integration framework.

[0097] S3, processes each task of the pipeline in sequence according to the configuration of the sentiment analysis application pipeline.

[0098] S4-S5, real-time monitoring of sentiment analysis application pipeline operation logs to obtain the running status of the current task.

[0099] S6, determine whether all tasks of the pipeline are successfully completed. If all tasks of the pipeline are successfully completed, go to step S13. If a task fails to execute, go to S7.

[0100] S7, if a task in the pipeline (e.g., model deployment and prediction tasks) fails, obtain the timestamp TS of the pipeline failure (e.g., 20200713143715001), the pipeline name P_name (e.g., NLP_SA_Pipeline_001), the failed task name T_name (e.g., Serving_Task_001), and combine the pipeline name P_name, task name T_name and error occurrence timestamp TS as the pipeline status name P_status_name (e.g., NLP_SA_Pipeline_001Serving_Task_00120200713143715001).

[0101] In addition, get the status T_status of all tasks before the failed task, including status value, output file name and address (for example: Build_Task_001: completed, SA_image, http: / / harbor.repository.com / library / SA_image:1.0; Train_Task_001: completed, SA_model, / home / CI / SA / v1 / ; Export_Task_001: completed, SA_model_artificate, http: / / modeldb.com / SA / 20200713143715001 / SA_model_artificate)

[0102] Then, connect to the state machine container, use the pipeline current state name P_status_name as the key, and the status of all tasks T_status as the value, and save them to the Redis database of the state machine container.

[0103] Through the above steps, this example completes the task status preservation work of the failed pipeline.

[0104] The following describes the specific processing flow of grading the importance of all tasks after the developer has handled the problem corresponding to the failed pipeline error (S8).

[0105] S9, for applications in the field of machine learning, the optimization system uses the following five pipeline task importance evaluation indicators:

[0106] 1) The number of source code files modified for each task;

[0107] 2) The number of configuration parameters modified for each task;

[0108] 3) The number of containers that need to be started for each task;

[0109] 4) Whether each task interacts with the Kubernetes cluster (e.g., 1 if there is interaction, 0 otherwise);

[0110] 5) Whether each task uses GPU resources (1 if using GPU, 0 otherwise).

[0111] For each task, first, the initial weight ratios are set for the above five importance evaluation indicators, which are w1 to w5. The principle of setting is that the greater the impact on the importance of the task, the higher the weight. If the modification of the source code file will affect the model quality and prediction accuracy, the task usually needs to be re-executed, and w1 has the highest weight; followed by w2 corresponding to the number of modifications to the task configuration parameters, which does not affect the model quality, but will affect the successful execution of the pipeline task; and finally, w3 corresponding to the number of containers to be started for the task has an impact on the high availability of the task. w4 corresponding to whether to interact with the Kubernetes cluster and w5 corresponding to whether to use GPU resources only affect the execution efficiency and have little effect on the success of the task, so the weight is the lowest (for example: w1 = 100, w2 = 50, w3 = 20, w4 = 10, w5 = 5). The weight values ​​of the importance indicators are set according to this distribution principle and written into the importance indicator configuration module.

[0112] After the developer completes the corresponding processing of the problems that occur in the failed pipeline (such as modifying the source code file code, configuring parameters, etc.), the optimization system obtains the index value of the importance evaluation index of each task of the sentiment analysis application pipeline through the server file modification log and the cluster hardware list (for example: the image generation task has no code parameter changes, the number of containers to be started is 2, without interaction and GPU; the model training task has no code parameter modification, the number of containers to be started is 3, with interaction and GPU use; the model output task has 2 source code modification files, 1 parameter modification number, 2 containers to be started, no interaction and GPU; the model deployment and prediction task has 3 file modification numbers, no parameter modification, 3 containers to be started, interaction, and no GPU).

[0113] Then, we get the weight values ​​of the five indicators from the importance indicator configuration module. For each task in the pipeline, we multiply the source file weight value by the number of modified files, the parameter weight value by the number of modified parameters, the container weight value by the number of started containers, the Kubernetes cluster interaction weight value by the interaction status value, and the GPU weight value by the GPU usage status value. Then, we add up the products of these five (modified files, embryo parameters, containers, interaction status, and GPU) to get the importance score of each task.

[0114] Example: Model packaging task score = 100*0+50*0+20*2+10*0+5*0=40;

[0115] Model training task score = 100*0+50*0+20*3+10*1+5*1=75;

[0116] Model output task score = 100*2+50*1+20*2+10*0+5*0=290.

[0117] Then, the importance of each task is evaluated based on its score. For example, a score over 100 is an important task and must be executed again, while a score below 100 is a general task and does not need to be executed again.

[0118] Through the above steps, the importance scoring process of all tasks in the sentiment analysis application pipeline is completed.

[0119] The following describes the specific processing flow of reading and re-executing the saved pipeline state.

[0120] S10, obtain the importance score of each task, update the pipeline configuration file, and update the classification of each task in the pipeline. (For example: the model output task score is 290, which is an important task) and use the pipeline current status name P_status_name in 5) as the new file name of the pipeline configuration file. Please refer to the previous description for the specific tasks that need to be executed in the pipeline configuration file, which will not be repeated here.

[0121] S11, connect the state machine container, use the updated pipeline configuration file name as key, obtain the status T_status of all tasks before the failed task corresponding to this pipeline, including the task status value, task output file name and address. The output of the previous task of the pipeline is the input of the next task, read the updated pipeline configuration file to determine the first important task, and use this task name as key to obtain the task input from T_status (for example: in the updated pipeline configuration file, the model output task is the first important task, and the output of the previous task of the model output task, i.e. the training task, Train_Task_001:completed, SA_model, / home / CI / SA / v1 / is the input of this task).

[0122] Then, determine the pipeline input and execute the updated pipeline configuration file on the Kubernetes platform.

[0123] S12, monitor the sentiment analysis application pipeline operation log in real time, obtain the running status of the current task, and determine whether all tasks of the pipeline are successfully completed. If all tasks of the pipeline are successfully completed, enter step S13. If a task fails to execute, return to S7.

[0124] S13, after all tasks are successfully executed, customers can access the deployed sentiment analysis application for prediction / analysis services.

[0125] In this example, users only need to configure the sentiment analysis application pipeline execution file. The continuous integration framework uses the pipeline optimization system to skip unnecessary tasks when the pipeline is restarted, and directly executes from important tasks so that the product can be delivered to customers more quickly. In this example, the state preservation technology is used to ensure the correct acquisition of task input when the pipeline is restarted. The task importance rating algorithm intelligently monitors and allocates the task execution order of the pipeline by distinguishing between general tasks and important tasks in the pipeline. While ensuring the successful delivery of the application, it also greatly reduces the time cost of the continuous integration pipeline execution and saves system resources.

[0126] Various methods of the embodiments of the present invention are introduced above, and a system for implementing the above methods is further provided below.

[0127] Please refer to Figure 4 , an optimization system 40 for a continuous integration pipeline provided by an embodiment of the present invention includes:

[0128] The pipeline operation module 41 is used to execute each task of the pipeline in sequence according to the pipeline configuration file of the target application;

[0129] The state management module 42 is used to save the pipeline state corresponding to the current task when the current task fails to execute, and the pipeline state includes the states of all tasks before the current task;

[0130] The importance rating module 43 is used to calculate the importance score of each task according to the importance index and corresponding weight of each task after the pipeline modification is completed; determine the remaining tasks that need to be continued in the pipeline according to the importance score, and update the pipeline configuration file;

[0131] The pipeline running module 41 is further used to execute the remaining tasks in the pipeline according to the updated pipeline configuration file and the saved pipeline status.

[0132] Through the above modules, the optimization system of the continuous integration pipeline of the embodiment of the present invention can skip unnecessary tasks when the pipeline is started again and directly execute from important tasks. While ensuring the successful delivery of the application, it also greatly reduces the time cost of the continuous integration pipeline execution, saves system resources, and improves the execution efficiency of the pipeline.

[0133] Optionally, the state management module is also used to trigger the state management module to save the pipeline state corresponding to the current task if any task fails to execute during the process of sequentially executing each remaining task in the pipeline.

[0134] Optionally, the optimization system further includes the following modules (not shown in the figure):

[0135] A deployment module is used to publish or deploy the target application after all tasks in the pipeline are successfully executed.

[0136] Optionally, the importance index of each task includes all of the following indicators:

[0137] The number of files modified in the source code files related to the task;

[0138] The number of modified parameters among the external parameters related to the task;

[0139] The number of containers that need to be started for the task;

[0140] whether the task interacts with the optimization platform of the continuous integration pipeline;

[0141] Whether the task uses graphics processor GPU resources.

[0142] Optionally, the importance rating module is also used to select important tasks whose importance scores exceed a preset threshold value, and determine the first task with the highest execution order in the pipeline from the important tasks; from the first and second tasks, select a target task with the highest execution order in the pipeline, and use all tasks in the pipeline starting from the target task as the remaining tasks that need to be continued in the pipeline, wherein the second task is the task that failed to execute before the pipeline was modified.

[0143] Optionally, the state management module is also used to combine the name of the pipeline, the timestamp when the current task fails to execute, and the name of the current task when saving the pipeline state corresponding to the current task, to generate the pipeline state name corresponding to the current task, and obtain the status of all tasks before the current task; the key is the pipeline state name, and the value is the key-value pair of the pipeline state corresponding to the current task, and is saved in the database of the state machine container.

[0144] Optionally, the pipeline running module is also used to obtain the pipeline state corresponding to the pipeline state name from the database of the state machine container according to the pipeline state name when executing the remaining tasks in the pipeline according to the updated pipeline configuration file and the saved pipeline state; and execute the remaining tasks in the pipeline according to the obtained pipeline state.

[0145] Optionally, the importance rating module is further used to further use the pipeline state name as the file name of the pipeline configuration file when updating the pipeline configuration file;

[0146] The pipeline operation module is also used to query the database of the state machine container according to the file name of the pipeline configuration file to obtain the pipeline state corresponding to the file name of the pipeline configuration file when obtaining the pipeline state corresponding to the pipeline state name from the database of the state machine container according to the pipeline state name.

[0147] like Figure 5 As shown, the embodiment of the present invention also provides another continuous integration pipeline optimization system 50, which specifically includes a processor 51, a memory 52, a bus system 53, a receiver 54 and a transmitter 55. The processor 51, the memory 52, the receiver 54 and the transmitter 55 are connected via the bus system 53, the memory 52 is used to store instructions, and the processor 51 is used to execute the instructions stored in the memory 52 to control the receiver 54 to receive signals and control the transmitter 55 to send signals;

[0148] The processor 51 is used to read the program in the memory and execute the following process:

[0149] According to the pipeline configuration file of the target application, execute each task of the pipeline in sequence;

[0150] When the current task fails to execute, the pipeline state corresponding to the current task is saved, and the pipeline state includes the states of all tasks before the current task;

[0151] After the pipeline modification is completed, the importance score of each task is calculated according to the importance index and corresponding weight of each task;

[0152] Determine the remaining tasks that need to be continued to be executed in the pipeline according to the importance score, and update the pipeline configuration file;

[0153] Execute the remaining tasks in the pipeline according to the updated pipeline configuration file and the saved pipeline state.

[0154] It should be understood that in the embodiment of the present invention, the processor 51 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0155] The memory 52 may include a read-only memory and a random access memory, and provides instructions and data to the processor 51. A portion of the memory 52 may also include a non-volatile random access memory. For example, the memory 52 may also store information on the device type.

[0156] The bus system 53 includes not only a data bus but also a power bus, a control bus, a status signal bus, etc. However, for the sake of clarity, various buses are labeled as the bus system 53 in the figure.

[0157] In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 51 or the instruction in the form of software. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as the execution of the hardware processor, or the execution is completed by the combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 52, and the processor 51 reads the information in the memory 52 and completes the steps of the above method in combination with its hardware. To avoid repetition, it is not described in detail here.

[0158] When the program is executed by the processor, it can implement all the implementation methods of the continuous integration pipeline optimization method in the above method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0159] In some embodiments of the present invention, a computer-readable storage medium is further provided, on which a program is stored, and when the program is executed by a processor, the following steps are implemented:

[0160] According to the pipeline configuration file of the target application, execute each task of the pipeline in sequence;

[0161] When the current task fails to execute, the pipeline state corresponding to the current task is saved, and the pipeline state includes the states of all tasks before the current task;

[0162] After the pipeline modification is completed, the importance score of each task is calculated according to the importance index and corresponding weight of each task;

[0163] Determine the remaining tasks that need to be continued to be executed in the pipeline according to the importance score, and update the pipeline configuration file;

[0164] Execute the remaining tasks in the pipeline according to the updated pipeline configuration file and the saved pipeline state.

[0165] When the program is executed by the processor, all implementation methods in the above-mentioned optimization method for the continuous integration pipeline applied to the wireless access point can be implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0166] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0167] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0168] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0169] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.

[0170] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0171] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0172] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for optimizing a continuous integration pipeline, characterized in that: include: According to the pipeline configuration file of the target application, execute each task of the pipeline in sequence; When the current task fails to execute, the pipeline state corresponding to the current task is saved, and the pipeline state includes the states of all tasks before the current task; After the pipeline modification is completed, the importance score of each task is calculated according to the importance index and corresponding weight of each task; Determine the remaining tasks that need to be continued to be executed in the pipeline according to the importance score, and update the pipeline configuration file; Execute the remaining tasks in the pipeline according to the updated pipeline configuration file and the saved pipeline state; The importance index of the task includes all the following indexes: The number of files modified in the source code files related to the task; The number of modified parameters among the external parameters related to the task; The number of containers that need to be started for the task; whether the task interacts with the optimization platform of the continuous integration pipeline; Whether the task uses graphics processor GPU resources; The step of determining the remaining tasks that need to be continued in the pipeline according to the importance score includes: Select the important tasks whose importance scores exceed a preset threshold value, and determine the first task with the highest execution order in the pipeline from the important tasks; From the first task and the second task, a target task that is executed earlier in the pipeline is selected, and all tasks in the pipeline starting from the target task are used as remaining tasks that need to be continued to be executed in the pipeline, wherein the second task is a task that failed to execute before the pipeline was modified.

2. The optimization method according to claim 1, characterized in that: In the process of sequentially executing each remaining task in the pipeline, if any task fails to execute, the process returns to the step of saving the pipeline state corresponding to the current task when the current task fails to execute.

3. The optimization method according to claim 1 or 2, characterized in that: Also includes: After all tasks in the pipeline are successfully executed, the target application is released or deployed.

4. The optimization method according to claim 1, characterized in that: When the current task fails to execute, the pipeline state corresponding to the current task is saved, including: Combine the name of the pipeline, the timestamp when the current task fails to execute, and the name of the current task to generate a pipeline state name corresponding to the current task, and obtain the states of all tasks before the current task; The keyword key is the pipeline state name, and the value value is the key-value pair of the pipeline state corresponding to the current task, which is saved in the database of the state machine container.

5. The optimization method according to claim 4, characterized in that: The executing the remaining tasks in the pipeline according to the updated pipeline configuration file and the saved pipeline state includes: According to the pipeline state name, obtaining the pipeline state corresponding to the pipeline state name from the database of the state machine container; According to the obtained pipeline status, the remaining tasks in the pipeline are executed.

6. The optimization method according to claim 5, characterized in that: When updating the pipeline configuration file, further using the pipeline state name as the file name of the pipeline configuration file; When obtaining the pipeline state corresponding to the pipeline state name from the database of the state machine container according to the pipeline state name, query the database of the state machine container according to the file name of the pipeline configuration file to obtain the pipeline state corresponding to the file name of the pipeline configuration file.

7. A continuous integration pipeline optimization system, characterized in that: include: A pipeline operation module, used to execute each task of the pipeline in sequence according to the pipeline configuration file of the target application; A state management module, used to save the pipeline state corresponding to the current task when the current task fails to execute, and the pipeline state includes the states of all tasks before the current task; An importance rating module, used to calculate the importance score of each task according to the importance index and corresponding weight of each task after the pipeline modification is completed; determine the remaining tasks that need to be continued in the pipeline according to the importance score, and update the pipeline configuration file; The pipeline operation module is further used to execute the remaining tasks in the pipeline according to the updated pipeline configuration file and the saved pipeline status; Among them, the importance index of each task includes all the following indicators: The number of files modified in the source code files related to the task; The number of modified parameters among the external parameters related to the task; The number of containers that need to be started for the task; whether the task interacts with the optimization platform of the continuous integration pipeline; Whether the task uses graphics processor GPU resources; Among them, the importance rating module is also used to select important tasks whose importance scores exceed a preset threshold value, and determine the first task with the highest execution order in the pipeline from the important tasks; from the first and second tasks, select the target task with the highest execution order in the pipeline, and use all tasks in the pipeline starting from the target task as the remaining tasks that need to be continued in the pipeline, wherein the second task is the task that failed to execute before the pipeline was modified.

8. The optimization system according to claim 7, characterized in that: The state management module is also used to trigger the state management module to save the pipeline state corresponding to the current task if any task fails to execute during the process of sequentially executing each remaining task in the pipeline.

Images