A method, apparatus, device and medium for processing pipeline data

Abstract

The present application provides a kind of processing method, device, equipment and medium of pipelined data, the processing method includes: obtaining template pipeline, template pipeline characterizes the dependency of each execution link in project pipeline;According to template pipeline, each execution link is sorted and handled to generate the task instruction of project pipeline;Task instruction is executed, and the resource value base and running time data of each execution link are obtained;Based on iterative incremental value, resource value base is improved, and each execution link is iteratively run to generate the minimum running time of running time data;And based on iterative decrement value and minimum running time, resource value base is reduced, and each execution link is iteratively run to generate optimal resource value, optimal resource value represents the minimum value of resource occupied by execution link under minimum running time.The present application can reasonably allocate node resources, improve pipelined running efficiency.

Description

Technical Field

[0001] This invention relates to the field of software development technology, and specifically to a method, apparatus, equipment, and medium for processing pipeline data. Background Technology

[0002] The software development and testing industry typically uses or bases its solutions on Jenkins to build CI (Continuous Integration) / CD (Continuous Delivery / Continuous Deployment) workflows for software projects. Current Jenkins Master-Slave deployment methods involve creating jobs uniformly on the Master node, which then manages Slave nodes and distributes job tasks to each node for execution, thus providing concurrent task capacity. Leveraging container technology, the Master node publishes continuous integration tasks, flexibly distributing multiple tasks from a pipeline to a single Slave container for processing, ensuring high availability of Slave container resources.

[0003] However, a single Master instance can only handle a limited number of pipeline tasks, and resource contention can easily occur under multi-task requests, leading to increasingly poor performance and slow build speed. In the case of multiple Master instances, the resource distribution method does not take into account the task status and performance of the Jenkins service nodes, which can easily lead to resource waste. Summary of the Invention

[0004] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a method, apparatus, device and medium for processing pipeline data, which can solve the problem of wasted Jenkins Master resources in the prior art and ensure high concurrency and high availability in multi-task situations in automated operation and maintenance CI / CD processes.

[0005] To achieve the above and other related objectives, the present invention provides a method for processing pipeline data, comprising:

[0006] A method for processing pipeline data, characterized in that it includes:

[0007] Obtain a template pipeline, which represents the dependencies of each execution stage in the project pipeline;

[0008] According to the template pipeline, each execution step is sorted to generate the task instructions for the project pipeline;

[0009] Execute the task instructions and obtain the resource value base and running time data for each execution stage;

[0010] Based on an iterative incrementing value, the base value of the resource is increased, and each execution stage is run iteratively to generate the minimum execution time for the execution time data; and

[0011] Based on the iterative decrease value and the minimum running time, the base value of the resource value is reduced, and each execution step is run iteratively to generate the optimal resource value, which represents the minimum value of resources occupied by the execution step under the minimum running time.

[0012] In one embodiment of the present invention, the step of sorting each execution stage according to the template pipeline to generate task instructions for the project pipeline includes:

[0013] Responding to the user's selection action, obtain the execution stage to be run; and

[0014] Based on the dependencies in the template pipeline, the execution stages to be run are automatically orchestrated to generate task instructions.

[0015] In one embodiment of the present invention, the step of executing the task instruction and obtaining the resource value base and running time data of each execution stage includes:

[0016] Each execution stage is tagged, with one tag corresponding to each project pipeline; and

[0017] Based on the tags, the resource value base and the running time data for each execution stage are monitored and stored.

[0018] In one embodiment of the present invention, the step of increasing the resource value base based on iterative increments and iteratively running each of the execution stages to generate the minimum execution time of the execution time data includes:

[0019] Based on the iterative increment value, the base value of the resource is increased, each execution stage is run iteratively, and intermediate time consumption data after iterative running is obtained; and

[0020] When the intermediate time data of two consecutive iterations are equal, the intermediate time data corresponding to the current iteration is stored as the minimum running time.

[0021] In one embodiment of the present invention, the step of reducing the base resource value based on the iterative decrease value and the minimum running time, and iteratively running each execution stage to generate an optimal resource value, wherein the optimal resource value represents the minimum resource occupied by the execution stage under the minimum running time, includes:

[0022] Obtain the base resource value corresponding to the minimum running time;

[0023] Based on the iterative decrease value, the base value of the resource value is reduced, and each execution stage is run iteratively, obtaining the intermediate time consumption data after the iterative run; and

[0024] If the intermediate execution time of the current iteration is greater than the minimum execution time, the resource value occupied by the previous iteration will be stored as the optimal resource value.

[0025] In one embodiment of the present invention, in the step of executing the task instruction and obtaining the resource value base and running time data of each execution stage:

[0026] The execution steps that have no parent node dependency are executed concurrently in the project pipeline.

[0027] In one embodiment of the present invention, in the step of sorting each execution stage according to the template pipeline to generate task instructions for the project pipeline:

[0028] The execution phase includes code retrieval, compilation, static scanning, unit testing, incremental code inspection, quality gate inspection, image building, and / or artifact delivery.

[0029] The present invention also provides a pipeline data processing apparatus, characterized in that it comprises:

[0030] The template acquisition module is used to acquire template pipelines, which represent the dependencies of each execution stage in the project pipeline;

[0031] The task generation module is used to sort each execution step according to the template pipeline to generate task instructions for the project pipeline.

[0032] The data statistics module is used to execute the task instructions and obtain the resource value base and running time data for each execution stage;

[0033] The resource adjustment module is used to increase the base value of the resource based on an iterative increment, and iteratively run each of the execution stages to generate the minimum execution time of the execution time data; and

[0034] The optimal value generation module is used to reduce the base value of the resource value based on the iterative decrease value and the minimum running time, and iteratively run each of the execution links to generate the optimal resource value, wherein the optimal resource value represents the minimum value of resources occupied by the execution link under the minimum running time.

[0035] The present invention also provides an electronic device, the electronic device comprising:

[0036] One or more processors; and

[0037] A storage device for storing one or more programs that, when executed by one or more processors, enable an electronic device to perform pipelined data processing methods as described above.

[0038] The present invention also provides a computer-readable storage medium, characterized in that it stores a computer program thereon, which, when executed by a computer processor, causes the computer to perform any of the above-described pipeline data processing methods.

[0039] As described above, the method, apparatus, device, and medium for processing pipeline data according to the present invention have the following beneficial effects:

[0040] (1) The present invention can improve the efficiency of the pipeline during triggering execution through a reasonable tree-like pipeline structure;

[0041] (2) This invention can monitor and collect data from running nodes, analyze the data, improve the utilization rate of the Jenkins-slave resource pool, and maximize the efficiency of each project in each execution stage. Attached Figure Description

[0042] Figure 1 This illustrates the application environment of a pipeline data processing method provided by the present invention.

[0043] Figure 2 The diagram shown is an exemplary flowchart of a pipeline data processing method according to the present invention.

[0044] Figure 3 The diagram shown is a schematic representation of the template production line in an exemplary embodiment of the present invention.

[0045] Figure 4 The diagram shown is an example of data analysis in this invention;

[0046] Figure 5 Displayed as Figure 2 A flowchart illustrating a specific implementation of step S202;

[0047] Figure 6 Displayed as Figure 2 A schematic diagram of a specific implementation method for step S203;

[0048] Figure 7 Displayed as Figure 2 A flowchart illustrating a specific implementation of step S204;

[0049] Figure 8 Displayed as Figure 2 A schematic diagram of a specific implementation method for step S205;

[0050] Figure 9 The diagram shown is a block diagram of a pipeline data processing apparatus as an example of the present invention.

[0051] Figure 10 A schematic diagram of a computer system suitable for implementing embodiments of the present invention is shown. Detailed Implementation

[0052] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0053] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0054] First, it's important to note that Jenkins is an open-source web platform developed in Java, used to support continuous integration and continuous deployment of software / testing projects. Jenkins is highly scalable and is typically used in conjunction with version control (Software Configuration Management, SCM) and build tools. Because of its distributed nature, Jenkins allows multiple computers to build / test simultaneously, so most existing technical solutions leverage this feature to execute jobs on a large scale, thereby improving CI / CD process efficiency.

[0055] Please see Figures 1 to 10 This invention provides a method, apparatus, device, and medium for processing pipeline data, relating to the field of software development, and specifically applicable to CICD (Continuous Build, Continuous Integration, Continuous Delivery) processes. This invention optimizes the conventional sequential execution of each project at each stage through a tree-like structured management approach. By monitoring data collection and analysis, it intelligently allocates the most efficient resource solutions for each project in the pipeline at each execution stage.

[0056] Please see Figure 1 As shown, the present invention can be applied to, for example... Figure 1 In this application environment, client 101 can communicate with server 102 via a network. Server 102 stores automated scripts. Client 101 can use the automated scripts provided by server 102 to monitor and collect data from Jenkins-slave running nodes, analyze the data, and achieve reasonable scheduling to maximize pipeline efficiency.

[0057] Please see Figure 2 As shown, Figure 2 This is a schematic flowchart illustrating an exemplary embodiment of the present invention of a pipeline data processing method, which may include the following steps:

[0058] Step S201: Obtain the template pipeline. The template pipeline represents the dependencies of each execution stage in the project pipeline.

[0059] Step S202: Based on the template pipeline, sort each execution stage to generate task instructions for the project pipeline;

[0060] Step S203: Execute the task instruction and obtain the resource value base and running time data for each execution stage;

[0061] Step S204: Based on the iterative increment value, increase the resource value base and iteratively run each execution stage to generate the minimum execution time for the execution time data; and

[0062] Step S205: Based on the iterative decrease value and the minimum running time, reduce the base value of the resource value and iteratively run each execution stage to generate the optimal resource value. The optimal resource value represents the minimum value of resources occupied by the execution stage under the minimum running time.

[0063] In one embodiment of the present invention, when step S201 is executed, a template pipeline is obtained. The template pipeline represents the dependencies of each execution stage in the project pipeline. Specifically, a tree-like parallel pipeline can be pre-established for each execution stage of the project pipeline. Parallel pipelines differ from traditional pipelines in that they have multiple parallel and serial nodes simultaneously. The template pipeline clearly describes the root node of each execution stage in the project pipeline, as well as the parent and child nodes that depend on the root node.

[0064] Please see Figure 3As shown, in one embodiment of the present invention, the parallel pipeline may include eight execution stages: a code fetch stage, an execution compilation stage, a static scanning stage, a unit testing stage, an incremental code inspection stage, a quality gate check stage, an image building stage, and an artifact push stage. The code fetch stage is the root node, and its child nodes include the execution compilation stage, the incremental code inspection stage, and the static scanning stage. The execution compilation stage's child node is the unit testing stage; the static scanning stage's child node is the quality gate check stage; and the image building stage's child node is the artifact push stage, which is a common child node of the unit testing stage, the incremental code inspection stage, and the quality gate check stage.

[0065] In one embodiment of the present invention, the pipeline is a visualized automated task scheduling platform that needs to be used in conjunction with automated tasks in the software development production line, such as compilation and building, code inspection, test planning, and deployment. Depending on the user's needs, such as application deployment in development and testing environments or production environments, these automated tasks can be customized and orchestrated. Once configured, they can be automatically triggered and executed with a single click, avoiding frequent and inefficient manual operations.

[0066] In one embodiment of the present invention, when step S202 is executed, each execution stage is sorted according to the template pipeline to generate task instructions for the project pipeline. Specifically, after the business personnel select the stages to be executed in the current project pipeline, the stages can be automatically arranged according to the tree-like structure information of the template pipeline to generate task instructions. It should be noted that nodes without parent node dependencies in the task instructions can be executed concurrently. The present invention can effectively improve execution efficiency by continuously optimizing and maintaining the information within the tree-like structure of the parallel pipeline.

[0067] In one embodiment of the present invention, when step S203 is executed, a task instruction is executed, and the resource value base and running time data for each execution stage are obtained. Specifically, when each project pipeline runs for the first time, a default resource value can be set to be allocated as the resource value base for each execution stage. Based on this resource value base, the task instruction generated in step S202 is executed, thereby obtaining the running time data for each execution stage.

[0068] In one embodiment of the present invention, different project pipelines can be tagged, with each project pipeline corresponding to a single tag. Then, through a container monitoring tool, the actual resource consumption data and running time data of each project during each execution stage are stored in the database.

[0069] Please see Figure 4As shown, in one embodiment of the present invention, when step S204 is executed, the resource value base is increased based on the iterative increment value, and each execution stage is run iteratively to generate the minimum running time for the running time data. Specifically, through the label settings in step S203, resource consumption data analysis is performed on the corresponding data under each label for a single execution stage in a single project pipeline. The resources consumed by the execution stage in the previous run or the resources allocated by the Jenkins system by default are obtained and used as the resource value base. Then, this resource value base is increased according to the iterative increment value. For example, the resource value base in the previous run is increased by 5% as the resource value base for the current run of this execution stage. This project pipeline runs based on the increased resource value base and records the running time data after the increased resource value base. Then compare the time taken in the previous run with the time taken in the current run; if the time taken in the previous run is greater than the time taken in the current run, continue to increase the allocated resources to serve as the base for the resource value of the next execution stage; if the time taken in the previous run is equal to the time taken in the current run, store the time taken in the current iteration as the minimum running time.

[0070] In one embodiment of the present invention, when step S205 is executed, the resource value base is reduced based on the iterative decrease value and the minimum running time, and each execution step is run iteratively to generate the optimal resource value. The optimal resource value represents the minimum resource occupied by the execution step under the minimum running time. Specifically, the shortest running time data can be associated with multiple resource value bases, wherein the optimal resource value represents the minimum resource occupied by the execution step under the minimum running time, that is, the one with the smallest resource value base is the optimal resource value of that execution step. According to the minimum running time and the iterative decrease value, the resource value base is reduced and each execution step is run iteratively. For example, the resources allocated to the execution step are reduced by 10%, which is used as the resource value base for the next execution step. The iterative operation is performed according to the above steps. When the intermediate running time data of the current iteration is greater than the minimum running time, the resource value occupied by the previous iteration is stored as the optimal resource value.

[0071] Through the aforementioned iterative comparison steps, optimal resource values ​​for each execution stage of different project pipelines can be generated. This data is saved for subsequent resource allocation during runtime. This enables the monitoring, collection, and analysis of resource consumption data for each pipeline stage under each project, thereby obtaining intelligently customized resource requirements for each pipeline stage under each project. This maximizes resource utilization and ensures that pipeline efficiency is not limited by unreasonable resource allocation, thus achieving peak operational efficiency.

[0072] Please refer to Figure 5 As shown, Figure 5 yes Figure 2 A schematic flowchart of a specific implementation of step S202 may include the following steps:

[0073] Step S501: Respond to the user's selection operation and obtain the execution stage to be run;

[0074] Step S502: Based on the dependencies in the template pipeline, automatically orchestrate the execution stages to be run to generate task instructions.

[0075] In one embodiment of the present invention, when steps S501 to S502 are executed, specifically, the business personnel select the various stages to be executed in the current project pipeline through the pipeline front-end orchestrator. The back-end processor then parses the template pipeline to obtain tree-like structure information; and then automatically orchestrates the various stages based on the tree-like structure information to generate task instructions. In this embodiment, the template pipeline contains at least one execution stage, each execution stage having a different function. Furthermore, the execution order of each execution stage in the template pipeline has been pre-set.

[0076] Please refer to Figure 6 As shown, Figure 6 yes Figure 2 A schematic flowchart of a specific implementation of step S203 may include the following steps:

[0077] Step S601: Mark each execution stage, with one label corresponding to each project pipeline;

[0078] Step S602: Based on tags, monitor and store the resource value base and running time data of each execution stage.

[0079] In one embodiment of the present invention, when steps S601 to S602 are executed, specifically, different project pipelines are tagged, with each project pipeline corresponding to a single tag. Then, using a container monitoring tool, the actual resource consumption data and runtime data of each project during each execution stage are stored in a database. It should be noted that when each project pipeline runs for the first time, a default resource value can be set as the resource base for each execution stage. Based on the resource base value, task instructions are executed, and the runtime data for each execution stage is obtained and stored using the container monitoring tool.

[0080] Please refer to Figure 7 As shown, Figure 7 yes Figure 2 A schematic flowchart of a specific implementation of step S204 may include the following steps:

[0081] Step S701: Based on the iterative increment value, increase the base resource value, iteratively run each execution stage, and obtain the intermediate time consumption data after the iterative run; and

[0082] Step S702: When the intermediate time data of two adjacent iterations are equal, store the intermediate time data corresponding to the current iteration as the minimum running time.

[0083] In one embodiment of the present invention, when steps S701 to S702 are executed, specifically, through the label setting in step S203, resource consumption data analysis is performed on the corresponding data under each label for a single-project pipeline execution step. The resources consumed in the previous run or the default resources allocated by the Jenkins system are obtained and used as the resource value base. Then, this resource value base is increased according to the iterative increment. For example, the resource value base from the previous run is increased by 5% as the resource value base for the current run of this execution step. This project pipeline performs the current run based on the increased resource value base and records the runtime data after the increased resource value base. The runtime data from the previous run is then compared with the runtime data from the current run; if the runtime data from the previous run is greater than the runtime data from the current run, the allocated resources are further increased to serve as the resource value base for the next execution step; if the runtime data from the previous run is equal to the runtime data from the current run, the runtime data corresponding to the current iteration is stored as the minimum runtime.

[0084] Please refer to Figure 8 As shown, Figure 8 yes Figure 2 A schematic flowchart of a specific implementation of step S205 may include the following steps:

[0085] Step S801: Obtain the base resource value corresponding to the minimum running time;

[0086] Step S802: Based on the iterative decrease value, reduce the base value of the resource value, iteratively run each execution stage, and obtain the intermediate time data after the iterative run;

[0087] Step S803: When the intermediate time consumption data of this iteration is greater than the minimum running time, store the resource value occupied by the previous iteration as the optimal resource value.

[0088] In one embodiment of the present invention, when executing steps S801 to S803, specifically, the resource value base corresponding to the minimum running time in step S702 is obtained. Based on this resource value base, the resource value base is reduced according to the iterative decrease value. Each execution stage is iteratively run, and intermediate running time data is obtained after iterative running. If the intermediate running time data of this iteration is greater than the minimum running time, the resource value occupied by the previous iteration is stored as the optimal resource value. For example, the resource value base corresponding to the minimum running time in step S702 is reduced by 10% to be used as the resource allocated for the next run. Iterative running is performed accordingly, and intermediate running time data is obtained. If the intermediate running time data is equal to the minimum running time, the resource value base is further reduced. If the intermediate running time data is greater than the minimum running time, the resource value occupied by the previous iteration is stored as the optimal resource value. By repeatedly iterating in this way, the minimum value of resources occupied by each execution stage of each project under the minimum running time can be obtained.

[0089] Please see Figure 9 As shown, Figure 9 A block diagram of a pipeline data processing device shown in an exemplary embodiment of the present invention may include a template acquisition module 901, a task generation module 902, a data statistics module 903, a resource adjustment module 904, and an optimal value generation module 905. The functional modules are described in detail below.

[0090] The template acquisition module 901 is used to acquire the template pipeline, which represents the dependency relationship of each execution step in the project pipeline.

[0091] In one embodiment of the present invention, the template acquisition module 901 can be used to acquire a template pipeline, which represents the dependencies of each execution stage in the project pipeline. Specifically, the template acquisition module 901 can pre-establish a tree-like parallel pipeline for each execution stage of the project pipeline. The parallel pipeline differs from a traditional pipeline in that it has multiple parallel and serial nodes simultaneously. The template pipeline can clearly describe the root node of each execution stage in the project pipeline during execution, as well as the parent and child nodes that depend on the root node.

[0092] The task generation module 902 is used to sort each execution stage according to the template pipeline in order to generate task instructions for the project pipeline.

[0093] In one embodiment of the present invention, the task generation module 902 can be used to sort each execution stage according to the template pipeline to generate task instructions for the project pipeline. Specifically, after the business personnel select the various stages to be executed in the current project pipeline, the stages can be automatically arranged according to the tree-like structure information of the template pipeline, thereby generating task instructions. It should be noted that nodes without parent node dependencies in the task instructions can be executed concurrently. The present invention can effectively improve execution efficiency by continuously optimizing and maintaining the information within the tree-like structure of the parallel pipeline.

[0094] In one specific embodiment, the task generation module 902 may also be specifically used for:

[0095] Responding to the user's selection action, obtain the execution stage to be run; and

[0096] Based on the dependencies in the template pipeline, the execution stages to be run are automatically orchestrated to generate task instructions.

[0097] The data statistics module 903 is used to execute task instructions and obtain the resource value base and running time data for each execution stage.

[0098] In one embodiment of the present invention, the data statistics module 903 can be used to execute task instructions and obtain the resource value base and running time data for each execution stage. Specifically, when each project pipeline runs for the first time, a default resource value can be set to be allocated as the resource value base for each execution stage. Based on this resource value base, the task instructions generated by the task generation module 902 are executed, thereby obtaining the running time data for each execution stage.

[0099] In one embodiment of the present invention, different project pipelines can be tagged, with each project pipeline corresponding to a single tag. Then, through a container monitoring tool, the actual resource consumption data and running time data of each project during each execution stage are stored in the database.

[0100] In one specific embodiment, the data statistics module 903 may also be specifically used for:

[0101] Each execution stage is tagged, with one tag corresponding to each project pipeline; and

[0102] Based on tags, monitor and store the resource value base and runtime data for each execution stage.

[0103] The resource adjustment module 904 is used to increase the base value of resources based on iterative increments and iteratively run each execution stage to generate the minimum running time of the running time data.

[0104] In one embodiment of the present invention, the resource adjustment module 904 can be used to increase the resource value base based on an iterative increment, and iteratively run each execution stage to generate the minimum running time for the running time data. Specifically, through the label settings of the data statistics module 903, resource consumption data analysis is performed on the corresponding data under each label for a single execution stage in a single project pipeline. The resources consumed by the execution stage in the previous run or the resources allocated by the Jenkins system by default are obtained and used as the resource value base. Then, this resource value base is increased according to the iterative increment. For example, the resource value base in the previous run is increased by 5% as the resource value base for the current run of this execution stage. This project pipeline runs based on the increased resource value base and records the running time data after the increased resource value base. Then compare the time taken in the previous run with the time taken in the current run; if the time taken in the previous run is greater than the time taken in the current run, continue to increase the allocated resources to serve as the base for the resource value of the next execution stage; if the time taken in the previous run is equal to the time taken in the current run, store the time taken in the current iteration as the minimum running time.

[0105] In one specific embodiment, the resource adjustment module 904 may also be specifically used for:

[0106] Based on iterative increments, the resource value base is increased, each execution stage is run iteratively, and intermediate time consumption data is obtained after each iteration; and

[0107] When the intermediate time data of two consecutive iterations are equal, the intermediate time data corresponding to the current iteration is stored as the minimum running time.

[0108] The optimal value generation module 905 is used to reduce the base value of the resource value based on the iterative decrease value and the minimum running time, and iteratively run each execution stage to generate the optimal resource value. The optimal resource value represents the minimum value of resources occupied by the execution stage under the minimum running time.

[0109] In one embodiment of the present invention, the optimal value generation module 905 can be used to reduce the resource value base based on the iterative decrease value and the minimum running time, and iteratively run each execution step to generate an optimal resource value. The optimal resource value represents the minimum resource occupied by the execution step under the minimum running time. Specifically, the shortest running time data can be associated with multiple resource value bases, wherein the optimal resource value represents the minimum resource occupied by the execution step under the minimum running time, that is, the one with the smallest resource value base is the optimal resource value of that execution step. The resource value base is reduced according to the minimum running time and each execution step is iteratively run. For example, the resources allocated to the execution step are reduced by 10%, which is used as the resource value base for the next execution step. The iteration is performed according to the above steps. When the intermediate running time data of the current iteration is greater than the minimum running time, the resource value occupied by the previous iteration is stored as the optimal resource value.

[0110] In one specific embodiment, the optimal value generation module 905 may also be specifically used for:

[0111] Obtain the base resource value corresponding to the minimum running time;

[0112] Based on the iterative decreasing value, the resource value base is reduced, each execution stage is run iteratively, and intermediate time data after each iteration is obtained; and

[0113] If the intermediate execution time of this iteration is greater than the minimum execution time, the resource value occupied by the previous iteration will be stored as the optimal resource value.

[0114] It should be noted that the pipeline data processing apparatus and the pipeline data processing method provided in the above embodiments belong to the same concept. The specific ways in which each module and unit performs operations have been described in detail in the method embodiments and will not be repeated here. In practical applications, the pipeline data processing apparatus provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above. This is not a limitation here.

[0115] Embodiments of the present invention also provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device enables the pipelined data processing method provided in the above embodiments.

[0116] Figure 10 A schematic diagram of a computer system suitable for implementing embodiments of the present invention is shown. It should be noted that... Figure 10The computer system 1000 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0117] like Figure 10 As shown, the computer system 1000 includes a Central Processing Unit (CPU) 1001, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on programs stored in Read-Only Memory (ROM) 1002 or programs loaded from Storage Unit 10010 into Random Access Memory (RAM) 1003. The RAM 1003 also stores various programs and data required for system operation. The CPU 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An Input / Output (I / O) interface 1005 is also connected to the bus 1004.

[0118] The following components are connected to I / O interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to I / O interface 1005 as needed. Removable media 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1010 as needed so that computer programs read from them can be installed into storage section 1008 as needed.

[0119] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing computer programs for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1009, and / or installed from removable medium 1011. When the computer program is executed by central processing unit (CPU) 1001, it performs various functions defined in the system of the present invention.

[0120] It should be noted that the computer-readable medium shown in the embodiments of the present invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0121] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0122] The units described in the embodiments of the present invention can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0123] Another aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer's processor, causes the computer to perform the pipelined data processing method described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not incorporated into the electronic device.

[0124] Another aspect of the present invention provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the pipelined data processing method provided in the various embodiments described above.

[0125] In summary, this invention provides a method, apparatus, equipment, and medium for processing pipeline data. This invention monitors, collects, and analyzes resource consumption data during the operation of each pipeline step under each project, thereby obtaining intelligent customized resource requirements for each pipeline step under each project. This allows resources to be utilized to the maximum extent and ensures that the operating efficiency of the pipeline is not limited by unreasonable resource allocation, thus improving operating efficiency.

[0126] The above embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention.

Claims

1. A method for processing pipeline data, characterized in that, include: Obtain a template pipeline, which represents the dependencies of each execution stage in the project pipeline; According to the template pipeline, each execution step is sorted to generate the task instructions for the project pipeline; Execute the task instructions and obtain the resource value base and running time data for each execution stage through the container monitoring tool; Based on the iterative increment value, the base value of the resource is increased, each execution stage is run iteratively, and the intermediate time consumption data after the iterative run is obtained; And when the intermediate time data of two adjacent iterations are equal, the intermediate time data corresponding to the current iteration is stored as the minimum running time; as well as Based on the iterative decrease value and the minimum running time, the base value of the resource value is reduced, each execution stage is iteratively run, and intermediate running time data is obtained after the iterative run. And when the intermediate time consumption data of the current iteration is greater than the minimum running time, the resource value occupied by the previous iteration is stored as the optimal resource value, which represents the minimum value of resources occupied by the execution link under the minimum running time.

2. The method for processing pipeline data according to claim 1, characterized in that, The step of sorting each execution stage according to the template pipeline to generate task instructions for the project pipeline includes: In response to the user's selection, obtain the execution stage to be run; and Based on the dependencies in the template pipeline, the execution stages to be run are automatically orchestrated to generate task instructions.

3. The method for processing pipeline data according to claim 1, characterized in that, The steps of executing the task instruction and obtaining the resource value base and running time data for each execution stage include: Each execution stage is tagged, with one tag corresponding to each project pipeline; and Based on the tags, the resource value base and the running time data for each execution stage are monitored and stored.

4. The method for processing pipeline data according to claim 1, characterized in that, The step of reducing the base value of the resource value based on the iterative decrease value and the minimum running time includes: Obtain the base resource value corresponding to the minimum running time; The base value of the resource is reduced based on the iterative decreasing value.

5. The method for processing pipeline data according to claim 1, characterized in that, In the step of executing the task instruction and obtaining the resource value base and running time data for each execution stage: The execution steps that have no parent node dependency are executed concurrently in the project pipeline.

6. The method for processing pipeline data according to claim 1, characterized in that, In the step of sorting each execution stage according to the template pipeline to generate the task instructions for the project pipeline: The execution phase includes code retrieval, compilation, static scanning, unit testing, incremental code inspection, quality gate inspection, image building, and / or artifact delivery.

7. A pipeline data processing apparatus, characterized in that, include: The template acquisition module is used to acquire template pipelines, which represent the dependencies of each execution stage in the project pipeline; The task generation module is used to sort each execution step according to the template pipeline to generate task instructions for the project pipeline. The data statistics module is used to execute the task instructions and obtain the resource value base and running time data of each execution stage through the container monitoring tool; The resource adjustment module is used to increase the base value of the resource based on the iterative increment, iteratively run each execution stage, and obtain the intermediate time data after the iterative run; And when the intermediate time data of two adjacent iterations are equal, the intermediate time data corresponding to the current iteration is stored as the minimum running time; as well as The optimal value generation module is used to reduce the base value of the resource value based on the iterative decrease value and the minimum running time, iteratively run each of the execution stages, and obtain the intermediate running time data after the iterative run; And when the intermediate time consumption data of the current iteration is greater than the minimum running time, the resource value occupied by the previous iteration is stored as the optimal resource value, which represents the minimum value of resources occupied by the execution link under the minimum running time.

8. An electronic device, characterized in that, The electronic device includes: One or more processors; and A storage device for storing one or more programs, which, when executed by one or more processors, cause the electronic device to implement the pipelined data processing method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by the computer's processor, causes the computer to perform the pipeline data processing method according to any one of claims 1 to 6.

Images