Streamline task execution method and server

Abstract

The embodiment of the invention provides a pipeline task execution method, which is characterized in that a shared data directory and a workspace directory are preset in a node, and the shared data directory and the workspace directory are arranged in the same disk. The shared data directory comprises shared data, and the shared data is universal data used when the assembly line task is executed. When the assembly line task is executed, a Pod is created in the node, the shared data directory and the workspace directory are mapped into the Pod, and then shared data in the shared data directory is moved to the workspace directory to support the Pod to execute the assembly line task. The shared data is stored by setting the shared data directory, so that the situation that a new data copy is created in Pod for repeated storage every time the assembly line is executed is avoided.

Description

Technical Field

[0001] The present application relates to the field of server technology, and in particular to a method and server for executing a pipeline task. Background Art

[0002] In modern software development, continuous integration and continuous delivery (CI / CD) processes (referred to as pipeline tasks) play an important role in improving development efficiency and software quality. Many companies use solutions based on Kubernetes (K8S) clusters to implement these processes to take advantage of the elasticity and scalability of Kubernetes. Among them, Jenkins, as a widely used automation server, is used to trigger and manage pipeline tasks to achieve automated building, testing, and deployment.

[0003] In a Kubernetes cluster, the choice of persistent storage is critical to the efficiency and resource management of pipeline tasks. Especially when HostPath is used as a persistent storage solution, the Kubernetes plug-in will select a node in the cluster to create a Pod after Jenkins triggers a pipeline task. When the Pod executes a Jenkins pipeline task, it stores the task's related data in the Pod. Although the Pod will be destroyed after the end of its life cycle, the data stored in the host directory will still be retained. This means that every time a Pod creates and executes a Jenkins pipeline task, the host will store a new copy of the data.

[0004] Since the data of Jenkins pipeline tasks are often similar or repetitive, the data generated by each Pod when executing the task is stored repeatedly on the host, resulting in serious waste of disk resources. This repeated storage not only wastes valuable disk space, but may also affect the overall performance and storage management efficiency of the Kubernetes cluster.

[0005] Therefore, in order to improve the utilization efficiency of storage resources and optimize data management during the construction process, it is necessary to introduce innovative solutions into the existing storage mechanism to reduce duplicate storage and waste of data. Summary of the invention

[0006] The embodiments of the present application provide a method and a server for executing a pipeline task.

[0007] In a first aspect, an embodiment of the present application provides a method for executing a pipeline task. The method comprises:

[0008] Creating a service unit Pod for executing the pipeline task in a host machine; wherein a shared data directory and a workspace directory are mounted in a disk of the host machine, and the shared data directory includes shared data for executing the pipeline task;

[0009] Mapping the shared data directory and the workspace directory to the Pod;

[0010] The shared data in the shared data directory is moved to the workspace directory, so that the Pod executes the pipeline task according to the shared data.

[0011] In an embodiment of the present application, a service unit Pod for executing pipeline tasks is created in the host machine, and the execution environment of the pipeline tasks is isolated in an independent Pod, thereby avoiding mutual interference between tasks and providing a stable and repeatable execution environment. After the Pod is created, the shared data directory and the workspace directory are mapped to the Pod to ensure that the Pod can directly access the shared data and workspace data on the host machine without the need to store these data repeatedly in the Pod, thereby reducing the consumption of disk resources. Then, the shared data in the shared data directory is moved to the workspace directory so that the Pod executes the pipeline task based on these shared data. By moving the shared data to the workspace for execution, the Pod can efficiently use the existing data to complete the task, thereby avoiding the need to re-store the same data each time the task is executed, reducing the need for repeated data storage.

[0012] By moving shared data directly into the workspace directory for task execution, it not only avoids the problem of repeatedly storing the same data each time a task is executed, but also effectively reduces the waste of disk resources. This approach optimizes the use of storage resources, improves the execution efficiency of pipeline tasks in the Kubernetes cluster, and helps maintain the overall performance of the cluster and the stability of storage management.

[0013] In a possible implementation, mapping the shared data directory and the workspace directory to the Pod includes:

[0014] The shared data directory and the workspace directory are mapped to the first disk of the Pod, and the workspace directory is mapped to the second disk of the Pod; wherein the workspace directory mapped in the first disk is synchronized with the workspace directory mapped in the second disk.

[0015] In an embodiment of the present application, the shared data directory and the workspace directory are mapped to the first disk of the Pod together, and the shared data can be moved inside the disk, avoiding data transmission across disks and improving data transmission efficiency. In addition, the workspace directory data in the first disk and the second disk are synchronized to ensure the consistency of the workspace data on different disks, so that when the Pod processes tasks, no matter which disk it accesses the shared data of the workspace directory from, the data can be synchronized. It effectively avoids task errors or data inconsistencies caused by data asynchrony, and improves the reliability and accuracy of task execution.

[0016] In a possible implementation, mapping the shared data directory and the workspace directory to the Pod includes:

[0017] The shared data directory is mapped to the first disk of the Pod, and the workspace directory is mapped to the second disk of the Pod; wherein the workspace directory mapped in the first disk is synchronized with the workspace directory mapped in the second disk.

[0018] In a possible implementation, creating a service unit Pod for executing the pipeline task in a host machine includes:

[0019] Determining a label of the pipeline task;

[0020] Generate configuration information of the Pod according to the Pod template matching the label;

[0021] Create the Pod according to the configuration information.

[0022] In the embodiment of the present application, by introducing the mechanism of label matching and Pod template, the flexibility and efficiency of Pod creation during the execution of pipeline tasks are enhanced. By selecting the matching Pod template through labels, the optimal allocation of resources and the simplification of management are achieved, while the creation speed of Pod and the scalability of the system are improved, and the efficient execution and management of pipeline tasks in the Kubernetes cluster are achieved.

[0023] In a possible implementation, before moving the shared data in the shared data directory to the workspace directory, the method further includes:

[0024] Querying the workspace directory to determine whether the shared data exists;

[0025] The moving the shared data in the shared data directory to the workspace directory comprises:

[0026] If the shared data is not found in the workspace directory, moving the shared data in the shared data directory to the workspace directory;

[0027] When the shared data is found in the workspace directory, the method further includes:

[0028] The pipeline task is executed using the shared data in the workspace directory.

[0029] In an embodiment of the present application, before moving shared data from a shared data directory to a workspace directory, first query whether the shared data already exists in the workspace directory to avoid unnecessary data movement operations. If the shared data already exists in the workspace directory, there is no need to move it repeatedly, and the shared data in the workspace directory can be directly used to execute the pipeline task, thereby saving time and resources for disk I / O operations. This method optimizes the use and management of shared data by performing query operations before moving shared data. By avoiding unnecessary data movement, storage overhead and system load are reduced, and task processing efficiency is improved. At the same time, this method simplifies the data management process and enhances data consistency. Overall, this implementation method improves system performance and resource utilization efficiency while ensuring data validity.

[0030] In a possible implementation manner, moving the shared data in the shared data directory to the workspace directory includes:

[0031] Traversing the shared data in the shared data directory, selecting the shared data that has no corresponding empty directory in the shared data directory as the target shared data; the empty directory is used to indicate that the shared data corresponding to the empty directory has been occupied;

[0032] An empty directory corresponding to the target shared data is created in the shared data directory, and the target shared data is moved to the workspace directory.

[0033] In the embodiment of the present application, an empty directory is used in the shared data directory to mark whether the shared data is occupied, thereby avoiding data conflict problems during the shared data movement process.

[0034] In a possible implementation, the Pod processes the pipeline task through the workspace directory in the second disk.

[0035] In a possible implementation, after the Pod completes the pipeline task, the method further includes:

[0036] The shared data is moved from the workspace directory back to the shared data directory; and the business data generated by executing the pipeline task in the workspace directory is deleted.

[0037] In the embodiment of the present application, when the pipeline task is completed, the shared data previously moved from the shared data directory to the workspace directory is moved back to the original directory, so that the shared data can continue to be used by other Pods. In addition, after the pipeline task is completed, the business data generated in the workspace directory is deleted to free up disk space. Provide a clean environment for the next task execution, ensure that there is an independent workspace for each task execution, and avoid task conflicts or errors caused by residual data.

[0038] In a possible implementation manner, moving the shared data from the workspace directory back to the shared data directory includes:

[0039] In the shared data directory, an empty directory corresponding to the shared data is deleted, and the shared data is moved back to the shared data directory.

[0040] In the embodiment of the present application, when the target shared data is moved from the workspace directory back to the shared data directory, the empty directory corresponding to the target shared data in the shared data directory is first deleted, and then the target shared data is migrated. The accuracy of the shared data directory structure is ensured so that when the target shared data is migrated back, the shared data directory can correctly reflect the actual status of the shared data, ensuring that the shared data can continue to be used by other Pods.

[0041] In a second aspect, an embodiment of the present application provides a pipeline task execution device, the device comprising:

[0042] A creation module, used to create a service unit Pod for executing the pipeline task in a host machine; wherein a shared data directory and a workspace directory are mounted in the disk of the host machine, and the shared data directory includes shared data for executing the pipeline task;

[0043] A mapping module, used to map the shared data directory and the workspace directory to the Pod;

[0044] The moving module is used to move the shared data in the shared data directory to the workspace directory, so that the Pod executes the pipeline task according to the shared data.

[0045] In one possible implementation, the mapping module is specifically used to map the shared data directory and the workspace directory to the first disk of the Pod, and map the workspace directory to the second disk of the Pod; wherein the workspace directory mapped in the first disk is synchronized with the workspace directory mapped in the second disk.

[0046] In one possible implementation, the mapping module is specifically used to map the shared data directory to the first disk of the Pod, and map the workspace directory to the second disk of the Pod; wherein the workspace directory mapped in the first disk is synchronized with the workspace directory mapped in the second disk.

[0047] In a possible implementation, the creation module is specifically used to determine a label of the pipeline task; generate configuration information of the Pod according to a Pod template matching the label; and create the Pod according to the configuration information.

[0048] In a possible implementation, the device further includes a query module, and the query module is used to query whether the workspace directory contains the shared data before moving the shared data in the shared data directory to the workspace directory;

[0049] The moving module is specifically used to move the shared data in the shared data directory to the workspace directory when the shared data is not found in the workspace directory;

[0050] The device further comprises a task execution module. When the shared data is found in the workspace directory, the task execution module is used to execute the pipeline task using the shared data in the workspace directory.

[0051] In a possible implementation, the moving module is specifically used to traverse the shared data in the shared data directory, select the shared data that has no corresponding empty directory in the shared data directory as the target shared data; the empty directory is used to indicate that the shared data corresponding to the empty directory has been occupied; create an empty directory corresponding to the target shared data in the shared data directory, and move the target shared data to the workspace directory.

[0052] In a possible implementation, the Pod processes the pipeline task through the workspace directory in the second disk.

[0053] In a possible implementation, after the Pod completes the pipeline task, the dynamic module is also used to move the shared data from the workspace directory back to the shared data directory; and delete the business data generated by executing the pipeline task in the workspace directory.

[0054] In a possible implementation, the activation module is further configured to delete an empty directory corresponding to the shared data in the shared data directory, and move the shared data back to the shared data directory.

[0055] In a third aspect, an embodiment of the present application provides a server, comprising a processor, and a memory communicatively connected to the processor;

[0056] The memory is used to store computer-executable instructions;

[0057] The processor is used to execute the computer-executable instructions stored in the memory to implement the pipeline task execution method described in any embodiment of the first aspect above.

[0058] In a fourth aspect, an embodiment of the present application provides a computer storage medium, in which code is stored. When the code is executed, the device executing the code implements the execution method of the pipeline task described in any one of the first aspect or the second aspect.

[0059] It should be understood that the description of technical features, technical solutions, beneficial effects or similar language in this application does not imply that all features and advantages can be realized in any single embodiment. On the contrary, it is understood that the description of features or beneficial effects means that specific technical features, technical solutions or beneficial effects are included in at least one embodiment. Therefore, the description of technical features, technical solutions or beneficial effects in this specification does not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions and beneficial effects described in the present embodiment can also be combined in any appropriate manner. Those skilled in the art will understand that the embodiment can be realized without one or more specific technical features, technical solutions or beneficial effects of a specific embodiment. In other embodiments, additional technical features and beneficial effects can also be identified in a specific embodiment that does not embody all embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0061] Figure 1 A schematic diagram of a computing system provided in an embodiment of the present application;

[0062] Figure 2 A flowchart of a pipeline task execution method provided in an embodiment of the present application;

[0063] Figure 3 A schematic diagram of a configuration page provided in an embodiment of the present application;

[0064] Figure 4 A schematic diagram of a directory mapping provided in an embodiment of the present application;

[0065] Figure 5 A schematic diagram of a process for executing a Jenkins pipeline task provided in an embodiment of the present application;

[0066] Figure 6 A flowchart of another pipeline task execution method provided in an embodiment of the present application;

[0067] Figure 7 A schematic diagram of the structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] It should be noted that the embodiments described in this application are only part of the embodiments of this application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0069] In order to make the following embodiments clear, the technical terms involved in this application are first introduced.

[0070] Kubernetes, also known as K8S or kube, is an open source container scheduling platform used to automatically deploy, scale, and manage containerized applications.

[0071] The service unit (Pod) is the basic scheduling unit in Kubernetes. A Pod can include one or more tightly coupled containers that share the same network namespace and storage volume. Kubernetes can allocate Pods to different nodes in the Kubernetes cluster (referred to as K8S cluster) through the scheduler to run and manage containerized applications in the K8S cluster. Pod provides container encapsulation and management functions, allowing containers to share network and storage resources.

[0072] Jenkins is an open source automation server that is widely used for continuous integration (CI) and continuous deployment (CD). Jenkins allows developers to improve the efficiency of software development by automating the build, test, and deployment processes. Jenkins plugins enable it to integrate with many other tools and platforms to support flexible build and deployment pipelines. Jenkins can run in a K8S cluster to take advantage of the scalability and elasticity of K8S. By using K8S, Jenkins can dynamically create and manage Pods in a K8S cluster.

[0073] A pipeline task is the process of combining multiple steps into a task, or implementing a component to perform a specified task. A pipeline task can be considered as a combination of multiple steps that share a workspace and work together to complete a specified task.

[0074] Jenkins pipeline tasks (Pipeline) are the core tasks in Jenkins for automating software building, testing, and deployment processes. Jenkins pipeline tasks achieve the goals of continuous integration (CI) and continuous delivery (CD) by defining and managing automated steps and stages.

[0075] HostPath is a type of persistent storage volume in Kubernetes that allows Pods to access file directories on the host. HostPath volumes can store files that need to be shared between multiple containers. In one implementation, HostPath volumes can be used to store Jenkins build data or caches in a file directory on the host.

[0076] In a K8S cluster that uses HostPath as a persistent storage solution, when a pipeline task is triggered on Jenkins, the K8S plug-in selects a node in the K8S cluster that meets the task execution requirements, creates a Pod in the node, and uses the file directory of the node as the Pod's mounted volume. The Pod executes the pipeline task based on the file directory of the host (the node where the Pod is located). During the execution of the Jenkins pipeline task, the Pod stores the relevant data of the Jenkins pipeline task in the file directory.

[0077] After the Pod lifecycle ends, that is, after the Pod completes the execution of the Jenkins pipeline task, the Pod is destroyed, but the relevant data stored in the host during the Pod's execution of the Jenkins pipeline task will still be retained. The nodes in the K8S cluster serve as the host of the container, and usually store the relevant data during the execution of each Jenkins pipeline task. In other words, each time a Pod is created in the node to execute a Jenkins pipeline task, a copy of the Jenkins pipeline task data will be stored in the node. However, the data of the Jenkins pipeline task may be similar or duplicated, and repeated storage of the data of the Jenkins pipeline task will cause serious waste of disk resources.

[0078] An embodiment of the present application provides a method for executing a pipeline task, in which a shared data directory is pre-set in each node, and in each node, the shared data directory and the workspace directory are set in the same disk. The shared data directory includes multiple sets of identical shared data, and the shared data is the common data used when executing Jenkins pipeline tasks. When the Jenkins pipeline task is triggered, a Pod is created in the node, and the shared data directory and the workspace directory are mapped to the Pod, and then the shared data in the shared data directory is moved to the workspace directory to support the Pod to execute the Jenkins pipeline task. After the Jenkins pipeline task is executed, the shared data is moved back to the shared data directory so that it can be reused when other Jenkins pipeline tasks are executed later.

[0079] The method provided in the embodiment of the present application stores multiple sets of the same common data by setting up a shared data directory, thereby avoiding the creation of a new copy of the data in the Pod for repeated storage each time the pipeline is executed. When the Pod executes a task, the shared data is moved from the shared data directory to the Pod's workspace directory so that the data can be used during the life cycle of the Pod. After the task is completed, the data is moved back to the shared data directory. This mechanism effectively reduces the need to store duplicate data on the node. In addition, by centrally storing shared data in the shared data directory, managing and updating data becomes more centralized and simple. All Pods on the same node read data from the same directory, which can simplify the data update and maintenance process.

[0080] Furthermore, the embodiment of the present application also sets the shared data directory and the workspace directory in the same disk of the node, and when the shared data directory and the workspace directory are mapped to the Pod, they are also mapped to the same disk of the Pod. When the Pod executes the Jenkins pipeline task, the shared data can be moved from the shared data directory to the workspace directory in the disk, which can reduce the data access latency compared to moving data across disks.

[0081] The method provided in the embodiments of the present application can be applied to Figure 1 The computing system shown.

[0082] like Figure 1 As shown, the K8S cluster includes multiple computing devices (referred to as nodes), and a shared data directory and a workspace directory are set in each node, and the shared data directory and the workspace directory can be set in the same disk in each node. The shared data directory includes multiple sets of the same shared data, and the number of shared data in the shared data directory can be configured according to the computing resources of the node setting the shared data directory.

[0083] In the Pod created in the node, the shared data directory and workspace directory in the node are mapped to the Pod, and the shared data directory and workspace directory can also be mapped to the same disk of the Pod. When the Pod executes the Jenkins pipeline task, it traverses the unused shared data in the shared data directory, such as Figure 1 The shared data 1 shown in the figure is then moved to the workspace directory for use. When the Pod completes the Jenkins pipeline task, the shared data is moved back to the workspace directory.

[0084] Below, the pipeline task execution method provided in the embodiment of the present application is introduced in conjunction with the accompanying drawings.

[0085] It should be noted that the pipeline task execution method provided in the embodiment of the present application is not limited to executing Jenkins pipeline tasks, but is also applicable to executing other pipeline tasks. In addition, shared data refers to common data required to be used in the process of executing multiple pipeline tasks, and does not specifically refer to common data in the process of executing Jenkins pipeline tasks. The following embodiments are only examples.

[0086] like Figure 2 As shown, Figure 2 A flowchart of a pipeline task execution method provided in an embodiment of the present application.

[0087] S201: Users build K8S clusters.

[0088] like Figure 1 As shown, the K8S cluster includes multiple nodes, which are used to perform various tasks. In each node, sufficient disk space is configured for data storage, and the workspace directory and shared data directory are set in the disk space.

[0089] The workspace directory is a directory for storing temporary files and data, and is mainly used for data management during construction, testing, and operation. In an embodiment of the present application, the workspace directory is used to store temporary data and intermediate results during the execution of Jenkins pipeline tasks. The workspace directory provides an isolated environment to avoid conflicts between the temporary data and intermediate results of the Jenkins pipeline task and other tasks.

[0090] It should be noted that temporary data is not the same as shared data. Although some repeated data, i.e., shared data in the embodiment of the present application, is needed in the process of executing the Jenkins pipeline task, there may still be certain differences between the temporary files and intermediate results generated in each execution of the Jenkins pipeline task. Therefore, in the process of executing the Jenkins pipeline task, the temporary files and intermediate results are stored in the workspace directory. Moreover, after the Jenkins pipeline task is executed, the temporary files and intermediate results in the workspace directory are usually cleared.

[0091] The shared data directory is used to store multiple sets of shared data. Shared data refers to common data required to perform each task. In an embodiment of the present application, the shared data may be data required to perform each Jinkens pipeline task. The shared data directory makes the management of shared data centralized and simplified. When shared data needs to be updated, these shared data can be uniformly updated and maintained through the shared data directory, such as updating build tools, dependent libraries, etc.

[0092] Storing commonly used shared data in the shared data directory in advance avoids the need to store the same data each time a Pod is created to execute a task, thus reducing disk space usage. In addition, when executing a task, the Pod can directly access the required data from the shared data directory instead of generating or downloading the data each time.

[0093] In the embodiment of the present application, K8S can be integrated in Jenkins, and the construction of the K8S cluster can be realized by configuring the underlying cloud platform (Configure Clouds). The configuration page is as follows: Figure 3 As shown below. Figure 3 The contents that need to be configured are introduced one by one:

[0094] K8S Name: The name defined by the Kubernetes cloud configuration, which is used to identify and manage different Kubernetes clusters in Jenkins.

[0095] K8S address (node ​​address): refers to the address of the Kubernetes API server, usually an IP address or domain name. Jenkins will communicate with the Kubernetes cluster through this address.

[0096] K8S service certificate key: The Secure Socket Layer (SSL) certificate key used for secure communication with the Kubernetes API server to ensure the security of data during transmission.

[0097] K8S Namespace: In Kubernetes, namespaces are used to logically group cluster resources. Specifying a namespace in the Jenkins configuration ensures that Jenkins creates and manages resources in a specific namespace.

[0098] Credentials (K8S token value): A token used for authentication. Jenkins uses this token to interact with the Kubernetes API server to perform operations such as deployment and scaling.

[0099] Disable HTTPS certificate checking: A security option. If you check "Yes", Jenkins will not check the validity of the SSL certificate when communicating with the Kubernetes API server. If you check "No", Jenkins needs to check the validity of the SSL certificate when communicating with the Kubernetes API server.

[0100] Jenkins address: The address of the Jenkins server, used when the Kubernetes cluster calls back Jenkins.

[0101] Jenkins channel: refers to the specific channel or port for communication between Jenkins and Kubernetes.

[0102] Pod template: A template that defines a Pod running in Kubernetes, including settings such as Pod name, image, command, parameters, etc. It defines the Pod environment when the Jenkins job runs in a Kubernetes cluster.

[0103] Pod quantity: The number of Pod instances started for the Jenkins job in the Kubernetes cluster. You can adjust this value as needed to scale up or down the running instances of the job.

[0104] Read Timeout: The read timeout setting defines the maximum time Jenkins waits for a response from the Kubernetes API server. If no response is received within this time, Jenkins will consider the request failed.

[0105] Direct Connection: Direct connection option. If you check "Yes", Jenkins will connect directly to the Kubernetes API server; if you check "No", Jenkins will connect to the Kubernetes API server through some kind of proxy or middleware.

[0106] S202: Jenkins triggers the Jenkins pipeline task.

[0107] The core idea of ​​Jenkins pipeline tasks is to regard the entire build and deployment process as a continuous "pipeline", and each stage represents a step or stage in the process. Pipelines can be used to define the entire process from code submission to final deployment, including all build, test, integration and release steps.

[0108] Jenkins pipeline tasks provide a flexible and powerful way to manage and automate the various steps in the software development process. Through Jenkins pipeline tasks, the entire build, test, and deployment process can be standardized and automated, thereby improving development efficiency, reducing human errors, and ensuring software quality and stability.

[0109] The scenarios for triggering Jenkins pipeline tasks are as follows: When developers submit code to a version control system (such as Git, Subversion), Jenkins can trigger pipeline tasks through Webhook or scheduled polling. When a pull request is submitted to the code repository, Jenkins can trigger pipeline tasks through Webhook for building and testing.

[0110] S203: Jenkins determines a computing node that meets the Jenkins pipeline task execution requirements and creates a Pod in the computing node.

[0111] When Jenkins triggers a Jenkins pipeline task, Jenkins needs to select a computing node that meets the requirements from the K8S cluster to run the Jenkins pipeline task. Specifically, Jenkins can select a suitable Pod based on the following factors:

[0112] (1) Tag matching:

[0113] In the compute node, you can specify the label of the compute node, such as jenkins-agent. These labels are used to identify compute nodes suitable for running certain types of tasks. The agent part in the Jenkins pipeline task can specify labels to ensure that the task runs on the compute node with a specific label. Jenkins can select the compute node with the same or corresponding label to execute the Jenkins pipeline task based on the label in the Jenkins pipeline task.

[0114] (2) Resource requirements:

[0115] When selecting a compute node, Jenkins considers the resources required to execute the Jenkins pipeline task and matches the resource configuration of the Pod template. Jenkins determines the compute node based on the resource availability of the compute node and the resources required to execute the Jenkins pipeline task.

[0116] (3) Computing node availability:

[0117] Jenkins will check the status of each computing node in the K8S cluster to ensure that the computing nodes are healthy and available. If some computing nodes are unavailable (such as restarting or terminated), Jenkins will avoid assigning Jenkins pipeline tasks to these computing nodes.

[0118] (4) Task requirements:

[0119] Some Jenkins pipeline tasks may have special requirements for the execution environment, such as specific software versions, dependent libraries, etc. Jenkins will check the execution environment of each computing node in the K8S cluster, and the configuration of the Pod template needs to meet these requirements to ensure that the task can be executed smoothly.

[0120] After determining the compute node for executing the Jenkins pipeline task, create a Pod in the compute node to execute the Jenkins pipeline task.

[0121] Jenkins selects the appropriate Pod template based on the agent configuration in the Jenkins pipeline task definition. For example, if the Jenkins pipeline task defines labels, Jenkins selects the Pod template with matching labels. Jenkins generates the configuration information for the Pod using the selected Pod template. The configuration information includes the specifications of the Pod, such as container images, resource limits, and environment variables. Jenkins submits the configuration information to the selected compute node to start the Pod on the selected compute node. The containers in the Pod start running and execute the Jenkins pipeline task.

[0122] During the Pod creation process, the compute node maps the workspace directory and shared data directory to the Pod so that the Pod executes the Jenkins pipeline tasks based on the workspace directory and shared data directory.

[0123] In a possible implementation, the workspace directory and shared data directory in the computing node can be mapped to two disks in the Pod respectively. When the Pod executes the Jenkins pipeline task, a set of shared data is selected from the shared data directory and moved across disks to the workspace directory.

[0124] In another possible implementation, Figure 4 As shown in the figure, the workspace directory and shared data directory in the computing node belong to the same general directory. The workspace directory and shared data directory are mapped to the same disk in the Pod, and the workspace directory is mapped separately to another disk in the Pod. When the Pod executes the Jenkins pipeline task, a set of shared data is selected from the shared data directory and moved to the workspace directory on the disk. The shared data is also updated synchronously in the workspace directory on the other disk. Moving shared data from the shared data directory on the same disk to the workspace directory is faster than cross-disk operations. The shared data movement is limited to the same disk to increase the transmission speed.

[0125] S204: Pod executes Jenkins pipeline tasks.

[0126] Pod executes Jenkins pipeline tasks based on the shared data in the shared data directory and the workspace directory. When executing Jenkins pipeline tasks, Pod can use the shared data pre-stored in the shared data directory to execute tasks, without having to download and store the common data in the Jenkins pipeline tasks again.

[0127] In an embodiment of the present application, by pre-setting a shared data directory including multiple sets of shared data in the computing node, when a Pod is created in the computing node to execute a task, the shared data in the shared data directory can be used to execute the task without the need to additionally cache the shared data, thereby saving disk space on the host machine.

[0128] In an embodiment of the present application, the number of shared data in the shared data directory can be determined based on the computing resources of the computing node and the computing resources configured for the Pod in the Pod template. As an example, the computing resources of the computing node are 16U36G, that is, a 16-core central processing unit (CPU) and 36GB of memory; the computing resources of the Pod configured in the Pod template are 8U13G, that is, 8 cores and 18GB of memory are allocated. In this computing device, up to two Pods can be created, and two sets of shared data can be stored in the shared data directory. When two Pods in the computing node execute tasks at the same time, a set of shared data can be allocated to each Pod.

[0129] With respect to the above step S204, the present application embodiment further provides an implementation process of Pod executing Jenkins pipeline tasks, as follows: Figure 5 As shown:

[0130] S501: The Pod traverses the workspace directory to see if it includes shared data.

[0131] In an embodiment of the present application, after completing a Jenkins pipeline task, the Pod will typically move the shared data in the workspace directory back to the shared data directory so that it can be reused the next time the Jenkins pipeline task is executed.

[0132] However, in some special cases (such as the last time the Pod failed to execute the Jenkins pipeline task), the Pod does not have time to move the shared data back to the shared data directory. When the Pod is destroyed, the shared data still remains in the workspace directory. Therefore, the Pod first determines whether there is any shared data in the workspace directory that was used last time but was not moved back to the shared data directory in time.

[0133] If the workspace directory includes shared data, the Pod does not need to move data from the shared data directory to the workspace directory, and can directly use the shared data in the workspace directory as the target shared data to execute the Jenkins pipeline task, that is, execute step S503. If no shared data is found in the workspace directory, execute step S502.

[0134] S502: The Pod selects target shared data from the shared data directory, and moves the target shared data to the workspace directory.

[0135] If the shared data is not found in the workspace directory, the Pod needs to select unoccupied shared data from the shared data directory as the target shared data and move the target shared data to the target space directory.

[0136] In the shared data directory, each shared data has a corresponding number, such as shared data 1, shared data 2, and shared data X. When shared data is being used, the shared data directory also includes a corresponding empty directory Used. For example, if shared data 2 is being used, the shared data directory also includes shared data 2_Used (an empty directory Used corresponding to shared data 2).

[0137] The Pod traverses each shared data in the shared data directory and determines the unused shared data as the target shared data, that is, selects the shared data without a corresponding Used empty directory in the shared data directory as the target shared data, and creates a Used empty directory corresponding to the target shared data in the shared data directory to indicate that the target data is being used. After determining the target shared data, move the target shared data from the shared data directory to the workspace directory. As an implementation method, if the workspace directory and the shared data directory are mapped to two disks of the Pod respectively when creating the Pod, the movement of the target shared data is a cross-disk move. As another implementation method, if the workspace directory and the shared data directory are mapped to the same disk of the Pod when creating the Pod, the movement of the target shared data is an intra-disk move, and the movement latency of the target shared data is lower and the response speed is faster.

[0138] If all the shared data in the shared data directory that the Pod traverses are being used, it means that there is no free shared data in the shared data directory. The Pod can temporarily stop executing the Jenkins pipeline task and continue to execute after there is free shared data in the shared data directory. The Pod can also generate a new set of shared data as the target shared data stored in the working directory for use. The newly generated shared data is the same as the shared data in the shared data directory.

[0139] In another implementation, if all shared data in the shared data directory traversed by the Pod are being used, the Pod can also search for unused shared data in the workspace directories of other Pods executing Jenkins pipeline tasks. If so, the free shared data is moved from the workspace directories of other Pods to the workspace directory of the Pod for use.

[0140] In the embodiment of the present application, by selecting the idle shared data in the shared data directory for use, the risk of conflict when multiple Pods use the same set of shared data at the same time is avoided.

[0141] S503: Pod executes the Jenkins pipeline task based on the workspace directory.

[0142] The Pod switches back to the workspace directory and uses the shared data in the workspace directory to execute the Jenkins pipeline task.

[0143] The Pod needs to be switched back to the workspace directory on the independent disk to execute the Jenkins pipeline task in the independent workspace. Figure 4 Taking the architecture shown in the figure as an example, if the workspace directory and the shared data directory are mapped to the same disk (first disk) at the same time, and the workspace directory is also mapped to another disk (second disk) separately, when the Pod executes the Jenkins pipeline task, it is necessary to switch the workspace directory on the second disk. The pipeline task is executed using the workspace directory on the second disk.

[0144] After the Pod successfully executes the Jenkins pipeline task, step S504 is executed. If the Pod executes the Jenkins pipeline task, the task execution is directly exited, and the target shared data is also retained in the workspace directory.

[0145] S504: The Pod processes the target shared data.

[0146] If the target shared data is taken from the shared data directory, the Pod needs to move the target shared data from the workspace directory back to the shared data directory and delete the Used empty directory corresponding to the target shared data in the shared data directory. There are two cases where the target shared data is taken from the shared data directory: first, the target shared data is the shared data left in the workspace directory after the last execution of the Jenkins pipeline task; second, the target shared data is the shared data selected from the shared data directory when the Jenkins pipeline task is executed this time.

[0147] In both cases, the target shared data is taken from the shared data directory, that is, the target shared data has a corresponding number and a Used empty directory in the shared data directory. Therefore, after moving the target shared data back to the shared data directory, the Used empty directory corresponding to the target shared data needs to be deleted.

[0148] If the target shared data is newly generated shared data by the Pod, the shared data has no corresponding number set in the shared data directory, and there is no corresponding Used empty directory.

[0149] In a possible implementation, the Pod traverses the shared data directory to see if there is any shared data missing. For example, the shared data directory includes shared data 1, shared data 3, and shared data 4. The Pod may determine that shared data 2 is missing from the shared data directory, and then set a number for the target shared data, and move the target shared data back to the shared data directory as shared data 2. If there is no shared data missing from the shared data directory, the target shared data may be directly cleared.

[0150] In another possible implementation, the Pod may also directly move the target shared data back to the shared data directory, and set a number for the target shared data in the shared data directory. For example, if the shared data directory includes shared data 1, shared data 2, and shared data 3, the target shared data is set to shared data 4.

[0151] In an embodiment of the present application, after executing the Jenkins pipeline task, the Pod will also clear the business data generated by executing the Jenkins pipeline task in the workspace directory to release storage space. Finally, the Pod is deleted. Business data refers to business data generated by executing the Jenkins pipeline task. The execution of different Jenkins pipeline tasks may require the same shared data, but the generated business data may be different. Therefore, after executing the Jenkins pipeline task, it is necessary to clear the business data in the workspace directory to release disk space, while also avoiding affecting the next execution of the Jenkins pipeline task.

[0152] Through the above steps S501-S504, the Pod can move the shared data in the shared data directory to the workspace directory to execute the Jenkins pipeline task. There is no need to store an additional set of shared data each time a Jenkins pipeline task is executed. In addition, if the shared data directory and the workspace directory are mapped to the same disk of the Pod, the Pod can move the shared data from the shared data directory to the workspace directory on the disk to reduce the data movement latency. After the Pod executes the Jenkins pipeline task, the shared data can also be moved back to the shared data directory for use the next time the Jenkins pipeline task is executed, thereby realizing the reuse of shared data.

[0153] In the above embodiments, the shared data may be a git repository, which is used as a code source in the Jenkins pipeline task to perform operations such as building and testing. The shared data may also be a ccache cache or a binary file cache.

[0154] Next, we will take the shared data as a git repository as an example to introduce the execution method of the above pipeline tasks.

[0155] The proposed host machine hardware configuration is 16-core CPU, 36G memory and 300G data disk. Two sets of git repositories are stored in the data disk. Two Pods are created in the host machine, and each Pod is allocated 8-core CPU and 13G memory.

[0156] The directory names corresponding to the two git repositories are / usr1 / P / codeshare / share1 and / usr1 / P / codeshare / share2, and corresponding numbers, share1 and share2, are created for the two git repositories. / usr1 / P / codeshare / share1 and / usr1 / P / codeshare / share2 are subdirectories in the shared data directory "cd / sp / codeshare". The pipeline task execution method is as follows Figure 6 shown.

[0157] S601: Jerkirs triggers the pipeline task.

[0158] S602: K8S selects a computing node and creates a Pod to execute the pipeline in the computing node.

[0159] S603: Pod queries whether there is shared data in the workspace directory.

[0160] Pod queries whether there is a git repository under the workspace directory " / home / workspace" in Pod. If not, execute step S604; if yes, execute step S608.

[0161] S604: The Pod queries the shared data directory to see whether there is available shared data.

[0162] Pod searches for the directory corresponding to the git repository in the shared data directory "cd / sp / codeshare". If the directory corresponding to the git repository is found in the shared data directory "cd / sp / codeshare", step S605 is executed; if the directory corresponding to the git repository is not found in the shared data directory "cd / sp / codeshare", step S606 is executed.

[0163] S605: The Pod moves the target shared data to the workspace directory, and creates an empty Used directory corresponding to the target shared data in the shared data directory.

[0164] Pod moves the corresponding directory to the workspace directory " / home / workspace" and creates an empty directory Used in the workspace directory " / home / workspace".

[0165] S606: The Pod queries whether the shared data in other workspace directories is being used.

[0166] If the Pod does not find the directory corresponding to the git repository in the shared data directory "cd / sp / codeshare", it searches the workspaces of other pipeline tasks to check whether the git repositories in the directories of other workspaces are being used.

[0167] If the git repository in the other workspace directory is being used, the process returns to step S604 after waiting for a while; if the git repository in the other workspace directory is not being used, the process proceeds to step S607.

[0168] S607: Move the shared data from other workspace directories to the current workspace directory.

[0169] S608: Pod executes pipeline tasks.

[0170] Pod uses the git repository in the workspace directory " / home / workspace" to execute pipeline tasks.

[0171] When the Pod fails to execute the pipeline task, the target shared data will remain in the current workspace directory, and the Used empty directory corresponding to the git repository will be saved in the shared data directory. After the Pod successfully executes the pipeline task, step S609 is executed.

[0172] S609: The Pod determines whether the target shared data is taken from the shared data directory.

[0173] After the pipeline task is compiled successfully, the Pod determines whether the git repository is taken from the shared data directory. If the git repository is taken from the shared data directory, execute step S610; if the git repository is a newly generated git repository, execute step S611.

[0174] S610: The Pod moves the target shared data back to the shared data directory and deletes the Used empty directory corresponding to the target shared data.

[0175] If the git repository is taken from the shared data directory, move the directory corresponding to the git repository from the workspace directory back to the shared data directory "cd / sp / codeshare" and delete the corresponding Used empty directory.

[0176] S611: The Pod determines whether the shared data directory lacks shared data.

[0177] If yes, move the target shared data back to the shared data directory and complete the shared data number; if no, delete the target shared data directly.

[0178] The above are some specific implementations of the pipeline task execution method provided in the embodiment of the present application. Based on this, the present application also provides a corresponding pipeline task execution device. The device provided in the embodiment of the present application is introduced below. The device includes:

[0179] A creation module, used to create a service unit Pod for executing the pipeline task in a host machine; wherein a shared data directory and a workspace directory are mounted in the disk of the host machine, and the shared data directory includes shared data for executing the pipeline task;

[0180] A mapping module, used to map the shared data directory and the workspace directory to the Pod;

[0181] The moving module is used to move the shared data in the shared data directory to the workspace directory, so that the Pod executes the pipeline task according to the shared data.

[0182] The embodiments of the present application also provide corresponding computing devices and computer storage media for implementing the solutions provided by the embodiments of the present application.

[0183] like Figure 7 As shown, the computing device 100 includes a processor 110 and a memory 120, wherein the memory 120 is used to store instructions or codes, and the processor 110 is used to execute the instructions or codes so that the device executes the execution method of the pipeline task described in any embodiment of the present application.

[0184] The computer storage medium stores codes, and when the codes are executed, the device executing the codes implements the method for executing the pipeline tasks described in any embodiment of the present application.

[0185] The "first" and "second" in the names such as "first" and "second" (if any) mentioned in the embodiments of the present application are only used as name identifiers and do not represent the first or second in order.

[0186] Through the description of the above implementation methods, it can be known that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus a general hardware platform. Based on such an understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a storage medium, such as a read-only memory (ROM) / RAM, a magnetic disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network communication device such as a router) to execute the methods described in each embodiment of the present application or some parts of the embodiments.

[0187] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0188] The above description is merely an exemplary embodiment of the present application and is not intended to limit the protection scope of the present application.

Claims

1. A method for executing a pipeline task, characterized in that: include: Creating a service unit Pod for executing the pipeline task in a host machine; wherein a shared data directory and a workspace directory are mounted in a disk of the host machine, and the shared data directory includes shared data for executing the pipeline task; Mapping the shared data directory and the workspace directory to the Pod; The shared data in the shared data directory is moved to the workspace directory, so that the Pod executes the pipeline task according to the shared data.

2. The method according to claim 1, characterized in that Mapping the shared data directory and the workspace directory to the Pod includes: The shared data directory and the workspace directory are mapped to the first disk of the Pod, and the workspace directory is mapped to the second disk of the Pod; wherein the workspace directory mapped in the first disk is synchronized with the workspace directory mapped in the second disk.

3. The method according to claim 1, characterized in that Mapping the shared data directory and the workspace directory to the Pod includes: The shared data directory is mapped to the first disk of the Pod, and the workspace directory is mapped to the second disk of the Pod; wherein the workspace directory mapped in the first disk is synchronized with the workspace directory mapped in the second disk.

4. The method according to claim 1, characterized in that: The step of creating a service unit Pod in a host machine for executing the pipeline task includes: Determining a label of the pipeline task; Generate configuration information of the Pod according to the Pod template matching the label; Create the Pod according to the configuration information.

5. The method according to claim 1, characterized in that Before moving the shared data in the shared data directory to the workspace directory, the method further includes: Querying the workspace directory to determine whether the shared data exists; The moving the shared data in the shared data directory to the workspace directory comprises: If the shared data is not found in the workspace directory, moving the shared data in the shared data directory to the workspace directory; When the shared data is found in the workspace directory, the method further includes: The pipeline task is executed using the shared data in the workspace directory.

6. The method according to any one of claims 1 to 5, characterized in that: The moving the shared data in the shared data directory to the workspace directory comprises: Traversing the shared data in the shared data directory, selecting the shared data without a corresponding empty directory in the shared data directory as the target shared data; the empty directory is used to indicate that the shared data corresponding to the empty directory has been occupied; An empty directory corresponding to the target shared data is created in the shared data directory, and the target shared data is moved to the workspace directory.

7. The method according to claim 2 or 3, characterized in that: The Pod processes the pipeline task through the workspace directory in the second disk.

8. The method according to claim 1, characterized in that After the Pod completes the pipeline task, the method further includes: The shared data is moved from the workspace directory back to the shared data directory; and the business data generated by executing the pipeline task in the workspace directory is deleted.

9. The method according to claim 8, characterized in that The step of moving the shared data from the workspace directory back to the shared data directory comprises: In the shared data directory, an empty directory corresponding to the shared data is deleted, and the shared data is moved back to the shared data directory.

10. A server, characterized in that: comprising a processor, and a memory communicatively connected to the processor; The memory is used to store computer-executable instructions; The processor is used to execute the computer-executable instructions stored in the memory to implement the execution method of the pipeline task described in any one of claims 1-9.

Images