Jenkins-based task construction method, system and storage medium

Abstract

The application relates to a Jenkins-based task construction method, system and storage medium. The method comprises the following steps: a slave node obtains a cache dependent file required for task construction from a master node; and a task construction slave node constructs a pipeline task by using the cache dependent file, wherein the task construction slave node is one of the slave nodes. According to the method, the master node uniformly manages dependent packages, ensures that the dependent packages in the task nodes and the master node are always consistent, and the dependent packages can also be used in the pods of the task nodes, so that the efficiency of building and pulling dependencies is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a Jenkins-based task building method, system, and storage medium. Background Technology

[0002] Jenkins is a widely used visual web tool for continuous building and service delivery, essentially enabling the "automated" compilation, packaging, distribution, and deployment of various projects. Automated deployment offers numerous advantages, such as reduced costs, increased productivity, high availability, enhanced reliability, and performance optimization. Pipeline tasks are widely used in Kubernetes service deployment, from cloning the original code and building to generating an image and deploying it as a usable application. Currently, Jenkins' task execution mode in ICKS uses a slave node corresponding to the agent to start task pods. The master node schedules the pods, which exist alongside the pipeline tasks. Once a task completes, regardless of success or failure, it is automatically destroyed.

[0003] When executing the `mavenbuild` command locally, the required dependency files are cached by default in the `~ / .m2 / repository` directory. Subsequent dependencies will be searched for in this local cache first. Only if not found locally will they be fetched from a configured private repository, significantly optimizing the build process. Similarly, when executing a Maven build task in Jenkins, a cache is also generated in the `~ / .m2 / repository` directory of the pod containing the task. However, since the lifecycle of a task pod in Jenkins is tied to the task itself, the corresponding pod is destroyed after a pipeline task completes, regardless of success or failure. Therefore, even if dependencies are cached, they won't be used in subsequent executions; they will be destroyed along with the pod. Re-executing a task and fetching dependencies would consume significant time and resources. To address this, we consider using these caches by mounting the cached files generated in the pod to the slave host node via a shared storage volume. This way, regardless of which pipeline task is used or how many times it is executed, the cached files can be used on the host node via the mounted storage volume, prioritizing the use of the cache when fetching dependencies. However, this approach presents another problem: mounting the storage volume will overwrite the entire file, unlike traditional copy-paste which saves the existing file and adds new dependencies. This global overwrite method also fails to achieve efficient caching. Summary of the Invention

[0004] Therefore, it is necessary to provide a task building method, system, and storage medium based on Jenkins to address the aforementioned technical issues.

[0005] On the one hand, a task building method based on Jenkins is provided, the method comprising:

[0006] The slave node retrieves the cached dependency files required for task building from the master node;

[0007] The task build slave node uses the cached dependency file to build pipeline tasks, wherein the task build slave node is one of the slave nodes.

[0008] In one embodiment, the slave node obtains the cached dependency files required for task construction from the master node, including:

[0009] The master node communicates with at least one slave node and listens to whether the slave node pod generates build dependency packages;

[0010] If so, then synchronize the dependency packages built by each slave node to the master node, and build the dependency library;

[0011] The slave node retrieves the cached dependency files required for task building from the master node's dependency library.

[0012] In one embodiment, the slave node retrieves the cached dependency files required for task construction from the master node's dependency library, including:

[0013] The master node synchronizes the dependency libraries to the dependency cache paths of each slave node;

[0014] The node retrieves the cached dependency files required for task building from its own dependency cache path.

[0015] In one embodiment, the task building slave node utilizes the cached dependency file to build pipelined tasks, including:

[0016] Configure the pipeline task flow, including cached dependency file fetch addresses;

[0017] Retrieve the cached dependency files required for task build;

[0018] The build task is executed according to the pipeline task flow using the cached dependency file.

[0019] In one embodiment, obtaining the cached dependency file includes:

[0020] Obtain the cached dependency files required for task building from its own dependency cache path;

[0021] If the cached dependency files required for task building are not present in the cached path, the required cached dependency files will be retrieved from the configured cached dependency file retrieval address.

[0022] In one embodiment, the task building slave node uses the cached dependency file to build pipeline tasks, and further includes:

[0023] Once the build task is complete, the master node synchronization mechanism is triggered to update the master node's dependency libraries.

[0024] In one embodiment, the update of the master node dependency library includes:

[0025] The task build process synchronizes the new cached dependency files generated by the node in the build pipeline task to the dependency library.

[0026] In one embodiment, the update of the master node dependency library further includes:

[0027] The task builds the necessary cached dependency files from the configured cached dependency file pull address on the node and updates them to the main node's dependency library.

[0028] On the other hand, a Jenkins-based task building system is provided, the system comprising a master node and at least one slave node, wherein:

[0029] The master node is used to communicate with at least one slave node, listen to whether the slave node pod generates build dependency packages, and obtain the build dependency packages from each slave node to build the dependency library;

[0030] The slave node retrieves the cached dependency files required for task building from the master node;

[0031] The task build slave node uses the cached dependency file to build pipeline tasks, wherein the task build slave node is one of the slave nodes.

[0032] In another aspect, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, performs the following steps:

[0033] The slave node retrieves the cached dependency files required for task building from the master node;

[0034] The task build slave node uses the cached dependency file to build pipeline tasks, wherein the task build slave node is one of the slave nodes.

[0035] The aforementioned Jenkins-based task building method, system, and storage media manage dependency packages uniformly through the master node, ensuring that the dependency packages in the task nodes and the master node remain consistent. At the same time, dependency packages can also be used in the pods on the task nodes to improve the efficiency of building and pulling dependencies. Attached Figure Description

[0036] Figure 1This is an application environment diagram of a Jenkins-based task building method in one embodiment;

[0037] Figure 2 This is a flowchart illustrating a Jenkins-based task building method in one embodiment;

[0038] Figure 3 Here is a block diagram of a Jenkins-based task building system in one embodiment;

[0039] Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0041] The Jenkins-based task building method provided in this application can be applied to, for example... Figure 1 The application environment shown depicts a master node (node1) connected to and communicating with multiple slave nodes (node2, node3, node4, and node5). The master node (node1) uses a synchronous pull and distribution mechanism to ensure that the master node (node1) and each of the slave nodes (node2, node3, node4, and node5) executing pipeline tasks have the same, up-to-date, and complete dependency packages. The master node (node1) communicates with the task nodes, monitors the build tasks in the pipeline running on those nodes, synchronizes the dependencies of the ~ / .m2 / repository mounted in the pods on the slave nodes in real time, and distributes them to other nodes, ensuring the efficient use of dependency caching.

[0042] In one embodiment, such as Figure 2 As shown, a task building method based on Jenkins is provided, including the following steps:

[0043] S1, the slave node retrieves the cached dependency files required for task construction from the master node.

[0044] In one embodiment, step S1 includes:

[0045] S11, the master node communicates with at least one slave node and listens to whether the slave node pod generates build dependency packages;

[0046] S12, If so, synchronize the dependency packages built by each slave node to the master node, and build the dependency library;

[0047] S13, the slave node retrieves the cached dependency files required for task building from the master node's dependency library.

[0048] In one embodiment, step S13 includes:

[0049] S131, the master node synchronizes the dependency library to the dependency cache path of each slave node;

[0050] S132, retrieve the cached dependency files required for task construction from the node's own dependency cache path.

[0051] Specifically, to address the issue of build tasks in a Jenkins pipeline using locally cached dependencies, and to ensure that newly cached dependencies can be saved to task nodes via mounted volumes in the pod and simultaneously synchronized to the master node, this application primarily utilizes storage volume mounting technology and Kubernetes' list & watch monitoring and push technologies.

[0052] The master node communicates with the slave nodes, using `list` and `watch` to monitor the build dependencies generated by each pod on the slave node (by default, these dependencies are cached in ` / root / .m2 / repository` of the corresponding pod). The master node then synchronizes the files from the pods to the corresponding files on the master node. The dependency files on the master node are then managed, with only new dependency packages added, not overwritten. This ensures that the dependency library becomes increasingly comprehensive.

[0053] The master node synchronizes the latest cached packages to the / root / .m2 / repository directory on each slave node. When a build task is executed on this node, the dependency packages are mounted to the pod using storage volumes, thus enabling the use of dependency caching.

[0054] The newly generated cached dependency packages will be monitored by the master node and updated on the master node.

[0055] By repeating this process, dependency caching can be implemented. This workflow is suitable for Maven, Go, and other build processes that require dependency packages.

[0056] S2, the task building slave node uses the cached dependency file to build pipeline tasks, wherein the task building slave node is one of the slave nodes.

[0057] In one embodiment, step S2 includes:

[0058] S21, Configure the pipeline task flow, including cache dependency file fetch addresses;

[0059] S22, retrieve the cached dependency files required for task building;

[0060] In one embodiment, step S22 includes:

[0061] S221, Obtain the cached dependency files required for task building from its own dependency cache path;

[0062] S222, If the cached dependency file required for task building does not exist in the cached path of the task itself, then the required cached dependency file is retrieved from the configured cached dependency file retrieval address.

[0063] S23, the build task is executed according to the pipeline task flow using the cached dependency file.

[0064] In one embodiment, step S2 further includes:

[0065] S24. After the build task is completed, the master node synchronization mechanism is triggered to update the master node dependency library.

[0066] In one embodiment, updating the master node dependency library includes: synchronizing new cached dependency files generated by the task build slave node in the build pipeline task to the dependency library.

[0067] In one embodiment, updating the master node dependency library further includes updating the master node dependency library by having the task build slave node pull the required cached dependency files from the configured cached dependency file pull address.

[0068] Specifically, the process of building pipeline tasks from nodes using the cached dependency files mainly includes the following steps:

[0069] 1. Configure pipeline tasks

[0070] Configure the task flow in the pipeline task, including pulling code, building, generating images and pushing, and generating application deployment flow based on the images.

[0071] 2. Pre-construction tasks

[0072] The master node synchronizes cached dependency data, ensuring that the cached dependencies of each task node are the latest and most comprehensive.

[0073] 3. Build the task

[0074] Jenkins task builds mount the pod to the dependency cache path / root / .m2 / repository on the task node as a storage volume. When executing build commands, such as Maven build, the cache is used first by default. If the dependency is not found in the cache, the configured private repository address is used to fetch it.

[0075] 4. Post-build tasks

[0076] The built task will trigger the synchronization mechanism of the master node, so that the new cached dependencies pulled in the task pod built on the task node can be synchronized to the master node. The master node will ensure that the new data synchronized this time will also be synchronized to each task node.

[0077] The above Jenkins-based task building method can achieve the following:

[0078] (1) The Master node manages dependency packages uniformly, ensuring that the dependency packages in the task nodes and the Master node are always consistent; at the same time, the dependency packages can also be used in the pods of the task nodes to improve the efficiency of building and pulling dependencies.

[0079] (2) The Master node monitors the build process in the pods running on the slave nodes and promptly updates the cache of newly pulled dependency packages in the corresponding node paths. At the same time, it uses the list && watch mechanism to synchronize new dependency packages to the paths of each node.

[0080] It should be understood that, although Figure 2 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 2 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0081] In one embodiment, such as Figure 3 As shown, a Jenkins-based task building system is provided, including a master node and at least one slave node, wherein:

[0082] The master node is used to communicate with at least one slave node, listen to whether the slave node pod generates build dependency packages, and obtain the build dependency packages from each slave node to build the dependency library;

[0083] The slave node retrieves the cached dependency files required for task building from the master node;

[0084] The task build slave node uses the cached dependency file to build pipeline tasks, wherein the task build slave node is one of the slave nodes.

[0085] In one embodiment, the master node is used to communicate with at least one slave node and listen to whether the slave node pod generates build dependency packages;

[0086] If so, the dependency packages built by each slave node will be synchronized to the master node, and the dependency library will be built.

[0087] In one embodiment, the master node is used to synchronize the dependency library to the dependency cache path of each slave node; the slave node is used to obtain the cached dependency files required for task building from its own dependency cache path.

[0088] In one embodiment, task building from the node is used for:

[0089] Configure the pipeline task flow, including cached dependency file fetch addresses;

[0090] Retrieve the cached dependency files required for task build;

[0091] The build task is executed according to the pipeline task flow using the cached dependency file.

[0092] In one embodiment, task building from the node is used for:

[0093] Obtain the cached dependency files required for task building from its own dependency cache path;

[0094] If the cached dependency files required for task building are not present in the cached path, the required cached dependency files will be retrieved from the configured cached dependency file retrieval address.

[0095] In one embodiment, task building from the node is used for:

[0096] Once the build task is complete, the master node synchronization mechanism is triggered to update the master node's dependency libraries.

[0097] In one embodiment, task building from the node is used for:

[0098] New cached dependency files generated during the build pipeline task will be synchronized to the master node dependency library.

[0099] In one embodiment, task building from the node is used for:

[0100] The required cached dependency files will be pulled from the configured cached dependency file pull address and synchronized to the master node dependency library during the build pipeline task.

[0101] For specific limitations regarding the Jenkins-based task creation system, please refer to the limitations of the Jenkins-based task creation method mentioned above, which will not be repeated here. Each module in the aforementioned Jenkins-based task creation system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, so that the processor can call and execute the corresponding operations of each module.

[0102] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a Jenkins-based task construction method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0103] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0104] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0105] The slave node retrieves the cached dependency files required for task building from the master node;

[0106] The task build slave node uses the cached dependency file to build pipeline tasks, wherein the task build slave node is one of the slave nodes.

[0107] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0108] The master node communicates with at least one slave node and listens to whether the slave node pod generates build dependency packages;

[0109] If so, then synchronize the dependency packages built by each slave node to the master node, and build the dependency library;

[0110] The slave node retrieves the cached dependency files required for task building from the master node's dependency library.

[0111] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0112] The master node synchronizes the dependency libraries to the dependency cache paths of each slave node;

[0113] The node retrieves the cached dependency files required for task building from its own dependency cache path.

[0114] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0115] Configure the pipeline task flow, including cached dependency file fetch addresses;

[0116] Retrieve the cached dependency files required for task build;

[0117] The build task is executed according to the pipeline task flow using the cached dependency file.

[0118] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0119] Obtain the cached dependency files required for task building from its own dependency cache path;

[0120] If the cached dependency files required for task building are not present in the cached path, the required cached dependency files will be retrieved from the configured cached dependency file retrieval address.

[0121] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0122] Once the build task is complete, the master node synchronization mechanism is triggered to update the master node's dependency libraries.

[0123] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0124] The task build process synchronizes the new cached dependency files generated by the node in the build pipeline task to the dependency library.

[0125] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0126] The task builds the necessary cached dependency files from the configured cached dependency file pull address on the node and updates them to the main node's dependency library.

[0127] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0128] The slave node retrieves the cached dependency files required for task building from the master node;

[0129] The task build slave node uses the cached dependency file to build pipeline tasks, wherein the task build slave node is one of the slave nodes.

[0130] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0131] The master node communicates with at least one slave node and listens to whether the slave node pod generates build dependency packages;

[0132] If so, then synchronize the dependency packages built by each slave node to the master node, and build the dependency library;

[0133] The slave node retrieves the cached dependency files required for task building from the master node's dependency library.

[0134] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0135] The master node synchronizes the dependency libraries to the dependency cache paths of each slave node;

[0136] The node retrieves the cached dependency files required for task building from its own dependency cache path.

[0137] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0138] Configure the pipeline task flow, including cached dependency file fetch addresses;

[0139] Retrieve the cached dependency files required for task build;

[0140] The build task is executed according to the pipeline task flow using the cached dependency file.

[0141] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0142] Obtain the cached dependency files required for task building from its own dependency cache path;

[0143] If the cached dependency files required for task building are not present in the cached path, the required cached dependency files will be retrieved from the configured cached dependency file retrieval address.

[0144] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0145] Once the build task is complete, the master node synchronization mechanism is triggered to update the master node's dependency libraries.

[0146] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0147] The task build process synchronizes the new cached dependency files generated by the node in the build pipeline task to the dependency library.

[0148] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0149] The task builds the necessary cached dependency files from the configured cached dependency file pull address on the node and updates them to the main node's dependency library.

[0150] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0151] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0152] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A task construction method based on Jenkins, characterized in that, include: The slave node retrieves the cached dependency files required for task building from the master node; The task build slave node uses the cached dependency file to build pipeline tasks, wherein the task build slave node is one of the slave nodes; The slave node obtains the cached dependency files required for task construction from the master node, including: The master node communicates with at least one slave node and listens to whether the slave node pod generates build dependency packages; If so, then synchronize the dependency packages built by each slave node to the master node, and build the dependency library; The slave node retrieves the cached dependency files required for task building from the master node's dependency library; The slave node obtains the cached dependency files required for task construction from the master node's dependency library, including: The master node synchronizes the dependency libraries to the dependency cache paths of each slave node; The node retrieves the cached dependency files required for task building from its own dependency cache path; If the cache dependency file required for task building does not exist in the cache path of the task itself, the required cache dependency file is pulled from the configured cache dependency file pull address, wherein the cache dependency file is mounted to the slave node pod through a storage volume.

2. The task construction method based on Jenkins according to claim 1, characterized in that, The task building process involves the node constructing pipelined tasks using the cached dependency files, including: Configure the pipeline task flow, including cached dependency file fetch addresses; Retrieve the cached dependency files required for task build; The build task is executed according to the pipeline task flow using the cached dependency file.

3. The task construction method based on Jenkins according to claim 1, characterized in that, The task building process, which involves building pipelined tasks from nodes using the cached dependency files, also includes: Once the build task is complete, the master node synchronization mechanism is triggered to update the master node's dependency libraries.

4. The task construction method based on Jenkins according to claim 3, characterized in that, The updated master node dependency libraries include: The task build process synchronizes the new cached dependency files generated by the node in the build pipeline task to the dependency library.

5. The task construction method based on Jenkins according to claim 4, characterized in that, The update of the master node dependency library also includes: The task builds the necessary cached dependency files from the configured cached dependency file pull address on the node and updates them to the main node's dependency library.

6. A Jenkins-based task building system for implementing the Jenkins-based task building method as described in any one of claims 1-5, characterized in that, The system includes a master node and at least one slave node, wherein: The master node is used to communicate with at least one slave node, listen to whether the slave node pod generates build dependency packages, and obtain the build dependency packages from each slave node to build the dependency library; The slave node retrieves the cached dependency files required for task building from the master node; The task build slave node uses the cached dependency file to build pipeline tasks, wherein the task build slave node is one of the slave nodes.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images