Communication method and system for self-defined Docker container of continuous integration assembly line

Abstract

The invention relates to the technical field of continuous integration, in particular to a communication method and system of a user-defined Docker container for a continuous integration assembly line, and the method comprises the steps: making a user-defined Docker mirror image, configuring a mirror image construction task, and distributing the mirror image construction task; the method comprises the following steps: creating a Pod based on a container arrangement interface, and respectively starting a side vehicle container from a side vehicle Docker mirror image address built in a platform and a construction container from a user-defined Docker mirror image address through a Pod controller; communication is carried out based on a command pipeline and a log pipeline which are connected between the side vehicle container and the construction container, so that the construction container executes the mirror image construction task, and a corresponding construction log is obtained and sent to a message queue; and reading the construction log in the message queue and displaying the construction log to the user terminal. The method has the effect of improving the flexibility and maintainability of the continuous integration platform.

Description

Technical Field

[0001] This application relates to the field of continuous integration technology, and in particular to a communication method and system for a custom Docker container used in a continuous integration pipeline. Background Technology

[0002] As software scales up and delivery pace accelerates, Continuous Integration (CI) has become a core component of modern software engineering. By automating processes such as code compilation, packaging, deployment, and testing, CI can significantly improve development efficiency, shorten product delivery cycles, and effectively ensure software quality.

[0003] Traditional build and test environments often rely on virtual machines (such as VMware, VirtualBox, etc.) for deployment. However, virtual machines have problems such as high resource consumption, cumbersome and time-consuming allocation and reclamation, the need for independent network IP resources, and resource competition among multiple applications in a virtual machine, making it difficult to meet the needs of rapid iteration.

[0004] In recent years, Docker container technology has been widely used in continuous integration scenarios due to its lightweight nature, strong environment isolation, and rapid deployment. Developers can package their build or test environments into Docker images, achieving standardization and reusability of the environment.

[0005] However, when current continuous integration platforms call user-defined Docker images, the lack of a unified communication standard means that the platform cannot directly interact with the processes inside the container. This usually requires modifying the image or installing additional services, increasing the complexity of use. Summary of the Invention

[0006] To improve the flexibility and maintainability of continuous integration platforms, this application provides a communication method and system for custom Docker containers used in continuous integration pipelines.

[0007] Firstly, this application provides a communication method for custom Docker containers used in continuous integration pipelines, employing the following technical solution: A communication method for a custom Docker container used in a continuous integration pipeline, comprising the following steps: Create a custom Docker image, configure the image build task, and distribute the image build task. Pods are created based on the container orchestration interface, and the Pod controller starts sidecar containers from the platform's built-in sidecar Docker image address and build containers from a custom Docker image address, respectively. Communication is achieved through the command and log pipelines connecting the sidecar container and the build container, enabling the build container to execute the image build task, obtain the corresponding build logs, and send them to the message queue. The build log is read from the message queue and displayed to the user terminal.

[0008] In some embodiments, creating a custom Docker image, configuring an image build task, and distributing the image build task includes the following steps: Obtain all the software environment needed to build a custom Docker image; Configure a custom Docker image address, build command, and timeout to generate the image build task.

[0009] In some embodiments, a Pod is created based on a container orchestration interface, and a sidecar container derived from a platform-built-in sidecar Docker image address and a build container derived from a custom Docker image address are launched through a Pod controller, including the following steps: Create a build host and configure the input parameters, including node name, number of executors, working directory, usage method, startup method, and node availability; Obtain environment variables based on the created host; Configure a shared disk volume that mounts the sidecar container and the build container together. The shared disk volume is used to store the files corresponding to the command pipeline and the log pipeline. Assemble the sidecar container Pod controller based on the built-in Docker image address, the environment variables, the connection command, and the shared disk volume, and download the sidecar Docker image according to the sidecar container Pod controller; Based on the custom Docker image address and the shared disk volume, a container Pod controller is assembled and built, and the custom Docker image is downloaded based on the built container Pod controller; A Pod is created based on the container orchestration interface, and the sidecar container and the build container are started through the sidecar container Pod controller and the build container Pod controller.

[0010] In some embodiments, the following steps are included before performing the image build task: Configure a container build scheduling task, the input parameters of which include the container host name corresponding to the unique identifier of the sidecar container, the message queue name, the build command, and the timeout; When the container building scheduling task is triggered, the image building task is assigned to the corresponding sidecar container.

[0011] In some embodiments, the input parameters corresponding to the container scheduling task include: The build command is base64 encoded; If the execution time of the build command exceeds the specified time limit, the current image build task is terminated.

[0012] In some embodiments, communication is made between the sidecar container and the build container via command and log pipes to enable the build container to execute the image build task, including the following steps: After the sidecar container starts, it creates the command pipeline and the log pipeline; The sidecar container receives the image building task and sends it to the command pipeline. When the build container determines that the command pipeline exists, it continuously retrieves and executes the image building task in the command pipeline.

[0013] In some embodiments, obtaining the corresponding build logs and sending them to a message queue includes the following steps: After the image building task is completed, the build container generates a build log based on the execution result and sends it to the log pipeline. The sidecar container reads the build log in the log pipeline and sends it to the message queue.

[0014] In some embodiments, the following steps are also included: After the image build task is completed, the build container sends an end marker to the log pipeline; After the sidecar container reads the end marker in the log pipeline, it terminates the image building task.

[0015] In some embodiments, after the image build task is completed, the following steps are also included: The sidecar container compresses all files in the shared disk volume to obtain a compressed package, and uploads the compressed package to the storage server. The shared disk volume contains the build results files shared between the sidecar container and the build container.

[0016] Secondly, this application provides a communication system for a custom Docker container used in a continuous integration pipeline, employing the following technical solution: A communication system for custom Docker containers used in continuous integration pipelines, comprising: A continuous integration platform for creating custom Docker images and configuring image build tasks; The Jenkins platform is used to receive and distribute the image build tasks. The Kubernetes platform is used to create Pods based on the container orchestration interface, and to launch sidecar containers from the platform's built-in sidecar Docker image address and build containers from custom Docker image addresses through the Pod controller. The pipeline execution engine is used to obtain the image build task issued by the Jenkins platform, communicate with the command pipeline and log pipeline connected between the sidecar container and the build container to enable the build container to execute the image build task, obtain the corresponding build log, and send the build log to the message queue; The RabbitMQ platform is used to read the build logs from the message queue and display them to the user terminal.

[0017] The technical solutions provided by the embodiments of this application have the following technical effects: 1. High-efficiency build execution capability: By customizing the communication methods of Docker containers, distributed calls and parallel task processing are achieved, enabling the build, test, and deployment tasks in the continuous integration pipeline to be executed quickly, significantly improving the overall build efficiency; 2. Easy-to-use and non-intrusive integration method: Users only need to configure the Docker image address and build command in the continuous integration platform. There is no need to modify the existing business logic or depend on additional components, which realizes a "non-intrusive" call mode that is completely transparent to users. 3. Controllable and reusable environment: By encapsulating and building environments using Docker images, a consistent runtime environment can be quickly reproduced on any node, ensuring the consistency and traceability of build results. Different tasks can flexibly load corresponding images, achieving standardized and versioned environment management; 4. Improved resource utilization and reduced costs: Containers can be destroyed immediately after the build task is completed, and runtime environments can be created and recycled as needed, which greatly reduces the occupation of server resources, reduces dependence on traditional virtual machines, and saves operation and maintenance costs. 5. Supports diverse and customized task scenarios: Custom images can meet the high environmental requirements of tasks such as cross-compilation, automated testing, and security scanning, and support flexible expansion of build environments for different languages ​​and toolchains. Attached Figure Description

[0018] Figure 1 This is a schematic diagram illustrating the steps of a communication method for a custom Docker container used in a continuous integration pipeline, as provided in this embodiment.

[0019] Figure 2 This is a schematic diagram of the dual-container communication process provided in an embodiment of this application.

[0020] Figure 3 This is a schematic diagram illustrating the sending of build logs to a message queue, as provided in an embodiment of this application.

[0021] Figure 4 This is a schematic diagram of the framework of a communication system for a custom Docker container used in a continuous integration pipeline, as provided in an embodiment of this application. Detailed Implementation

[0022] To better understand the purpose, technical solutions, and advantages of this application, it has been described and illustrated below with reference to the accompanying drawings and embodiments. However, those skilled in the art should understand that this application can be implemented without these details. In some cases, to avoid obscuring various aspects of this application due to unnecessary description, well-known methods, processes, systems, components, and / or circuits already described at a higher level will not be elaborated upon. It will be apparent to those skilled in the art that various modifications can be made to the embodiments disclosed in this application, and the general principles defined in this application can be applied to other embodiments and application scenarios without departing from the principles and scope of this application. Therefore, this application is not limited to the illustrated embodiments, but conforms to the broadest scope consistent with the scope of protection claimed in this application.

[0023] It should be noted that the descriptions of these embodiments are for the purpose of aiding understanding the present invention, but do not constitute a limitation thereof. Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0024] In the description of this application, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0025] In the description of this application, the terms "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples.

[0026] This application solution is based on a system implementation that includes a continuous integration platform, a Jenkins platform, a Kubernetes platform, a pipelined execution engine, and a RabbitMQ platform.

[0027] The continuous integration platform is a visual continuous integration pipeline orchestration system that integrates various types of build tasks, including code compilation, code scanning, software packaging, installation and deployment, automated testing, and security testing, simplifying user configuration and improving R&D efficiency.

[0028] The build server mounted on the pipeline execution engine is mounted on the Jenkins platform as a Jenkins Slave. The continuous integration platform triggers Jenkins build tasks and passes all the configuration parameters required for the build to the pipeline execution engine, which is responsible for executing all build tasks.

[0029] The Kubernetes platform is used to orchestrate Pods to specified worker nodes based on the Pod controller configuration information. The Pod runs Docker containers inside and is also responsible for establishing a stable and reliable TCP connection with the Jenkins platform for subsequent scheduling.

[0030] The Jenkins platform is an open-source continuous integration task distribution system developed in Java. In this application, it serves as a bridge connecting the pipeline execution engine and the sidecar containers, enabling the pipeline execution engine to seamlessly schedule the sidecar containers to pass various build instructions to the build containers.

[0031] The RabbitMQ platform is used to transmit build logs generated in the build container to the pipeline execution engine and display them to the client terminal through the Jenkins platform. This allows users to monitor the execution status of build commands in the build container in real time, preventing builds from going out of control and causing tasks to time out, freeze, or crash.

[0032] like Figure 1 and Figure 2 As shown in the figure, this application discloses a communication method for a custom Docker container used in a continuous integration pipeline, including the following steps: S100 creates custom Docker images, configures image build tasks, and distributes image build tasks.

[0033] First, users create custom Docker images based on their actual application scenarios. These custom Docker images contain all the software environments required for building, such as compilation, deployment, and testing environments.

[0034] Meanwhile, the continuous integration platform contains the build parameters required for the build tasks, including custom Docker image addresses, build commands, and timeouts.

[0035] The S200 creates Pods based on the container orchestration interface and starts sidecar containers from the platform's built-in sidecar Docker image address and build containers from a custom Docker image address through the Pod controller.

[0036] The Kubernetes platform uses the Kubernetes API (Container Orchestration Interface) to create Pods. Kubernetes is an open-source container orchestration platform that provides a range of functions for Docker containers, including deployment, execution, resource scheduling, load balancing, and automatic scaling. A Pod is the smallest unit of resource scheduling in a Kubernetes cluster, used to store a group of containers and their corresponding shared resources, such as PID namespaces, network namespaces, IPC namespaces, and Volumes shared disk volumes, enabling easy communication between multiple containers within a Pod.

[0037] The Pod controller is the middle layer of the Pod. After using the Pod controller, you only need to tell the Pod controller how many and what kind of Pods are needed to create Pods that meet the conditions.

[0038] This application defines two containers within a Pod. One container is a build container generated from a user-provided custom Docker image, originating from a user-configured Docker image address, and executes user-defined build commands. The other container is a sidecar container generated from the built-in Docker container of the continuous integration platform, originating from the built-in Docker container address, and is used to receive various commands from the outside. The sidecar container is embedded in a Docker image repository, containing various environments and tools required for subsequent connection to the Jenkins platform, such as the JDK environment and the Jenkinsagent.jar program.

[0039] Once a Pod starts, two containers run inside it. Through the shared disk volume mechanism provided by the Pod, the two containers within the Pod share the same disk directory on the host machine, and both containers have read and write permissions to the shared directory. Through the shared PID process namespace mechanism provided by the Pod, all containers in the Pod share the same process namespace. One container process can receive signals issued by other container processes in the Pod, so the two container processes in the Pod can communicate with each other by sharing pipe files.

[0040] S300 communicates with the sidecar container and the build container via command and log pipes to enable the build container to execute image build tasks, obtain the corresponding build logs, and send the build logs to the message queue.

[0041] A pipe, also known as a pipe file, is an inter-process communication mechanism. Essentially, it creates a special virtual file in the file system. This file does not actually store data (or temporarily caches a small amount of data), but rather serves as a channel for transferring byte streams between two or more processes. One process writes data to the pipe, and another process reads data from the pipe. Pipes possess one or more of the following characteristics: Following the FIFO (First-In, First-Out) principle, data written to the pipe first will be read first. Half-duplex communication means that by default, pipes can only communicate in one direction. Relying on shared storage, pipe files need to be created in a file system path that can be accessed by multiple processes / containers.

[0042] In this embodiment, there are two named pipes between the sidecar container and the build container: a command pipe and a log pipe. The command pipe is written to by the sidecar container and read from by the build container, and is used by the sidecar container to send image build tasks to the build container. The log pipe is written to by the build container and read from by the sidecar container, and is used by the build container to send the execution results corresponding to the build tasks to the sidecar container.

[0043] Once the sidecar container starts, it receives the image build task and sends it to the build container via the command pipeline. The build container executes the build command based on the image build task and sends the execution result back to the sidecar container via the log pipeline. The sidecar container then sends it to the RabbitMQ message list.

[0044] The S400 reads build logs from the message queue and displays them to the user terminal.

[0045] The build execution engine reads and displays build logs from the RabbitMQ message list in real time and displays them on the user terminal, enabling users to monitor the execution status of build commands in the build container in real time, avoiding build loss of control and causing tasks to time out, freeze, crash, or other issues.

[0046] In summary, users create Docker images based on their actual application scenarios. These Docker images contain all the software environments required for the build process. Then, a Docker environment build task is added to the continuous integration platform. Next, the build task is triggered within the continuous integration platform, using the Jenkins system as the command transmission medium to launch the pipeline execution engine. The pipeline execution engine calls the Kubernetes API to create a Pod controller, which then starts a sidecar container (using a built-in system image) and a build container (generated from a user-defined image). When the sidecar container starts, it automatically connects to the Jenkins Master. The build execution engine transmits build commands to the sidecar container via the Jenkins Master. The sidecar container then transmits the commands to the build container via a command pipeline file. The build container receives and executes the build commands, then sends the execution results to a log pipeline file. The sidecar container reads the build logs from the log pipeline file in real time and sends them to a RabbitMQ message queue. Finally, the build execution engine reads and displays the build logs from the RabbitMQ message queue to the user's terminal in real time.

[0047] This method realizes a continuous integration architecture that features "containerized building environment, distributed task execution, and seamless user experience," effectively improving build speed, environment controllability, and system resource utilization, and significantly optimizing the flexibility and maintainability of the continuous integration platform.

[0048] In other embodiments, creating a custom Docker image, configuring image build tasks, and distributing image build tasks includes the following steps: In other embodiments, a Pod is created based on a container orchestration interface, and a sidecar container derived from a platform-built-in sidecar Docker image address and a build container derived from a custom Docker image address are launched through a Pod controller, including the following steps: S210, Create the build host and configure the input parameters.

[0049] First, build the build host Jenkins Node: This is created based on the Jenkins platform's API interface. It serves as the node connection information for the sidecar container to connect to the Jenkins platform. The required input parameters include: Node name: also known as the build host name, is a unique identifier configured in the actual business scenario to identify the build container. It is used by the subsequent build container scheduling task build_container_job to accurately assign the image build task to the build container. Number of executors: In this embodiment, it is configured to be 1, allowing a maximum of one build task to be executed simultaneously; Working directory: Configured as / data1 / Jenkins, which is the directory where the Jenkins agent file corresponding to the sidecar container is located; Usage: Configure it to "Only build job with label expressions matching thisnode", which means only the container build jobs configured in the current pipeline will be executed; Startup method: Configure it as a launch agent by connecting it to the master, that is, use the Jenkins agent client method in the sidecar container to connect to the subsequent Jenkins service; Node availability: Set to Keep this agent online as much as possible, which means ensuring that the current agent is always online.

[0050] S220, retrieves environment variables based on the created build host.

[0051] After the Jenkins Node is created, the connection string corresponding to the sidecar container is obtained. This string includes two key configuration parameters, which are environment variables: jnlpUrl and secret. jnlpUrl is the network address of the sidecar container to connect to the Jenkins platform, and secret is the connection authentication key information that matches the network address jnlpUrl. These two parameters are used to establish a TCP connection with the Jenkins platform.

[0052] S230 configures a shared disk volume that mounts both the sidecar container and the build container. This shared disk volume is used to store the files corresponding to the command pipeline and the log pipeline.

[0053] The specific configuration for assembling the Pod controller is as follows: First, configure a shared disk volume, which is created on the host machine as a temporary shared space to store the communication pipe files of the Docker containers in the Pod. Both the sidecar container and the build container are mounted on the same shared disk volume. The pipe file is the communication channel between the build container and the sidecar container. The processes of the two containers pass interactive information to each other through the pipe.

[0054] The S240 assembles a sidecar container Pod controller based on the built-in Docker image address, environment variables, connection commands, and shared disk volume, and downloads the sidecar Docker image based on the sidecar container Pod controller.

[0055] Configuring the sidecar container Pod controller requires configuring the built-in Docker image address, the jnlpUrl and secret environment variables required to connect to the Jenkins platform, the connection command between the sidecar container and the Jenkins platform, and the mounted shared disk volume.

[0056] The sidecar container is started via the Pod controller and stored on the Jenkins platform as a build host. Its startup process specifically includes: Obtain the jnlpUrl and secret environment variables for connecting the sidecar container to the Jenkins platform from the sidecar container Pod controller, assemble the Jenkins Node connection string, and the Jenkins platform uses the JNLP protocol to establish a connection with the Jenkins client. The JNLP (Java Network Launching Protocol) protocol enables Java applications to maintain a TCP connection through a simple URL. The Kubernetes platform downloads the sidecar container image through the Pod controller and starts it with the Pod; By starting the command: java -jar agent.jar -jnlpUrl $jnlpUrl -secret $secret -workDir " / data1 / jenkins", the sidecar container establishes a connection with the Jenkins platform, and the Jenkins platform schedules the sidecar container by building the host.

[0057] The S250 assembles and builds a container Pod controller based on a custom Docker image address and a shared disk volume, and downloads a custom Docker image based on the built container Pod controller.

[0058] When configuring the container Pod controller, you need to configure a custom Docker image address and the mounted shared disk volume.

[0059] After creating the Docker image, the user configures the Docker image containing various software environments into the continuous integration pipeline, starts the continuous integration pipeline execution engine, and the pipeline execution engine schedules the Kubernetes API to create build containers.

[0060] S260 creates Pods based on the container orchestration interface and starts the sidecar container and build container through the sidecar container Pod controller and the build container Pod controller.

[0061] After the Pod controller is assembled, the Kubernetes API is called to create Pods, and the sidecar containers and build containers are started through the Pod controller.

[0062] It is important to note that the name of the RabbitMQ log queue is the same as the node name. After creation, it listens for messages in the message queue, extracts build logs from the queue in real time, and pushes them to the user terminal for real-time monitoring of the execution status of build tasks.

[0063] In other embodiments, the following steps are included before performing the image build task: S270, configure the container build scheduling task. The input parameters for the container build scheduling task include the container host name corresponding to the unique identifier of the sidecar container, the message queue name, the build command, and the timeout.

[0064] In the Jenkins platform, define a build container scheduling task `build_container_job`, which is used to bind to a build container and execute user-defined build commands within the build container. Its corresponding input parameters include: Container host name: This is the unique identifier of the sidecar container in the Jenkins platform. Based on this identifier, build tasks are scheduled to the specified sidecar container.

[0065] Log queue name: The name of the RabbitMQ message list, through which the pipeline execution engine and containers send and receive log information.

[0066] Build commands: Shell commands that need to be executed within the container. Furthermore, in order to receive multiple lines of build commands, the build commands need to be base64 encoded.

[0067] Specifically, the "multi-line" nature of shell commands relies on newline characters. While separating different instructions is possible, multiple links in the transmission chain have poor compatibility with special characters such as newlines. Base, on the other hand, is an encoding algorithm based on 64 printable ASCII characters. It can convert any binary data, including special symbols such as newlines, into a pure ASCII string, allowing it to be securely transmitted in APIs, Jenkins parameters, and shell variables.

[0068] Timeout: The maximum time the build command can be executed. If the timeout occurs, the Jenkins platform will automatically terminate the current build task.

[0069] S271, When the container building scheduling task is triggered, the image building task is assigned to the corresponding sidecar container.

[0070] When a build container scheduling task is triggered, the Jenkins platform assigns the build task to a sidecar container, which then performs the task.

[0071] like Figure 2 As shown, in some other embodiments, communication is made between the sidecar container and the build container via command and log pipelines to enable the build container to perform image build tasks, including the following steps: S310, after the sidecar container starts, creates command pipes and log pipes.

[0072] First, after the sidecar container starts, it creates a command pipeline ` / data1 / script / command_queue` and a log pipeline ` / data1 / script / log_queue`. Simultaneously, it creates a shared folder ` / data1 / script / output` on a shared disk volume between the sidecar container and the build container, used for sharing build output files between them.

[0073] S320, the sidecar container receives image build tasks and sends them to the command pipeline. When the build container determines that the command pipeline exists, it continuously retrieves and executes image build tasks from the command pipeline.

[0074] When passing image build tasks, the build pipeline execution engine first passes the image build task to the sidecar container via the basecmd parameter, which contains a user-written shell command. Then, the sidecar container passes the build command to the command pipeline. Finally, the build container retrieves and executes the image build task from the command pipeline, completing the continuous integration build task within the container.

[0075] like Figure 2 As shown, in some other embodiments, obtaining the corresponding build logs and sending them to the message queue includes the following steps: S330, after the image build task is completed, the build container generates build logs based on the execution results and sends them to the log pipeline. The sidecar container reads the build logs from the log pipeline and sends them to the message queue.

[0076] Before executing the image build task, the container must first determine whether the corresponding pipeline files exist, that is, whether the sidecar container has completed the creation of command pipelines and log pipelines.

[0077] If it does not exist, then wait; If present, the build container continuously retrieves and executes user-defined image build tasks from the command pipeline, and sends the execution results of the image build tasks to the log pipeline in the form of build logs.

[0078] like Figure 3 As shown, after the image build task is completed, the sidecar container reads the build logs in the log pipeline and pushes them to the specified RabbitMQ message queue. The pipeline execution engine listens to the specific RabbitMQ message queue and retrieves the build logs from it, and then sends the build logs to the user terminal in a unified manner.

[0079] The process of building a container and executing an image building task, taking the construction of a container for compiling Java code as an example, includes the following: Download the Java code to be compiled from the remote SVN repository to the local directory X1 of the build container, switch the working directory X2, execute the code compilation and packaging tasks, and after the compilation task is completed, copy the compiled binary results to the shared folder between the build container and the sidecar container.

[0080] The sidecar container and the build container communicate with each other via inter-process communication using pipe files. As a result, the data exchange efficiency is high. The sidecar container can send image build tasks to the build container in real time, and the build container can send build logs to the sidecar container in real time.

[0081] In other embodiments, the following steps are also included: S340, after the image build task is completed, the build container sends an end marker to the log pipeline.

[0082] S350: After the sidecar container reads the end marker in the log pipeline, it ends the image build task.

[0083] In order to avoid the image build task being triggered repeatedly and to inform the terminal that the build task has been completed, in this embodiment of the application, after the build container has completely executed the user-defined image build task, an end flag is sent to the log pipeline. After the sidecar container receives the end flag, it considers the task to be completed and ends the image build task.

[0084] In other embodiments, after the image build task is completed, the following steps are also included: S360: The sidecar container compresses all files in the shared disk volume to obtain a compressed package, and uploads the compressed package to the storage server. The shared disk volume contains the build output files shared between the sidecar container and the build container.

[0085] Once the sidecar container receives the end marker, it compresses all deliverable files in the shared folder, uploads the compressed package to the Object Storage Service (OSS) server, and then notifies the continuous integration platform of the URL generated after the upload, thus completing the continuous integration build tasks within all containers.

[0086] Furthermore, the technical solution of this application has the following application scenarios: Scenario 1: Compiling Domestically Produced Software In continuous integration pipelines, for software projects that need to be built in a domestically developed environment, it is usually necessary to deploy a dedicated domestic compilation server or adjust the main build environment to adapt to the corresponding dependencies and toolchains. This approach is not only complex and costly to configure, but it also affects the overall stability and versatility of the pipeline.

[0087] Based on the method of this application, it is possible to compile and build domestic software directly by configuring a domestic Docker image without modifying the main build service runtime environment.

[0088] Specific process: 1. Users configure the repository address of domestically compiled images in the continuous integration platform; 2. The pipeline execution engine pulls the image according to the configuration, starts the container, and executes the compilation instructions inside the container; 3. Build a system for real-time transmission of logs and artifacts back to the mainstream pipeline environment; 4. Once the container is built, it is destroyed, and resources are released immediately.

[0089] Technical effects: Zero intrusion: No need to modify the runtime environment of the main build service; High portability: Allows for quick switching between different domestically developed compilation environments on any server; High resource utilization: Containers are created and destroyed on demand, saving server resources; Strong unified management: Domestically produced compilation images can be centrally managed and version controlled, ensuring consistency of the build environment; Reduced maintenance costs: There is no longer a need to maintain multiple physical or virtual machine environments with different operating systems.

[0090] Scenario 2: Isolation and reuse of complex multi-version dependency environments, such as isolation of multi-version C++ build environments: In the software development process, especially for projects using compiled languages ​​such as C / C++, Go, and Rust, there are often many differences in the versions of dependent libraries and compilation toolchains. For example, different projects may depend on different versions of GCC, Clang compilers, or third-party dynamic libraries (such as OpenSSL, Boost, Protobuf, etc.).

[0091] In traditional continuous integration environments, if all projects share the same build node, different dependency versions are prone to interfering with each other, leading to build failures or unreproducible artifacts; while deploying a separate build server for each project will cause environmental redundancy and waste of resources.

[0092] The custom Docker container communication method provided by this invention can automatically load the corresponding version of the build image for different projects, achieving complete isolation and version control of the build environment.

[0093] Specific process: 1. Users configure corresponding Docker build images for different projects in the continuous integration platform, such as standard compilation images like gcc:9.3, gcc:11.2, clang:15, etc.; or custom-packaged internal C++ build images containing specific dependency libraries and compilation scripts.

[0094] 2. When the pipeline is executed, the system automatically pulls and starts the corresponding image; 3. All build, linking, and packaging operations are completed within this container; 4. Build logs and generated artifacts are returned to the main waterline in real time; 5. Once the container is built, it is destroyed and the environment is automatically cleaned up to avoid dependency residue.

[0095] Technical effects: Complete environment isolation: Dependency versions between different projects do not affect each other; Version controllability: Precisely specify the build toolchain version through image tags; Consistency Guarantee: The build environment is reusable and traceable, ensuring stable and consistent build results; Quick switching: The pipeline can flexibly switch between multiple version compilation environments; Resource savings: No need to maintain a separate server or virtual machine environment for each project.

[0096] Scenario 3: Automated Testing Scenario In continuous integration pipelines, automated testing is a crucial step in ensuring software quality. Different projects often rely on different versions of runtime environments, frameworks, and testing tools (such as Python, Node.js, Java, Go, C++, etc.), and also require specific dependency packages, databases, message queues, or middleware.

[0097] In traditional solutions, a large number of dependent components often need to be pre-installed on the build server to adapt to the testing environment. This not only makes the environment complex and difficult to maintain, but also easily leads to distorted test results due to dependency conflicts or version inconsistencies.

[0098] The communication method based on a custom Docker container provided by this invention can start automated test containers on demand in a continuous integration pipeline, realize the dynamic creation, isolated execution and automatic recycling of test environments, thereby ensuring the independence and reproducibility of the testing process.

[0099] Specific process: 1. Users configure automated test images in the continuous integration platform, such as images for integrating unit testing frameworks (e.g., pytest, jest, junit, etc.). 2. Images that integrate web testing tools (such as Selenium, Playwright, Cypress, etc.), or custom images that include a complete testing dependency environment; 3. The pipeline execution engine starts the corresponding test container according to the configuration; 4. Execute test cases, performance test scripts, or security scan tasks within the container; 5. Real-time transmission of test logs, reports, and results; 6. After the test is completed, the container will be automatically destroyed to release resources and ensure a clean main environment.

[0100] Technical effects: The testing environment is completely isolated: testing tasks from different projects and at different stages do not interfere with each other; High reproducibility: The same image environment can be reused multiple times to ensure consistent test results; Dynamic allocation on demand: Supports concurrent execution of multiple test tasks, improving testing efficiency; Reduce maintenance costs: No need to repeatedly configure dependent environments on the server; Flexible scalability: Supports containerized integration of complex testing scenarios (such as UI automation, API testing, compatibility verification, etc.).

[0101] like Figure 4 As shown, this application also discloses a communication system for a custom Docker container used in a continuous integration pipeline, comprising: A continuous integration platform for creating custom Docker images and configuring image build tasks; The Jenkins platform is used to receive and distribute image build tasks. The Kubernetes platform is used to create Pods based on the container orchestration interface, and to launch sidecar containers from the platform's built-in sidecar Docker image address and build containers from custom Docker image addresses through the Pod controller. The pipeline execution engine is used to obtain image build tasks issued by the Jenkins platform. It communicates with the command pipeline and log pipeline connected between the sidecar container and the build container to enable the build container to execute image build tasks, obtain the corresponding build logs, and send the build logs to the message queue. The RabbitMQ platform is used to read build logs from a message queue and display them to the user's terminal.

[0102] The implementation principle is as follows: Users create Docker images based on their actual application scenarios. These Docker images contain all the software environments required for the build process. Then, a Docker environment build task is added to the continuous integration platform. Next, the build task is triggered within the continuous integration platform, using the Jenkins system as the command transmission medium to launch the pipeline execution engine. The pipeline execution engine calls the Kubernetes API to create a Pod controller, which then starts a sidecar container (using a built-in system image) and a build container (generated from a user-defined image). When the sidecar container starts, it automatically connects to the Jenkins Master. The build execution engine transmits build commands to the sidecar container via the Jenkins Master. The sidecar container then transmits the commands to the build container via a command pipeline file. The build container receives and executes the build commands, then sends the execution results to a log pipeline file. The sidecar container reads the build logs from the log pipeline file in real time and sends them to a RabbitMQ message queue. Finally, the build execution engine reads and displays the build logs from the RabbitMQ message queue to the user's terminal in real time.

[0103] It should be understood that although the steps in the flowcharts in the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise expressly stated herein, there is no strict order in which these steps are performed, and they may be performed in other orders.

[0104] The above are all preferred embodiments of this application and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A communication method for a custom Docker container used in a continuous integration pipeline, characterized in that, Includes the following steps: Create a custom Docker image, configure the image build task, and distribute the image build task. Pods are created based on the container orchestration interface, and the Pod controller starts sidecar containers from the platform's built-in sidecar Docker image address and build containers from a custom Docker image address, respectively. Communication is achieved through the command and log pipelines connecting the sidecar container and the build container, enabling the build container to execute the image build task, obtain the corresponding build logs, and send them to the message queue. The build log is read from the message queue and displayed to the user terminal.

2. The communication method for a custom Docker container used in a continuous integration pipeline according to claim 1, characterized in that, Creating a custom Docker image, configuring the image build task, and distributing the image build task includes the following steps: obtaining all the software environment required for building the custom Docker image; Configure a custom Docker image address, build command, and timeout to generate the image build task.

3. The communication method for a custom Docker container used in a continuous integration pipeline according to claim 1, characterized in that, The process involves creating a Pod based on a container orchestration interface, and then launching a sidecar container from a platform-built-in sidecar Docker image address and a build container from a custom Docker image address via the Pod controller. This includes the following steps: Create a build host and configure the input parameters, including node name, number of executors, working directory, usage method, startup method, and node availability; Obtain environment variables based on the created host; Configure a shared disk volume that mounts the sidecar container and the build container together. The shared disk volume is used to store the files corresponding to the command pipeline and the log pipeline. Assemble the sidecar container Pod controller based on the built-in Docker image address, the environment variables, the connection command, and the shared disk volume, and download the sidecar Docker image according to the sidecar container Pod controller; Based on the custom Docker image address and the shared disk volume, a container Pod controller is assembled and built, and the custom Docker image is downloaded based on the built container Pod controller; A Pod is created based on the container orchestration interface, and the sidecar container and the build container are started through the sidecar container Pod controller and the build container Pod controller.

4. The communication method for a custom Docker container used in a continuous integration pipeline according to claim 2, characterized in that, Before executing the image build task, the following steps are also included: Configure a container build scheduling task, the input parameters of which include the container host name corresponding to the unique identifier of the sidecar container, the message queue name, the build command, and the timeout. When the container building scheduling task is triggered, the image building task is assigned to the corresponding sidecar container.

5. The communication method for a custom Docker container used in a continuous integration pipeline according to claim 4, characterized in that, The input parameters corresponding to the container scheduling task include: base64 encoding of the build command; If the execution time of the build command exceeds the specified time limit, the current image build task is terminated.

6. The communication method for a custom Docker container used in a continuous integration pipeline according to claim 1, characterized in that, The sidecar container and the build container communicate with each other via command and log pipes to enable the build container to execute the image build task, including the following steps: after the sidecar container starts, the command pipe and the log pipe are created; The sidecar container receives the image building task and sends it to the command pipeline. When the build container determines that the command pipeline exists, it continuously retrieves and executes the image building task from the command pipeline.

7. The communication method for a custom Docker container used in a continuous integration pipeline according to claim 1, characterized in that, Retrieving the corresponding build logs and sending them to the message queue includes the following steps: After the image building task is completed, the build container generates a build log based on the execution result and sends it to the log pipeline. The sidecar container reads the build log in the log pipeline and sends it to the message queue.

8. The communication method for a custom Docker container in a continuous integration pipeline according to claim 6 or 7, characterized in that, It also includes the following steps: After the image build task is completed, the build container sends an end marker to the log pipeline; After the sidecar container reads the end marker in the log pipeline, it terminates the image building task.

9. The communication method for a custom Docker container in a continuous integration pipeline according to claim 8, characterized in that, After the image building task is completed, the following steps are also included: The sidecar container compresses all files in the shared disk volume to obtain a compressed package, and uploads the compressed package to the storage server. The shared disk volume contains the build results files shared between the sidecar container and the build container.

10. A communication system for custom Docker containers used in continuous integration pipelines, characterized in that, For implementing the method as described in any one of claims 1-9, comprising: A continuous integration platform for creating custom Docker images and configuring image build tasks; The Jenkins platform is used to receive and distribute the image build tasks. The Kubernetes platform is used to create Pods based on the container orchestration interface, and to launch sidecar containers from the platform's built-in sidecar Docker image address and build containers from custom Docker image addresses through the Pod controller. The pipeline execution engine is used to obtain the image build task issued by the Jenkins platform, communicate with the command pipeline and log pipeline connected between the sidecar container and the build container to enable the build container to execute the image build task, obtain the corresponding build log, and send the build log to the message queue. The RabbitMQ platform is used to read the build logs from the message queue and display them to the user terminal.

Images