Container-based different-frame hybrid compiling method and system

Abstract

The invention discloses a container-based different-frame mixed compiling method and system. Compiling environments of different languages and different architectures are output as compiling mirror images of different languages, the compiling mirror images are built in a CICD platform, a user drags and arranges a compiling task, the task is issued to a job-center through Pipeline, the job-center assembles a Tekton task and transmits the Tekton task to a Tekton-agency, and the Tekton task and the Tekton-agency are combined to form the different-frame mixed compiling method and system. The Tekton-agent generates a task through a K8s API (Application Program Interface), and the kube-scheduler schedules the Pod to execute the Pod. Based on the characteristics of the K8s cluster and the container, the editing environments of various languages are preset in a mirroring mode in combination with the scheduling tool Tekton, the compiling task is completed in the container, the problems that the resource utilization rate is low, and the compiling environment is unique and unsafe are solved, and the working efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of language compilation technology, and in particular to a container-based heterogeneous hybrid compilation method and system. Background Art

[0002] In modern software development, cross-architecture compilation is a key link to support multi-platform release and compatibility. In order to adapt to the software requirements of different hardware architectures, development teams usually choose to compile through multiple machines with different architectures, or complete related tasks through offline host compilation. However, due to various technical limitations, this cross-architecture compilation method has not yet been fully integrated into the continuous integration / continuous delivery (CICD) automation process, which has brought many challenges to development and operation and maintenance.

[0003] Although the CICD process has been widely used in modern software development and can greatly improve development efficiency and release frequency, the automated integration of cross-architecture compilation processes still faces many challenges. Currently, most CICD tools and platforms (such as Jenkins, GitLab CI, CircleCI, etc.) are usually more mature and optimized for the compilation of the same architecture, and can automatically handle code building, testing, deployment and other links. However, in the scenario of multiple architectures, the integration of CICD processes faces the following technical difficulties:

[0004] (1) Low resource utilization. When there is an environment conflict, one machine can only serve as the compilation environment for one project, resulting in a huge waste of resources.

[0005] (2) The compilation environment cannot be completely isolated, which poses security issues.

[0006] (3) The compilation of each project requires the establishment of a corresponding compilation environment, which results in low overall efficiency. Summary of the invention

[0007] To this end, the present application provides a container-based heterogeneous hybrid compilation method and system to solve the problems of resource waste, low security and low efficiency in cross-architecture compilation in the prior art.

[0008] In order to achieve the above objectives, this application provides the following technical solutions:

[0009] In a first aspect, a container-based heterogeneous hybrid compilation method outputs compilation environments of different languages ​​and different architectures as compilation images of different languages ​​in advance, and builds them into a CICD platform. The container-based heterogeneous hybrid compilation method is applied to the CICD platform, including:

[0010] Receive compilation tasks arranged by users through drag and drop;

[0011] Send the compilation task to the job-center service through the Pipeline service;

[0012] The job-center assembles the Task and TaskRun required for the Tekton scheduling task according to the compilation task requirements and sends them to Tekton-agency, then determines the architecture of the scheduling machine where the compilation task needs to run, and sets the corresponding nodeSelector label according to the determined architecture; the Tekton-agency receives the assembled Task and TaskRun, and generates Task and TaskRun in the k8s cluster through the Kubernetes API Server; the kube-scheduler listens to new Pod resources, and selects the most suitable Node node for scheduling according to the pre-selected strategy and the preferred strategy, receives the scheduling instruction according to the kubelet on the selected Node node, starts the Pod through the container runtime, and executes the compilation task.

[0013] Preferably, the job-center determines the architecture of the scheduling machine on which the compilation task needs to run according to the plug-in identifier in the compilation task.

[0014] Preferably, the corresponding nodeSelector tag is set according to the determined architecture, specifically: if the compilation task needs to be run on an ARM architecture machine, kubernetes.io / arch=arm64 is set; if the compilation task needs to be run on an x86 architecture machine, kubernetes.io / arch=amd64 is set.

[0015] Preferably, the job-center carries the callback address when sending Task and TaskRun to Tekton-agency.

[0016] Preferably, the kube-scheduler monitors new Pod resources through the apiserver watch API.

[0017] Preferably, the new Pod resource is generated by the Tekton controller calling the Kubernetes API Server.

[0018] Preferably, the new Pod resource is written into etcd through apiserver.

[0019] Preferably, the most suitable Node node is selected for scheduling according to the pre-selection strategy and the preferred strategy. Specifically, kube-scheduler first filters out the Nodes that do not meet the pre-selection strategy according to the pre-selection strategy, and then scores and ranks the pre-selected Nodes according to the priority strategy, and selects the Node with the highest score as the most suitable Node node for scheduling.

[0020] Preferably, when the pre-selected Nodes are scored and ranked, the Node with more abundant resources and smaller load has a higher ranking.

[0021] In a second aspect, a container-based heterogeneous hybrid compilation system includes:

[0022] The compilation environment containerization module is used to output the compilation environments of different languages ​​and architectures into compilation images of different languages ​​in advance and build them into the CICD platform;

[0023] A compilation task receiving module is used to receive compilation tasks arranged by users by dragging and dropping;

[0024] The task delivery module is used to deliver the compilation task to the job-center service through the Pipeline service;

[0025] The node selection and task scheduling module is used by the job-center to assemble the Task and TaskRun required for the Tekton scheduling task according to the compilation task requirements and send them to the Tekton-agency, and then determine the architecture of the scheduling machine where the compilation task needs to run, and set the corresponding nodeSelector label according to the determined architecture; the Tekton-agency receives the assembled Task and TaskRun, and generates Task and TaskRun in the k8s cluster through the Kubernetes API Server; the kube-scheduler listens to new Pod resources, and selects the most suitable Node node for scheduling according to the pre-selected strategy and the preferred strategy, receives the scheduling instruction according to the kubelet on the selected Node node, starts the Pod through the container runtime, and executes the compilation task.

[0026] Compared with the prior art, this application has at least the following beneficial effects:

[0027] The present application provides a container-based heterogeneous hybrid compilation method and system. Based on the characteristics of K8s clusters and containers, and combined with the capabilities of the open source scheduling tool Tekton, the editing environments of various languages ​​are preset by mirroring. When the task is issued, Tekton schedules the task to the specified node of K8s and completes the compilation task in the container. When the task is executed, the resources can be automatically released, which solves the problems of low resource utilization, uniqueness and insecurity of the compilation environment, and improves work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more intuitively illustrate the prior art and the present application, exemplary drawings are given below. It should be understood that the specific shapes and structures shown in the drawings should not generally be regarded as limiting conditions for implementing the present application; for example, those skilled in the art are capable of easily making conventional adjustments or further optimizations to the addition / reduction / attribution division, specific shapes, positional relationships, connection methods, dimensional ratios, etc. of certain units (components) based on the technical concepts and exemplary drawings disclosed in the present application.

[0029] Figure 1 A flow chart of a container-based heterogeneous hybrid compilation method provided in Example 1 of the present application;

[0030] Figure 2 A schematic diagram of the structure of a container-based heterogeneous hybrid compilation method provided in Example 1 of the present application. DETAILED DESCRIPTION

[0031] The present application is further described below in detail through specific embodiments in conjunction with the accompanying drawings.

[0032] In the description of this application: unless otherwise specified, "plurality" means two or more. The terms "first", "second", "third", etc. in this application are intended to distinguish the objects referred to, and do not have special meanings in terms of technical connotations (for example, they should not be understood as emphasizing the importance or order, etc.). Expressions such as "including", "comprising", "having", etc. also mean "not limited to" (certain units, components, materials, steps, etc.).

[0033] The terms such as "upper", "lower", "left", "right", "middle", etc. cited in this application are usually used to indicate the general relative position relationship for the purpose of intuitive understanding by referring to the drawings, and are not absolute limitations on the position relationship in the actual product.

[0034] Embodiment 1

[0035] See also Figure 1 and Figure 2This embodiment provides a container-based heterogeneous hybrid compilation method, which outputs compilation environments of different languages ​​and different architectures as compilation images of different languages ​​in advance and builds them into the CICD platform. The container-based heterogeneous hybrid compilation method is applied to the CICD platform, including:

[0036] S1: receiving compilation tasks arranged by users by dragging and dropping;

[0037] Specifically, the user arranges the compilation tasks by dragging and dropping in the web ui (Web user interface). In this embodiment, the user side can specify the compilation architecture environment, and there is no need for human intervention to build the compilation environment, which improves work efficiency.

[0038] S2: Send the compilation task to the job-center service through the Pipeline service;

[0039] Specifically, when a compilation task is issued, whether it is triggered automatically or manually, the task will be issued to the Pipeline service, and the Pipeline service will issue the relevant information required for the current compilation task to the job-center service.

[0040] S3: job-center assembles the Task and TaskRun required for Tekton scheduling tasks according to the compilation task requirements and sends them to Tekton-agency, then determines the architecture of the scheduling machine where the compilation task needs to run, and sets the corresponding nodeSelector label according to the determined architecture; Tekton-agency receives the assembled Task and TaskRun, and generates Task and TaskRun in the k8s cluster through the Kubernetes API Server; kube-scheduler listens to new Pod resources, and selects the most suitable Node node for scheduling according to the pre-selected strategy and the preferred strategy, receives the scheduling instruction according to the kubelet on the selected Node node, starts the Pod through the container runtime, and executes the compilation task.

[0041] In this step, job-center specifically determines the architecture of the scheduling machine where the compilation task needs to run according to the plug-in identifier in the compilation task.

[0042] In this step, set the corresponding nodeSelector label according to the determined architecture. Specifically, if the compilation task needs to be run on an ARM architecture machine, set kubernetes.io / arch=arm64; if the compilation task needs to be run on an x86 architecture machine, set kubernetes.io / arch=amd64, for example:

[0043] X86 architecture environment:

[0044] nodeSelector:

[0045] kubernetes.io / arch: amd64

[0046] Arm architecture environment:

[0047] nodeSelector:

[0048] kubernetes.io / arch: arm64.

[0049] In this step, job-center sends the assembled Task (Tekton relies on the K8s environment to complete the execution of tasks. Task is a series of steps to execute commands, and each step is actually executed by a Pod. For example: compiling code, building images, pushing images, etc. Task only defines a template) and TaskRun (TaskRun represents an actual operation) to tekton-agency, and carries the callback address (used for changes in task execution status, callback to job-center). Tekton-agency receives the assembled Task and TaskRun, and calls Tekton's custom resource API. These APIs generate Task and TaskRun resources in the Kubernetes cluster through the Kubernetes API Server.

[0050] In this step, the Tekton controller calls the Kubernetes API Server to generate a new Pod, and then the apiserver receives the new Pod data and writes the data to etcd. Since kube-scheduler has been monitoring resource changes through the apiserver watch API, when a new Pod is found, but the Pod has not been bound to any Node node, kube-scheduler schedules and selects a suitable Node node. At this time, Pods select the most suitable Node node for the Pod one by one and write the binding information to etcd. The entire scheduling process is divided into two steps: preselection (Predicates) and priority (Priorities):

[0051] Preselection (Predicates): kube-scheduler filters out Nodes that do not meet the policy based on the preselection strategy (xxx Predicates). At this time, the nodeSelector labels kubernetes.io / arch: amd64 and kubernetes.io / arch: arm64 will be used to schedule the Pod to the node of the specified architecture.

[0052] Priorities: Priorities will score and rank the pre-selected Nodes according to the priority strategy (xxx Priority) and select the Node with the highest score. For example, the Node with more abundant resources and smaller load may have a higher ranking.

[0053] This step binds the Pod to the target Node, and then updates the message to etcd. At this time, the kubelet on the same target Node detects that a new Pod has been scheduled through the apiserver watch API, and it passes the relevant data of the Pod to the subsequent container runtime (container runtime), such as Docker, to let them run the Pod. In addition, the kubelet will obtain the status of the Pod through the container runtime, and then update it to the apiserver, and of course, it will also be written to etcd in the end.

[0054] In this embodiment, once the task is completed, the corresponding resources will be released; after the task is completed, the task status will be called back to the task issuing platform, and the task status will be intuitively reflected on the page through websocket and fed back to the user.

[0055] The container-based heterogeneous hybrid compilation method provided in this embodiment has the following advantages:

[0056] (1) On the CICD platform, users can complete heterogeneous compilation in multiple languages ​​simply by dragging and dropping configurations. There is no need to build a separate compilation environment, which reduces the complexity of user use and improves user experience and work efficiency.

[0057] (2) Relying on the characteristics of K8s cluster, open source component Tekon and container, the resource reusability and security of the compilation environment are guaranteed, greatly improving resource utilization. Specifically: the scheduling environment relies on the K8s cluster, using the K8sNodeSelector feature, according to user configuration, to schedule tasks to the specified K8s architecture Node node, and complete the compilation task in the container, ensuring the isolation and security of the environment; the underlying scheduling uses the open source component Tekton to execute tasks by defining "TaskRun". Once the task is completed, the corresponding resources will be released, achieving resource sharing and improving resource utilization.

[0058] In summary, the container-based heterogeneous hybrid compilation method provided in this embodiment is based on the characteristics of the K8s cluster and the container, and combines the capabilities of the open source scheduling tool Tekton to preset the editing environments of various languages ​​by mirroring; when a specific task is issued, the task is scheduled to the specified node of K8s through Tekton, and the compilation task is completed in the container. When the task is completed, the resources can be automatically released, which solves the problems of low resource utilization and the security and uniqueness of the compilation environment, thereby improving work efficiency.

[0059] Embodiment 2

[0060] This embodiment provides a container-based heterogeneous hybrid compilation system, including:

[0061] The compilation environment containerization module is used to output the compilation environments of different languages ​​and architectures into compilation images of different languages ​​in advance and build them into the CICD platform;

[0062] A compilation task receiving module is used to receive compilation tasks arranged by users by dragging and dropping;

[0063] The task delivery module is used to deliver the compilation task to the job-center service through the Pipeline service;

[0064] The node selection and task scheduling module is used by the job-center to assemble the Task and TaskRun required for the Tekton scheduling task according to the compilation task requirements and send them to the Tekton-agency, and then determine the architecture of the scheduling machine where the compilation task needs to run, and set the corresponding nodeSelector label according to the determined architecture; the Tekton-agency receives the assembled Task and TaskRun, and generates Task and TaskRun in the k8s cluster through the Kubernetes API Server; the kube-scheduler listens to new Pod resources, and selects the most suitable Node node for scheduling according to the pre-selected strategy and the preferred strategy, receives the scheduling instruction according to the kubelet on the selected Node node, starts the Pod through the container runtime, and executes the compilation task.

[0065] For the specific implementation content of each module in a container-based heterogeneous hybrid compilation system, please refer to the above definition of a container-based heterogeneous hybrid compilation method, which will not be repeated here.

[0066] The technical features of the above embodiments may be arbitrarily combined (as long as there is no contradiction in the combination of these technical features). In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described; these embodiments that are not explicitly written should also be considered to be within the scope of this specification.

Claims

1. A container-based heterogeneous hybrid compilation method, characterized in that: The compilation environments of different languages ​​and architectures are output as compilation images of different languages ​​in advance and built into the CICD platform. The container-based heterogeneous hybrid compilation method is applied to the CICD platform, including: Receive compilation tasks arranged by users through drag and drop; Send the compilation task to the job-center service through the Pipeline service; The job-center assembles the Task and TaskRun required for the Tekton scheduling task according to the compilation task requirements and sends them to Tekton-agency, then determines the architecture of the scheduling machine where the compilation task needs to run, and sets the corresponding nodeSelector label according to the determined architecture; the Tekton-agency receives the assembled Task and TaskRun, and generates Task and TaskRun in the k8s cluster through the Kubernetes API Server; the kube-scheduler listens to new Pod resources, and selects the most suitable Node node for scheduling according to the pre-selected strategy and the preferred strategy, receives the scheduling instruction according to the kubelet on the selected Node node, starts the Pod through the container runtime, and executes the compilation task.

2. The container-based heterogeneous hybrid compilation method according to claim 1, characterized in that: The job-center determines the architecture of the scheduling machine on which the compilation task needs to run according to the plug-in identifier in the compilation task.

3. The container-based heterogeneous hybrid compilation method according to claim 2, characterized in that: Set the corresponding nodeSelector label according to the determined architecture. Specifically, if the compilation task needs to be run on an ARM architecture machine, set kubernetes.io / arch=arm64; if the compilation task needs to be run on an x86 architecture machine, set kubernetes.io / arch=amd64.

4. The container-based heterogeneous hybrid compilation method according to claim 1, characterized in that: The job-center carries the callback address when sending Task and TaskRun to Tekton-agency.

5. The container-based heterogeneous hybrid compilation method according to claim 1, characterized in that: The kube-scheduler monitors new Pod resources through the apiserver watch API.

6. The container-based heterogeneous hybrid compilation method according to claim 1, characterized in that: The new Pod resource is generated by the Tekton controller calling the Kubernetes API Server.

7. The container-based heterogeneous hybrid compilation method according to claim 1, characterized in that: The new Pod resources are written to etcd through the apiserver.

8. The container-based heterogeneous hybrid compilation method according to claim 1, characterized in that: The method of selecting the most suitable Node node for scheduling according to the pre-selection strategy and the preferred strategy is as follows: kube-scheduler first filters out the Nodes that do not meet the pre-selection strategy according to the pre-selection strategy, then scores and ranks the pre-selected Nodes according to the preferred strategy, and selects the Node with the highest score as the most suitable Node node for scheduling.

9. The container-based heterogeneous hybrid compilation method according to claim 8, characterized in that: When the pre-selected Nodes are scored and ranked, the Node with more abundant resources and smaller load has a higher ranking.

10. A container-based hybrid compilation system, characterized in that: include: The compilation environment containerization module is used to output the compilation environments of different languages ​​and architectures into compilation images of different languages ​​in advance and build them into the CICD platform; A compilation task receiving module is used to receive compilation tasks arranged by users by dragging and dropping; The task delivery module is used to deliver the compilation task to the job-center service through the Pipeline service; The node selection and task scheduling module is used by the job-center to assemble the Task and TaskRun required for the Tekton scheduling task according to the compilation task requirements and send them to the Tekton-agency, and then determine the architecture of the scheduling machine where the compilation task needs to run, and set the corresponding nodeSelector label according to the determined architecture; the Tekton-agency receives the assembled Task and TaskRun, and generates Task and TaskRun in the k8s cluster through the Kubernetes API Server; the kube-scheduler listens to new Pod resources, and selects the most suitable Node node for scheduling according to the pre-selected strategy and the preferred strategy, receives the scheduling instruction according to the kubelet on the selected Node node, starts the Pod through the container runtime, and executes the compilation task.

Images