Compilation construction method and device based on container multiplexing
The container reuse-based compilation method optimizes build efficiency by reusing pod containers for subsequent builds, addressing the inefficiencies of traditional host machine builds and virtualization, thereby improving resource utilization and concurrency.
Patent Information
- Application Number
- CN202411267955.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-10
- Publication Date
- 2024-12-27
AI Technical Summary
Traditional builds on host machines suffer from low resource utilization due to the inability to isolate and perform multiple builds simultaneously, leading to inefficient use of resources and long build times, especially when virtualization is employed.
A container reuse-based compilation and construction method that utilizes a build management platform, scheduler, and pod manager to identify and reuse retained pod containers for subsequent builds, reducing the need to obtain code and scripts from scratch, and optimizing resource allocation.
This approach enhances build efficiency by allowing concurrent builds, fully utilizing host machine resources, and significantly reducing build times by reusing existing containers.
Smart Images
Figure CN119201127A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of compilation technology, and in particular to a compilation construction method and device based on container reuse. Background Art
[0002] Traditional builds are generally performed on the host machine, using all the host machine's resources for the build. Although the build efficiency is the highest, it is unable to isolate the builds of different applications, and only one build can be performed at the same time, resulting in low resource utilization. That is, the resources required for the build are usually less than the upper limit of the resources that the host machine can provide, resulting in the host machine's resources not being fully utilized.
[0003] In order to make full use of the host machine's resources, virtualization technology (such as virtual machines, containers, etc.) can be used to allocate appropriate resources to different builds, and support simultaneous builds to improve overall resource utilization and build efficiency. The higher the degree of virtualization, the more virtualized environments that can exist at the same time, and the higher the concurrency of the build. Virtualization technology naturally provides an isolated environment so that different builds will not affect each other, ensuring the reliability of the build product. However, when using virtualization technology to build, there are the following problems:
[0004] 1. Each build requires obtaining code, build scripts and other content. When there is a lot of content, it will take a long time;
[0005] 2. Each build starts from scratch and requires downloading dependencies. The intermediate products of the previous build (such as cache) cannot provide support for the next build. Summary of the invention
[0006] In view of the above problems, embodiments of the present application are proposed to provide a container reuse-based compilation and construction method and device that overcome the above problems or at least partially solve the above problems.
[0007] According to a first aspect of an embodiment of the present application, a compilation and construction method based on container reuse is provided, which includes:
[0008] The build management platform creates a compile and build task based on the build request initiated by the user, sets the status of the compile and build task to the initial status, and stores it in the database;
[0009] The build scheduler periodically queries the database to obtain the compilation and build tasks in the initial state; based on the application compiled by the compilation and build task, it queries the database to see whether there is a retained pod container that matches the application; the retained pod container is determined by the pod manager based on the application compilation frequency and / or cluster resource usage information;
[0010] If so, the build scheduler uses the retained pod container to perform the compilation and build tasks.
[0011] Optionally, the build management platform creates a compile build task according to the build request initiated by the user, sets the state of the compile build task to an initial state, and stores the state in the database, further comprising:
[0012] The build management platform obtains build information based on the build request initiated by the user; the build information includes the source code, image, and build script required for the compiled application;
[0013] The build management platform creates a compile build task according to the build information, sets the state of the compile build task to the initial state, and stores it in the database.
[0014] Optionally, the build scheduler periodically queries the database to obtain the compilation and build tasks in the initial state further including:
[0015] The first process of the build scheduler periodically queries the database to obtain the compilation and build tasks in the initial state in the database;
[0016] If there are multiple compilation and construction tasks in the initial state, the multiple compilation and construction tasks are stored in the construction task list, so that each compilation and construction task is executed in sequence according to the construction task list.
[0017] Optionally, according to the application compiled by the compile build task, querying the database whether there is a retained pod container matching the application further includes:
[0018] According to the application compiled by the compilation and build task, query the retained pod list in the database to determine whether there is a pod container in the retained pod list that meets the preset retention conditions; the preset retention conditions include: a retained pod container associated with the application, and the information of the retained pod container matches the build information, and the retained pod container is determined to be in a survival state;
[0019] If yes, update the status of the retained pod container in the database and set the status to in use;
[0020] If the update is successful, it is determined that there is a retained pod container that matches the application.
[0021] Optionally, if it is determined that there is no pod container that meets the preset retention condition in the retained pod list, or the status update of the retained pod container in the database fails, the method further includes:
[0022] A new pod container is built according to the build information. The build scheduler uses the new pod container to perform the compilation and build task, records the association between the new pod container and the compilation and build task, and sets the status of the compilation and build task to in progress.
[0023] After the compilation and build task is completed, the build scheduler sets the status of the compilation and build task to be cleaned up.
[0024] Optionally, the build scheduler uses the retained pod container to execute the compilation and build task further including:
[0025] The build scheduler records the association between the retained pod container and the compile and build task, and sets the status of the compile and build task to in progress;
[0026] After the compilation and build task is completed, the build scheduler sets the status of the compilation and build task to be cleaned up.
[0027] Optionally, the method further comprises:
[0028] The second process of the scheduler is constructed to query the database regularly to obtain the latest working instance;
[0029] Determine whether the work instance is a work instance for building a scheduler;
[0030] If so, update the heartbeat time of the working instance;
[0031] If not, determine whether the difference between the heartbeat time of the working instance and the current time exceeds a preset threshold;
[0032] If so, update the worker instance to the worker instance that builds the scheduler.
[0033] Optionally, the method further comprises:
[0034] The third process of the pod manager periodically queries the database to obtain the compilation and construction tasks that are in the status of being cleaned up;
[0035] According to the pod container and / or the associated application associated with the compilation and construction task to be cleaned up, the status of the compilation and construction task to be cleaned up is updated and / or the associated pod container is processed.
[0036] Optionally, according to the pod container and / or the associated application associated with the compile and build task to be cleaned up, updating the status of the compile and build task to be cleaned up and / or processing the associated pod container further includes:
[0037] Determine whether the pod container associated with the compilation and construction task to be cleaned up is alive;
[0038] If not, set the status of the compilation and build task to be cleaned up to cleaned up.
[0039] Optionally, according to the pod container and / or the associated application associated with the compile and build task to be cleaned up, updating the status of the compile and build task to be cleaned up and / or processing the associated pod container further includes:
[0040] Determine whether the application associated with the compilation and construction task to be cleaned is in a preset application list; the application compilation frequency of the application in the preset application list within a preset time exceeds a preset frequency threshold;
[0041] If not, set the status of the compilation and construction task to be cleaned up to cleaned up, and delete the pod container associated with the compilation and construction task to be cleaned up;
[0042] If yes, determine whether the pod container associated with the compilation and build task to be cleaned up is in the retained pod list;
[0043] If yes, set the status of the compilation and build task to be cleaned up to cleaned up, and set the associated pod container to unused;
[0044] If not, query the retained pod list to see whether there is a retained pod container associated with the application;
[0045] If so, set the status of the compilation and build task to be cleaned up to cleaned up; if the status of the retained pod container is in use, delete the pod container associated with the compilation and build task to be cleaned up; if the status of the retained pod container is not in use, delete the retained pod container and add the associated pod container to the retained pod list.
[0046] Optionally, the method further comprises:
[0047] The fourth process of the pod manager periodically queries the database to obtain a list of retained pods and / or obtains cluster resource usage information;
[0048] Sort the retained pod containers in the retained pod list in ascending order based on their usage frequency;
[0049] For each sorted retained pod container, the retained pod container in the retained pod list is processed according to the retained pod list and / or the cluster resource usage information.
[0050] Optionally, for each sorted retained pod container in turn, according to the retained pod list and / or the cluster usage resource information, processing the retained pod container in the retained pod list further includes:
[0051] For each sorted retained pod container, check whether the retained pod container in the cluster is alive.
[0052] If not, delete the remaining pod container;
[0053] If yes, determine whether the status of the retained pod container is in use;
[0054] If not, determine whether the unused time of the retained pod container exceeds the preset unused threshold;
[0055] If yes, delete the remaining pod container.
[0056] Optionally, for each sorted retained pod container in turn, according to the retained pod list and / or the cluster usage resource information, processing the retained pod container in the retained pod list further includes:
[0057] For each sorted retained pod container, the resource information occupied by the retained pod container is combined with the cluster resource usage information to obtain the cluster resource utilization information;
[0058] Determine whether the resource utilization information of the cluster exceeds the cluster utilization threshold;
[0059] If so, delete the retained pod container to release the resource information occupied by the retained pod container.
[0060] According to a second aspect of an embodiment of the present application, a compilation and construction device based on container reuse is provided, comprising:
[0061] The build management platform is used to create a compilation and build task based on the build request initiated by the user, set the status of the compilation and build task to the initial status, and store it in the database;
[0062] The build scheduler is used to query the database regularly to obtain the compilation and build tasks in the initial state; based on the application compiled by the compilation and build task, query the database to see whether there is a retained pod container that matches the application; if so, use the retained pod container to execute the compilation and build task;
[0063] The pod manager is used to determine the retained pod containers based on the application compilation frequency and / or cluster resource usage information.
[0064] According to a third aspect of an embodiment of the present application, there is provided a computing device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;
[0065] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the above-mentioned compilation and construction method based on container reuse.
[0066] According to a fourth aspect of an embodiment of the present application, a computer storage medium is provided, in which at least one executable instruction is stored, and the executable instruction enables a processor to perform operations corresponding to the above-mentioned compilation and construction method based on container reuse.
[0067] According to a fifth aspect of an embodiment of the present application, a computer program product is provided, comprising at least one executable instruction, wherein the executable instruction enables a processor to perform operations corresponding to the above-mentioned compilation and construction method based on container reuse.
[0068] According to the compilation and construction method and device based on container reuse provided by the present application, after the construction management platform constructs the compilation and construction task, the construction scheduler queries the retained pod container that matches the application of the compilation and construction task, and reuses the retained pod container to complete the compilation and construction task. This eliminates the need to obtain the application code, construction scripts, and other content from scratch for each compilation and construction, thereby improving subsequent construction efficiency.
[0069] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0070] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:
[0071] Figure 1 A flowchart of a compilation and construction method based on container reuse according to an embodiment of the present application is shown;
[0072] Figure 2 A schematic diagram of the architecture of the compilation and construction is shown;
[0073] Figure 3 A flowchart of a compilation and construction method based on container reuse according to another embodiment of the present application is shown;
[0074] Figure 4 A schematic diagram of the structure of a compilation and construction device based on container reuse according to an embodiment of the present application is shown;
[0075] Figure 5 A schematic diagram of the structure of a computing device according to an embodiment of the present application is shown. DETAILED DESCRIPTION
[0076] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.
[0077] First, the terms involved in one or more embodiments of the present application are explained.
[0078] Kubernetes: A container orchestration management system used to manage containerized applications on multiple hosts in a cloud platform, referred to as k8s.
[0079] Pod: The smallest unit level managed in k8s, which is a combination of one or more containers.
[0080] Build: The process of generating products or images from source code.
[0081] Build script: A series of commands that a build executes.
[0082] Code repository: a place where codes are stored uniformly.
[0083] Image warehouse: a place for unified storage of images.
[0084] OSS platform: Object Storage Service, object storage platform, a place for unified storage of products, logs and other content.
[0085] Dockerfile: A text file that contains a series of commands to execute the image.
[0086] Figure 1 A flowchart of a container reuse-based compilation and construction method according to an embodiment of the present application is shown. Figure 1 As shown, the method comprises the following steps:
[0087] Step S101: The build management platform creates a compile build task according to a build request initiated by a user, sets the state of the compile build task to an initial state, and stores the state in a database.
[0088] The build management platform is used to create compilation and build tasks, such as Figure 2As shown, users can initiate build requests based on the build management platform. The build management platform provides users with visual operations. Users can create new applications on the build management platform, specify the source code required for the application build, the image used for the build, and the build script, etc. The source code can be the path of the code repository, and the image and build script can be created on the build management platform, or existing ones can be used, such as images stored in the image repository, build scripts stored in the code repository, etc., which are not limited here.
[0089] When a user initiates a build request, if the application has multiple different branches, the user can specify the source code branch of the application used for this compilation and build. If not specified, the latest version of the source code of the application can be used for compilation and build. The build management platform creates a compilation and build task for the application based on the build request, sets the status of the created compilation and build task to the initial status, and stores it in the database to facilitate the subsequent build scheduler to find and execute the compilation and build task.
[0090] The build management platform facilitates user visual operations. It can also obtain and display the real-time log of the pod container during the execution of the compile and build task, making it easy to understand the execution progress of the compile and build task. After the compile and build task is completed, the build management platform can display the complete log of the pod container, making it easy for users to review the build process. Furthermore, the build association platform can also display the products or images of the compile and build, making it easy for users to download and use them.
[0091] Step S102, the build scheduler periodically queries the database to obtain the compilation and construction tasks in the initial state; based on the application compiled by the compilation and construction task, it is queried whether there is a retained pod container matching the application in the database.
[0092] The build management platform and the build scheduler use an asynchronous processing method. After the build management platform creates a compile and build task and stores it in the database, the build scheduler can query the database regularly to query the compile and build tasks in the initial state from the database in order to execute the compile and build tasks. If there are multiple compile and build tasks in the initial state, they can be stored in the build task list first, and the compile and build tasks in the build task list can be built and executed in turn.
[0093] When executing a compilation and build task, you need to select a pod container for it. When building in a pod container, you can select different build environments for different applications to implement a universal build method. When building in a pod container, users can select images, and different images can be used to build applications in different languages. The builds of different applications are isolated from each other to avoid mutual influence or pollution of the build environment. Different applications can be built at the same time, with a high degree of concurrency, which can fully utilize the resources of the host machine, improve build efficiency, and significantly shorten build time.
[0094] When selecting a pod container, in order to avoid having to obtain the application code, build scripts, and other content from scratch each time you build, you can reuse the pod container to improve build efficiency. Specifically, the compilation and build task includes the application to be built, and queries the database to see if there is a retained pod container that matches the application, so that the retained pod container can be reused for compilation and build.
[0095] Here, the retained pod container is determined by the pod manager based on the application compilation frequency and / or cluster resource usage information. For example, if the application is frequently compiled within a period of time, the used pod container can be retained according to the application compilation frequency, or when the resource usage information of the Kubernetes cluster is sufficient, the used pod container can be retained so that the retained pod container can be directly used in the next compilation and build without obtaining the source code, build script, etc. from scratch.
[0096] Step S103: The build scheduler uses the retained pod container to execute the compilation and build task.
[0097] The build scheduler uses the queried retained pod container to execute the compilation and build tasks in the retained pod container to compile and build the application until the build is completed.
[0098] According to the compilation and construction method based on container reuse provided in the present application, after the build management platform builds the compilation and construction task, the build scheduler queries the retained pod container that matches the application of the compilation and construction task, and reuses the retained pod container to complete the compilation and construction task. This eliminates the need to obtain the application code, build scripts, and other content from scratch for each compilation and construction, thereby improving subsequent construction efficiency.
[0099] Figure 3 A flowchart of a container reuse-based compilation and construction method according to an embodiment of the present application is shown. Figure 3 As shown, the method comprises the following steps:
[0100] Step S301, the build management platform obtains build information according to the build request initiated by the user; the build management platform creates a compile build task according to the build information, sets the state of the compile build task to the initial state, and stores it in the database.
[0101] The build management platform is visualized for users. Users can initiate build requests based on the build management platform. Based on the build request, the build information selected by the user for compiling the application can be obtained. The build information includes the source code, image, and build script required for the compiled application. The source code can be the path of the code repository, and the source code can be any branch. If the user does not specify a specific branch, the latest version of the source code in the code repository is obtained for compilation and construction. The image and build script can be specified by the user to use the existing image in the image repository and the existing build script in the code repository, or they can be created by the user.
[0102] The build management platform creates a compile build task for the application according to the build information and stores it in the database, wherein the state of the created compile build task is an initial state.
[0103] Step S302: The first process of the build scheduler periodically queries the database to obtain the compilation and build tasks in the database in the initial state.
[0104] The build management platform and the build scheduler have an asynchronous upstream and downstream relationship. After the build management platform creates a compilation and build task, the first process of the build scheduler can query the database regularly, such as querying the database every 5 seconds, to obtain the compilation and build tasks in the database with an initial status, so as to select a pod container for the compilation and build task for construction.
[0105] Furthermore, if there are multiple compilation and construction tasks in the database in the initial state when querying the database, the multiple compilation and construction tasks are stored in the construction task list, and each compilation and construction task is executed in sequence according to the construction task list, that is, for any compilation and construction task, the following steps are executed.
[0106] Step S303: query the database for a list of retained pods according to the application compiled by the compile and build task, and determine whether there is a pod container that meets the preset retention condition in the list of retained pods.
[0107] According to the application compiled in the compilation and build task, you can query the database's retained pod list to find and determine the pod containers that meet the preset retention conditions, so that you can reuse the retained pod containers to improve compilation efficiency.
[0108] The preset retention conditions include, for example, a retained pod container that is associated with the application, and the information of the retained pod container matches the build information, and it is determined that the retained pod container is in a live state. When using a pod container to build an application, the database will store the association between the pod container and the application. According to the application query data, the pod container associated with the application can be obtained. If the pod container is in the retained pod list, it is a retained pod container associated with the application. Considering the configuration of images, build scripts, etc. in the retained pod container, it is also necessary to determine that the information of the retained pod container matches the build information of the compilation and build task before the retained pod container can be directly reused. For example, the image used by the retained pod container needs to be the same as the image required by the application in the build information. Further, the build scheduler also needs to determine that the retained pod container is currently in a live state. If the live probe checks the retained pod container in the cluster and determines that it is in a live state, it is determined that there is a pod container that meets the preset retention conditions in the retained pod list, and step S304 is executed, otherwise step S306 is executed.
[0109] Step S304: Update the status of the pod container stored in the database, update the status to in use, and determine whether the update is successful.
[0110] When it is determined that there is a retained pod container that meets the preset retention condition, the state of the retained pod container in the database can be changed to occupy the retained pod container.
[0111] Before the change, the status of the retained pod container is unused. When the change is made, the status of the retained pod container is updated to in use. Furthermore, when there are two or more applications matching the same retained pod container, and the status of the retained pod container is updated, update snatch may occur. Therefore, after updating the retained pod container, the update result needs to be checked. If the update is successful, step S305 is executed. Otherwise, if the retained pod container cannot be used when the update fails, step S306 is executed. Alternatively, if the status of the retained pod container is already in use, it cannot be updated, and the update fails, and step S306 is executed.
[0112] Step S305: Determine whether there is a retained pod container matching the application, and the build scheduler records the association between the retained pod container and the compile and build task, and sets the status of the compile and build task to in progress.
[0113] After the status of the retained pod container is updated, it can be determined that there is a retained pod container that matches the application. The build scheduler records the association between the retained pod container and the compilation and build task, stores it in the database, sets the status of the compilation and build task to in progress, and compiles the application using the retained pod container.
[0114] During the specific construction, the main container of the pod container is the container for executing the construction. Its image can be specified by the user, or it can be automatically recommended according to the application. Different images can be used to build applications in different languages. Among them, one container in the pod container is responsible for obtaining the source code, construction scripts, and Dockerfile of the application from the code repository according to the path of the code repository; one container is responsible for packaging the built products and uploading them to the OSS platform for storage; one container is responsible for making the built products into images and uploading them to the image repository; one container is responsible for checking the execution status of other containers and controlling the execution order, and writing the execution results to the annotation of the pod container after the execution is completed. The annotation can record the various stages of the compilation and construction and the output of each stage. The various stages include different stages such as building and building completed.
[0115] Step S306: Build a new pod container according to the build information. The build scheduler uses the new pod container to execute the compile and build task, records the association between the new pod container and the compile and build task, and sets the status of the compile and build task to in progress.
[0116] If it is determined that there is no pod container that meets the preset retention conditions in the retained pod list, or if the status update of the retained pod container in the database fails, a new pod container can be built to compile the application. When creating a new pod container, the pod template can be filled according to the build information of the compile and build task, and a new pod container is created in the Kubernetes cluster. The build scheduler uses the new pod container to perform the compile and build task, and records the association between the new pod container and the compile and build task, that is, updates the compile and build task in the database, records the new pod container used for its construction, and sets the status of the compile and build task to in progress.
[0117] The subsequent execution process is the same as the process of compiling the application using the retained pod container, so it is not repeated here.
[0118] Step S307: After the compilation and construction task is completed, the construction scheduler sets the state of the compilation and construction task to be cleaned up.
[0119] For pod containers in the cluster, you can check whether the compilation and build task is completed or in progress by querying the annotation of the pod container. If the build is completed, the build scheduler can collect all logs of the pod container executing this compilation and build task and upload them to the OSS platform, such as Figure 2 shown.
[0120] After the compilation and construction task is completed, the build scheduler sets the status of the compilation and construction task in the database to pending cleanup.
[0121] Furthermore, the present embodiment adopts a backup mechanism. When a certain working instance stops working, another instance will continue to work. When multiple working instances are running at the same time, the second process of the build scheduler performs heartbeat updates, such as regularly querying the database, querying data every 5 seconds, and obtaining the latest working instance. Determine whether the working instance is the current working instance of the build scheduler. If so, only the heartbeat time of the working instance in the database is updated. If not, it means that the latest working instance is not the current working instance of the build scheduler. It can be determined whether the difference between the heartbeat time of the working instance and the current time exceeds the preset threshold; if it exceeds the preset threshold, it means that the working instance may have stopped working, and the working instance is updated to the working instance of the build scheduler; if it does not exceed the preset threshold, it means that the working instance may still be working, and no processing is required, waiting for the next heartbeat update of the second process.
[0122] The build management platform and build scheduler can set reasonable resource quotas for the builds of different applications, make the best use of the host machine's resources, and improve the concurrency of the build.
[0123] Step S308: The third process of the pod manager periodically queries the database to obtain the compilation and construction tasks whose status is to be cleaned up.
[0124] To make the pod container persistent, you do not need to obtain the application source code, build scripts, and other content from scratch every time you compile and build. You can set the pod container to be retained to improve the subsequent build efficiency. The retention or deletion of the pod container is completed by the pod manager. The retention of the pod container is determined by the pod manager based on the application compilation frequency, cluster resource usage information, etc. The pod manager sets a retention and deletion mechanism for the pod container, which also improves the actual resource utilization of the host machine.
[0125] The third process of the pod manager can query the database regularly, such as every 3 seconds. Figure 2 As shown, the compilation and construction tasks with the status of to be cleaned up are obtained from the database, that is, the compilation and construction tasks that have been built, and the corresponding pod containers have also been built.
[0126] Step S309: according to the pod container and / or the associated application associated with the compile and build task to be cleaned up, update the status of the compile and build task to be cleaned up and / or process the associated pod container.
[0127] For the compilation and build tasks to be cleaned up, it is necessary to further set the status of the compilation and build tasks, and further process the pod containers associated with the compilation and build tasks, that is, the pod containers used during execution.
[0128] Setting the status of the compilation and build task to be cleaned up to cleaned up indicates that the compilation and build task has been completed and the corresponding pod container is no longer in use. When setting the status of the compilation and build task to be cleaned up, the associated pod container needs to be processed at the same time.
[0129] Specifically, the following situations are handled:
[0130] Determine whether the pod container associated with the compilation and construction task to be cleaned up is alive, that is, whether the pod container is still alive in the cluster, which can be obtained through, for example, liveness probes. If not, that is, the pod container is no longer alive, then only set the status of the compilation and construction task to be cleaned up to cleaned up, and no further processing of the pod container is required. If the pod container is alive, continue to judge.
[0131] Determine whether the application associated with the compilation and build task to be cleaned up is in the preset application list. The applications in the preset application list are applications whose compilation frequency within the preset time exceeds the preset frequency threshold, that is, each application in the preset application list is an application that is frequently compiled and built. The preset frequency threshold is set according to the implementation situation and is not limited here. If not, it means that the frequency of application compilation and construction is low, and there is no need to persist the pod container used for compilation and construction. The status of the compilation and build task to be cleaned up can be set to cleaned up, and the pod container associated with the compilation and build task to be cleaned up can be deleted.
[0132] If the application is in the preset application list, it means that the application is frequently compiled and built. It is necessary to further determine whether the pod container associated with the compilation and construction task to be cleaned up is in the retained pod list, that is, whether the pod container used by this compilation and construction task is a retained pod container. If so, set the status of the compilation and construction task to be cleaned up to cleaned up, and set the associated pod container to unused, that is, set the retained pod container to unused, so that when the subsequent application is compiled and built again, it can find the unused retained pod container for reuse. If the pod container used in this compilation and build task is not in the retained pod list, that is, it is not a retained pod container, then query the retained pod list to see if there is a retained pod container associated with the application. If so, first set the status of the compilation and build task to be cleaned up to be cleaned up, and then determine the status of the retained pod container. If the retained pod container status is in use, delete the pod container associated with the compilation and build task to be cleaned up, that is, delete the new pod container used by this compilation and build task; if the retained pod container status is unused, considering that the build information of the application may be updated each time it is compiled and built, the new pod container better matches the image, build script, etc. required for the current compilation and build of the application, here, delete the retained pod container, and add the pod container associated with this compilation and build task, that is, the new pod container, to the retained pod list.
[0133] Step S310: The fourth process of the pod manager periodically queries the database to obtain a list of retained pods and / or obtains cluster resource usage information.
[0134] In addition to persisting pod containers, the pod manager also performs elimination management on the retained pod containers in the retained pod list. Elimination management eliminates pods based on the retained pod list, cluster resource usage information, etc. Elimination can be based on one or more of the above conditions.
[0135] Specifically, the fourth process of the pod manager queries the database periodically, such as every 3 seconds, to obtain the list of retained pods stored in the database, and can also obtain the current resource usage information of the Kubernetes cluster, such as CPU, memory and other resource usage information.
[0136] Step S311, sorting the retained pod containers in the retained pod list from small to large according to the usage frequency.
[0137] By querying the preset build time, such as the usage frequency of the retained pod container used by each application in the application list whose build frequency exceeds the threshold in the past week, or querying the usage frequency of the retained pod container associated with the compilation and build tasks in the past week, the usage frequency of each retained pod container in the retained pod list can be obtained.
[0138] The retained pod containers in the retained pod list are sorted in ascending order according to the usage frequency, so that the retained pod containers can be eliminated in ascending order according to the usage frequency.
[0139] Step S312: for each sorted retained pod container, the retained pod container in the retained pod list is processed according to the retained pod list and / or the cluster usage resource information.
[0140] For each sorted retained pod container, make judgments in turn so as to eliminate the retained pod container according to the frequency of use from small to large. Specifically, first query whether the retained pod container in the cluster is alive. If not, it means that the retained pod container no longer exists in the cluster, and then delete the retained pod container in the retained pod list in the database. If yes, it means that the retained pod container is still alive in the cluster, then determine whether the status of the retained pod container is in use. If yes, do not eliminate the retained pod container first; if not, further determine whether the unused time of the retained pod container exceeds the preset unused threshold. The preset unused threshold can be 2 days, which is set according to the implementation situation and is not limited here. If yes, that is, the retained pod container has not been used for 2 days. In order to save resources, the retained pod container can be deleted. If not, it means that the retained pod container may still be used, and it is necessary to continue to judge according to the next condition.
[0141] Furthermore, when the retained pod container is created, the cluster will pre-allocate the occupied resources to it, and combine the resource information occupied by the retained pod container with the cluster resource usage information to obtain the cluster resource utilization information, such as adding the CPU usage information occupied by the retained pod container to the CPU resource information already used by the cluster to obtain the cluster CPU resource usage information, and adding the memory usage information occupied by the retained pod container to the memory resource information already used by the cluster to obtain the cluster memory resource information. Determine whether the cluster resource utilization information exceeds the cluster utilization threshold. The cluster utilization threshold includes CPU utilization and memory utilization. If the obtained cluster resource utilization information has exceeded the cluster utilization threshold, such as exceeding the CPU utilization or memory utilization, it will affect the overall operation of the cluster, and then it is necessary to delete the retained pod container to release the resource information occupied by the retained pod container.
[0142] According to the compilation and construction method based on container reuse provided in this application, the pod manager persists the pod container, and uses the pod container used by the application with a higher compilation frequency as the retained pod container, so that the construction scheduler can determine the matching retained pod container according to the application of the compilation and construction task, and reuse the retained pod container to complete the compilation and construction task, thereby improving the compilation efficiency. And based on the compilation and construction of the pod container, different construction environments can be selected for different applications, a general application construction method can be implemented, the resources on the host machine can be fully utilized as much as possible, and the concurrency of the construction can be improved. Furthermore, the pod manager also sets a retention and elimination mechanism for the container to further improve the actual utilization rate of the host machine's resources.
[0143] Figure 4 The schematic diagram of the structure of the compiling and building device based on container reuse provided by an embodiment of the present application is shown. Figure 4 As shown, the device comprises:
[0144] The build management platform 410 is used to create a compile build task according to a build request initiated by a user, set the state of the compile build task to an initial state, and store it in a database;
[0145] The build scheduler 420 is used to query the database regularly to obtain the compilation and construction tasks in the initial state; according to the application compiled by the compilation and construction task, query whether there is a retained pod container matching the application in the database; if so, use the retained pod container to execute the compilation and construction task;
[0146] The pod manager 430 is used to determine the retained pod container according to the application compilation frequency and / or cluster resource usage information.
[0147] Optionally, the build management platform 410 is further used to: obtain build information based on a build request initiated by a user; the build information includes source code, image and build script required for the compiled application; create a compilation build task based on the build information, set the status of the compilation build task to the initial state, and store it in a database.
[0148] Optionally, the build scheduler 420 is further used to: the first process of the build scheduler periodically queries the database to obtain the compilation and construction tasks in the database in the initial state; if there are multiple compilation and construction tasks in the initial state, the multiple compilation and construction tasks are stored in the build task list to execute each compilation and construction task in sequence according to the build task list.
[0149] Optionally, the construction scheduler 420 is further configured to:
[0150] According to the application compiled by the compilation and build task, query the retained pod list in the database to determine whether there is a pod container in the retained pod list that meets the preset retention conditions; the preset retention conditions include: a retained pod container associated with the application, and the information of the retained pod container matches the build information, and the retained pod container is determined to be in a survival state;
[0151] If yes, update the status of the retained pod container in the database and set the status to in use;
[0152] If the update is successful, it is determined that there is a retained pod container that matches the application.
[0153] Optionally, the construction scheduler 420 is further configured to:
[0154] A new pod container is built according to the build information. The build scheduler uses the new pod container to execute the compilation and build task, records the association between the new pod container and the compilation and build task, and sets the status of the compilation and build task to in progress. After the compilation and build task is completed, the build scheduler sets the status of the compilation and build task to be cleaned up.
[0155] Optionally, the build scheduler 420 is further used for: the build scheduler records the association between the retained pod container and the compilation and construction task, and sets the status of the compilation and construction task to in progress; after the compilation and construction task is completed, the build scheduler sets the status of the compilation and construction task to be cleaned up.
[0156] Optionally, the construction scheduler 420 is further used to: construct a second process of the scheduler to query the database regularly to obtain the latest working instance;
[0157] Determine whether the work instance is a work instance for building a scheduler;
[0158] If so, update the heartbeat time of the working instance;
[0159] If not, determine whether the difference between the heartbeat time of the working instance and the current time exceeds a preset threshold;
[0160] If so, update the worker instance to the worker instance that builds the scheduler.
[0161] Optionally, the pod manager 430 is further configured to:
[0162] The third process of the pod manager periodically queries the database to obtain the compilation and construction tasks that are in the status of being cleaned up;
[0163] According to the pod container and / or the associated application associated with the compilation and construction task to be cleaned up, the status of the compilation and construction task to be cleaned up is updated and / or the associated pod container is processed.
[0164] Optionally, the pod manager 430 is further configured to:
[0165] Determine whether the pod container associated with the compilation and construction task to be cleaned up is alive;
[0166] If not, set the status of the compilation and build task to be cleaned up to cleaned up.
[0167] Optionally, the pod manager 430 is further configured to:
[0168] Determine whether the application associated with the compilation and construction task to be cleaned is in a preset application list; the application compilation frequency of the application in the preset application list within a preset time exceeds a preset frequency threshold;
[0169] If not, set the status of the compilation and construction task to be cleaned up to cleaned up, and delete the pod container associated with the compilation and construction task to be cleaned up;
[0170] If yes, determine whether the pod container associated with the compilation and build task to be cleaned up is in the retained pod list;
[0171] If yes, set the status of the compilation and build task to be cleaned up to cleaned up, and set the associated pod container to unused;
[0172] If not, query the retained pod list to see whether there is a retained pod container associated with the application;
[0173] If so, set the status of the compilation and build task to be cleaned up to cleaned up; if the status of the retained pod container is in use, delete the pod container associated with the compilation and build task to be cleaned up; if the status of the retained pod container is not in use, delete the retained pod container and add the associated pod container to the retained pod list.
[0174] Optionally, the pod manager 430 is further configured to:
[0175] The fourth process of the pod manager periodically queries the database to obtain a list of retained pods and / or obtains cluster resource usage information;
[0176] Sort the retained pod containers in the retained pod list in ascending order based on their usage frequency;
[0177] For each sorted retained pod container, the retained pod container in the retained pod list is processed according to the retained pod list and / or the cluster resource usage information.
[0178] Optionally, the pod manager 430 is further configured to:
[0179] For each sorted retained pod container, check whether the retained pod container in the cluster is alive.
[0180] If not, delete the remaining pod container;
[0181] If yes, determine whether the status of the retained pod container is in use;
[0182] If not, determine whether the unused time of the retained pod container exceeds the preset unused threshold;
[0183] If yes, delete the remaining pod container.
[0184] Optionally, the pod manager 430 is further configured to:
[0185] For each sorted retained pod container, the resource information occupied by the retained pod container is combined with the cluster resource usage information to obtain the cluster resource utilization information;
[0186] Determine whether the resource utilization information of the cluster exceeds the cluster utilization threshold;
[0187] If so, delete the retained pod container to release the resource information occupied by the retained pod container.
[0188] The description of each module above refers to the corresponding description in the method embodiment and will not be repeated here.
[0189] According to the compilation and construction device based on container reuse provided by the present application, after the construction management platform constructs the compilation and construction task, the construction scheduler queries the retained pod container that matches the application of the compilation and construction task, and reuses the retained pod container to complete the compilation and construction task. This eliminates the need to obtain the application code, construction scripts, and other content from scratch for each compilation and construction, thereby improving subsequent construction efficiency.
[0190] The present application also provides a non-volatile computer storage medium, which stores at least one executable instruction, and the executable instruction can execute operations corresponding to the compilation and construction method based on container reuse in any of the above method embodiments.
[0191] The present application also provides a computer program product, which includes at least one executable instruction or computer program, which can enable a processor to perform operations corresponding to the compilation and construction method based on container reuse in any of the above method embodiments.
[0192] Figure 5 A schematic diagram of the structure of a computing device according to an embodiment of the present application is shown. The specific embodiment of the present application does not limit the specific implementation of the computing device.
[0193] like Figure 5 As shown, the computing device may include: a processor (processor) 502 , a communications interface (Communications Interface) 504 , a memory (memory) 506 , and a communication bus 508 .
[0194] in:
[0195] The processor 502 , the communication interface 504 , and the memory 506 communicate with each other via a communication bus 508 .
[0196] The communication interface 504 is used to communicate with other devices such as clients or other servers.
[0197] The processor 502 is used to execute the program 510, and specifically can execute the relevant steps in the above-mentioned container reuse-based compilation and construction method embodiment.
[0198] Specifically, the program 510 may include program codes, which include computer operation instructions.
[0199] The processor 502 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the present application. The one or more processors included in the computing device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.
[0200] The memory 506 is used to store the program 510. The memory 506 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.
[0201] Program 510 can be specifically used to enable processor 502 to execute the compilation and construction method based on container reuse in any of the above-mentioned method embodiments. The specific implementation of each step in program 510 can refer to the corresponding descriptions in the corresponding steps and units in the above-mentioned compilation and construction embodiment based on container reuse, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the aforementioned method embodiment, which will not be repeated here.
[0202] The algorithm or display provided here is not inherently related to any specific computer, virtual system or other device. Various general systems can also be used together with the teaching based on this. According to the above description, it is obvious to construct the structure required for this type of system. In addition, the application is not directed to any specific programming language either. It should be understood that various programming languages can be utilized to realize the content of the application described here, and the above description of specific languages is to disclose the preferred embodiment of the application.
[0203] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.
[0204] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following intention: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the claims below, the inventive aspects are less than all the features of the single embodiment disclosed above. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.
[0205] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.
[0206] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.
[0207] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all functions of some or all components of the present application. The present application can also be implemented as a device or apparatus program (e.g., computer program and computer program product) for executing a part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.
[0208] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be constructed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The present application may be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be understood as limitations on the order of execution.
Claims
1. A compilation and construction method based on container reuse, comprising: The build management platform creates a compile build task according to the build request initiated by the user, sets the state of the compile build task to an initial state, and stores it in a database; The build scheduler periodically queries the database to obtain the compilation and construction tasks in the initial state; based on the application compiled by the compilation and construction task, queries the database to see whether there is a retained pod container matching the application; the retained pod container is determined by the pod manager based on the application compilation frequency and / or cluster resource usage information; If so, the build scheduler uses the retained pod container to execute the compilation and build task.
2. The method according to claim 1, wherein: The build management platform creates a compile build task according to a build request initiated by a user, sets the state of the compile build task to an initial state, and stores the state in a database, further comprising: The build management platform obtains build information according to the build request initiated by the user; the build information includes source code, image and build script required for the compiled application; The build management platform creates a compile build task according to the build information, sets the state of the compile build task to an initial state, and stores the state in a database.
3. The method according to claim 1, wherein: The build scheduler periodically queries the database to obtain the compilation and build tasks in the initial state, further comprising: The first process of the build scheduler periodically queries the database to obtain the compilation and build tasks in the initial state in the database; If there are multiple compilation and construction tasks in the initial state, the multiple compilation and construction tasks are stored in a construction task list, so that each compilation and construction task is executed in sequence according to the construction task list.
4. The method according to any one of claims 1 to 3, wherein: The querying of a database for whether there is a retained pod container matching the application compiled by the compile and build task further includes: According to the application compiled by the compilation and construction task, query the retained pod list of the database to determine whether there is a pod container that meets the preset retention condition in the retained pod list; the preset retention condition includes: a retained pod container that has an association relationship with the application, and the information of the retained pod container matches the construction information, and it is determined that the retained pod container is in a survival state; If yes, update the status of the retained pod container in the database to in use; If the update is successful, it is determined that there is a retained pod container matching the application.
5. The method according to claim 4, wherein: If the pod container that meets the preset retention condition does not exist in the determined retained pod list, or the status update of the retained pod container in the update database fails, the method further includes: A new pod container is built according to the build information, and the build scheduler uses the new pod container to execute the compile and build task, records the association between the new pod container and the compile and build task, and sets the status of the compile and build task to be in progress; After the compilation and construction task is executed and completed, the construction scheduler sets the state of the compilation and construction task to be cleaned up.
6. The method according to any one of claims 1 to 5, wherein: The build scheduler further includes: The build scheduler records the association between the retained pod container and the compile and build task, and sets the status of the compile and build task to in progress; After the compilation and construction task is executed and completed, the construction scheduler sets the state of the compilation and construction task to be cleaned up.
7. The method according to any one of claims 1 to 6, wherein: The method further comprises: The second process of the scheduler is constructed to query the database regularly to obtain the latest working instance; Determine whether the work instance is a work instance for building a scheduler; If so, update the heartbeat time of the working instance; If not, determine whether the difference between the heartbeat time of the working instance and the current time exceeds a preset threshold; If so, update the working instance to the working instance of the construction scheduler.
8. The method according to any one of claims 1 to 7, wherein: The method further comprises: The third process of the pod manager periodically queries the database to obtain the compilation and construction tasks that are in the status of being cleaned up; According to the pod container and / or the associated application associated with the compilation and construction task to be cleaned up, the status of the compilation and construction task to be cleaned up is updated and / or the associated pod container is processed.
9. The method according to claim 8, wherein: The updating of the status of the compilation and construction task to be cleaned up and / or processing of the associated pod container according to the pod container and / or the associated application associated with the compilation and construction task to be cleaned up further includes: Determine whether the pod container associated with the compilation and construction task to be cleaned up is alive; If not, the status of the compilation and construction task to be cleaned up is set to cleaned up.
10. The method according to claim 8, wherein: The updating of the status of the compilation and construction task to be cleaned up and / or processing of the associated pod container according to the pod container and / or the associated application associated with the compilation and construction task to be cleaned up further includes: Determine whether the application associated with the compilation and construction task to be cleared is in a preset application list; the application compilation frequency of the application in the preset application list within a preset time exceeds a preset frequency threshold; If not, set the status of the compilation and construction task to be cleaned up to cleaned up, and delete the pod container associated with the compilation and construction task to be cleaned up; If so, determine whether the pod container associated with the compilation and construction task to be cleaned up is in the retained pod list; If yes, set the status of the compilation and construction task to be cleaned up to cleaned up, and set the associated pod container to unused; If not, query the retained pod list to see whether there is a retained pod container associated with the application; If so, set the status of the compilation and construction task to be cleaned up to cleaned up; if the status of the retained pod container is in use, delete the pod container associated with the compilation and construction task to be cleaned up; if the status of the retained pod container is not in use, delete the retained pod container and add the associated pod container to the retained pod list.
11. The method according to any one of claims 1 to 10, wherein: The method further comprises: The fourth process of the pod manager periodically queries the database to obtain a list of retained pods and / or obtains cluster resource usage information; Sort the retained pod containers in the retained pod list according to the usage frequency from small to large; For each sorted retained pod container, the retained pod container in the retained pod list is processed in turn according to the retained pod list and / or the cluster usage resource information.
12. The method according to claim 11, wherein: The step of sequentially processing the sorted retained pod containers according to the retained pod list and / or the cluster resource usage information further includes: For each of the sorted retained pod containers, query in turn whether the retained pod container in the cluster is alive; If not, delete the retained pod container; If yes, determine whether the status of the retained pod container is in use; If not, determine whether the unused time of the retained pod container exceeds a preset unused threshold; If so, delete the remaining pod container.
13. The method according to claim 11, wherein: The step of sequentially processing the sorted retained pod containers according to the retained pod list and / or the cluster resource usage information further includes: For each of the sorted retained pod containers, the resource information occupied by the retained pod containers is combined with the resource usage information of the cluster to obtain resource utilization information of the cluster; Determining whether the resource utilization information of the cluster exceeds a cluster utilization threshold; If so, delete the retained pod container to release the resource information occupied by the retained pod container.
14. A compilation and construction device based on container reuse, comprising: The build management platform is used to create a compile build task according to the build request initiated by the user, set the state of the compile build task to the initial state, and store it in the database; A build scheduler is used to query the database regularly to obtain the compilation and construction tasks in the initial state; based on the application compiled by the compilation and construction task, query the database whether there is a retained pod container matching the application; if so, use the retained pod container to execute the compilation and construction task; The pod manager is used to determine the retained pod containers based on the application compilation frequency and / or cluster resource usage information.
15. A computing device comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the compilation and construction method based on container reuse as described in any one of claims 1-13.
16. A computer storage medium, wherein at least one executable instruction is stored in the storage medium, and the executable instruction enables a processor to execute operations corresponding to the compilation and construction method based on container reuse as described in any one of claims 1 to 13.
17. A computer program product, comprising at least one executable instruction, wherein the executable instruction enables a processor to execute operations corresponding to the compilation and construction method based on container reuse as described in any one of claims 1 to 13.