Container management method and device and cloud platform

Abstract

The invention discloses a container management method and device and a cloud platform, and relates to the field of distributed systems. The control node receives the request message and generates a container wake-up message in response to the request message. And the working node obtains the container wakeup message and wakes up the pod to be waken up in response to the container wakeup message. Therefore, when the to-be-awakened pod is awakened, compared with a mode of starting a container based on a virtual machine cache, due to the fact that the thread of the vCPU of the to-be-awakened pod in the hibernation state stops running and the pod service data of the to-be-awakened pod is stored in a memory except the memory of the virtual machine to which the to-be-awakened pod belongs, the to-be-awakened pod occupies less resources, and resource waste is avoided. Compared with a mode of starting the container based on the virtual machine cache, the virtual machine process of the to-be-awakened pod in the hibernation state is reserved, so that the thread of the virtual machine is directly used when the to-be-awakened pod is awakened, the starting time delay of the container is reduced, and the starting time delay requirement of the container can be met under the condition of reducing resource waste.

Description

Technical Field

[0001] This application relates to the field of distributed systems, and more particularly to a container management method, apparatus, and cloud platform. Background Technology

[0002] Serverless architecture is a technology that allows users to directly create container workloads in the cloud. Users no longer manage clusters and nodes; instead, they directly call the application programming interface (API) to create their own container workloads. For cloud platforms, when the number of cloud container instances created using serverless architecture is zero, the cloud platform launches a cloud container instance upon receiving an external request and then uses that instance to serve the request.

[0003] However, in current serverless architectures, if fast container startup is required, the cloud platform needs to reserve virtual machine (VM) cache, leading to significant resource waste. If the cloud platform uses a cold start method to start containers, although there is no resource waste due to reserved VM cache, the container startup time is too long, failing to meet container startup latency requirements. Therefore, how to meet container startup latency requirements while reducing resource waste has become an urgent technical problem to be solved. Summary of the Invention

[0004] This application provides a container management method, apparatus, and cloud platform that can meet container startup latency requirements and reduce resource waste.

[0005] Firstly, a container management method is provided, comprising: acquiring a container wake-up message, and responding to the container wake-up message to wake up a container unit (pod) to be woken up. The pod to be woken up is in a hibernation state. In the hibernation state, the threads of the virtual central processing unit (vCPU) of the pod to be woken up stop running, the pod business data of the pod to be woken up is stored in a storage device other than the memory of the virtual machine to which the pod to be woken up belongs, and the virtual machine process of the pod to be woken up is preserved.

[0006] Based on the container management method described above, when a pod is to be woken up, it is in a hibernation state. The threads of the vCPU of the pod stop running, and the business data of the pod is stored in storage outside the memory of the virtual machine to which the pod belongs. In this way, the storage resources and central processing unit (CPU) resources of the pod are released. Compared to starting containers based on virtual machine caching, the resources occupied by the pod in hibernation state are less than those occupied by the cached virtual machine, thus reducing resource waste. Furthermore, compared to starting containers based on cold start, since the virtual machine process of the pod in hibernation state is preserved, it is not necessary to restart the virtual machine threads when waking up the pod. The container startup latency is greatly shortened, thus meeting the container startup latency requirements while reducing resource waste.

[0007] As one possible implementation, the pod to be woken up is at least one hibernating pod within a worker node. The worker node is determined based on load. A hibernating pod refers to a pod in a hibernating state, meaning a new hibernation (podhibernated) state (podcondition) has been added to the pod state set. Since the worker node to which the pod to be woken up belongs is determined based on the load of each node, it can be the worker node with the lowest load, or at least one worker node with a relatively low load, thereby improving the resource utilization efficiency of the worker nodes and avoiding resource waste.

[0008] As one possible implementation, the initialization details of the virtual machine process's memory and network are preserved. Preserving these initialization details consumes very little memory, but re-initializing them takes a considerable amount of time. Therefore, preserving the initialization details of the virtual machine process's memory and network can reduce container startup latency while consuming fewer resources.

[0009] As one possible implementation, the container wake-up message could be the first modification event of the pod to be woken up. The first modification event is the deletion of the first pod comment, which indicates that the pod to be woken up is in a hibernation state. Thus, when the first pod comment indicating that the pod to be woken up is deleted is detected, it can be determined that the pod needs to be woken up, thereby enabling the container to be used according to its needs, avoiding situations where the container is idle, and improving container utilization efficiency.

[0010] As one possible implementation, the container wake-up message can be a wake-up command. Since the wake-up command is used to instruct the pod to be woken up, waking up the pod only when the wake-up command is received ensures that the container is used according to the required conditions, avoiding the situation where the container is idle and improving the utilization efficiency of the container.

[0011] As one possible implementation, the container wake-up message includes an identifier for the pod to be woken up. In response to the identifier of the pod to be woken up in the container wake-up message, the pod to be woken up is determined. Thus, the pod to be woken up is determined by the control node, which can select the pod to be woken up from hibernating pods among multiple worker nodes, avoiding selection from worker nodes with high load, thereby improving overall resource utilization efficiency. The pod is woken up by restarting the vCPU thread of the pod to be woken up and loading the pod's business data into the memory of the virtual machine to which the pod belongs. Since the wake-up process does not require initialization of the vCPU thread and the virtual machine to which the pod belongs, container startup latency is reduced.

[0012] One possible implementation involves responding to a container wake-up message and determining the pod to be woken up. In this way, the pod to be woken up is determined by the worker node itself. Since the worker node selects the pod to be woken up only from its own dormant pods, the time required to determine the pod to be woken up is reduced, further reducing container startup latency. The pod is then woken up by restarting the vCPU thread of the pod to be woken up and loading the pod's business data into the memory of the virtual machine to which the pod belongs. Because the wake-up process does not require initialization of the vCPU thread or the virtual machine to which the pod belongs, container startup latency is further reduced.

[0013] As one possible implementation, before obtaining the container wake-up message, obtain the container hibernation message, and in response to the container hibernation message, control the pod to be hibernated to enter the hibernation state.

[0014] Based on the above implementation method, the pods to be hibernated are controlled to enter the hibernation state. Since the storage and CPU resources of the pods to be hibernated are released, the waste of resources is reduced.

[0015] Optionally, a second modification event of the pod to be hibernated can be detected. The second modification event is the addition of a first pod annotation, which indicates that the pod to be hibernated is in a hibernation state. In this way, when the addition of the first pod annotation to indicate that the pod to be hibernated is detected, the pod to be hibernated can be identified, thereby controlling the pod to enter the hibernation state, avoiding the situation where the container is idle, and improving the utilization efficiency of the container.

[0016] Optionally, a hibernation command can be received. Since the hibernation command is used to instruct the pod to enter a hibernation state, controlling the pod to enter a hibernation state upon receiving the hibernation command improves container utilization efficiency.

[0017] Optionally, the pod to be hibernated is determined in response to the identifier of the pod in the container hibernation message. In this way, the pod to be hibernated is determined by the control node, which can identify all pods in the worker nodes, thus improving the efficiency of pod identification. By stopping the thread execution of the vCPU of the pod to be hibernated, the CPU resources of the pod are released, thereby reducing CPU resource waste. By transferring the pod's business data to storage, the memory occupied by the pod's business data in the virtual machine is released, thereby reducing memory resource waste.

[0018] Optionally, in response to a container hibernation message, a pod to be hibernated is identified. The pod to be hibernated is at least one idle pod within the worker node. Thus, the worker node determines the pod to be hibernated by identifying at least one idle pod within its own pods, thereby reducing resource waste on the worker node. By stopping the thread execution of the vCPU of the pod to be hibernated, the CPU resources of the pod are released, thus reducing CPU resource waste. Furthermore, by transferring the pod's business data to storage, the memory occupied by the pod's business data in the virtual machine is released, thereby reducing memory resource waste.

[0019] Optionally, in response to a container hibernation message, before a pod to be hibernated enters hibernation, if the resources consumed by a hibernating pod in at least one pod reach a preset proportion of the total resources, hibernation of the pod to be hibernated is stopped. This avoids excessive resource consumption by hibernating pods within worker nodes, thereby reducing resource waste.

[0020] One possible implementation involves receiving a container migration request. The request includes the identifier of the pod to be migrated and the identifier of the destination worker node. The pod to be migrated is a pod awaiting wake-up due to workload exceeding a preset threshold. Workload includes at least one of the following: load, pod wake-up frequency, and container concurrency. Thus, when the workload of a worker node exceeds the preset threshold, if the worker node still has pods awaiting wake-up, these pods are migrated as pods to be migrated, preventing further increases in resource consumption on the worker node. In response to the container migration request, the pod to be migrated is migrated to the destination worker node. By migrating the pod to be migrated to the destination worker node, further increases in resource consumption on the worker node are avoided. Furthermore, the pod to be migrated utilizes the resources of the destination worker node, preventing excessively high resource consumption on any single worker node and ensuring a more balanced resource consumption across multiple worker nodes, thereby improving overall resource utilization efficiency.

[0021] Secondly, a container management method is provided, comprising: receiving a request message and, in response to the request message, generating a container wake-up message. The container wake-up message is used to instruct the pod to be woken up. The pod to be woken up is in a hibernation state. In the hibernation state, the threads of the vCPU of the pod to be woken up stop running, the business data of the pod to be woken up is stored in storage outside the memory of the virtual machine to which the pod to be woken up belongs, and the virtual machine process of the pod to be woken up is preserved.

[0022] Based on the container management method described above, when a container wake-up message is generated to wake up the pod to be woken up, the pod is in a hibernation state. The threads of the pod's vCPUs stop running, and the pod's business data is stored in storage outside the memory of the virtual machine to which the pod belongs. In this way, the storage and CPU resources of the pod to be woken up are released. Compared to starting containers based on virtual machine caching, the resources occupied by the pod in hibernation state are less than those occupied by the cached virtual machine, thus reducing resource waste. Furthermore, compared to starting containers based on cold start, since the virtual machine process of the pod to be woken up is preserved, it is not necessary to restart the virtual machine's threads when waking up the pod. This significantly shortens the container startup latency, thus meeting the container startup latency requirements while reducing resource waste.

[0023] As one possible implementation, the initialization details of the virtual machine process's memory and network are preserved. Preserving these initialization details consumes very little memory, but re-initializing them takes a considerable amount of time. Therefore, preserving the initialization details of the virtual machine process's memory and network can reduce container startup latency while consuming fewer resources.

[0024] As one possible implementation, the container wake-up message includes the identifier of the pod to be woken up. In response to the request message, the identifier of the pod to be woken up is determined from at least one dormant pod on the lowest-loaded worker node. Thus, upon receiving a user's request message, the pod to be woken up is selected from the lowest-loaded worker node, thereby reducing resource waste on the lowest-loaded worker node and improving overall resource utilization efficiency. Based on the identifier of the pod to be woken up, a container wake-up message is generated, which wakes up the dormant pod on the lowest-loaded worker node, thereby enabling the container to be used according to the required specifications, putting the woken-up container into a running state, and improving container utilization efficiency.

[0025] As one possible implementation, the container wake-up message includes the identifier of the pod to be woken up. In response to the request message, a first preset number of worker nodes are determined based on load from low to high. Thus, when a user request message is received, the pod to be woken up is selected from the first preset number of worker nodes with lower load, thereby ensuring a more balanced distribution of resources across multiple worker nodes and improving overall resource utilization efficiency. The first preset number of worker nodes includes a first worker node. The identifier of the pod to be woken up in at least one dormant pod within the first worker node is determined. Since the number of pods to be woken up in each worker node of the first preset number of worker nodes does not exceed a second preset number, the product of the first preset number and the second preset number equals the number of pods the request message needs to wake up. This avoids excessive resource consumption by a single worker node while meeting user needs, reducing resource waste. A container wake-up message is generated based on the identifier of the pod to be woken up. The container wake-up message is used to wake up the pods to be woken up in the first worker node. The container wake-up message wakes up the dormant pods to be woken up in the first worker node, thereby enabling the container to be used as required, ensuring that all woken-up pods are in a running state, and improving container utilization efficiency.

[0026] Optionally, based on the identifier of the pod to be woken up, the first pod annotation of the pod to be woken up is deleted. The first pod annotation is used to indicate that the pod to be woken up is in a hibernation state. In this way, by deleting the first pod annotation of the pod to be woken up, the container wake-up message can be the first modification event of the deletion of the first pod annotation of the pod to be woken up, so that the worker node can determine to wake up the pod to be woken up, realize the use of the container according to the required conditions, avoid the situation where the pod is in an idle state, and reduce resource waste.

[0027] Optionally, a wake-up command is sent to the worker node where the pod to be woken up resides, based on the identifier of the pod. The container wake-up message can be a wake-up command. Since the wake-up command is used to instruct the pod to be woken up, sending the wake-up command to the worker node where the pod resides ensures that the worker node wakes up the pod upon receiving the wake-up command. This achieves the container's required usage, avoids pods being idle, and reduces resource waste.

[0028] One possible implementation is to receive a container hibernation request before receiving the request message. In response to the container hibernation request, a container hibernation message is generated. The container hibernation message is used to indicate that the pod to be hibernated is entering a hibernation state.

[0029] Based on the above implementation, a container hibernation message is generated according to the user's container hibernation request, causing the pod to enter hibernation state. Since the storage and CPU resources of the pod to be hibernated are released, resource waste is reduced.

[0030] One possible implementation involves identifying the pod to be hibernated in response to a container hibernation request. The pod to be hibernated is an idle pod among at least one pod on a worker node. In this way, the control node identifies all idle pods across all worker nodes as pods to be hibernated, thus improving the efficiency of identifying pods to be hibernated. Based on the identifier of the pod to be hibernated, a container hibernation message is generated. This message is used to put the pod into a hibernation state, releasing its storage and CPU resources, thereby reducing resource waste.

[0031] Optionally, based on the identifier of the pod to be hibernated, a first pod annotation is added to the pod to be hibernated. The first pod annotation is used to indicate that the pod to be hibernated is in a hibernation state. In this way, by adding the first pod annotation to indicate that the pod to be hibernated is in a hibernation state, the worker nodes can identify the pod to be hibernated, thereby controlling the pod to enter the hibernation state, avoiding the situation where the container is idle, and reducing resource waste.

[0032] Optionally, a hibernation command is sent to the worker nodes based on the identifier of the pod to be hibernated. In this way, the hibernation command instructs the pod to enter a hibernation state, allowing the worker nodes to control the pod to enter hibernation upon receiving the command, thereby reducing resource waste.

[0033] Optionally, in response to a container hibernation request, before generating a container hibernation message, if the resources consumed by the hibernating pod in at least one pod on the worker node reach a preset proportion of the total resources of the worker node, then the generation of container hibernation messages for the worker node is stopped. This avoids excessive resource consumption by hibernating pods within the worker node, thereby reducing resource waste.

[0034] One possible implementation involves identifying the pod to be migrated as a second worker node whose workload exceeds a preset threshold, based on the workload of the worker node hosting the pod to be woken up. Workload includes at least one of the following: load, pod wake-up frequency, and container concurrency. When the workload of the worker node hosting the pod exceeds the preset threshold, it is determined that the worker node has a hotspot issue, meaning its resource consumption is higher compared to other worker nodes. A container migration request is generated based on the identifier of the pod to be migrated. This request includes the identifier of the pod to be migrated and the identifier of the destination worker node. Thus, the pod to be woken up is migrated as the pod to be migrated, preventing further increases in the workload of the worker node. A container migration request is sent to the second worker node to migrate the pod to the destination worker node. By migrating the pod to the destination worker node, further increases in the workload of the worker node are avoided, and the pod uses the resources of the destination worker node, preventing excessive resource consumption by a single worker node and ensuring a more balanced resource consumption across multiple worker nodes, thereby improving overall resource utilization efficiency.

[0035] Thirdly, a container management apparatus is provided, the apparatus comprising modules for performing the container management method of the first aspect or any possible implementation thereof.

[0036] The container management device described in the third aspect can be a terminal device or a network device, or it can be a chip (system), network card or other component or assembly that can be set in a terminal device or a network device, or it can be a device that includes a terminal device or a network device. This application does not limit it in this regard.

[0037] Furthermore, the technical effects of the container management device described in the third aspect can be referred to the technical effects of the container management method described in the first aspect, and will not be repeated here.

[0038] Fourthly, a container management apparatus is provided, the apparatus comprising modules for performing the container management method of the second aspect or any possible implementation thereof.

[0039] The container management device described in the fourth aspect can be a terminal device or a network device, or it can be a chip (system), network card or other component or assembly that can be set in a terminal device or a network device, or it can be a device that includes a terminal device or a network device. This application does not limit it in this regard.

[0040] Furthermore, the technical effects of the container management device described in the fourth aspect can be referred to the technical effects of the container management method described in the second aspect, and will not be repeated here.

[0041] Fifthly, an electronic device is provided, comprising a processor and a memory; the processor is coupled to the memory; the memory is used to store computer instructions, which are loaded and executed by the processor to enable the computing device to implement the operation steps of the method described in any possible implementation of the first aspect above, or to implement the operation steps of the method described in any possible implementation of the second aspect above.

[0042] A sixth aspect provides a computer-readable storage medium comprising: computer software instructions; when the computer software instructions are executed in a computer, causing the computer to perform operational steps of the method as described in any possible implementation of the first aspect, or to perform operational steps of the method as described in any possible implementation of the second aspect.

[0043] In a seventh aspect, a chip system is provided. The chip system includes a memory and at least one processor. The memory stores a set of computer instructions, which, when executed by the processor, perform operation steps of the method described in any possible implementation of the first aspect, or perform operation steps of the method described in any possible implementation of the second aspect.

[0044] Eighthly, a computer program product is provided that, when the computer program product is run on a computer, causes the computer to perform the operation steps of the method as described in any possible implementation of the first aspect, or to perform the operation steps of the method as described in any possible implementation of the second aspect.

[0045] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0046] Figure 1 A schematic diagram of the architecture of a cloud platform provided for an embodiment of this application;

[0047] Figure 2 A schematic diagram of the layered structure of a cloud platform provided in an embodiment of this application;

[0048] Figure 3An architecture diagram of a computer system 300 provided in this application embodiment;

[0049] Figure 4 A flowchart illustrating a container management method provided in an embodiment of this application;

[0050] Figure 5 A schematic flowchart illustrating a container wake-up process provided in an embodiment of this application;

[0051] Figure 6 A schematic flowchart illustrating another container wake-up process provided in an embodiment of this application;

[0052] Figure 7 A schematic diagram illustrating a container wake-up process provided in an embodiment of this application;

[0053] Figure 8 A flowchart illustrating another container management method provided in an embodiment of this application;

[0054] Figure 9 A flowchart illustrating a service creation and container hibernation process provided in an embodiment of this application;

[0055] Figure 10 A flowchart illustrating a service creation process provided in an embodiment of this application;

[0056] Figure 11 A schematic flowchart illustrating a container hibernation process provided in this application embodiment;

[0057] Figure 12 A schematic diagram of a possible container management device 1200 provided for an embodiment of this application;

[0058] Figure 13 A schematic diagram of a possible container management device 1300 provided for an embodiment of this application;

[0059] Figure 14 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0060] This application can be applied not only to existing cloud platform technologies and virtual resource management scenarios, but also to future cloud platform technologies and virtual resource management scenarios. The terminology used in the implementation section of this application is only used to explain specific embodiments of this application and is not intended to limit this application. For ease of understanding, the relevant terminology involved in the embodiments of this application will be introduced below.

[0061] (1) Container

[0062] Containers are a lightweight and portable virtualization technology.

[0063] A container packages an application and its related dependencies into a single unit.

[0064] The container contains the necessary elements for the application to run smoothly, such as code, environment variables, processes, runtime environment, and software dependencies.

[0065] (2)pod

[0066] A pod is the smallest scheduling unit in a container cluster management system (Kubernetes, K8s). A pod contains at least one container, and the containers in a pod share the pod's network, storage, and other resources.

[0067] Because dozens, hundreds, or thousands of temporary containers can run on hardware devices, there is a need for container management. Kubernetes (K8s) is used to automatically manage and schedule applications defined by individual containers.

[0068] In Kubernetes, a node is a machine that hosts a pod. A node can be a physical machine or a virtual machine.

[0069] Kubernetes (K8s) is a distributed system whose core architecture includes master nodes and worker nodes (mode nodes). The master node is also called the control node. The control node and worker nodes work together to build and manage a cluster of containerized applications. The control node is used for control plane management, while the worker nodes are used to host containers and run applications.

[0070] (3) Virtual Machine

[0071] A virtual machine is a complete computer system simulated by software, possessing full hardware system functionality and running in a completely isolated environment.

[0072] The main difference between containers and virtual machines is their architecture. Virtual machines contain a complete instance of an operating system and run on virtual hardware. Containers, on the other hand, share the host operating system kernel and only contain applications and their dependencies.

[0073] (4) Cloud Container Instance (CCI)

[0074] Cloud container instances are a type of container service provided by cloud platforms that does not require the management of clusters and nodes. Users can directly call the Kubernetes interface to create pods.

[0075] (5) Cloud computing platform

[0076] A cloud platform, also known as a cloud computing platform, refers to a service that provides computing, networking, and storage capabilities based on hardware and software resources. In other words, the platform provider combines cloud (remote hardware resources) and computing (remote software resources) to form a platform that provides various services to users.

[0077] Cloud platforms offer services such as Software as a Service (SaaS), Platform as a Service (PaaS), and Infrastructure as a Service (IaaS). SaaS involves service providers deploying application software on cloud platform servers, with users subscribing to these services via the internet. PaaS provides a development environment as a service, offering users a development environment, server platform, and hardware resources. Users develop applications within this cloud platform environment and share them with other users via the cloud platform and the internet. IaaS provides users with a cloud infrastructure comprised of multiple servers, integrating memory, input / output devices, storage, and computing resources into a virtual resource pool, providing storage resources and virtualized servers.

[0078] To make the technical issues easier to understand, the existing container management methods are described below.

[0079] Because the deployment information of serverless containers (i.e., cloud containers) built using serverless containers is not visible to users, cloud containers are generally deployed using a secure container approach to ensure that containers from different tenants can run securely on the same host machine. This means that each Kubernetes pod is isolated by a virtual machine. Cloud containers typically use kernel-based virtual machine (KVM) isolation technology to isolate different pods and run them in different virtual machines.

[0080] KVM is a Linux kernel feature that enables Linux hosts to create virtual machines. The CPU and memory management of these virtual machines rely on the Linux kernel module. KVM uses either an enhanced page table (EPT) or a nested page table (NPT) provided by the host hardware to manage the virtual machine's memory, with each virtual machine having its own dedicated extended page table.

[0081] The hypervisor uses device control interface functions (ioctl) to set the mapping between the physical memory of the virtual machine and the virtual addresses of the virtual machine processes on the host. `ioctl` is used to set the virtual machine memory region (KVM_SET_MEMORY_REGION). The hypervisor can be open-source software such as emulators, virtual machine supervisors (QEMU), and cloud hypervisors. The hypervisor manages the physical memory of the virtual machine just like it manages its own virtual machine memory; for example, the hypervisor can dump the contents of all virtual machine physical memory.

[0082] Serverless containers have the ability to scale to zero, meaning that when there are no requests, the number of cloud container instances is scaled down to 0, i.e., there are no cloud container instances. When a request is received, Kubernetes will start a cloud container instance and then use that instance to serve the request. However, this method requires that the container startup latency be controlled within a set time limit.

[0083] When cloud containers employ a container isolation solution based on a Virtual Machine Manager (VMM), the VMM is software running on the node that manages the lifecycle of virtual machines. If Kubernetes (K8s) uses a cold start method to start cloud containers, it needs to first start the virtual machines and the guest kernel, resulting in a startup latency of hundreds of milliseconds. This causes the startup latency of the cloud containers to exceed the set time, failing to meet the startup latency requirements. If K8s uses a virtual machine caching method to start cloud containers, it needs to pre-cache a certain number of virtual machines. The cached virtual machines consume a lot of resources, and since they do not execute tasks before being started, this leads to a significant waste of resources.

[0084] To address the aforementioned issues, this application provides a container management method, particularly a method for hibernating a container and waking it up upon receiving a request. A control node receives a request message and generates a container wake-up message in response. A worker node obtains the container wake-up message generated by the control node and wakes up the pod to be woken up in response. Since the pod to be woken up is in a hibernation state, the vCPU threads of the pod stop running, and the pod's business data is stored in storage outside the memory of the virtual machine to which the pod belongs. Thus, the storage and CPU resources of the pod to be woken up are released. Compared to starting containers based on virtual machine caching, the pod to be woken up in a hibernation state occupies fewer resources, reducing resource waste. Furthermore, compared to starting containers based on cold start, since the virtual machine process of the pod to be woken up in a hibernation state is preserved, it is not necessary to restart the virtual machine's threads when waking up the pod, significantly shortening the container startup latency. This allows for meeting container startup latency requirements while reducing resource waste.

[0085] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0086] Figure 1 This is a schematic diagram of a cloud platform architecture provided as an embodiment of this application. Figure 1 As shown, the cloud platform 100 includes a computing server cluster 110, a storage server cluster 120, a management server cluster 130, a network device cluster 140, and a user terminal 150. The computing server cluster 110, storage server cluster 120, and management server cluster 130 communicate with the user terminal 150 through the network device cluster 140.

[0087] Computing server cluster 110 includes one or more computing servers ( Figure 1 The diagram shows two computing servers, namely computing server 111 and computing server 112, but is not limited to two computing servers.

[0088] Computing servers, such as servers and desktop computers, are computing resources within the cloud platform 100. They are used to generate and allocate computing resources based on user needs using virtualization technology. At the hardware level, computing servers are equipped with processors and memory (…). Figure 1 (Not shown in the diagram), the computing server's computing functions are implemented by the processor running programs in memory. The computing server can also read / write data on various storage servers in the storage server cluster 120 according to user needs.

[0089] Storage server cluster 120 includes one or more storage servers ( Figure 1The image shows two storage servers, namely storage server 121 and storage server 122, but is not limited to two storage servers.

[0090] Storage servers, serving as storage resources within the cloud platform 100, include components such as servers, desktop computers, storage array controllers, and hard disk enclosures. They provide logical disk storage, unstructured data storage, and integrated backup services for cloud virtual machines within the cloud platform 100. Hardware-wise, storage servers include network interface cards (NICs), processors, and memory. The processor in the storage server processes data from external sources. The NIC controls access to the memory, such as address signals, data signals, and various command signals, enabling the storage server to provide memory as a storage resource to users. Memory stores data and may include RAM and / or hard disks. RAM refers to internal storage that directly exchanges data with the processor; it can quickly read and write data at any time, serving as temporary data storage for the operating system or other running programs. Unlike RAM, hard disks are slower to read and write data and are typically used for persistent data storage.

[0091] Management server cluster 130 includes one or more management servers ( Figure 1 The diagram shows two management servers, namely management server 131 and management server 132, but is not limited to two management servers.

[0092] The management server is used to manage all computing services, shared storage, and network of the entire cloud platform 100, and also provides users or administrators with an API to manage the entire node.

[0093] Network device cluster 140 includes one or more switches and routers, such as Figure 1 As shown, in this embodiment, the network device cluster 140 includes a router 141, a switch 142, a switch 143, a switch 144, and a switch 145. The user terminal 150 is connected to the router 141 via the Internet. The router 141 is then connected to the switch 143 via the switch 142. The switch 143 is connected to each computing server in the computing server cluster 110. The switch 144 is connected to each computing server in the computing server cluster 110 and each storage server in the storage server cluster 120. The switch 145 is connected to each computing server in the computing server cluster 110, each storage server in the storage server cluster 120, and each management server in the management server cluster 130.

[0094] Optionally, the number and type of switches included in the network device cluster 140 can be adjusted according to the needs of the cloud platform 100. Switches 142, 143, 144, and 145 can be switches with different functions. For example, switch 142 is a core switch, while switches 143, 144, and 145 are used to manage specific network segments. For instance, switch 142 can be a core switch, switch 143 can be an internal / external switching network segment switch, switch 144 can be a storage network segment switch, and switch 145 can be a management network segment switch.

[0095] User terminal 150 includes one or more user terminals ( Figure 1 The diagram shows two user terminals, namely user terminal 151 and user terminal 152, but is not limited to two user terminals. The user terminals contain the interfaces and applications required for accessing the cloud platform 100.

[0096] It is worth noting that, Figure 1 This is merely an illustration and should not be construed as limiting the scope of this application. The cloud platform 100 may also include other devices. Figure 1 It is not shown in the middle.

[0097] In such Figure 1 Based on the equipment of the cloud platform 100 shown, the cloud platform 100 implements the functions of service nodes based on IaaS, PSSS, and SaaS, and provides services (e.g., computing services, storage services, and network services) to users through the service nodes. The service node can be a cloud node (e.g., control node, worker node, etc.) obtained by virtualizing the resources (e.g., computing resources and storage resources) of the cloud platform 100.

[0098] Next, combine Figure 2 The layered structure of the cloud platform 200 is explained.

[0099] Figure 2 This is a schematic diagram of a layered structure of a cloud platform provided in an embodiment of this application. Figure 2 As shown, the Infrastructure as a Service platform 210 is used to virtualize all infrastructure resources in the cloud platform 100, providing virtual resources (e.g., computing resources, network resources, and storage resources) to users in a software-defined manner. Infrastructure resources refer to the resources provided by the computing server cluster 110, storage server cluster 120, management server cluster 130, and / or network device cluster 140 in the cloud platform 100.

[0100] Platform as a Service (PaaS) platform 220 provides the runtime environment and application support functions for cloud platform 100, allowing users to request computing units within their quotas instead of virtual resources to run their services. Optionally, the computing units can be containers, and cloud platform 100 deploys and runs user code by scheduling containers. It should be noted that the number of containers in platform as a Service 220 can be one or more. Figure 2 The example uses only two containers.

[0101] As one possible implementation, cloud platform 100 can inject one or more components into a container to deploy and run code. Optionally, the resources (computing or storage resources) used by multiple components in the same container can belong to the same hardware device (e.g., a compute server, storage server, or management server) or different hardware devices within cloud platform 100.

[0102] The components may include database components, message queue components, orchestration components, logging components, virtualization components, business database components, configuration database components, and permission management components.

[0103] Software as a Service (SaaS) application 230 is used to provide services to users by composing user-deployed applications as API responses, based on Infrastructure as a Service (IaaS) platform 210 and Platform as a Service (PAS) platform 220. The application of SaaS application 230 and the container can communicate with each other through a web server.

[0104] It is worth noting that, Figure 2 This is merely an illustration and should not be construed as limiting the scope of this application. The layered structure of the cloud platform 100 may also include other modules. Figure 2 It is not shown in the middle.

[0105] The steps of the container management method provided in this application embodiment are executed by the service nodes (e.g., control nodes, worker nodes) provided by the cloud platform 100. The following steps are combined with... Figure 3 The container management method provided in the embodiments of this application will be described.

[0106] The container management method provided in this application can be applied to computer systems, such as Kubernetes cluster network systems, distributed cluster network systems, and other computer systems that utilize the Kubernetes system. These computer systems can consist of multiple physical nodes or virtual nodes, for example... Figure 3 As shown, Figure 3 This application provides an architectural diagram of a computer system 300. The computer system 300 includes a control node 310 and at least one worker node (…). Figure 3The diagram shows three computing nodes: worker node 320, worker node 330, and worker node 340 (but is not limited to three worker nodes). Control node 310, worker node 320, worker node 330, and worker node 340 can have management or computing functions. Figure 2 The service node shown. Control node 310 can be... Figure 1 The management server in the management server cluster 130 shown, worker nodes 320, 330, and 340 can be... Figure 1 The computing servers in the computing server cluster 110 shown.

[0107] Control node 310 is communicatively connected to worker nodes 320, 330, and 340, respectively. For example, control node 310 can communicate with worker nodes 320, 330, and 340 via a wireless network or a wired network. The wireless network can be a local area network, wide area network, metropolitan area network, Wi-Fi, Bluetooth, etc., and the wired network can be a network connected by optical fiber, cable, twisted pair, etc. The embodiments of this application do not limit this.

[0108] Control node 310 is used to receive request messages and, in response to the request messages, generate container wake-up messages.

[0109] The request message may include the number of pods that need to be woken up, and the container wake-up message may include the identifier of the pod to be woken up.

[0110] Worker nodes 320, 330, and 340 are used to obtain container wake-up messages and, in response, wake up the pods to be woken up.

[0111] In this process, the pod to be woken up is in a hibernation state. In the hibernation state, the threads of the vCPU of the pod to be woken up stop running, the business data of the pod to be woken up is stored in a storage device other than the memory of the virtual machine to which the pod to be woken up belongs, and the virtual machine process of the pod to be woken up is preserved.

[0112] The pod to be woken up may be located on any of the worker nodes 320, 330 and 340.

[0113] The following is based on Figure 3 The following explanation uses the execution of container management methods by control node 310, worker node 320, worker node 330, and worker node 340 as an example. Figure 4 This is a flowchart illustrating a container management method provided in an embodiment of this application. About Figure 4 The control node 310, worker node 320, worker node 330 and worker node 340 can be referred to the foregoing. Figures 1-3 The relevant descriptions will not be repeated here. For example... Figure 4 As shown, the container management method includes steps 401 to 404.

[0114] Step 401: Control node 310 receives the request message.

[0115] When a user needs the container to execute a user's request, a request message is sent from the user terminal 150 to the control node 310. Correspondingly, the control node 310 receives the request message.

[0116] As one possible implementation, the control node 310 receives a request message sent by the user terminal 150 through the K8s system, or the control node 310 receives a request message input by the user through the K8s system's knative extension platform, or the user inputs the request message to the knative-activator of the control node 310's K8s system's knative extension platform.

[0117] As one possible implementation, after receiving a user's request, if the number of pods required by the request exceeds the number of pods that can provide services, the control node 310 needs to wake up the hibernating pods. Here, pods that can provide services refer to pods that are not in a hibernating state.

[0118] For example, the knative-activator of control node 310 triggers the knative-autoscaler component of the Kubernetes system's scaling platform to scale up pods. The knative-autoscaler identifies dormant pods and initiates the wake-up process. The knative-autoscaler modifies the comment "io.serverless.k8s.hibernate.count" in the Kubernetes system's deployment controller. This comment represents the number of dormant serverless containers in Kubernetes. By modifying the count in this comment, the modified count is smaller than the original count, thereby reducing the number of dormant pods listed in the comment.

[0119] Step 402: Control node 310 responds to the request message and generates a container wake-up message.

[0120] As one possible implementation, control node 310 may respond to a request message, determine the pod to be woken up that can execute the request message, and generate a container wake-up message for the pod. The container wake-up message may include an identifier for the pod to be woken up, which refers to a pod in a worker node that is in a hibernation state and can execute the request message. The pod to be woken up can be located on any worker node.

[0121] As another possible implementation, the control node 310 can also directly generate a container wake-up message in response to the request message, and the worker node can determine the pod to be woken up.

[0122] Based on the two possible implementations mentioned above, optionally, the pod to be woken up is at least one hibernating pod within a worker node, and the worker node is determined based on the load, for example, the worker node is determined from worker node 320, worker node 330 and worker node 340 based on the load.

[0123] In a first alternative example, the control node 310 responds to the request message by determining the identifier of the pod to be woken up in at least one hibernating pod of the worker node with the lowest load.

[0124] The load can be at least one of the following: worker node memory utilization, CPU utilization, memory usage, etc., and this application embodiment does not limit this. The identifier of the pod to be woken up is used to uniquely indicate the pod to be woken up, and the identifier can be the number, name, or other information that uniquely indicates the pod to be woken up.

[0125] In a second alternative example, in response to the request message, control node 310 determines a first preset number of worker nodes based on the load from low to high, the first preset number of worker nodes including a first worker node. Control node 310 determines the identifier of at least one hibernating pod of the first worker node that needs to be woken up. Control node 310 generates a container wake-up message based on the identifier of the pod to be woken up.

[0126] In this context, the number of pods to be woken up in each of the first preset number of worker nodes does not exceed the second preset number, and the product of the first preset number and the second preset number is equal to the number of pods that the request message needs to wake up. Each pod to be woken up in the first preset number of worker nodes can correspond to the same deployment.

[0127] For example, the second preset number can be 1, meaning that only one hibernating pod is identified as the pod to be woken up in each worker node. In this case, the first preset number is equal to the number of pods that need to be woken up by the request message. When each pod to be woken up in each worker node corresponds to the same deployment, pods corresponding to the same deployment implement anti-affinity scheduling, thereby preventing a certain node from becoming a hotspot node.

[0128] The two optional examples above are merely optional methods provided by the embodiments of this application and should not be construed as limiting this application. For example, depending on the number of worker nodes, the method by which the control node 310 determines the pod to be woken up can change or be adjusted.

[0129] In this context, the pod to be woken up is in a hibernation state. In hibernation, the threads of the vCPU of the pod to be woken up stop running, the business data of the pod to be woken up is stored in storage outside the memory of the virtual machine to which the pod belongs, and the virtual machine process of the pod to be woken up is preserved. The memory and network initialization of the virtual machine process are preserved. A hibernating pod refers to a pod being in a hibernation state; that is, a new state is added to the pod's state set, and the type of the new state is pod hibernation.

[0130] As one possible implementation, control node 310 removes the first pod comment of the pod to be woken up based on the identifier of the pod to be woken up.

[0131] The `<first-pod>` annotation indicates that the pod to be woken up is in a hibernation state. For example, if the `<first-pod>` annotation contains `"io.serverless.k8s.hibernate":true`, it means that the serverless container in Kubernetes has entered a hibernation state. Removing the `<first-pod>` annotation indicates that the pod to be woken up needs to be woken up.

[0132] As another possible implementation, the control node 310 sends a wake-up command to the worker node where the pod to be woken up is located, based on the identifier of the pod to be woken up.

[0133] The wake-up command is used to instruct the user to wake up the pod to be woken up. The wake-up command can be a command entered by the user through the command-line tool (kubectl). For example, the wake-up command is kubectl wakeup pod-123-test, where pod-123-test is the identifier of the pod to be woken up.

[0134] Optionally, the control node 310 determines the pod to be woken up through its own hibernation controller and generates a container wake-up message for the pod.

[0135] For example, if the hibernation controller of control node 310 detects changes in the controller through the interface, it will determine the pod to be woken up and delete the first comment of the pod to be woken up.

[0136] The specific implementation process of the hibernation controller of control node 310 determining the pod to be woken up and generating the container wake-up message can be found in the following text. Figures 5-7 The relevant descriptions will not be repeated here.

[0137] Step 403: The worker node obtains the container wake-up message.

[0138] The pod to be woken up could be located on any of worker nodes 320, 330, and 340; therefore, worker nodes 320, 330, and 340 can all receive the container wake-up message. For example, Figure 4 Taking the example of worker node 320 obtaining the container wake-up message.

[0139] One possible implementation is that worker node 320 listens for the first modification event of the pod to be woken up. That is, if worker node 320 hears the first modification event of the pod to be woken up and determines that the pod is located on worker node 320, it means it has received the container wake-up message for the pod. If worker node 320 determines that the pod is not located on worker node 320, it means it has not received the container wake-up message.

[0140] Optionally, worker node 320 listens for the first modification event of the pod to be woken up through the node's agent (kubelet). The first modification event is the deletion of the comment on the first pod.

[0141] As another possible implementation, worker node 320 receives the wake-up command for the pod to be woken up. For example, worker node 320 receives `kubectl wakeup pod-123-test`. If worker node 320 determines that the pod to be woken up is located on worker node 320, it means it has obtained the container wake-up message for the pod. If worker node 320 determines that the pod to be woken up is not located on worker node 320, it means it has not obtained the container wake-up message.

[0142] Optionally, worker node 320 receives the wake-up command for the pod to be woken up via kubectl.

[0143] Step 404: The worker node responds to the container wake-up message and wakes up the pod waiting to be woken up.

[0144] As one possible implementation, worker nodes respond to container wake-up messages and identify the pod to be woken up. The worker node restarts the vCPU thread of the pod to be woken up and loads the pod's business data into the memory of the virtual machine to which the pod belongs. For example, Figure 4 Taking the response of worker node 320 to the container wake-up message as an example.

[0145] In the first alternative example, worker node 320 may determine the pod to be woken up in response to the identifier of the pod to be woken up in the container wake-up message. That is, worker node 320 determines the pod to be woken up based on the identifier of the pod to be woken up determined by control node 310.

[0146] In the second alternative example, worker node 320 may also determine the pod to be woken up in response to a container wake-up message. That is, worker node 320 determines the pod to be woken up itself. Specifically, worker node 320 identifies the pod among its at least one hibernating pods that can execute the user request in the container wake-up message as the pod to be woken up.

[0147] In the third alternative example, worker node 320 can wake up the pod to be woken up by calling the newly added first container runtime interface (CRIAPI) via kubelet.

[0148] The first CRI API interface is used to wake up pods located in the sandbox.

[0149] The definition of the first CRI API interface is as follows:

[0150] Wake up a pod located in the sandbox. Wake up a hibernated pod.Runtimes will recover.

[0151] Resources, such as the pod's CPU and memory.

[0152] Wake up a pod in the sandbox (Wake up pod in the sandbox request) returns (Wake up pod in the sandbox response) (rpc WakeupPodSandbox(WakeupPodSandboxRequest)returns(WakeupPodSandboxResponse)).

[0153] The requirements for waking up a pod located in a sandbox are as follows:

[0154] message WakeupPodSandboxRequest{

[0155] / / ID of the PodSandbox to wakeup

[0156] string pod_sandbox_id = 1;

[0157] }

[0158] The response for waking up a pod located in the sandbox is shown below:

[0159] message WakeupPodSandboxResponse{}.

[0160] In the fourth alternative example, the container runtime component of worker node 320 can restart the vCPU threads of the pod to be woken up and load the pod's business data into the memory of the virtual machine to which the pod belongs through the cloud-hypervisor and hostOS.

[0161] For example, the multi-sandbox container runtime component on worker node 320 restarts the vCPU threads of the pod to be woken up using the `cpusetcgroup thaw` command in the `cpuset` subsystem. The multi-sandbox container runtime component wakes up the virtual machine to which the pod belongs by calling the virtual machine wakeup interface of the hypervisor. It then loads the pod's business data into the memory of the virtual machine using mmap virtual memory with a file backend, and restores the virtual machine's memory through mechanisms such as page faults or preloading.

[0162] The specific implementation process of worker node 320 restarting the vCPU thread of the pod to be woken up, and loading the pod's business data into the memory of the virtual machine to which the pod belongs, can be found in the following text. Figures 5-7 The relevant descriptions will not be repeated here.

[0163] Based on the above Figure 4 As described in steps 401 to 404, since the pod to be woken up is in a hibernation state, the threads of the vCPU of the pod to be woken up stop running, and the business data of the pod to be woken up is stored in storage outside the memory of the virtual machine to which the pod belongs. In this way, the storage and CPU resources of the pod to be woken up are released. Compared with the method of starting the container based on the virtual machine cache, the pod to be woken up in the hibernation state occupies fewer resources, reducing resource waste. Furthermore, compared with the method of starting the container based on cold start, since the virtual machine process of the pod to be woken up in the hibernation state is preserved, it is not necessary to restart the virtual machine threads when waking up the pod, greatly shortening the container startup latency. Thus, the container startup latency requirements can be met while reducing resource waste.

[0164] The above describes the container wake-up process in a container management method. The following describes the container wake-up process through two specific examples, taking the control node 310 and the worker node 320 as examples.

[0165] Figure 5 This is a flowchart illustrating a container wake-up process provided in an embodiment of this application.

[0166] like Figure 5 As shown, control node 310 may include an activator for the K8s system's extension platform, an auto-scaling component for the K8s system's extension platform, an interface for the K8s system (kube-apiserver), and a hibernation controller. Worker node 320 may include an agent, a container runtime component (containerd), a multi-sandbox container runtime component (kuasar), a host system, and a cloud hypervisor.

[0167] Figure 5 The container wake-up process shown may include steps 501 to 518.

[0168] Step 501: The activator of control node 310 receives a user request sent by the user terminal.

[0169] Step 502: The activator of control node 310 sends an expansion command (scale) to the automatic scaling component.

[0170] Step 503: The auto-scaling component of control node 310 calls the interface to send a command to modify the controller (modifydeployment). The auto-scaling component of control node 310 modifies the comment "io.serverless.k8s.hibernate.count" in the deployment, reducing the number of dormant pods in the comment.

[0171] Step 504: The hibernation controller of control node 310 listens for the controller change event (getdeploy change message) through the interface.

[0172] Step 505: The hibernation controller of control node 310 selects the pod to be woken up from hibernation.

[0173] Step 506: The hibernation controller of control node 310 will evict low-priority pods if necessary.

[0174] Step 507: The hibernation controller of control node 310 sets the state of the pod to be woken up to running.

[0175] Step 508: The hibernation controller of control node 310 deletes the first comment of the pod to be woken up.

[0176] Step 509: The agent of worker node 320 listens for the modification event (pod changeevent) of the pod to be woken up through the interface.

[0177] Step 510: The agent of worker node 320 calls the interface of the container runtime component and the multi-sandbox container runtime component to wake up the pod located in the sandbox.

[0178] Step 511: The container runtime component and the multi-sandbox container runtime component of worker node 320 send a virtual machine wakeup command (vm-wakeup) to the cloud virtual machine hypervisor.

[0179] Step 512: The cloud virtual machine management program of worker node 320 responds to the virtual machine wake-up command and performs virtual machine memory mapping with file backend (mmap vm memory with file backend).

[0180] Step 513: The cloud virtual machine hypervisor of worker node 320 returns a response message to the container runtime component and the multi-sandbox container runtime component.

[0181] Step 514: The container runtime component and multi-sandbox container runtime component of worker node 320 send a thaw-cgroup command to the host system.

[0182] In step 515, after responding to the command to unfreeze the subsystem and restart the vCPU thread of the pod to be woken up, the host system of worker node 320 returns a response message to the container runtime component and the multi-sandbox container runtime component.

[0183] Step 516: The container runtime component and the multi-sandbox container runtime component of worker node 320 return response messages to the agent.

[0184] Step 517: The agent of worker node 320 returns the pod running information (setpod running) to the interface of control node 310.

[0185] Step 518: The cloud virtual machine management program of worker node 320 receives the send request input by the activator to the application running in the virtual machine.

[0186] The above Figure 5 In the illustrated embodiment, the hibernation controller of control node 310 generates a container wake-up message in response to the received request message. In response to the container wake-up message, the container runtime component of worker node 320 wakes up the hibernating pod through the control host system and the cloud virtual machine hypervisor. Since it is not necessary to restart the virtual machine's threads, the container startup latency is greatly reduced, thus meeting the container startup latency requirements.

[0187] The container management method provided in this application does not limit the devices where the control node 310 and the worker node 320 are located. For example, the control node 310 and the worker node 320 can be located on the same electronic device, or they can be located on different electronic devices. Figure 6 As shown, the control node 310 and the worker node 320 are service nodes virtualized by the cloud platform based on the hardware resources of different devices, and the container in the worker node 320 is a serverless container.

[0188] Figure 6 This is a flowchart illustrating a container wake-up process provided in an embodiment of this application.

[0189] like Figure 6 As shown, control node 310 includes an activator for the Kubernetes (K8s) system's extension platform, an auto-scaling component for the K8s system's extension platform, and an interface for the K8s system. Worker node 320 includes an agent, a container runtime component, a multi-sandbox container runtime component, and a virtual machine. The virtual machine includes a task component and at least one container. Figure 6 The image shows two containers, but is not limited to two containers.

[0190] Figure 6 The container wake-up process shown includes steps 601 to 605.

[0191] Step 601: The activator of control node 310 receives the user request.

[0192] Step 602: Control node 310 sends a wake-up command (kubectlwakeup pod xxx) to the agent of worker node 320 through the interface.

[0193] Step 603: The agent of worker node 320 calls the interface of the container runtime component to wake up the pod located in the sandbox and wakes up the pod to be woken up.

[0194] Step 604: The container runtime component of worker node 320 wakes up the virtual machine's task component.

[0195] Step 605: The multi-sandbox container runtime component of worker node 320 sends a virtual machine wake-up command to the virtual machine.

[0196] exist Figure 6 In the illustrated embodiment, control node 310 and worker node 320 employ the container management method of this application to wake up pods in a dormant state, thereby resolving the contradiction between the rapid startup requirements and resource efficiency of serverless containers. That is, it reduces the time required to start the container while minimizing resource consumption.

[0197] The above text used two specific examples to illustrate the flowchart of the container wake-up process. The following text describes the scenarios of the container wake-up process.

[0198] Figure 7 This is a schematic diagram of a container wake-up process provided in an embodiment of this application.

[0199] like Figure 7 As shown, Figure 7 The larger rectangle shown in the upper center indicates a virtual machine that is running. Figure 7 The smaller rectangles shown in the lower center indicate virtual machines in hibernation mode, with only one hibernating pod in each hibernating virtual machine.

[0200] Figure 7 The left side of the text indicates waking up the virtual machine that is in hibernation. Figure 7 The right side of the diagram indicates that a virtual machine in a hibernation state is awakened, thus increasing the number of running virtual machines by one and decreasing the number of hibernation virtual machines by one.

[0201] The above text Figures 3-7 The content shown describes the container wake-up process in the container management method. Next, we will give an overall description of the container management method based on the above, and focus on describing the container hibernation process.

[0202] Figure 8 This is a flowchart illustrating another container management method provided in an embodiment of this application.

[0203] like Figure 8 As shown, the container management method includes steps 801 to 813.

[0204] Step 801: Control node 310 receives the container hibernation request.

[0205] When the number of requests that users need to execute in containers decreases, i.e. when there are pods in an idle state, the control node 310 receives a container hibernation request input by the user.

[0206] As one possible implementation, the control node 310 receives the container hibernation request from the user through the K8s system, or the control node 310 receives the container hibernation request from the user through the K8s system's knative extension platform.

[0207] For example, control node 310 receives annotations input when creating a knative service. For instance, the annotation might include a maximum of 18 pods to hibernate (autoscaling.knative.dev / max-hibernete: "18") and a minimum of 3 pods to hibernate (autoscaling.knative.dev / min-hibernete: "3"). Control node 310's knative call interface creates a deployment, using the annotations as part of the deployment's annotations.

[0208] As one possible implementation, before receiving a container hibernation request from the user, the control node 310 creates the number of pods defined in the comments of the Knative service. The method by which the control node 310 creates the pods is prior art and will not be elaborated upon in this embodiment; however, please refer to the following text for details. Figure 8 and Figure 9 Related descriptions.

[0209] Step 802: Control node 310 responds to the container hibernation request and generates a container hibernation message.

[0210] As one possible implementation, control node 310 may respond to a container hibernation request, determine the identifier of the pod to be hibernated, and generate a container hibernation message based on the identifier. The container hibernation message indicates that the pod to be hibernated is entering a hibernation state, and includes the identifier of the pod to be hibernated, which uniquely identifies the pod. For example, the identifier of the pod to be hibernated could be its number, name, etc. Alternatively, control node 310 may directly generate the container hibernation message in response to a container hibernation request.

[0211] Optionally, control node 310 adds a first pod annotation to the pod to be hibernated based on the identifier of the pod to be hibernated.

[0212] The first pod annotation indicates that the pod to be hibernated is in a hibernation state.

[0213] For example, the hibernation controller of control node 310 determines the required number of pods to be hibernated based on the annotations of the deployent. Control node 310 calls the modified interface (pod update API) of the pods to be hibernated and adds the first pod annotation to the pods to be hibernated.

[0214] Among them, control node 310 can add the first pod annotation of the pod to be hibernated through the hibernation controller. The specific implementation process is described in the following text. Figure 9 and Figure 11 The relevant descriptions are not repeated here in the embodiments of this application.

[0215] Optionally, the control node 310 sends a hibernation command to the worker nodes based on the identifier of the pod to be hibernated.

[0216] The hibernation command is used to instruct a pod to enter a hibernation state. For example, the hibernation command is `kubectl hibernate pod-123-test`, where `pod-123-test` is the identifier of the pod to be hibernated.

[0217] The control node 310 determines the working node where the pod to be hibernated is located based on the identifier of the pod to be hibernated, and sends a hibernation command to the working node where the pod to be hibernated is located.

[0218] Optionally, after identifying a pod to be hibernated, control node 310 can also stop performing health checks on the pod. Health checks are a Kubernetes (K8s) mechanism used to automatically restart containers and remove the application from the service when application anomalies are detected, ensuring high availability.

[0219] As another possible implementation, the control node 310 can also stop generating container hibernation messages for a worker node when the resources occupied by a hibernating pod in at least one pod of the worker node reach a preset proportion of the total resources of the worker node.

[0220] To prevent a worker node from becoming a hotspot, the control node 310 can set a resource water level for the worker node, which is a preset percentage of the total resources of the worker node. This preset percentage can be pre-set, for example, 32%, 40%, 54%, 60%, 71%, etc.

[0221] For example, the control node 310 can set the memory resource waterline to 50% of the total memory resources. When the memory occupied by the hibernating pod in at least one pod of the worker node reaches 50% of the total memory of the worker node, the control node 310 stops generating container hibernation messages for the worker node.

[0222] Step 803: The worker node obtains the container hibernation message.

[0223] A pod awaiting hibernation may reside on any of the worker nodes 320, 330, and 340. Therefore, worker nodes 320, 330, and 340 can all receive the container hibernation message. For example, Figure 8 Taking the example of worker node 320 obtaining container hibernation messages.

[0224] One possible implementation is that worker node 320 listens for the second modification event of the pod to be hibernated. Specifically, if worker node 320 listens for the second modification event of the pod to be woken up and determines that the pod is located on worker node 320, it means it has received the container hibernation message. If worker node 320 listens for the second modification event of the pod to be woken up and determines that the pod is not located on worker node 320, it means it has not received the container hibernation message.

[0225] The second modification event is the addition of a first pod annotation, which indicates that the pod to be hibernated is in a hibernation state.

[0226] Optionally, worker node 320 listens for the second modification event of the pod to be woken up via kubelet.

[0227] As one possible implementation, worker node 320 receives the hibernation command for the pod to be hibernated. For example, worker node 320 receives `kubectl hibernate pod-123-test`. If worker node 320 determines that the pod to be hibernated is located on worker node 320, it means it has obtained the container hibernation message for the pod. If worker node 320 determines that the pod to be hibernated is not located on worker node 320, it means it has not obtained the container hibernation message for the pod.

[0228] The hibernation command is used to instruct a pod to enter hibernation mode.

[0229] As one possible implementation, worker node 320 stops hibernating pods when the resources occupied by hibernating pods in at least one pod reach a preset proportion of the total resources.

[0230] To avoid becoming a hotspot node, worker node 320 can set its own resource water level. The resource water level of each worker node among worker nodes 320, 330 and 340 can be the same or different.

[0231] For example, worker node 320 sets its own memory resource threshold to 60% of the total memory resources. When worker node 320 determines that at least one of its pods has a hibernating pod occupying 60% of the total memory, it stops hibernating the pod to be hibernated. Worker node 330 sets its own memory resource threshold to 55% of the total memory resources. When worker node 330 determines that at least one of its pods has a hibernating pod occupying 55% of the total memory, it stops hibernating the pod to be hibernated.

[0232] Step 804: The worker node responds to the container hibernation message and controls the pod to enter hibernation state.

[0233] As one possible implementation, worker nodes respond to container hibernation messages and identify pods to be hibernated. The worker nodes stop the thread execution of the vCPUs of the pods to be hibernated, transfer the pod's business data to storage, and release the memory occupied by the pod's business data in the virtual machine. For example, Figure 8 Taking the response of worker node 320 to the container wake-up message as an example.

[0234] As one possible implementation, worker node 320 responds to the container hibernation message and identifies the pod to be hibernated.

[0235] Optionally, worker node 320 responds to the identifier of the pod to be hibernated in the container hibernation message to determine the pod to be hibernated.

[0236] Optionally, worker node 320 responds to the container hibernation message and identifies the pod to be hibernated.

[0237] Optionally, worker node 320 can control the pod to enter hibernation state by calling the newly added second CRI API interface via kubelet.

[0238] The second CRI API interface is used to hibernate pods located in the sandbox (hibernatepodsandbox).

[0239] The definition of the second CRI API interface is as follows:

[0240] Hibernate a pod located in a sandbox. Hibernate a running sandbox. Runtimes will try its best to hibernate.

[0241] Release resources, such as CPU and memory.

[0242] This makes the resource available for running other tasks.

[0243] The request to hibernate a pod in a sandbox returns (the response of the hibernate pod in a sandbox) {}(rpc HibernatePodSandbox(HibernatePodSandboxRequest)returns(HibernatePodSandboxResponse) {}).

[0244] The requirements for a pod hibernating in a sandbox are as follows:

[0245] message HibernatePodSandboxRequest{

[0246] / / ID of the PodSandbox to hibernate

[0247] string pod_sandbox_id = 1;

[0248] }

[0249] The response of a pod hibernating in a sandbox is shown below:

[0250] message HibernatePodSandboxResponse{}.

[0251] Optionally, the container runtime component of worker node 320 can freeze the threads of the vCPU of the pod to be hibernated through the cloud hypervisor and the host system, and transfer the pod business data in the memory of the virtual machine to which the pod belongs to be hibernated to the storage, thereby releasing the memory occupied by the pod business data in the virtual machine. The specific implementation process can be found in the following text. Figure 9 and Figure 11The relevant descriptions are not repeated here in the embodiments of this application.

[0252] Step 805: Control node 310 receives the request message.

[0253] Step 806: Control node 310 responds to the request message and generates a container wake-up message.

[0254] Step 807: The worker node obtains the container wake-up message.

[0255] Step 808: The worker node responds to the container wake-up message and wakes up the pod to be woken up.

[0256] Among them, steps 805 to 808 and Figure 4 Steps 402 to 404 are similar and will not be described in detail here.

[0257] Step 809: Control node 310 determines the pod to be migrated as the pod to be woken up in the second working node whose business volume exceeds the preset threshold, based on the business volume of the working node where the pod to be woken up is located.

[0258] If a worker node experiences high concurrency in waking up containers, high frequency of waking up multiple containers within the node, or a load exceeding a preset threshold, its resource consumption will be higher than that of other worker nodes. This imbalance in resource consumption will cause the worker node to become a hotspot node in the cluster.

[0259] To avoid hotspot nodes, control node 310 can monitor the workload of the worker node where the pod to be woken up resides. The workload can include at least one of the node's load, the frequency of pod wake-ups, and container concurrency. Control node 310 determines a second worker node whose workload exceeds a preset threshold. This second worker node may be any one of worker nodes 320, 330, or 340. The following description will use worker node 330 as an example of the second worker node.

[0260] Step 810: Control node 310 generates a container migration request based on the identifier of the pod to be migrated.

[0261] After determining that worker node 330 is a hotspot node, control node 310 identifies the pods to be woken up in worker node 330 as pods to be migrated. Control node 310 generates a container migration request based on the identifier of the pod to be migrated.

[0262] The container migration request includes the identifier of the pod to be migrated and the identifier of the destination worker node. The destination worker node refers to the worker node to which the pod will be migrated. The destination worker node can be pre-selected by the user or determined based on the load of each worker node. For example, the destination worker node can be the worker node with the lowest load.

[0263] Step 811: Control node 310 sends a container migration request to the second worker node.

[0264] Step 812: The second worker node receives the container migration request.

[0265] Step 813: The second worker node responds to the container migration request and migrates the pod to be migrated to the destination worker node.

[0266] Worker node 330 responds to the identifier of the pod to be migrated and the identifier of the destination worker node in the container migration request, and migrates the pod to be migrated from itself to the destination worker node. For example, worker node 330 can be based on the hot migration capability of secure containers, and migrate the pod to be migrated from itself to the destination worker node by rescheduling the pod to be migrated.

[0267] Based on the above Figure 8 As described in steps 801 to 813, since the control node 310 controls the idle pods in the worker nodes to enter a hibernation state, the vCPU threads of the hibernation pods stop running, and the pod business data of the hibernation pods are stored in storage outside the memory of the virtual machine to which the hibernation pods belong. Thus, the storage and CPU resources of the hibernation pods are released. Compared with the method of starting containers using virtual machine caching, the method of this embodiment consumes fewer resources in the hibernation state, reducing resource waste. Furthermore, compared with the method of starting containers using cold start, since the virtual machine processes of the hibernation pods are preserved, it is not necessary to restart the virtual machine threads when waking up the pods. This greatly shortens the container startup latency, thereby meeting the container startup latency requirements. Moreover, preserving the virtual machine threads does not consume too many resources, thus avoiding increased resource waste.

[0268] Furthermore, since the resource consumption of worker nodes is considered during the hibernation and revival of pods, pods awaiting hibernation can be left alone or migrated, thus preventing worker nodes from becoming hotspot nodes.

[0269] The above describes the overall process of container management. Next, we will describe the service creation and container hibernation process in the container management method through specific examples, taking control node 310 and worker node 320 as examples to introduce the service creation and container hibernation process.

[0270] Figure 9 This is a flowchart illustrating a service creation and container hibernation process provided in an embodiment of this application.

[0271] like Figure 9 As shown, control node 310 includes an extension platform, interfaces, and hibernation controller for the Kubernetes system. Worker node 320 includes an agent, container runtime components, multi-sandbox container runtime components, a host system, and a cloud hypervisor.

[0272] Figure 9 The service creation and container hibernation process shown may include steps 901 to 922.

[0273] Step 901: The extended platform of control node 310 receives a request to add a service (createservice) sent by the user terminal.

[0274] Step 902: The extended platform of control node 310 calls the interface to create a controller (create deployment).

[0275] Step 903: The controller of control node 310 creates pods for the number of instances required by the service request.

[0276] Step 904: The controller of control node 310 sends a command to the agent of worker node 320 to create a new pod (newpod).

[0277] Step 905: In response to the add pod command, the agent of worker node 320 calls the interface of the container runtime component and the pod run (runpodsandbo) of the multi-sandbox container runtime component located in the sandbox to add a pod in worker node 320.

[0278] Step 906: The container runtime component and the multi-sandbox container runtime component of worker node 320 send the virtual machine start command (start vm) to the cloud virtual machine hypervisor.

[0279] Step 907: After the virtual machine is started, the cloud virtual machine management program of worker node 320 returns a response message to the container runtime component and the multi-sandbox container runtime component.

[0280] Step 908: The container runtime component and multi-sandbox container runtime component of worker node 320 send a subsystem command to the host system to set the virtual CPU cgroup.

[0281] Step 909: After responding to the subsystem command to set the virtual central processing unit and completing the setup, the host system of worker node 320 returns a response message to the container runtime component and the multi-sandbox container runtime component.

[0282] Step 910: The agent of worker node 320 sends the startcontainer command to the container runtime component and the multi-sandbox container runtime component.

[0283] Step 911: After the container runtime component and multi-sandbox container runtime component of worker node 320 start the container, they return a response message to the agent.

[0284] Step 912: The agent of worker node 320 calls the interface of control node 310 and returns a response message.

[0285] Step 913: After the hibernation controller of control node 310 determines the pod to be hibernated, it calls the interface to send the pod to be hibernated (hibernate pod).

[0286] Step 914: The controller of control node 310 modifies the interface of the pod to be hibernating.

[0287] Step 915: The hibernation controller of control node 310 calls the modified interface (pod update api) and adds a comment for the first pod.

[0288] Step 916: The agent of worker node 320 listens for the second modification event of the pod to be hibernated.

[0289] Step 917: After the agent of worker node 320 determines that the pod to be hibernated is in the running state, it calls the interface of the container runtime component and the multi-sandbox container runtime component for hibernating pods located in the sandbox (hibernatepodsndbox) to control the pod to be hibernated to enter the hibernation state.

[0290] Step 918: The container runtime component and multi-sandbox container runtime component of worker node 320 freeze the vCPU subsystem of the pod to be hibernated (freeze vCPU cgroup).

[0291] Step 919: The container runtime component and the multi-sandbox container runtime component of worker node 320 send a virtual machine hibernation command (vm-hibernate) to the cloud virtual machine hypervisor.

[0292] Step 920: The cloud virtual machine management program of worker node 320 dumps the virtual machine memory to the solid-state drive (dump vmmemory to ssd).

[0293] Step 921: The cloud virtual machine hypervisor of worker node 320 releases the virtual machine's memory (munmap vmmemory).

[0294] Step 922: The cloud virtual machine hypervisor of worker node 320 sends response messages to the container runtime component and the multi-sandbox container runtime component.

[0295] The above Figure 9 In the illustrated embodiment, the hibernation controller of control node 310 generates a container hibernation message in response to a received container hibernation request. Worker node 320, in response to the container hibernation message, uses its container runtime component to hibernate the pod through the control host system and the cloud virtual machine hypervisor. The vCPU threads of the pod to be hibernated stop running, and the pod's business data is stored in storage outside the memory of the virtual machine to which the pod belongs. Thus, the storage and CPU resources of the pod to be hibernated are released. Compared to the method of starting containers using virtual machine caching, the method of this embodiment consumes fewer resources in the hibernation state, reducing resource waste.

[0296] The container management method provided in this application does not limit the devices where the control node 310 and the worker node 320 are located. For example, the control node 310 and the worker node 320 can be located on the same electronic device, or they can be located on different electronic devices. Figure 10 and Figure 11 As shown. In Figure 10 and Figure 11 In the process, the control node 310 and the worker node 320 are service nodes virtualized by the cloud platform based on the hardware resources of different devices, and the container in the worker node 320 is a serverless container.

[0297] Figure 10 This is a flowchart illustrating a service creation process provided in an embodiment of this application.

[0298] like Figure 10 As shown, control node 310 includes an interface to the Kubernetes system. Worker node 320 includes an agent, a container runtime component, a multi-sandbox container runtime component, and a virtual machine, which includes a task component and at least one container. Figure 10 The image shows two containers, but is not limited to two containers.

[0299] Figure 10The service creation process shown may include steps 1001 to 1004.

[0300] Step 1001: Control node 310 calls the interface to send a pod creation command (kubectl create-fpod.yaml) to the agent of worker node 320.

[0301] Step 1002: The agent of worker node 320 sends the pod run command (runpodsandbox), the container create command (createcontainer), and the container start command (startcontainer) located in the sandbox to the container runtime component.

[0302] Step 1003: The container runtime component of worker node 320 wakes up the virtual machine's task component.

[0303] Step 1004: The multi-sandbox container runtime component of worker node 320 starts the virtual machine through the cloud virtual machine management program.

[0304] exist Figure 10 In the illustrated embodiment, the control node 310 sends a pod creation command to the worker node 320, which creates a pod on the worker node 320 to execute the application, thereby enabling the execution of user requests through the pod.

[0305] The container management method provided in this application does not limit the state of the pod, except as described above. Figure 10 In addition to the regular states of newly created pods, other states such as sleep, hibernation, hibernation, wake-up, and pre-start can also be added. For example... Figure 11 As shown, the pods in this container management method can also include a hibernation state.

[0306] Figure 11 This is a schematic flowchart illustrating a container hibernation process provided in an embodiment of this application.

[0307] like Figure 11 As shown, control node 310 includes an activator for the K8s system's extension platform, an auto-scaling component for the K8s system's extension platform, and an interface for the K8s system. Worker node 320 includes an agent, a container runtime component, a multi-sandbox container runtime component, and a virtual machine.

[0308] Figure 11 The container hibernation process shown may include steps 1101 to 1104.

[0309] Step 1101: The activator of control node 310 receives a hibernation request from the container.

[0310] Step 1102: Control node 310 sends a hibernation command (kubectl hibernate pod xxx) to the agent of worker node 320 through the interface.

[0311] Step 1103: The agent of worker node 320 calls the interface of the hibernation pod in the sandbox of the container runtime component to control the pod to enter the hibernation state.

[0312] Step 1104: The multi-sandbox container runtime component of worker node 320 sends a virtual machine hibernation command to the virtual machine.

[0313] exist Figure 11 In the illustrated embodiment, the control node 310 and the worker node 320 adopt the container management method of this application embodiment to control the pod to hibernate to enter the hibernation state, realize on-demand charging for cloud platform resources, and release cloud platform resources when they are not needed, thereby reducing costs and improving the resource utilization efficiency of the cloud platform.

[0314] The container management method provided in this embodiment has been described in detail above. The following will combine... Figure 12 This describes the container management device provided in this embodiment.

[0315] Figure 12 This is a schematic diagram of a possible container management device 1200 provided in an embodiment of this application. The container management device 1200 can be used to implement the container management function of the working node in the above method embodiments, and therefore can also achieve the beneficial effects of the above method embodiments. In this embodiment, the container management device 1200 can be... Figure 3 Any worker node used to implement container management functions can also be a module (such as a chip) applied to a worker node.

[0316] The container management device 1200 includes an acquisition module 1201 and a wake-up module 1202.

[0317] The acquisition module 1201 is used to acquire the container wake-up message. For example, the acquisition module 1201 is used to execute... Figure 4 Step 403, or Figure 8 Step 807 in the process.

[0318] The wake-up module 1202 is used to wake up the container unit pod to be woken up in response to the container wake-up message. The pod to be woken up is in a hibernation state. In the hibernation state, the threads of the virtual CPU (vCPU) of the pod to be woken up stop running, the pod business data of the pod to be woken up is stored in a memory other than the memory of the virtual machine to which the pod to be woken up belongs, and the virtual machine process of the pod to be woken up is preserved. For example, the acquisition module 1201 is used to execute... Figure 4Step 404, or Figure 8 Step 808 in the process.

[0319] As one possible implementation, the pod to be woken up is at least one hibernating pod within a worker node, which is determined based on the load.

[0320] As one possible implementation, the initialization content of the virtual machine process's memory and network is preserved.

[0321] As one possible implementation, the acquisition module 1201 is specifically used to listen for the first modification event of the pod to be woken up. The first modification event is the deletion of the first pod comment, which is used to indicate that the pod to be woken up is in the hibernation state.

[0322] As one possible implementation, the acquisition module 1201 is specifically used to receive a wake-up command; the wake-up command is used to instruct the pod to be woken up to be woken up.

[0323] As one possible implementation, the container wake-up message includes the identifier of the pod to be woken up. The wake-up module 1202 is specifically used to respond to the identifier of the pod to be woken up in the container wake-up message, determine the pod to be woken up; restart the vCPU thread of the pod to be woken up; and load the pod's business data into the memory of the virtual machine to which the pod to be woken up belongs.

[0324] As one possible implementation, the wake-up module 1202 is specifically used to respond to the container wake-up message, determine the pod to be woken up; restart the vCPU thread of the pod to be woken up; and load the pod's business data into the memory of the virtual machine to which the pod to be woken up belongs.

[0325] As one possible implementation, the acquisition module 1201 is also used to acquire a container hibernation message before acquiring the container wake-up message; and in response to the container hibernation message, control the pod to be hibernated to enter the hibernation state.

[0326] As one possible implementation, the acquisition module 1201 is specifically used to listen for a second modification event of the pod to be hibernated, the second modification event being the addition of a first pod annotation, the first pod annotation being used to indicate that the pod to be hibernated is in the hibernation state.

[0327] As one possible implementation, the acquisition module 1201 is specifically used to receive a hibernation command, which is used to instruct the pod to hibernate to enter the hibernation state.

[0328] As one possible implementation, the apparatus further includes: a hibernation module, configured to, in response to the identifier of the pod to be hibernated in the container hibernation message, identify the pod to be hibernated; stop the thread execution of the vCPU of the pod to be hibernated; transfer the pod's business data to the memory; and release the memory occupied by the pod's business data in the virtual machine.

[0329] As one possible implementation, the hibernation module is specifically used to respond to the container hibernation message, determine the pod to be hibernated; the pod to be hibernated is at least one pod that is in an idle state; stop the thread execution of the vCPU of the pod to be hibernated; transfer the pod business data of the pod to be hibernated to the memory; and release the memory occupied by the pod business data in the virtual machine.

[0330] As one possible implementation, the hibernation module is further configured to stop hibernating the pod to be hibernated before controlling the pod to enter the hibernation state in response to the container hibernation message, provided that the resources occupied by the hibernating pod in at least one pod reach a preset proportion of the total resources.

[0331] As one possible implementation, the acquisition module 1201 is further configured to receive a container migration request, the container migration request including an identifier of the pod to be migrated and an identifier of the destination worker node, wherein the pod to be migrated is a pod to be woken up whose workload exceeds a preset threshold, and the workload includes at least one of load, pod wake-up frequency, and container concurrency. The apparatus further includes a migration module, configured to migrate the pod to be migrated to the destination worker node in response to the container migration request.

[0332] It should be understood that the container management device 1200 of this application embodiment can be implemented in hardware. For example, the hardware can refer to a processor or controller. For example, the processor or controller may be implemented using a GPU, NPU, application-specific integrated circuit (ASIC), or programmable logic device (PLD). The PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof. Software implementation is also possible. Figure 4 and Figure 8In the method shown, the container management device 1200 and its various modules can also be software modules.

[0333] Figure 13 This is a schematic diagram of a possible container management device 1300 provided in an embodiment of this application. The container management device 1300 can be used to implement the container management function of the control node 310 in the above method embodiments, and therefore can also achieve the beneficial effects of the above method embodiments. In this embodiment, the container management device 1300 can be... Figure 3 The control node 310 used to implement container management functions can also be a module (such as a chip) applied to the control node 310.

[0334] The container management device 1300 includes a receiving module 1301 and a generating module 1302.

[0335] The receiving module 1301 is used to receive request messages. For example, the obtaining module 1201 is used to execute... Figure 4 Step 401, or Figure 8 Step 805 in the process.

[0336] The generation module 1302 is used to generate a container wake-up message in response to the request message; the container wake-up message is used to instruct the pod to be woken up to be woken up; the pod to be woken up is in a hibernation state, in which the threads of the vCPU of the pod to be woken up stop running, the pod business data of the pod to be woken up is stored in a memory other than the memory of the virtual machine to which the pod to be woken up belongs, and the virtual machine process of the pod to be woken up is preserved. For example, the acquisition module 1201 is used to execute Figure 4 Step 402, or Figure 8 Step 806 in the process.

[0337] As one possible implementation, the initialization content of the virtual machine process's memory and network is preserved.

[0338] As one possible implementation, the container wake-up message includes an identifier of the pod to be woken up, and the pod to be woken up can execute the request corresponding to the request message. The generation module 1302 is specifically configured to, in response to the request message, determine the identifier of the pod to be woken up in at least one hibernating pod of the worker node with the lowest load; and generate a container wake-up message based on the identifier of the pod to be woken up; the container wake-up message includes the identifier of the pod to be woken up.

[0339] As one possible implementation, the container wake-up message includes an identifier of the pod to be woken up, and the pod to be woken up can execute the request corresponding to the request message. The generation module 1302 is specifically configured to, in response to the request message, determine a first preset number of worker nodes according to the load order from low to high; the first preset number of worker nodes includes a first worker node; determine the identifier of the pod to be woken up in at least one hibernating pod of the first worker node; the number of pods to be woken up in each of the first preset number of worker nodes does not exceed a second preset number, and the product of the first preset number and the second preset number is equal to the number of pods to be woken up in the request message; generate a container wake-up message based on the identifier of the pod to be woken up; the container wake-up message is used to wake up the pod to be woken up in the first worker node.

[0340] As one possible implementation, the generation module 1302 is specifically used to delete the first pod annotation of the pod to be woken up based on the identifier of the pod to be woken up. The first pod annotation is used to indicate that the pod to be woken up is in the hibernation state.

[0341] As one possible implementation, the generation module 1302 is specifically used to send a wake-up command to the worker node where the pod to be woken up is located based on the identifier of the pod to be woken up; the wake-up command is used to instruct the pod to be woken up to be woken up.

[0342] As one possible implementation, the receiving module 1301 is further configured to receive a container hibernation request before receiving the receiving request message. The generating module 1302 is further configured to generate a container hibernation message in response to the container hibernation request; the container hibernation message is used to indicate that the pod to be hibernated enters a hibernation state.

[0343] As one possible implementation, the generation module 1302 is specifically used to respond to the container hibernation request, determine the identifier of the pod to be hibernated; the pod to be hibernated is at least one pod in the worker node that is in an idle state; and generate a container hibernation message based on the identifier of the pod to be hibernated.

[0344] As one possible implementation, the generation module 1302 is specifically used to add a first pod annotation to the pod to be hibernated based on the identifier of the pod to be hibernated. The first pod annotation is used to indicate that the pod to be hibernated is in the hibernation state.

[0345] As one possible implementation, the generation module 1302 is specifically used to send a hibernation command to the worker node according to the identifier of the pod to be hibernated, the hibernation command being used to instruct the pod to be hibernated to enter the hibernation state.

[0346] As one possible implementation, the generation module 1302 is further configured to stop generating the container hibernation message for the worker node before generating the container hibernation message in response to the container hibernation request, provided that the resources occupied by the hibernating pod in at least one pod of the worker node reach a preset proportion of the total resources of the worker node.

[0347] As one possible implementation, the generation module 1302 is further configured to determine, based on the workload of the worker node where the pod to be woken up is located, that the pod to be migrated is a pod to be woken up in a second worker node whose workload exceeds a preset threshold; the workload includes at least one of load, pod wake-up frequency and container concurrency; and generate a container migration request based on the identifier of the pod to be migrated; the container migration request includes the identifier of the pod to be migrated and the identifier of the destination worker node.

[0348] The device further includes a sending module, configured to send a container migration request to the second worker node, the container migration request being used to migrate the pod to be migrated to the destination worker node.

[0349] It should be understood that the container management device 1300 of this application embodiment can be implemented in hardware. For example, the hardware can refer to a processor or controller. For example, the processor or controller may be implemented using a GPU, NPU, application-specific integrated circuit (ASIC), or programmable logic device (PLD). The PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof. Software implementation is also possible. Figure 4 and Figure 8 In the method shown, the container management device 1300 and its various modules can also be software modules.

[0350] This application also provides an electronic device, please refer to... Figure 14 , Figure 14This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 1400 includes a memory 1401, a processor 1402, a communication interface 1403, and a bus 1404. The memory 1401, the processor 1402, and the communication interface 1403 are interconnected via the bus 1404.

[0351] Memory 1401 may be a read-only memory, a static storage device, a dynamic storage device, or a random access memory. Memory 1401 may store computer instructions, and when the computer instructions stored in memory 1401 are executed by processor 1402, processor 1402 and communication interface 1403 are used to execute the steps in the container management method. For example, processor 1402 is used to execute the above-described... Figure 4 and Figure 8 The steps in the container management method shown above, and the above Figure 12 The container management device 1200 or Figure 13 The functions of the container management device 1300.

[0352] Processor 1402 may be a general-purpose CPU, an application-specific integrated circuit (ASIC), a GPU, or any combination thereof. Processor 1402 may include one or more chips.

[0353] The communication interface 1403 uses a transceiver module, such as, but not limited to, a transceiver, to enable communication between the electronic device 1400 and other devices or communication networks.

[0354] Bus 1404 may include a pathway for transmitting information between various components of electronic device 1400 (e.g., memory 1401, processor 1402, communication interface 1403).

[0355] Electronic device 1400 can be a computer (e.g., a server) in a cloud data center, or a computer in an edge data center, or a terminal.

[0356] The method steps in this embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Alternatively, the ASIC can reside in a terminal device. Of course, the processor and storage medium can also exist as discrete components in a network device or terminal device.

[0357] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of this application are performed entirely or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, or other programmable device. The computer program or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; it can also be an optical medium, such as a digital video disc (DVD); or it can be a semiconductor medium, such as a solid-state drive (SSD). The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A container management method characterized by, The method comprises: acquiring a container wake-up message; in response to the container wake-up message, waking up a to-be-woken-up container unit pod; the to-be-woken-up pod is in a hibernation state, in which a thread of a virtual central processing unit (vCPU) of the to-be-woken-up pod stops running, pod service data of the to-be-woken-up pod is stored in a storage device other than a memory of a virtual machine to which the to-be-woken-up pod belongs, and a virtual machine process of the to-be-woken-up pod is reserved.

2. The method of claim 1, wherein, The to-be-woken-up pod is at least one hibernation pod in a worker node, and the worker node is determined according to a load.

3. The method of claim 1, wherein, Initialization contents of the virtual machine process and a network are reserved.

4. The method according to any one of claims 1 to 3, characterized in that, The acquiring of the container wake-up message comprises: listening to a first modification event of the to-be-woken-up pod, the first modification event being that a first pod annotation is deleted, and the first pod annotation being used to indicate that the to-be-woken-up pod is in the hibernation state.

5. The method according to any one of claims 1 to 3, characterized in that, The acquiring of the container wake-up message comprises: receiving a wake-up command, the wake-up command being used to indicate to wake up the to-be-woken-up pod.

6. The method according to any one of claims 1 to 5, characterized in that, The container wake-up message comprises an identifier of the to-be-woken-up pod, and the waking up of the to-be-woken-up pod in response to the container wake-up message comprises: determining the to-be-woken-up pod in response to the identifier of the to-be-woken-up pod in the container wake-up message; restarting a thread of a vCPU of the to-be-woken-up pod; loading the pod service data in a memory of a virtual machine to which the to-be-woken-up pod belongs.

7. The method according to any one of claims 1 to 5, characterized in that, The waking up of the to-be-woken-up pod in response to the container wake-up message comprises: determining the to-be-woken-up pod in response to the container wake-up message; restarting a thread of a vCPU of the to-be-woken-up pod; loading the pod service data in a memory of a virtual machine to which the to-be-woken-up pod belongs.

8. The method according to any one of claims 1 to 7, characterized in that, Before the acquiring of the container wake-up message, the method further comprises: acquiring a container hibernation message; in response to the container hibernation message, controlling a to-be-hibernation pod to enter the hibernation state.

9. The method of claim 8, wherein, The acquiring of the container hibernation message comprises: listening to a second modification event of the to-be-hibernation pod, the second modification event being that a first pod annotation is added, and the first pod annotation being used to indicate that the to-be-hibernation pod is in the hibernation state.

10. The method of claim 8, wherein, The acquiring of the container hibernation message comprises: receiving a hibernation command, the hibernation command being used to indicate that the to-be-hibernation pod enters the hibernation state.

11. The method according to any one of claims 8-10, characterized in that, The controlling of the to-be-hibernation pod to enter the hibernation state in response to the container hibernation message comprises: determining the to-be-hibernation pod in response to an identifier of the to-be-hibernation pod in the container hibernation message; stopping a thread of a vCPU of the to-be-hibernation pod from running; storing the pod service data in the storage device; and releasing a memory occupied by the pod service data in the virtual machine.

12. The method according to any one of claims 8-10, characterized in that, The controlling of the to-be-hibernation pod to enter the hibernation state in response to the container hibernation message comprises: determining the to-be-hibernation pod in response to the container hibernation message, the to-be-hibernation pod being a pod in an idle state among at least one pod; stopping a thread of a vCPU of the to-be-hibernation pod from running; copying the pod service data of the pod to be hibernated into the memory; releasing the memory occupied by the pod service data in the virtual machine.

13. The method according to any one of claims 8-12, characterized in that, Before the method controls the pod to be hibernated to enter the hibernation state in response to the container hibernation message, the method further comprises: stopping hibernation of the pod to be hibernated when the resources occupied by the hibernated pods in at least one pod reach a preset proportion of total resources.

14. The method according to any one of claims 1 to 13, characterized in that, The method further comprises: receiving a container migration request, the container migration request comprising an identifier of a pod to be migrated and an identifier of a destination worker node, the pod to be migrated being a pod to be woken up whose service volume exceeds a preset threshold, the service volume comprising at least one of load, pod wake-up frequency and container concurrency; migrating the pod to be migrated to the destination worker node in response to the container migration request.

15. A container management method characterized by, The method comprises: receiving a request message; generating a container wake-up message in response to the request message, the container wake-up message being used to instruct a pod to be woken up to wake up, the pod to be woken up being in a hibernation state, in the hibernation state, threads of vCPUs of the pod to be woken up stop running, pod service data of the pod to be woken up is stored in a memory other than a memory of a virtual machine to which the pod to be woken up belongs, and a virtual machine process of the pod to be woken up is reserved.

16. The method of claim 15, wherein, Initialization contents of memory and network of the virtual machine process are reserved.

17. The method according to claim 15 or 16, characterized in that, The container wake-up message comprises an identifier of the pod to be woken up, the pod to be woken up being capable of executing a request corresponding to the request message, and the generating of the container wake-up message in response to the request message comprises: determining, in response to the request message, an identifier of a pod to be woken up in at least one hibernated pod of a worker node with the lowest load; generating a container wake-up message according to the identifier of the pod to be woken up.

18. The method of claim 15 or 16, wherein, The container wake-up message comprises an identifier of the pod to be woken up, the pod to be woken up being capable of executing a request corresponding to the request message, and the generating of the container wake-up message in response to the request message comprises: determining, in response to the request message, a first preset number of worker nodes in a sequence from low to high load; the first preset number of worker nodes comprises a first worker node; determining an identifier of a pod to be woken up in at least one hibernated pod of the first worker node; a number of pods to be woken up in each worker node of the first preset number of worker nodes does not exceed a second preset number, and a product of the first preset number and the second preset number is equal to a number of pods to be woken up required by the request message; generating a container wake-up message according to the identifier of the pod to be woken up; the container wake-up message is used to wake up the pod to be woken up in the first worker node.

19. The method of claim 17 or 18, wherein, The generating of the container wake-up message according to the identifier of the pod to be woken up comprises: deleting, according to the identifier of the pod to be woken up, a first pod annotation of the pod to be woken up, the first pod annotation being used to indicate that the pod to be woken up is in the hibernation state.

20. The method of claim 17 or 18, wherein, The container wake-up message is generated according to the identifier of the pod to be woken up, and the container wake-up message comprises: According to the identifier of the pod to be woken up, a wake-up command is sent to the worker node where the pod to be woken up is located; the wake-up command is used to instruct to wake up the pod to be woken up.

21. The method of any one of claims 15-20, wherein, Before the receiving request message, the method further comprises: Receiving a container hibernation request; In response to the container hibernation request, a container hibernation message is generated; the container hibernation message is used to instruct the pod to be hibernated to enter a hibernation state.

22. The method of claim 21, wherein, The container hibernation message is generated in response to the container hibernation request, and the container hibernation message comprises: In response to the container hibernation request, the identifier of the pod to be hibernated is determined; the pod to be hibernated is a pod in an idle state in at least one pod of a worker node; According to the identifier of the pod to be hibernated, a container hibernation message is generated.

23. The method of claim 22, wherein, The container hibernation message is generated according to the identifier of the pod to be hibernated, and the container hibernation message comprises: According to the identifier of the pod to be hibernated, a first pod annotation of the pod to be hibernated is added, and the first pod annotation is used to indicate that the pod to be hibernated is in the hibernation state.

24. The method of claim 22, wherein, The container hibernation message is generated according to the identifier of the pod to be hibernated, and the container hibernation message comprises: According to the identifier of the pod to be hibernated, a hibernation command is sent to the worker node, and the hibernation command is used to instruct the pod to be hibernated to enter the hibernation state.

25. The method of any one of claims 21-24, wherein, Before the container hibernation message is generated in response to the container hibernation request, the method further comprises: In a case where the resources occupied by the hibernation pod in at least one pod of a worker node reach a preset proportion of total resources of the worker node, the generation of the container hibernation message of the worker node is stopped.

26. The method of any one of claims 15-25, wherein, The method further comprises: According to the amount of business of the worker node where the pod to be woken up is located, the pod to be migrated is determined to be the pod to be woken up in a second worker node whose amount of business exceeds a preset threshold; the amount of business comprises at least one of load, pod wake-up frequency and container concurrency; According to the identifier of the pod to be migrated, a container migration request is generated; the container migration request comprises the identifier of the pod to be migrated and the identifier of the destination worker node; The container migration request is sent to the second worker node, and the container migration request is used for the pod to be migrated to migrate to the destination worker node.

27. A container management apparatus characterized by comprising: The device comprises: An acquisition module is configured to acquire a container wake-up message; A wake-up module is configured to wake up a pod to be woken up in response to the container wake-up message; the pod to be woken up is in a hibernation state, in which state the threads of a virtual central processing unit (vCPU) of the pod to be woken up stop running, the pod business data of the pod to be woken up is stored in a storage device other than the memory of a virtual machine to which the pod to be woken up belongs, and the virtual machine process of the pod to be woken up is retained.

28. A container management apparatus characterized by comprising: The device comprises: A receiving module is configured to receive a request message; A control module is configured to generate a container wake-up message in response to the request message; the container wake-up message is configured to instruct to wake up a pod to be woken up; the pod to be woken up is in a hibernation state, in which a thread of a vCPU of the pod to be woken up is stopped from running, pod service data of the pod to be woken up is stored in a memory other than a memory of a virtual machine to which the pod to be woken up belongs, and a virtual machine process of the pod to be woken up is reserved.

29. A cloud platform, characterized by The system comprises a control node and at least one worker node, any worker node of the at least one worker node is configured to perform the operation steps of the method of any one of claims 1-14; and the control node is configured to perform the operation steps of the method of any one of claims 15-26.

30. An electronic device, comprising: The system comprises a memory and a processor, the memory is configured to store at least one set of computer instructions; when the processor executes the at least one set of computer instructions, the operation steps of the method of any one of claims 1-14 are performed; or the operation steps of the method of any one of claims 15-26 are performed.

31. A computer readable storage medium, characterized in that, The system comprises: computer software instructions; when the computer software instructions run in a controller, the controller performs the method of any one of claims 1-14; or performs the method of any one of claims 15-26.

Images