Container cluster management method and device

Abstract

The embodiment of the present application provides a method for managing a container cluster, the method comprising: the container cluster management CCM receives an instantiation request message of a container cluster from a management entity, the request message carries the instantiation parameters of the container cluster; the CCM instantiates the container cluster according to the instantiation parameters of the container cluster; the instantiation parameters of the container cluster are determined by the management entity by accessing the container cluster descriptor CCD. Through the scheme of the embodiment of the present invention, by defining a container cluster descriptor template and a container cluster node descriptor template, dynamic management of container clusters is supported, and consistent deployment and batch replication of large-scale container clusters are achieved.

Description

Technical Field

[0001] The present application relates to the field of communications, and in particular to a method and device for managing a container cluster. Background Art

[0002] Network Function Virtualization (NFV) refers to the use of virtualization technology in the field of information technology (IT) by telecom network operators to decouple the implementation of some telecom network functions (e.g. core network functions) in software and hardware in general-purpose servers, switches and storage, thereby achieving rapid and efficient deployment and operation of network services (NS), while achieving the goal of saving network investment costs CAPEX and operating costs OPEX. By applying NFV technology, telecom network functions are implemented in software and can run on general-purpose server hardware. They can be migrated, instantiated, and deployed in different physical locations of the network as needed, without the need to install new equipment.

[0003] The standardization work of NFV mainly focuses on network services, virtualized network functions (VNF) and dynamic management and orchestration (MANO) of virtual resources. The interface and architecture (IFA) working group of the NFV industry standard group under the European Telecommunications Standards Institute (ETSI) completes the functional formulation work within the MANO framework. Its functional architecture is as follows: Figure 1 As shown, the NFV system 100 mainly includes the following functional entities:

[0004] NFV orchestrator (NFVO) 102 is mainly responsible for the lifecycle management of NS, and for the allocation and scheduling of virtual resources in network functions virtualisation infrastructure (NFVI) 104. NFVO 102 can communicate with one or more virtualised network function managers (VNFM) 106 to perform operations related to instantiating NS, such as sending corresponding configuration information to VNFM 106 and requesting status information of one or more VNF 108 from VNFM 106. In addition, NFVO 102 can also communicate with virtual infrastructure manager (VIM) 110 to allocate and / or reserve various resources in NFVI 104, exchange resource configuration and status information, etc.

[0005] VNFM106 is mainly responsible for the life cycle management of one or more VNF108, such as instantiating VNF108, updating VNF108, querying VNF108, elastically scaling VNF108, terminating VNF108, etc. VNFM106 can communicate with VNF108 to manage the life cycle of VNF108, exchange configuration information and status information with VNF, etc. It can be understood that the NFV system 100 can include one or more VNFM106, and each VNFM106 performs life cycle management on different types of VNF108.

[0006] NFVI 104 refers to the infrastructure of NFV system 100, including hardware components, software components and their combination, so as to establish a virtualized environment, deploy, manage and implement VNF ​​108 in the virtualized environment. NFVI 104 may at least include computing hardware 1041, storage hardware 1042, and network hardware 1043; the virtualization layer 1044 of NFVI 104 may abstract the aforementioned hardware, decouple the hardware from VNF 108, and obtain corresponding virtual computing resources 1045, virtual storage resources 1046, and virtual network resources 1047, thereby providing virtual machines and other forms of virtualized containers for VNF 108.

[0007] VIM110 is mainly used to control and manage the interaction between VNF108 and computing hardware 1041, storage hardware 1042, network hardware 1043, virtual computing resources 1045, virtual storage resources 1046, and virtual network resources 1047. For example, VIM110 can perform resource management functions, such as adding corresponding virtual resources to virtual machines or other forms of virtual containers, collecting fault information of NFVI104 during system operation, etc. In addition, VIM110 can communicate with VNFM106, such as receiving resource allocation requests from VNFM106, and feeding back resource configuration and status information to VNFM106.

[0008] VNF108, VNF108 includes one or more VNFs (usually multiple VNFs), which can run one or more virtual machines or other forms of virtual containers, corresponding to a set of network functions originally implemented by dedicated devices.

[0009] The element management system (EMS) 112 may be used to configure and manage the VNF 108, and initiate lifecycle management operations such as instantiation of a new VNF 108 to the VNFM 106. It is understood that the NFV system 100 may include one or more EMSs 112.

[0010] The operations support system (OSS) or business support system (BSS) 114 can support various end-to-end telecommunications services. Among them, the management functions supported by OSS may include network configuration, service provision, fault management, etc.; BSS can be used to process orders, payments, revenue and other related services, and support product management, order management, revenue management and customer management functions. It should be noted that OSS / BSS114 can act as a service requester to request NFVO to instantiate NS, and OSS / BSS114 or the computing device on which OSS / BSS114 depends can usually be referred to as a service requester.

[0011] It is understandable that Figure 1 In the NFV system 100 shown, the aforementioned functional entities may be deployed in different computing devices respectively, or some functional entities may be integrated into the same computing device.

[0012] Currently, the network transformation in the telecommunications field is undergoing an evolution from Network Function Virtualisation (NFV) to Cloud-Native. Cloud-Native is a new system implementation paradigm for building, running and managing software in a cloud environment, making full use of cloud infrastructure and platform services, adapting to the cloud environment, and possessing key architectural features such as (micro)services, elastic scaling, distribution, high availability, multi-tenancy and automation. In this transformation, the introduction of container management in the reference architecture of NFV Management and Orchestration (MANO) is a key link in the many practices of NFV moving towards Cloud-Native.

[0013] Container as a Service (CaaS) is a specific type of Platform as a Service (PaaS). Generally speaking, containers are a virtualization technology at the operating system level that isolates different processes through operating system isolation technologies such as CGroup and NameSpace under Linux. Container technology is different from hardware virtualization (Hypervisor) technology. There is no virtual hardware, and there is no operating system inside the container, only processes. It is precisely because of this important feature of container technology that containers are lighter and easier to manage than virtual machines. In the running state of the container, a set of common management operations are defined, such as: start, stop, pause and delete, etc., to uniformly manage the life cycle of the container. The Kubernetes project of the Cloud Native Computing Foundation is currently recognized in the industry as the de facto standard for container management and orchestration.

[0014] The introduction of container-as-a-service architecture in the evolution of telecom network cloud native has brought about an agile change in the development and operation (DevOps) of the telecom industry. Corresponding to this change, traditional large-grained single-body network functions are gradually being deconstructed into services, and even further into microservices. Each service-based function is independently developed, delivered, and maintained, and version upgrades become more frequent; but on the other hand, the surge in the number of containerized network functions will not bring an exponential increase in the workload of interoperability testing, and the stable API interface definition ensures the consistency and reliability of interface function calls.

[0015] Currently, the most popular application in the field of container management and orchestration is Google's Kubernetes (K8S) container cluster management technology based on the open source platform. Its core idea is "everything is service-centric and everything revolves around services". Following this idea, the container application system built on Kubernetes can not only run independently on physical machines, virtual machines or enterprise private clouds, but can also be hosted on public clouds. Another feature of Kubernetes is automation. A service can scale up and down, self-diagnose, and is easy to upgrade.

[0016] The functional scope of container cluster management includes management of container clusters (creating / deleting container clusters) and management of container cluster nodes (increasing / reducing nodes in the cluster, elastically updating the size of the cluster). Container clusters can be created dynamically on demand, that is, NFV MANO determines the number of container clusters to be created and the capacity of each cluster based on the scale and reliability strategy of the managed containerized VNFs.

[0017] In the dynamic management mode of container clusters, how to manage container clusters so that the creation or update of container clusters is simple and fast, and batch operations are more efficient is particularly important in the large-scale containerized VNF management and orchestration of telecom clouds. Currently, there are some basic prototype tools for container cluster management in the open source community, such as Google Kubeadm, but these prototypes are not sufficient to support the needs of large-scale deployment and management of container clusters in telecom clouds. Summary of the invention

[0018] In order to solve the technical problems in the above-mentioned prior art, the embodiments of the present application provide a method and device for managing a container cluster node resource pool, which are as follows:

[0019] The present application provides a method for managing a container cluster node resource pool, the method comprising:

[0020] The container cluster management CCM receives an instantiation request message of a container cluster from a management entity, where the request message carries instantiation parameters of the container cluster; the CCM instantiates the container cluster according to the instantiation parameters of the container cluster; the instantiation parameters of the container cluster are determined by the management entity by accessing a container cluster descriptor CCD.

[0021] The method further comprises:

[0022] The CCM receives an instantiation request message of a container cluster node from a management entity, where the request message carries instantiation parameters of the container cluster node, where the instantiation parameters of the container cluster node are determined by the management entity accessing a container cluster node descriptor CCND; alternatively, the CCM accesses CCND to determine the instantiation parameters of the container cluster node.

[0023] The CCM instantiates the container cluster node according to the instantiation parameters of the container cluster node; and instantiates the CISM instance and the CIS instance on the container cluster node according to the instantiation parameters of the container cluster.

[0024] The method further includes: the CCM receives an update request message of the container cluster from the management entity, the request message carries parameters of the updated container cluster instance, and the CCM updates the container cluster instance according to the parameters of the updated container cluster instance.

[0025] The method further includes: the CCM receives a container cluster deletion request message from a management entity, the deletion request message carrying identification information of a container cluster instance to be deleted and / or a type of deletion operation; and the CCM deletes the container cluster instance.

[0026] An embodiment of the present application provides a management system for a container cluster, the system comprising:

[0027] The management entity is used to determine the instantiation parameters of the container cluster from the container cluster descriptor CCD, and send the instantiation parameters of the container cluster to the container cluster management CCM; the CCM is used to instantiate the container cluster according to the instantiation parameters of the container cluster.

[0028] The management entity is further used to access the container cluster node descriptor CCND to determine the instantiation parameters of the container cluster node; and send the instantiation parameters of the container cluster node to the CCM.

[0029] The CCM instantiates the container cluster node according to the instantiation parameters of the container cluster node and instantiates the CISM instance and the CIS instance on the container cluster node according to the instantiation parameters of the container cluster.

[0030] The embodiment of the present application also provides a container cluster management device, including a module for executing the above method steps.

[0031] An embodiment of the present application further provides a container cluster management device, characterized in that it includes a processor and a memory, the processor and the memory are coupled, and a computer program is stored in the memory; the processor is used to call the computer program in the memory so that the management device executes the above method.

[0032] An embodiment of the present application further provides a computer-readable storage medium, characterized in that a computer program is stored in the storage medium, and when the computer program is executed, the above method is performed.

[0033] A computer program product is also provided in an embodiment of the present application. The computer program product includes a computer program code. When the computer program code runs on a computing device, the computing device executes the above method.

[0034] Through the solution of the embodiment of the present invention, by defining a container cluster descriptor template and a container cluster node descriptor template, dynamic management of container clusters is supported, and consistent deployment and batch replication of large-scale container clusters are achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The following is a brief introduction to the drawings required for describing the embodiments or prior art.

[0036] Figure 1 This is a NFV system framework diagram in the prior art.

[0037] Figure 2 An architectural diagram of a Kubernetes (K8S) container management and orchestration system provided in an embodiment of the present application.

[0038] Figure 3 An architecture diagram of an NFV management and orchestration system for managing container clusters provided in an embodiment of the present application.

[0039] Figure 4 A logical relationship diagram of a container cluster, container cluster nodes, and namespace provided in an embodiment of the present application

[0040] Figure 5 A schematic diagram of a process for creating a container cluster provided in an embodiment of the present application.

[0041] Figure 6 A schematic diagram of a process for updating a container cluster provided in an embodiment of the present application.

[0042] Figure 7 A schematic diagram of a process for deleting a container cluster provided in an embodiment of the present application.

[0043] Figure 8 A schematic diagram of a management entity device module provided in an embodiment of the present application.

[0044] Fig. 9 A schematic diagram of a CCM device module provided in an embodiment of the present application.

[0045] Fig.10 A schematic diagram of the hardware structure of a management entity device provided in an embodiment of the present application.

[0046] Fig.11 A schematic diagram of the hardware structure of a CCM device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0048] Please refer to Figure 2 , Figure 2 An architectural diagram of the Kubernetes (K8S) container management and orchestration system.

[0049] Kubernetes divides the infrastructure resources in the container cluster into a Kubernetes master node (master) and a group of worker nodes (Node). Among them, the master node (also called the management node) runs a group of processes related to container cluster management, such as the Application Programming Interface Server (API Server), Replication Controller (RC), etc. These processes implement resource management, container warehouse (Pod) scheduling, elastic scaling, security control, system monitoring and error correction and other management functions of the entire container cluster. Kubelet, Proxy, and Docker run on each worker node, which are responsible for managing the life cycle of the Pod on this node and realizing the function of the service proxy. Figure 2 As shown, a Pod can include at least one container, so a Pod can be understood as a container warehouse composed of one or more containers.

[0050] The API Server provides the only operation entry for resource objects. All other components must operate resource data through the API interface it provides, and complete related business functions by "full query" and "change monitoring" of related resource data.

[0051] Controller Manager is the management and control center of the container cluster. Its main purpose is to realize the automatic fault detection and recovery of the Kubernetes cluster. For example, it can complete the replication or removal of Pod according to the definition of RC to ensure that the number of Pod instances meets the definition of RC, and complete the creation and update of the endpoint object of the service according to the management relationship between the service (SerVice) and the Pod, the discovery, management and status monitoring of the node, and the cleaning of the locally cached image files.

[0052] The Kubelet component is responsible for the creation, modification, monitoring, and deletion of Pods on this node throughout their life cycle. At the same time, Kubelet periodically reports the status information of this node to the API Server.

[0053] The Proxy component is used to implement service proxy and software mode load balancing.

[0054] The Docker component is the operating environment for the container.

[0055] The NFV industry standards group under the European Telecommunications Standards Institute (ETSI) has defined standardized functions for the NFV management and orchestration system management container in the Release 4 feature work, such as Figure 3 As shown in the reference function framework, the management plane on the right has two newly introduced logical functions as follows:

[0056] Container Infrastructure Service Management (CISM) (also known as CaaS management, the open source prototype is Kubernetes) is responsible for managing the container objects called by the containerized VNF, including the creation, update and deletion of container objects, and scheduling container objects to the corresponding node resources (computing, storage and network) in the container cluster node resource pool managed by it (CISM). The corresponding concept of container objects in the ETSI standard is managed container infrastructure objects (MCIO).

[0057] Container Cluster Management (CCM) is responsible for managing container clusters, including the creation of node resource pools used by container clusters and node scaling. A container cluster consists of a monitoring and management system (e.g. Figure 2 Kubernetes Master in ) and a set of compute nodes (for example, Figure 2 The nodes in the container cluster can be physical servers, bare metal or virtual machines. The container cluster is a dynamic system in which multiple containers can be deployed. The status of these containers and the communication between containers can be monitored by the system. The corresponding concept of the container cluster in the ETSI standard is the container infrastructure service cluster (CISCluster).

[0058] Containerized VNF can be understood as a containerized workload that encapsulates NFVI resources such as computing, storage, and networking. The container object MCIO called by the workload is scheduled to run on the node of the container cluster. The container cluster is loaded with the image of the CISM instance (CaaS management plane function, such as: Kubernetes Master) or the image of the Container Infrastructure Service (CIS) instance (CaaS user plane function, such as: kubelet, kube-proxy and docker on the Kubernetes worker node). In the ETSI NFV standard, the CISM in each container cluster provides management functions such as namespace creation, query, update and deletion (Create / Read / Update / Delete, CRUD). A namespace is a logical grouping composed of a set of specific identifiers, resources, policies, and authorizations. It acts like a folder in a server. NFVO can create multiple namespaces in a container cluster, and use namespaces to implement resource and identity isolation of the container object MCIO of multiple tenants (i.e., containerized VNF) in the container cluster. The relationship between the container cluster (CIS cluster), container cluster node (CIS cluster node) and namespace (namespace) is as follows Figure 4 As shown in Figure 2, CISM and CCM provide management services for NFVO or VNFM to call their functions on the northbound interface.

[0059] The solution of the present invention proposes a container cluster management method based on NFV template, which supports dynamic management of container clusters and realizes consistent deployment and batch replication of large-scale container clusters by defining a container cluster descriptor template and a container cluster node descriptor template and applying the newly defined descriptor template in the container cluster management process.

[0060] The container cluster descriptor (CCD, CIS Cluster Descriptor) is a type of NFV template file defined in the embodiment of the present invention that describes the deployment and operation behavior requirements of the container cluster. CCD can refer to or use a template similar to VNFD (Virtual Network Function Descriptor), which includes but is not limited to the following basic deployment information:

[0061] The name, identifier, provider, version information, etc. of the container cluster descriptor.

[0062] The size of the container cluster, that is, the maximum number of CISM instances and / or the maximum number of CIS instances contained in the container cluster.

[0063] The basic characteristics of the container cluster's scale operation, including the minimum step size, maximum step size, and / or achievable scale level that the container cluster can perform in the scale operation.

[0064] The affinity / anti-affinity rules of the container cluster as a whole refer to the identification information of the affinity / anti-affinity group to which the container cluster instance created based on the CCD belongs, and are used to indicate the affinity / anti-affinity relationship between the container cluster instance created based on the CCD and the container cluster instance created based on other CCDs. An affinity group is a logical relationship group formed around resource proximity. Objects belonging to the same affinity group use similar resources during deployment, for example, all objects in an affinity group are deployed in the same data center. An anti-affinity group is a logical relationship group formed around resource distance. Objects belonging to the same anti-affinity group use dissimilar resources during deployment, for example, each object in an anti-affinity group is deployed in a different data center.

[0065] The affinity / anti-affinity rules between CISM instances deployed in the container cluster refer to the identification information of the affinity / anti-affinity group to which the CISM instance in the container cluster instance created based on the CCD is located, and are used to indicate the affinity / anti-affinity relationship between the CISM instance in the container cluster instance created based on the CCD and other CISM instances in the same container cluster instance created based on the CCD.

[0066] The affinity / anti-affinity rules between CIS instances deployed in a container cluster refer to the identification information of the affinity / anti-affinity group to which the CIS instance in the container cluster instance created based on the CCD belongs, and are used to indicate the affinity / anti-affinity relationship between the CIS instance in the container cluster instance created based on the CCD and other CIS instances in the same container cluster instance created based on the CCD.

[0067] The affinity / anti-affinity rules between the CISM instance and the CIS instance deployed in the container cluster refer to the identification information of the affinity / anti-affinity group to which the CISM instance and the CIS instance in the container cluster instance created based on the CCD are located, and are used to indicate the affinity / anti-affinity relationship between the CISM instance in the container cluster instance created based on the CCD and the CIS instance in the container cluster instance created based on the CCD.

[0068] The characteristics of the primary CIS cluster external network refer to the basic configuration information of the primary external network of the container cluster instance created based on the CCD, such as the characteristic requirements of the IP address and port for the containers in the container cluster to connect to the external network; among them, the primary container cluster external network refers to the network open to the outside of the container cluster, and the containers (OScontainer) in the container cluster are indirectly connected to the external public network through the native network capabilities of the underlying container infrastructure layer.

[0069] The characteristics of the secondary CIS cluster external network refer to the basic configuration information of the secondary external network of the container cluster instance created based on the CCD, such as the characteristic requirements of the container network interface (CNI) used by the container cluster; the secondary container cluster external network refers to the network exposed outside the container cluster, and the containers (OS containers) in the container cluster are directly interconnected through other network interfaces except the primary network interface.

[0070] Container Cluster Node Descriptor (CCND, CIS Cluster Node Descriptor) is a type of NFV template file that describes the deployment and operation behavior requirements of container cluster nodes. CCND is similar to the definition of virtual computing or storage resource descriptors, including but not limited to the following deployment information:

[0071] The type of the container cluster node created based on the CCND, for example, indicating whether the node type is a physical machine (bare metal) or a virtual machine.

[0072] Requirements for hardware acceleration, network interfaces, and local storage of container cluster nodes created based on the CCND.

[0073] The affinity / anti-affinity rules between nodes in the container cluster created based on the CCND refer to the identification information of the affinity / anti-affinity group in which the container cluster node instance created based on the CCND is located, and are used to indicate the affinity / anti-affinity relationship between the container cluster node (or container cluster node instance) created based on the CCND and other container cluster node instances created based on the CCND.

[0074] Based on the above template file, the first embodiment of the present invention provides a method for creating (or instantiating) a container cluster, such as Figure 5 As shown, it specifically includes the following steps:

[0075] Step 501: The NFV MANO management entity (or simply referred to as management entity, the same below) accesses the container cluster descriptor CCD, and obtains the deployment information of the container cluster to be created (or called container cluster instance) from the CCD file.

[0076] The management entity may be NFVO or VNFM. Which one executes all the steps of the method of this embodiment depends on the system configuration and is not specifically limited here.

[0077] Step 502: The management entity determines the instantiation parameters of the container cluster instance to be created based on the deployment information of the container cluster in the CCD, such as: the name or identification information of the container cluster descriptor CCD, the scale of the container cluster, the number of CISM instances and the number of CIS instances created by initializing the container cluster, and the affinity / anti-affinity rules between CISM instances within the container cluster, between CIS instances, and between CISM instances and CIS instances.

[0078] The management entity may use the deployment information of the container cluster in the container cluster descriptor CCD as instantiation parameters of the container cluster instance, or may determine the instantiation parameters of the container cluster instance by referring to the input of other network element systems (such as OSS / BSS) on the basis of satisfying the deployment information.

[0079] Step 503: The management entity sends a container cluster creation request to the container cluster management CCM. The request message carries the scale of the container cluster to be created, the number of CISM instances and CIS instances initially created for the container cluster, and the affinity / anti-affinity rules between CISM instances within the container cluster, between CIS instances, and between CISM instances and CIS instances.

[0080] Step 504: The CCM returns a container cluster creation response to the management entity, indicating that the container cluster creation request message is received successfully.

[0081] Step 505: The CCM sends a container cluster management process change notification to the management entity, indicating to the management entity that the container cluster instantiation process starts.

[0082] Step 506: The management entity obtains identification information of the container cluster node descriptor CCND of the container cluster node instance to be created from the container cluster descriptor CCD, and obtains a CCND file through the identification information of CCND; the management entity accesses CCND to obtain deployment information of the container cluster node instance to be created.

[0083] Step 507: The management entity determines the instantiation parameters of the container cluster node instance to be created according to the deployment information of the container cluster node in the CCND, for example: the type of the container cluster node and the affinity / anti-affinity group to which the container cluster node belongs.

[0084] Step 508: The management entity sends a container cluster node creation request to the container cluster management CCM. The request message carries the name or identification information of the container cluster node descriptor to be created, the type of the container cluster node, and the affinity / anti-affinity group to which the container cluster node belongs.

[0085] Step 509: The CCM returns a container cluster node creation response to the management entity, indicating that the container cluster node creation request message is received successfully.

[0086] Optionally, as an alternative method of steps 506 to 509, the CCM obtains identification information of the container cluster node descriptor CCND from the container cluster descriptor CCD, and determines the instantiation parameters of the container cluster node by accessing the container cluster node descriptor CCND.

[0087] Similarly, CCM can use the deployment information of the container cluster in the container cluster descriptor CCD as the instantiation parameters of the container cluster instance, or it can determine the instantiation parameters of the container cluster instance by referring to the input of other network element systems (such as OSS / BSS) based on the deployment information.

[0088] Step 510: CCM completes the creation process of the container cluster node initialized in the container cluster to be created, thereby completing the creation of the container cluster instance locally. Further, CCM accesses the container cluster descriptor CCD to obtain the software image information of the CISM instance and / or CIS instance to be deployed, deploys the CISM instance and CIS instance on the container cluster node (optionally, the CIS instance can also be created by the created CISM instance), and at the same time, CCM creates the information of the container cluster instance, such as: CCD identification information and version used by the instantiated container cluster instance, instantiation status, elastic scaling status, maximum elastic scaling level allowed, external network information, node resource information, etc.

[0089] It should be noted that the software image of the CISM instance and / or CIS instance can be stored in the package file of the container cluster in the NFV-MANO management domain, or in the software image library (image registry) outside the NFV-MANO management domain. The container cluster descriptor CCD contains index information pointing to the package file of the container cluster storing the software image of the CISM instance and / or CIS instance or the directory address of the external software image library.

[0090] Step 511: The CCM sends a container cluster management process change notification to the management entity, and sends a container cluster instantiation completion notification message to the management entity.

[0091] Embodiment 2 of the present invention provides a method for updating a container cluster. Figure 6 As shown, it specifically includes the following steps:

[0092] Step 601: The management entity sends a container cluster update request to the container cluster management CCM. The request message carries the identification information of the container cluster instance to be updated, the type of update operation is elastic scaling, the scale or elastic scaling level of the target container cluster achieved by the elastic scaling, and the affinity / anti-affinity rules between the target container cluster nodes of the elastic scaling.

[0093] Step 602: The CCM returns a container cluster update response to the management entity, indicating that the container cluster update request message is received successfully.

[0094] Step 603: The CCM sends a container cluster management process change notification to the management entity, indicating to the management entity that the container cluster update process starts.

[0095] Step 604: the management entity obtains identification information of the container cluster node descriptor CCND of the container cluster node instance to be updated from the container cluster descriptor CCD, and obtains the CCND file through the identification information of CCND; the management entity accesses CCND to obtain deployment information of the container cluster node instance to be created;

[0096] Step 605: The management entity determines the instantiation parameters of the container cluster node instance to be created according to the deployment information of the container cluster node instance in CCND, such as the name or identification information of the container cluster node descriptor, the type of the container cluster node, and the affinity / anti-affinity group to which the container cluster node belongs.

[0097] Step 606: The management entity sends a container cluster node creation request message to the container cluster management CCM. The request message carries the type of the container cluster node instance to be created and the affinity / anti-affinity group to which the container cluster node instance belongs.

[0098] Step 607: The CCM returns a container cluster node creation response to the management entity, indicating that the container cluster node creation request message is received successfully.

[0099] Optionally, as an alternative method of steps 604 to 607, the CCM obtains identification information of the container cluster node descriptor CCND from the container cluster descriptor CCD, and determines the instantiation parameters of the container cluster node by accessing the container cluster node descriptor CCND.

[0100] Step 608: The CCM completes the creation process of the container cluster node instance in the container cluster to be updated, and locally generates information of the newly created container cluster node instance.

[0101] Step 609: The CCM returns a container cluster update completion notification message to the management entity, indicating to the management entity that the container cluster update process is completed.

[0102] Embodiment 3 of the present invention provides a method for deleting a container cluster, such as Figure 7 As shown, it specifically includes the following steps:

[0103] Step 701: The management entity sends a container cluster deletion request message to the container cluster management CCM. The request message carries the identification information of the container cluster instance to be deleted and / or the type of deletion operation, such as forced deletion or graceful deletion.

[0104] Step 702: The CCM locally uninstalls the CISM instance and / or CIS instance in the container cluster to be deleted according to the type of deletion operation in the request message, releases the I-layer resources occupied by the container cluster node, deletes the container cluster node instance, and deletes the container cluster instance. At the same time, the CCM deletes the information of the container cluster instance.

[0105] Step 703: The CCM returns a container cluster deletion response to the management entity, indicating that the container cluster instance is deleted successfully.

[0106] The embodiment of the present invention adds an information model that defines a container cluster descriptor CCD and a container cluster node descriptor CCND in the NFV template. CCD mainly includes the scale of the cluster, elastic scaling properties, and affinity / anti-affinity rules of object instances in the cluster. CCND includes the type of node, the node's requirements for hardware acceleration, network interface, local storage, and affinity / anti-affinity rules between nodes in the container cluster. In the process of creating / updating / deleting a container cluster, the management entity obtains information about the container cluster to be created / updated / deleted by accessing CCD, obtains information about the nodes in the container cluster by accessing CCND, and sends a create / update / delete container cluster request to CCM based on this information. After the CCM completes the creation / update / deletion of the container cluster, it returns a response to the management entity.

[0107] By implementing the methods of the above-mentioned various embodiments, the solution of the embodiment of the present invention can support dynamic management of container clusters and realize consistent deployment and batch replication of large-scale container clusters.

[0108] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the interaction between various network elements. It can be understood that in order to realize the above functions, the above NFVO, VNFM or CCM, etc. include hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm operations of each example described in the embodiments disclosed in this document, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0109] The embodiment of the present application can divide the functional modules of NFVO, VNFM or CCM according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0110] For example, when the functional modules are divided in an integrated manner, Figure 8 The structure diagram of a communication device 80 is shown. The communication device 80 includes a transceiver module 801 and a processing module 802 .

[0111] Exemplarily, the communication device 80 is used to implement the functions of NFVO or VNFM. Figure 5 The embodiment shown, Figure 6 The embodiment shown or Figure 7 The NFVO or VNFM described in the illustrated embodiment.

[0112] In the embodiment of the present application, the communication device 80 may be a NFVO or a VNFM, or may be a chip applied to NFVO or VNFM or other combined devices, components, etc. having the above-mentioned NFVO or VNFM functions. When the communication device 80 is a NFVO or VNFM, the transceiver module 801 may be a transceiver, which may include an antenna and a radio frequency circuit, etc., and the processing module 802 may be a processor (or a processing circuit), such as a baseband processor, which may include one or more CPUs. When the communication device 80 is a component having the above-mentioned NFVO or VNFM functions, the transceiver module 801 may be a radio frequency unit, and the processing module 802 may be a processor (or a processing circuit), such as a baseband processor. When the communication device 80 is a chip system, the transceiver module 801 may be an input and output interface of a chip (such as a baseband chip), and the processing module 802 may be a processor (or a processing circuit) of the chip system, which may include one or more central processing units. It should be understood that the transceiver module 801 in the embodiment of the present application can be implemented by a transceiver or a transceiver-related circuit component, and the processing module 802 can be implemented by a processor or a processor-related circuit component (or, referred to as a processing circuit).

[0113] For example, the transceiver module 801 can be used to perform Figure 5 In the embodiment shown, all the transceiver operations performed by the NFVO or VNFM, such as S503, and / or other processes used to support the technology described herein. The processing module 802 can be used to perform Figure 5 All operations except the sending and receiving operations performed by the NFVO or VNFM in the illustrated embodiment, such as S501, S502, S505 and / or other processes for supporting the technology described in this document.

[0114] For another example, the transceiver module 801 may be used to execute Figure 6 In the embodiment shown, all the transceiver operations performed by the NFVO or VNFM, such as S603, and / or other processes for supporting the technology described herein. The processing module 802 may be used to perform Figure 6 All operations except the sending and receiving operations performed by the NFVO in the illustrated embodiment, such as S601, S602, S605 and / or other processes for supporting the technology described in this document.

[0115] For another example, the transceiver module 801 may be used to execute Figure 7 In the embodiment shown, all the transceiver operations performed by the NFVO or VNFM, such as S701, and / or other processes for supporting the technology described herein. The processing module 802 may be used to perform Figure 6All operations except the sending and receiving operations performed by the NFVO in the illustrated embodiment, such as S703, and / or other processes for supporting the technology described herein.

[0116] Similarly, the communication device 80 can also be used to implement the function of CCM. Figure 5 The embodiment shown, Figure 6 The embodiment shown or Figure 7 The CCM of the embodiment shown in the figure performs the Figure 5-Figure 7 All operations performed by the CCM in the illustrated embodiment will not be described in detail.

[0117] Fig. 9 A schematic diagram of the composition of a communication system is shown in FIG. Fig. 9 As shown, the communication system 90 may include: a management entity 901 and a CCM 902. It should be noted that: Fig. 9 This is only an exemplary figure, and the embodiments of this application are not limited to Fig. 9 The communication system 90 shown includes network elements and the number of network elements.

[0118] Among them, NFVO 901 is used to achieve the above Figure 5-Figure 7 Functions of the management entity 901 in the method embodiment shown. For example, the management entity 901 can be used to access the container cluster descriptor file CCD, obtain the deployment information of the container cluster to be created from the file, determine the instantiation parameters of the container cluster according to the deployment information of the container cluster in the CCD, and send a container cluster creation request to the container cluster management CCM, where the request message carries the instantiation parameters of the container cluster to be created.

[0119] CCM 902 is used to achieve the above Figure 5-Figure 7 Functions of the CCM in the method embodiment shown. For example, the CCM 902 returns a container cluster creation response to the management entity 901, indicating the success or failure of the container cluster creation and the reason for the creation failure, and creates a container cluster instance locally, and completes the initial creation of a specified number of container cluster nodes.

[0120] It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding network element of the communication system 90, and will not be repeated here.

[0121] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0122] In an embodiment of the present application, a computing device 1000 is provided, such as Fig.10 As shown, at least one memory 1030 is included for storing program instructions and / or data. The memory 1030 is coupled to the processor 1020. The processor 1020 implements corresponding functions by running the stored program instructions and / or processing the stored data. The computing device 1000 may be Figure 5 to Figure 7 The NFVO or VNFM in the embodiment shown can implement the functions of the NFVO or VNFM in the method provided in the embodiment. The computing device 1000 can be a chip system. In the embodiment of the present application, the chip system can be composed of a chip, or can include a chip and other discrete devices.

[0123] The computing device 1000 may also include a communication interface 1010 for communicating with other devices via a transmission medium. For example, the other device may be a control device. The processor 1020 may use the communication interface 1010 to send and receive data.

[0124] The specific connection medium between the communication interface 1010, the processor 1020 and the memory 1030 is not limited in the embodiment of the present application. Fig.10 In the embodiment, the memory 1030, the processor 1020 and the communication interface 1010 are connected via a bus 1040. Fig.10 The connections between the other components are shown in bold lines, which are only for illustration and are not intended to be limiting. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0125] In the embodiment of the present application, the processor 1020 may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiment of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0126] In an embodiment of the present application, the memory 1030 may be a non-volatile memory, such as a hard disk drive (HDD) or a solid-state drive (SSD), etc., or a volatile memory (volatile memory), such as a random access memory (RAM). The memory is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory in the embodiment of the present application may also be a circuit or any other device that can implement a storage function, for storing program instructions and / or data.

[0127] The present application also provides a computing device 1100, such as Fig.11 As shown, the computing device 1000 includes at least one memory 1130 for storing program instructions and / or data, and the memory 1130 is coupled to the processor 1120. The processor 1120 implements corresponding functions by running the stored program instructions and / or processing the stored data. The computing device 1000 may be Figure 5 to Figure 7 The CCM in the illustrated embodiment can implement the functions of the CCM in the method provided in the embodiment.

[0128] The computing device 1100 also includes a communication interface 1110 for communicating with other devices via a transmission medium. The processor 1120 can use the communication interface 1110 to send and receive data.

[0129] Other functions and structures are similar to those of the above-mentioned computing device 1000 and will not be repeated here.

[0130] A computer-readable storage medium is also provided in an embodiment of the present application for storing instructions. When the instructions are executed by a processor of a computing device, the computing device implements a method provided in any embodiment of the present application.

[0131] A computer program product is provided in an embodiment of the present application. The computer program product includes a computer program code. When the computer program code is executed on a computing device, the computing device executes a method provided in any one of the embodiments of the present application.

[0132] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present application.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions provided in the aforementioned embodiments can still be modified, or some of the technical features therein can be replaced by equivalents, and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions provided in the various embodiments of the present application.

Claims

1. A method for managing a container cluster, characterized in that: The method comprises: The container cluster management CCM receives an instantiation request message of a container cluster from a management entity, where the request message carries instantiation parameters of the container cluster, where the instantiation parameters of the container cluster include affinity / anti-affinity rules between container infrastructure service management CISM instances within the container cluster, between container infrastructure service CIS instances, and between CISM instances and CIS instances; The CCM instantiates the container cluster according to the instantiation parameters of the container cluster; The instantiation parameters of the container cluster are determined by the management entity by accessing a container cluster descriptor CCD.

2. The management method according to claim 1, characterized in that: The CCM instantiating the container cluster according to the instantiation parameters of the container cluster includes: The CCM receives an instantiation request message of a container cluster node from a management entity, where the request message carries instantiation parameters of the container cluster node, where the instantiation parameters of the container cluster node are determined by the management entity accessing a container cluster node descriptor CCND; or, CCM accesses CCND to determine the instantiation parameters of the container cluster nodes; The CCM instantiates the container cluster node according to the instantiation parameters of the container cluster node, and instantiates the CISM instance and / or CIS instance on the container cluster node according to the instantiation parameters of the container cluster.

3. The management method according to claim 2, characterized in that: The CCM instantiates the CISM instance and the CIS instance on the container cluster node according to the instantiation parameters of the container cluster, including: The CCM creates a container infrastructure service management CISM instance and / or a container infrastructure service CIS instance on the container cluster node; or The CCM creates a container infrastructure service management CISM instance on the container cluster node, and the CISM instance further creates a CIS instance on the container cluster node.

4. The management method according to claim 1, characterized in that: The instantiation parameters of the container cluster include one or more of the following: the name or identification information of the container cluster descriptor CCD, the scale of the container cluster, the number of CISM instances and the number of CIS instances created by initialization of the container cluster.

5. The management method according to claim 2, characterized in that: The instantiation parameters of the container cluster node include one or more of the following: a name or identification information of a container cluster node descriptor, a type of the container cluster node, and an affinity / anti-affinity group to which the container cluster node belongs.

6. The management method according to claim 1, characterized in that: The CCM receives an update request message of a container cluster from a management entity, where the request message carries parameters of an updated container cluster instance; The CCM updates the container cluster according to the parameters of the updated container cluster instance.

7. The management method according to claim 6, characterized in that: The CCM updating the container cluster according to the parameters of the updated container cluster instance includes: The CCM receives a container cluster node creation request message from a management entity, where the creation request message carries instantiation parameters of the container cluster node; or, CCM accesses CCND to determine the instantiation parameters of the container cluster nodes; The CCM creates a container cluster node according to the instantiation parameters of the container cluster node.

8. The management method according to claim 6, characterized in that: The parameters of the updated container cluster instance include one or more of the following: The identification information of the container cluster instance, the type of the update operation is elastic scaling, the scale or elastic scaling level of the target container cluster achieved by the elastic scaling, and the affinity / anti-affinity rules between the target container cluster nodes of the elastic scaling.

9. The management method according to claim 1, characterized in that: The CCM receives a container cluster deletion request message from the management entity, where the deletion request message carries identification information of the container cluster instance to be deleted and / or the type of the deletion operation; The CCM deletes the container cluster instance.

10. The management method according to claim 9, characterized in that: The CCM deletes the container cluster instance specifically including deleting each container cluster node included in the container cluster and the CISM instance and / or CIS instance on the node, releasing the I-layer resources occupied by the container cluster node, and deleting the container cluster instance.

11. The management method according to any one of claims 1 to 10, characterized in that: The management entity is a network function virtualization orchestrator NFVO or a virtual network function manager VNFM.

12. A method for managing a container cluster, characterized in that: The method comprises: The management entity accesses the container cluster descriptor CCD and determines the instantiation parameters of the container cluster, wherein the instantiation parameters of the container cluster include affinity / anti-affinity rules between CISM instances, between CIS instances, and between CISM instances and CIS instances within the container cluster; The management entity sends the instantiation parameters of the container cluster to the container cluster management CCM; The CCM instantiates the container cluster according to the instantiation parameters of the container cluster.

13. The management method according to claim 12, characterized in that: The CCM instantiating the container cluster according to the instantiation parameters of the container cluster includes: The management entity accesses the container cluster node descriptor CCND and determines instantiation parameters of the container cluster node; The management entity sends the instantiation parameters of the container cluster node to the CCM; or, CCM accesses CCND to determine the instantiation parameters of the container cluster nodes; The CCM instantiates the container cluster node according to the instantiation parameters of the container cluster node and instantiates the container infrastructure service management CISM instance and / or the container infrastructure service CIS instance on the container cluster node according to the instantiation parameters of the container cluster.

14. The management method according to claim 13, characterized in that: The CCM instantiates the CISM instance and the CIS instance on the container cluster node according to the instantiation parameters of the container cluster, including: The CCM creates a CISM instance and / or a CIS instance on a container cluster node; or The CCM creates a CISM instance on the container cluster node, and the CISM instance further creates a CIS instance on the container cluster node.

15. The management method according to claim 12, characterized in that: The instantiation parameters of the container cluster include one or more of the following: the name or identification information of the container cluster descriptor CCD, the scale of the container cluster, the number of CISM instances and the number of CIS instances created by initialization of the container cluster.

16. The management method according to claim 13, characterized in that: The instantiation parameters of the container cluster node include one or more of the following: a name or identification information of a container cluster node descriptor CCND, a type of the container cluster node, and an affinity / anti-affinity group to which the container cluster node belongs.

17. A management system for a container cluster, characterized in that: The system comprises: A management entity, used to access the container cluster descriptor CCD and determine the instantiation parameters of the container cluster, and send the instantiation parameters of the container cluster to the container cluster management CCM, where the instantiation parameters of the container cluster include affinity / anti-affinity rules between CISM instances, between CIS instances, and between CISM instances and CIS instances within the container cluster; The CCM is used to instantiate a container cluster according to the instantiation parameters of the container cluster.

18. The management system according to claim 17, characterized in that: The management entity is further used to access the container cluster node descriptor CCND and determine the instantiation parameters of the container cluster node; The management entity sends the instantiation parameters of the container cluster node to the CCM; The CCM instantiates the container cluster node according to the instantiation parameters of the container cluster node, and instantiates the container infrastructure service management CISM instance and / or the container infrastructure service CIS instance on the container cluster node according to the instantiation parameters of the container cluster.

19. The management system according to claim 17, characterized in that: The CCM is also used to access the CCND to determine the instantiation parameters of the container cluster node; The CCM instantiates the container cluster node according to the instantiation parameters of the container cluster node, and instantiates the container infrastructure service management CISM instance and / or the container infrastructure service CIS instance on the container cluster node according to the instantiation parameters of the container cluster.

20. The management system according to claim 18, characterized in that: The CCM instantiates the CISM instance and the CIS instance on the container cluster node according to the instantiation parameters of the container cluster, including: The CCM creates a CISM instance and / or a CIS instance on a container cluster node; or The CCM creates a CISM instance on the container cluster node, and the CISM instance further creates a CIS instance on the container cluster node.

21. A container cluster management device, comprising a module for executing the steps in the method according to any one of claims 1 to 11.

22. A container cluster management device, characterized in that: It includes a processor and a memory, the processor and the memory are coupled, and a computer program is stored in the memory; the processor is used to call the computer program in the memory so that the management device executes any method as claimed in claims 1 to 11.

23. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 11 is implemented.

Images