Management method and apparatus for a container cluster - Patents.com

Abstract

An embodiment of the present application provides a management method for a container cluster, the method includes: a container cluster management CCM receives a container cluster instantiation request message from a management entity, the request message carrying instantiation parameters of the container cluster; and the CCM instantiates the container cluster based on the container cluster instantiation parameters, the container cluster instantiation parameters being determined by the management entity by accessing a container cluster descriptor CCD. Using the solution of the embodiment of the present invention, a container cluster descriptor template and a container cluster node descriptor template are defined, and dynamic management for the container cluster is supported to implement consistent deployment and batch replication of large-scale container clusters.

Description

[Technical Field]

[0001] The present application relates to the field of communications, and in particular to a management method and apparatus for a container cluster. [Background technology]

[0002] Network Function Virtualization (NFV) refers to the decoupling of software and hardware when implementing some of the telecommunications network functions (such as core network functions) on general-purpose servers, switches, and memory using virtualization technology in the field of information technology (IT) to enable telecommunications network operators to implement fast and efficient deployment and operation of network services (NS) while achieving the goal of saving on network capital expenses (CAPEX) and operational expenses (OPEX). By applying NFV technology, telecommunications network functions can be implemented in the form of software, run on general-purpose server hardware, and can be migrated to, instantiated at, or deployed at various physical locations in the network as needed without installing new devices.

[0003] The standardization work of NFV mainly focuses on network services, virtualized network functions (VNFs), and dynamic management and orchestration of virtual resources (MANO). The function definition work in the MANO framework has been completed by the Interface and Architecture (IFA) Working Group of the NFV Industry Standards Committee under the European Telecommunications Standards Institute (ETSI). The functional architecture of NFV is shown in Figure 1. An NFV system 100 mainly includes the following functional entities:

[0004] NFV orchestrator (NFVO) 102: Primarily responsible for the lifecycle management of NSs, it is responsible for allocating and scheduling virtual resources in the network functions virtualization infrastructure (NFVI) 104. The NFVO 102 may communicate with one or more virtualized network function managers (VNFMs) 106 and perform operations related to NS instantiation, such as sending corresponding configuration information to the VNFMs 106 or requesting status information of one or more VNFs 108 from the VNFMs 106. Additionally, the NFVO 102 may further communicate with a virtualized infrastructure manager (VIM) 110 to perform allocation and / or reservation for each of the resources in the NFVI 104, exchange resource configuration and status information, etc.

[0005] VNFM 106: Primarily responsible for lifecycle management of one or more VNFs 108, e.g., instantiating, updating, querying, scaling, and terminating the VNFs 108. The VNFM 106 may communicate with the VNFs 108 to manage the lifecycle of the VNFs 108 and exchange configuration information, status information, and the like with the VNFs. It may be understood that the NFV system 100 may comprise one or more VNFMs 106, with each VNFM 106 performing lifecycle management for a different type of VNF 108.

[0006] NFVI 104: An infrastructure of the NFV system 100, including hardware components, software components, and combinations thereof, for establishing a virtualized environment and deploying, managing, and implementing the VNFs 108 in the virtualized environment. The NFVI 104 may include at least computing hardware 1041, storage hardware 1042, and network hardware 1043. A virtualization layer 1044 of the NFVI 104 may abstract the aforementioned hardware to provide another form of virtual machine and virtual container for the VNFs 108, decoupling the hardware from the VNFs 108 and obtaining corresponding virtual computing resources 1045, virtual storage resources 1046, and virtual network resources 1047.

[0007] VIM 110: Mainly configured to control and manage interactions between VNFs 108 and computing hardware 1041, storage hardware 1042, network hardware 1043, virtual computing resources 1045, virtual storage resources 1046, and virtual network resources 1047. For example, VIM 110 may perform resource management functions, such as adding corresponding virtual resources to another form of virtual machine or virtual container, and collecting fault information of NFVI 104 in the system operation process. In addition, VIM 110 may communicate with VNFM 106, such as receiving resource allocation requests from VNFM 106 and feeding back resource configuration and status information to VNFM 106.

[0008] VNF108: The VNF108 may include one or more VNFs (usually multiple VNFs) and may run one or more virtual machines or virtual containers of other types to correspond to groups of network functions originally implemented by dedicated devices.

[0009] An element management system (EMS) 112: may be configured to configure and manage the VNFs 108 and initiate lifecycle management operations such as instantiating new VNFs 108 to the VNFM 106. It may be understood that the NFV system 100 may include one or more EMSs 112.

[0010] Operations support system (OSS) or business support system (BSS) 114: Can support various end-to-end telecommunications businesses. Management functions supported by the OSS may include network configuration, business provisioning, fault management, etc., and the BSS may be configured to process related business such as orders, payments, and revenue to support functions such as product management, order management, revenue management, and customer management. It should be noted that the OSS / BSS 114 may be used as a business requester to request the NFVO to instantiate an NS, and that the OSS / BSS 114, or the computing device on which the OSS / BSS 114 relies, may be correspondingly referred to as a business requester.

[0011] It may be understood that in the NFV system 100 shown in FIG. 1, the aforementioned functional entities may be deployed separately in different computing devices, or some functional entities may be integrated into the same computing device.

[0012] Currently, network transformation in the telecommunications field is undergoing a process of evolving from Network Function Virtualization (NFV) to Cloud Native. Cloud Native is a new system implementation paradigm for building, running, and managing software in a cloud environment. It is an architectural practice that fully utilizes cloud infrastructure and platform services, adapts to the cloud environment, and has key characteristics such as (micro)servitization, scaling, decentralization, high availability, multi-tenancy, and automation. In this transformation, the introduction of container management into the reference architecture for NFV Management and Orchestration (MANO) is a key link in many practices of the evolution from NFV to Cloud Native.

[0013] Container as a Service (CaaS) is a specific type of Platform as a Service (PaaS). Generally, containers are operating system-level virtualization technologies that isolate various processes using operating system isolation techniques such as CGroup and NameSpace in Linux. Unlike hardware virtualization (hypervisor) technologies, container technology does not have virtual hardware, and inside a container there is only a process, not an operating system. This key feature of container technology makes containers lighter and easier to manage than virtual machines. In the running state of a container, a group of common management operations, such as start, stop, pause, and delete, is defined to manage the container lifecycle in a unified manner. The Cloud Native Computing Foundation's Kubernetes project is the currently recognized de facto standard for container management and orchestration.

[0014] The introduction of container-as-a-service architecture in the evolution process of telecommunication networks to cloud native brings about an agility transformation in the development and operations (DevOps) of the telecommunication industry. The corresponding change in this transformation is that traditional, coarse-grained single network functions are gradually decomposed into services, or even microservices. Each service-based function is developed, delivered, and maintained independently, and version upgrades become more frequent. However, the proliferation of containerized network functions does not exponentially increase the workload of interoperability testing. Stable API interface definitions ensure the consistency and reliability of interface function calls.

[0015] Currently, the most popular application in the field of container management and orchestration is Google's Kubernetes (abbreviated as K8S), a container cluster management technology based on an open source platform. The core idea of ​​Kubernetes is that "everything is service-centric and revolves around services." Based on this idea, container application systems built on Kubernetes can run independently on physical machines, virtual machines, or private enterprise clouds, and can also be hosted on public clouds. Another feature of Kubernetes is automation, which allows services to be automatically scaled, automatically diagnosed, and easily upgraded.

[0016] The functional scope of container cluster management includes container cluster management (creating or deleting a container cluster) and container cluster node management (adding / removing nodes in a cluster and elastically updating the size of a cluster). Container clusters can be created dynamically as needed, i.e., the NFV MANO decides the amount of container clusters to be created and the respective capacity of the clusters based on the size and reliability policies of the containerized VNFs to be managed.

[0017] In the dynamic management mode of container clusters, how to manage container clusters is particularly important in large-scale containerized VNF management and orchestration in telecommunication clouds to make the creation or update of container clusters simple and quick and batch operations more efficient. Currently, the open source community has several basic container cluster management prototype tools, such as Google Kubeadm. However, these prototype tools cannot meet the requirements of large-scale container cluster deployment and management in telecommunication clouds. Summary of the Invention

[0018] To solve the aforementioned technical problems in the prior art, the embodiments of the present application provide a management method and apparatus for a container cluster node resource pool, the details of which are as follows:

[0019] An embodiment of the present application provides a management method for a container cluster node resource pool, which includes:

[0020] Container Cluster Management ( CCM ) receives a container cluster instantiation request message from a management entity, the request message carrying instantiation parameters of the container cluster. The CCM instantiates the container cluster based on the container cluster instantiation parameters, and the container cluster instantiation parameters are stored by the management entity in a container cluster descriptor. ( CCD ) is determined by accessing

[0021] The method further includes:

[0022] The CCM receives a container cluster node instantiation request message from a management entity, the request message carrying instantiation parameters for the container cluster node. The instantiation parameters for the container cluster node are stored by the management entity in a container cluster node descriptor. ( CCND ) Alternatively, the CCM accesses the CCND to determine the instantiation parameters of the container cluster node.

[0023] The CCM instantiates a container cluster node based on instantiation parameters of the container cluster node, and instantiates a CISM instance and a CIS instance on the container cluster node based on instantiation parameters of the container cluster.

[0024] The method further includes: a CCM receiving a container cluster update request message from a management entity, the request message carrying parameters of the container cluster instance to be updated; and the CCM updating the container cluster instance based on the parameters of the container cluster instance to be updated.

[0025] The method further includes: receiving a container cluster deletion request message from a management entity, the deletion request message carrying identification information of the container cluster instance to be deleted and / or a type of deletion operation; and deleting the container cluster instance.

[0026] An embodiment of the present application provides a management system for a container cluster, the system comprising: Container Cluster Descriptor ( CCD ) Determine instantiation parameters of the container cluster based on the container cluster management parameter. ( CCM ) and a CCM configured to instantiate a container cluster based on instantiation parameters of the container cluster.

[0027] The management entity is a container cluster node descriptor ( CCND ) to determine instantiation parameters of the container cluster node, and sending the instantiation parameters of the container cluster node to the CCM.

[0028] The CCM instantiates a container cluster node based on instantiation parameters of the container cluster node, and instantiates a CISM instance and a CIS instance on the container cluster node based on instantiation parameters of the container cluster.

[0029] An embodiment of the present application further provides a management device for a container cluster, including modules configured to perform the aforementioned method steps.

[0030] An embodiment of the present application further provides a management device for a container cluster, including a processor and a memory, the processor being coupled to the memory and storing a computer program, the processor being configured to invoke the computer program in the memory to enable the management device to implement the aforementioned method.

[0031] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, which, when executed, performs the aforementioned method.

[0032] An embodiment of the present application further provides a computer program product, which includes computer program code that, when run on a computing device, enables the computing device to perform the aforementioned method.

[0033] Using the solutions of the embodiments of the present invention, a container cluster descriptor template and a container cluster node descriptor template are defined, dynamic management for container clusters is supported, and consistent deployment and batch replication of large-scale container clusters is implemented. [Brief explanation of the drawings]

[0034] The accompanying drawings that are necessary to be used in describing the embodiments or the prior art are briefly described below.

[0035] [Figure 1] FIG. 1 is a framework diagram of an NFV system in the prior art. [Figure 2]FIG. 1 is an architecture diagram of a Kubernetes (K8S) container management and orchestration system according to an embodiment of the present application. [Figure 3] FIG. 1 is an architecture diagram of an NFV management and orchestration system for managing a container cluster according to an embodiment of the present application. [Figure 4] FIG. 2 is a diagram of a logical relationship between a container cluster, a container cluster node, and a namespace according to an embodiment of the present application. [Figure 5] 1 is a schematic flowchart of creating a container cluster according to an embodiment of the present application; [Figure 6] 1 is a schematic flowchart of updating a container cluster according to an embodiment of the present application; [Figure 7] 1 is a schematic flowchart of deleting a container cluster according to an embodiment of the present application; [Figure 8] 1 is a schematic module diagram of a management entity device according to an embodiment of the present application; [Figure 9] 1 is a schematic module diagram of a CCM device according to an embodiment of the present application. [Figure 10] 1 is a schematic diagram of a hardware structure of a management entity device according to an embodiment of the present application; [Figure 11] 1 is a schematic diagram of a hardware structure of a CCM device according to an embodiment of the present application; DETAILED DESCRIPTION OF THE INVENTION

[0036] Hereinafter, the technical solutions in the embodiments of the present application will be described with reference to the accompanying drawings in the embodiments of the present application.

[0037] Please refer to Figure 2, which is an architecture diagram of the Kubernetes (K8S) container management and orchestration system.

[0038] Kubernetes divides infrastructure resources in a container cluster into a Kubernetes master node and a group of worker nodes. A group of processes related to container cluster management, such as an Application Programming Interface Server (API Server) and a Replication Controller (RC), runs on the master node (also called a management node). These processes implement management functions for the entire container cluster, such as resource management, container pod scheduling, scaling, security control, system monitoring, and error correction. Each worker node runs three components: Kubelet, Proxy, and Docker, which are responsible for managing the lifecycle of pods on the current node and implementing service proxy functionality. As shown in Figure 2, a pod may contain at least one container. In this case, a pod may be understood as a container pod containing one or more containers.

[0039] The API server provides a unique operational entry for resource objects, and all other components must operate on the resource data through the API interfaces provided by the API server to implement the associated business functions by performing "full queries" and "change monitoring" on the associated resource data.

[0040] The Controller Manager is the management and control center of a container cluster, and its main purpose is to implement automatic failure detection and recovery for a Kubernetes cluster. For example, pod replication or removal can be performed based on the RC definition to ensure that the amount of pod instances conforms to the RC definition. Service endpoint object creation and update, node discovery, management, and status monitoring, and locally cached image file cleaning can be performed based on the management relationship between Services and Pods.

[0041] The Kubelet component is responsible for the complete lifecycle management of pods on the current node, including creating, modifying, monitoring, and deleting them. Additionally, Kubelet periodically reports the status information of the current node to the API server.

[0042] The Proxy component is configured to implement load balancing between the proxy pattern and the software pattern of services.

[0043] The Docker component is the environment in which the container runs.

[0044] The NFV Industry Standards Working Group under the European Telecommunications Standards Institute (ETSI) is defining standard functions for the NFV Management and Orchestration system management container in its Release 4 feature work. As shown in Figure 3, in this standard function framework, the management plane on the right side has two newly introduced logical functions:

[0045] Container Infrastructure Service Management (CISM) (also known as CaaS management, whose open source prototype is Kubernetes) is responsible for managing the container objects invoked by containerized VNFs, including creating, updating, and deleting container objects, and scheduling them to the corresponding node resources (compute, storage, and network resources) in the container cluster node resource pool managed by CISM. The corresponding concept of container object in the ETSI standard is Managed Container Infrastructure Object (MCIO).

[0046] Container Cluster Management (CCM) is responsible for managing container clusters, including creating node resource pools used by the container cluster and scaling nodes. A container cluster is a set formed by a monitoring and management system (e.g., the Kubernetes Master in Figure 2) and a set of computing nodes (e.g., the nodes in Figure 2, which can be physical servers, bare metal, or virtual machines). A container cluster is a dynamic system in which multiple containers can be deployed, and the system can monitor the status of these containers and the communication between them. The corresponding concept of a container cluster in the ETSI standard is a Container Infrastructure Service Cluster (CIS Cluster).

[0047] A containerized VNF can be understood as a containerized workload that encapsulates NFVI resources such as compute, storage, and network resources. A container object MCIO called by the workload is scheduled on a node of the container cluster where it will run. The container cluster loads an image of a CISM instance (a CaaS management plane function such as Kubernetes Master) or a Container Infrastructure Service (CIS) instance (a CaaS user plane function such as kubelet, kube-proxy, and docker on a Kubernetes worker node) on the node. In the ETSI NFV standard, CISM in each container cluster provides management functions such as creating, reading, updating, and deleting (CRUD) namespaces. A namespace is a logical group formed by a group of specific identifiers, resources, policies, and permissions, and has functions similar to those of a folder on a server. The NFVO may create multiple namespaces in a container cluster to isolate resources and identifiers of container objects MCIOs of multiple tenants (i.e., containerized VNFs) in the container cluster through the namespaces. Figure 4 shows the relationship between a container cluster (CIS cluster), a container cluster node (CIS cluster node), and namespaces. The CISM and CCM provide management services to the NFVO or VNFM to invoke their functions over the northbound interface.

[0048] The solution of the present invention provides a management method for container clusters based on NFV templates, and by defining container cluster descriptor templates and container cluster node descriptor templates and applying the newly defined descriptor templates in the container cluster management process, dynamic management of container clusters is supported and consistent deployment and batch replication of large-scale container clusters is implemented.

[0049] A Container Cluster Descriptor (CCD) is a type of NFV template file defined in an embodiment of the present invention that describes the deployment and operational behavior requirements of a container cluster. For a CCD, refer to or use a template similar to a Virtual Network Function Descriptor (VNFD) template. The template includes basic deployment information, including but not limited to:

[0050] The name, identity, provider, and version information of the container cluster descriptor.

[0051] The size of the container cluster, i.e., the maximum number of CISM instances and / or the maximum number of CIS instances that the container cluster will contain.

[0052] Basic characteristics of the container cluster's scale operation, including the minimum step, maximum step, and / or achievable scale level that may be performed by the container cluster during the scaling operation.

[0053] The affinity / anti-affinity rule for the entire container cluster is identification information of an affinity / anti-affinity group in which a container cluster instance created based on a CCD is located, and is for indicating an affinity / anti-affinity relationship between a container cluster instance created based on a CCD and a container cluster instance created based on another CCD. An affinity group is a logical relationship group formed based on resource similarity, and objects belonging to the same affinity group use similar resources during deployment, for example, all objects in an affinity group are deployed in the same data center. An anti-affinity group is a logical relationship group formed based on resource difference, and objects belonging to the same anti-affinity group use different resources during deployment, for example, all objects in an anti-affinity group are deployed in different data centers.

[0054] The affinity / anti-affinity rules between CISM instances deployed in a container cluster refer to the identification information of the affinity / anti-affinity group in which a CISM instance in a container cluster instance created based on CCD is located, and are intended to indicate the affinity / anti-affinity relationship between a CISM instance in a container cluster instance created based on CCD and another CISM instance in the same container cluster instance created based on CCD.

[0055] The affinity / anti-affinity rules between CIS instances deployed in a container cluster refer to the identification information of the affinity / anti-affinity group in which a CIS instance in a container cluster instance created based on a CCD is located, and are intended to indicate the affinity / anti-affinity relationship between a CIS instance in a container cluster instance created based on a CCD and another CIS instance in the same container cluster instance created based on a CCD.

[0056] The affinity / anti-affinity rules between CISM instances and CIS instances deployed in a container cluster refer to the identification information of the affinity / anti-affinity group in which the CISM instances and CIS instances in the container cluster instance created based on the CCD are located, and are intended to indicate the affinity / anti-affinity relationship between the CISM instances in the container cluster instance created based on the CCD and the CIS instances in the container cluster instance created based on the CCD.

[0057] The characteristics of the primary container cluster external network (primary CIS cluster external network) refer to the basic configuration information of the primary external network of a container cluster instance created based on CCD, such as the characteristic requirements of IP addresses and ports for containers in the container cluster to connect to the external network. The primary container cluster external network is the external public network of the container cluster, and containers (OS containers) in the container cluster are indirectly connected to the external public network through the native network capabilities of the underlying container infrastructure layer.

[0058] The characteristics of the secondary container cluster external network (secondary CIS cluster external network) refer to the basic configuration information of the secondary external network of a container cluster instance created based on CCD, for example, the characteristic requirements of the container network interface (CNI) used by the container cluster. The secondary container cluster external network refers to the externally exposed network of the container cluster, and the containers (OS containers) in the container cluster are directly interconnected through a network interface different from the primary network interface.

[0059] A Container Cluster Node Descriptor (CCND) is a type of NFV template file that describes the deployment and operational behavior requirements of a container cluster node. A CCND is similar to a virtual computing or storage resource descriptor definition and includes the following deployment information, but is not limited to:

[0060] The type of container cluster node that will be created based on CCND, for example, whether the node type is a physical machine (bare metal) or a virtual machine.

[0061] Requirements for container cluster nodes created based on CCND regarding hardware acceleration, network interfaces, and local storage.

[0062] The affinity / anti-affinity rules between nodes in a container cluster created based on CCND refer to the identification information of the affinity / anti-affinity group in which a container cluster node instance created based on CCND is located, and are intended to indicate the affinity / anti-affinity relationship between a container cluster node (also called a container cluster node instance) created based on CCND and another container cluster node instance created based on CCND.

[0063] Based on the above template file, embodiment 1 of the present invention provides a container cluster creation (also called instantiation) method. As shown in Figure 5, this method specifically includes the following steps:

[0064] Step 501: The NFV MANO management entity (or management entity for short, hereinafter the same) generates a container cluster descriptor ( CCD ) to obtain deployment information for the container cluster (also called container cluster instance) that will be created from the CCD file.

[0065] The management entity may be an NFVO or a VNFM, and which specifically implements all steps of the method in this embodiment depends on the system configuration, which is not particularly limited in this specification.

[0066] Step 502: Based on the deployment information of the container cluster in the CCD, the management entity determines the instantiation parameters of the container cluster instance to be created, e.g., the container cluster descriptor. ( CCD )The container cluster management unit 100 determines the name or identification information of the container cluster, the size of the container cluster, the number of CISM instances and the number of CIS instances to be created during initialization of the container cluster, and affinity / anti-affinity rules between the CISM instances, between the CIS instances, and between the CISM instances and the CIS instances in the container cluster.

[0067] The management entity is a container cluster descriptor ( CCD ) The deployment information of the container cluster in the network may be used as the instantiation parameters of the container cluster instance, or the instantiation parameters of the container cluster instance may be determined by referring to the input of another network element system (such as OSS / BSS) based on satisfying the deployment information.

[0068] Step 503: The management entity sends the container cluster creation request to the container cluster management ( CCM ) The request message contains the size of the container cluster to be created, the number of CISM instances and the number of CIS instances to be created during the initialization of the container cluster, and affinity / anti-affinity rules between the CISM instances, between the CIS instances, and between the CISM and CIS instances in the container cluster.

[0069] Step 504: The CCM returns a container cluster creation response to the management entity, indicating that the container cluster creation request message has been successfully received.

[0070] Step 505: The CCM sends a container cluster management process change notification to the management entity, indicating to the management entity that the container cluster instantiation process will begin.

[0071] Step 506: The management entity receives the container cluster node descriptor of the container cluster node instance to be created. ( CCND ) The identification information of the container cluster descriptor ( CCD ) to obtain a CCND file using the identification information of the CCND, and the management entity accesses the CCND to obtain deployment information for the container cluster node instance to be created.

[0072] Step 507: Based on the deployment information of the container cluster node in the CCND, the management entity determines the instantiation parameters of the container cluster node instance to be created, such as the type of container cluster node and the affinity / anti-affinity group to which the container cluster node belongs.

[0073] Step 508: The management entity sends the container cluster node creation request to the container cluster management ( CCM ) , and the request message carries the name or identification of the descriptor of the container cluster node to be created, the type of the container cluster node, and the affinity / anti-affinity group to which the container cluster node belongs.

[0074] Step 509: The CCM returns a container cluster node creation response to the management entity, indicating that the container cluster node creation request message has been successfully received.

[0075] Optionally, as an alternative to steps 506 to 509, the CCM may create a container cluster node descriptor ( CCND ) The identification information of the container cluster descriptor ( CCD ) Container cluster node descriptor obtained from ( CCND )The instantiation parameters of the container cluster node are determined by accessing the

[0076] Similarly, the CCM uses the container cluster descriptor ( CCD ) The deployment information of the container cluster in the network may be used as the instantiation parameters of the container cluster instance, or the instantiation parameters of the container cluster instance may be determined by referring to the input of another network element system (such as OSS / BSS) based on satisfying the deployment information.

[0077] Step 510: The CCM completes the process of creating an initialized container cluster node in the container cluster to be created, thereby locally completing the creation of the container cluster instance. Additionally, the CCM creates a container cluster descriptor ( CCD ) to obtain software image information of the CISM instance and / or CIS instance to be deployed, and deploys the CISM instance and CIS instance on the container cluster node (optionally, a CIS instance may also be created using the created CISM instance), and the CCM creates information about the container cluster instance, such as the CCD identification information and version used by the instantiated container cluster instance, instantiation status, scaling status, maximum allowed scaling level, external network information, and node resource information.

[0078] The software image of the CISM instance and / or the software image of the CIS instance may be stored in a package file of a container cluster in the NFV-MANO management domain, or may be stored in a software image registry outside the NFV-MANO management domain, and the container cluster descriptor( CCD ) Note that includes index information that points to a package file of a container cluster that stores software images of a CISM instance and / or a CIS instance, or points to a directory address of an external software image registry.

[0079] Step 511: The CCM sends a change notification of the container cluster management process to the management entity, and sends a container cluster instantiation termination notification message to the management entity.

[0080] Embodiment 2 of the present invention provides a container cluster update method. As shown in Figure 6, the method specifically includes the following steps:

[0081] Step 601: The management entity sends a container cluster update request to the container cluster management ( CCM ) The request message carries the identification information of the container cluster instance to be updated, the type of update operation which is scaling, the size of the target container cluster to be reached by scaling or scaling level, and the affinity / anti-affinity rules between nodes of the target container cluster of the scaling.

[0082] Step 602: The CCM returns a container cluster update response to the management entity to indicate that the container cluster update request message has been successfully received.

[0083] Step 603: The CCM sends a container cluster management process change notification to the management entity, indicating to the management entity that a container cluster update process will begin.

[0084] Step 604: The management entity receives the container cluster node descriptor of the container cluster node instance to be updated. (CCND ) The identification information of the container cluster descriptor ( CCD ) to obtain a CCND file using the identification information of the CCND, and the management entity accesses the CCND to obtain deployment information for the container cluster node instance to be created.

[0085] Step 605: Based on the deployment information of the container cluster node instance in the CCND, the management entity determines the instantiation parameters of the container cluster node instance to be created, such as the name or identification information of the container cluster node descriptor, the type of the container cluster node, and the affinity / anti-affinity group to which the container cluster node belongs.

[0086] Step 606: The management entity sends the container cluster node creation request message to the container cluster management ( CCM ) , and the request message carries the type of container cluster node instance to be created and the affinity / anti-affinity group to which the container cluster node instance belongs.

[0087] Step 607: The CCM returns a container cluster node creation response to the management entity, indicating that the container cluster node creation request message has been successfully received.

[0088] Optionally, as an alternative to steps 604 to 607, the CCM may create a container cluster node descriptor ( CCND ) The identification information of the container cluster descriptor (CCD ) Container cluster node descriptor obtained from ( CCND ) The instantiation parameters of the container cluster node are determined by accessing the

[0089] Step 608: The CCM completes the process of creating a container cluster node instance in the container cluster to be updated, and locally generates information about the newly created container cluster node instance.

[0090] Step 609: The CCM sends a container cluster update complete notification message to the management entity to indicate to the management entity that the container cluster update process is finished.

[0091] Embodiment 3 of the present invention provides a container cluster deletion method. As shown in Figure 7, the method specifically includes the following steps:

[0092] Step 701: The management entity sends a container cluster deletion request message to the container cluster management ( CCM ) , and the request message carries the identity of the container cluster instance to be deleted and / or the type of deletion operation, for example, forceful deletion or graceful deletion.

[0093] Step 702: Based on the type of deletion operation in the request message, the CCM locally uninstalls the CISM instance and / or the CIS instance in the container cluster to be deleted, releases Layer I resources occupied by the container cluster node, deletes the container cluster node instance, and deletes the container cluster instance. In addition, the CCM deletes information about the container cluster instance.

[0094] Step 703: The CCM returns a container cluster deletion response to the management entity to indicate that the container cluster instance has been successfully deleted.

[0095] In this embodiment of the invention, the container cluster descriptor (CCD ) and container cluster node descriptor ( CCND ) An information model that defines the following is added to the NFV template. The CCD mainly includes the size of the cluster, scaling attributes, and affinity / anti-affinity rules for object instances in the cluster. The CCND includes the node type, node requirements for hardware acceleration, network interface, and local storage, and affinity / anti-affinity rules between nodes in the container cluster. In the process of creating, updating, or deleting a container cluster, the management entity accesses the CCD to obtain information about the container cluster to be created, updated, or deleted, accesses the CCND to obtain information about the nodes in the container cluster, and sends a request to the CCM to create, update, or delete the container cluster based on the information. After completing the creation, update, or deletion of the container cluster, the CCM returns a response to the management entity.

[0096] By implementing the methods in the aforementioned embodiments, the solution in the embodiments of the present invention can support dynamic management of container clusters and implement consistent deployment and batch replication of large-scale container clusters.

[0097] The above describes the solutions provided in the embodiments of the present application mainly from the perspective of interactions between network elements. It can be understood that, to implement the aforementioned functions, the NFVO, VNFM, CCM, etc., include corresponding hardware structures and / or software modules for performing the functions. In combination with the example units and algorithm operations described in the embodiments disclosed herein, those skilled in the art will readily recognize that the present application can be implemented by hardware or a combination of hardware and computer software. Whether the functions are implemented by hardware or by hardware driven by computer software depends on the specific application and design constraints of the technical solution. Those skilled in the art may implement the described functions using various methods for each specific application, but this implementation should not be considered to exceed the scope of the present application.

[0098] In the embodiments of the present application, functional modules in the NFVO, VNFM, CCM, etc. may be divided based on the above-mentioned exemplary method. For example, the functional modules may be divided according to their functions, or two or more functions may be integrated into one processing module. The integrated module may be implemented in the form of hardware or in the form of a software functional module. It should be noted that the division of modules in the embodiments of the present application is an example and merely represents the division of logical functions. In actual implementation, other division methods may be used.

[0099] For example, when the functional modules are divided in an integrated manner, Figure 8 shows a schematic diagram of the structure of a communication device 80. The communication device 80 includes a transceiver module 801 and a processing module 802.

[0100] For example, the communication device 80 is configured to implement the functionality of an NFVO or a VNFM, for example, the communication device 80 is the NFVO or the VNFM in the embodiment shown in FIG. 5, the embodiment shown in FIG. 6, or the embodiment shown in FIG. 7.

[0101] In an embodiment of the present application, the communication device 80 may be an NFVO or VNFM, or may be a chip applied to the NFVO or VNFM, or may be another combined device or component having the functionality of the aforementioned NFVO or VNFM. When the communication device 80 is an NFVO or VNFM, the transceiver module 801 may be a transceiver that may include an antenna, a radio frequency circuit, etc., and the processing module 802 may be a processor (or processing circuit), for example, a baseband processor that may have one or more CPUs. When the communication device 80 is a component having the functionality of the aforementioned NFVO or VNFM, the transceiver module 801 may be a radio frequency unit, and the processing module 802 may be a processor (or processing unit), for example, a baseband processor. When the communication device 80 is a chip system, the transceiver module 801 may be an input / output interface of a chip (e.g., a baseband chip), and the processing module 802 may be a processor (or processing circuit) of the chip system and may include one or more central processing units. It should be understood that the transceiver module 801 in the embodiments of the present application may be implemented by a transceiver or by circuit components associated with a transceiver, and the processing module 802 may be implemented by a processor or by circuit components associated with a processor (also referred to as processing circuitry).

[0102] For example, the transceiver module 801 may be configured to perform all of the receive and transmit operations, e.g., S503, performed by the NFVO or VNFM in the embodiment shown in Figure 5, and / or to support other processes of the techniques described herein. The processing module 802 may be configured to perform all of the operations, other than the receive and transmit operations, performed by the NFVO or VNFM in the embodiment shown in Figure 5, e.g., S501, S502, and S505, and / or to support other processes of the techniques described herein.

[0103] In another example, the transceiver module 801 may be configured to perform all of the receive and transmit operations, e.g., S603, performed by the NFVO or VNFM in the embodiment shown in Figure 6, and / or to support other processes of the techniques described herein. The processing module 802 may be configured to perform all of the operations, other than the receive and transmit operations, performed by the NFVO in the embodiment shown in Figure 6, e.g., S601, S602, and S605, and / or to support other processes of the techniques described herein.

[0104] In another example, the transceiver module 801 may be configured to perform all of the receive and transmit operations, e.g., S701, performed by the NFVO or VNFM in the embodiment shown in FIG. 7 and / or to support other processes of the techniques described herein. 7

[0046] It may be configured to perform all of the operations, except for the receive and transmit operations, performed by the NFVO in the embodiment shown in Figure 1, e.g., S703, and / or to support other processes of the techniques described herein.

[0105] Similarly, the communication device 80 may also be configured to implement the functions of the CCM that is the CCM in the embodiment shown in FIG. 5, the embodiment shown in FIG. 6, or the embodiment shown in FIG. 7, and to perform all operations performed by the CCM in the embodiments shown in FIGS. 5 to 7, and the details will not be described again.

[0106] 9 shows a schematic composition diagram of a communication system. As shown in FIG. 9, the communication system 90 may include a management entity 901 and a CCM 902. It should be noted that FIG. 9 is only an example of the accompanying drawings, and the network elements and the amount of network elements included in the communication system 90 shown in FIG. 9 are not limited in the embodiments of the present application.

[0107] The NFVO 901 is configured to implement the functions of the management entity 901 in the method embodiments shown in Figures 5 to 7. For example, the management entity 901 may (CCD) Phi To Access and obtain deployment information of the container cluster to be created from the file, determine instantiation parameters of the container cluster based on the deployment information of the container cluster in the CCD, and submit the container cluster creation request to the container cluster management. ( CCM ) , where the request message carries instantiation parameters of the container cluster to be created.

[0108] The CCM 902 is configured to implement the functions of the CCM in the method embodiments shown in Figures 5 to 7. For example, the CCM 902 returns a container cluster creation response to the management entity 901 to indicate the success or failure of the container cluster creation and the cause of the creation failure, creates a container cluster instance locally, and completes the initial creation of a specified amount of container cluster nodes.

[0109] It should be noted that all the associated contents of the steps in the foregoing method embodiments may be referred to for the functional description of the corresponding network elements of the communication system 90, and will not be described again in detail here.

[0110] The above description of the implementation allows those skilled in the art to understand that the division of the above functional modules is taken as an example for illustration purposes for the purpose of simple description. In actual application, the above functions can be allocated to different modules and implemented according to requirements, that is, the internal structure of the device is divided into different functional modules to implement all or some of the above functions.

[0111] An embodiment of the present application provides a computing device 1000 as shown in FIG. 10 , which includes at least one memory 1030 configured to store program instructions and / or data. The memory 1030 is coupled to a processor 1020. The processor 1020 implements corresponding functions by running the stored program instructions and / or processing the stored data. The computing device 1000 may be an NFVO or a VNFM in the embodiments shown in FIGS. 5 to 7 and may implement the functions of the NFVO or VNFM in the methods provided in the embodiments. The computing device 1000 may be a chip system. In the embodiment of the present application, the chip system may include a chip, or may include a chip and another discrete device.

[0112] The computing device 1000 may further include a communication interface 1010 configured to communicate with another device using a transmission medium. For example, the other device may be a control device. The processor 1020 may receive and transmit data through the communication interface 1010.

[0113] The specific connection medium between the communication interface 1010, the processor 1020, and the memory 1030 is not limited in the embodiment of the present application. In this embodiment of the present application, the memory 1030, the processor 1020, and the communication interface 1010 are connected to each other using a bus 1040 in FIG. 10. The bus is represented using a thick line in FIG. 10. The manner of connection between other components is only described schematically and is not used as a limitation. The bus may be classified as an address bus, a data bus, a control bus, etc. For ease of explanation, the bus in FIG. 10 is represented using only one thick line, but this does not indicate that there is only one bus or one type of bus.

[0114] In the embodiments of the present application, the processor 1020 may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or perform the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor, any conventional processor, etc. The steps of the methods disclosed in the embodiments of the present application may be performed and completed directly using a hardware processor, or may be performed and completed using a combination of hardware and software modules in the processor.

[0115] In an embodiment of the present application, the memory 1030 may be a non-volatile memory such as a hard disk drive (HDD) or a solid-state drive (SSD), or may be a volatile memory, for example, a random-access memory (RAM). The memory is any medium capable of holding or storing expected program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. The memory according to an embodiment of the present application may also be a circuit or any other device capable of implementing a storage function and configured to store program instructions and / or data.

[0116] An embodiment of the present application further provides a computing device 1100 as shown in Figure 11, where the computing device 1100 includes at least one memory 1130 configured to store program instructions and / or data. The memory 1130 is coupled to the processor 1120. The processor 1120 implements corresponding functions by executing the stored program instructions and / or processing the stored data. Computing Device 1100 may be the CCM in the embodiments shown in FIGS. 5 to 7 and may implement the functions of the CCM in the methods provided in the embodiments.

[0117] The computing device 1100 also includes a communication interface 1110 configured to communicate with another device using a transmission medium. The processor 1020 may receive and transmit data through the communication interface 1110.

[0118] Other functions and structures are similar to those of the computing device 1000 described above, and will not be described in detail again here.

[0119] An embodiment of the present application further provides a computer-readable storage medium configured to store instructions that, when executed by a processor of a computing device, enable the computing device to implement the methods provided in any of the embodiments of the present application.

[0120] An embodiment of the present application further provides a computer program product, which includes computer program code that, when run on a computing device, enables the computing device to perform the methods provided in any embodiment of the present application.

[0121] Those skilled in the art may recognize that the present application may be implemented using electronic hardware or a combination of computer software and electronic hardware in combination with the example units and algorithm steps described in the embodiments disclosed herein. Whether a function is performed in hardware mode or software mode depends on the specific application and design constraints of the technical solution. Those skilled in the art may implement the described functions using various methods for each specific application, but this implementation should not be considered to exceed the scope of the embodiments of the present application.

[0122] Finally, please note that the above embodiments are only intended to explain the technical solutions of the present application, and are not intended to limit the present application. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications may still be made to the technical solutions provided in the above embodiments, or equivalent substitutions may be made to some technical features thereof, without departing from the scope of the technical solutions provided in the embodiments of the present application.

Claims

1. 1. A management method for a container cluster, the method comprising: receiving a container cluster instantiation request message from a management entity by a container cluster management (CCM), the request message carrying instantiation parameters for the container cluster; instantiating, by the CCM, the container cluster based on the instantiation parameters of the container cluster; Including, A management method for a container cluster, wherein the instantiation parameters of the container cluster are determined by the management entity by accessing a container cluster descriptor (CCD), and the instantiation parameters of the container cluster include affinity / anti-affinity rules between CISM instances, between CIS instances, and between the CISM instances and the CIS instances in the container cluster.

2. The step of instantiating the container cluster by the CCM based on the instantiation parameters of the container cluster includes: receiving, by the CCM, a container cluster node instantiation request message from the management entity, the request message carrying instantiation parameters of the container cluster node, and the instantiation parameters of the container cluster node are determined by the management entity by accessing a Container Cluster Node Descriptor (CCND); or accessing the CCND by the CCM to determine the instantiation parameters of the container cluster node; instantiating, by the CCM, the container cluster node based on the instantiation parameters of the container cluster node, and instantiating a container infrastructure services management (CISM) instance and / or a container infrastructure services (CIS) instance on the container cluster node based on the instantiation parameters of the container cluster. The management method according to claim 1 ,

3. instantiating a CISM instance and a CIS instance on the container cluster node based on the instantiation parameters of the container cluster by the CCM, creating the Container Infrastructure Services Management (CISM) instance and / or the Container Infrastructure Services (CIS) instance on the container cluster node by the CCM; or creating the container infrastructure services management (CISM) instance on the container cluster node by the CCM, and further creating the CIS instance on the container cluster node by the CISM instance. The management method according to claim 2 .

4. 2. The method of claim 1, wherein the instantiation parameters of the container cluster further include one or more of a name or identification of the container cluster descriptor (CCD), a size of the container cluster, and an amount of CISM instances and an amount of CIS instances created during initialization of the container cluster.

5. 3. The method of claim 2, wherein the instantiation parameters of the container cluster node include one or more of a name or identification of the container cluster node descriptor, a type of container cluster node, and an affinity / anti-affinity group to which the container cluster node belongs.

6. receiving, by the CCM, an update request message for the container cluster from the management entity, the request message carrying parameters of a container cluster instance to be updated; updating, by the CCM, the container cluster based on the parameters of the container cluster instance to be updated; The method of claim 1 , further comprising:

7. The step of updating the container cluster based on the parameters of the container cluster instance to be updated by the CCM comprises: receiving, by the CCM, a container cluster node creation request message from the management entity, the creation request message carrying instantiation parameters of the container cluster node; or accessing a CCND by the CCM to determine the instantiation parameters of the container cluster node; creating, by the CCM, the container cluster node based on the instantiation parameters of the container cluster node; The management method according to claim 6 , comprising:

8. The parameters of the container cluster instance to be updated are:

7. The method of claim 6, further comprising one or more of: an identification of the container cluster instance; a type of update operation that is scaling; a size of a target container cluster to be reached by the scaling or scaling level; and affinity / anti-affinity rules between nodes of the target container cluster for the scaling.

9. receiving, by the CCM, a delete request message for the container cluster from the management entity, the delete request message carrying an identification of a container cluster instance to be deleted and / or a type of delete operation; deleting the container cluster instance by the CCM; The method of claim 1 , further comprising:

10. The management method of claim 9, wherein the step of deleting the container cluster instance by the CCM specifically includes the steps of deleting each of the container cluster nodes included in the container cluster and a CISM instance and / or a CIS instance on the node, releasing layer I resources occupied by the container cluster node, and deleting the container cluster instance.

11. The method of claim 1 , wherein the management entity is a Network Function Virtualization Orchestrator (NFVO) or a Virtualized Network Function Manager (VNFM).

12. 1. A management method for a container cluster, the method comprising: accessing, by a management entity, a container cluster descriptor (CCD) to determine instantiation parameters of said container cluster; sending, by the management entity, the instantiation parameters of the container cluster to a container cluster management (CCM); instantiating, by the CCM, the container cluster based on the instantiation parameters of the container cluster; the instantiation parameters of the container cluster include affinity / anti-affinity rules between CISM instances, between CIS instances, and between the CISM and CIS instances in the container cluster; A management method for a container cluster, including:

13. The step of instantiating the container cluster by the CCM based on the instantiation parameters of the container cluster includes: accessing, by the management entity, a container cluster node descriptor (CCND) to determine instantiation parameters of the container cluster node; sending the instantiation parameters of the container cluster node to the CCM by the management entity; or accessing the CCND by the CCM to determine the instantiation parameters of the container cluster node; instantiating, by the CCM, the container cluster node based on the instantiation parameters of the container cluster node, and instantiating a container infrastructure services management (CISM) instance and / or a container infrastructure services (CIS) instance on the container cluster node based on the instantiation parameters of the container cluster node. The management method according to claim 12 , comprising:

14. instantiating a CISM instance and a CIS instance on the container cluster node based on the instantiation parameters of the container cluster by the CCM, creating the CISM instance and / or the CIS instance on the container cluster node by the CCM; or creating the CISM instance on the container cluster node by the CCM and further creating the CIS instance on the container cluster node by the CISM instance. The management method according to claim 13 , comprising:

15. 13. The method of claim 12, wherein the instantiation parameters of the container cluster further include one or more of a name or identification of the container cluster descriptor (CCD), a size of the container cluster, and an amount of CISM instances and an amount of CIS instances created during initialization of the container cluster.

16. 14. The method of claim 13, wherein the instantiation parameters of the container cluster node include one or more of a name or identification of the container cluster node descriptor (CCND), a type of container cluster node, and an affinity / anti-affinity group to which the container cluster node belongs.

17. 1. A management system for a container cluster, the system comprising: a management entity configured to access a container cluster descriptor (CCD) to determine instantiation parameters of the container cluster and to send the instantiation parameters of the container cluster to a container cluster management (CCM); the CCM configured to instantiate the container cluster based on the instantiation parameters of the container cluster; the instantiation parameters of the container cluster include affinity / anti-affinity rules between CISM instances, between CIS instances, and between the CISM and CIS instances in the container cluster; A management system.

18. The management entity is further configured to access a container cluster node descriptor (CCND) to determine instantiation parameters of the container cluster node; The management entity sends the instantiation parameters of the container cluster node to the CCM; the CCM instantiates the container cluster node based on the instantiation parameters of the container cluster node, and instantiates a container infrastructure services management (CISM) instance and / or a container infrastructure services (CIS) instance on the container cluster node based on the instantiation parameters of the container cluster node. The management system according to claim 17.

19. The CCM is further configured to access a CCND to determine instantiation parameters of a container cluster node; the CCM instantiates the container cluster node based on the instantiation parameters of the container cluster node, and instantiates a container infrastructure services management (CISM) instance and / or a container infrastructure services (CIS) instance on the container cluster node based on the instantiation parameters of the container cluster node. The management system according to claim 17.

20. instantiating, by the CCM, a CISM instance and a CIS instance on the container cluster node based on the instantiation parameters of the container cluster node, creating the CISM instance and / or the CIS instance on the container cluster node by the CCM; or creating, by the CCM, the CISM instance on the container cluster node; and further creating, by the CISM instance, the CIS instance on the container cluster node. The management system of claim 18 , comprising:

21. A management device for a container cluster, comprising modules adapted to implement the steps of the method according to any one of claims 1 to 11.

22. 12. A management device for a container cluster comprising a processor and a memory, the processor coupled to the memory, the memory storing a computer program, and the processor configured to invoke the computer program in the memory to enable the management device to implement the method of any one of claims 1 to 11.

23. A computer readable storage medium, the storage medium storing a computer program, the computer program implementing the method of any one of claims 1 to 11 when executed.

Images