Methods, apparatus and computer-readable media related to container-as-a-service computing systems

Abstract

A computer-implemented method is described, performed by an operator entity in a container-as-a-Service (CaaS) computing system. The CaaS computing system comprises a server storing definitions of a plurality of software resources for use in the CaaS computing system. The method comprises: determining that a first custom resource definition (CRD) has been created for installation in the CaaS computing system; and, responsive to a determination that an identity of the first CRD matches an identity of a second CRD stored in the server, checking definitions of the first and second CRDs for compatibility with each other.

Description

[0001] METHODS, APPARATUS AND COMPUTER-READABLE MEDIA RELATED TO CONTAINER-AS-A-SERVICE COMPUTING SYSTEMS

[0002] Technical field

[0003] Embodiments of the disclosure relate to container-as-a-service (CaaS) computing systems, and particularly to methods, apparatus and computer-readable media for managing custom resource definitions used by CaaS computing systems.

[0004] Kubernetes (K8s) is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. The Kubernetes container orchestration platform supports users to extend the built-in resource types with custom resource types; that is, resource types that are not necessarily available in a standard Kubernetes cluster installation.

[0005] A Custom Resource type specification, referred to as a Custom Resource Definition (CRD), can be dynamically installed and deleted in a running Kubernetes cluster, and one can update custom resource definitions independently of the cluster itself (i.e., the Kubernetes control plane). A Custom resource type is created by installing (e.g., POSTing in the Kubernetes API server) a CRD object that defines the properties and data schema for the desired custom resource type.

[0006] Once a given custom resource definition is installed, users can create and manage objects of that resource type, using the normal Kubernetes Command Line Interface (CLI) and / or the corresponding REST operations towards the Kubernetes application programming interface (API) server, just as users do for Kubernetes built-in resources types such as Deployments, StatefulSets, ConfigMaps, Services, etc.

[0007] On their own, custom resource instances enable users to create data objects where structured data can be stored and retrieved. However, an associated custom resource controller is also commonly installed. This is a special privileged workload (often referred to as an Operator) that controls some runtime state and uses the resource data schema as a declarative API for those runtime resources.

[0008] The installation of applications is commonly automated by use of different templating technologies such as Helm charts, Terraform domain-specific language (DSL), etc. A user parameterizes portable orchestration templates for the application, and runs them through a tool that produces Kubernetes native resource manifests. These manifests are then sent to the Kubernetes API server, which will create the requested workloads, and assign to them the required compute, storage and networking resources.

[0009] When using these auto installation tools to deploy a cloud native application that is dependent on one or more custom resource types, it cannot be taken for granted that such custom resource types are already available in the platform. However, it is also desirable not to break the automation by first manually interacting with the cloud admin to ensure that the CRD(s) needed are installed. Instead it is desirable to install the CRD objects needed by the application in the first steps of the automated application installation logic, and so also manage any optionally required CRD update (in case the CRD object already exists and some additions on the custom resource type are required) as part of normal application installations and / or updates.

[0010] This process works acceptably as long as only a single application in a Kubernetes cluster attempts to install and / or update a given CRD object. If several applications to be deployed in the same Kubernetes cluster are dependent on the same custom resource type and more than one of them manages the CRD, there is a risk that an application will update the “shared” CRD object in a way that is non-backwards compatible. Doing so may result in other applications relying on the CRD object experiencing errors.

[0011] Embodiments of the disclosure seek to address these and other problems.

[0012] A first aspect of the disclosure provides a computer-implemented method performed by an operator entity in a container-as-a-Service (CaaS) computing system. The CaaS computing system comprises a server storing definitions of a plurality of software resources for use in the CaaS computing system. The method comprises: determining that a first custom resource definition (CRD) has been created for installation in the CaaS computing system; and, responsive to a determination that an identity of the first CRD matches an identity of a second CRD stored in the server, checking definitions of the first and second CRDs for compatibility with each other. A second aspect of the disclosure provides a non-transitory computer-readable storage medium. A CaaS computing system comprises a server storing definitions of a plurality of software resources for use in the CaaS computing system. The storage medium comprises instructions or code which, when executed by processing circuitry of a computing device implementing an operator entity in the CaaS computing system, cause the computing device to: determine that a first custom resource definition (CRD) has been created for installation in the CaaS computing system; and, responsive to a determination that an identity of the first CRD matches an identity of a second CRD stored in the server, check definitions of the first and second CRDs for compatibility with each other.

[0013] A third aspect of the disclosure provides a computing device implementing an operator entity in a CaaS computing system. The CaaS computing system comprises a server storing definitions of a plurality of software resources for use in the CaaS computing system. The computing device comprises processing circuitry and a non-transitory computer-readable storage medium storing instructions or code which, when executed by the processing circuitry, cause the computing device to: determine that a first custom resource definition (CRD) has been created for installation in the CaaS computing system; and, responsive to a determination that an identity of the first CRD matches an identity of a second CRD stored in the server, check definitions of the first and second CRDs for compatibility with each other.

[0014] Thus embodiments of the disclosure provide methods, computer-readable media and apparatus for the management and installation of CRDs in a CaaS system such as a K8s system. Some embodiments make it possible for multiple applications, in the same K8s cluster, to manage (e.g., install, update, etc) the same CRD, without risk of breaking backward compatibility of the CRD in case a newer version of the CRD is already installed, or a different CRD with same name was already installed. In this way, the management of CRDs that are not platform specific can be made part of each application’s lifecycle management and therefore does not require involvement of the cluster administrator.

[0015] Further, many applications are installed using Helm charts, and embodiments of the disclosure enable an application to ensure that the custom resource types it is dependent on are available. The first Helm chart to be deployed for an application can be configured to include a CRD-LCM resource manifest for a CRD the application is dependent on in accordance with the embodiments described herein.

[0016] Brief description of the drawings

[0017] For a better understanding of embodiments of the disclosure, and to show how they may be put into effect, reference will now be made, by way of example, to the accompanying drawings, in which:

[0018] Figure 1 shows a computing system according to embodiments of the disclosure;

[0019] Figure 2 shows a method according to embodiments of the disclosure; and Figure 3 shows an apparatus according to embodiments of the disclosure.

[0020] Detailed description

[0021] Some of the embodiments contemplated herein will now be described more fully with reference to the accompanying drawings. Other embodiments, however, are contained within the scope of the subject-matter disclosed herein, the disclosed subject-matter should not be construed as limited to only the embodiments set forth herein; rather, these embodiments are provided by way of example to convey the scope of the subject-matter to those skilled in the art.

[0022] As mentioned earlier, there are described herein advantageous techniques for managing the installation of and / or updates to custom resource definitions with a container-as-a-service (CaaS) computing system such as Kubernetes (K8s). Figure 1 shows a computing system 100 according to embodiments of the disclosure. The computing system 100 may correspond to a Kubernetes system, and in particular may be implemented within a Kubernetes cluster, e.g., a group of one or more network nodes or entities within which a common set of applications are installed and updated. The network nodes within the Kubernetes cluster may comprise a master node and one or more worker nodes.

[0023] A K8s cluster of worker nodes is a set of worker nodes that run containerised applications. Although K8s is agnostic to the underlying network infrastructure used to connect devices, in certain embodiments of the disclosure the worker nodes are one or more of: computing devices; network nodes (e.g., for a radio network, such as radio access network nodes or core network nodes); and wireless devices. Further, although a K8s cluster of worker nodes has been provided as one example of the type of cluster of worker nodes, it will be understood that any other type of cluster of worker nodes may be used (such as OpenStack, Docker Swarm, etc.). Specifically, the infrastructure described herein is intended to operate irrespective of the orchestration system.

[0024] As noted above the worker nodes referred to herein can comprise one or more devices, such as one or more wireless devices. The one or more devices can, for example, comprise one or more robots, surveillance equipment, small control devices, and / or any other wireless devices. The location of the worker nodes referred to herein can be distributed according to some embodiments, such as across a building or premises (e.g. an industrial building or premises).

[0025] The worker nodes referred to herein can be responsible for providing a computational resource in a network. The network referred to herein can be any type of network (e.g. a telecommunications network). For example, the network referred to herein can be a cellular or mobile network, such as a fourth generation (4G) mobile network, a fifth generation (5G) mobile network, a sixth generation (6G) mobile network, or any other generation mobile network. In some embodiments, the network referred to herein can be a radio access network (RAN), or any other type of network. In some embodiments, the network referred to herein can be a local network, such as a local area network (LAN). In some embodiments, the network referred to herein can be an edge cloud infrastructure. In some embodiments, the network referred to herein can be a virtual network or an at least partially virtual network.

[0026] The system 100 comprises a server 102 (such as a Kubernetes application programming interface (API) server), a database 104, which is in communication with the server 102, an operator entity 106, and an application namespace 108. The application namespace 108 is a virtual environment within which a group of resources in the cluster can be isolated. Within a namespace, the names of resources should be unique, but the same name may be re-used in different namespaces. Although only one namespace is illustrated, the system 100 may comprise multiple namespaces.

[0027] The operator entity 106 may also be referred to herein as a CRD lifecycle manager (LCM), in view of its functionality to install and manage the updates to custom resource definitions within the system 100 and particularly within the namespace 108, i.e., to manage the lifecycle of custom resources definitions. As noted below, the operator entity 106 may be defined using software, hardware or a combination thereof. The operator entity 106 may be installed within the database 104, and may thus act on all namespaces 108 within the cluster.

[0028] In one embodiment, the operator entity 106 is itself installed within the system as a custom resource definition (CRD) resource type. In such an embodiment, the operator entity 106 may be installed, and when needed upgraded, as part of the cluster infrastructure. The operator entity 106 may be installed using a plain Kubernetes CustomResourceDefinition manifest. An associated controller (not illustrated) for the resource type may also be installed and assigned administrator privileges.

[0029] In one embodiment, the CRD for the CRD-LCM Resource type may be defined as follows: apiVersion: apiextensions.k8s.io / v1 kind: CustomResourceDefinition metadata: name: crdlifecyclemanagers.company.com spec: group: company.com scope: Namespaced names: plural: crdlifecyclemanagers singular: crdlifecyclemanager kind: CRDLifecycleManager shortNames:

[0030] - crdlcm versions:

[0031] - name: v1 served: true storage: true schema: openAPIV3Schema: type: object properties: spec: type: object properties: crdManifest: type: string required:

[0032] - crdManifest status: type: object properties: manifestApplied: type: boolean errorMessage: type: string required:

[0033] - manifestApplied subresources: status: {}

[0034] Operation of the system 100 will be described with respect to Figure 2, which is a flowchart of a method according to embodiments of the disclosure. The method may be performed by the operator entity 106 described above with respect to Figure 1.

[0035] In step 200, the operator entity determines that a first custom resource definition (CRD) has been created for installation in the system 100 (where “installation” in this context potentially includes upgrades to existing CRDs). The first CRD may be implemented within the namespace 108 (i.e., one or more particular namespaces), or may be implemented at the level of the cluster of worker nodes.

[0036] A CRD is typically installed as part of the installation of an application or a cloud-native network function (CNF) container or package. The CNF package (not illustrated) comprises one or more sub-packages (e.g., Helm charts) that contain or define the resources necessary to deploy and / or implement an application within the cluster of worker nodes. An installation tool 114 receives the CNF package, and forwards the sub-packages to the server 102 for installation in the cluster or the namespace 108.

[0037] According to embodiments of the disclosure, CRDs included within the sub-packages are defined using a resource type that is configured to be managed by the operator entity 106 (referred to herein as a CRD-LCM resource type). That is, the sub-packages have a resource type which is monitored by the operator entity 106, and which encapsulate the CRD for installation in the namespace 108. Thus the operator entity 106 subscribes to the server 102 to receive notifications once a CRD having the particular resource type is installed or updated within the cluster.

[0038] The server 102 uses the sub-packages received from the installation tool 114 to create one or more data objects 110 within a namespace 108. The data objects 110 may have or be given a CRD-LCM resource type (e.g., by the server 102), and encapsulate a manifest 112 defining a CRD to be installed within the namespace 108.

[0039] Step 200 may therefore comprise the operator entity receiving a notification from the server 102 that a first CRD, having a resource type that is managed by the operator entity, has been received for installation in the cluster or the namespace 108. The operator entity may watch all namespaces 108 in the cluster it is running on, looking for custom resource object instances being created, updated or deleted.

[0040] In step 202, the operator entity checks whether the first CRD has a name that is already present in the server 102, i.e., whether another CRD, referred to herein as a second CRD, having the same name as the first CRD, has already been installed. For example, where the first CRD is defined by a manifest, the name may be indicated in the crdManifest field of the CRD-LCM resource. The operator entity 106 thus compares the contents of the crdManifest field of the CRD-LCM resource with the corresponding fields of those CRDs already installed in the server 102.

[0041] If the name does not exist (has not previously been installed), the method proceeds to step 204 and the first CRD is installed on the server 102. That is, the contents of the CRD-LCM resource are decapsulated, and the manifest 112 used to define and install a custom resource definition in the server 102 and one or more namespaces 108.

[0042] If the name does exist on the server 102 (i.e., there is a second CRD having the same name as the first CRD already installed), installing the first CRD in addition to the second CRD or replacing the second CRD with the first CRD may cause errors in the running of other applications in the cluster. The method therefore proceeds to step 206, in which the operator entity determines whether the first CRD and the second CRD are compatible with each other. For example, the first CRD may be deemed compatible with the second CRD responsive to one or more of:

[0043] - the functionality of the first CRD comprising the functionality of the second CRD and additional functionality. In this way, those applications relying on the second CRD will not be adversely affected by replacing the second CRD with the first CRD, as the existing functionality of the second CRD is replicated within the first CRD. Any application relying on the first CRD is able to use the additional functionality defined in the first CRD.

[0044] - The first CRD operating on a first set of data types, and the second CRD operating on a second set of data types which is a subset of the first set of data types. Similarly, in this way, those applications relying on the second CRD will not be adversely affected by replacing the second CRD with the first CRD, as the second set of data types of the second CRD is operated on by the first CRD. Any application relying on the first CRD is able to operate on any additional data types contained within the first set.

[0045] - The first CRD comprising a first set of attributes and the second CRD comprising a second set of attributes which is a subset of the first set of attributes. Similarly, in this way, those applications relying on the second CRD will not be adversely affected by replacing the second CRD with the first CRD, as the second set of attributes is included within the first set. Any application relying on the first CRD is able to rely on any additional attributes contained within the first set of attributes.

[0046] In other words, the second CRD may be determined to be compatible with the first CRD when the first CRD contains all of the functionality, attributes and / or operates on all the data types that the second CRD does (and optionally provides additional functionality, has additional attributes and / or operates on additional data types to the second CRD).

[0047] If the first and second CRDs are determined to be compatible with each other, the method proceeds to step 208, in which the operator entity attempts to patch the second CRD so as to achieve the functionality, attributes or data types defined in the first CRD (e.g., as specified in the CRD-LCM resource instance 110). For example, the second CRD may be patched so as to be identical to the first CRD. The second CRD may be patched by applying one of a plurality of predetermined patching procedures to the second CRD to produce the patched second CRD, such as one or more of: adding an attribute specified in the definition of the second CRD to the definition of the first CRD; adding an attribute specified in the definition of the first CRD to the definition of the second CRD; and removing an attribute, which is absent from the definition of the second CRD, from the definition of the first CRD. Additionally or alternatively, the plurality of predetermined patching procedures may comprise one or more of: adding a function specified in the definition of the second CRD to the definition of the first CRD; adding a function specified in the definition of the first CRD to the definition of the second CRD; and removing a function, which is absent from the definition of the second CRD, from the definition of the first CRD. Additionally or alternatively, the plurality of predetermined patching procedures may comprise one or more of: adding a data type on which the CRD operates specified in the definition of the second CRD to the definition of the first CRD; adding a data type specified in the definition of the first CRD to the definition of the second CRD; and removing a data type, which is absent from the definition of the second CRD, from the definition of the first CRD.

[0048] For example, in one embodiment, the operator entity may run through each possible patching procedure, adding or removing one or more attributes, data types and / or functionalities. In other embodiments, the operator entity may first determine the differences between the first and second CRDs, and only apply those patching procedures which might be expected to remedy the differences. For example, the operator entity may only apply a patching procedure that adds an attribute to the second CRD where the only difference between the first and second CRDs is that the first CRD comprises an additional attribute that the second CRD does not. Those skilled in the art will appreciate how such a process could be extended to patch other differences.

[0049] In step 210, the operator entity determines whether the patching procedure(s) were successful. For example, the operator entity may determine whether the first CRD and the patched second CRD are identical to each other, or whether they provide the same functionality, or whether they have the same attributes, or whether they operate on the same set of data types. If so, the patch can be deemed successful, and the process moves to step 212 in which the patched second CRD is installed on the server 102. The patched second CRD may replace the second CRD previously installed on the server 102, or it may be stored as a second version of the second CRD (i.e., having the same name but a different version number).

[0050] If the patch is not successful, or not possible (i.e. updates to the second CRD to fulfill the application expectations on the custom resource type will result in a non-backward compatible custom resource type) then the process moves to step 214 and the first CRD is discarded by the operator entity without being installed. The operator entity may additionally record an error in an installation file, e.g., the “status” stanza of the CRD-LCM resource object.

[0051] When a Helm chart is used to install a CRD-LCM resource, then the failure to update a CRD (e.g., due to non-backwards compatibility as described above) should be propagated to Helm level, so that the function invoking the Helm upgrade command will get to know about the failure. This can, for example, be done by the installation tool 114 (e.g., the Helm chart), in addition to having a CRD-LCM resource manifest, also including a post-upgrade Helm hook job that waits for the status stanza of the CRDLifeCycleManagement object to be updated. Thus, if the status is updated to “ok” or some other positive indication, the installation job has finished successfully. If the status is updated to “not ok” or some other negative indication, the installation job fails, which results in the Helm upgrade command completing with a failure indication.

[0052] In some embodiments of the disclosure, it may be apparent from step 206 that the first and second CRDs are not compatible with each other and cannot be patched to be compatible with each other. For example, the second CRD may operate on data types which are not included in the first CRD or which conflict with the data types that are included in the first CRD (or vice versa). Additionally or alternatively, the second CRD may provide a functionality that is not included in the first CRD or which conflicts with the functionality that is included in the first CRD (or vice versa). Additionally or alternatively, the second CRD may have one or more attributes that are not included in the first CRD or which conflict with the attributes that are included in the first CRD (or vice versa).

[0053] In such a case, the method may proceed directly to step 214 from step 206, without attempting any patching procedures. It will be understood that at least some or all of the method steps described herein can be automated in some embodiments. That is, in some embodiments, at least some or all of the method steps described herein can be performed automatically. The method described herein can be a computer-implemented method.

[0054] The techniques described herein include advantageous techniques for managing the installation of custom resource definitions within a cluster of worker nodes or other CaaS computing system. In particular, the advantageous techniques enable multiple applications, in the same cluster, to manage (e.g., install, update, etc) the same CRD, without risk of breaking backward compatibility of the CRD in case a newer version of the CRD is already installed, or a different CRD with same name was already installed. In this way, the management of CRDs that are not platform specific can be made part of each applications lifecycle management and do not require involvement of the cluster administrator.

[0055] Figure 3 illustrates a schematic block diagram of an apparatus 300 for implementing an operator entity in a CaaS computing system (such as a system implementing Kubernetes). The apparatus 300 may comprise a computing device. The CaaS computing system comprises a server storing definitions of a plurality of software resources for use in the CaaS computing system.

[0056] The apparatus 300 may implement an operator entity as discussed above with relation to Figure 1.

[0057] Apparatus 300 is operable to carry out the example method described with reference to Figure 2 and possibly any other processes or methods disclosed herein. It is also to be understood that the method of Figure 2 is not necessarily carried out solely by apparatus 300. At least some operations of the method can be performed by one or more other entities.

[0058] The apparatus 300 comprises processing circuitry 302 (such as one or more processors, digital signal processors, general purpose processing units, etc), a machine-readable medium 304 (e.g., memory such as read-only memory (ROM), random-access memory, cache memory, flash memory devices, optical storage devices, etc) and one or more interfaces 306. In one embodiment, the machine-readable medium 304 contains (e.g. stores) instructions which are executable by the processing circuitry 302 such that the apparatus determines that a first CRD has been created for installation in the CaaS computing system; and, responsive to a determination that an identity of the first CRD matches an identity of a second CRD stored in the server, check definitions of the first and second CRDs for compatibility with each other.

[0059] Additionally or alternatively, the machine-readable medium 304 may further contain instructions which are executable by the processing circuitry such that the apparatus 300 is operable to, responsive to a determination that the definitions of the first and second CRDs are compatible with each other, patch the second CRDs to produce a patched second CRD. The patched second CRD provides the same functionality as the first CRD. For example, the patched second CRD may be stored as a second version of the second CRD in the server. In a further example, the patched second CRD may be identical to the first CRD.

[0060] The apparatus 300 may be operable to patch the second CRD by applying one of a plurality of predetermined patching procedures to the second CRD to produce the patched second CRD, such as one or more of: adding an attribute specified in the definition of the second CRD to the definition of the first CRD; adding an attribute specified in the definition of the first CRD to the definition of the second CRD; and removing an attribute, which is absent from the definition of the second CRD, from the definition of the first CRD.

[0061] Additionally or alternatively, the machine-readable medium 304 may further contain instructions which are executable by the processing circuitry such that the apparatus 300 is operable to, responsive to a determination that an identity of the first CRD is not present in the plurality of software resources for which definitions are stored in the server, add the first CRD to the server.

[0062] Additionally or alternatively, the machine-readable medium 304 may further contain instructions which are executable by the processing circuitry such that the apparatus 300 is operable to determine that a first CRD has been created by monitoring the CaaS computing system for data objects being created or updated and having a resource type managed by the operator entity. For example, data objects having the resource type managed by the operator entity may encapsulate the CRD for installation in the CaaS computing system. In a further example, the data objects having the resource type managed by the operator entity may further comprise a manifest describing the CRD to be installed in the CaaS computing system.

[0063] Additionally or alternatively, the first CRD may be deemed compatible with the second CRD responsive to one or more of: the functionality of the first CRD comprising the functionality of the second CRD and additional functionality; the first CRD operating on a first set of data types, and the second CRD operating on a second set of data types which is a subset of the first set of data types; and the first CRD comprising a first set of attributes and the second CRD comprising a second set of attributes which is a subset of the first set of attributes.

[0064] Thus, the machine-readable medium may store instructions which, when executed by the processing circuitry 302, cause the apparatus 300 to perform the steps described above.

[0065] In other embodiments, the processing circuitry 302 may be configured to directly perform the method, or to cause the apparatus 300 to perform the method, without executing instructions stored in the non-transitory machine-readable medium 304, e.g., through suitably configured dedicated circuitry.

[0066] The one or more interfaces 306 may comprise hardware and / or software suitable for communicating with other nodes of the communication network using any suitable communication medium. For example, the interfaces 306 may comprise one or more wired interfaces, using optical or electrical transmission media. Such interfaces may therefore utilize optical or electrical transmitters and receivers, as well as the necessary software to encode and decode signals transmitted via the interface. In a further example, the interfaces 306 may comprise one or more wireless interfaces. Such interfaces may therefore utilize one or more antennas, baseband circuitry, etc. The components are illustrated coupled together in series; however, those skilled in the art will appreciate that the components may be coupled together in any suitable manner (e.g., via a system bus or suchlike).

[0067] In further embodiments of the disclosure, the apparatus 300 may comprise power circuitry (not illustrated). The power circuitry may comprise, or be coupled to, power management circuitry and is configured to supply the components of apparatus 300 with power for performing the functionality described herein. Power circuitry may receive power from a power source. The power source and / or power circuitry may be configured to provide power to the various components of apparatus 300 in a form suitable for the respective components (e.g., at a voltage and current level needed for each respective component). The power source may either be included in, or external to, the power circuitry and / or the apparatus 300. For example, the apparatus 300 may be connectable to an external power source (e.g., an electricity outlet) via an input circuitry or interface such as an electrical cable, whereby the external power source supplies power to the power circuitry. As a further example, the power source may comprise a source of power in the form of a battery or battery pack which is connected to, or integrated in, the power circuitry. The battery may provide backup power should the external power source fail. Other types of power sources, such as photovoltaic devices, may also be used.

[0068] There is also provided a computer program comprising instructions which, when executed by processing circuitry (such as the processing circuitry 302 of the apparatus 300 described herein), cause the processing circuitry or the apparatus within which it is comprised to perform at least part of the method described herein. There is provided a computer program product, embodied on a non-transitory machine-readable medium, comprising instructions which are executable by processing circuitry (such as the processing circuitry 302) to cause the processing circuitry or the apparatus within which it is comprised to perform at least part of the method described herein. There is provided a computer program product comprising a carrier containing instructions for causing processing circuitry (such as the processing circuitry 302) to perform at least part of the method described herein. In some embodiments, the carrier can be any one of an electronic signal, an optical signal, an electromagnetic signal, an electrical signal, a radio signal, a microwave signal, or a computer-readable storage medium.

[0069] In some embodiments, the operator entity described herein can be performed by hardware. Thus, in some embodiments, the operator entity 106 and / or the apparatus 300 described herein can be hardware. However, it will also be understood that optionally at least part or all of the entity functionality described herein can be virtualized. For example, the functions performed by the operator entity 106 described herein can be implemented in software running on generic hardware that is configured to orchestrate them. Thus, in some embodiments, the operator entity 106 described herein can be virtual. In some embodiments, at least part or all of the entity functionality, node functionality, and / or device functionality described herein may be performed in a network enabled cloud. Thus, the method described herein can be realised as a cloud implementation according to some embodiments. The entity functionality described herein may all be at the same location or at least some of the functionality may be distributed.

[0070] It should be noted that the above-mentioned examples illustrate rather than limit embodiments of the disclosure, and that those skilled in the art will be able to design many alternative examples without departing from the scope of the appended claims. The word “comprising” does not exclude the presence of elements or steps other than those listed in a claim, “a” or “an” does not exclude a plurality, and a single processor or other unit may fulfil the functions of several units recited in the statements below. Where the terms, “first”, “second” etc. are used they are to be understood merely as labels for the convenient identification of a particular feature. In particular, they are not to be interpreted as describing the first or the second feature of a plurality of such features (i.e. the first or second of such features to occur in time or space) unless explicitly stated otherwise. Steps in the methods disclosed herein may be carried out in any order unless expressly otherwise stated. Any reference signs in the statements shall not be construed so as to limit their scope.

Claims

CLAIMS1.A computer-implemented method performed by an operator entity in a container- as-a-Service, CaaS, computing system (100), the CaaS computing system comprising a server (102) storing definitions of a plurality of software resources for use in the CaaS computing system, the method comprising: determining (200) that a first custom resource definition, CRD, has been created for installation in the CaaS computing system; and responsive to a determination (202) that an identity of the first CRD matches an identity of a second CRD stored in the server, checking (206) definitions of the first and second CRDs for compatibility with each other.

2. The method according to claim 1 , further comprising: responsive to a determination that the definitions of the first and second CRDs are compatible with each other, patching (208) the second CRD to produce a patched second CRD, wherein the patched second CRD provides the same functionality as the first CRD.

3. The method according to claim 2, further comprising: storing (212) the patched second CRD as a second version of the second CRD in the server.

4. The method according to claim 2 or 3, wherein the patched second CRD is identical to the first CRD.

5. The method according to any one of claims 2 to 4, wherein patching (208) the second CRD comprises applying one of a plurality of predetermined patching procedures to the second CRD to produce the patched second CRD.

6. The method according to claim 5, wherein the plurality of predetermined patching procedures comprise one or more of: adding an attribute specified in the definition of the second CRD to the definition of the first CRD; adding an attribute specified in the definition of the first CRD to the definition of the second CRD; and removing an attribute, which is absent from the definition of the second CRD, from the definition of the first CRD.

7. The method according to any one of the preceding claims, further comprising: responsive to a determination that the first and second CRDs are not compatible, discarding (214) the first CRD and recording an error in a status file.

8. The method according to claim 7, wherein the first CRD is discarded and an error recorded in a status file further responsive to a determination that it is not possible to patch the second CRD for compatibility with the first CRD.

9. The method according to any one of the preceding claims, further comprising: responsive to a determination that an identity of the first CRD is not present in the plurality of software resources for which definitions are stored in the server, adding (204) the first CRD to the server.

10. The method according to any one of the preceding claims, wherein the step of determining (200) that a first CRD has been created comprises monitoring the CaaS computing system for data objects being created or updated and having a resource type managed by the operator entity.

11. The method according to claim 10, wherein data objects having the resource type managed by the operator entity encapsulate the CRD for installation in the CaaS computing system.

12. The method according to claim 10 or 11 , wherein data objects having the resource type managed by the operator entity further comprise a manifest describing the CRD to be installed in the CaaS computing system.

13. The method according to any one of the preceding claims, wherein the first CRD is compatible with the second CRD where the functionality of the first CRD comprises the functionality of the second CRD and additional functionality.

14. The method according to any one of the preceding claims, wherein the first CRD is compatible with the second CRD where the first CRD operates on a first set of data types, and the second CRD operates on a second set of data types which is a subset of the first set of data types.

15. The method according to any one of the preceding claims, wherein the first CRD is compatible with the second CRD where the first CRD comprises a first set of attributes and the second CRD comprises a second set of attributes which is a subset of the first set of attributes.

16. The method according to any one of the preceding claims, wherein the CaaS computing system comprises a Kubernetes system.

17. A non-transitory computer-readable storage medium (304) storing code which, when executed by processing circuitry (302) of a computing device implementing an operator entity (106) in a container-as-a-Service, CaaS, computing system, the CaaS computing system comprising a server storing definitions of a plurality of software resources for use in the CaaS computing system, causes the computing device to: determine (200) that a first custom resource definition, CRD, has been created for installation in the CaaS computing system; and responsive to a determination that an identity of the first CRD matches an identity of a second CRD stored in the server, check (206) definitions of the first and second CRDs for compatibility with each other.

18. The non-transitory computer-readable storage medium according to claim 17, wherein the computing device is further caused to: responsive to a determination that the definitions of the first and second CRDs are compatible with each other, patch (208) the second CRDs to produce a patched second CRD, wherein the patched second CRD provides the same functionality as the first CRD.

19. The non-transitory computer-readable storage medium according to claim 18, wherein the computing device is further caused to: store (212) the patched second CRD as a second version of the second CRD in the server (102).

20. The non-transitory computer-readable storage medium according to claim 18 or 19, wherein the patched second CRD is identical to the first CRD.

21. The non-transitory computer-readable storage medium according to any one of claims 18 to 20, wherein the computing device is caused to patch the secondCRD by applying one of a plurality of predetermined patching procedures to the second CRD to produce the patched second CRD.

22. The non-transitory computer-readable storage medium according to claim 21, wherein the plurality of predetermined patching procedures comprise one or more of: adding an attribute specified in the definition of the second CRD to the definition of the first CRD; adding an attribute specified in the definition of the first CRD to the definition of the second CRD; and removing an attribute, which is absent from the definition of the second CRD, from the definition of the first CRD.

23. The non-transitory computer-readable storage medium according to any one of claims 17 to 22, wherein the computing device is further caused to: responsive to a determination that the first and second CRDs are not compatible, discard (214) the first CRD and record an error in a status file.

24. The non-transitory computer-readable storage medium according to claim 23, wherein the first CRD is discarded and an error recorded in a status file further responsive to a determination that it is not possible to patch the second CRD for compatibility with the first CRD.

25. The non-transitory computer-readable storage medium according to any one of claims 17 to 24, wherein the computing device is further caused to: responsive to a determination that an identity of the first CRD is not present in the plurality of software resources for which definitions are stored in the server, add (204) the first CRD to the server.

26. The non-transitory computer-readable storage medium according to any one of claims 17 to 25, wherein the computing device is caused to determine that a first CRD has been created by monitoring the CaaS computing system for data objects being created or updated and having a resource type managed by the operator entity.

27. The non-transitory computer-readable storage medium according to claim 26, wherein data objects having the resource type managed by the operator entity encapsulate the CRD for installation in the CaaS computing system.

28. The non-transitory computer-readable storage medium according to claim 26 or 27, wherein data objects having the resource type managed by the operator entity further comprise a manifest describing the CRD to be installed in the CaaS computing system.

29. The non-transitory computer-readable storage medium according to any one of claims 17 to 28, wherein the first CRD is compatible with the second CRD where the functionality of the first CRD comprises the functionality of the second CRD and additional functionality.

30. The non-transitory computer-readable storage medium according to any one of claims 17 to 29, wherein the first CRD is compatible with the second CRD where the first CRD operates on a first set of data types, and the second CRD operates on a second set of data types which is a subset of the first set of data types.

31. The non-transitory computer-readable storage medium according to any one of claims 17 to 30, wherein the first CRD is compatible with the second CRD where the first CRD comprises a first set of attributes and the second CRD comprises a second set of attributes which is a subset of the first set of attributes.

32. The non-transitory computer-readable storage medium according to any one of claims 17 to 31, wherein the CaaS computing system comprises a Kubernetes system.

33. A computing device (300) implementing an operator entity (106) in a container- as-a-Service, CaaS, computing system, the CaaS computing system comprising a server storing definitions of a plurality of software resources for use in the CaaS computing system, the computing device comprising processing circuitry (302) and a non-transitory computer-readable storage medium (304) storing code which, when executed by the processing circuitry, causes the computing device to: determine (200) that a first custom resource definition, CRD, has been created for installation in the CaaS computing system; andresponsive to a determination that an identity of the first CRD matches an identity of a second CRD stored in the server, check (206) definitions of the first and second CRDs for compatibility with each other.

Images