Dynamic multi-cluster management

Abstract

A method of dynamic multi-cluster management may include sending a first request to a management service producer to create a set of clusters and train a machine learning model for one or more clusters in the set of clusters for a plurality of contexts. The method may also include sending a second request to the management service producer to dynamically monitor the set of clusters. The method may also include receiving, after at least one of the first request or the second request, a cluster report from the management service producer as a result of the creation of the set of clusters. The method may also include receiving, after at least one of the first request or the second request, a training report from the management service producer as a result of the dynamic monitoring of the cluster set and the training of the machine learning model.

Description

Technical Field

[0001] Some example embodiments may generally relate to mobile or wireless telecommunication systems, such as Long Term Evolution (LTE) or fifth generation (5G) New Radio (NR) access technology or technologies above 5G, or may relate to other communication systems. For example, certain example embodiments may relate to apparatus, systems, and / or methods for dynamic multi-cluster management. Background Art

[0002] Examples of mobile or wireless telecommunication systems may include Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access Network (UTRAN), LTE Evolved UTRAN (E-UTRAN), Advanced LTE (LTE-A), MulteFire, LTE-A Pro, fifth generation (5G) radio access technology or NR access technology, and / or Advanced 5G. 5G wireless systems refer to next generation (NG) radio systems and network architectures. 5G network technology is primarily based on NR technology, but 5G (or NG) networks can also be built on E-UTRAN radios. It is estimated that NR will provide bit rates of about 10-20Gbit / s or higher and will support at least enhanced mobile broadband (eMBB) and ultra-reliable low latency communications (URLLC) and massive machine type communications (mMTC). NR is expected to provide extreme broadband and ultra-robust low latency connections and large-scale networks to support IoT. Summary of the invention

[0003] Some example embodiments may relate to a method. The method may include sending a first request to a management service producer to create a cluster set and train a machine learning model for one or more clusters in the cluster set for multiple contexts. According to certain example embodiments, each cluster may include multiple network nodes, network functions, and management functions. The method may also include sending a second request to the management service producer to dynamically monitor the cluster set. The method may also include receiving a cluster report from the management service producer as a result of the creation of the cluster set after at least one of the first request or the second request. In addition, the method may include receiving a training report from the management service producer as a result of the dynamic monitoring of the cluster set and the training of the machine learning model after at least one of the first request or the second request.

[0004] Other example embodiments may relate to an apparatus. The apparatus may include at least one processor and at least one memory including computer program code. The at least one memory and the computer program code may be configured, together with the at least one processor, so that the apparatus at least: sends a first request to a management service producer to create a cluster set and train a machine learning model for one or more clusters in the cluster set for multiple contexts. According to certain example embodiments, each cluster may include multiple network nodes, network functions, and management functions. The apparatus may also be caused to send a second request to the management service producer to dynamically monitor the cluster set. The apparatus may also be caused to receive a cluster report from the management service producer as a result of the creation of the cluster set after at least one of the first request or the second request. In addition, the apparatus may also be caused to receive a training report from the management service producer as a result of the dynamic monitoring of the cluster set and the training of the machine learning model after at least one of the first request or the second request.

[0005] Other example embodiments may relate to an apparatus. The apparatus may include a component for sending a first request to a management service producer to create a cluster set and train a machine learning model for one or more clusters in the cluster set for multiple contexts. According to certain example embodiments, each cluster may include multiple network nodes, network functions, and management functions. The apparatus may also include a component for sending a second request to the management service producer to dynamically monitor the cluster set. The apparatus may also include a component for receiving a cluster report from the management service producer as a result of the creation of the cluster set after at least one of the first request or the second request. In addition, the apparatus may include a component for receiving a training report from the management service producer as a result of the dynamic monitoring of the cluster set and the training of the machine learning model after at least one of the first request or the second request.

[0006] According to other example embodiments, a non-transitory computer-readable medium may be encoded with instructions that, when executed in hardware, may perform a method. The method may include sending a first request to a management service producer to create a cluster set and train a machine learning model for one or more clusters in the cluster set for multiple contexts. According to certain example embodiments, each cluster may include multiple network nodes, network functions, and management functions. The method may also include sending a second request to the management service producer to dynamically monitor the cluster set. The method may also include receiving a cluster report from the management service producer as a result of the creation of the cluster set after at least one of the first request or the second request. In addition, the method may include receiving a training report from the management service producer as a result of the dynamic monitoring of the cluster set and the training of the machine learning model after at least one of the first request or the second request.

[0007] Other example embodiments may relate to a computer program product for executing a method. The method may include sending a first request to a management service producer to create a cluster set and train a machine learning model for one or more clusters in the cluster set for multiple contexts. According to certain example embodiments, each cluster may include multiple network nodes, network functions, and management functions. The method may also include sending a second request to the management service producer to dynamically monitor the cluster set. The method may also include receiving a cluster report from the management service producer as a result of the creation of the cluster set after at least one of the first request or the second request. In addition, the method may include receiving a training report from the management service producer after at least one of the first request or the second request as a result of the dynamic monitoring of the cluster set and the training of the machine learning model.

[0008] Other example embodiments may be directed to a device that may include a circuit system configured to send a first request to a management service producer to create a cluster set and train a machine learning model for one or more clusters in the cluster set for multiple contexts. According to certain example embodiments, each cluster may include multiple network nodes, network functions, and management functions. The device may also include a circuit system configured to send a second request to the management service producer to dynamically monitor the cluster set. The device may also include a circuit system configured to receive a cluster report from the management service producer as a result of the creation of the cluster set after at least one of the first request or the second request. In addition, the device may include a circuit system configured to receive a training report from the management service producer as a result of the dynamic monitoring of the cluster set and the training of the machine learning model after at least one of the first request or the second request.

[0009] Another example embodiment may relate to a method. The method may include receiving a first request from a management service consumer to create a cluster set and train a machine learning model for one or more clusters in the cluster set for multiple different contexts. According to certain example embodiments, each cluster may include multiple network nodes, network functions, and management functions. The method may also include receiving a second request from the management service consumer to dynamically monitor the cluster set. The method may also include creating a cluster set based on the first request. According to certain example embodiments, each cluster in the cluster set may be associated with a machine learning model. In addition, the method may include training a machine learning model for one or more clusters in the cluster set for multiple different contexts based on the first request. In addition, the method may include performing a dynamic monitoring process based on the second request to monitor the clusters in the cluster set. The method may also include sending a cluster report to the management service consumer based on the creation of the cluster set. The method may also include sending a training report to the management service consumer based on performing dynamic monitoring of the cluster set and training of the machine learning model.

[0010] Other example embodiments may relate to an apparatus. The apparatus may include at least one processor and at least one memory including computer program code. The at least one memory and the computer program code may be configured to, together with the at least one processor, cause the apparatus to at least: receive a first request from a management service consumer to create a cluster set and train a machine learning model for one or more clusters in the cluster set for multiple different contexts. According to certain example embodiments, each cluster may include multiple network nodes, network functions, and management functions. The apparatus may also be caused to receive a second request from a management service consumer to dynamically monitor the cluster set. The apparatus may also be caused to create a cluster set based on the first request. According to certain example embodiments, each cluster in the cluster set may be associated with a machine learning model. In addition, the apparatus may be caused to train a machine learning model for one or more clusters in the cluster set for multiple different contexts based on the first request. In addition, the apparatus may be caused to perform a dynamic monitoring process based on the second request to monitor clusters in the cluster set. The apparatus may also be caused to send a cluster report to the management service consumer based on the creation of the cluster set. The apparatus may also be caused to send a training report to the management service consumer based on the execution of dynamic monitoring of the cluster set and the training of the machine learning model.

[0011] Other example embodiments may relate to a device. The device may include a component for receiving a first request from a management service consumer to create a cluster set and train a machine learning model for one or more clusters in the cluster set for multiple different contexts. According to certain example embodiments, each cluster may include multiple network nodes, network functions, and management functions. The device may also include a component for receiving a second request from a management service consumer to dynamically monitor the cluster set. The device may also include a component for creating a cluster set based on the first request. According to certain example embodiments, each cluster in the cluster set may be associated with a machine learning model. In addition, the device may include a component for training a machine learning model for one or more clusters in the cluster set for multiple different contexts based on the first request. In addition, the device may include a component for performing a dynamic monitoring process to monitor clusters in the cluster set based on the second request. The device may also include a component for sending a cluster report to the management service consumer based on the creation of the cluster set. The device may also include a component for sending a training report to the management service consumer based on performing dynamic monitoring of the cluster set and training of the machine learning model.

[0012] According to other example embodiments, a non-transitory computer-readable medium may be encoded with instructions that, when executed in hardware, may perform a method. The method may include receiving a first request from a management service consumer to create a cluster set and train a machine learning model for one or more clusters in the cluster set for multiple different contexts. According to certain example embodiments, each cluster may include multiple network nodes, network functions, and management functions. The method may also include receiving a second request from a management service consumer to dynamically monitor the cluster set. The method may also include creating a cluster set based on the first request. According to certain example embodiments, each cluster in the cluster set may be associated with a machine learning model. In addition, the method may include training a machine learning model for one or more clusters in the cluster set for multiple different contexts based on the first request. In addition, the method may include performing a dynamic monitoring process based on the second request to monitor clusters in the cluster set. The method may also include sending a cluster report to the management service consumer based on the creation of the cluster set. The method may also include sending a training report to the management service consumer based on performing dynamic monitoring of the cluster set and training of the machine learning model.

[0013] Other example embodiments may relate to a computer program product for executing a method. The method may include receiving a first request from a management service consumer to create a cluster set and train a machine learning model for one or more clusters in the cluster set for multiple different contexts. According to certain example embodiments, each cluster may include multiple network nodes, network functions, and management functions. The method may also include receiving a second request from a management service consumer to dynamically monitor the cluster set. The method may also include creating a cluster set based on the first request. According to certain example embodiments, each cluster in the cluster set may be associated with a machine learning model. In addition, the method may include training a machine learning model for one or more clusters in the cluster set for multiple different contexts based on the first request. In addition, the method may include performing a dynamic monitoring process based on the second request to monitor clusters in the cluster set. The method may also include sending a cluster report to the management service consumer based on the creation of the cluster set. The method may also include sending a training report to the management service consumer based on performing dynamic monitoring of the cluster set and training of the machine learning model.

[0014] Other example embodiments may relate to a device that may include a circuit system configured to receive a first request from a management service consumer to create a cluster set and train a machine learning model for one or more clusters in the cluster set for multiple different contexts. According to certain example embodiments, each cluster may include multiple network nodes, network functions, and management functions. The device may also include a circuit system configured to receive a second request from a management service consumer to dynamically monitor the cluster set. The device may also include a circuit system configured to create a cluster set based on the first request. According to certain example embodiments, each cluster in the cluster set may be associated with a machine learning model. In addition, the device may include a circuit system configured to train a machine learning model for one or more clusters in the cluster set for multiple different contexts based on the first request. In addition, the device may include a circuit system configured to perform a dynamic monitoring process to monitor clusters in the cluster set based on the second request. The device may also include a circuit system configured to send a cluster report to the management service consumer based on the creation of the cluster set. The device may also include a circuit system configured to send a training report to the management service consumer based on performing dynamic monitoring of the cluster set and training of the machine learning model. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] For a proper understanding of the example embodiments, reference should be made to the accompanying drawings, in which:

[0016] Figure 1 illustrates an example signal diagram for a cluster creation request according to certain example embodiments;

[0017] Figure 2 illustrates an example signal diagram of another cluster creation request according to certain example embodiments;

[0018] Figure 3 illustrates an example signal diagram of a dynamic cluster monitoring request according to certain example embodiments;

[0019] Figure 4 An example signal diagram illustrating the process of a machine learning training request according to certain example embodiments;

[0020] Figure 5 illustrates a Unified Modeling Language diagram class according to certain example embodiments;

[0021] Figure 6 illustrates another type of Unified Modeling Language diagram according to certain example embodiments;

[0022] Figure 7 illustrates an example flow chart of a method according to certain example embodiments;

[0023] Figure 8illustrates an example flow chart of another method according to certain example embodiments; and

[0024] Fig. 9 A set of apparatuses according to certain example embodiments is illustrated. DETAILED DESCRIPTION

[0025] It will be readily appreciated that the components of certain example embodiments, as generally described and illustrated herein in the accompanying drawings, may be arranged and designed in a variety of different configurations. The following is a detailed description of some example embodiments of systems, methods, apparatus, and computer program products for dynamic multi-cluster management. In certain example embodiments, dynamic multi-cluster management may involve management of artificial intelligence / machine learning (AIML) training.

[0026] The features, structures, or characteristics of the example embodiments described in this specification may be combined in any suitable manner in one or more example embodiments. For example, the use of the phrases "certain embodiments," "example embodiments," or "some embodiments" throughout this specification refers to the fact that a particular feature, structure, or characteristic described in conjunction with one embodiment may be included in at least one embodiment. Therefore, the appearance of the phrases "in certain embodiments," "example embodiments," "in some embodiments," "in other embodiments," or other similar language throughout this specification does not necessarily refer to the same set of embodiments, and the described features, structures, or characteristics may be combined in any suitable manner in one or more example embodiments. In addition, the terms "base station," "cell," "node," "gNB," "network," or other similar language may be used interchangeably throughout this specification.

[0027] As used herein, “at least one of: ” and “at least one of <a list of two or more elements]” and similar expressions (wherein a list of two or more elements is connected by “and” or “or”) mean at least any one of these elements, or at least any two or more of these elements, or at least all of these elements.

[0028] According to the specifications of the Third Generation Partnership Project (3GPP), communication networks can be built by instances of the same functions and entities deployed in different locations in different situations to achieve common goals. However, if these functions and entities are to be equipped with artificial intelligence (AI) (i.e., AI models), each entity or function may require different training data due to different deployment contexts and environments. 3GPP describes a way to achieve this training through attributes such as MLContext (i.e., AIMLContext). AIMLContext can represent states and conditions related to AIMLEntity. Specifically, it can be one of three types of contexts, including, for example, ExpectedRunTimeContext, TrainingContext, and RunTimeContext. However, over time, the maintenance and retraining of these models are unavailable. It can be assumed that it is based on ad-hoc requests by instances or operators. 3GPP provides methods for requesting training processes and managing (i.e., starting, pausing, and restarting) these processes. There are also some reports on the training process. However, the current ad hoc and tunnel views of training are not enough to train AIML models on a large scale.

[0029] There is currently no monitoring and maintenance method for updating one or several MLEntities (ML entities, i.e., AIML entities). An AIMLEntity can be an AIML model or an AIML-enabled function. AIMLTraining can be requested for an AIML model or an AIML-enabled function. For each AIMLEntity being trained, one or more AIMLTrainingProcesses can be instantiated. An EIMLEntity can include 3 types of contexts: TrainingContext (i.e., the context under which the AIMLEntity is trained), ExpectedRunTimeContext (i.e., the context in which the AIMLEntity is expected to be applied), or RunTimeContext (i.e., the context in which the model is applied).

[0030] Currently, there is no monitoring and maintenance method for updating one or several MLEntities. Although the situations and contexts in which AIML entities can operate are diverse, it is reasonable to assume that there can be similarities in the experiences of a certain entity. Therefore, there may be an opportunity to identify entities or groups of functions that perform the same function in different locations, which are similar enough in terms of operational state to share a trained model for their specific use case.

[0031] Before deploying AIML models in an operational environment for reasoning or prediction, AIML models need to be properly trained. Traditional methods for training these models can include deploying a trained model to each entity / node and then performing a round of retraining to customize the model for that specific entity / node and context. However, this approach faces various challenges, including challenges related to scalability and challenges related to analysis. Scalability-related challenges can include the need for a large amount of computation to train models for each entity / node, and therefore are not energy efficient. In addition, more machine learning entities increase the complexity of maintaining the performance of each model. The management of the model is also far from efficient and ideal. As for analytical challenges, although the maintenance work is more complicated, there is no statistical way to obtain understanding and insight into the impact of network performance, changes, and the environment on network entities.

[0032] Given the various challenges, one of the barriers to adoption and integration of AIML capabilities in 3GPP networks is the management and maintenance of models. This issue will grow in importance and severity as networks are required to use AI on a larger scale.

[0033] In view of the above challenges and shortcomings, certain example embodiments may provide a method for dynamically managing and monitoring the training of AIML models of a network (e.g., a 3GPP network). In doing so, certain example embodiments may use the concept of clustering and introduce management operations (i.e., BreakNMerge, expansion, classic, and reset) to maintain trained clusters through some form of re-clustering. The capabilities of this service may include an authorized clustering management service (MnS) consumer sending a clustering request for a group of nodes (i.e., functions including AIML models, entities, etc.) having different data networks (DNs) / identifiers or identifications (IDs) for a given MLEntityId (ML entity Id), and an information set indicating clustering criteria. This step may or may not include whether a service producer (e.g., an MnS producer) requests to instantiate the various stages of training. If the MnS producer is to instantiate the training request, it may perform clustering operations and determine the node-to-cluster association. The MnS producer may also generate MLEntities with corresponding MLContext and version assignments. MnS Producers can also make requests for training created MLEntities. After the training process is completed, MnS Producers can deactivate already operational MLEntities that were previously provided for the same node.

[0034] According to certain example embodiments, a service producer (e.g., MnS producer) may provide a report of descriptions / details of clusters, node associations, and clustering criteria. According to other example implementations, an authorized consumer may request monitoring of a set of clusters to be maintained and managed (e.g., a set of MLEntities with similar IDs and different versions), which is managed and maintained in the following manner: considering a set of conditions, performing one or several clustering operations (e.g., expansion, decomposition, merging, classification, and reset) based on a self-organizing manner by providing a common re-clustering condition, an event-triggered manner by providing an event handle, and / or by considering a hybrid perception of both conditions. According to certain example embodiments, in this step, the service producer may be responsible for triggering the training process without the need for a training request from the consumer.

[0035] In certain example embodiments, the training process may be triggered based on conditions of the clustering operation provided by the consumer or based on default values. Alternatively, the monitoring process may be triggered based on enabling cluster monitoring capabilities on ad hoc MLEntities. In this example, the service producer may also group the MLEntities based on their IDs and be responsible for monitoring each unique ID. The latter approach allows new versions of MLEntities to be integrated to be inserted into the clustering process and versions that are no longer needed in the cluster to be deleted. In other example embodiments, the clustering MnS producer may provide reports to authorized consumers on any rearrangements or triggering of any clustering operations and provide statistics and descriptions for the triggers of the recalculations.

[0036] According to certain example embodiments, MnS may correspond to a set of provided capabilities for management and orchestration of networks and services. An entity that produces MnS is referred to as a MnS producer, and an entity that consumes MnS is referred to as a MnS consumer. MnS provided by a MnS producer may be consumed by any entity with appropriate authorization and authentication. In some example embodiments, a MnS producer may provide its services via a standardized service interface consisting of individually specified MnS components.

[0037] In certain example embodiments, a dynamic multi-cluster management mechanism may enable cluster training and management of these clusters. Cluster training may refer to a process in which a given AIML entity (identified by its inputs, outputs, and ML models, such as, for example, MLEntityId) may be expected to be trained for a set of nodes / entities (e.g., functions, terminals, etc.), for different contexts and / or different inference locations based on its operational requirements. In other words, in a cluster, it may be possible to perform training and manually replace multiple training requests corresponding to different MLexpectedRuntimeContexts for a given MLmodelMLentity, which are intended for different interface entities.

[0038] According to certain example embodiments, the cluster training criteria may determine the appropriate association of nodes / instances (inference nodes) with clusters formed based on the conditions and requirements in the clustering criteria. Clustering may be based on, for example, a multivariate clustering approach or a univariate approach. The multivariate clustering approach may use input data of the MLEntity, or a set of provided features that are known to have a significant impact on model performance. In a univariate approach, the approach may be based on a unique feature from among the features provided in the model input or request as domain and model design knowledge.

[0039] In certain example embodiments, the clustering method may provide a number of clusters, where each cluster may have a trained model with tuned hyperparameters. In each cluster, the same trained model corresponding to its cluster may be replicated and instantiated on the nodes forming the cluster. Certain example embodiments may define certain properties that can affect the creation and dynamic monitoring of clusters.

[0040] Services associated with dynamic management of clusters for model training may be introduced as an extension of a training request, or may be modeled and provisioned as a separate cluster monitoring request. However, certain example embodiments provide a method for performing dynamic cluster monitoring, in which nodes (e.g., model instances or ML entities) that were not previously part of a cluster (hereinafter referred to as external nodes) may enter dynamic clustering and maintenance. Cluster dynamic monitoring may also provide nodes within a cluster (e.g., model instances or ML entities), hereinafter referred to as internal nodes, to exit the clustering service when attributes of a training request change.

[0041] According to certain example embodiments, in order to enable clustering of MLEntities with similar IDs (i.e., similar IDs may be defined as having at least one of the following: same input, same output, or same architecture) as a service, appropriate information should be provided. For example, certain example embodiments may provide a set of attributes that provide a means for requesting a dynamic clustering request (see Table 1). Attributes for issuing a clustering request may be provided and may include: MLEntityId; ClusteringCriteriaFeatures; ClusteringCriteriaOnly; MinClusterSize; MinExpectedAccuracy; AcceptedAccuracyMargin; MaxNodeOutlierPercentage; and MethodologyId.

[0042] MLEntityId can represent information about input features, output format and ML algorithm category (Convolutional Neural Network (CNN), Artificial Neural Network (ANN), Virtual Space Model (VSM), etc.), and the version of the model, or the MLEntity identified by the ID is associated with all of these items. In some example embodiments, this attribute may be mandatory. ClusteringCriteriaFeatures may be a way to introduce one or more features to be used for clustering, and this attribute may be optional. ClusteringCriteriaOnly may be a Boolean attribute that is used to indicate a clustering method. It may be used to indicate that clustering is performed only based on the features provided in ClusteringCriteriaFeatures. If the ClusteringCriteriaOnly attribute is not specified, the input features of the model will be used as input to the clustering criteria. Therefore, this feature may affect the data source in the clustering method, and the default value may be FALSE.

[0043] MinClusterSize may represent the minimum number of nodes that can form a cluster. Nodes that do not fit into any cluster with appropriate accuracy may be excluded. MinExpectedAccuracy may represent the minimum allowed accuracy value for a cluster. In addition, AcceptedAccuracyMargin may represent the margin that allows flexibility in achieving accuracy on different clusters, and this may be an optional attribute. MaxNodeOutlierPercentage may correspond to the maximum percentage of nodes that are allowed to be outliers during the clustering process. These nodes may be excluded from cluster training if they violate the minimum expected accuracy for the cluster or are not similar enough based on the clustering criteria. These nodes may be assigned a dedicated model that can be trained on its own data and customized to its special case. MethodologyId may represent the ID of the clustering method. This information may be disclosed by the clustering MnS producer after obtaining the supported clustering methods. expectedRuntimeContextSet may be a collection of expectedRuntimeContexts for which the referenced MLEntity will be trained.

[0044] Table 1: Properties used to make clustering requests

[0045]

[0046]

[0047] In other example embodiments, as an alternative to clustering requests, clustering methods can be supported by providing a new attribute called EnableClustering in the training request. MLEntities with the same ID and EnableClustering enabled can be treated as different clusters and can be appropriately grouped and re-clustered based on default values. For dynamic monitoring of clusters created in this way, requests can be issued.

[0048] According to certain example embodiments, after a cluster set is created, it can be monitored to maintain the inference quality and performance of the models in the network. In order to provide the necessary control handles to consumers, certain attributes can be provided in the cluster dynamic monitoring request to perform cluster dynamic monitoring on the cluster set (see Table 2). These attributes may include ClusterDynMonitoringRequestId, MLEntityId, MLEntityVersions, ClusterId, ClusteringOperationConditions, and ReportTimeWindow. ClusterDynMonitoringRequestID may represent the ID of the request for dynamic monitoring of the cluster set. MLEntityId may represent the MLEntityId pointing to the parent ML model with specific inputs, outputs, and architecture. In other words, it points to the parent ML model.

[0049] MLEntityVersions may correspond to a list of versions of parent models generated as a result of a clustering request, or grouped because they indicated a willingness to be part of the cluster. ClusterId may correspond to a list of clusterIds for which the dynamic monitoring request is processing operations. ClusteringOperationConditions may refer to a list of operation objects / OperationIds that provide information needed to manage and control each operation. Additionally, ReportTimeWindow is a time window used to calculate statistics in reports and operations when no time frame is specified. It may also be expressed in terms of number of inferences or number of operations.

[0050] Table 2: Attributes of cluster dynamic monitoring request

[0051]

[0052] According to certain example embodiments, management operations may be used to dynamically manage the created clusters. Management of these clusters may be performed to keep the performance of the model within an acceptable region based on clustering criteria and conditions. Management and control may provide the opportunity to share trained models with several nodes while meeting their performance requirements. In addition, since a model is generalized to several nodes rather than customized for each node, a higher level of management may be required to maintain the performance of the nodes. In order to accommodate the nuances arising from clustering and the resulting level of generalization, certain example embodiments may provide operations including, but not limited to, for example, Classify, Expand, Decompose, BreakNMerge, and Reset.

[0053] In a classification operation, a node may change its cluster if the conditions of the operation are met. In this operation, the MLEntity version (i.e., model details) in each cluster may not change, and therefore no retraining is required. This operation may change nodes within a cluster, which means that the same model may be associated with different nodes. An expansion operation may refer to a scenario in which a cluster may need to be expanded by further training on new observations to accommodate higher accuracy in later reasoning, or to keep up with the accuracy required by the node. The cluster may also be trained based on observations from another set of nodes to accommodate more nodes in one cluster (merge). A decomposition operation may refer to a cluster that can be decomposed into two or more clusters when certain conditions of the operation are met. In this case, clustering may be required to perform retraining or change hyperparameters for the new cluster to meet the clustering criteria. Additional training (e.g., 1 or more times) may be required based on the number of clusters generated by the decomposition operation.

[0054] The BreakNMerge operation can refer to a scenario in which a cluster can first be appropriately broken down into smaller clusters based on the conditions and then merged accordingly to form a better, more consistent, and more efficient set of clusters. This operation can be triggered so that the alternation in classification is reduced (i.e., a classification operation is triggered between two clusters), thereby improving the accuracy of both the managed and merged clusters. In the reset operation, a complete re-clustering can be performed, and this can be similar in process to the initial creation of the clusters.

[0055] According to certain example embodiments, each operation may have certain attributes to enable cluster management at the operation level and provide a level of flexibility in reporting. For example, the management of the cluster may be based on potential conditions that can be assumed based on the attributes of the operation. As an example embodiment, if the maxClusterStabilitylevel of an operation (e.g., decomposition) is set to 70%, then if the value of any cluster is less than this value, the decomposition operation will separate the cluster so that the stability level of the new cluster is greater than this value. In order to create these new clusters, the minimum acceptable accuracy of the client nodes and the clusters under consideration may be considered. Nodes may also be regrouped so that the conditions of the clusters meet the requirements and no other operations are triggered. The triggering of an operation is usually when one or more attributes of the operation are checked and these attributes are violated. The frequency of checking the attributes of the operation may be based on the conditionFullfillmentCheckRate attribute. Additionally, these attributes provide an explanation of how cluster management makes decisions and manages the training of its managed nodes (see Table 3). As shown in Table 3, the operation attributes may include OperationId, type, EventType, MinAccuracy, ConditionFullfillmentCheckRate, MaxClusterStability, and OutliersCount.

[0056] Table 3: Operational attributes

[0057]

[0058]

[0059] As shown in Table 3, OperationId can correspond to an Id pointing to a specific operation running under a specific cluster monitoring request. The Type attribute can correspond to a classification, expansion, decomposition, merger and / or reset operation. EventType can correspond to the type of event or output defined in the analysis, which can be the trigger of the operation in question. In some example embodiments, the EventType attribute can be optional. MinAccuracy can correspond to the minimum acceptable accuracy of the operation to be triggered and cause changes within the cluster. ConditionFullfillmentCheckRate can indicate the frequency at which the operating condition should be checked. It can have a higher frequency for classification operations and can extend operations compared to reset operations. This attribute can be defined both in terms of time and in terms of the number of inferences. If not specified, a default value can be assigned to it. The MaxClusterStabilityLevel attribute can indicate the maximum stability level that can trigger an operation. In some example embodiments, the stability of a cluster can be defined as the ratio of nodes that have never changed their cluster compared to the original size of the cluster. This attribute can also be an indicator driven in the data. In addition, the OutliersCount attribute can indicate the number of outliers generated by performing the operation. This property can be an indicator of improperly set precision or operating conditions, or allocation of incorrect or insufficient features for the operation or clustering criteria. OutliersCount can also indicate that not all nodes requesting to be part of the cluster can benefit from the service, and it is best to exclude them from the service to maintain its performance.

[0060] In certain example embodiments, after a request for dynamic monitoring of a cluster is issued, a cluster monitoring process may be instantiated / executed. The cluster monitoring process may provide a report of any changes in the cluster, as well as multi-cluster management information to provide visibility into the state of the model. These attributes are summarized in Table 4 and may be used to provide reports that are initiated based on triggering any operations and changes in the cluster. For example, these attributes may be used in any manner as long as they do not violate their definitions.

[0061] Table 4: Cluster monitoring process attributes

[0062]

[0063] As shown in Table 4, ClusterDynMonitoringProcessId can correspond to the process ID assigned to a specific ClusterDynMonitoringRequestId. In certain example embodiments, a ClusterDynMonitoringRequest can be associated with only one monitoring process. MinOveralAccuracy can correspond to the minimum observed accuracy value among all clusters in the past ReportTimeWindow. The averageAccuracy attribute can correspond to the average accuracy value over clusters in the past ReportTimeWindow. In addition, the NodeOutlierPercentage attribute can correspond to the percentage of nodes that have been declared as outliers of the clustering process and have exited the process. This may not be reported for a specific time window, but based on the original number of nodes in the cluster and the current participating nodes. ClusterReportRef can refer to the ID of a cluster report that provides detailed internal statistics for each cluster. In addition, the OperationsHistory attribute can refer to the last several operations that have been performed.

[0064] In some example embodiments, a set of attributes may be provided that may reflect the state of the cluster and act as an indicator of cluster information. In some example embodiments, a cluster may have monitoring attributes (see Table 5) that signal the overall performance of the cluster and thus trigger any of the above actions to keep the performance of the cluster within expected ranges.

[0065] Table 5: Attributes in the cluster report

[0066]

[0067]

[0068] As shown in Table 5, ClusterId may correspond to the ID of a cluster pointing to an existing running cluster, and thus corresponds to a unique combination of MLEntityId and version. AvgClusterAccuracy may correspond to the average accuracy of nodes / instances in a cluster with cluster ID ClusterId within the last ReportTimeWindow. MinClusterAccuracy may correspond to the accuracy of the node with the minimum accuracy level in the cluster with ID ClusterId within the last ReportTimeWindow. ClusterAccuracyQuartiles may represent the quartiles of the accuracy of nodes in the cluster with ID ClusterId within the last ReportTimeWindow. ClusterStabilitylevel may represent the cluster stability level as a percentage or ratio. It may be defined as the ratio of nodes that have never changed their cluster (compared to the original cluster) to the original cluster size. This value may be mandatory and may be reset when any operation other than classification is performed on the cluster. Therefore, the stability level of a newly generated cluster (i.e., a new version of MLEntity) is always 1 or 100. In certain example embodiments, classification may affect this value and result in a modification, but not a reset of the value. If the version of the MLEntity has not been modified (thus, no training has been performed), then the value may not be modified.

[0069] As further shown in Table 5, GuestNodes refers to a list of node IDs that have been classified in a cluster and whose model is used. These nodes can be considered transient nodes. The list shows the history of cluster visitors and users within the ReportTimeWindow. It can also be the entire operational life of the model. OriginalNodes refers to the id of the node assigned to the cluster when the cluster is formed. Combining information from GuestNodes, ClusterStabilityLevel, and ClusterAccuracyQuartiles with this property can implement a variety of extensive algorithms and mechanisms to distinguish operations and identify the best fitting operations. In addition, NeighborCluster represents the ID of the cluster (multiple) clusters where most classification operations have fallen in the last ReportTimeWindow, except itself.

[0070] According to certain example embodiments, the ML training MnS producer and the ML clustering MnS producer may be two separate management entities. However, the ML clustering service may be part of the ML training service, and both may be part of the same management service.

[0071] Figure 1 An example signal diagram of a cluster creation request according to certain example embodiments is illustrated. Figure 1 As shown, the signal diagram may involve an ML clustering MnS consumer 100, an ML clustering MnS producer 105, and an ML training MnS producer 110. The ML clustering MnS consumer 100 may be an authorized user of the service and may include, for example, a network function, a management function, an operator, or other functional distinctions. The ML clustering MnS producer 105 may provide management services for clustering, and the ML training MnS producer 110 may represent a function with ML training capabilities, such as, for example, an ML training function that performs ML model training. At 115, the ML clustering MnS consumer 100 instantiates a request to create a set of clusters associated with an MLEntity (MLEntity ID). In certain example embodiments, the cluster creation request may require various attributes to provide the information required to perform appropriate clustering (see the above examples). At 120, the ML clustering MnS producer 205 creates multiple node clusters based on the received request, where one MLEntity can be trained separately on each node cluster. At 125, after creating the clusters, the ML Clustering MnS Producer 105 instantiates one / multiple ML Training Requests for each cluster. At 130, the ML Training MnS Producer 110 runs the ML training process, and at 135 and 140, compiles the ML training report and notifies the ML Clustering MnS Producer 105 and / or the ML Clustering MnS Consumer 100 when the training is complete (i.e., a new version of the MLEntities is available). At 145, the ML Clustering MnS Producer 105 compiles (multiple) cluster reports, one for each cluster, summarizing the characteristics of the different clusters. The ML Clustering MnS Producer 110 also notifies the ML Clustering MnS Consumer 100 when the report is ready.

[0072] Figure 2 An example signal diagram of another cluster creation request according to certain example embodiments is illustrated. Specifically, Figure 2 An alternative process for requesting cluster creation by an authorized ML clustering MnS consumer (eg, operator) is illustrated. Figure 2 In the example shown, the services of the clustering MnS producer can be added to the ML training MnS producer (i.e., encapsulating the ML clustering responsibilities), and the training request can encapsulate the cluster report as well as the existing attributes. Figure 2As shown, at 210, the ML clustering MnS consumer 200 instantiates a request to create a set of clusters associated with an MLEntity. At 215, the ML training MnS producer 205 creates multiple node clusters, where one MLEntity can be trained separately on each node cluster. At 220, the ML training MnS producer 205 sends an ML training report to the ML clustering MnS consumer 200, where the report can include a summary of different cluster characteristics for each created cluster, and notifies the ML clustering MnS consumer 200 when the report is ready. At 225, the ML training MnS producer 205 runs the ML training process, and at 230, compiles the ML training report, and notifies the ML clustering MnS consumer 200 when the training is complete (a new version of the MLEntities is available).

[0073] Figure 3 An example signal diagram of a dynamic cluster monitoring request according to certain example embodiments is illustrated. At 320, the ML clustering MnS consumer 300 instantiates a request for cluster dynamic monitoring, where the request can be associated with a cluster creation request ID. At 325, the ML clustering MnS producer 305 instantiates the clusteringDynMonitoringProcess to dynamically monitor the cluster based on some monitoring criteria and conditions as described above. At 330 and 335, the ML clustering MnS producer 305 controls different operation triggers. When the conditionFulfilmentCheckRate is triggered for one or more clusters, a data collection mechanism can be triggered to collect any data required to evaluate the minAccuracy and maxClusterStability levels of the relevant clusters. At 340, the ML clustering MnS producer 305 checks whether an operation is triggered.

[0074] like Figure 3 As shown, if a classification operation is triggered at 345, nodes with ML models with accuracy levels below minAccuracy are identified and reclassified to be associated with different clusters (matching higher accuracy requirements). At 350, the context of MLEntities can be updated to take into account adding / removing nodes from one of the multiple clusters. At 355, the ML clustering MnS producer 305 sends a cluster report to the ML clustering MnS consumer 300. A cluster report can be generated to summarize the status of the new cluster, and the ML clustering MnS consumer 300 is notified when the cluster report is ready. Figure 3As further shown, when another operation (e.g., expand, decompose, breakNmerge, or reset) is triggered, at 360, the ML clustering MnS producer 305 performs the clustering operation selected based on the operation triggering condition (e.g., expand, decompose, breakNmerge, reset). Any of these operations can create / decompose / merge / remove one / multiple clusters. Therefore, ML model training can be instantiated.

[0075] At 365, based on the type of operation triggered, the ML clustering MnS producer 305 instantiates one / multiple training requests to train the ML model on the modified cluster(s). At 370, the ML training MnS producer 315 runs the training process. At 375, when the ML training process is complete, the ML training MnS producer 315 notifies the ML clustering MnS consumer 300, and the ML training report is ready and is notified to the ML clustering MnS consumer 300. At 380, when the cluster report is ready, the ML clustering MnS consumer 300 is notified.

[0076] Figure 4 An example signal diagram illustrating the process of a machine learning training request according to certain example embodiments is shown. Specifically, Figure 4 The diagram shows Figure 3 At 415, the ML clustering MnS producer 410 executes the ML dynamic cluster monitoring process. At 420, the ML training MnS consumer 400 instantiates a request to train the ML model. According to certain example embodiments, the cluster dynamic monitoring process may already be running, and the ML training request ML context may be part of the clustering ML context.

[0077] At 425, the ML Training MnS Producer 405 instantiates a change request to the ClusteringContext portion of the ClusterDynMonitoringProcess. This attribute may be a list of all Node / DN portions of the monitored cluster / clusters. At 430, the ML Clustering MnS Producer 410 performs a classification operation to define the clusters that the node involved in the ML training request should be a part of. At 435, the ML Clustering MnS Producer 410 updates the MLEntity of the cluster (the runTimeContext attribute is updated). At 440, the ML Training MnS Producer 405 compiles a report and notifies the ML Training MnS Consumer 400 of the report.

[0078] In certain example embodiments, certain high-level requirements may be defined. For example, in Req_1, the 3GPP Management System may support the ability to perform cluster-based ML training of MLEntitites. In Req_2, the 3GPP Management System may support the ability to create a set of clusters associated with an MLEntity (MLEntity ID) based on a selected set of similarities described by a set of features or a group of different expectRuntimeContexts. Furthermore, in Req_3, the 3GPP Management System may support the ability to monitor the performance of a set of clusters associated with an MLEntity.

[0079] Figure 5 A class Unified Modeling Language diagram is illustrated according to certain example embodiments. Specifically, Figure 5 Provides a visual representation of the relationships between the various entities defined in this article. Figure 5 As shown, the class unified modeling diagram may include ProxyClass (proxy class) 500, InformationObjectClass (information object class) 505, 510, 515, 520, 525 and 530, which respectively define MLClusteringFunction (ML clustering function), ClusterTrainingRequest (cluster training request), ClusterDynMonitoringRequest (cluster dynamic monitoring request), MLEntity (ML entity), ClusterDynMonitoringProcess (cluster dynamic monitoring process) and ClusterReport (cluster report). The class unified modeling diagram may also include datatype (data type) 535 that defines ClusterOperation.

[0080] like Figure 5 As shown, a ManagedEntity 500 may be associated with multiple MLClusteringFunctions 505. MLClusteringFunction 505 may be responsible for performing all clustering related tasks, such as processing requests for cluster training, monitoring clusters, and providing reports on cluster and model performance.

[0081] In some example embodiments, the MLClusteringFunction 505 may handle multiple ClusterTrainingRequests 510. However, each ClusterTrainingRequest 510 may be associated with only one ClusterDynMonitoringProcess 525. The ClusterDynMonitoringProcess 525 may be the process that performs all duties for the maintenance and management of the cluster so that they may be within the defined framework conditions defined in its associated ClusterDynMonitoringRequest 515. A ClusterDynMonitoringRequest 515 (one or more) may also be associated with the MLClusteringFunction 505. In some example embodiments, the MLClusteringFunction 505 may manage all cluster-related tasks for one or more MLEntities 520.

[0082] According to some example embodiments, each ClusterDynMonitoringProcess 525 may generate several ClusterReports 530, and it may also have several ClusterOperation 535 objects that define the boundary and action conditions for each operation. According to some example embodiments, each ClusterDynMonitoringRequest 515 may be associated with one or more ClusterOperations 535 that reflect the desired conditions and operations that may be used to satisfy the request.

[0083] also, Figure 6 Another type of unified modeling language diagram is illustrated in accordance with certain example embodiments. Figure 6As shown, the class unified modeling diagram may include a ProxyClass 600 defining a ManagedEntity, an MLTrainingFunction 605, an InformationObjectClass 610 defining an MLTrainingRequest, an InformationObjectClass 615 defining a ClusterTrainingRequest, an InformationObjectClass 620 defining an MLTrainingReport, an InformationObjectClass 625 defining an MLEntity, and an InformationObjectClass 630 defining a ClusterReport.

[0084] like Figure 6 As shown, MLTrainingFunction 605 may be responsible for training MLEntitites 625. According to certain example embodiments, MLTrainingFunction 605 may be extended to support an improved version of MLTrainingRequest 610. In addition, the new MLTrainingRequest 610 may encapsulate the markup of ClusterTrainingRequest 615. In addition, ClusterTrainingRequest 615 may then generate MLTrainingReports 620 for training and clustering. Thus, the new MLTrainingReport may be associated with ClusterReport 630. In some example embodiments, the responsibilities and features of the cluster-related blocks may remain the same. Figure 5 However, MLTrainingRequest and MLTrainingReport can be extended as described in this article.

[0085] like Figure 5 and Figure 6As shown, the information object class (IOC) MLClusteringFunction represents the entity that undertakes clustering operations (creation, dynamic monitoring) and is a container for (multiple) ClusterTrainingRequest and ClusterDynMonitoringRequest IOCs. The entity represented by the MLClusteringFunction MOI supports clustering of ML models associated with one or more MLEntities.

[0086] Table 6: MLClusteringFunction (ML clustering function) properties

[0087]

[0088] like Figure 5 and Figure 6 As shown in Table 7, the IOC can represent the properties of the ClusterTrainingRequest. For each request to cluster and train an ML model, a consumer can instantiate a new ClusterTrainingRequest on the MLClusteringFunction. Each ClusterTrainingREquest can be associated with an MLEntity (e.g., an MLEntityId, and any one of multiple MLEntityVersions). The ClusterTrainingRequest can include an attribute MinClusterSize to indicate the minimum number of nodes that can form a cluster. In addition, the ClusterTrainingRequest can include an attribute MinExpectedAccuracy to indicate the minimum allowed accuracy value of the cluster to be formed. The ClusterTrainingRequest can also include an attribute MaxNodeOutlierPercentage to indicate the maximum percentage of nodes allowed to be outliers in the clustering process. The ClusterTrainingRequest can also include an attribute clusteringMethodId, which indicates the ID of the clustering method to be used. In some example embodiments, the expectedRuntimeContextSet can be an ExpectedRunTimeContext collection for which the referenced MLEntity will be trained.

[0089] Table 7: ClusterTrainingRequest (cluster training request) properties

[0090]

[0091] like Figure 5 and Figure 6 As shown in Table 8, IOC ClusterDynMonitoringRequest can represent a request for dynamic monitoring of a set of clusters. IOC ClusterDynMonitoringRequest can be instantiated by an authorized ML clustering MnS consumer and can include clusteringOperationConditions and reportTimeWindow. clusteringOperationConditions can indicate the operational characteristics of providing information required to manage and control each operation. In addition, ReportTimeWindow can indicate the time window considered for calculating statistics in reports and operations without specifying a time frame. Each ClusterTrainingRequest can be associated with an MLEntity (an mlEntityID, multiple mlEntityVersions). ClusterTrainingRequest can also be associated with one / multiple clusterIds, which indicate a list of clusters to be processed by the monitoring request.

[0092] Table 8: ClusterDynMonitoringRequest (cluster dynamic monitoring request) attributes

[0093]

[0094] like Figure 5 and Figure 6 As shown in Table 9, the IOC ClusterDynMonitoringProcess may represent a dynamic monitoring process for a cluster set. In some example embodiments, a ClusterDynMonitoringProcess managed object instance (MOI) may be instantiated for each ClusterDynMonitoringRequest. In other example embodiments, the ClusterDynMonitoringProcess may be associated with only one MLEntity, but with different versions of the MLEntity.

[0095] Table 9: ClusterDynMonitoringProcess (cluster dynamic monitoring process) attributes

[0096]

[0097] like Figure 5 and Figure 6As shown in Table 10, the ClusterReport data type can capture the results of performing a clustering operation on a cluster. The ClusterReport can include multiple attributes that can reflect the state of the cluster and act as an indicator of the information of the cluster. In some example embodiments, a ClusterReport can be generated and each type of clustering operation can be performed. In some example embodiments, the content of the ClusterReport can be part of the MLTrainingReport.

[0098] Table 10: ClusterReport properties

[0099]

[0100] like Figure 5 and Figure 6 As shown in Table 11, ClusterOperation can represent the characteristics of the clustering operation.

[0101] Table 11: ClusterOperation properties

[0102]

[0103] Figure 7 An example flow chart of a method according to certain example embodiments is illustrated. In one example embodiment, Figure 7 The method may be performed by a network entity or a group of multiple network elements in a 3GPP system (such as LTE or 5G-NR). For example, in an example embodiment, Figure 7 The method can be similar to Fig. 9 The ML clustering MnS consumer is performed by one of the devices 10 or 20 shown.

[0104] According to certain example embodiments, Figure 7 The method may include, at 700, sending a first request to a management service producer to create a cluster set and train a machine learning model for one or more clusters in the cluster set for multiple contexts. According to certain example embodiments, each cluster includes multiple network nodes, network functions, and management functions. The method may also include, at 705, sending a second request to the management service producer to dynamically monitor the cluster set. The method may also include, at 710, receiving a cluster report from the management service producer as a result of the creation of the cluster set after at least one of the first request or the second request. In addition, the method may include, at 715, receiving a training report from the management service producer as a result of the dynamic monitoring of the cluster set and the training of the machine learning model after at least one of the first request or the second request.

[0105] According to certain example embodiments, the cluster report may include a report for each cluster, and the report for each cluster may include characteristics of the corresponding cluster, and the status of the corresponding cluster. According to some example embodiments, each of the multiple network nodes, network functions, and management functions may be associated with the same machine learning model. According to other example embodiments, the machine learning model may be identified by a machine learning entity identifier, which is associated with the input, output, and architecture. According to further example embodiments, the first request and the second request may include multiple attributes for cluster creation and cluster dynamic monitoring, respectively.

[0106] In some example embodiments, the multiple attributes within the first request for cluster creation may include at least one of the following: a machine learning model identifier, a clustering criteria feature, a Boolean attribute indicating whether clustering should be performed based solely on the clustering criteria feature, a minimum cluster size requirement, a minimum allowed precision value for a cluster, an acceptable precision margin, a maximum network node outlier percentage, an identification of a clustering method, or an expected runtime context set. In some example embodiments, the multiple attributes within the second request for cluster dynamic monitoring may include at least one of the following: an identification of a dynamic monitoring request, an identifier of a machine learning model, a list of versions of a machine learning model, a list of cluster identifiers that a dynamic monitoring request is processing, a list of operation objects that provide information for managing and controlling management operations, or a time window for calculation of statistical information in a report.

[0107] Figure 8 An example flow chart of a method according to certain example embodiments is illustrated. In one example embodiment, Figure 8 The method may be performed by a network entity or a group of multiple network elements in a 3GPP system (such as LTE or 5G-NR). For example, in an example embodiment, Figure 8 The method can be similar to Fig. 9 The ML clustering of the MnS producers is performed by one of the apparatuses 10 or 20 shown.

[0108] According to certain example embodiments, Figure 8The method may include, at 800, receiving a first request from a management service consumer to create a cluster set and train a machine learning model for one or more clusters in the cluster set for multiple different contexts. According to certain example embodiments, each cluster may include multiple network nodes, network functions, and management functions. The method may also include, at 805, receiving a second request from a management service consumer to dynamically monitor the cluster set. The method may also include, at 810, creating a cluster set based on the first request. According to certain example embodiments, each cluster in the cluster set may be associated with a machine learning model. In addition, the method may include, at 815, training a machine learning model for one or more clusters in the cluster set for multiple different contexts based on the first request. In addition, the method may include, at 820, performing a dynamic monitoring process based on the second request to monitor clusters in the cluster set. The method may also include, at 825, sending a cluster report to the management service consumer based on the creation of the cluster set. The method may also include, at 830, sending a training report to the management service consumer based on performing dynamic monitoring of the cluster set and training of the machine learning model.

[0109] According to certain example embodiments, the dynamic monitoring process may be performed depending on the presence of a trigger, and the trigger may include the satisfaction of a set of conditions specified in an operation for managing one or more clusters, the operation including elements defined for managing operation attributes. According to some example embodiments, the operation for managing one or more clusters may include at least one of: a classification operation of network nodes with a machine learning model, an expansion operation of a cluster, a decomposition operation of a cluster, a decomposition and merging operation of a cluster, or a reset operation of a cluster. According to other example embodiments, each of the operations for managing one or more clusters may include: one or more attributes that enable management of a set of clusters.

[0110] In some example embodiments, the cluster report may include a report for each cluster summarizing different characteristics of each cluster, a status of each cluster, and an indication that the report is ready. In some example embodiments, each of the plurality of network nodes, network functions, and management functions may be associated with the same machine learning model. In other example embodiments, the machine learning model may be identified by a machine learning entity identifier, the machine learning identifier being associated with the inputs, outputs, and architecture.

[0111] According to certain example embodiments, the first request and the second request may include multiple attributes for cluster creation and cluster dynamic monitoring, respectively. According to some example embodiments, the multiple attributes for cluster creation within the first request may include at least one of the following: a machine learning model identifier, a clustering standard feature, a Boolean attribute indicating whether clustering should be performed based only on the clustering standard feature, a minimum cluster size requirement, a minimum allowed precision value for the cluster, an acceptable precision margin, a maximum percentage of network node outliers, an identification of the clustering method, or an expected runtime context set. According to some example embodiments, the multiple attributes for cluster dynamic monitoring within the second request may include at least one of the following: an identification of a dynamic monitoring request, an identifier of a machine learning model, a list of versions of the machine learning model, a list of cluster identifiers that the dynamic monitoring request is processing, a list of operation objects that provide information for managing and controlling management operations, or a time window for calculating statistical information in a report.

[0112] Fig. 9 A set of apparatuses 10 and 20 according to certain example embodiments is illustrated. In certain example embodiments, apparatuses 10 and 20 may be elements in a communication network or associated with such a network. For example, apparatus 10 may be an ML clustering MnS consumer or other similar radio communication computer device, and apparatus 20 may be an ML clustering MnS producer.

[0113] In some example embodiments, the apparatuses 10 and 20 may include one or more processors, one or more computer-readable storage media (e.g., memory, storage, etc.), one or more radio access components (e.g., modems, transceivers, etc.), and / or user interfaces. In some example embodiments, the apparatuses 10 and 20 may be configured to operate using one or more radio access technologies, such as GSM, LTE, LTE-A, NR, 5G, WLAN, WiFi, NB-IoT, Bluetooth, NFC, MulteFire, and / or any other radio access technology. It should be noted that a person of ordinary skill in the art will understand that the apparatuses 10 and 20 may include Fig. 9 Components or features not shown.

[0114] like Fig. 9 As shown in the example of , the devices 10 and 20 may include or be coupled to processors 12 and 22 for processing information and executing instructions or operations. The processors 12 and 22 may be any type of general or special purpose processor. In fact, as examples, the processors 12 and 22 may include one or more of a general purpose computer, a special purpose computer, a microprocessor, a DSP, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), and a processor based on a multi-core processor architecture. Although Fig. 91 and 22, but according to other example embodiments, multiple processors may be used. For example, it should be understood that in some example embodiments, the devices 10 and 20 may include two or more processors that may form a multiprocessor system that may support multiprocessing (e.g., in this case, the processor 12 may represent a multiprocessor). According to some example embodiments, the multiprocessor system may be tightly coupled or loosely coupled (e.g., to form a computer cluster).

[0115] Processors 12 and 22 may perform functions related to the operation of devices 10 and 20, including, as some examples, precoding of antenna gain / phase parameters, encoding and decoding of individual bits forming communication messages, formatting of information, and overall control of devices 10 and 20, including Figure 1-Figure 8 The process and examples shown.

[0116] The devices 10 and 20 may also include or be coupled to memory 14 and 24 (internal or external), which may be coupled to the processors 12 and 24, respectively, for storing information and instructions that can be executed by the processors 12 and 24. The memory 14 and 24 may be one or more memories and may be of any type suitable for the local application environment and may be implemented using any suitable volatile or non-volatile data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and / or removable memory. For example, the memory 14 and 24 may include any combination of random access memory (RAM), read-only memory (ROM), static memory such as a magnetic disk or optical disk, a hard disk drive (HDD), or any other type of non-transitory machine or computer readable medium. The instructions stored in the memory 14 and 24 may include program instructions or computer program code that, when executed by the processors 12 and 22, enables the devices 10 and 20 to perform the tasks described herein.

[0117] In certain example embodiments, the devices 10 and 20 may also include or be coupled to a (internal or external) drive or port configured to accept and read an external computer-readable storage medium, such as an optical disk, a USB drive, a flash drive, or any other storage medium. For example, the external computer-readable storage medium may store a computer program or software for execution by the processors 12 and 22 and / or the devices 10 and 20 to perform Figure 1-Figure 8 Any methods and examples shown.

[0118] In some example embodiments, the apparatuses 10 and 20 may further include or be coupled to one or more antennas 15 and 25 for receiving downlink signals and for transmitting from the apparatuses 10 and 20 via the UL. The apparatuses 10 and 20 may further include transceivers 18 and 28 configured to transmit and receive information. The transceivers 18 and 28 may further include a radio interface (e.g., a modem) coupled to the antennas 15 and 25. The radio interface may correspond to a variety of radio access technologies, including one or more of GSM, LTE, LTE-A, 5G, NR, WLAN, NB-IoT, Bluetooth, BT-LE, NFC, RFID, UWB, etc. The radio interface may include other components such as filters, converters (e.g., digital-to-analog converters, etc.), symbol demappers, signal shaping components, inverse fast Fourier transform (IFFT) modules, etc., to process symbols carried by the downlink or UL, such as OFDMA symbols.

[0119] For example, transceivers 18 and 28 may be configured to modulate information onto a carrier waveform for transmission by antennas 15 and 25, and demodulate information received via antennas 15 and 25 for further processing by other elements of apparatuses 10 and 20. In other example embodiments, transceivers 18 and 28 may be capable of directly sending and receiving signals or data. Additionally or alternatively, in some example embodiments, apparatus 10 may include input and / or output devices (I / O devices). In certain example embodiments, apparatuses 10 and 20 may also include a user interface, such as a graphical user interface or a touch screen.

[0120] In certain example embodiments, the memories 14 and 34 store software modules that provide functionality when executed by the processors 12 and 22. These modules may include, for example, an operating system that provides operating system functionality for the devices 10 and 20. The memory may also store one or more functional modules, such as applications or programs, that are used to provide additional functionality for the devices 10 and 20. The components of the devices 10 and 20 may be implemented in hardware, or as any suitable combination of hardware and software. According to certain example embodiments, the devices 10 and 20 may optionally be configured to communicate with each other (in any combination) via a wireless or wired communication link 70 according to any radio access technology (such as NR).

[0121] According to some example embodiments, processors 12 and 22 and memories 14 and 24 may be included in processing circuitry or control circuitry, or may form part of processing circuitry or controller circuitry. In addition, in some example embodiments, transceivers 18 and 28 may be included in transceiver circuitry, or may form part of transceiver circuitry.

[0122] For example, in certain example embodiments, the device 10 may be controlled by the memory 14 and the processor 12 to send a first request to a management service producer to create a cluster set and train a machine learning model for one or more clusters in the cluster set for multiple contexts. According to certain example embodiments, each cluster may include multiple network nodes, network functions, and management functions. The device 10 may also be controlled by the memory 14 and the processor 12 to send a second request to the management service producer to dynamically monitor the cluster set. The device 10 may also be controlled by the memory 14 and the processor 12 to receive a cluster report from the management service producer as a result of the creation of the cluster set after at least one of the first request or the second request. In addition, the device 10 may be controlled by the memory 14 and the processor 12 to receive a training report from the management service producer as a result of the dynamic monitoring of the cluster set and the training of the machine learning model after at least one of the first request or the second request.

[0123] In other example embodiments, the device 20 may be controlled by the memory 24 and the processor 22 to receive a first request from a management service consumer to create a cluster set and train a machine learning model for one or more clusters in the cluster set for multiple different contexts. According to certain example embodiments, each cluster may include multiple network nodes, network functions, and management functions. The device 20 may also be controlled by the memory 24 and the processor 22 to receive a second request from a management service consumer to dynamically monitor the cluster set. The device 20 may also be controlled by the memory 24 and the processor 22 to create a cluster set based on the first request. According to certain example embodiments, each cluster in the cluster set may be associated with a machine learning model. In addition, the device 20 may be controlled by the memory 24 and the processor 22 to train a machine learning model for one or more clusters in the cluster set for multiple different contexts based on the first request. In addition, the device 20 may be controlled by the memory 24 and the processor 22 to perform a dynamic monitoring process based on the second request to monitor the clusters in the cluster set. The device 20 may also be controlled by the memory 24 and the processor 22 to send a cluster report to the management service consumer based on the creation of the cluster set. The device 20 can also be controlled by the memory 24 and the processor 22 to send training reports to management service consumers based on dynamic monitoring of the execution cluster set and training of the machine learning model.

[0124] In some example embodiments, an apparatus (e.g., apparatus 10 and / or apparatus 20) may include components for performing the methods, processes, or any variants discussed herein. Examples of the components may include one or more processors, memories, controllers, transmitters, receivers, and / or computer program codes for causing the execution of operations.

[0125] Certain example embodiments may relate to an apparatus comprising components for performing any of the methods described herein, including, for example, components for sending a first request to a management service producer to create a cluster set and train a machine learning model for one or more clusters in the cluster set for multiple contexts. According to certain example embodiments, each cluster may include multiple network nodes, network functions, and management functions. The apparatus may also include components for sending a second request to the management service producer to dynamically monitor the cluster set. The apparatus may also include components for receiving a cluster report from the management service producer as a result of the creation of the cluster set after at least one of the first request or the second request. In addition, the apparatus may include components for receiving a training report from the management service producer as a result of the dynamic monitoring of the cluster set and the training of the machine learning model after at least one of the first request or the second request.

[0126] Other example embodiments may relate to an apparatus comprising components for performing any of the methods described herein, including, for example, components for receiving a first request from a management service consumer to create a cluster set and train a machine learning model for one or more clusters in the cluster set for multiple different contexts. According to certain example embodiments, each cluster may include multiple network nodes, network functions, and management functions. The apparatus may also include components for receiving a second request from a management service consumer to dynamically monitor the cluster set. The apparatus may also include components for creating a cluster set based on the first request. According to certain example embodiments, each cluster in the cluster set may be associated with a machine learning model. In addition, the apparatus may include components for training a machine learning model for one or more clusters in the cluster set for multiple different contexts based on the first request. In addition, the apparatus may include components for performing a dynamic monitoring process to monitor clusters in the cluster set based on the second request. The apparatus may also include components for sending a cluster report to the management service consumer based on the creation of the cluster set. The apparatus may also include components for sending a training report to the management service consumer based on performing dynamic monitoring of the cluster set and training of the machine learning model.

[0127] Certain example embodiments described herein provide several technical improvements, enhancements and / or advantages. For example, in some example embodiments, a management system may be enabled to optimize computing resources by performing a training of an ML entity once for an entire cluster. In other example embodiments, an authorized consumer may be enabled to monitor available clusters and adjust them according to preferred criteria. In other example embodiments, the complexity of model maintenance may be reduced, and the ability to obtain further understanding and insight into network performance changes and environmental impacts on network entities in a statistical and tangible manner may be provided.

[0128] A computer program product may include one or more computer executable components that are configured to perform some example embodiments when the program is run. The one or more computer executable components may be at least one software code or part thereof. Modifications and configurations required to implement the functionality of some example embodiments may be performed as (multiple) routines, which may be implemented as added or updated (multiple) software routines. (Multiple) software routines may be downloaded to the device.

[0129] For example, software or computer program code or parts thereof may be in source code form, object code form or some intermediate form and may be stored in some carrier, distribution medium or computer readable medium, which may be any entity or device capable of carrying the program. For example, such carriers may include recording media, computer memory, read-only memory, optoelectronic and / or electrical carrier signals, telecommunication signals and software distribution packages. Depending on the processing power required, the computer program may be executed in a single electronic digital computer or may be distributed among multiple computers. The computer readable medium or computer readable storage medium may be a non-transitory medium.

[0130] In other example embodiments, the functions may be performed by hardware or circuitry included in a device (e.g., device 10 or device 20), such as by using an application specific integrated circuit (ASIC), a programmable gate array (PGA), a field programmable gate array (FPGA), or any other combination of hardware and software. In yet another example embodiment, the functions may be implemented as signals, non-tangible components that may be carried by electromagnetic signals downloaded from the Internet or other networks.

[0131] According to certain example embodiments, an apparatus (such as a node, a device or a corresponding component) may be configured as a circuit system, a computer or a microprocessor (such as a single-chip computer element) or a chipset, which includes at least a memory for providing storage capacity for arithmetic operations and an operation processor for performing arithmetic operations.

[0132] Those of ordinary skill in the art will readily appreciate that the above disclosure may be practiced with procedures in a different order, and / or with hardware elements in a configuration different from that disclosed. Therefore, although the present disclosure has been described based on these example embodiments, certain modifications, variations, and alternative constructions will be apparent to those skilled in the art while remaining within the spirit and scope of the example embodiments. Although the above embodiments relate to 5G NR and LTE technologies, the above embodiments may also be applied to any other current or future 3GPP technologies, such as Advanced LTE and / or fourth generation (4G) technologies.

[0133] Partial glossary:

[0134] 3GPP Third Generation Partnership Project

[0135] 5G Fifth Generation

[0136] 5GCN 5G Core Network

[0137] 5GS 5G System

[0138] AI

[0139] BS Base Station

[0140] DL Downlink

[0141] eNB Enhanced Node B

[0142] E-UTRAN Evolved UTRAN

[0143] gNB 5G or Next Generation NodeB

[0144] LTE Long Term Evolution

[0145] ML Machine Learning

[0146] NR New Radio

[0147] PRACH Physical Random Access Channel

[0148] RACH Random Access Channel

[0149] UE User Equipment

[0150] UL Uplink

Claims

1. A method for use in a wireless network, comprising: Sending a first request to a management service producer to create a set of clusters and train a machine learning model for one or more clusters in the set of clusters for a plurality of contexts, wherein each cluster includes a plurality of network nodes, a network function, and a management function; Sending a second request for dynamically monitoring the set of clusters to the management service producer; receiving a cluster report from the management service producer as a result of the creation of the cluster set after at least one of the first request or the second request; as well as After at least one of the first request or the second request, a training report is received from the management service producer as a result of the dynamic monitoring of the set of clusters and the training of the machine learning model.

2. The method according to claim 1, wherein the cluster report includes a report for each cluster, and The report for each cluster includes: Characteristics of the corresponding cluster, and the status of the corresponding cluster.

3. The method according to any one of claims 1 to 2, wherein each of the plurality of network nodes, network functions, and management functions is associated with the same machine learning model, and The machine learning model is identified by a machine learning entity identifier, and the machine learning entity identifier is associated with inputs, outputs, and architecture.

4. The method according to any one of claims 1 to 3, wherein the first request and the second request respectively comprise: Multiple properties for cluster creation and cluster dynamic monitoring.

5. The method according to claim 4, wherein the plurality of attributes for cluster creation in the first request comprises at least one of the following: Machine learning model identifier, Clustering standard features, a Boolean property indicating whether clustering should be performed based only on the clustering criteria feature, Minimum cluster size requirement, The minimum allowed precision value of the cluster, Acceptable accuracy margin, Maximum network node outlier percentage, the identifier of the clustering method, or The expected runtime context collection.

6. The method according to claim 4, wherein the plurality of attributes for cluster dynamic monitoring in the second request include at least one of the following: an identifier of the dynamic monitoring request, an identifier of the machine learning model, a list of versions of the machine learning model, The list of cluster identifiers for which the dynamic monitoring request is being processed, A list of operation objects that provide information used to manage and control administrative operations, or The time window used for calculation of statistics in the report.

7. A method for use in a wireless network, comprising: receiving a first request from a management service consumer to create a set of clusters and train a machine learning model for one or more clusters in the set of clusters for a plurality of different contexts, wherein each cluster includes a plurality of network nodes, network functions, and management functions; receiving a second request from the management service consumer to dynamically monitor the set of clusters; creating the set of clusters based on the first request, wherein each cluster in the set of clusters is associated with the machine learning model; training the machine learning model for the one or more clusters in the set of clusters for the plurality of different contexts based on the first request; performing a dynamic monitoring process based on the second request to monitor the cluster in the set of clusters; Based on the creation of the cluster set, sending a cluster report to the management service consumer; as well as Based on performing the dynamic monitoring of the set of clusters and the training of the machine learning model, a training report is sent to the management service consumer.

8. The method according to claim 7, wherein the dynamic monitoring process is performed depending on the presence of a trigger, and The triggers include: Satisfaction of a set of conditions specified in an operation for managing the one or more clusters, the operation including elements defined for managing operation attributes.

9. The method of claim 8, wherein the operations for managing the one or more clusters include at least one of: classification operations of network nodes having said machine learning model, Cluster expansion operations, The decomposition operation of the cluster, Splitting and merging operations of said clusters, or A reset operation for the cluster.

10. The method of claim 8, wherein each of the operations for managing the one or more clusters comprises: One or more attributes that enable management of the set of clusters.

11. A method according to any one of claims 7 to 10, wherein the cluster report comprises a report for each cluster summarizing different characteristics of each cluster, a status of each cluster and an indication that the report is ready.

12. The method according to any one of claims 7 to 11, wherein each of the plurality of network nodes, network functions, and management functions is associated with the same machine learning model, and The machine learning model is identified by a machine learning entity identifier, and the machine learning identifier is associated with inputs, outputs, and architecture.

13. The method according to any one of claims 7 to 12, wherein the first request and the second request respectively comprise: Multiple properties for cluster creation and cluster dynamic monitoring.

14. The method according to claim 13, wherein the plurality of attributes for cluster creation in the first request comprises at least one of the following: Machine learning model identifier, Clustering standard features, a Boolean property indicating whether clustering should be performed based only on the clustering criteria feature, Minimum cluster size requirement, The minimum allowed precision value of the cluster, Acceptable accuracy margin, Maximum network node outlier percentage, the identifier of the clustering method, or The expected runtime context collection.

15. The method according to claim 13, wherein the plurality of attributes for cluster dynamic monitoring in the second request include at least one of the following: an identifier of the dynamic monitoring request, an identifier of the machine learning model, a list of versions of the machine learning model, The list of cluster identifiers for which the dynamic monitoring request is being processed, A list of operation objects that provide information used to manage and control administrative operations, or The time window used for calculation of statistics in the report.

16. An apparatus for use in a wireless network, comprising: at least one processor; as well as At least one memory comprising computer program code which, when executed by the at least one processor, causes the apparatus to at least perform the method according to any one of claims 1 to 15.

Images