Robust AI / ML based network control with trust building

Abstract

An apparatus, e.g. a Network Control Function (NCF) 110, receives an optimal network configuration suggestion 412, e.g. optimal network configuration parameters and / or expected network performance corresponding to the optimal network configuration parameters, from a cognitive function (CF) 120, sends an evaluation request 414 for the optimal network configuration suggestion to a reliability check cognitive function (RCCF) 122 and receives an evaluation response 422 from the RCCF 122 containing a quality estimate and an acceptance criterion for the optimal network configuration suggestion. The apparatus, e.g. NCF 110, may decide 424 to accept or reject the optimal network configuration suggestion based on the evaluation response 422 and send an acknowledge message 428 to the CF 120, the acknowledge message 428 containing at least one of the following: the decision of accepting or rejecting the optimal network configuration suggestion or a decision cause. The CF 120 may use the decision and decision cause as feedback information to improve 430 an algorithm or model that generates the optimal network configuration suggestion.

Description

TECHNICAL FIELD

[0001] Various example embodiments described herein generally relate to communication technologies, and more particularly, to devices, methods, apparatuses and computer readable mediums supporting robust artificial intelligence / machine learning based network control with trust building. BACKGROUND

[0002] Certain abbreviations that may be found in the description and / or in the figures are herewith defined as follows: AI / ML Artificial Intelligence / Machine Learning CAN Cognitive Autonomous Network CCL Closed Control Loop CF Cognitive Function CIO Cell Individual Offset KPI Key Performance Indicator MLB Mobility Load Balancing MNO Mobile Network Operator MRO Mobility Robustness Optimization NCF Network Control Function RAN Radio Access Network RCCF Reliability Check Cognitive Function TTT Time to Triger

[0003] Cognitive Autonomous Network (CAN) is a promising approach for advancing network management automation using artificial intelligence / machine learning (AI / ML) based functions known as cognitive functions (CFs). The CFs interact with network environment to learn and predict optimal network configuration parameters to optimize their objectives. To minimize conflicts among actions of multiple CFs, the CFs send their proposed network configurations to a central network control function (NCF), which assesses the network configurations and decides to accept or reject them. The NCF can consider the network configurations from different CFs together and compute the final network configuration parameters that optimize the combined interests of all the CFs. SUMMARY

[0004] A brief summary of example embodiments is provided below to provide basic understanding of some aspects of various embodiments. It should be noted that this summary is not intended to identify key features of essential elements or define scopes of the embodiments, and its sole purpose is to introduce some concepts in a simplified form as a preamble for a more detailed description provided below.

[0005] In a first aspect, an example embodiment of an apparatus is provided. The apparatus may comprise at least one processor and at least one memory. The at least one memory may store instructions that, when executed by the at least one processor, cause the apparatus at least to receive an optimal network configuration suggestion from a cognitive function, to send an evaluation request for the optimal network configuration suggestion to a reliability check cognitive function, and to receive an evaluation response from the reliability check cognitive function containing a quality estimate and an acceptance criterion for the optimal network configuration suggestion.

[0006] In a second aspect, an example embodiment of an apparatus is provided. The apparatus may comprise at least one processor and at least one memory. The at least one memory may store instructions that, when executed by the at least one processor, cause the apparatus at least to send an optimal network configuration suggestion to a network control function, and to receive an acknowledge message from the network control function. The acknowledge message contains at least one of the following: a decision of accepting or rejecting the optimal network configuration suggestion; or a decision cause.

[0007] In a third aspect, an example embodiment of an apparatus is provided. The apparatus may comprise at least one processor and at least one memory. The at least one memory may store instructions that, when executed by the at least one processor, cause the apparatus at least to receive an evaluation request for an optimal network configuration suggestion from a network control function, to determine a quality estimate and an acceptance criterion for the optimal network configuration suggestion, and to send an evaluation response to the network control function containing the quality estimate and the acceptance criterion for the optimal network configuration suggestion.

[0008] In a fourth aspect, an example embodiment of an apparatus is provided. The apparatus may comprise at least one processor and at least one memory. The at least one memory may store instructions that, when executed by the at least one processor, cause the apparatus at least to receive an optimal network configuration suggestion from a cognitive function, to determine a quality estimate and an acceptance criterion for the optimal network configuration suggestion, and to decide to accept or reject the optimal network configuration suggestion based on the determined quality estimate and acceptance criterion.

[0009] Example embodiments of methods, apparatuses and computer readable mediums are also provided, which generally correspond to the above-described example embodiments and a repetitive description thereof is omitted here for convenience.

[0010] Other features and advantages of the example embodiments of the present disclosure will also be apparent from the following description of specific embodiments when read in conjunction with the accompanying drawings, which illustrate, by way of example, the principles of example embodiments of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Some example embodiments will now be described, by way of non-limiting examples, with reference to the accompanying drawings.

[0012] Fig 1 is a schematic block diagram illustrating an example framework for network control and optimization in which example embodiments of the present disclosure can be implemented.

[0013] Fig 2 is a schematic block diagram illustrating an example framework for robust network control according to example embodiments of the present disclosure.

[0014] Fig. 3 is a schematic signaling diagram illustrating a process according to example embodiments of the present disclosure.

[0015] Fig 4 is a schematic signaling diagram illustrating a process according to example embodiments of the present disclosure.

[0016] Fig 5 is a schematic signaling diagram illustrating a process according to example embodiments of the present disclosure.

[0017] Fig. 6 is a schematic signaling diagram illustrating a process according to example embodiments of the present disclosure.

[0018] Fig. 7 is a schematic flowchart illustrating a method according to example embodiments of the present disclosure.

[0019] Fig. 8 is a schematic flowchart illustrating a method according to example embodiments of the present disclosure.

[0020] Fig 9 is a schematic flowchart illustrating a method according to example embodiments of the present disclosure.

[0021] Fig 10 is a schematic flowchart illustrating a method according to example embodiments of the present disclosure.

[0022] Fig. 11 is a schematic block diagram illustrating an apparatus according to example embodiments of the present disclosure.

[0023] Fig 12 is a schematic block diagram illustrating an apparatus according to example embodiments of the present disclosure.

[0024] Fig 13 is a schematic block diagram illustrating an apparatus according to example embodiments of the present disclosure.

[0025] Fig. 14 is a schematic block diagram illustrating an apparatus according to example embodiments of the present disclosure.

[0026] Fig 15 is a schematic block diagram illustrating devices in a network control system according to example embodiments of the present disclosure.

[0027] Throughout the drawings, same or similar reference numbers indicate same or similar elements. A repetitive description on the same elements would be omitted. DETAILED DESCRIPTION

[0028] Herein below, some example embodiments are described in detail with reference to the accompanying drawings. The following description includes specific details for the purpose of providing a thorough understanding of various concepts. However, it will be apparent to those skilled in the art that these concepts may be practiced without these specific details. In some instances, well known circuits, techniques and components are shown in block diagram form to avoid obscuring the described concepts and features.

[0029] Fig 1 illustrates an example framework 100 for network control and optimization in which example embodiments of the present disclosure can be implemented. As shown in Fig. 1, the network control and optimization framework 100 may include plural cognitive functions (CFs) 120, shown as CFi 120a, CF2 120b and CF3 120c, which observe and learn behavior of a network environment e.g. a radio access network (RAN) 130. Given the network states / features X=[Xi, X2, ..., Xn], the network performance, which is represented by a set of key performance indicators (KPIs) Y=[Yi, Y2, ..., Yn], can be controlled by some parameters p=[pi, P2, ..., pn]. The true network mapping may be represented by T: (X, p) —> Y. The CFs 120 may periodically check whether the network state or performance has changed. If the network state or performance has changed, the CFs 120 computes optimal control parameters p* based on its learning and sends the optimal control parameters p* to a central network control function (NCF) 110 if the optimal control parameters p* are different from the actual control parameters p currently configured at the RAN 130. The NCF 110 assesses and decides to adopt or abandon the optimal control parameters p* suggested by the CFs 120. The NCF 110 may or may not have access to network data from the RAN 130. Fig. 1 shows relevant interfaces between the NCF 110, the CFs 120 and the RAN 130.

[0030] The CFs 120 may be third party applications and they may not directly communicate with each other. For instance, the CFs 120 may only indirectly communicate with each other through the NCF 110. A given CFn 120 may be realized using a hidden AI / ML model Mn: (Xn, pn) —> Yn, where Xn is the input including features describing the state of the RAN 130, pn is the parameter(s) of interest, and Yn is the output of the AI / ML model. In usual cases, Yn may represent some target KPIs, e.g., cell load for mobility load balancing (MLB) or successful handover ratio for mobility robustness optimization (MRO). This model Mn approximates the true RAN mapping Tn: (Xn, pn) —’Yn, and different models Mn may be designed to optimize different KPIs Yn. Some control parameters p may be shared by multiple CFs 120. For instance, a first CF for performing mobility load balancing (MLB) and a second CF for performing mobility robustness optimization (MRO) may share control parameters such as time-to-trigger (TTT) and cell individual offset (CIO). When the NCF 110 receives TTT and CIO values proposed by one or both of the first CF and the second CF, the NCF 110 may evaluate how good these values are for both MLB and MRO and compute final TTT and CIO values to be used to config the RAN 130.

[0031] Fig 2 illustrate an example framework 200 for robust network control according to example embodiments of the present disclosure. For ease of notation, in the following description, the index n is dropped and a single CF is discussed. It would be appreciated that the description relative to the signal CF is also applicable to other CFs. Throughout the present disclosure, variables may have a single value (e.g. a scalar) or plural values (e.g. a vector or matrix).

[0032] Referring to Fig. 2, given network data (X, p, Y), the CF 120 may use its model M to approximate the network environment T and optimize the target KPI Y=M(X, p) by computing the optimal value of p This framework is general, and most of the AI / ML optimization methods, e.g., those with supervised learning or unsupervised learning, can be represented by the framework with minor adjustments. The inherent complexity of the approximation, i.e., how well in principle a certain mapping T: (X, p) —> Y can be learned given the available (even high quality) data (X, p, Y), may be large. This is especially the case in present and future wireless networks which are complex and dynamic, and there are many features in X. In this case, even small changes in X and p may cause large changes in Y. If the true RAN mapping T is sufficiently complex, e.g., it shows large local variability on X, then generalizations made by M may be very error prone.

[0033] The CF 120 may enable a performance monitoring mechanism to detect a suboptimal configuration of the RAN 130 by using the AI / ML model M. If the received network configuration parameter p is optimal in the network state X, the CF 120 may continue its learning of the model M from the network environment T. If the received network configuration parameter p becomes suboptimal as the network state X has changed, the CF 120 may compute optimal control parameter value p* based on the model M. The optimal parameter value p* is communicated as a “suggestion” to the NCF 110 by the CF 120 over the interface shown in Fig. 1, and it is claimed by the CF 120 that the suggestion shall result in the KPI value Y* which is better than the current KPI value Ycurr in the network in the current network state. The NCF 110 will take this suggestion into account before changing the current parameter value pcl,IT. The current parameter value pcurr can either be set by default by the mobile network operator (MNO) or it can be set based on long-term performance optimization by some CAN mechanism in the network. In this sense, the NCF 110 considers pcurr to be reliable in terms of maintaining longterm network performance. The MNO usually prefers that only small changes in pcurr are made in order to fix time-critical performance issues in the network. The NCF 110 makes small changes in pcurr based on the suggestion it receives from the CF 120. From the perspective of the NCF 110, the CF 120 is a black-box function where the following assumptions are valid: • The CF model M: (X, p) —> Y is hidden; • The NCF 110 is not aware of how well the CF model M approximates the true network environment T relevant to the CF.

[0034] These assumptions are applicable in a multi-vendor environment where different CFs are maybe supplied by different vendors. Under these assumptions, the NCF 110 has to be confident that if it makes the suggested change A*=|pcurr - p*| to the current parameter value peurr, following the suggestion from the CF 120, this will result in a KPI value Y in the RAN 130 close to the expected KPI value Y* computed by the CF 120. In this sense, the NCF must have some “level of trust” in the CF 120.

[0035] As shown in Fig. 2, the framework for building the trust in the CFs 120 may be divided into following aspects: a) A CF performance learning mechanism enabling the NCF 110 to assess the quality (i.e., accuracy / reliability) of suggestions made by the CFs 120 independently and reliably. Since this mechanism is independent, the NCF 110 cannot be manipulated by the CFs 120. b) A criterion computation and check mechanism for computing a criterion by which the NCF 110 can decide to accept or reject the parameter change A*.

[0036] The above two aspects will be discussed in more detail below. In the first aspect, the quality of the suggestion p* depends on how well the CF 120 can learn the behavior of RAN 130, i.e., how well the model M approximates the true network mapping T. The CF 120 can improve its approximation over time, i.e., as it is exposed to more and more network data (X, p, Y) sampled from the RAN 130. As an example, the quality of this approximation may be estimated by calculating a prediction or confidence interval around Y*, i.e., the KPI prediction made by the CF 120 as a function of X and p*. The prediction interval conveys the following information: the true future KPI value Y achieved by applying the parameter change A* in the RAN 130 is within the prediction interval r(X, p*) = Yint = [Ymin, Ymax] with a probability at least a close to 1, where Ymin and Ymax are lower and upper bounds of the prediction interval Yint, respectively. Typically, the KPI prediction Y* is contained within the prediction interval Yint. The prediction interval function r: (X, p*) Yint depends on X and p* and it is, therefore, sensitive to the network features / states X and the parameter suggestion p*. Furthermore, even if no time variable is introduced explicitly, the RAN environment (i.e. the function T: (X, p) —> Y) changes over time, and therefore the prediction interval Yint would be updated over time in an online fashion.

[0037] Obviously, the tighter the interval r(X, p*) = Yint is, the more confidence it conveys in Y* and ultimately the suggestion p*. The prediction intervals can be obtained by using data-driven methods such as conformal regression which deliver prediction intervals that hold with valid probability at least a, where the value of a is generally fixed to be close to 1 (e.g., 0.95). The exact / true length of Yint ultimately depends on how well the CF model M approximates the RAN environment T at the network state X and the network configuration parameter p* However, the NCF 110 can only estimate r(X, p) with the help of a finite amount of data. Since the estimation is data-driven, generally, for a fixed probability a, the prediction intervals become more accurate as more data is available, i.e., the NCF 110 can better estimate its confidence in the CF 120 overtime. Any data-driven method is subject to this limitation.

[0038] In the second aspect, an appropriate criterion is needed for the NCF 110 to decide to accept or reject the parameter change A*. If the prediction intervals Yint can be maintained accurately over time, then the NCF 110 can use them to evaluate the parameter change A*. Now the question is how to impose the acceptance criterion c on the size of Yint that is the output of r(X, p*). On one hand, if the acceptance criterion c is too lenient, i.e., relatively large values of Yint would be allowed, then it may result in unreliable changes in network parameters. On the other hand, an overly stringent criterion that requires very tight intervals Yint would inhibit the CF’s ability to alleviate time-critical performance problems in Ycurr. In prior arts, it is suggested that the criterion on the intervals is set a priori by the MNO, obviously independent of X, Y*, and A* because these variables are not known a priori. Such general criteria must be stringent to rule out large and unreliable parameter changes in the preferred reliable value pcurr.

[0039] The previous approaches have following drawbacks. First, for certain values of X and A*, it may be impossible for the CF 120 to meet the fixed acceptance criteria c set by the MNO on r(X, p*). The reason is that the network mapping T depends locally on the network states X and the configuration parameters p, and it can be arbitrarily difficult to approximate the network mapping T for certain values of X and p, even if a large amount of data is available to the CF 120.

[0040] Second, in the early phase of its learning, the CF 120 may not exhibit much accuracy as it has not learnt the network mapping T sufficiently. In this case, the CF 120 may not be allowed to make any change in pcurr no matter how small it is, if the criterion c is very stringent and fixed.

[0041] Third, the data-driven estimation of r(X, p*) by the NCF 110 is dependent on network data and suggestions made by the CF 120. It means that in the early phase of operation, the NCF 110 cannot be itself too confident on how accurate the estimated intervals r(X, p*) are.

[0042] Example embodiments of the present disclosure propose methods for robust network control by developing trust and collaboration between NCF and CF over time. On one hand, some methods enable the AI / ML based CF to approximate the behavior of RAN with taking into account as many external factors as possible to improve quality of the AI / ML model and hence quality of the configuration suggestion derived from the model. On the other hand, some methods help the NCF to obtain accurate prediction intervals on the expected KPI value Y* computed by the CF, and flexible policy / criterion c for the prediction intervals by which the NCF can properly regulate the configuration suggestions p* received from the CF. In some example embodiments, a dynamic criterion denoted by c(A*, Ycurr, Y*) may be computed, which depends on the suggested parameter change A*, the expected KPI value Y* computed by the CF, and the current KPI value Ycurr that the CF aims to improve. In some example embodiments, the criterion c may be inversely proportional to the suggested parameter change A* and directly proportional to the expected KPI improvement |Ycurr- Y*|. It is believed that significant time-critical improvements in the KPI Y can result from the CF configuration parameter suggestion p* even if the suggested parameter change A* is very small, i.e., the parameter value p* suggested by the CF is not far from the current value pcurr actually configured at the network. If the deterioration in the current KPI value YCU1T is significant, the expected KPI value Y* is significantly better than the current KPI value Ycurr, and the parameter change A* is small, then the NCF should be less stringent in its requirement on the prediction interval r(X, p*). On the contrary, if the parameter change A* is large, i.e., the CF suggests a large change to the preferred parameter value pcurr, then the NCF should be stringent in its requirement on the prediction interval r(X, p*). In this way, a flexible acceptance criterion c may be determined depending on the suggested parameter change A*, the current KPI value Ycuri and the expected KPI value Y*. The NCF can decide to accept the suggested parameter change A* if the corresponding prediction interval r(X, p*) has a size less than or equal to the criterion c, or reject the parameter change A* if the prediction interval r(X, p*) has a size larger than the criterion c. The decision made by the NCF also indicates whether the CF model M is judged to be good enough. In some example embodiments, both the criterion c and the prediction interval r(X, p*) may be sent to the CF along with the decision made by the NCF. The CF can use this information as feedback that it can utilize to improve its estimate M of the true network environment T over time.

[0043] Fig 3 illustrates a process 300 for CF performance evaluation according to example embodiments of the present disclosure. The process 300 may be implemented at the NCF 110, the CF 120, the RAN 130, and a reliability check cognitive function (RCCF) 122 which may reside alongside the CF 120 in the system. It is assumed that the NCF 110 does not have access to network data of the RAN 130, and therefore the RCCF 122 may be adapted to perform the CF performance evaluation function.

[0044] Referring to Fig. 3, the CF 120 may receive network data from the RAN 130 at 310, and the RCCF 122 may receive network data from the RAN 130 at 312. It is worth noting that the network data available to the CF 120 is specific to the CF 120, e.g. the network data (Xn, pn, Yn) where n represents the index of the CF 120, while the RCCF 122 may have access to network data that it can use to evaluate performance of any CF in the system, e.g. the network data (Xi, pi, Yi) where i = 1, ..., N and N represents the number of CFs in the system.

[0045] At 314, the CF 120 may use the network data to train its AI / ML model M that approximate the network environment of the RAN 130. Depending on the AI / ML model M, appropriate training methods including but not limited to supervised training or unsupervised training may be used. The CF 120 may periodically or continuously receive the network data from the RAN 130 and train its AI / ML model M. As more network data is available, the model M learns the network mapping T better, i.e., it can more accurately approximate the network mapping T.

[0046] When the AI / ML model M is trained to a satisfactory level, e.g., the loss of the model output compared with the ground-truth data is smaller than a predetermined quantity or within a minimal range, the CF 120 may inform at 316 theNCF 110 that the model learning is completed. Then the NCF 110 may initiate performance evaluation of the CF model M. In some example embodiments, the CF 120 may continue to train and update the model M after it sends the learning complete message to the NCF 110, i.e., the model M may be trained in an online manner.

[0047] At 318, the NCF 110 may request the RCCF 122 to evaluate the AI / ML model M of the CF 120. As mentioned above, the NCF 110 does not have access to the network data, and therefore it requests the RCCF 122 which has access to the network data to evaluate the CF model M.

[0048] In response to the request, the RCCF 122 may send a request for data and / or information needed for the model evaluation to the NCF 110. For instance, in addition to the network data (X, p, Y) received from the RAN 130 at 312, the RCCF 122 may need to know at least the inference output of the model M in order to evaluate the model performance. It is assumed that the CFs including the CFs 120 and the RCCF 122 cannot communicate with each other directly. Therefore, the RCCF 122 may request the data and / or information needed for the model evaluation from the NCF 110.

[0049] The NCF 110 may acquire the data and / or information for model evaluation from the CF 120 through a data exchange procedure at 322, e.g., a request and response procedure. The acquired data and / or information includes but is not limited to the inference output of the model M, i.e., the predicted optimal configuration parameter p* and the expected KPI value Y*. Then the NCF 110 may forward the data and / or information to the RCCF 122 at 324. In this way, the NCF 110 facilitates indirect communication between the CF 120 and the RCCF 122. It is important to note that the RCCF 122 receives the network data from the RAN 130, but not through the CF 120, to ensure that the model evaluation is performed independently of the CF 120 and to prevent any manipulative behavior by CF.

[0050] The RCCF 122 may perform appropriate methods using the network data from the RAN 130 and the inference output of the model M to estimate the performance of the model M at 326. In an example embodiment, the RCCF 122 may be implemented as an AI / ML based function which uses an AI / ML model to estimate the performance of the CF model M. For example, conformal regression models may be used at the RCCF 122, which is a method that uses past experience to determine precise levels of confidence in new predictions by converting a given prediction into a prediction set / interval which may contain the prediction and is valid with probability at least a. In an example embodiment, the RCCF 122 may produce a prediction interval r(X, p*) = Yim = [Ymin, Ymax] which may contain the expected / predicted KPI value Y* and it is valid with probability at least a close to 1. The probabilistic guarantee parameter a may be set a priori by the MNO, and a typical value of a is e.g. 0.95. Apparently, the tighter the prediction interval r(X, p*) = Yint is, the more confidence it conveys in Y* and ultimately the suggested parameter p*. Therefore, the prediction interval r(X, p*) may be used to evaluate the quality of the model M. In the learning phase, the RCCF 120 may learn the prediction intervals for the CF model M using the network data periodically received from the RAN 130 and the model M inference output periodically received from the NCF 110. In an example embodiment, the process 300 may be continuously performed in an on-line fashion. In addition, it is worth noting that the RCCF 122 can produce the prediction intervals before the CF 120 makes any actual configuration suggestion to the NCF 110.

[0051] Fig. 4 illustrates a process 400 for network control based on dynamic criterion generation according to example embodiments of the present disclosure. The process 400 may be performed at the NCF 110, the CF 120, the RCCF 122 and the RAN 130. It is also assumed that the NCF 110 does not have access to the network data of the RAN 130, and the CF 120 and the RCCF 122 cannot directly communicate with each other.

[0052] Referring to Fig. 4, at 410, the CF 120 may detect a suboptimal configuration in the RAN 130. For example, the CF 120 may periodically receive network data (X, p, Y) specific to the CF 120 from the RAN 130 and predict optimal configuration parameter p* for the RAN 130 that results in expected KPI value Y* using the AI / ML model M that approximates the network environment T of the RAN 130. If the predicted optimal configuration parameter p* differs from the currently configured parameter pclUT and the expected KPI value Y* is better than the current KPI value Ycurr, the CF 120 may determine that a suboptimal configuration is detected in the RAN 130. In other words, the CF 120 may determine that the predicted configuration parameter p* with the expected KPI value Y* is the optimal configuration for the RAN 130.

[0053] At 412, the CF 120 may send the optimal network configuration including the configuration parameter p* and the KPI value Y* as a suggestion to the NCF 110. For instance, the CF 120 may send the suggested parameter value p* to the NCF 110, or indicate a change amount (p*-pclirr) for the parameter pcutT to the NCF 110.

[0054] Then the NCF 110 may request at 414 the RCCF 122 to evaluate the optimal network configuration suggested by the CF 120. The NCF 110 may forward the optimal network configuration including the configuration parameter p* and the expected KPI value Y* to the RCCF 122 for evaluation at 414.

[0055] In response to the evaluation request, the RCCF 122 may request current network data from the RAN 130 at 416 and receive the network data at 418. In an example embodiment, the RCCF 122 may periodically receive the network data (X, p, Y) from the RAN 130, and the step 416 may be omitted or be performed only once.

[0056] At 420, the RCCF 122 may estimate quality of the optimal configuration by computing a prediction interval r(X, p*) = Yint = [Ymin, Ymax] for the expected KPI value Y*, and also determine an acceptance criterion for the optimal configuration. For example, the RCCF 122 may use the conformal regression model trained in the process 300 to compute the prediction intervals. In some example embodiments, the acceptance criterion may be represented as c(A*, ycurr, y*^ whjch may depend on the suggested parameter change A*=|pcurr - p*|, the expected KPI value Y* and the current KPI value Yclirr. In an example, the acceptance criterion c may be inversely proportional to the suggested parameter change A* and directly proportional to the expected KPI improvement |Ycurr - Y*|. It is believed that significant time-critical improvements in the KPI Y can result from the CF configuration parameter suggestion p* even if the suggested parameter change A* is very small, i.e., the parameter value p* suggested by the CF 120 is not far from the current value pcurr actually configured at the network. If the deterioration in the current KPI value Ycurr is significant, the expected KPI value Y* is significantly better than the current KPI value Ycurr, and the parameter change A* is small, then the NCF 110 may be less stringent in its requirement on the prediction interval r(X, p*), and the determined acceptance criterion c(A*, Yclirr, Y*) may have a relatively large value. Hence the prediction interval even with a relatively large size would be acceptable. On the contrary, if the parameter change A* is large, i.e., the CF 120 suggests a large change to the preferred parameter value pcl,IT, then the NCF 110 may be stringent in its requirement on the prediction interval r(X, p*), and the determined acceptance criterion c(A*, Yclirr, Y*) may have a relatively small value. Hence the prediction interval with a relatively small / tight size would be acceptable. In this way, a dynamic acceptance criterion c may be determined depending on the suggested parameter change A*, the current KPI value Ycurr and the expected KPI value Y*. Compared with a fixed criterion set by the MNO, the determined acceptance criterion c(A*, Ycurr, Y*) can flexibly filter the proposed optimal configurations. For example, the dynamic acceptance criterion c(A*, Ycun', Y*) could be stringent to strictly prevent large and unreliable configuration changes A*, and it could be less stringent to permit small configuration changes A* with significant KPI improvement even if the performance estimate of the proposed configuration is not very high. The dynamic acceptance criterion is beneficial for the network performance improvement especially in the early phase of the models training because it would allow promising small changes even if the NCF 110 does not have much confidence in the models.

[0057] At 422, the RCCF 122 may send an evaluation response message to the NCF 110 including the quality estimate, i.e., the prediction interval r(X, p*), and the acceptance criterion c(A*, Ycurr, Y*) for the optimal configuration. With the prediction interval r(X, p*) and the acceptance criterion c(A*, Ycurr, Y*), the NCF 110 may make at 424 the final decision of accepting or rejecting the optimal configuration suggested by the CF 120. For example, if the prediction interval r(X, p*) has a size less than or equal to the criterion c, the NCF 110 may accept the suggested parameter change A*. If the prediction interval r(X, p*) has a size larger than the criterion c, the NCF 110 may reject the parameter change A*.

[0058] If the suggested parameter change A* is accepted at 424, then the NCF 110 may make the parameter change in the RAN 130 at 426. The NCF 110 may send the optimal configuration directly or via an intermediate entity to the RAN 130.

[0059] The NCF 110 may send an acknowledge message to the CF 120 to confirm receipt of the optimal configuration suggestion at 428. In an example embodiment, the acknowledge message may indicate the decision that the suggested optimal configuration is accepted or rejected. The acknowledge message may also indicate a cause of the decision. For instance, the decision cause may include the prediction interval r(X, p*) and the acceptance criterion c for the optimal configuration.

[0060] The decision and decision cause can reflect whether the CF model M is judged to be good enough. Therefore, the CF 120 may use the information as feedback to improve its model M at 430. For example, the CF 120 may re-train or fine-tune the model M based on the feedback and optionally new network data to improve accuracy and acceptance chance of the optimal configuration predicted by the model.

[0061] Fig 5 illustrates a process 500 for CF performance evaluation according to example embodiments of the present disclosure. The process 500 may be implemented at the NCF 110, the CF 120 and the RAN 130. In the process 500, it is assumed that the NCF 110 has access to network data of the RAN 130, therefore the NCF 110 can perform the CF performance evaluation by itself, and the RCCF 122 is not needed. Some steps in the process 500 may be similar to those in the process 300 and they will be briefly described below.

[0062] Referring to Fig. 5, the CF 120 may receive network data from the RAN 130 at 510, and the NCF 110 may receive network data from the RAN 130 at 512. It is worth noting that the network data available to the CF 120 is specific to the CF 120, e.g. the network data (Xn, pn, Yn) where n represents the index of the CF 120, while the NCF 110 may have access to network data that it can use to evaluate performance of any CF in the system, e.g. the network data (Xi, pi, Yi) where i = 1, ..., N and N represents the number of CFs in the system.

[0063] At 514, the CF 120 may use the network data to train its AI / ML model M that approximate the network environment of the RAN 130. The CF 120 may periodically or continuously receive the network data from the RAN 130 and train its AI / ML model M. As more network data is available, the model M learns the network mapping T better, i.e., it can more accurately approximate the network mapping T.

[0064] When the AI / ML model M is trained to a satisfactory level, the CF 120 may inform at 516 the NCF 110 that the model learning is completed.

[0065] At 518, the NCF 110 may acquire data and / or information for model performance evaluation from the CF 120 through a data exchange procedure, e.g., a request and response procedure. The acquired data and / or information may include but is not limited to the inference output of the model M, i.e., the predicted optimal configuration parameter p* and the expected KPI value Y*.

[0066] The NCF 110 may estimate the performance of the model M using the network data and the inference output of the model M at 520. In an example embodiment, the NCF 110 may use an AI / ML model e.g. the conformal regression model to estimate the performance of the CF model M. The conformal regression model may produce a prediction interval r(X, p*) = Ymt = [Ymin, Ymax] that contains the expected / predicted KPI value Y* with the probability at least a close to 1. The probabilistic guarantee parameter a may be set a priori by the MNO, and a typical value of a is e.g. 0.95. Apparently, the tighter the prediction interval r(X, p*) = Yimis, the more confidence it conveys in Y* and ultimately the suggested parameter p*. Therefore, the prediction interval r(X, p*) may be used to evaluate the quality of the model M. In the learning phase, the NCF 110 may learn the prediction intervals for the CF model M using the network data periodically received from the RAN 130 and the model M inference output periodically received from the NCF 110. In an example embodiment, the process 500 may be continuously performed in an on-line fashion. In addition, it is worth noting that the NCF 110 can produce the prediction intervals before the CF 120 makes any actual configuration suggestion to the NCF 110.

[0067] Fig 6 illustrates a process 600 for network control based on dynamic criterion generation according to example embodiments of the present disclosure. The process 600 may be performed at the NCF 110, the CF 120 and the RAN 130. It is assumed that the NCF 110 has access to the network data of the RAN 130, therefore the NCF 110 can perform the CF performance evaluation and criterion generation by itself, and the RCCF 122 is not needed. Some steps in the process 600 may be similar to those in the process 400 and they will be briefly described below.

[0068] Referring to Fig. 6, at 610, the CF 120 may detect a suboptimal configuration in the RAN 130. For example, if the CF 120 predicts an optimal configuration parameter p* different from the currently configured parameter pcurr and the expected KPI value Y* corresponding to the predicted optimal parameter p* is better than the current KPI value Ycurr, the CF 120 may determine that a suboptimal configuration is detected.

[0069] At 612, the CF 120 may send the optimal network configuration including the optimal configuration parameter p* and the expected KPI value Y* as a suggestion to the NCF 110.

[0070] Upon receiving the optimal configuration suggestion, the NCF 110 may request at 614 current network data from the RAN 130 for evaluating the optimal configuration, and receive the network data at 616. In an example embodiment, the NCF 110 may periodically receive the network data (X, p, Y) from the RAN 130, and the step 614 may be omitted or be performed only once. The NCF 110 may use the latest network data to evaluate the optimal configuration received from the CF 120 and to compute the acceptance criterion c.

[0071] At 618, the NCF 110 may estimate quality of the optimal configuration suggested by the CF 120 by computing a prediction interval r(X, p*) = Yim = [Ymin, Ymax] for the expected KPI value Y*, and also determine an acceptance criterion for the optimal configuration. For example, the NCF 110 may use the conformal regression model trained in the process 500 to compute the prediction interval. In some example embodiments, the acceptance criterion may be represented as c(A*, Ycurr, Y*), which may depend on the suggested parameter change A*=|pcurr - p*|, the expected KPI value Y* and the current KPI value YCUIT In an example, the acceptance criterion c may be inversely proportional to the suggested parameter change A* and directly proportional to the expected KPI improvement |Ycurr- Y*|. If the deterioration in the current KPI value Ycurr is significant, the expected KPI value Y* is significantly better than the current KPI value Ycurr, and the proposed parameter change A* is small, then the NCF 110 may be less stringent in its requirement on the prediction interval r(X, p*), and the determined acceptance criterion c(A*, Yclirr, Y*) may have a relatively large value. Hence the prediction interval even with a relatively large size would be acceptable. On the contrary, if the proposed parameter change A* is large, i.e., the CF 120 suggests a large change to the preferred parameter value pcurr, then the NCF 110 may be stringent in its requirement on the prediction interval r(X, p*), and the determined acceptance criterion c(A*, Ycurr, Y*) may have a relatively small value. Hence the prediction interval with a relatively small / tight size would be acceptable. In this way, a dynamic acceptance criterion c may be determined depending on the suggested parameter change A*, the current KPI value Ycurrand the expected KPI value Y*. Compared with a fixed criterion set a priori by the MNO, the dynamic acceptance criterion c(A*, Ycun, Y*) can flexibly filter the proposed optimal configurations. For example, the dynamic acceptance criterion c(A*, Ycurr, Y*) could be stringent to strictly prevent large and unreliable configuration changes A*, and it could be less stringent to permit small configuration changes A* with significant KPI improvement even if the performance estimate of the proposed configuration is not very high. The dynamic acceptance criterion is also beneficial for the network performance improvement especially in the early phase of the models training because it would allow promising small changes even if the NCF 110 does not have much confidence in the models.

[0072] At 620, the NCF 110 may make a decision of accepting or rejecting the optimal configuration suggested by the CF 120 based on the quality estimate i.e., the prediction interval r(X, p*) and the acceptance criterion c determined at 618. For example, if the prediction interval r(X, p*) has a size less than or equal to the criterion c, which reflects that the NCF 110 has sufficient confidence in the optimal configuration suggested by the CF 120, then the NCF 110 may decide to accept the optimal configuration. If the prediction interval r(X, p*) has a size larger than the criterion c, which reflects that the NCF 110 has insufficient confidence in the optimal configuration, then the NCF 110 may decide to reject the optimal configuration.

[0073] If the suggested optimal configuration is accepted at 620, then the NCF 110 may make the parameter change A* in the RAN 130 at 622. The NCF 110 may send the optimal configuration directly or via an intermediate entity to the RAN 130.

[0074] The NCF 110 may send an acknowledge message to the CF 120 to confirm receipt of the optimal configuration suggestion at 624. In an example embodiment, the acknowledge message may indicate the decision that the suggested optimal configuration is accepted or rejected. The acknowledge message may also indicate a cause of the decision. For instance, the decision cause may include the prediction interval r(X, p*) and the acceptance criterion c computed for the optimal configuration.

[0075] The decision and decision cause can reflect whether the CF model M is judged to be good enough. Therefore, the CF 120 may use the information as feedback to improve its model M at 626. For example, the CF 120 may re-train or fine-tune the model M based on the feedback and optionally new network data to improve accuracy and acceptance chance of the optimal configuration predicted by the model.

[0076] Fig. 7 illustrates a method 700 according to example embodiments of the present disclosure. The method 700 may be implemented at an NCF, e.g. the NCF 110 discussed above.

[0077] Referring to Fig. 7, the method 700 may include a step 710 of receiving an optimal network configuration suggestion from a CF, e.g. the CF 120 discussed above. In an example embodiment, the optimal network configuration suggestion may contain one or more optimal network configuration parameters p* and expected network performance e.g. KPI values Y* when the one or more optimal network configuration parameters p* are applied.

[0078] The method 700 may further include a step 712 of sending an evaluation request for the optimal network configuration suggestion to a RCCF, e.g. the RCCF 122 discussed above. In the step 712, the NCF 110 may forward the optimal network configuration suggestion received from the CF 120 to the RCCF 122.

[0079] The method 700 may further include a step 714 of receiving an evaluation response from the RCCF 122. The evaluation response may contain a quality estimate and an acceptance criterion for the optimal network configuration suggestion. In an example embodiment, the quality estimate of the optimal network configuration suggestion may be represented by a prediction interval r(X, p*) = Yint = [Ymin, Ymax] that contains the expected network performance e.g. KPI values Y* with a predetermined probability a. The acceptance criterion may be represented by a threshold size for the prediction interval. In some example embodiments, the acceptance criterion may be inversely proportional to a difference between the one or more optimal network configuration parameters p* indicated in the optimal network configuration suggestion and corresponding network configuration parameters pcurr currently configured for the network. In some example embodiments, the acceptance criterion may be directly proportional to the expected performance improvement |YCUIT^Y*| when the optimal network configuration suggestion is applied, where Ycurr is the current network performance e.g. KPI value in the network.

[0080] In some example embodiments, the method 700 may further include a step 716 of deciding to accept or reject the optimal network configuration suggestion based on the evaluation response received from the RCCF 122, i.e., based on the quality estimate and the acceptance criterion for the optimal network configuration suggestion, and a step 718 of sending an acknowledge message to the CF 120. The acknowledge message may contain the decision of accepting or rejecting the optimal network configuration suggestion, and optionally a decision cause. For instance, the decision cause may include the quality estimate and the acceptance criterion for the optimal network configuration suggestion. As shown by the dash-line boxes, the steps 716, 718 may be optional.

[0081] Fig 8 illustrates a method 800 according to example embodiments of the present disclosure. The method 800 may be implemented at an NCF, e.g. the NCF 110 discussed above.

[0082] Referring to Fig. 8, the method 800 may include a step 810 of receiving an optimal network configuration suggestion from a CF e.g. the CF 120 discussed above. In an example embodiment, the optimal network configuration suggestion may contain one or more optimal network configuration parameters p* predicted by the CF 120 and expected network performance e.g. KPI values Y* when the one or more optimal network configuration parameters p* are applied.

[0083] The method 800 may further include a step 812 of determining a quality estimate and an acceptance criterion for the optimal network configuration suggestion. In an example embodiment, the quality estimate of the optimal network configuration suggestion may be determined as a prediction interval r(X, p*) = Yint = [Ymin, Ymax] that contains the expected network performance e.g. KPI values Y* with a predetermined probability a The acceptance criterion may be determined as a threshold size for the prediction interval. In some example embodiments, the acceptance criterion may be inversely proportional to a difference between the one or more optimal network configuration parameters p* indicated in the optimal network configuration suggestion and corresponding network configuration parameters pclUT currently configured for the network. In some example embodiments, the acceptance criterion may be directly proportional to the expected performance improvement when the optimal network configuration suggestion is applied, where YCU1T is the current network performance e.g. KPI value in the network.

[0084] In some example embodiments, the NCF 110 may receive network data from the network e.g. the RAN 130 before determining the quality estimate and the acceptance criterion for the optimal network configuration suggestion. Then the NCF 110 may use the received network data to determine the quality estimate and the acceptance criterion for the optimal network configuration suggestion. In some example embodiments, the NCF 110 may send a request for the network data to the RAN 130 before it receives the network data from the RAN 130.

[0085] The method 800 may further include a step 814 of deciding to accept or reject the optimal network configuration suggestion based on the determined quality estimate and acceptance criterion. For instance, the NCF 110 may decide to accept the optimal network configuration suggestion if the quality estimate of the optimal network configuration suggestion satisfies the acceptance criterion. Otherwise, the NCF 110 may decide to reject the optimal network configuration suggestion.

[0086] In some example embodiments, the method 800 may further include a step 816 of sending an acknowledge message to the CF 120. The acknowledge message may contain the decision of accepting or rejecting the optimal network configuration suggestion, and optionally a decision cause. For instance, the decision cause may include the quality estimate and the acceptance criterion for the optimal network configuration suggestion. As shown by the dashline box, the step 816 may be optional.

[0087] Fig 9 illustrates a method 900 according to example embodiments of the present disclosure. The method 900 may be implemented at a CF, e.g. the CF 120 discussed above.

[0088] Referring to Fig. 9, the method 900 may include a step 910 of sending an optimal network configuration suggestion to an NCF e.g. the NCF 110 discussed above. As discussed above, the optimal network configuration suggestion may contain one or more optimal network configuration parameters p* precited by the CF 120 and expected network performance e.g. KPI values Y* when the one or more optimal network configuration parameters p* are applied.

[0089] The method 900 may further include a step 912 of receiving an acknowledge message from the NCF 110. In some example embodiments, the acknowledge message may contain a decision of accepting or rejecting the optimal network configuration suggestion, and optionally a decision cause. The decision cause may include a quality estimate and an acceptance criterion for the optimal network configuration suggestion. In some example embodiments, the quality estimate of the optimal network configuration suggestion may be represented by a prediction interval r(X, p*) = Yim = [Ymin, Ymax] that contains the expected network performance e.g. KPI values Y* with a predetermined probability u. The acceptance criterion may be represented by a threshold size for the prediction interval. In some example embodiments, the acceptance criterion may be inversely proportional to a difference between the one or more optimal network configuration parameters p* indicated in the optimal network configuration suggestion and corresponding network configuration parameters pCUIT currently configured for the network. In some example embodiments, the acceptance criterion may be directly proportional to the expected performance improvement |Ycurr-Y*| when the optimal network configuration suggestion is applied, where Ycurr is the current network performance e.g. KPI value in the network.

[0090] In some example embodiments, the method 900 may further include a step 914 of using the decision and the decision cause as feedback information to improve an algorithm or model that generates the optimal network configuration suggestion. For instance, the CF may re-train or fine-tune the algorithm or model using the feedback information and optionally additional network data. As shown by the dash-line box, the step 914 is optional.

[0091] Fig 10 illustrates a method 1000 according to example embodiments of the present disclosure. The method 1000 may be implemented at a RCCF, e.g. the RCCF 122 discussed above.

[0092] Referring to Fig. 10, the method 1000 may include a step 1010 of receiving an evaluation request for an optimal network configuration suggestion from an NCF, e.g. the NCF 110 discussed above. The request message may contain the optimal network configuration suggestion. In some example embodiments, the optimal network configuration suggestion may contain one or more optimal network configuration parameters p* and expected network performance e.g. KPI values Y* when the one or more optimal network configuration parameters p* are applied.

[0093] The method 1000 may further include a step 1012 of determining a quality estimate and an acceptance criterion for the optimal network configuration suggestion. In an example embodiment, the quality estimate of the optimal network configuration suggestion may be determined as a prediction interval r(X, p*) = Yint = [Ymin, Ymax] that contains the expected network performance e.g. KPI values Y* with a predetermined probability a The acceptance criterion may be determined as a threshold size for the prediction interval. In some example embodiments, the acceptance criterion may be inversely proportional to a difference between the one or more optimal network configuration parameters p* indicated in the optimal network configuration suggestion and corresponding network configuration parameters pclUT currently configured for the network. In some example embodiments, the acceptance criterion may be directly proportional to the expected performance improvement when the optimal network configuration suggestion is applied, where Ycurr is the current network performance e.g. KPI value in the network.

[0094] In some example embodiments, the RCCF 122 may receive network data from the network e.g. the RAN 130 before determining the quality estimate and the acceptance criterion for the optimal network configuration suggestion. Then the RCCF 122 may use the received network data to determine the quality estimate and the acceptance criterion for the optimal network configuration suggestion. In some example embodiments, the RCCF 122 may send a request for the network data to the RAN 130 before it receives the network data from the RAN 130.

[0095] The method 1000 may further include a step 1014 of sending an evaluation response to the NCF 110. The evaluation response may contain the quality estimate and the acceptance criterion for the optimal network configuration suggestion.

[0096] The various example embodiments discussed above produce many beneficial effects. For example, small changes A* may be permitted by the NCF by relaxing the acceptance criterion c(A*, Yclirr, Y*) if the performance improvement |Y* - Yeurr| is significant. It ensures that in the early phase, the CF is allowed to make small changes to the reliable parameter pcurr. Furthermore, the same applies to the case if, for certain values of X and p, it is difficult for the CF model M to accurately approximate the network mapping T that depends locally on X and p. On the other hand, the CF is also allowed to make large parameter changes A* if the prediction interval r(X, p*) satisfies a stringent criterion, i.e., the NCF only allows large changes to its reliable default parameter pcurr if it has considerable confidence / trust in the CF. It ensures that large parameter changes A* are not permitted in the early phase. But, as both CF and NCF get better at their tasks with time, the NCF can confidently allow the CF to make large parameter changes. In addition, the CF can improve its model M and therefore its configuration suggestion p* over time by considering the decisions made by the NCF, the predictions intervals r, and the criteria c. Such information should help the CF to improve its performance (better accuracy) or chances of acceptance (taking into account the criteria c provided by the NCF). The CF can improve acceptance of its suggestions by making the suggested configuration p* satisfy (i.e., less or smaller than) the criteria provided by the NCF or the RRCF

[0097] Fig. 11 illustrates an apparatus 1100 according to example embodiments of the present disclosure. The apparatus 1100 may be implemented to comprise or to form at least a part of an NCF such as the NCF 110 discussed above to perform at least a part of operations related to the NCF 110.

[0098] Referring to Fig. 11, the apparatus 1100 may comprise a first means 1110, a second means 1112 and a third means 1114. The first means 1110 may be adapted to perform the step 710 of the method 700 shown in Fig. 7, i.e. receiving an optimal network configuration suggestion from a CF. The second means 1112 may be adapted to perform the step 712 of the method 700, i.e. sending an evaluation request for the optimal network configuration suggestion to a RCCF. The third means 1114 may be adapted to perform the step 714 of the method 700, i.e. receiving an evaluation response from the RCCF containing a quality estimate and an acceptance criterion for the optimal network configuration suggestion.

[0099] The apparatus 1100 may optionally comprise a fourth means 1116 and a fifth means 1118. The fourth means 1116 may be adapted to perform the step 716 of the method 700, i.e. deciding to accept or reject the optimal network configuration suggestion based on the evaluation response. The fifth means 1118 may be adapted to perform the step 718 of the method 700, i.e. sending an acknowledge message to the CF containing the decision of accepting or rejecting the optimal network configuration suggestion and optionally a decision cause.

[00100] Since the steps of the method 700 have been described in detail with reference to Fig. 7, the blocks in the apparatus 1100 are briefly described here for simplicity of the description.

[00101] Fig 12 illustrates an apparatus 1200 according to example embodiments of the present disclosure. The apparatus 1200 may be implemented to comprise or to form at least a part of an NCF such as the NCF 110 discussed above to perform at least a part of operations related to the NCF 110.

[00102] Referring to Fig. 12, the apparatus 1200 may comprise a first means 1210, a second means 1212 and a third means 1214. The first means 1210 may be adapted to perform the step 810 of the method 800 shown in Fig. 8, i.e. receiving an optimal network configuration suggestion from a CF. The second means 1212 may be adapted to perform the step 812 of the method 800, i.e. determining a quality estimate and an acceptance criterion for the optimal network configuration suggestion. The third means 1214 may be adapted to perform the step 814 of the method 800, i.e. deciding to accept or reject the optimal network configuration suggestion based on the determined quality estimate and acceptance criterion.

[00103] The apparatus 1200 may optionally comprise a fourth means 1216. The fourth means 1216 may be adapted to perform the step 816 of the method 800, i.e. sending an acknowledge message to the CF containing the decision of accepting or rejecting the optimal network configuration suggestion and optionally a decision cause.

[00104] In some example embodiments, the apparatus 1200 may further comprise a means (not shown) for requesting network data from the network, and a means (not shown) for receiving the requested network data from the network. The second means 1212 may be adapted to use the received network data to determine the quality estimate and the acceptance criterion for the optimal network configuration suggestion.

[00105] Since the steps of the method 800 have been described in detail with reference to Fig. 8, the blocks in the apparatus 1200 are briefly described here for simplicity of the description.

[00106] Fig 13 illustrates an apparatus 1300 according to example embodiments of the present disclosure. The apparatus 1300 may be implemented to comprise or to form at least a part of a CF such as the CF 120 discussed above to perform at least a part of operations related to the CF 120.

[00107] Referringto Fig. 13, the apparatus 1300 may comprise a first means 1310 and a second means 1312. The first means 1310 may be adapted to perform the step 910 of the method 900 shown in Fig. 9, i.e. sending an optimal network configuration suggestion to an NCF. The second means 1312 may be adapted to perform the step 912 of the method 900, i.e. receiving an acknowledge message from the NCF containing a decision of accepting or rejecting the optimal network configuration suggestion and optionally a decision cause.

[00108] In some example embodiments, the apparatus 1300 may optionally comprise a third means 1314. The third means 1314 may be adapted to perform the step 916 of the method 900, i.e. using the decision and decision cause as feedback information to improve an algorithm or model that generates the optimal network configuration suggestion.

[00109] Since the steps of the method 900 have been described in detail with reference to Fig. 9, the blocks in the apparatus 1300 are briefly described here for simplicity of the description.

[00110] Fig 14 i illustrates an apparatus 1400 according to example embodiments of the present disclosure. The apparatus 1400 may be implemented to comprise or to form at least a part of a RCCF such as the RCCF 122 discussed above to perform at least a part of operations related to the RCCF 122.

[00111] Referring to Fig. 14, the apparatus 1400 may comprise a first means 1410, a second means 1412, and a third means 1414. The first means 1410 may be adapted to perform the step 1010 of the method 1000 shown in Fig. 10, i.e. receiving an evaluation request for an optimal network configuration suggestion from an NCF. The second means 1412 may be adapted to perform the step 1012 of the method 1000, i.e. determining a quality estimate and an acceptance criterion for the optimal network configuration suggestion. The third means 1414 may be adapted to perform the step 1014 of the method 1000, i.e. sending an evaluation response to the NCF containing the quality estimate and the acceptance criterion for the optimal network configuration suggestion.

[00112] In some example embodiments, the apparatus 1400 may further comprise a means (not shown) for requesting network data from the network, and a means (not shown) for receiving the requested network data from the network. The second means 1412 may be adapted to use the received network data to determine the quality estimate and the acceptance criterion for the optimal network configuration suggestion.

[00113] Since the steps of the method 1000 have been described in detail with reference to Fig. 10, the blocks in the apparatus 1400 are briefly described here for simplicity of the description.

[00114] Fig 15 is a schematic block diagram illustrating devices in a network control system 1500 according to example embodiments of the present disclosure. As shown in Fig. 15, the network control system 1500 may include a network control unit 1510 and one or more cognitive units 1520.

[00115] The network control unit 1510 may comprise one or more processors 1511, one or more memories 1512 and one or more network interfaces 1514 interconnected through one or more buses 1515. The one or more network interfaces 1514 may provide wired or wireless communication links through which the network control unit 1510 may communicate with other devices, entities, elements or functions. For example, the network control unit 1510 may communicate with the one or more cognitive units 1520 and with a network e.g. a wireless / mobile / cellular communication network (not shown). The one or more memories 1512 may include instructions 1513 which, when executed by the one or more processors 1511, may cause the network control unit 1510 to perform operations and procedures relating to the NCF 110 as described above.

[00116] The one or more cognitive units 1520 each may comprise one or more processors 1521, one or more memories 1522 and one or more network interfaces 1524 interconnected through one or more buses 1525. The one or more network interfaces 1524 may provide wired or wireless communication links through which the cognitive unit 1520 may communicate with other devices, entities, elements or functions. For example, the cognitive unit 1520 may communicate with the network control unit 1510 and with a network e.g. a wireless / mobile / cellular communication network (not shown). The one or more memories 1522 may include instructions 1523 which, when executed by the one or more processors 1521, may cause the cognitive units 1520 to perform operations and procedures relating to the CF 120 as described above, or to perform operations and procedures relating to the RCCF 122 as described above.

[00117] The one or more processors 1511,1521 discussed above may be of any appropriate type that is suitable for the local technical network, and may include one or more of general purpose processors, special purpose processor, microprocessors, a digital signal processor (DSP), one or more processors in a processor based multi-core processor architecture, as well as dedicated processors such as those developed based on Field Programmable Gate Array (FPGA) and Application Specific Integrated Circuit (ASIC). The one or more processors 1511, 1521 may be configured to control other elements of the network control unit 1510 and the cognitive units 1520 and operate in cooperation with them to implement the procedures discussed above.

[00118] The one or more memories 1512, 1522 may include at least one storage medium in various forms, such as a transitory memory and / or a non-transitory memory. The transitory memory may include, but not limited to, for example, a random access memory (RAM) or a cache. The non-transitory memory may include, but not limited to, for example, a read only memory (ROM), a hard disk, a flash memory, and the like. The term “non-transitory,” as used herein, is a limitation of the medium itself (i. e., tangible, not a signal) as opposed to a limitation on data storage persistency (e.g., RAM vs. ROM). Further, the one or more memories 1512, 1522 may include but not limited to an electric, a magnetic, an optical, an electromagnetic, an infrared, or a semiconductor system, apparatus, or device or any combination of the above.

[00119] Some example embodiments of the present disclosure may contribute to the 3rd Generation Partnership Project (3GPP) Technical Report (TR) 28.867 which captures study on closed control loops (CCLs). The contribution may be as follows. «<Contribution Start»» 4 Concepts and background 4.X Closed Control Loop Coordination 4.X. 1 CCL coordination scenarios Multiple CCLs could co-exist and concurrently act within the same environment. The CCLs can affect one another, in the worst cases leading to conflicts requiring the CCLs to be coordinated. The coordination of Closed Control Loops includes the services needed to detect, resolve, or avoid conflicts among goals, processes, control subspaces or action of the CCL. The control subspaces of a CCL are the set of managed entities and controlled parameters on those managed entities for which a CCL instance takes responsibility. Actions of the CCL are changes that a CCL can perform over a managed entity such as configuring an attribute. To address different situations and coordination needs, coordination capabilities could be required for the following scenarios: • / / ... skip ... / / • Capability to develop / establish trust in the recommendations made by CCLs before they are allowed to have directly control over the network • / / ... skip ... H / / / ... skip ...II 5 Use Cases I I... skip ... / / 5.X2 Use case X2: CCL performance Trust development 5.X2.1 Description After a CCL is instantiated into the system, e.g., when composed at runtime from multiple management functions, it is possible that the performance of the CCL at all the applicable actiondecision points is not as expected, e.g., where large actions are involved. The CCL needs to make a decision regarding its action and execute it through a controller functions (say the CCL coordinator). The coordinator must ensure to minimize the negative impact on the network and protect against a malfunctioning CCL. Therefore, before the CCL may be allowed to have direct control over the network, the coordinator needs to develop “trust” in the decision regarding network actions (e.g., parameter changes) made by the CCL. The 3GPP management system should have a capability enabling the coordinator to develop / establish trust in the recommendations made by CCL. The CCL should provide its planned action-decision to the coordinator and the coordinator should provide a report to the CCL indicating how trustworthy (and good for the network) the planned action-decision are and, in the cases, where the decisions are untrustworthy, the report should indicate the "trust" criteria which are not fulfilled. 5.X2.2 Potential Requirements REQ-CCL-TRIG-COORD-1: The 3GPP management system should support a capability to inform the CCL (e.g. via a decision trust report) of whether the action-decision recommendation by the CCL is acceptable or not. REQ-CCL-TRIG-COORD-1: The 3GPP management system should support a capability to inform the CCL in the case of an untrustworthy / bad recommendation by the CCL of the criteria on why its decisions are considered untrustworthy / bad, e.g. how far the CCL prediction on impact / effect on the network metrics is likely to be from the true value and what is the maximum change in the current network that is allowed. REQ-CCL-TRIG-COORD-1: The 3GPP management system should support a capability in the case that the CCL has consistently made good decisions and achieved ultimate trust to inform the CCL that decisions are good to fulfil the required trust level. 5.X2.3 Potential Solutions introduce a request for recommendation execution or evaluation (say called CCLRecommendation) which may be sent by the CCL to the coordinator. • The request should include information on the desired change as well as the information on the predicted impact / effect of the decision on the related metrics. introduce a decision trust report to inform the CCL of whether the decision is acceptable or not. • It is assumed that after the CCL makes a recommendation to Coordinator, including its predicted impact / effect on the metrics, the coordinator evaluates the recommendation - e.g. based on its internal model or past execution of action and their impact. • if the decision is good, Coordinator responds to the CCL (via the decision trust report) that action-decision recommendation is acceptable, allowing the CCL to reuse that recommendation. Otherwise, the coordinator informs the CCL (via the decision trust report) that the decision is unacceptable, the response including criteria on why the decision is bad / untrustworthy, e.g. how far the CCL prediction on impact / effect on the network metrics is likely to be from the true value and what is the maximum change (in the current network parameters) that is allowed. • Subsequent, CCL updates its model and makes a smaller action, e.g. a smaller change in the parameter. Include in the decision trust report an indication for when the CCL has consistently made good decisions and achieved ultimate trust. • It is assumed that based on feedback on the quality of its decisions, the CCL updates it decision-making engine and repeats the decision evaluation process. Then if CCL has consistently made good large action-decisions, the coordinator can consider the CCL as trusted to make such large decisions. The coordinator informs the CCL (via the decision trust report) that the CCL has consistently made good decisions and achieved its ultimate trust. The report may include an indication for how the CCL may behave thereafter - e.g. that the CCLs decisions will go without checking via the coordinator or that the CCL may directly execute its decisions on to the network. «<Contrib END»»

[00120] Some example embodiments of the present disclosure may contribute to the 3GPP Technical Specification (TS) 28.536 which specifies management services for communication service assurance. The contribution may be as follows. «<Contribution Start»» 4 Communication service assurance service 4.1.2.3 Class definitions 4.1.2.3.1 AssuranceClosedControlLoop 4.1.2.3.1.1 Definition This class represents the information for controlling and monitoring an assurance closed control loop associated with a Networkslice or NetworkSliceSubnet. It can be name-contained by SubNetwork or ManagedElement. To express the assurance closed control loop goals, the MnS consumer needs to request the MnS producer to create an AssuranceClosedControlLoop on the MnS producer. The MnS producer may trigger to create the AssuranceClosedControlLoop as well, for example, when an instance of Networkslice or NetworkSliceSubnet is created, MnS producer may create an instance of AssuranceClosedControlLoop associated to the instance of Networkslice or NetworkSliceSubnet to assure the target described in ServiceProf ile or SliceProf ile. For the deletion of the assurance closed control loop, the MnS consumer needs to request the MnS producer to delete the AssuranceClosedControlLoop to free up resources on the MnS producer. MnS producer also can trigger to delete AssuranceClosedControlLoop to free up resources by itself. For temporary deactivation of the assurance closed control loop, the MnS consumer can modify the value of the administrative state attribute to “LOCKED”. The MnS producer may disable the assurance closed control loop, for example in conflict situations, by setting the operational state attribute to “disabled”. When a closed control loop is enabled by the MnS producer, the operational state is set again to “enabled”. For the activation of an assurance closed control loop, the MnS consumer can modify the value of the administrative state attribute to “UNLOCKED”. An AssuranceClosedControlLoop can name-contain multiple instances of AssuranceGoal which represents the assurance goal and corresponding observed or predicted goal fulfilment information (see clause 4.1.2.3.2). The AssuranceGoal may optionally include an assurance scope in terms of location (see clause 4.1.2.3.2). The attribute “controlLoopLifeCyclePhase” is used to keep track of the lifecycle of an AssuranceClosedControlLoop. The attribute aCCLDisallowedList is used to descope the ACCL. See clause 6.1.6 of TS 28.535

[17] , Each entry in the list indicates a specific list of attributes belonging to a managedEntity identified by the managedEntityldentifier which the ACCL is not allowed to modify. 4.1.2.3.1.2 Attributes The AssuranceClosedControlLoop IOC includes attributes inherited from Top IOC (defined TS 28.622(5]) and the following attributes: Attribute name s isReadable is\\ ritable islnsariant isNotihablc operationalState M T F F T administrativeState M T T F T controlLoopLifeCyclePhase M T T F T aCCLDisallowedList 0 T T F T Attributes related to role networkSliceRef CM T T F T networkSliceSubnetRef CM T T F T decisionTrustReport M T T F T 4.1.2.3.1.3 Constraints Name Definition networkSliceSubnetRef Condition: the AssuranceGoal applies to aNetworkSliceSubNet networkSliceRef Condition: the AssuranceGoal applies to a Networkslice 4.1.2.3.1.4 Notifications The common notifications defined in clause 4.1.2.5 are valid for this IOC, without exceptions or additions. -------CCL coordinator start 4.1.2.3.Y CCL Coordinator «IOC» 4.1.2.3.Y.1 Definition This IOC represents the properties of a CCL coordinator. 4.1.2.3.Y.2 Attributes The CCL coordinator IOC includes the following attributes: Attribute name Support Qualifier isReadable isWritable islmariant isNotifyablc Attributes related to role cCLRecommendation M T T F T ---------CCL Recommendation Start X.X.l CCLRecommendation « dataType» X.X.1.1 Definition This «IOC» represents the properties of a recommendation regarding an action-decision (e.g., change of network parameter) by the CCL. It contains the session ID, the ID of the network managed object, parameter values of this object, and predicted performance effect / metrics as the result of the action in the network object. X.X.1.2 Attributes The cCLRecommendation «dataType» includes the following attributes: Attribute name Support Qualifier isReadable is Writable islnvariant isNotifyable Sessionld M T F T T ManagedObj ectID M T F T T MetricValuePredictions M T T F T Parametervalues M T T F T X.X.X.l MetricVauePredictions « dataType» X.X.X.1.1 Definition This «dataType» represents the predictions in terms of performance metrics of the managed entity that the CCL does performance assurance for. The CCL produces these values by using its AI / ML model representing a mapping between managed object state (including its parameters) and the performance metrics. X.X.X.l.2 Attributes The MetricVauePredictions «dataType» includes the following attributes: Attribute name Support Qualifier isReadable isWritable islnvariant isNotifyable NamesList M T F T T predictedValuesList M T F T T X.X.X.l Parametervalues « dataType» X.X.X.2.1 Definition This «dataType» represents the parameters and their values that the CCL recommends to be configured in the managed object for performance assurance X.X.X.2.2 Attributes The ParameterValues «dataType» includes the following attributes: Attribute name Support Qualifier isReadable isWritable islnvariant isNotifyable NamesList M T F T T ParameterValuesList M T F T T ---------CCL Recommendation End 4.1.2.3.Y.3 Attribute constraints None 4.1.2.3.Y.4 Notifications The common notifications defined in clause 4.1.2.5 are valid for this data type, without exceptions or additions. CCL coordinator end — Decision Trust Report 4.1.2.3.Z DecisionTrustReport« dataType» 4.1.2.3.Z.1 Definition This «datatype» represents the properties of the report on the evaluation carried out by the coordinator. It contains an evaluation regarding whether the CCL is allowed to take action on the 5 managed object (directly or indirectly through the coordinator) or not, the criteria which was used to make the evaluation. The default value is the current value of the parameters that the CCL asked to change, and maximum allowed changed is the maximum change allowed for this CCL (based on its trust performance). Furthermore, it also contains prediction error intervals on the predictions provided by the CCL. 10 4.1.2.3.Z.2 Attributes The decisionTrustReport «datatype» includes the following attributes: Attribute name Support Qualifier isReadable isW ritable isinvariant isNotihablc Sessionld M T F T T ManagedObj ectID M T T T T evaluation M T T F T > DirectActionAllowed M T T F T > ActionAccepted M T T F T criteria M T T F T > defaultvalues M T T F T > MaxChangesAllowed M T T F T predict!onErrorIntervals M T T F T X.Z.l Evaluation « dataType» X.Z.1.1 Definition 15 This «dataType» represents the evaluation and decision taken by the coordinator after getting CCL recommendations. Direction action allowed, if true, allows direct access for the CCL to managed object parameter. Action accepted, if true, means that the coordinator accepts the CCL recommendation regarding its proposed / planned actions on the managed object. X.Z.1.2 Attributes 20 The Decisions « dataType» includes the following attributes: Attribute name Support Qualifier is Readable isWritable isinvariant isNntifyable DirectActionAllowed M T F T T ActionAccepted M T F T T X.Z.l Critera « dataType» X.Z.1.1 Definition This «dataType» represents the criteria that the coordinator uses to accept or reject CCL 25 recommendation X.Z.1.2 Attributes The Cri tera « dataType» includes the following attributes: Attribute name Support Qualifier isReadable isWritable islnvariant isNotifyable defaultvalues M T F T T MaxChangesAllowed M T F T T -------decision report end 4.1.2.4 Attribute definitions 5 4.1.2.4.1 Attribute properties The following table defines the properties of attributes that are specified in the present document. Attribute Name Documentation and Allowed Values Properties «skip» «skip» «skip» cCLRecommendation It indicates the planned action-decision of the CCL to the CCL coordinator. It consists of session ID, managed object ID, predicted affect of CCL actions if they are allowed by the coordinator and the actual changes to the managed object parameters, the CCL is recommending type: c CLRecommend ation multiplicity: 1 isOrdered: N / A isUnique: N / A defaultValue: NA isNullable: False Sessionld The Sessionld is used to associate a recommendation to the response from the coordinator. Allowed values: unique integers type: Integer multiplicity: 1 isOrdered: N / A isUnique: N / A defaultValue: None isNullable: False ManagedObj etID The ID of the network node / resource that is the subject / target of CCL performance assurance type: Integer multiplicity: 1 isOrdered: N / A isUnique: N / AdefaultV alue: None isNullable: False MetricValuePredic tions The predicted values of a list of performance assurance KPIs by the CCL as a result of the cCLRecommendation. Allowed Values: List of (Name, Value) tuples. type: MelricValuePredictions multiplicity: 1 isOrdered: N / A isUnique: N / AdefaultV alue: None isNullable: False ParameterValues This «dataType» represents the parameters and their values that the CCL recommends to be configured in the managed object for performance assurance type: ParameterValues multiplicity: 1 isOrdered: N / A isUnique: N / AdefaultV alue: None isNullable: False NamesList List of names of metric or parameters type: String multiplicity” I isOrdered: N / A isUnique: N / AdefaultV alue: None isNullable: False predictedValuesLi st List of predicted metric values by the CCL type: Real multiplicity” 1 isOrdered: N / A isUnique: N / AdefaultV alue: None isNullable: False parameterValuesLi st List of recommended pramater values type: Real multiplicity: 1 isOrdered: N / A isUnique: N / AdefaultV alue: None isNullable: False evaluation List of evaluations for action-decision recommendations provided by the CCL in th CCL recommedation type: Evaluation multiplicity: 1 isOrdered: N / A isUnique: N / AdefaultV alue: None isNullable: False DirectActionAllow ed Whether a direct access is granted to the CCL by the coordinator Allowed values: true or false type: String multiplicity” 1 isOrdered: N / A isUnique: N / AdefaultV alue: False isNullable: False ActionAccepted Whether the action-decisions in the (parameter value list) are accepted or rejected by the coordinator Allowed values: accepted, or rejected type: String multiplicity" 1 isOrdered: N / A isUnique: N / AdefaultV alue: rejected isNullable: False criteria The criteria the coordinator uses (parameter and metricwise) to set actionacceptedIE defaultvalues The default values of the parameters in the CCL recommendation provided by the CCL type: Real multiplicity” 1 isOrdered: N / A isUnique: N / AdefaultV alue:NA isNullable: False MaxChangesAllowed The maximum allowed changes for parameters in the CCL recommendation provided by the CCL type: Real multiplicity: 1 isOrdered: N / A isUnique: N / AdefaultV alue:NA isNullable: False predictionErrorln tervals The error in predictions for metric values in CCL recommendation. The error is calculated as the positive-distance between the metricValuePredictions and true values of the associated metrics. The true values could be direct observation by coordinator on the managed object or its estimations of the values. type: Real multiplicity: 1 isOrdered: N / A isUnique: N / AdefaultV alue:NA isNullable: False NOTE 1: Void NOTE 2: Void «<Contribution END»»

[00121] It would be understood that blocks in the drawings may be implemented in various manners, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more blocks may be implemented using software and / or firmware, for example, machine-executable instructions stored in the storage medium. In addition to or instead of machine-executable instructions, parts or all of the blocks in the drawings may be implemented, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application-Specific Standard Products (ASSPs), System-on-Chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[00122] Some exemplary embodiments further provide program instructions which, when executed by one or more processors, may cause a device or apparatus to perform the procedures described above. The program instructions for carrying out procedures of the exemplary embodiments may be written in any combination of one or more programming languages. The program instructions may be provided to one or more processors or controllers of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program instructions, when executed by the processor or controller, cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program instructions may execute entirely on a machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

[00123] Some exemplary embodiments further provide a computer program product or a computer readable medium having the program instructions stored therein. The computer readable medium may be any tangible medium that may contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine readable medium may be a machine readable signal medium or a machine readable storage medium. A machine readable medium may include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine readable storage medium would include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[00124] As used herein, “at least one of the following: ” and “at least one of ” and similar wording, where the list of two or more elements are joined by “and” or “or”, mean at least any one of the elements, or at least any two or more of the elements, or at least all the elements.

[00125] Throughout the present disclosure, reference to “one embodiment,” “an embodiment,” “some embodiments,” “other embodiments,” etc. indicates that one or more particular features, structures, steps, concepts, and / or characteristics in accordance with principles of the present disclosure may be included in connection with the embodiment. However, such references do not necessarily mean that all embodiments include the particular features, structures, steps, concepts, and / or characteristics, or that an embodiment includes all features, structures, steps, concepts, and / or characteristics. Some embodiments may include one or more such features, structures, steps, concepts, and / or characteristics, in various combinations thereof. It should be understood that one or more of the features, structures, steps, concepts, and / or characteristics described with reference to one embodiment can be combined with one or more of the features, structures, steps, concepts, and / or characteristics of any of the other embodiments provided herein. That is, any of the features, structures, steps, concepts, and / or characteristics described herein can be mixed and matched to create hybrid embodiments, and such hybrid embodiments are within the scope of the present disclosure. Moreover, references to “one embodiment,” “an embodiment,” “some embodiments,” “other embodiments,” etc. in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments necessarily mutually exclusive of other embodiments. It should further be understood that various features, structures, steps, concepts, and / or characteristics of disclosed embodiments are independent of and separate from one another, and may be used or present individually or in various combinations with one another to create alternative embodiments which are considered part of the present disclosure. Therefore, the present disclosure is not limited to only the embodiments specifically described herein, as it would be too cumbersome to describe all of the numerous possible combinations and subcombinations of features, structures, steps, concepts, and / or characteristics, and the examples of embodiments disclosed herein are not intended as limiting the broader aspects of the present disclosure.

[00126] Further, while operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are contained in the above discussions, these should not be construed as limitations on the scope of the present disclosure, but rather as descriptions of features that may be specific to particular embodiments. Certain features that are described in 5 the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable sub-combination.

[00127] Al though the subject matter has been described in a language that is specific to structural 10 features and / or method actions, it is to be understood the subject matter defined in the appended claims is not limited to the specific features or actions described above. On the contrary, the above-described specific features and actions are disclosed as an example of implementing the claims. 15

Claims

1. An apparatus comprising:at least one processor; andat least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to perform:receiving an optimal network configuration suggestion from a cognitive function;sending an evaluation request for the optimal network configuration suggestion to a reliability check cognitive function; andreceiving an evaluation response from the reliability check cognitive function, the evaluation response containing a quality estimate and an acceptance criterion for the optimal network configuration suggestion.

2. The apparatus of claim 1, wherein the instructions, when executed by the at least one processor, further cause the apparatus at least to perform:deciding to accept or reject the optimal network configuration suggestion based on the evaluation response; andsending an acknowledge message to the cognitive function, the acknowledge message containing at least one of the following:the decision of accepting or rejecting the optimal network configuration suggestion; or a decision cause.

3. The apparatus of claim 1 or 2, wherein the optimal network configuration suggestion contains at least one of the following:one or more optimal network configuration parameters suggested by the cognitive function;orexpected network performance corresponding to the one or more optimal network configuration parameters.

4. The apparatus of any of claims 1-3, wherein the quality estimate of the optimal networkconfiguration suggestion is represented by a prediction interval for network performance with a predetermined probability in case the optimal network configuration suggestion is applied.

5. The apparatus of any of claims 1-4, wherein the acceptance criterion for the optimal network configuration suggestion is represented by a threshold size for the prediction interval.

6. An apparatus comprising:at least one processor; andat least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to perform:sending an optimal network configuration suggestion to a network control function;andreceiving an acknowledge message from the network control function, the acknowledge message containing at least one of the following:a decision of accepting or rejecting the optimal network configuration suggestion;ora decision cause.

7. The apparatus of claim 6, wherein the instructions, when executed by the at least one processor, further cause the apparatus at least to perform:using the decision and decision cause as feedback information to improve an algorithm or model that generates the optimal network configuration suggestion.

8. An apparatus comprising:at least one processor; andat least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to perform:receiving an evaluation request for an optimal network configuration suggestion from a network control function;determining a quality estimate and an acceptance criterion for the optimal networkconfiguration suggestion; andsending an evaluation response to the network control function, the evaluation response containing the quality estimate and the acceptance criterion for the optimal network configuration suggestion.

9. The apparatus of claim 8, wherein the acceptance criterion for the optimal network configuration suggestion is determined inversely proportional to a difference between one or more optimal network configuration parameters indicated in the optimal network configuration suggestion and corresponding network configuration parameters currently configured for the network.

10. The apparatus of claim 8 or 9, wherein the acceptance criterion for the optimal network configuration suggestion is determined directly proportional to performance improvement expected in case the optimal network configuration suggestion is applied.

11. The apparatus of any of claims 8-10, wherein the instructions, when executed by the at least one processor, further cause the apparatus at least to perform:receiving network data from the network; andusing the received network data to determine the quality estimate and the acceptance criterion for the optimal network configuration suggestion.

12. The apparatus of any of claims 8-11, wherein the instructions, when executed by the at least one processor, further cause the apparatus at least to perform:requesting for the network data from the network.

13. An apparatus comprising:at least one processor; andat least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to perform:receiving an optimal network configuration suggestion from a cognitive function;determining a quality estimate and an acceptance criterion for the optimal network configuration suggestion; anddeciding to accept or reject the optimal network configuration suggestion based on the determined quality estimate and acceptance criterion.

14. The apparatus of claim 13, wherein the instructions, when executed by the at least one processor, further cause the apparatus at least to perform:receiving network data from the network; andusing the received network data to determine the quality estimate and the acceptance criterion for the optimal network configuration suggestion.

15. The apparatus of claim 13 or 14, wherein the instructions, when executed by the at least one processor, further cause the apparatus at least to perform:requesting for the network data from the network.

16. The apparatus of any of claims 13-15, wherein the instructions, when executed by the at least one processor, further cause the apparatus at least to perform:sending an acknowledge message to the cognitive function, the acknowledge message containing at least one of the following:the decision of accepting or rejecting the optimal network configuration suggestion; or a decision cause.

17. The apparatus of any of claims 13-16, wherein the acceptance criterion for the optimal network configuration suggestion is determined inversely proportional to a difference between one or more optimal network configuration parameters indicated in the optimal network configuration suggestion and corresponding network configuration parameters currently configured for the network and directly proportional to performance improvement expected in case the optimal network configuration suggestion is applied.

18. A method comprising:receiving an optimal network configuration suggestion from a cognitive function;sending an evaluation request for the optimal network configuration suggestion to a reliability check cognitive function; andreceiving an evaluation response from the reliability check cognitive function, the evaluation response containing a quality estimate and an acceptance criterion for the optimal network configuration suggestion.

19. The method of claim 18, further comprising:deciding to accept or reject the optimal network configuration suggestion based on the evaluation response; andsending an acknowledge message to the cognitive function, the acknowledge message containing at least one of the following:the decision of accepting or rejecting the optimal network configuration suggestion; or a decision cause.

20. The method of claim 18 or 19, wherein the optimal network configuration suggestion contains at least one of the following:one or more optimal network configuration parameters suggested by the cognitive function; orexpected network performance corresponding to the one or more optimal network configuration parameters.

21. The method of any of claims 18-20, wherein the quality estimate of the optimal network configuration suggestion is represented by a prediction interval for network performance with a predetermined probability in case the optimal network configuration suggestion is applied.

22. The method of any of claims 18-21, wherein the acceptance criterion for the optimal network configuration suggestion is represented by a threshold size for the prediction interval.

23. A method comprising:sending an optimal network configuration suggestion to a network control function; and receiving an acknowledge message from the network control function, the acknowledge message containing at least one of the following:a decision of accepting or rejecting the optimal network configuration suggestion; or a decision cause.

24. The method of claim 23, further comprising:using the decision and decision cause as feedback information to improve an algorithm or model that generates the optimal network configuration suggestion.

25. A method comprising:receiving an evaluation request for an optimal network configuration suggestion from a network control function;determining a quality estimate and an acceptance criterion for the optimal network configuration suggestion; andsending an evaluation response to the network control function, the evaluation response containing the quality estimate and the acceptance criterion for the optimal network configuration suggestion.

26. The method of claim 25, wherein the acceptance criterion for the optimal network configuration suggestion is determined inversely proportional to a difference between one or more optimal network configuration parameters indicated in the optimal network configuration suggestion and corresponding network configuration parameters currently configured for the network.

27. The method of claim 25 or 26, wherein the acceptance criterion for the optimal network configuration suggestion is determined directly proportional to performance improvement expected in case the optimal network configuration suggestion is applied.

28. The method of any of claims 25-27, further comprising:receiving network data from the network; andusing the received network data to determine the quality estimate and the acceptance criterion for the optimal network configuration suggestion.

29. The method of any of claims 25-28, further comprising:requesting for the network data from the network.

30. A method comprising:receiving an optimal network configuration suggestion from a cognitive function;determining a quality estimate and an acceptance criterion for the optimal network configuration suggestion; anddeciding to accept or reject the optimal network configuration suggestion based on the determined quality estimate and acceptance criterion.

31. The method of claim 13, further comprising:receiving network data from the network; andusing the received network data to determine the quality estimate and the acceptance criterion for the optimal network configuration suggestion.

32. The method of claim 13 or 14, further comprising:requesting for the network data from the network.

33. The method of any of claims 30-32, further comprising:sending an acknowledge message to the cognitive function, the acknowledge message containing at least one of the following:the decision of accepting or rejecting the optimal network configuration suggestion; or a decision cause.

34. The method of any of claims 30-33, wherein the acceptance criterion for the optimal network configuration suggestion is determined inversely proportional to a difference betweenone or more optimal network configuration parameters indicated in the optimal network configuration suggestion and corresponding network configuration parameters currently configured for the network and directly proportional to performance improvement expected in case the optimal network configuration suggestion is applied.

35. An apparatus for a network control function, comprising means for performing the method of any of claims 18-22, 30-34.

36. An apparatus for a cognitive function, comprising means for performing the method of any of claims 23-24.

37. An apparatus for a reliability check cognitive function, comprising means for performing the method of any of claims 25-29.

38. A computer readable medium comprising instructions that, when executed by an apparatus for a network control function, cause the apparatus to perform the method of any of claims 18-22, 30-34.

39. A computer readable medium comprising instructions that, when executed by an apparatus for a cognitive function, cause the apparatus to perform the method of any of claims 23-24.

40. A computer readable medium comprising instructions that, when executed by an apparatus for a reliability check cognitive function, cause the apparatus to perform the method of any of claims 25-29.44

Images