Robust ai / ml based network control with trust building

Abstract

Various example embodiments relate to devices, methods, apparatuses, and computer-readable media that support robust artificial intelligence / machine learning based network control with trust building. An apparatus can be adapted to receive an optimal network configuration suggestion from a cognitive function, send an evaluation request for the optimal network configuration suggestion to a reliability check cognitive function, and receive an evaluation response from the reliability check cognitive function containing a quality estimate for the optimal network configuration suggestion and an acceptance criterion.

Description

Technical Field

[0001] The various example embodiments described herein generally relate to communication technologies, and more specifically, to devices, methods, apparatuses, and computer-readable media that support network control based on robust artificial intelligence / machine learning with trust building. Background Technology

[0002] Some abbreviations that may be found in the instruction manual and / or accompanying drawings are defined here as follows: AI / ML Artificial Intelligence / Machine Learning CAN Cognitive Autonomous Network CCL Closed Control Loop CF cognitive function CIO's individual cell offset KPIs (Key Performance Indicators) MLB Mobility Load Balancing MNO mobile network operator MRO Mobility Robustness Optimization NCF Network Control Function RAN (Radio Access Network) RCCF Reliability Verification Cognitive Function TTT Trigger Time Cognitive Autonomous Networks (CAN) are a promising approach to advancing the automation of network management using artificial intelligence / machine learning (AI / ML)-based functions called Cognitive Functions (CFs). CFs interact with the network environment to learn and predict optimal network configuration parameters to optimize their objectives. To minimize conflicts between the actions of multiple CFs, the CFs send their proposed network configurations to a Central Network Control Function (NCF), which evaluates the configurations and decides to accept or reject them. The NCFs can then collectively consider network configurations from different CFs and compute the final network configuration parameters that optimize the combined interests of all CFs. Summary of the Invention

[0003] The following provides a brief overview of exemplary embodiments to offer a basic understanding of some aspects of the various embodiments. It should be noted that the content of this invention is not intended to identify key features of essential elements or define the scope of the embodiments, and its sole purpose is to introduce some concepts in a simplified form as a prelude to the more detailed description provided below.

[0004] In a first aspect, an example embodiment of an apparatus is provided. The apparatus may include at least one processor and at least one memory. The at least one memory may store instructions that, when executed by the at least one processor, cause the apparatus to at least: receive an optimal network configuration recommendation from a cognitive function; send an evaluation request for the optimal network configuration recommendation to a reliability verification cognitive function; and receive an evaluation response from the reliability verification cognitive function, the evaluation response including a quality estimate and acceptance criteria for the optimal network configuration recommendation.

[0005] In a second aspect, an example embodiment of an apparatus is provided. The apparatus may include at least one processor and at least one memory. The at least one memory may store instructions that, when executed by the at least one processor, cause the apparatus to at least: send an optimal network configuration recommendation to a network control function; and receive an acknowledgment message from the network control function. The acknowledgment message includes at least one of the following: a decision to accept or reject the optimal network configuration recommendation; or a reason for the decision.

[0006] In a third aspect, an example embodiment of an apparatus is provided. The apparatus may include at least one processor and at least one memory. The at least one memory may store instructions that, when executed by the at least one processor, cause the apparatus to at least: receive an evaluation request for an optimal network configuration recommendation from a network control function; determine a quality estimate and acceptance criteria for the optimal network configuration recommendation; and send an evaluation response to the network control function, the evaluation response including the quality estimate and acceptance criteria for the optimal network configuration recommendation.

[0007] In a fourth aspect, an example embodiment of an apparatus is provided. The apparatus may include at least one processor and at least one memory. The at least one memory may store instructions that, when executed by the at least one processor, cause the apparatus to at least: receive an optimal network configuration recommendation from a cognitive function; determine a quality estimate and acceptance criteria for the optimal network configuration recommendation; and decide whether to accept or reject the optimal network configuration recommendation based on the determined quality estimate and acceptance criteria.

[0008] Example embodiments of methods, apparatuses and computer-readable media are also provided, which generally correspond to the example embodiments described above, and for convenience, repeated descriptions thereof are omitted herein.

[0009] Other features and advantages of exemplary embodiments of this disclosure will also become apparent when read in conjunction with the accompanying drawings, which illustrate the principles of exemplary embodiments of this disclosure by way of example. Attached Figure Description

[0010] Some exemplary embodiments will now be described by way of non-limiting example with reference to the accompanying drawings.

[0011] Figure 1 This is a schematic block diagram illustrating an example framework for network control and optimization in which example embodiments of the present disclosure can be implemented.

[0012] Figure 2 This is a schematic block diagram illustrating an example framework for robust network control according to an example embodiment of the present disclosure.

[0013] Figure 3 This is a schematic signaling diagram illustrating a process according to an example embodiment of the present disclosure.

[0014] Figure 4 This is a schematic signaling diagram illustrating a process according to an example embodiment of the present disclosure.

[0015] Figure 5 This is a schematic signaling diagram illustrating a process according to an example embodiment of the present disclosure.

[0016] Figure 6 This is a schematic signaling diagram illustrating a process according to an example embodiment of the present disclosure.

[0017] Figure 7 This is a schematic flowchart illustrating a method according to an example embodiment of the present disclosure.

[0018] Figure 8 This is a schematic flowchart illustrating a method according to an example embodiment of the present disclosure.

[0019] Figure 9 This is a schematic flowchart illustrating a method according to an example embodiment of the present disclosure.

[0020] Figure 10 This is a schematic flowchart illustrating a method according to an example embodiment of the present disclosure.

[0021] Figure 11 This is a schematic block diagram illustrating an apparatus according to an exemplary embodiment of the present disclosure.

[0022] Figure 12 This is a schematic block diagram illustrating an apparatus according to an exemplary embodiment of the present disclosure.

[0023] Figure 13 This is a schematic block diagram illustrating an apparatus according to an exemplary embodiment of the present disclosure.

[0024] Figure 14 This is a schematic block diagram illustrating an apparatus according to an exemplary embodiment of the present disclosure.

[0025] Figure 15 This is a schematic block diagram illustrating a device in a network control system according to an exemplary embodiment of the present disclosure.

[0026] In all the accompanying drawings, the same or similar reference numerals denote the same or similar elements. Repeated descriptions of the same elements will be omitted. Detailed Implementation

[0027] In the following description, some exemplary embodiments are described in detail with reference to the accompanying drawings. Specific details are included in the description for the purpose of providing a thorough understanding of the various concepts. However, it will be apparent to those skilled in the art that these concepts can be practiced without these specific details. In some instances, well-known circuits, technologies, and components are shown in block diagram form to avoid obscuring the described concepts and features.

[0028] Figure 1 An example framework 100 for network control and optimization is shown, in which example embodiments of this disclosure can be implemented. (See example...) Figure 1 As shown, the network control and optimization framework 100 may include multiple cognitive functions (CFs) 120, such as CF1120a, CF2120b, and CF3120c, which observe and learn the behavior of the network environment (e.g., radio access network (RAN) 130). Given a network state / feature X = [X1, X2, ..., X...], ... n ], consisting of a set of key performance indicators (KPIs) Y=[Y1, Y2, ..., Y n The network performance represented by ] can be determined by some parameters p = [p1, p2, ..., p n Control. The real network mapping can be represented by T: (X, p) → Y. The CF120 can periodically check whether the network state or performance has changed. If the network state or performance has changed, the CF120 calculates the optimal control parameter p based on its learning. And if the optimal control parameter p If the actual control parameter p is different from the one currently configured at RAN130, then the optimal control parameter p will be... The data is sent to the Central Network Control Function (NCF) 110. The NCF 110 evaluates and decides whether to adopt or reject the optimal control parameter p recommended by the CF 120. NCF110 may or may not have access to network data from RAN130. Figure 1 The relevant interfaces between NCF110, CF120 and RAN130 are shown.

[0029] The CF120 can be a third-party application, and they don't necessarily need to communicate directly with each other. For example, the CF120 can communicate indirectly with each other only through the NCF110. Given a CF... n 120 can use hidden AI / ML model M n :(X n p n →Yn To achieve this, where X n It is an input that includes features describing the state of RAN130, p n These are the (multiple) parameters of interest, and Y n This is the output of the AI / ML model. Typically, Y... n This can represent some target KPIs, such as cell load for mobility load balancing (MLB) or the successful handover ratio for mobility robustness optimization (MRO). The model M... n Approximate true RAN mapping T n :(X n p n →Y n And different models M n It can be designed to optimize different KPIs. n Some control parameters p can be shared by multiple CF120s. For example, a first CF used to perform Mobility Load Balancing (MLB) and a second CF used to perform Mobility Robustness Optimization (MRO) can share control parameters such as Trigger Time (TTT) and Cell Individual Offset (CIO). When the NCF110 receives TTT and CIO values ​​proposed by one or both of the first and second CFs, the NCF110 can evaluate how good these values ​​are for both MLB and MRO, and calculate the final TTT and CIO values ​​to be used to configure the RAN130.

[0030] Figure 2 An example framework 200 for robust network control according to an example embodiment of this disclosure is shown. For ease of labeling, the index n is discarded and a single CF is discussed in the following description. It should be understood that the description of a single CF also applies to other CFs. Throughout this disclosure, variables may have a single value (e.g., a scalar) or multiple values ​​(e.g., a vector or matrix).

[0031] refer to Figure 2Given network data (X, p, Y), the CF120 can approximate the network environment T using its model M and optimize the objective KPIY=M(X, p) by computing the optimal value of p. This framework is general, and most AI / ML optimization methods (e.g., AI / ML optimization methods with supervised or unsupervised learning) can be represented by this framework with minor adjustments. The inherent complexity of approximation lies in the fact that, in principle, the extent to which a specific mapping T: (X, p) → Y can be learned given available (even high-quality) data (X, p, Y) can be large. This is especially true in current and future complex and dynamic wireless networks, where many features exist in X. In such cases, even small changes in X and p can lead to large changes in Y. If the real RAN mapping T is sufficiently complex, for example, exhibiting large local variability on X, the generalization by M can be highly prone to error.

[0032] The CF120 enables the performance monitoring mechanism to detect suboptimal configurations of the RAN130 using an AI / ML model M. If the received network configuration parameter p is optimal in network state X, the CF120 can continue learning model M from the network environment T. If the received network configuration parameter p becomes suboptimal as network state X changes, the CF120 can calculate the optimal control parameter value p based on model M. The optimal parameter value p As a "recommendation" adopted by CF120 Figure 1 The interface shown transmits to NCF110, and CF120 claims that the suggestion will cause the KPI value Y to be affected. The KPI value Y Better than the current KPI value Y in the current network state curr The NCF110 will change the current parameter value p. curr This suggestion was considered previously. Current parameter value p. curr This can be configured by default by the mobile network operator (MNO), or it can be configured based on long-term performance optimizations through some CAN mechanism in the network. In this sense, NCF110 considers p curr It is reliable in maintaining long-term network performance. MNO is generally preferred only for p. curr Minor changes were made to address time-critical performance issues in the network. The NCF110 made adjustments to p based on recommendations received from the CF120. curr Make minor changes. From the perspective of NCF110, CF120 is a black-box function, where the following assumptions hold: CF model M: (X, p) → Y is hidden; NCF110 does not know the extent to which the CF model M closely approximates the real network environment T related to CF.

[0033] These assumptions apply to multi-vendor environments, where different CFs can be provided by different vendors. Under these assumptions, NCF110 must be confident that if it follows the advice from CF120 regarding the current parameter value p... curr Make the suggested changes Δ =|p curr -p This will cause the KPI value Y in RAN130 to be close to the expected KPI value Y calculated by CF120. In this sense, the NCF must have a certain "trust level" in CF120.

[0034] like Figure 2 As shown, the framework for establishing trust in CF120 can be divided into the following aspects: a) The CF performance learning mechanism enables the NCF110 to independently and reliably evaluate the quality (i.e., accuracy / reliability) of the recommendations made by the CF120. Because this mechanism is independent, the NCF110 cannot be manipulated by the CF120.

[0035] b) Standard calculation and checking mechanism used to calculate NCF110 can determine whether to accept or reject parameter changes Δ The standard.

[0036] The above two aspects will be discussed in more detail below. Regarding the first aspect, it is suggested that p... The quality depends on how well CF120 can learn the behavior of RAN130, i.e., how well model M approximates the real network mapping T. CF120 can improve its approximation over time, i.e., as it is exposed to more and more network data (X, p, Y) sampled from RAN130. As an example, this can be achieved by computing Y... Nearby prediction or confidence intervals (i.e., defined by CF120 as X and p) The quality of this approximation is estimated by using the KPI predictions made by the function. The prediction interval conveys the following information: by applying parameter variation Δ in RAN130. The realized true future KPI value Y in the prediction interval r(X, p) )=Y int =[Y min Y max Within [ ], the probability is at least α close to 1, where Y min and Y max These are the prediction intervals Y int The lower and upper limits of KPI forecasts. Typically, KPI forecasts for Y... Included in the prediction interval Y int Internal. Prediction interval function r: (X, p) →Yint Depends on X and p And therefore, suggestions for network features / states X and parameters p. Sensitive. Furthermore, even without explicitly introducing a time variable, the RAN environment (i.e., the function T: (X, p) → Y) changes over time, thus affecting the prediction interval Y. int It will be updated online over time.

[0037] Clearly, the interval r(X, p) )=Y int The tighter it is, the more it is in Y The greater the confidence level conveyed, the better the final recommendation will be. Predictive intervals can be obtained using data-driven methods such as conformal regression, which delivers predictive intervals maintained with an effective probability of at least α, where the value of α is typically fixed close to 1 (e.g., 0.95). int The exact / true length ultimately depends on the CF model M in the network state X and the network configuration parameter p. The NCF110 approximates the RAN environment T to a certain extent. However, the NCF110 can only estimate r(X, p) with the help of a limited amount of data. Since the estimation is data-driven, the prediction interval generally becomes more accurate for a fixed probability α as more data becomes available; that is, the NCF110 can better estimate its confidence over time with respect to the CF120. Any data-driven approach is subject to this limitation.

[0038] In the second aspect, NCF110 requires appropriate criteria to determine whether to accept or reject the parameter change Δ. If the prediction interval Y can be accurately maintained over time... int Then NCF110 can use them to evaluate parameter changes Δ Now, the question is how to treat r(X, p) as... The output Y int The size of Y is subject to an acceptance criterion c. On the one hand, if the acceptance criterion c is too loose, that is, relatively large values ​​of Y will be allowed. int This could lead to unreliable changes in network parameters. On the other hand, it requires a very strict interval Y. int Overly stringent standards will inhibit CF and reduce Y curr The ability to address time-critical performance issues. In existing technologies, it is suggested that the criterion for the interval be set a priori by the MNO, which is clearly independent of X and Y. and Δ This is because these variables are not known a priori. Such a general criterion must be rigorous to exclude the preferred reliable value p. curr Large and unreliable parameter variations in the medium.

[0039] The previous method has the following drawbacks. First, for X and Δ For certain values, CF120 may not satisfy the condition given by MNO in r(X, p) The fixed acceptance standard c is set on the network. The reason is that the network mapping T depends locally on the network state X and the configuration parameter p, and even if the CF120 has a large amount of data, it may be arbitrarily difficult to approximate the network mapping T for certain values ​​of X and p.

[0040] Secondly, in the early stages of its learning, CF120 may not exhibit high accuracy because it has not fully learned the network mapping T. In this case, if the standard c is very strict and fixed, then regardless of p... curr No matter how small, it may not allow CF120 to output to p. curr Make any changes.

[0041] Third, NCF110 for r(X, p) The data-driven estimation depends on network data and recommendations from CF120. This means that in the early stages of operation, NCF110 itself cannot estimate the interval r(X, p). It's too confident in its accuracy.

[0042] Example embodiments of this disclosure propose methods for robust network control by evolving trust and cooperation between the NCF and CF over time. On one hand, some methods enable the AI / ML-based CF to approximate the behavior of the RAN by considering as many external factors as possible, thereby improving the quality of the AI / ML model and consequently the quality of the configuration recommendations derived from the model. On the other hand, some methods help the NCF obtain information about the expected KPI value Y calculated by the CF. The accurate prediction interval, and the flexible strategy / standard c for the prediction interval, through which the NCF can appropriately adjust the configuration suggestions p received from the CF. In some example embodiments, the result can be calculated by c(Δ). Y curr Y The dynamic standard is represented by Δ, which depends on the proposed parameter change Δ. The expected KPI value Y calculated by CF And the current KPI value Y that CF aims to improve. curr In some example embodiments, the standard c can vary with the suggested parameter Δ. Inversely proportional to, and with, the expected KPI improvement |Y curr -Y |Proportional. It is believed that even if the suggested parameter changes by Δ Very small, that is, the parameter value p suggested by CF. The current value p that is actually configured at the network. curr Not far, or you can get suggestions from the CF configuration parameters. Significant, time-critical improvements to KPI Y. If the current KPI value Y... curr The degradation is significant, and the expected KPI value Y is expected to be lower. Significantly better than the current KPI value Y curr And the parameter changes Δ If the value is smaller, then the NCF is smaller in its relationship to the prediction interval r(X, p). The requirements for Δ should not be too strict. Conversely, if the parameter changes by Δ The value is relatively large, meaning CF recommends a preferred parameter value p. curr If a large change is made, the NCF will be affected by the prediction interval r(X, p). The requirements should be strict. In this way, Δ can be varied according to the suggested parameters. Current KPI value Y curr And the expected KPI value Y To determine the flexible acceptance criterion c. If the corresponding prediction interval r(X, p) If the value of Δ is less than or equal to the standard c, then the NCF can decide to accept the proposed parameter change Δ. Or if the prediction interval r(X, p) If the value of Δ is greater than the standard c, then the NCF can decide to reject the parameter change Δ. The decision made by the NCF also indicates whether the CF model M is judged to be good enough. In some example embodiments, the criterion c and the prediction interval r(X, p) are... Both can be sent to the CF along with the decision made by the NCF. The CF can use this information as feedback to improve its estimation of the real network environment T over time.

[0043] Figure 3 A process 300 for CF performance evaluation according to an example embodiment of this disclosure is illustrated. Process 300 may be implemented at NCF 110, CF 120, RAN 130, and Reliability Verification Cognitive Function (RCCF) 122, which may reside in the system along with CF 120. It is assumed that NCF 110 does not have access to network data of RAN 130, and therefore RCCF 122 may be adapted to perform CF performance evaluation functions.

[0044] See Figure 3 The CF120 can receive network data from the RAN130 at point 310, and the RCCF122 can receive network data from the RAN130 at point 312. It is worth noting that the network data available to the CF120 is CF120-specific, for example, network data (X...n p n Y n ), where n represents the index of CF120, and RCCF122 can access its network data that can be used to evaluate the performance of any CF in the system, such as network data (X). i p i Y i ), where i=1,…,N, and N represents the number of CFs in the system.

[0045] At point 314, CF120 can use network data to train its AI / ML model M, which approximates the network environment of RAN130. Based on the AI / ML model M, appropriate training methods can be used, including but not limited to supervised or unsupervised training. CF120 can periodically or continuously receive network data from RAN130 and train its AI / ML model M. As more network data becomes available, model M learns the network mapping T better; that is, it can more accurately approximate the network mapping T.

[0046] When the AI / ML model M is trained to a satisfactory level, for example, when the loss of the model output is less than a predetermined amount or within a minimum range compared to the ground truth data, the CF120 can notify the NCF110 at point 316 that the model learning is complete. The NCF110 can then initiate a performance evaluation of the CF model M. In some example embodiments, the CF120 can continue training and updating the model M after sending the learning complete message to the NCF110; that is, the model M can be trained online.

[0047] At point 318, NCF110 can request RCCF122 to evaluate CF120's AI / ML model M. As mentioned above, NCF110 cannot access network data, therefore it requests RCCF122, which has access to network data, to evaluate CF model M.

[0048] In response to a request, RCCF122 may send a request for data and / or information required for model evaluation to NCF110. For example, in addition to the network data (X, p, Y) received from RAN130 at 312, RCCF122 may need to know at least the inference output of model M in order to evaluate model performance. It is assumed that the CFs, including CF120 and RCCF122, cannot communicate directly with each other. Therefore, RCCF122 can request the data and / or information required for model evaluation from NCF110.

[0049] The NCF110 can acquire data and / or information from the CF120 at point 322 via a data exchange procedure (e.g., a request and response procedure) for model evaluation. The acquired data and / or information includes, but is not limited to, the inference output of model M, i.e., the predicted optimal configuration parameters p. And the expected KPI value Y Next, the NCF110 can forward data and / or information to the RCCF122 at point 324. In this way, the NCF110 facilitates indirect communication between the CF120 and the RCCF122. It is important to note that the RCCF122 receives network data from the RAN130, but not through the CF120, to ensure that model evaluation is performed independently of the CF120 and to prevent any manipulation by the CF.

[0050] At 326, RCCF122 can use network data from RAN130 and the inference output of model M to perform an appropriate method to estimate the performance of model M. In an example embodiment, RCCF122 can be implemented as an AI / ML-based function that uses an AI / ML model to estimate the performance of CF model M. For example, a conformal regression model can be used at RCCF122. Conformal regression is a method that uses past experience to determine the exact confidence level in new predictions by transforming a given prediction into a prediction set / interval, which can contain predictions and be valid with at least α probability. In an example embodiment, RCCF122 can produce a prediction interval r(X, p) )=Y int =[Y min Y max The forecast interval can include the expected / forecasted KPI value Y. And it is effective with a probability that at least α is close to 1. The probability guarantee parameter α can be set a priori by MNO, and a typical value of α is, for example, 0.95. Clearly, the prediction interval r(X, p) )=Y int The tighter it is, the more it is in Y The middle and final recommended parameter p The greater the confidence level conveyed, the better. Therefore, the prediction interval r(X, p) This can be used to evaluate the quality of model M. During the learning phase, RCCF122 can learn prediction intervals for CF model M using network data periodically received from RAN130 and inference outputs of model M periodically received from NCF110. In the example embodiment, process 300 can be continuously executed online. Additionally, it is worth noting that RCCF122 can generate prediction intervals before CF120 makes any actual configuration recommendations to NCF110.

[0051] Figure 4A process 400 for network control based on dynamically generated standards, according to an example embodiment of this disclosure, is illustrated. Process 400 can be executed at NCF110, CF120, RCCF122, and RAN130. It is also assumed that NCF110 does not have access to network data of RAN130, and that CF120 and RCCF122 cannot communicate directly with each other.

[0052] See Figure 4 At position 410, CF120 can detect suboptimal configurations in RAN130. For example, CF120 can periodically receive CF120-specific network data (X, p, Y) from RAN130 and use an AI / ML model M that approximates the network environment T of RAN130 to predict the expected KPI value Y. The optimal configuration parameters p of RAN130 If the predicted optimal configuration parameter p The parameter p is different from the currently configured parameter. curr And the expected KPI value Y Better than the current KPI value Y curr Then CF120 can determine that a suboptimal configuration has been detected in RAN130. In other words, CF120 can determine that the expected KPI value Y is... Predicted configuration parameter p This is the optimal configuration for RAN130.

[0053] At position 412, the CF120 can include configuration parameters p. and KPI value Y The optimal network configuration is sent as a suggestion to the NCF110. For example, the CF120 can send the suggested parameter value p Send to NCF110, or indicate parameter p to NCF110 curr Change in amount (p) -p curr ).

[0054] Then, the NCF110 can request the RCCF122 to evaluate the optimal network configuration recommended by the CF120 at point 414. At point 414, the NCF110 can include configuration parameters p And the expected KPI value Y The optimal network configuration is forwarded to RCCF122 for evaluation.

[0055] In response to an evaluation request, RCCF122 may request current network data from RAN130 at 416 and receive network data at 418. In an example embodiment, RCCF122 may periodically receive network data (X, p, Y) from RAN130, and step 416 may be omitted or performed only once.

[0056] At 420, RCCF122 can calculate the expected KPI value Y. The prediction interval r(X, p) )=Y int =[Y min Y max This is used to estimate the quality of the optimal configuration and also to determine the acceptance criteria for the optimal configuration. For example, the RCCF122 can use a conformal regression model trained in process 300 to compute the prediction interval. In some example embodiments, the acceptance criterion can be expressed as c(Δ Y curr Y The value of Δ can vary depending on the suggested parameters. =|p curr -p | Expected KPI value Y And the current KPI value Y curr In the example, the accepted standard c can be varied with the suggested parameter Δ. Inversely proportional to, and with, the expected KPI improvement |Y curr -Y |Proportional. It is believed that even if the suggested parameter changes by Δ Very small, that is, the parameter value p recommended by CF120. The current value p that is actually configured at the network. curr Not far, or you can get suggestions from the CF configuration parameters. Significant time-critical improvements to KPI Y. If the current KPI value Y... curr The degradation is significant, and the expected KPI value Y is expected to be lower. Significantly better than the current KPI value Y curr And the parameter changes Δ If the value is smaller, then NCF110 is smaller in its prediction interval r(X, p) The requirements for ) may not be very strict, and the established acceptance criteria c(Δ) Y curr Y The value of Δ may be relatively large. Therefore, even with a relatively large size, the prediction interval will be acceptable. Conversely, if the parameter changes by Δ... The value is relatively large, meaning CF120 recommends a preferred parameter value p. currIf there is a significant change, then NCF110 will be affected by the prediction interval r(X, p). The requirements may be stringent, and the determined acceptance criteria c(Δ) Y curr Y The value may be relatively small. Therefore, a prediction interval with a relatively small / close size would be acceptable. In this way, Δ can be varied according to the suggested parameters. Current KPI value Y curr And the expected KPI value Y To determine the dynamic acceptance criterion c. Compared with the fixed criterion set by the MNO, the determined acceptance criterion c (Δ Y curr Y This allows for flexible filtering of proposed optimal configurations. For example, dynamically accepting the criterion c(Δ) Y curr Y It may be strict, to rigorously prevent large and unreliable configuration changes Δ Furthermore, even if the performance estimate of the proposed configuration is not very high, it may not be very strict to allow small configuration changes Δ that could result in significant KPI improvements. Dynamic acceptance criteria are beneficial for improving network performance, especially in the early stages of model training, because even if NCF110 does not have much confidence in the model, dynamic acceptance criteria will allow for promising small changes.

[0057] At 422, RCCF122 can send an evaluation response message to NCF110, which contains a quality estimate for the optimal configuration (i.e., the prediction interval r(X, p)). )) and acceptance criteria c(Δ Y curr Y Using the prediction interval r(X, p) ) and acceptance criteria c(Δ Y curr Y At position 424, the NCF110 can make a final decision to accept or reject the optimal configuration suggested by the CF120. For example, if the prediction interval r(X, p) If the value is less than or equal to the standard c, then NCF110 can accept the suggested parameter change Δ. If the prediction interval r(X, p) If the value of Δ is greater than the criterion c, then NCF110 can reject the parameter change Δ. .

[0058] If the suggested parameter change Δ is accepted at 424... Therefore, the NCF110 can make parameter changes in RAN130 at position 426. The NCF110 can send the optimal configuration to RAN130 directly or via an intermediate entity.

[0059] At 428, NCF110 can send an acknowledgment message to CF120 to confirm receipt of the optimal configuration recommendation. In the example embodiment, the acknowledgment message may indicate whether the recommended optimal configuration is accepted or rejected. The acknowledgment message may also indicate the reason for the decision. For example, the reason for the decision may include the prediction interval r(X, p) for the optimal configuration. ) and acceptance criteria c.

[0060] The decision and the reasons for the decision can reflect whether the CF model M is judged to be good enough. Therefore, the CF120 can use this information at 430 as feedback to improve its model M. For example, the CF120 can retrain or fine-tune the model M based on the feedback and optionally based on new network data to improve the accuracy and acceptability of the optimal configuration predicted by the model.

[0061] Figure 5 A process 500 for CF performance evaluation according to an example embodiment of this disclosure is shown. Process 500 may be implemented at NCF 110, CF 120, and RAN 130. In process 500, it is assumed that NCF 110 has access to network data of RAN 130, therefore NCF 110 can perform CF performance evaluation itself without requiring RCCF 122. Some steps in process 500 may be similar to those in process 300, and they will be briefly described below.

[0062] See Figure 5 The CF120 can receive network data from RAN130 at position 510, and the NCF110 can receive network data from RAN130 at position 512. It is worth noting that the network data available to the CF120 is CF120-specific; for example, network data (X...) n p n Y n ), where n represents the index of CF120, and NCF110 can access its network data that can be used to evaluate the performance of any CF in the system, such as network data (X). i p i Y i ), where i=1,…,N, and N represents the number of CFs in the system.

[0063] At point 514, the CF120 can use network data to train its AI / ML model M, which approximates the network environment of RAN130. The CF120 can periodically or continuously receive network data from RAN130 and train its AI / ML model M. As more network data becomes available, model M learns the network mapping T better; that is, it can more accurately approximate the network mapping T.

[0064] When the AI / ML model M is trained to a satisfactory level, the CF120 can notify the NCF110 model that learning is complete at position 516.

[0065] At point 518, NCF110 can obtain data and / or information from CF120 via a data exchange procedure (e.g., a request and response procedure) for model performance evaluation. The obtained data and / or information may include, but is not limited to, the inference output of model M, i.e., the predicted optimal configuration parameters p. And the expected KPI value Y .

[0066] At 520, the NCF110 can use network data and the inference output of model M to estimate the performance of model M. In an example embodiment, the NCF110 can use an AI / ML model (e.g., a conformal regression model) to estimate the performance of the CF model M. The conformal regression model can produce a prediction interval r(X, p) )=Y int =[Y min Y max The prediction interval r(X, p) )=Y int =[Y min Y max [Includes expected / predicted KPI values ​​Y with a probability of at least α close to 1] The probability guarantee parameter α can be set a priori by the MNO, and a typical value for α is, for example, 0.95. Clearly, the prediction interval r(X, p) )=Y int The tighter it is, the more it is in Y The final recommended parameter p The greater the confidence level conveyed, the higher the prediction interval r(X, p). This can be used to evaluate the quality of model M. During the learning phase, NCF110 can learn the prediction interval of CF model M using network data periodically received from RAN130 and the inference output of model M periodically received from NCF110. In the example embodiment, process 500 can be continuously executed online. Additionally, it is worth noting that NCF110 can generate the prediction interval before CF120 makes any actual configuration recommendations to NCF110.

[0067] Figure 6 A process 600 for network control based on dynamic criteria generation, according to an example embodiment of this disclosure, is illustrated. Process 600 can be executed at NCF 110, CF 120, and RAN 130. Assuming that NCF 110 has access to network data from RAN 130, NCF 110 can perform CF performance evaluation and criterion generation itself, without requiring RCCF 122. Some steps in process 600 may be similar to those in process 400, and they will be briefly described below.

[0068] See Figure 6 At position 610, CF120 can detect suboptimal configurations in RAN130. For example, if CF120 predicts a parameter p that is similar to the current configuration... curr Different optimal configuration parameters p And with the predicted optimal parameter p The corresponding expected KPI value Y Better than the current KPI value Y curr Then the CF120 can determine that a suboptimal configuration has been detected.

[0069] At position 612, CF120 can include the optimal configuration parameter p. And the expected KPI value Y The optimal network configuration is sent as a suggestion to NCF110.

[0070] Upon receiving the optimal configuration recommendation, NCF110 can request current network data from RAN130 at 614 for evaluating the optimal configuration, and receive network data at 616. In an example embodiment, NCF110 can periodically receive network data (X, p, Y) from RAN130, and step 614 can be omitted or performed only once. NCF110 can use the latest network data to evaluate the optimal configuration received from CF120 and calculate the acceptance criterion c.

[0071] At position 618, NCF110 can calculate the expected KPI value Y. The prediction interval r(X, p) )=Y int =[Y min Y max The NCF110 estimates the quality of the optimal configuration recommended by CF120 and also determines the acceptance criteria for the optimal configuration. For example, NCF110 can use a conformal regression model trained in process 500 to compute the prediction interval. In some example embodiments, the acceptance criterion can be expressed as c(Δ Y curr Y The value of Δ can vary depending on the suggested parameters. =|p curr -p | Expected KPI value Y And the current KPI value Y curr In the example, the accepted standard c can be varied with the suggested parameter Δ. Inversely proportional to, and with, the expected KPI improvement |Y curr -Y | Proportional. If the current KPI value Y curr The degradation is significant, and the expected KPI value Y is expected to be lower. Significantly better than the current KPI value Y curr And the proposed parameter change Δ If the value is smaller, then NCF110 is smaller in its prediction interval r(X, p) The requirements for ) may not be very strict, and the established acceptance criteria c(Δ) Y curr Y The value of Δ may be relatively large. Therefore, even with a relatively large size, the prediction interval will be acceptable. Conversely, if the proposed parameter changes by Δ... The value is relatively large, meaning CF120 recommends a preferred parameter value p. curr If there is a significant change, then NCF110 will be affected by the prediction interval r(X, p). The requirements may be stringent, and the determined acceptance criteria c(Δ) Y curr Y The value may be relatively small. Therefore, a prediction interval with a relatively small / close size would be acceptable. In this way, Δ can be varied according to the suggested parameters. Current KPI value Y curr And the expected KPI value Y To determine the dynamic acceptance criterion c. Compared with the fixed criterion set a priori by the MNO, the dynamic acceptance criterion c(Δ) Y curr Y This allows for flexible filtering of proposed optimal configurations. For example, dynamically accepting the criterion c(Δ) Y curr Y It may be strict, to rigorously prevent large and unreliable configuration changes Δ Furthermore, even if the performance estimate of the proposed configuration is not very high, it may not be very strict to allow small configuration changes Δ that could result in significant KPI improvements. Dynamic acceptance criteria are also beneficial for improving network performance, especially in the early stages of model training, because even if NCF110 does not have much confidence in the model, dynamic acceptance criteria will allow for promising small changes.

[0072] At 620, NCF110 can be based on the quality estimate (i.e., the prediction interval r(X, p) determined at 618). The decision to accept or reject the optimal configuration recommended by CF120 is made using the prediction interval r(X, p) and acceptance criterion c). For example, if the prediction interval r(X, p) is... If the prediction interval r(X, p) has a size less than or equal to the standard c, this reflects that NCF110 has sufficient confidence in the optimal configuration recommended by CF120, and NCF110 can decide to accept the optimal configuration. If the value of ) is greater than the standard c, it reflects that the NCF110 has insufficient confidence in the optimal configuration, and the NCF110 can decide to reject the optimal configuration.

[0073] If the suggested optimal configuration is accepted at 620, then the NCF110 can make parameter changes Δ in RAN130 at 622. The NCF110 can send the optimal configuration to the RAN130 directly or via an intermediate entity.

[0074] At position 624, NCF110 can send an acknowledgment message to CF120 to confirm receipt of the optimal configuration recommendation. In an example embodiment, the acknowledgment message may indicate whether the recommended optimal configuration is accepted or rejected. The acknowledgment message may also indicate the reason for the decision. For example, the reason for the decision may include the prediction interval r(X, p) calculated for the optimal configuration. ) and acceptance criteria c.

[0075] The decision and the reasons for the decision can reflect whether the CF model M is judged to be good enough. Therefore, the CF120 can use this information as feedback at 626 to improve its model M. For example, the CF120 can retrain or fine-tune the model M based on the feedback and optionally based on new network data to improve the accuracy and acceptability of the optimal configuration predicted by the model.

[0076] Figure 7 A method 700 according to an example embodiment of the present disclosure is shown. Method 700 may be implemented at an NCF (e.g., NCF110 discussed above).

[0077] refer to Figure 7 Method 700 may include step 710 of receiving an optimal network configuration recommendation from a CF (e.g., CF120 discussed above). In an example embodiment, the optimal network configuration recommendation may include one or more optimal network configuration parameters p. And when one or more optimal network configuration parameters p are applied Expected network performance at that time, such as KPI value Y .

[0078] Method 700 may further include step 712 of sending an evaluation request for an optimal network configuration recommendation to an RCCF (e.g., RCCF 122 discussed above). In step 712, NCF 110 may forward the optimal network configuration recommendation received from CF 120 to RCCF 122.

[0079] Method 700 may further include step 714 of receiving an evaluation response from RCCF 122. The evaluation response may contain a quality estimate and acceptance criteria for the optimal network configuration recommendation. In an example embodiment, the quality estimate of the optimal network configuration recommendation may be determined by the prediction interval r(X, p). )=Y int =[Y min Y max This indicates that the prediction interval r(X, p) )=Y int =[Y min Y max This includes the expected network performance with a predetermined probability α, such as the KPI value Y. The acceptance criterion can be represented by a threshold size for the prediction interval. In some example embodiments, the acceptance criterion can be related to one or more optimal network configuration parameters p indicated in the optimal network configuration recommendation. The corresponding network configuration parameter p for the current network configuration curr The difference between them is inversely proportional. In some example embodiments, when applying optimal network configuration recommendations, the acceptance criterion can be inversely proportional to the expected performance improvement |Y|. curr -Y | proportional, where Y curr It refers to the current network performance, such as the network's KPI values.

[0080] In some example embodiments, method 700 may further include step 716 of deciding to accept or reject the optimal network configuration recommendation based on the evaluation response received from RCCF 122 (i.e., based on the quality estimate and acceptance criteria for the optimal network configuration recommendation), and step 718 of sending an acknowledgment message to CF 120. The acknowledgment message may contain the decision to accept or reject the optimal network configuration recommendation, and optionally include a reason for the decision. For example, the reason for the decision may include the quality estimate and acceptance criteria for the optimal network configuration recommendation. As shown in the dashed boxes, steps 716 and 718 may be optional.

[0081] Figure 8A method 800 according to an example embodiment of the present disclosure is shown. Method 800 may be implemented at an NCF (e.g., NCF110 discussed above).

[0082] refer to Figure 8 Method 800 may include step 810 of receiving an optimal network configuration recommendation from a CF (e.g., CF120 discussed above). In an example embodiment, the optimal network configuration recommendation may include one or more optimal network configuration parameters p predicted by CF120. And when one or more optimal network configuration parameters p are applied Expected network performance, such as KPI value Y .

[0083] Method 800 may further include step 812 of determining a quality estimate and acceptance criteria for the optimal network configuration recommendation. In an example embodiment, the quality estimate of the optimal network configuration recommendation may be determined as the prediction interval r(X, p). )=Y int =[Y min Y max The prediction interval r(X, p) )=Y int =[Y min Y max This includes expected network performance, such as a KPI value Y with a predetermined probability α. The acceptance criterion can be determined as a threshold size for the prediction interval. In some example embodiments, the acceptance criterion can be related to one or more optimal network configuration parameters p indicated in the optimal network configuration recommendation. The corresponding network configuration parameter p for the current network configuration curr The difference between them is inversely proportional. In some example embodiments, the acceptance criterion can be related to the expected performance improvement |Y| when the optimal network configuration recommendation is applied. curr -Y | proportional, where Y curr It refers to the current network performance, such as the network's KPI values.

[0084] In some example embodiments, NCF110 may receive network data from the network (e.g., RAN130) before determining the quality estimate and acceptance criteria for the optimal network configuration recommendation. NCF110 can then use the received network data to determine the quality estimate and acceptance criteria for the optimal network configuration recommendation. In some example embodiments, NCF110 may send a request for network data to RAN130 before receiving network data from RAN130.

[0085] Method 800 may further include step 814 of deciding whether to accept or reject the optimal network configuration recommendation based on the determined quality estimate and acceptance criteria. For example, if the quality estimate of the optimal network configuration recommendation meets the acceptance criteria, NCF 110 may decide to accept the optimal network configuration recommendation. Otherwise, NCF 110 may decide to reject the optimal network configuration recommendation.

[0086] In some example embodiments, method 800 may further include step 816 of sending an acknowledgment message to CF120. The acknowledgment message may contain a decision to accept or reject the optimal network configuration recommendation, and optionally include a reason for the decision. For example, the reason for the decision may include a quality estimate and acceptance criteria for the optimal network configuration recommendation. As shown in the dashed box, step 816 may be optional.

[0087] Figure 9 A method 900 according to an example embodiment of this disclosure is shown. Method 900 may be implemented at a CF (e.g., CF120 discussed above).

[0088] refer to Figure 9 Method 900 may include step 910 of sending an optimal network configuration recommendation to the NCF (e.g., NCF110 discussed above). As described above, the optimal network configuration recommendation may include one or more optimal network configuration parameters p predicted by CF120. And when one or more optimal network configuration parameters p are applied Expected network performance, such as KPI value Y .

[0089] Method 900 may further include step 912 of receiving an acknowledgment message from NCF 110. In some example embodiments, the acknowledgment message may contain a decision to accept or reject the optimal network configuration recommendation, and optionally include a reason for the decision. The reason for the decision may include a quality estimate and acceptance criteria for the optimal network configuration recommendation. In some example embodiments, the quality estimate of the optimal network configuration recommendation may be determined by the prediction interval r(X, p). )=Y int =[Y min Y max This indicates that the prediction interval r(X, p) )=Y int =[Y min Y max This includes the expected network performance with a predetermined probability α, such as the KPI value Y. The acceptance criterion can be represented by a threshold value for the prediction interval. In some example embodiments, the acceptance criterion can be related to one or more optimal network configuration parameters p indicated in the optimal network configuration recommendation. The corresponding network configuration parameter p for the current network configurationcurr The difference between them is inversely proportional. In some example embodiments, the acceptance criterion can be related to the expected performance improvement |Y| when the optimal network configuration recommendation is applied. curr -Y | proportional, where Y curr It refers to the current network performance, such as the network's KPI values.

[0090] In some example embodiments, method 900 may further include step 914, which uses the decisions and reasons for those decisions as feedback information to improve the algorithm or model that generates optimal network configuration recommendations. For example, CF may use the feedback information and optionally, based on additional network data, to retrain or fine-tune the algorithm or model. As shown in the dashed box, step 914 is optional.

[0091] Figure 10 A method 1000 according to an example embodiment of the present disclosure is shown. Method 1000 may be implemented at an RCCF (e.g., RCCF122 discussed above).

[0092] refer to Figure 10 Method 1000 may include step 1010 of receiving an evaluation request for an optimal network configuration recommendation from an NCF (e.g., NCF 110 discussed above). The request message may contain the optimal network configuration recommendation. In some example embodiments, the optimal network configuration recommendation may contain one or more optimal network configuration parameters p. And when one or more optimal network configuration parameters p are applied Expected network performance, such as KPI value Y .

[0093] Method 1000 may further include step 1012 of determining a quality estimate and acceptance criteria for the optimal network configuration recommendation. In an example embodiment, the quality estimate of the optimal network configuration recommendation may be determined as the prediction interval r(X, p). )=Y int =[Y min Y max The prediction interval r(X, p) )=Y int =[Y min Y max This includes expected network performance, such as a KPI value Y with a predetermined probability α. The acceptance criterion can be determined as a threshold size for the prediction interval. In some example embodiments, the acceptance criterion can be related to one or more optimal network configuration parameters p indicated in the optimal network configuration recommendation. The corresponding network configuration parameter p for the current network configuration currThe difference between them is inversely proportional. In some example embodiments, the acceptance criterion can be related to the expected performance improvement |Y| when the optimal network configuration recommendation is applied. curr -Y | proportional, where Y curr It refers to the current network performance, such as the network's KPI values.

[0094] In some example embodiments, RCCF 122 may receive network data from the network (e.g., RAN 130) before determining the quality estimate and acceptance criteria for the optimal network configuration recommendation. RCCF 122 can then use the received network data to determine the quality estimate and acceptance criteria for the optimal network configuration recommendation. In some example embodiments, RCCF 122 may send a request for network data to RAN 130 before receiving the network data from RAN 130.

[0095] Method 1000 may further include step 1014 of sending an evaluation response to NCF110. The evaluation response may include a quality estimate and acceptance criteria for the optimal network configuration recommendation.

[0096] The various example embodiments discussed above produce many beneficial effects. For example, if performance is improved |Y -Y curr If | is significant, then NCF can be improved by relaxing the acceptance criterion c(Δ). Y curr Y To allow for small changes Δ This ensures that, in the early stages, CF is allowed to work with reliable parameters p. curr Make small changes. Furthermore, this also applies when the CF model M struggles to accurately approximate a network mapping T that is locally dependent on X and p for certain values ​​of X and p. On the other hand, if the prediction interval r(X, p) If strict criteria are met, then large parameter changes Δ in the CF are also allowed. That is, if the NCF has a considerable degree of confidence / trust in the CF, then the NCF only allows its reliable default parameter p. curr Make major changes. This ensures that large parameter changes Δ are not allowed in the early stages. However, since both CF and NCF improve over time on their respective tasks, NCF can confidently allow CF to undergo large parameter changes. Furthermore, CF can improve its model M over time by incorporating the decisions made by NCF, the prediction interval r, and the standard c, and thus improve its configuration recommendation p. Such information should help CF improve its performance (better accuracy) or increase its likelihood of acceptance (considering the standard provided by NCF). CF can improve its performance by implementing the recommended configuration. To improve acceptance of their recommendations, it is necessary to meet (i.e., be less than or lower than) the standards provided by the NCF or RCCF.

[0097] Figure 11 An apparatus 1100 according to an example embodiment of the present disclosure is shown. The apparatus 1100 may be implemented to include or form at least a portion of an NCF (e.g., NCF 110 discussed above) to perform at least a portion of the operations associated with the NCF 110.

[0098] Reference Figure 11 The device 1100 may include a first component 1110, a second component 1112, and a third component 1114. The first component 1110 may be adapted to perform... Figure 7 Step 710 of method 700 shown is receiving an optimal network configuration recommendation from the RCCF. The second component 1112 can be adapted to perform step 712 of method 700, which is sending an evaluation request for the optimal network configuration recommendation to the RCCF. The third component 1114 can be adapted to perform step 714 of method 700, which is receiving an evaluation response from the RCCF containing a quality estimate and acceptance criteria for the optimal network configuration recommendation.

[0099] The apparatus 1100 may optionally include a fourth component 1116 and a fifth component 1118. The fourth component 1116 may be adapted to perform step 716 of method 700, namely, deciding whether to accept or reject the optimal network configuration recommendation based on the evaluation response. The fifth component 1118 may be adapted to perform step 718 of method 700, namely, sending a confirmation message to the CF containing a decision to accept or reject the optimal network configuration recommendation and optionally containing a reason for the decision.

[0100] Since it has already been referenced Figure 7 The steps of method 700 have been described in detail, so for the sake of simplicity, the blocks in device 1100 are briefly described here.

[0101] Figure 12 An apparatus 1200 according to an example embodiment of the present disclosure is shown. The apparatus 1200 may be implemented to include or form at least a portion of an NCF (e.g., NCF 110 discussed above) to perform at least a portion of the operations associated with the NCF 110.

[0102] Reference Figure 12 The device 1200 may include a first component 1210, a second component 1212, and a third component 1214. The first component 1210 may be adapted to perform... Figure 8Step 810 of method 800 shown is receiving an optimal network configuration recommendation from the CF. The second component 1212 can be adapted to perform step 812 of method 800, namely, determining a quality estimate and acceptance criteria for the optimal network configuration recommendation. The third component 1214 can be adapted to perform step 814 of method 800, namely, deciding whether to accept or reject the optimal network configuration recommendation based on the determined quality estimate and acceptance criteria.

[0103] The apparatus 1200 may optionally include a fourth component 1216. The fourth component 1216 may be adapted to perform step 816 of method 800, namely sending a confirmation message to the CF containing a decision to accept or reject the optimal network configuration recommendation and optionally containing a reason for the decision.

[0104] In some example embodiments, apparatus 1200 may further include components (not shown) for requesting network data from the network, and components (not shown) for receiving the requested network data from the network. The second component 1212 may be adapted to use the received network data to determine a quality estimate and acceptance criteria for an optimal network configuration recommendation.

[0105] Since it has already been referenced Figure 8 The steps of method 800 have been described in detail, so for the sake of simplicity, the blocks in device 1200 are briefly described here.

[0106] Figure 13 An apparatus 1300 according to an example embodiment of the present disclosure is shown. The apparatus 1300 may be implemented to include or form at least a portion of a CF (e.g., CF120 discussed above) to perform at least a portion of the operations associated with CF120.

[0107] Reference Figure 13 The device 1300 may include a first component 1310 and a second component 1312. The first component 1310 may be adapted to perform... Figure 9 Step 910 of method 900 shown is sending an optimal network configuration recommendation to the NCF. The second component 1312 can be adapted to perform step 912 of method 900, which is receiving from the NCF a confirmation message containing a decision to accept or reject the optimal network configuration recommendation and optionally a reason for the decision.

[0108] In some example embodiments, apparatus 1300 may optionally include a third component 1314. The third component 1314 may be adapted to perform step 914 of method 900, i.e., using the decision and the reasons for the decision as feedback information to improve the algorithm or model that generates the optimal network configuration recommendation.

[0109] Since it has already been referenced Figure 9The steps of method 900 have been described in detail, so for the sake of simplicity, the blocks in device 1300 are briefly described here.

[0110] Figure 14 An apparatus 1400 according to an example embodiment of the present disclosure is shown. The apparatus 1400 may be implemented to include or form at least a portion of an RCCF (such as RCCF122 discussed above) to perform at least a portion of the operations associated with RCCF122.

[0111] Reference Figure 14 The device 1400 may include a first component 1410, a second component 1412, and a third component 1414. The first component 1410 may be adapted to perform... Figure 10 Step 1010 of method 1000 shown is receiving an evaluation request for an optimal network configuration recommendation from the NCF. The second component 1412 can be adapted to perform step 1012 of method 1000, namely, determining a quality estimate and acceptance criteria for the optimal network configuration recommendation. The third component 1414 can be adapted to perform step 1014 of method 1000, namely, sending an evaluation response containing the quality estimate and acceptance criteria for the optimal network configuration recommendation to the NCF.

[0112] In some example embodiments, apparatus 1400 may further include components (not shown) for requesting network data from the network, and components (not shown) for receiving the requested network data from the network. The second component 1412 may be adapted to use the received network data to determine a quality estimate and acceptance criteria for an optimal network configuration recommendation.

[0113] Since it has already been referenced Figure 10 The steps of method 1000 have been described in detail, so for the sake of simplicity, the blocks in device 1400 are briefly described here.

[0114] Figure 15 This is a schematic block diagram illustrating devices in a network control system 1500 according to an example embodiment of the present disclosure. Figure 15 As shown, the network control system 1500 may include a network control unit 1510 and one or more cognitive units 1520.

[0115] The network control unit 1510 may include one or more processors 1511, one or more memories 1512, and one or more network interfaces 1514 interconnected via one or more buses 1515. The one or more network interfaces 1514 may provide wired or wireless communication links through which the network control unit 1510 can communicate with other devices, entities, components, or functions. For example, the network control unit 1510 may communicate with one or more cognitive units 1520 and with networks such as wireless / mobile / cellular communication networks (not shown). The one or more memories 1512 may include instructions 1513, which, when executed by the one or more processors 1511, may cause the network control unit 1510 to perform operations and processes related to the NCF 110 as described above.

[0116] One or more cognitive units 1520 may each include one or more processors 1521, one or more memories 1522, and one or more network interfaces 1524 interconnected via one or more buses 1525. The one or more network interfaces 1524 may provide wired or wireless communication links through which the cognitive unit 1520 may communicate with other devices, entities, components, or functions. For example, the cognitive unit 1520 may communicate with a network control unit 1510 and with networks such as wireless / mobile / cellular communication networks (not shown). The one or more memories 1522 may include instructions 1523, which, when executed by one or more processors 1521, may cause the cognitive unit 1520 to perform the operations and procedures related to CF120 as described above, or to perform the operations and procedures related to RCCF122 as described above.

[0117] The one or more processors 1511, 1521 discussed above can be any suitable type for a local technology network and can include one or more processors such as general-purpose processors, special-purpose processors, microprocessors, digital signal processors (DSPs), processor-based multi-core processor architectures, and special-purpose processors (such as those developed based on field-programmable gate arrays (FPGAs) and application-specific integrated circuits (ASICs). The one or more processors 1511, 1521 can be configured to control and cooperate with other elements of the network control unit 1510 and the cognitive unit 1520 to perform the processes described above.

[0118] One or more memories 1512, 1522 may include at least one storage medium of various forms, such as transient and / or non-transient memories. Transient memories may include, but are not limited to, random access memory (RAM) or cache. Non-transient memories may include, but are not limited to, read-only memory (ROM), hard disks, flash memory, etc. As used herein, the term “non-transient” is a limitation on the medium itself (i.e., tangible, not tactile) rather than a limitation on the persistence of data storage (e.g., RAM and ROM). Furthermore, one or more memories 1512, 1522 may include, but are not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any combination thereof.

[0119] Some example embodiments of this disclosure may contribute to 3GPP Technical Report (TR) 28.867, which captures the study of closed control loops (CCLs). Contributions may be made as follows.

[0120] <<<Start Contributing>>>> 4. Concepts and Background 4.X Closed-Loop Control Loop Coordination 4.X.1 CCL Coordination Scenario Multiple Control Logic Containers (CCLs) can coexist and function simultaneously in the same environment. In the worst-case scenario, CCLs may interfere with each other, leading to conflicts that require coordination. Coordination of closed control loops includes the services needed to detect, resolve, or avoid conflicts between CCL targets, processes, control subspaces, or actions. A CCL's control subspace is the set of managed entities that the CCL instance is responsible for, along with the controlled parameters on those entities. A CCL's actions are changes that the CCL can perform on managed entities, such as configuration attributes.

[0121] To address different situations and coordination needs, coordination capabilities may be required in the following scenarios: ● / / …Skip… / / ● The ability to develop / establish trust in recommendations made by the CCL before allowing the CCL to directly control the network. ● / / …Skip… / / / / …jump over… / / 5 use cases / / …jump over… / / 5. X2 Use Case X2: CCL Performance Trust Development 5.X2.1 Description After a Control Center Controller (CCL) is instantiated into the system, its performance may not be as expected at all applicable action decision points, for example, when it consists of multiple management functions at runtime, such as in the case of large actions. The CCL needs to make decisions about its actions and execute them through controller functions (e.g., the CCL coordinator). The coordinator must ensure that the negative impact on the network is minimized and prevent faulty CCLs from occurring. Therefore, before allowing the CCL to have direct control over the network, the coordinator needs to establish “trust” in the decisions made by the CCL regarding network actions (e.g., parameter changes). The 3GPP management system should have the capability to enable the coordinator to develop / establish trust in the recommendations made by the CCL. The CCL should provide the coordinator with its planned action decisions, and the coordinator should provide the CCL with a report indicating how credible (and how good for the network) the planned action decisions are, and in cases where the decisions are not credible, the report should indicate the unmet “trust” criteria.

[0122] 5.X2.2 Potential Requirements REQ-CCL-TRIG-COORD-1: The 3GPP management system should support the ability to notify the CCL (e.g., by deciding on a trust report) the CCL's actions to determine whether a recommendation is acceptable.

[0123] REQ-CCL-TRIG-COORD-1: The 3GPP management system should support the ability to notify the CCL of why its decision is considered unreliable / unfavorable when the CCL makes an unreliable / unfavorable recommendation, for example, how far the CCL's prediction of the impact / effect on network metrics may be from the truth and what the maximum allowable change is in the current network.

[0124] REQ-CCL-TRIG-COORD-1: If the CCL consistently makes good decisions and achieves final trust, the 3GPP management system should support the ability to notify the CCL that the decisions are good and meet the required level of trust.

[0125] 5.X2.3 Potential Solutions —Introducing a request for recommended execution or evaluation (e.g., called CCLRecommendation), which can be sent by CCL to the coordinator.

[0126] ● The request should include information about the desired changes and information about the predicted impact / effect of the decision on the relevant metrics.

[0127] —Introduce a decision trust report to inform the CCL of whether the decision is acceptable.

[0128] ● It is assumed that after the CCL makes a recommendation to the coordinator (including its predictive impact / effect on the metric), the coordinator evaluates the recommendation, for example, based on its internal model or the past execution of actions and their effects.

[0129] ● If the decision is good, the coordinator (via the decision trust report) responds to the CCL that the action decision recommendation is acceptable, thus allowing the CCL to reuse the recommendation. Otherwise, the coordinator informs the CCL (via the decision trust report) that the decision is unacceptable, and the response includes criteria for why the decision is bad / unreliable, such as how far the CCL's prediction of the impact / effect on network metrics might deviate from the truth and what the maximum permissible variation (in the current network parameters) is.

[0130] ● Subsequently, CCL updates its model and makes minor actions, such as small changes to the parameters.

[0131] —The trust decision report includes instructions on when CCL has consistently made good decisions and achieved ultimate trust.

[0132] ● Assuming that based on feedback on the quality of its decisions, the CCL updates its decision engine and repeats the decision evaluation process. Then, if the CCL consistently makes good decisions on large actions, the coordinator can consider the CCL trustworthy enough to make such large decisions. The coordinator notifies the CCL (via a decision trust report) that the CCL has consistently made good decisions and has achieved its final trust. This report may include instructions on how the CCL can subsequently behave—for example, that CCL decisions will be made without coordinator oversight, or that the CCL can directly execute its decisions on the network.

[0133] <<<Contribution complete>>>> Some example embodiments of this disclosure may contribute to 3GPP Technical Specification (TS) 28.536, which specifies management services for communication service assurance. Contributions may be made as follows.

[0134] <<<Start Contributing>>>> 4. Communication Service Guarantee Service 4.1.2.3 Class Definition 4.1.2.3.1AssuranceClosedControlLoop 4.1.2.3.1.1 Definition This class represents information used to control and monitor the guaranteed closed control loop associated with a NetworkSlice or NetworkSliceSubnet. It can be contained in the name of a SubNetwork or ManagedElement.

[0135] To express the goal of ensuring a closed control loop, the MnS consumer needs to request the MnS producer to create an AssuranceClosedControlLoop on the MnS producer. The MnS producer can also trigger the creation of an AssuranceClosedControlLoop; for example, when creating an instance of NetworkSlice or NetworkSliceSubnet, the MnS producer can create an instance of AssuranceClosedControlLoop associated with the NetworkSlice or NetworkSliceSubnet instance to ensure the goal described in the ServiceProfile or SliceProfile. To remove the guaranteed closed control loop, the MnS consumer needs to request the MnS producer to delete the AssuranceClosedControlLoop to release resources on the MnS producer. The MnS producer can also trigger the deletion of the AssuranceClosedControlLoop to release resources automatically.

[0136] To temporarily deactivate a guaranteed closed control loop, an MnS consumer can change the value of the `administrativeState` property to "LOCKED". An MnS producer can disable the guaranteed closed control loop, for example, by setting the `operational state` property to "disabled" in case of a conflict. When the MnS producer enables the closed control loop, the `operational state` is set to "enabled" again. To activate a guaranteed closed control loop, an MnS consumer can change the value of the `administrativeState` property to "UNLOCKED".

[0137] The AssuranceClosedControlLoop may name multiple instances containing an AssuranceGoal, which represents the guaranteed objective and the corresponding observed or predicted objective fulfillment information (see Clause 4.1.2.3.2). The AssuranceGoal may optionally include the scope of the guarantee in terms of location (see Clause 4.1.2.3.2).

[0138] The property "controlLoopLifeCyclePhase" is used to track the lifecycle of the AssuranceClosedControlLoop.

[0139] The attribute aCCLDisallowedList is used to narrow down the scope of ACCL. See Clause 6.1.6 of TS28.535

[17] . Each entry in the list indicates a specific list of attributes belonging to a managedEntity identified by a managedEntityIdentifier that is not allowed to be modified by ACCL.

[0140] 4.1.2.3.1.2 Attributes AssuranceClosedControlLoopIOC includes properties inherited from TopIOC (defined in TS28.622[5]) and the following properties:

[0141] 4.1.2.3.1.3 Constraints

[0142] 4.1.2.3.1.4 Notification Unless otherwise specified or added, the public notices defined in Clause 4.1.2.5 are valid for this IOC.

[0143] --------CCL Coordinator Start 4.1.2.3. YCCL Coordinator <ioc>> 4.1.2.3.Y.1 Definition The IOC indicates the nature of the CCL coordinator.

[0144] 4.1.2.3.Y.2 Attribute The CCL coordinator IOC includes the following properties:

[0145] -----------CCL Recommendation Begins XX1CCLRecommendation< <datatype>> XX1.1 Definition The <ioc>This represents the recommended attributes for CCL action decisions (e.g., changes to network parameters). It includes the session ID, the ID of the network-managed object, the parameter values ​​of that object, and the predicted performance effect / metric as a result of the action in the network object.

[0146] XX1.2 Attributes cCLRecommendation< <datatype>>Includes the following attributes:

[0147] XXX1MetricVauePredictions< <datatype>> XXX1.1 Definition The <datatype>> represents a prediction of the performance metrics of the managed entity, based on the performance guarantees provided by the CCL. The CCL generates these values ​​using its AI / ML model, which represents the mapping between the state of the managed object (including its parameters) and the performance metrics.

[0148] XXX1.2 Attributes MetricVauePredictions< <datatype>>Includes the following attributes:

[0149] XXX2ParameterValues< <datatype>> XXX2.1 Definition The <datatype>> This indicates the parameters and their values ​​that CCL recommends configuring in the managed objects for performance guarantees. XXX2.2 Attributes ParameterValues< <datatype>>Includes the following attributes:

[0150] -----------CCL recommendation ends 4.1.2.3.Y.3 Attribute Constraints none 4.1.2.3.Y.4 Notification Unless otherwise specified or added, public notices as defined in Clause 4.1.2.5 are valid for this data type.

[0151] --------CCL Coordinator End --------Decision Trust Report 4.1.2.3.ZDecisionTrustReport< <datatype>> 4.1.2.3.Z.1 Definition The <datatype>The > attribute represents a report on the assessment performed by the coordinator. It contains an assessment of whether the CCL is allowed to take action (directly or indirectly through the coordinator) on the managed object, which is the standard used to conduct the assessment. The default value is the current value of the parameter that the CCL requires to be changed, and the maximum allowed change is the maximum change allowed by the CCL (based on its trust performance). Additionally, it includes the prediction error range on the predictions provided by the CCL.

[0152] 4.1.2.3.Z.2 Attributes decisionTrustReport <datatype>>Includes the following attributes:

[0153] XZ1Evaluation< <datatype>> XZ1.1 Definition The <datatype>> Indicates the coordinator's evaluation and decision after receiving the CCL's recommendation. Permitted directional actions (if true) allow the CCL direct access to the managed object's parameters. If true, an action being accepted means the coordinator accepts the CCL's recommendation regarding its proposed / planned actions for the managed object.

[0154] XZ1.2 Attributes Decisions <datatype>>Includes the following attributes:

[0155] XZ1Critera< <datatype>> XZ1.1 Definition The <datatype>>Indicates the coordinator's use to accept or reject the CCL recommended standards. XZ1.2 Attributes Critera< <datatype>>Includes the following attributes:

[0156] --------End of report 4.1.2.4 Attribute Definition 4.1.2.4.1 Attribute Characteristics The following table defines the properties of the attributes specified in this document.

[0157]

[0158] <<Contribution complete>>>> It should be understood that the boxes in the figures can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more boxes may be implemented using software and / or firmware (e.g., machine-executable instructions stored in a storage medium). In addition to or in lieu of machine-executable instructions, some or all of the boxes in the figures may be implemented at least partially by one or more hardware logic components. Examples, but not limited to, illustrative types of hardware logic components that may be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), etc. Some example embodiments also provide program instructions that, when executed by one or more processors, cause a device or apparatus to perform the processes described above. The program instructions for performing the processes of the example embodiments can be written in any combination of one or more programming languages. The program instructions can be provided to one or more processors or controllers of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that, when executed by the processor or controller, the program instructions cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program instructions can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0159] Some example embodiments also provide a computer program product or computer-readable medium in which program instructions are stored. A computer-readable medium can be any tangible medium that can contain or store a program used by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. More specific examples of machine-readable storage media will include electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable optical disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0160] As used herein, "at least one of the following: " and "at least one of " and similar wording (where the list of two or more elements is connected by "and" or "or") means at least any one of the elements, at least any two or more of the elements, or all the elements.

[0161] Throughout this disclosure, references to "one embodiment," "embodiment," "some embodiments," "other embodiments," etc., indicate that one or more specific features, structures, steps, concepts, and / or characteristics based on the principles of this disclosure may be included in that embodiment. However, such references do not necessarily imply that all embodiments include a specific feature, structure, step, concept, and / or characteristic, or that an embodiment includes all features, structures, steps, concepts, and / or characteristics. Some embodiments may include one or more such features, structures, steps, concepts, and / or characteristics in various combinations thereof. It should be understood that one or more of the features, structures, steps, concepts, and / or characteristics described with reference to one embodiment may be combined with one or more of the features, structures, steps, concepts, and / or characteristics of any other embodiment provided herein. That is, any features, structures, steps, concepts, and / or characteristics described herein may be mixed and matched to create hybrid embodiments, and such hybrid embodiments are within the scope of this disclosure. Furthermore, references to "one embodiment," "embodiment," "some embodiments," "other embodiments," etc., in various places in the specification do not necessarily refer to the same embodiment, nor are they necessarily separate or alternative embodiments that are mutually exclusive with other embodiments. It should also be understood that the various features, structures, steps, concepts, and / or characteristics of the disclosed embodiments are independent and separate from each other, and may be used or present individually or in various combinations thereof to create alternative embodiments that are considered part of this disclosure. Therefore, this disclosure is not limited to the embodiments specifically described herein, as describing all the many possible combinations and sub-combinations of features, structures, steps, concepts, and / or characteristics would be too cumbersome, and the examples of embodiments disclosed herein are not intended to limit the broader aspects of this disclosure.

[0162] Furthermore, although the operations are described in a specific order, this should not be construed as requiring that such operations be performed in the specific order shown or sequentially, or that all the operations shown be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Similarly, while the foregoing discussion contains details of several specific embodiments, these details should not be construed as limiting the scope of the invention, but rather as descriptions of features that may be specific to particular embodiments. Certain features described in the context of a single embodiment may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0163] Although the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter defined in the appended claims is not limited to the specific features or actions described above. Rather, the specific features and actions described above are disclosed as examples of implementing the claims.< / datatype> < / datatype> < / datatype> < / datatype> < / datatype> < / datatype> < / datatype> < / datatype> < / datatype> < / datatype> < / datatype> < / datatype> < / datatype> < / datatype> < / datatype> < / datatype> < / ioc> < / datatype> < / ioc>

Claims

1. An apparatus comprising: At least one processor; as well as At least one memory, the at least one memory storing instructions that, when executed by the at least one processor, cause the device to perform at least the following: Receive optimal network configuration recommendations from cognitive functions; Send an evaluation request for the optimal network configuration recommendation to the reliability verification cognitive function; and The system receives an evaluation response from the reliability verification cognitive function, the evaluation response containing a quality estimate and acceptance criteria for the optimal network configuration recommendation.

2. The apparatus of claim 1, wherein the instructions, when executed by the at least one processor, further cause the apparatus to perform at least: Based on the evaluation response, a decision is made to accept or reject the optimal network configuration recommendation; and Send a confirmation message to the cognitive function, the confirmation message containing at least one of the following: The decision to accept or reject the optimal network configuration recommendation; or Reasons for the decision.

3. The apparatus according to claim 1 or 2, wherein, The optimal network configuration recommendation includes at least one of the following: One or more optimal network configuration parameters suggested by the cognitive function; or The expected network performance corresponding to the one or more optimal network configuration parameters.

4. The apparatus according to any one of claims 1-3, wherein, When applying the optimal network configuration recommendation, the quality estimate of the optimal network configuration recommendation is represented by a prediction interval for network performance with a predetermined probability.

5. The apparatus according to any one of claims 1-4, wherein, The acceptance criterion for the optimal network configuration recommendation is represented by a threshold value for the prediction interval.

6. An apparatus comprising: At least one processor; as well as At least one memory, the at least one memory storing instructions that, when executed by the at least one processor, cause the device to perform at least the following: Send optimal network configuration recommendations to the network control function; and Receive an acknowledgment message from the network control function, the acknowledgment message containing at least one of the following: The decision to accept or reject the optimal network configuration recommendation; or Reasons for the decision.

7. The apparatus of claim 6, wherein the instructions, when executed by the at least one processor, further cause the apparatus to perform at least: The decision and the reasons for the decision are used as feedback information to improve the algorithm or model that generates the optimal network configuration recommendation.

8. An apparatus comprising: At least one processor; as well as At least one memory, the at least one memory storing instructions that, when executed by the at least one processor, cause the device to perform at least the following: Receive an evaluation request from the network control function for recommendations on optimal network configuration; Determine the quality estimate and acceptance criteria for the optimal network configuration recommendation; and An evaluation response is sent to the network control function, the evaluation response including the quality estimate and the acceptance criteria for the optimal network configuration recommendation.

9. The apparatus according to claim 8, wherein, The acceptance criterion for the optimal network configuration recommendation is determined to be inversely proportional to the difference between one or more optimal network configuration parameters indicated in the optimal network configuration recommendation and the corresponding network configuration parameters for the current network configuration.

10. The apparatus according to claim 8 or 9, wherein, The acceptance criteria for the optimal network configuration recommendation are determined to be proportional to the expected performance improvement when the optimal network configuration recommendation is applied.

11. The apparatus according to any one of claims 8-10, wherein, When executed by the at least one processor, the instructions also cause the device to perform at least the following: Receive network data from the network; and The received network data is used to determine the quality estimate and acceptance criteria for the optimal network configuration recommendation.

12. The apparatus according to any one of claims 8-11, wherein, When executed by the at least one processor, the instructions also cause the device to perform at least the following: Request the network data from the network.

13. An apparatus comprising: At least one processor; as well as At least one memory, the at least one memory storing instructions that, when executed by the at least one processor, cause the device to perform at least the following: Receive optimal network configuration recommendations from cognitive functions; Determine the quality estimate and acceptance criteria for the optimal network configuration recommendation; and The decision to accept or reject the optimal network configuration recommendation is based on the determined quality estimate and the acceptance criteria.

14. The apparatus according to claim 13, wherein, The instructions, when executed by the at least one processor, also cause the device to perform at least the following: Receive network data from the network; and The received network data is used to determine the quality estimate and acceptance criteria for the optimal network configuration recommendation.

15. The apparatus according to claim 13 or 14, wherein, The instructions, when executed by the at least one processor, also cause the device to perform at least the following: Request the network data from the network.

16. The apparatus according to any one of claims 13-15, wherein, When executed by the at least one processor, the instructions also cause the device to perform at least the following: Send a confirmation message to the cognitive function, the confirmation message containing at least one of the following: The decision to accept or reject the optimal network configuration recommendation; or Reasons for the decision.

17. The apparatus according to any one of claims 13-16, wherein, The acceptance criteria for the optimal network configuration recommendation are determined to be inversely proportional to the difference between one or more optimal network configuration parameters indicated in the optimal network configuration recommendation and the corresponding network configuration parameters for the current network configuration, and directly proportional to the expected performance improvement when the optimal network configuration recommendation is applied.

18. A method comprising: Receive optimal network configuration recommendations from cognitive functions; Send an evaluation request for the optimal network configuration recommendation to the reliability verification cognitive function; as well as The system receives an evaluation response from the reliability verification cognitive function, the evaluation response containing a quality estimate and acceptance criteria for the optimal network configuration recommendation.

19. The method of claim 18, further comprising: The decision to accept or reject the optimal network configuration recommendation will be based on the evaluation response. as well as Send a confirmation message to the cognitive function, the confirmation message containing at least one of the following: The decision to accept or reject the optimal network configuration recommendation; or Reasons for the decision.

20. The method according to claim 18 or 19, wherein, The optimal network configuration recommendation includes at least one of the following: One or more optimal network configuration parameters suggested by the cognitive function; or The expected network performance corresponding to the one or more optimal network configuration parameters.

21. The method according to any one of claims 18-20, wherein, When applying the optimal network configuration recommendation, the quality estimate of the optimal network configuration recommendation is represented by a prediction interval for network performance with a predetermined probability.

22. The method according to any one of claims 18-21, wherein, The acceptance criterion for the optimal network configuration recommendation is represented by a threshold value for the prediction interval.

23. A method comprising: Send optimal network configuration recommendations to the network control function; as well as Receive an acknowledgment message from the network control function, the acknowledgment message containing at least one of the following: The decision to accept or reject the optimal network configuration recommendation; or Reasons for the decision.

24. The method of claim 23, further comprising: The decision and the reasons for the decision are used as feedback information to improve the algorithm or model that generates the optimal network configuration recommendation.

25. A method comprising: Receive an evaluation request from the network control function for recommendations on optimal network configuration; Determine the quality estimate and acceptance criteria for the optimal network configuration recommendation; as well as An evaluation response is sent to the network control function, the evaluation response including the quality estimate and the acceptance criteria for the optimal network configuration recommendation.

26. The method according to claim 25, wherein, The acceptance criterion for the optimal network configuration recommendation is determined to be inversely proportional to the difference between one or more optimal network configuration parameters indicated in the optimal network configuration recommendation and the corresponding network configuration parameters for the current network configuration.

27. The method according to claim 25 or 26, wherein, The acceptance criteria for the optimal network configuration recommendation are determined to be proportional to the expected performance improvement when the optimal network configuration recommendation is applied.

28. The method according to any one of claims 25-27, further comprising: Receive network data from the network; as well as The received network data is used to determine the quality estimate and acceptance criteria for the optimal network configuration recommendation.

29. The method according to any one of claims 25-28, further comprising: Request the network data from the network.

30. A method comprising: Receive optimal network configuration recommendations from cognitive functions; Determine the quality estimate and acceptance criteria for the optimal network configuration recommendation; as well as The decision to accept or reject the optimal network configuration recommendation is based on the determined quality estimate and the acceptance criteria.

31. The method of claim 13, further comprising: Receive network data from the network; as well as The received network data is used to determine the quality estimate and acceptance criteria for the optimal network configuration recommendation.

32. The method according to claim 13 or 14, further comprising: Request the network data from the network.

33. The method according to any one of claims 30-32, further comprising: Send a confirmation message to the cognitive function, the confirmation message containing at least one of the following: The decision to accept or reject the optimal network configuration recommendation; or Reasons for the decision.

34. The method according to any one of claims 30-33, wherein, The acceptance criteria for the optimal network configuration recommendation are determined to be inversely proportional to the difference between one or more optimal network configuration parameters indicated in the optimal network configuration recommendation and the corresponding network configuration parameters for the current network configuration, and directly proportional to the expected performance improvement when the optimal network configuration recommendation is applied.

35. An apparatus for network control functions, comprising components for performing the method according to any one of claims 18-22, 30-34.

36. An apparatus for cognitive function, comprising components for performing the method according to any one of claims 23-24.

37. An apparatus for a reliability verification cognitive function, comprising components for performing the method according to any one of claims 25-29.

38. A computer-readable medium comprising instructions that, when executed by a means for network control functions, cause the means to perform the method according to any one of claims 18-22, 30-34.

39. A computer-readable medium comprising instructions that, when executed by a means for cognitive function, cause the means to perform the method according to any one of claims 23-24.

40. A computer-readable medium comprising instructions that, when executed by a means for a reliability verification cognitive function, cause the means to perform the method according to any one of claims 25-29.

Images