Method and apparatus to evaluate performance of a closed control loop in wireless communication system

Abstract

The disclosure relates to a 5G or 6G communication system for supporting a higher data transmission rate. Embodiments of the disclosure describe a method and system to evaluate the performance of a closed control loop (CCL). The method comprises: in case that a service level specification (SLS) performance starts degrading for a first metric, obtaining, from a MnS producer, first information on attributes of all assurance closed control loops (ACCLs); identifying a CCL which contains the first metric among the ACCLs, based on the first information; and obtaining, from the MnS producer, second information on performance metrics of the identified CCL.

Description

METHOD AND APPARATUS TO EVALUATE PERFORMANCE OF A CLOSED CONTROL LOOP IN WIRELESS COMMUNICATION SYSTEM

[0001] The present disclosure generally relates to a field of network automation, and more particularly to a method and apparatus to evaluate performance of a closed control loop (CCL).

[0002] 5G mobile communication technologies define broad frequency bands such that high transmission rates and new services are possible, and can be implemented not only in "Sub 6GHz" bands such as 3.5GHz, but also in "Above 6GHz" bands referred to as mmWave including 28GHz and 39GHz. In addition, it has been considered to implement 6G mobile communication technologies (referred to as Beyond 5G systems) in terahertz bands (for example, 95GHz to 3THz bands) in order to accomplish transmission rates fifty times faster than 5G mobile communication technologies and ultra-low latencies one-tenth of 5G mobile communication technologies.

[0003] At the beginning of the development of 5G mobile communication technologies, in order to support services and to satisfy performance requirements in connection with enhanced Mobile BroadBand (eMBB), Ultra Reliable Low Latency Communications (URLLC), and massive Machine-Type Communications (mMTC), there has been ongoing standardization regarding beamforming and massive MIMO for mitigating radio-wave path loss and increasing radio-wave transmission distances in mmWave, supporting numerologies (for example, operating multiple subcarrier spacings) for efficiently utilizing mmWave resources and dynamic operation of slot formats, initial access technologies for supporting multi-beam transmission and broadbands, definition and operation of BWP (BandWidth Part), new channel coding methods such as a LDPC (Low Density Parity Check) code for large amount of data transmission and a polar code for highly reliable transmission of control information, L2 pre-processing, and network slicing for providing a dedicated network specialized to a specific service.

[0004] Currently, there are ongoing discussions regarding improvement and performance enhancement of initial 5G mobile communication technologies in view of services to be supported by 5G mobile communication technologies, and there has been physical layer standardization regarding technologies such as V2X (Vehicle-to-everything) for aiding driving determination by autonomous vehicles based on information regarding positions and states of vehicles transmitted by the vehicles and for enhancing user convenience, NR-U (New Radio Unlicensed) aimed at system operations conforming to various regulation-related requirements in unlicensed bands, NR UE Power Saving, Non-Terrestrial Network (NTN) which is UE-satellite direct communication for providing coverage in an area in which communication with terrestrial networks is unavailable, and positioning.

[0005] Moreover, there has been ongoing standardization in air interface architecture / protocol regarding technologies such as Industrial Internet of Things (IIoT) for supporting new services through interworking and convergence with other industries, IAB (Integrated Access and Backhaul) for providing a node for network service area expansion by supporting a wireless backhaul link and an access link in an integrated manner, mobility enhancement including conditional handover and DAPS (Dual Active Protocol Stack) handover, and two-step random access for simplifying random access procedures (2-step RACH for NR). There also has been ongoing standardization in system architecture / service regarding a 5G baseline architecture (for example, service based architecture or service based interface) for combining Network Functions Virtualization (NFV) and Software-Defined Networking (SDN) technologies, and Mobile Edge Computing (MEC) for receiving services based on UE positions.

[0006] As 5G mobile communication systems are commercialized, connected devices that have been exponentially increasing will be connected to communication networks, and it is accordingly expected that enhanced functions and performances of 5G mobile communication systems and integrated operations of connected devices will be necessary. To this end, new research is scheduled in connection with eXtended Reality (XR) for efficiently supporting AR (Augmented Reality), VR (Virtual Reality), MR (Mixed Reality) and the like, 5G performance improvement and complexity reduction by utilizing Artificial Intelligence (AI) and Machine Learning (ML), AI service support, metaverse service support, and drone communication.

[0007] Furthermore, such development of 5G mobile communication systems will serve as a basis for developing not only new waveforms for providing coverage in terahertz bands of 6G mobile communication technologies, multi-antenna transmission technologies such as Full Dimensional MIMO (FD-MIMO), array antennas and large-scale antennas, metamaterial-based lenses and antennas for improving coverage of terahertz band signals, high-dimensional space multiplexing technology using OAM (Orbital Angular Momentum), and RIS (Reconfigurable Intelligent Surface), but also full-duplex technology for increasing frequency efficiency of 6G mobile communication technologies and improving system networks, AI-based communication technology for implementing system optimization by utilizing satellites and AI (Artificial Intelligence) from the design stage and internalizing end-to-end AI support functions, and next-generation distributed computing technology for implementing services at levels of complexity exceeding the limit of UE operation capability by utilizing ultra-high-performance communication and computing resources.

[0008] A wireless 5G system consists of a 5G Access Network (AN), 5G Core Network (CN) and User Equipment (UE). The Wireless 5G system is expected to provide optimized support for a variety of different communication services, different traffic loads, and different end user communities. For example, communication services using network slicing may include vehicle to everything (V2X) services. The wireless 5G system aims to enhance its capability to meet key performance indicators (KPIs) that emerging V2X applications require. For these advanced applications, the requirements, such as data rate, reliability, latency, communication range and speed, are made more stringent.

[0009] As one of the key technologies to enable network slicing is fixed mobile convergence (FMC). FMC includes wireless-to-the-everything (WTTx) and fibre-to-the-everything (FTTx) and is expected to provide native support for network slicing. For optimization and resource efficiency, 5G wireless system will select a most appropriate third generation partnership project (3GPP) or non-3GPP access technology for a communication service, potentially allowing multiple access technologies to be used simultaneously for one or more services active on a UE and massive internet of things (mIoT) connections. Support for mIoT brings many new requirements in addition to mobile broad band (MBB) enhancements. Communication services with massive IoT connections such as smart households, smart grid, smart agriculture and smart meter will require the support of a large number and high density IoT devices to be efficient and cost effective. Operators may use one or more network slice instances to provide these communication services, which require similar network characteristics, to different vertical industries.

[0010] 3GPP TS 28.530 and 3GPP TS 28.531 define the management of network slice in 5G networks. They also define the concept of communication services, which are provided using one or multiple network slices. Network slice instance (NSI) may support multiple communication service instances (CSI). Similarly, a CSI may utilize multiple NSIs. The present work on CSI Service Level Specification (SLS) assurance is undergoing as part of eCOSLA (Closed Loop SLS Assurance) work in 3GPP SA5 working group.

[0011] Closed control loops (CCLs) are being defined where there is no direct involvement of a human operator or other management entity in the control loop, i.e., the control loop is fully automated. The human operator or management entity may not directly control the details inside the process steps but may provide control outside the loop. For example, configuring goals for the control loop to make autonomous decisions within the boundaries of the set goal. Once the control loop is configured with the goal, the controlled entity is adjusted according to the set goals. In a CCL, the input to the control loop provided by human operator or other management entity may include the goal or policies. The output of the CCL may include CCL status to a human operator or other management entity. Typically, the goal is set within certain parameter boundaries, the CCL may automatically monitor the network and ascertain if the defined goals are being breached. If the goal is breached the loop may re-configure the network to mitigate the breach.

[0012] However, there is no mechanism to evaluate the performance of the deployed CCL itself while in its operational phase to achieve a particular assurance goal for a desired SLS performance. This limitation highlights a critical gap in the ability to monitor and validate whether the deployed CCL in the communication network is functioning effectively to pursue its goals. Furthermore, a consumer may want to know the performance of the CCL and decide upon its merit with respect to other vendor's CCLs to choose the best one for deployment in its network.

[0013] Therefore, there lies at least a need to provide a solution to one or more aforesaid problems or improve an existing solution, product, or process.

[0014] The information disclosed in this background of the disclosure section is only for enhancement of understanding of the general background of the invention and should not be taken as an acknowledgement or any form of suggestion that this information forms the prior art already known to a person skilled in the art.

[0015] This summary is provided to introduce a selection of concepts, in a simplified format, which is further described in the detailed description of the invention. This summary is neither intended to identify key or essential inventive concepts of the invention nor is it intended for determining the scope of the invention.

[0016] The present disclosure describes a method and apparatus to evaluate performance of a closed control loop (CCL). In an embodiment of the present disclosure the method comprising: in case that a service level specification (SLS) performance starts degrading for a first metric, obtaining, from a MnS producer, first information on attributes of all assurance closed control loops (ACCLs); identifying a CCL which contains the first metric among the ACCLs, based on the first information; and obtaining, from the MnS producer, second information on performance metrics of the identified CCL.

[0017] The present disclosure describes a method and apparatus to evaluate performance of a closed control loop (CCL). In an embodiment of the present disclosure the method comprising: in case that a service level specification (SLS) performance starts degrading for a first metric, transmitting, to a MnS consumer, first information on attributes of all assurance closed control loops (ACCLs); and transmitting, to the MnS consumer, second information on the performance metrics of the an identified selected CCL which contains the first metric among the ACCLs based on the first information.

[0018] The present disclosure describes a method and apparatus to evaluate performance of a closed control loop (CCL). In an embodiment of the present disclosure the apparatus comprises at least one processor and a memory coupled to the at least one processor. The at least one processor may be configured to send a first request, to a Management service (MnS) producer, to retrieve attributes of a plurality of CCLs associated with the MnS consumer, when a desired Service Level Specification (SLS) performance degrades. The at least one processor may be then configured to receive the attributes of the plurality of CCLs from the MnS producer and identify a CCL among the plurality of CCLs responsible for maintaining the desired SLS performance based on the received attributes. Further, the at least one processor may be configured to send a second request, to the MnS producer, to obtain performance metrics of the identified CCL. For example, the performance metrics are TotalAssuranceGoalBreach, TimeBreachedGoalRecovery, and TotalCclConflicts_Filter. The at least one processor may be then configured to receive the performance metrics of the identified CCL from the MnS producer and evaluate performance of the identified CCL based on the received performance metrics. The received performance metrics TotalAssuranceGoalBreach, TimeBreachedGoalRecovery, and TotalCclConflicts_Filter indicate a total number of occurrences of an assurance goal breach, time taken by the CCL to meet a breached goal, and a total number of conflicts occurred by the identified CCL, respectively.

[0019] In another non-limiting embodiment of the present disclosure, the number of occurrences of goal breach as indicated by the performance metric TotalAssuranceGoalBreach provides measurement of a total number of occurrences of breach of an assurance goal, happened in the identified CCL, during an observation period. The number of occurrences of the assurance goal breach is measured by counting each incidence when the assurance goal is breached and incrementing a corresponding cumulative counter by one for each such occurrence within the observation period.

[0020] In yet another non-limiting embodiment of the present disclosure, the time taken to meet the breached goal as indicated by the performance metric TimeBreachedGoalRecovery provides measurement of time taken by the identified CCL to meet the breached goal after re-activating the identified CCL.

[0021] In yet another non-limiting embodiment of the present disclosure, the time taken to meet the breached goal is indicated by a difference between a first time stamp and a second time stamp. For example, the first time stamp represents a time instance when an assurance goal is breached and the second time stamp represents a time instance when the assurance goal is met after re-activating the identified CCL with required changes.

[0022] In yet another non-limiting embodiment of the present disclosure, the number of conflicts occurred by the CCL as indicated by the performance metric TotalCclConflicts_Filter provides measurement of a total number of conflicts that occur between the identified CCL and one or more CCLs other than the identified CCL during an observation period.

[0023] In yet another non-limiting embodiment of the present disclosure, the number of conflicts occurred by the CCL is measured by counting each incidence when a conflict occurs between the identified CCL and the one or more CCLs other than the identified CCL, and incrementing a corresponding cumulative counter by one for each such occurrence within the observation period.

[0024] In yet another non-limiting embodiment of the present disclosure, the conflict is at least one of an implicit conflict and an explicit conflict. For example, the implicit conflict represents an action conflict between the identified CCL and one or more existing CCLs other than identified CCL, and the explicit conflict represents a conflict between the identified CCL and a new requested CCL.

[0025] In yet another non-limiting embodiment of the present disclosure, based on the evaluation of the performance of the identified CCL, the at least one processor is configured to either send a modification request, to the MnS producer, to modify one or more parameters of the identified CCL for the desired SLS, or send a creation request, to the MnS producer, to create a new CCL for the desired SLS. In an embodiment, upon sending the modification request, a modification response is received from the MnS producer, containing information about modifications done for the identified CCL. In an embodiment, upon sending the creation request, a creation response, is received from the MnS producer, for the created new CCL for the desired SLS.

[0026] In yet another embodiment of the present disclosure, the present disclosure describes a yet another apparatus to evaluate performance of a closed control loop (CCL). In an embodiment of the present disclosure the apparatus comprises at least one processor and a memory coupled to the at least one processor. The at least one processor may be configured to receive a first request, from a MnS consumer, to retrieve attributes of a plurality of CCLs associated with the MnS consumer, when a desired Service Level Specification (SLS) performance degrades and send the retrieved attributes of the plurality of CCLs to the MnS consumer. The at least one processor may be further configured to receive a second request, from the MnS consumer, to obtain performance metrics of a selected CCL among the plurality of CCLs and send the performance metrics of the selected CCL to the MnS Consumer. For example, the performance metrics are TotalAssuranceGoalBreach, TimeBreachedGoalRecovery, and TotalCclConflicts_Filter. The performance metrics TotalAssuranceGoalBreach, TimeBreachedGoalRecovery, and TotalCclConflicts_Filter indicate a total number of occurrences of an assurance goal breach, time taken by the CCL to meet a breached goal, and a total number of conflicts occurred by the selected CCL, respectively.

[0027] In yet another non-limiting embodiment of the present disclosure, the number of occurrences of goal breach as indicated by the performance metric TotalAssuranceGoalBreach provides measurement of a total number of occurrences of breach of an assurance goal, happened in the selected CCL, during an observation period. The number of occurrences of the assurance goal breach is measured by counting each incidence when the assurance goal is breached and incrementing a corresponding cumulative counter by one for each such occurrence within the observation period.

[0028] In yet another non-limiting embodiment of the present disclosure, the time taken to meet the breached goal as indicated by the performance metric TimeBreachedGoalRecovery provides measurement of time taken by the selected CCL to meet the breached goal after re-activating the selected CCL.

[0029] In yet another non-limiting embodiment of the present disclosure, to measure the time taken to meet the breached goal, the at least one processor is configured to monitor a first time stamp when an assurance goal is breached and monitor a second time stamp　when the assurance goal is met after re-activating the selected CCL. The at least one processor is further configured to calculate a difference between the first time stamp and the second time stamp. For example, the difference represents the time taken to meet the breached goal.

[0030] In yet another non-limiting embodiment of the present disclosure, the number of conflicts occurred by the CCL as indicated by the performance metric TotalCclConflicts_Filter provides measurement of a total number of conflicts that occur between the selected CCL and one or more CCLs other than the selected CCL during an observation period.

[0031] In yet another non-limiting embodiment of the present disclosure, the number of conflicts occurred by the CCL is measured by counting each incidence when a conflict occurs between the selected CCL and the one or more CCLs other than the selected CCL, and incrementing a corresponding cumulative counter by one for each such occurrence within the observation period.

[0032] In yet another non-limiting embodiment of the present disclosure, the conflict is at least one of an implicit conflict and an explicit conflict. For example, the implicit conflict represents an action conflict between the selected CCL and one or more existing CCLs other than selected CCL, and the explicit conflict represents a conflict between the selected CCL and a new requested CCL.

[0033] In yet another non-limiting embodiment of the present disclosure, the at least one processor is further configured to either receive a modification request, from the MnS consumer, to modify one or more parameters of the identified CCL for the desired SLS, or receive a creation request, from the MnS consumer, to create a new CCL for the desired SLS. In an embodiment, upon receiving the modification request, a modification response is sent, to the MnS consumer, containing information about modifications done for the identified CCL. In an embodiment, in response to receiving the creation request, a creation response is sent to the MnS consumer, for the created new CCL for the desired SLS.

[0034] In yet another non-limiting embodiment, the present disclosure describes a method of evaluating performance of a closed control loop (CCL), being performed by a Management service (MnS) consumer in a communication network. The method comprises sending a first request, to a MnS producer, to retrieve attributes of a plurality of CCLs associated with the MnS consumer, when a desired Service Level Specification (SLS) performance degrades. The method further comprises receiving the attributes of the plurality of CCLs from the MnS producer and identifying a CCL among the plurality of CCLs responsible for maintaining the desired SLS performance based on the received attributes. The method further comprises sending a second request, to the MnS producer, to obtain performance metrics of the identified CCL. For example, the performance metrics are TotalAssuranceGoalBreach, TimeBreachedGoalRecovery, and TotalCclConflicts_Filter. Further the method comprises receiving the performance metrics of the identified CCL from the MnS producer. The received performance metrics TotalAssuranceGoalBreach, TimeBreachedGoalRecovery, and TotalCclConflicts_Filter indicates a total number of occurrences of an assurance goal breach, time taken by the CCL to meet a breached goal, and a total number of conflicts occurred by the identified CCL, respectively. The method further comprises evaluating performance of the identified CCL based on the received performance metrics.

[0035] In yet another non-limiting embodiment of the present disclosure, the present disclosure describes a method of evaluating performance of a closed control loop (CCL), being performed by a Management service (MnS) producer in a communication network. The method comprises receiving a first request, from a MnS consumer, to retrieve attributes of a plurality of CCLs associated with the MnS consumer, when a desired Service Level Specification (SLS) performance degrades and sending the retrieved attributes of the plurality of CCLs to the MnS consumer. The method further comprises receiving a second request, from the MnS consumer, to obtain performance metrics of a selected CCL among the plurality of CCLs. For example, the performance metrics are TotalAssuranceGoalBreach, TimeBreachedGoalRecovery, and TotalCclConflicts_Filter. The method further comprises sending the performance metrics of the selected CCL to the MnS Consumer. The performance metrics TotalAssuranceGoalBreach, TimeBreachedGoalRecovery, and TotalCclConflicts_Filter indicate a total number of occurrences of an assurance goal breach, time taken by the CCL to meet a breached goal, and a total number of conflicts occurred by the selected CCL, respectively.

[0036] To further clarify the advantages and features of the present invention, a more particular description of the invention will be rendered by reference to specific embodiments thereof, which is illustrated in the appended drawings. It is appreciated that these drawings depict only typical embodiments of the invention and are therefore not to be considered limiting of its scope. The invention will be described and explained with additional specificity and detail with the accompanying drawings.

[0037] The embodiments of the disclosure itself, as well as a preferred mode of use, further objectives and advantages thereof, will best be understood by reference to the following detailed description of an illustrative embodiment when read in conjunction with the accompanying drawings. One or more embodiments are now described, by way of example only, with reference to the accompanying drawings in which:

[0038] Figure 1 illustrates an architecture of a closed control loop (CCL) governance and monitoring, in accordance with an existing art;

[0039] Figure 2 illustrates a block diagram of an apparatus to evaluate performance of a closed control loop (CCL), in accordance with an embodiment of the present disclosure;

[0040] Figure 3 illustrates a yet another block diagram of an apparatus to evaluate performance of a closed control loop (CCL), in accordance with an embodiment of the present disclosure;

[0041] Figure 4 illustrates a method of evaluating performance of a closed control loop (CCL), being performed by a Management service (MnS) consumer in a communication network, in accordance with an embodiment of the present disclosure;

[0042] Figure 5 illustrates a method of evaluating performance of a closed control loop (CCL), being performed by a Management service (MnS) producer in a communication network, in accordance with an embodiment of the present disclosure; and

[0043] Figure 6 illustrates a sequence flow diagram between the Management service (MnS) consumer and the Management service (MnS) producer to evaluate performance of the closed control loop (CCL), in accordance with an embodiment of the present disclosure.

[0044] The figures depict embodiments of the disclosure for purposes of illustration only. One skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles of the disclosure described herein.

[0045] It should be appreciated by those skilled in art that any block diagrams herein represent conceptual views of illustrative systems embodying the principles of the present subject matter. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, pseudo code, and the like represent various processes which may be substantially represented in computer readable medium and executed by a computer or processor, whether or not such computer or processor is explicitly shown.

[0046] As disclosed in various studies, 5G system consists of 5G Access Network (AN), 5G Core Network and UE (For instance, see TS 23.501 [3]). 5G system is expected to be able to provide optimized support for a variety of different communication services, different traffic loads, and different end user communities. For example, the communication services using network slicing may include V2X services. The 5G system aims to enhance its capability to meet KPIs that emerging V2X applications require. For these advanced applications, the requirements, such as data rate, reliability, latency, communication range and speed, are made more stringent, 5G seamless eMBB. As one of the key technologies to enable network slicing, fixed mobile convergence (FMC) which includes wireless-to-the-everything (WTTx) and fibre-to-the- everything (FTTx), is expected to provide native support for network slicing. For optimization and resource efficiency, the 5G system will select the most appropriate 3GPP or non-3GPP access technology for a communication service, potentially allowing multiple access technologies to be used simultaneously for one or more services active on a UE, massive IoT connections. Support for massive Internet of Things (mIoT) brings many new requirements in addition to MBB enhancements. Communication services with massive IoT connections such as smart households, smart grid, smart agriculture and smart meter will require the support of a large number and high density IoT devices to be efficient and cost effective. Operators may use one or more network slice instances to provide these communication services, which require similar network characteristics, to different vertical industries. 3GPP TS 28.530 and 28.531 define the management of Network Slice in 5G networks. It also defined the concept of Communication Services, which are provided using one or multiple Network Slice. A Network Slice Instance (NSI) may support multiple Communication Service Instances (CSI). Similarly, a CSI may utilize multiple NSIs.

[0047] The present work on CSI SLS assurance is undergoing as part of eCOSLA (Closed Loop SLS Assurance) work in 3GPP SA5 working group. The closed control loops are being defined where there is no direct involvement of a human operator or other management entity in the control loop, the control loop is fully automated. The human operator or management entity is not directly controlling the details inside the process steps but provides control outside the loop. For example, configuring goals for the control loop to make autonomous decisions within the boundaries of the set goal. Once the control loop is configured with the goal, the controlled entity is adjusted according to the set goals. In a closed control loop the input to the control loop provided by human operator or other management entity may include the goal or policies. The output of the closed control loop may include closed control loop status to a human operator or other management entity. Typically, the goal is set within certain parameter boundaries, the closed control loop may automatically monitor the network and ascertain if the defined goals are being breached. If the goal is breached the loop may re-configure the network to mitigate the breach.

[0048] The present disclosure relates generally to a field of network automation, and more particularly to a system and a method for providing performance evaluation of a closed control loop.

[0049] It proposes to address the problem where the monitoring activity for a Closed Control Loop may result in further actions that happen in the operation phase, e.g. evaluate and update, in order to change the closed control loop settings and improve its performance. But currently there is no method and metrics to evaluate performance of a Closed Control Loop itself. Such metrics are important to understand and change a CCL's behaviour and to improve its performance to pursue the assigned goal(s). Furthermore, a consumer may want to know the performance of a CCL and decide upon its merit with respect to other vendor's CCLs to choose the best one for deployment in its network.

[0050] To address the above identified challenges, the present disclosure aims to provide a new performance metric, in one non-limiting embodiment, which may be defined to determine total number of incidences when an assurance goal is breached by a CCL during an observation time period. In another non-limiting embodiment, a new performance metric may be defined to determine time taken by a CCL to meet a breached goal after its reactivation with required changes. In yet another non-limiting embodiment, a new performance metric may be defined to determine the total number of conflicts that occur between a CCL under consideration and any other CCL during an observation time period, thereby facilitating better performance evaluation of a CCL by the management consumer. Based on the performance of the existing CCL, CCLs may be updated or new CCLs may be created.

[0051] The terms "CCL (Closed Control Loop)" and "ACCL (Assurance Closed Control Loop)" have been used interchangeably in the present disclosure. The invention includes defining new metrics to evaluate performance of a CCL for its optimal execution. This enables operators to track the effectiveness of closed loop automation, identify areas for improvement, and make informed adjustments to CCL functionalities.

[0052] These new metrics are as follows -

[0053] 1) Total number of occurrences of an assurance goal breach

[0054] 2) Time taken by CCL to meet a breached goal

[0055] 3) Total number of conflicts occurred by a CCL

[0056] In one non-limiting embodiment the new metrics "Total number of occurrences of an assurance goal breach" may be defined as follows:-

[0057] a) This measurement provides the total number of incidences when an assurance goal, as defined in CCL is breached during an observation time period.

[0058] b) CC.

[0059] c) This is measured by counting each incidence when an assurance goal is breached and incrementing the corresponding counter by one for each such occurrence within an observation time period.

[0060] d) An integer value.

[0061] e) The measurement name has the form TotalAssuranceGoalBreach.

[0062] f) CCL Provider.

[0063] g) Valid for packet switched traffic.

[0064] h) 5GS.

[0065] In another non-limiting embodiment the new metrics "Time taken by CCL to meet a breached goal may be defined as follows:-

[0066] a) This measurement provides the time taken by a CCL to meet a breached goal after its reactivation.

[0067] b) DER.

[0068] c) This is measured by considering the time stamp when an assurance goal is breached and subtracting it from the time stamp when that goal is met after re-activating the CCL with required changes.

[0069] d) Each measurement is an integer representing the mean delay in milliseconds.

[0070] e) The measurement name has the form TimeBreachedGoalRecovery.

[0071] f) CCL Provider.

[0072] g) Valid for packet switched traffic.

[0073] h) 5GS.

[0074] In yet another non-limiting embodiment the new metrics "Total number of conflicts occurred by a CCL" may be defined as follows:-

[0075] a) This measurement provides the the total number of conflicts that occur between a CCL under consideration and any other CCL during an observation time period.

[0076] b) CC

[0077] c) This is measured by counting each incidence when conflict occurs between a CCL under consideration and the other CCL and incrementing the corresponding counter by one for each such occurrence within an observation time period.

[0078] d) An integer value.

[0079] e) The measurement name has the form TotalCclConflicts_Filter, where filter is either Implicit or Explicit. Implicit represents the implicit conflict i.e. conflict between two existing CCL and Explicit represents the explicit conflict i.e. conflict between an existing CCL and a requested CCL.

[0080] f) CCL Provider

[0081] g) Valid for packet switched traffic

[0082] h) 5GS

[0083] In view of this, in an embodiment, the present invention illustrates a procedural flow step for performance evaluation of a closed control loop.

[0084] 1. Step 1 - PA / CCL MnS consumer notices that SLS performance starts degrading for certain metric.

[0085] 2. Step 2 - PA / CCL MnS consumer sends getMOIAttributeRequest message to PA / CCL MnS producer for getting information about all ACCLs attributes.

[0086] 3. Step 3 - PA / CCL MnS producer provides this information of all ACCLs to the consumer in getMOIAttributeResponse message.

[0087] 4. Step 4 - PA / CCL MnS consumer identifies the ACCL (n) which contains that SLS metric.

[0088] 5. Step 5 - PA / CCL MnS consumer sends createMOI (PerfMetricJob) request to PA / CCL MnS producer for obtaining status of following performance metrics (as defined by this disclosure) - TotalAssuranceGoalBreach, TimeCorrectiveGoalMeet, TotalCclConflicts_Filter) for that particular ACCL n (as identified in step 4).

[0089] 6. Step 6 - PA / CCL MnS producer provides requested performance metric values via createMOI () Response message to PA / CCL MnS consumer.

[0090] After this step the PA / CCL MnS consumer has two choices -

[0091] Choice 1:

[0092] 1. Step 7 - PA / CCL MnS consumer updates an existing ACCL n (of step 4) by sending modifyMOIAttributes request message for that ACCL or it may also update by sending changeMOIs request message to PA / CCL MnS producer.

[0093] 2. Step 8 - Accordingly, PA / CCL MnS producer sends modifyMOIAttributes Response or changeMOIs response message to PA / CCL MnS consumer for the updated attributes of ACCL n.

[0094] Choice 2:

[0095] 1. Step 9 - If PA / CCL MnS consumer chooses to create a new ACCL for the desired SLS / assurance goal, it does so by sending createMOI Request message to PA / CCL MnS producer.

[0096] 2. Step 10 - PA / CCL MnS producer provides createMOI () Response message for the newly created ACCL MOI to PA / CCL MnS consumer.

[0097] Technical Advantages offered by the present disclosure:

[0098] The invention may enable:

[0099] ●CCL consumer to obtain performance of a CCL with respect to total number of assurance goal breached over a period of time.

[0100] ●CCL consumer to obtain performance of a CCL with respect to time taken to meet a breached goal after its reactivation with required changes.

[0101] ●CCL consumer to obtain performance of a CCL with respect to total number of conflicts occurred by a CCL over a period of time.

[0102] ●CCL consumer to compare different vendors' CCLs for these performance areas and choose the best one for its network deployment.

[0103] All this may result in better performance evaluation of a CCL by the management consumer. Based on the performance of the existing CCL, CCLs may be updated or new CCLs may be created.

[0104] In the present document, the word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or implementation of the present subject matter described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

[0105] While the disclosure is susceptible to various modifications and alternative forms, specific embodiment thereof has been shown by way of example in the drawings and will be described and will be described in detail below. It should be understood that, however it is not intended to limit the disclosure is to cover all modifications, equivalents, and alternatives falling within the spirit and the scope of the disclosure.

[0106] The terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a setup, device or method that comprises a list of components or steps does not include only those components or steps but may include other components or steps not expressly listed in a device or system or apparatus proceeded by "comprises...a" does not, without more constraints, preclude the existence of other elements or additional elements in the device or system or apparatus.

[0107] In the following detailed description of the embodiments of the disclosure, reference is made to the accompanying drawings that form a part hereof, and which are shown by way of illustration specific embodiments in which the disclosure may be practiced. These embodiments are described in sufficient detail to enable those skilled in art to practice the disclosure, and it is to be understood that other embodiments may be utilized and that changes may be made without departing from the scope of the disclosure. The following description is, therefore, not to be taken in a limiting sense.

[0108] Embodiments of the present disclosure describe method(s) and apparatus(es) to evaluate performance of a closed control loop (CCL). It may be noted that the terms "CCL (Closed Control Loop)" and "ACCL (Assurance Closed Control Loop)" have been used interchangeably in the present disclosure.

[0109] Figure 1 illustrates an architecture 100 of a closed control loop (CCL) governance and monitoring, in accordance with an existing art. In the architecture 100, the CCL may be viewed as an entity that monitors and regulates a set of managed entities with an objective of achieving a specific goal to be managed. However, some management capabilities (e.g. closed control loop governance and closed control loop monitoring) may be exposed by a MnS producer 104, which implements the CCLs to enable a MnS consumer 102 to manage the CCLs 106. The CCL governance describes a set of capabilities to allow the MnS consumer 102 to govern the CCL 106, including lifecycle management of the CCL 106, including create, modify, activate / deactivate, delete the CCL 106 and configure goals for the CCL 106. Further, closed control loop monitoring describes a set of capabilities to allow the MnS consumer 102 to monitor the progress and result of the CCL 106, which includes but not limited to, monitoring the goal fulfillment of the CCL 106.

[0110] However, there is no mechanism to evaluate the performance of the CCL 106 itself while in its operational phase, to achieve a particular assurance goal for a desired SLS performance. The present disclosure defines new performance metrics to evaluate performance of the deployed CCL in the communication network for its optimal functioning in pursuing its goals.

[0111] Figure 2 illustrates a block diagram of an apparatus 200 to evaluate performance of a closed control loop (CCL), in accordance with an embodiment of the present disclosure. The apparatus 200 may comprise a processing unit 202 comprising at least one processor, a transceiver comprising an input / output (I / O) interface 204, and a memory 206, but not limited thereto. The processing unit 202 may comprise at least one processor for executing program components for executing user or system-generated processes. The processing unit 202 may comprise specialized processing units such as integrated system (bus) controllers, memory management control units, digital signal processing units, etc.

[0112] The processing unit 202 may be disposed in communication with one or more input / output (I / O) devices via transceiver comprising the I / O interface 204. The I / O interface 204 may employ communication protocols / methods such as, without limitation, audio, analog, digital, stereo, IEEE-1394, serial bus, Universal Serial Bus (USB), infrared, PS / 2, BNC, coaxial, component, composite, Digital Visual Interface (DVI), high-definition multimedia interface (HDMI), Radio Frequency (RF) antennas, S-Video, Video Graphics Array (VGA), IEEE 802.n / b / g / n / x, Bluetooth, cellular (e.g., Code-Division Multiple Access (CDMA), High-Speed Packet Access (HSPA+), Global System For Mobile Communications (GSM), Long-Term Evolution (LTE) or the like), etc. Using the I / O interface 204, the apparatus 200 may communicate with one or more I / O devices.

[0113] The apparatus 200 may also include suitable logic, circuitry, and interfaces that may be configured to evaluate the performance of the CCL 106. In an embodiment, the apparatus 200 may be implemented in a computing device. A person skilled in the art may appreciate that the computing device may be a smartphone, a cellular phone, a mobile phone, a mainframe machine, a computer workstation, a laptop and / or a consumer electronic (CE) device. Moving on, a detailed explanation of the working of the apparatus 200 is explained in forthcoming paragraphs.

[0114] In an embodiment, the processor 202 of the apparatus 200 may send a first request, to the Management service (MnS) producer 104, to retrieve attributes of a plurality of CCLs associated with the MnS consumer 102, when a desired Service Level Specification (SLS) performance degrades. Further, the processor 202 may receive the attributes of the plurality of CCLs from the MnS producer 104. In one implementation, the received attributes may be stored in the memory 206. Further, the processor 202 may identify a CCL 106 among the plurality of CCLs responsible for maintaining the desired SLS performance based on the received attributes and send a second request, to the MnS producer 104, to obtain performance metrics of the identified CCL 106. For example, the performance metrics may be TotalAssuranceGoalBreach, TimeBreachedGoalRecovery, and TotalCclConflicts_Filter. The processor 202 may receive the performance metrics of the identified CCL 106 from the MnS producer 104. The received performance metrics TotalAssuranceGoalBreach, TimeBreachedGoalRecovery, and TotalCclConflicts_Filter may indicate a number of occurrences of goal breach, time taken to meet a breached goal, and a number of conflicts occurred by the identified CCL 106, respectively. The processor 202 may then evaluate performance of the identified CCL 106 based on the received performance metrics.

[0115] In an embodiment, the processor 202 may be configured to send the first request, to the Management service (MnS) producer 104, to retrieve attributes of the plurality of CCLs associated with the MnS consumer 102, when the desired Service Level Specification (SLS) performance degrades. The 3rdGeneration Partnership Project (3GPP)　Technical Specifications (TS 32.156) defines semantic of a Network Resource Model (NRM) implemented by the MnS Producer, such as an information object class (IOC), Attribute, Attribute properties etc. The MnS consumer 102 may want to evaluate the performance of the CCL 106 itself. So, whenever the MnS consumer 102 feels that there is a performance degradation for the desired SLS performance, the MnS consumer 102 may request the MnS producer 104 to retrieve attributes of the plurality of CCLs associated with the MnS consumer 102.

[0116] In an embodiment, the processor 202 may be configured to receive the attributes of the plurality of CCLs from the MnS producer 104. Further the processor 202 may be then configured to identify a CCL 106 among the plurality of CCLs responsible for maintaining the desired SLS performance based on the attributes received.

[0117] In an embodiment, the processor 202 may be configured to send the second request, to the MnS producer 104, to obtain the performance metrics of the identified CCL 106. In an embodiment of the present disclosure, the performance metrics may be TotalAssuranceGoalBreach, TimeBreachedGoalRecovery, and TotalCclConflicts_Filter, as explained in subsequent paragraphs.

[0118] In an embodiment, the number of occurrences of goal breach as indicated by the performance metric TotalAssuranceGoalBreach may provide measurement of a total number of occurrences of breach of an assurance goal, happened in the identified CCL 106, during an observation period. The number of occurrences of the assurance goal breach may be measured by counting each incidence when the assurance goal is breached and incrementing a corresponding cumulative counter (CC) by one to get an integer value for each such occurrence within the observation period. It may be noted that the CC is reset to a well-defined value (usually "0") at the beginning of the measurement and further incremented by one when the assurance goal is breached.

[0119] In an embodiment, the time taken to meet the breached goal as indicated by the performance metric TimeBreachedGoalRecovery may provide measurement of time taken by the identified CCL 106 to meet the breached goal after re-activating the identified CCL 106. The time taken to meet the breached goal may be indicated by a difference between a first time stamp and a second time stamp. For example, the first time stamp may represent a time instance when an assurance goal is breached and may be captured in a discrete event registration (DER). Further the second time stamp　may represent a time instance when the assurance goal is met after re-activating the identified CCL 106 with required changes and may be captured in a DER.　In an embodiment, each measurement in the TimeBreachedGoalRecovery performance metric is an integer representing the mean delay in milliseconds.

[0120] In an embodiment, the number of conflicts occurred by the CCL 106 as indicated by the performance metric TotalCclConflicts_Filter may provide measurement of a total number of conflicts that occur between the identified CCL and one or more CCLs other than the identified CCL during an observation period. The number of conflicts occurred by the CCL 106 may be measured by counting each incidence when a conflict occurs between the identified CCL and the one or more CCLs other than the identified CCL and incrementing a corresponding cumulative counter (CC) by one to get an integer value for each such occurrence within the observation period. It may be noted that the CC is reset to a well-defined value (usually "0") at the beginning of the measurement and further incremented by one, when a conflict occurs between the identified CCL and the one or more CCLs.

[0121] In an embodiment, the number of conflicts occurred by the CCL 106 as indicated by the performance metric TotalCclConflicts_Filter may be at least implicit conflict or explicit conflict. The implicit conflict may represent an action conflict between the identified CCL 106 and one or more existing CCLs other than identified CCL 106, and the explicit conflict may represent a conflict between the identified CCL 106 and a new requested CCL 106.

[0122] In an embodiment, the processor 202 may be configured to receive the performance metrics of the identified CCL 106 from the MnS producer 104. The received performance metrics TotalAssuranceGoalBreach, TimeBreachedGoalRecovery, and TotalCclConflicts_Filter may indicate a total number of occurrences of the assurance goal breach, time taken by the CCL to meet a breached goal, and a total number of conflicts occurred by the identified CCL 106, respectively.

[0123] In an embodiment, the processor 202 may be then configured to evaluate the performance of the identified CCL 106 based on the received performance metrics. Based on the performance evaluation of the identified CCL 106, the processor 202 may be further configured either to send a modification request, to the MnS producer 104, to modify one or more parameters of the identified CCL 106 or send a creation request, to the MnS producer 104, to create a new CCL 106 for the desired SLS, as described below in subsequent paragraphs.

[0124] In an embodiment, the processor 202 may be configured to update the identified CCL 106 by sending a modifyMOIAttributes request message for that identified CCL 106, or the processor 202 may be configured to update by sending changeMOIs request message to performance assurance (PA) / CCL MnS producer 104, for the identified CCL 106. Accordingly, the MnS producer 104 may send a modifyMOIAttributes Response or changeMOIs response message to the processor 202 for the updated attributes of the selected CCL 106.

[0125] In an embodiment, the processor 202 may be configured to create a new CCL 106 for the desired assurance goal / SLS by sending a createMOI Request message to the PA / CCL MnS producer 104. Accordingly, the MnS producer 104 may provide a createMOI () Response message for the newly created CCL 106 to the processor 202.

[0126] Figure 3 illustrates a yet another block diagram of an apparatus 300 to evaluate performance of a closed control loop (CCL), in accordance with an embodiment of the present disclosure. The apparatus 300 may comprise a processing unit 302 comprising at least one processor, a transceiver comprising an input / output (I / O) interface 304, and a memory 306, but not limited thereto. The processing unit 302 may comprise at least one data processor for executing program components for executing user or system-generated processes. The processing unit 302 may comprise specialized processing units such as integrated system (bus) controllers, memory management control units, digital signal processing units, etc.

[0127] The processing unit 302 may be disposed in communication with one or more input / output (I / O) devices via transceiver comprising an I / O interface 304. The I / O interface 304 may employ communication protocols / methods such as, without limitation, audio, analog, digital, stereo, IEEE-1394, serial bus, Universal Serial Bus (USB), infrared, PS / 2, BNC, coaxial, component, composite, Digital Visual Interface (DVI), high-definition multimedia interface (HDMI), Radio Frequency (RF) antennas, S-Video, Video Graphics Array (VGA), IEEE 802.n / b / g / n / x, Bluetooth, cellular (e.g., Code-Division Multiple Access (CDMA), High-Speed Packet Access (HSPA+), Global System For Mobile Communications (GSM), Long-Term Evolution (LTE) or the like), etc. Using the I / O interface 304, the apparatus 300 may communicate with one or more I / O devices.

[0128] The apparatus 300 may also include suitable logic, circuitry, and interfaces that may be configured to evaluate the performance of the CCL 106. In an embodiment, apparatus 300 may be implemented in a computing device. A person skilled in the art may appreciate that the computing device may be a smartphone, a cellular phone, a mobile phone, a mainframe machine, a computer workstation, a laptop and / or a consumer electronic (CE) device. Moving on, a detailed explanation of the working of the apparatus 300 is explained in forthcoming paragraphs.

[0129] In an embodiment, the present disclosure relates to a yet another apparatus 300 to evaluate the performance of the CCL 106, where the processor 302 of the apparatus 300 may receive a first request, from a MnS consumer 102, to retrieve attributes of a plurality of CCLs associated with the MnS consumer 102, when a desired Service Level Specification (SLS) performance degrades and send the retrieved attributes of the plurality of CCLs to the MnS consumer 102. The processor 302 may receive a second request, from the MnS consumer 102, to obtain performance metrics of a selected CCL 106 among the plurality of CCLs and send the performance metrics of the selected CCL 106 to the MnS consumer 102.

[0130] In an embodiment, the processor 302 may be configured to receive the first request, from the MnS consumer 102, to retrieve the attributes of the plurality of CCLs associated with the MnS consumer 102, when the desired Service Level Specification (SLS) performance degrades. The 3rdGeneration Partnership Project (3GPP)　Technical Specifications (TS 32.156) defines semantic of the NRM implemented by the MnS Producer, such as the information object class (IOC), Attribute, Attribute properties etc. During the SLS assurance closed control loop operation phase, the MnS consumer 102 may request the MnS producer 104 to retrieve the attributes of the plurality of CCLs associated with the MnS consumer 102, as the task of the MnS producer 104 is to monitor the deployed CCLs performance and report the required attributes of the plurality of CCLs as and when asked by the MnS consumer 102.

[0131] In an embodiment, the processor 302 may be configured to send the attributes of the plurality of CCLs to the MnS consumer 102. Further the processor 302 may be configured to receive the second request, from the MnS consumer 102, to obtain the performance metrics of a selected CCL 106 among the plurality of CCLs. In an embodiment of the present disclosure, the performance metrics may be TotalAssuranceGoalBreach, TimeBreachedGoalRecovery, and TotalCclConflicts_Filter.

[0132] In an embodiment, the number of occurrences of goal breach as indicated by the performance metric TotalAssuranceGoalBreach may provide measurement of a total number of occurrences of breach of an assurance goal, happened in the selected CCL 106, during an observation period. The processor 302 of the apparatus 300 may be configured to measure the number of occurrences of the assurance goal breach by counting each incidence when the assurance goal is breached and incrementing a corresponding cumulative counter (CC) by one to get an integer value for each such occurrence within the observation period. It may be noted that the CC is reset to a well-defined value (usually "0") at the beginning of the measurement and further incremented by one when the assurance goal is breached.

[0133] In an embodiment, the time taken to meet the breached goal as indicated by the performance metric TimeBreachedGoalRecovery may provide measurement of time taken by the selected CCL 106 to meet the breached goal after re-activating the selected CCL 106. To measure the time taken to meet the breached goal, the processor 302 may be configured to monitor a first time stamp when an assurance goal is breached. The processor 302 may be then configured to monitor a second time stamp　when the assurance goal is met after re-activating the selected CCL 106. For example, the first time stamp may represent a time instance when an assurance goal is breached and may be captured in a discrete event registration (DER). Further the second time stamp　may represent a time instance when the assurance goal is met after re-activating the selected CCL 106 and may be captured in the DER. Further, the processor 302 may be configured to calculate a difference between the first time stamp and the second time stamp. The difference represents the time taken to meet the breached goal.

[0134] In an embodiment, the number of conflicts occurred by the CCL 106 as indicated by the performance metric TotalCclConflicts_Filter may provide measurement of a total number of conflicts that occur between the selected CCL and one or more CCLs other than the selected CCL during an observation period. The number of conflicts occurred by the CCL 106 may be measured by counting each incidence when a conflict occurs between the selected CCL and the one or more CCLs other than the selected CCL, and incrementing a corresponding cumulative counter (CC) by one to get an integer value for each such occurrence within the observation period. It may be noted that the CC is reset to a well-defined value (usually "0") at the beginning of the measurement and further incremented by one, when a conflict occurs between the selected CCL and the one or more CCLs other than the selected CCL.

[0135] In an embodiment, the number of conflicts occurred by the CCL 106 may be at least implicit conflict or explicit conflict. The implicit conflict may represent an action conflict between the selected CCL and one or more existing CCLs other than selected CCL, and the explicit conflict may represent a conflict between the selected CCL and a new requested CCL.

[0136] In an embodiment, the processor 302 may be configured to send the performance metrics of the selected CCL 106 to the MnS Consumer 102. The sent performance metrics may indicate the number of occurrences of assurance goal breach, time taken by the CCL to meet a breached goal, and a total number of conflicts occurred by the selected CCL.

[0137] In an embodiment of the present disclosure, the processor 302 may be further configured either to receive a modification request, from the MnS consumer 102, to modify one or more parameters of the identified CCL 106 or receive a creation request, from the MnS consumer 102, to create a new CCL 106 for the desired SLS, as described below in subsequent paragraphs.

[0138] In an embodiment, the processor 302 may be configured to receive the modifyMOIAttributes request message for that identified CCL 106, or the processor 302 may receive the changeMOIs request message from the performance assurance (PA) / CCL MnS consumer 102, for the identified CCL 106. Accordingly, the processor 302 may send a modifyMOIAttributes Response or changeMOIs response message to the MnS consumer 102 for the updated attributes of the selected CCL 106.

[0139] In an embodiment, the processor 302 may be configured to receive a createMOI Request message from the PA / CCL MnS consumer 102 to create a new CCL 106 for the desired assurance goal / SLS. Accordingly, the processor 302 may provide a createMOI () Response message for the newly created CCL 106 to the MnS consumer 102.

[0140] Figure 4 illustrates a method 400 of evaluating performance of a closed control loop (CCL), being performed by a Management service (MnS) consumer 102 in a communication network, in accordance with an embodiment of the present disclosure. Although example method 400 depicts a particular sequence of operations, the sequence may be altered without departing from the scope of the present disclosure. For example, some of the operations depicted may be performed in parallel or in a different sequence that does not materially affect the function of method 400.

[0141] According to some examples, at block 402 method 400 includes sending the first request, to the MnS producer 104, to retrieve attributes of the plurality of CCLs associated with the MnS consumer 102, when the desired Service Level Specification (SLS) performance degrades. The 3rdGeneration Partnership Project (3GPP)　Technical Specifications (TS 32.156) defines semantic of the NRM implemented by the MnS Producer, such as the information object class (IOC), Attribute, Attribute properties etc. The MnS consumer 102 may want to evaluate the performance of the CCL 106 itself. So, whenever the MnS consumer 102 feels that there is a performance degradation for a desired SLS performance, the MnS consumer 102 may request the MnS producer 104 to retrieve attributes of a plurality of CCLs associated with the MnS consumer 102.

[0142] According to some examples, at block 404, method 400 includes receiving the attributes of the plurality of CCLs from the MnS producer 104. The role of the MnS producer 104 is to monitor the deployed CCLs to achieve their assurance goal. Hence, once the MnS consumer 102 identifies any degradation in the desired Service Level Specification (SLS) performance, the MnS consumer 102 may initially request the MnS producer 104 to report the attributes of the plurality of CCLs to the MnS consumer 102, as the task of the MnS producer 104 is to keep a track of all the deployed CCLs in the communication network and collect the required measurements corresponding to the plurality of CCLs and report as and when asked by the MnS consumer 102.

[0143] According to some examples, at block 406, method 400 includes identifying a CCL 106 among the plurality of CCLs responsible for maintaining the desired SLS performance based on the received attributes.

[0144] According to some examples, at block 408, method 400 includes sending the second request, to the MnS producer 104, to obtain the performance metrics of the identified CCL 106. The MnS consumer 102 on identifying the CCL 106 among the plurality of CCLs responsible for maintaining the desired SLS performance may then send the second request to the MnS producer 104 to obtain the performance metrics of the identified CCL 106. In an embodiment of the present disclosure, the performance metrics may be TotalAssuranceGoalBreach, TimeBreachedGoalRecovery and TotalCclConflicts_Filter. The subsequent paragraphs explains the various performance metrics obtained by the MnS producer 104 and further report to the MnS consumer 102 to evaluate the performance of the deployed CCL 106 in the communication network.

[0145] In an embodiment, the number of occurrences of goal breach as indicated by the performance metric TotalAssuranceGoalBreach may provide measurement of a total number of occurrences of breach of an assurance goal, happened in the identified CCL 106, during an observation period. The number of occurrences of the assurance goal breach may be measured by counting each incidence when the assurance goal is breached and incrementing a corresponding cumulative counter (CC) by one to get an integer value for each such occurrence within the observation period. It may be noted that the CC is reset to a well-defined value (usually "0") at the beginning of the measurement and further incremented by one when the assurance goal is breached.

[0146] In an embodiment, the time taken to meet the breached goal as indicated by the performance metric TimeBreachedGoalRecovery may provide measurement of time taken by the identified CCL 106 to meet the breached goal after re-activating the identified CCL 106. The time taken to meet the breached goal may be indicated by a difference between a first time stamp and a second time stamp. For example, the first time stamp may represent a time instance when an assurance goal is breached and may be captured in a discrete event registration (DER). Further the second time stamp　may represent a time instance when the assurance goal is met after re-activating the identified CCL with the required changes 106 and may be captured in a DER.　In an embodiment, each measurement in the TimeBreachedGoalRecovery performance metric is an integer representing the mean delay in milliseconds.

[0147] In an embodiment, the number of conflicts occurred by the CCL 106 as indicated by the performance metric TotalCclConflicts_Filter may provide measurement of a total number of conflicts that occur between the identified CCL 106 and one or more CCLs other than the identified CCL during an observation period. The number of conflicts occurred by the CCL 106 may be measured by counting each incidence when a conflict occurs between the identified CCL and the one or more CCLs and incrementing a corresponding cumulative counter (CC) by one to get an integer value for each such occurrence within the observation period. It may be noted that the CC is reset to a well-defined value (usually "0") at the beginning of the measurement and further incremented by one, when a conflict occurs between the identified CCL and the one or more CCLs.

[0148] In an embodiment, the number of conflicts occurred by the CCL 106 as indicated by the performance metric TotalCclConflicts_Filter may be at least implicit conflict or explicit conflict. The implicit conflict may represent an action conflict between the identified CCL and one or more CCLs other than identified CCL, and the explicit conflict may represent a conflict between the identified CCL and a new requested CCL.

[0149] According to some examples, at block 410, method 400 includes receiving the performance metrics of the identified CCL 106 from the MnS producer 104. The MnS producer 104 on receiving the request from the MnS consumer 102, may send the performance metrics of the identified CCL 106 to the MnS consumer 102. The received performance metrics TotalAssuranceGoalBreach, TimeBreachedGoalRecovery and TotalCclConflicts_Filter may indicate a total number of occurrences of the assurance goal breach, time taken by the CCL to meet a breached goal, and a totalnumber of conflicts occurred by the identified CCL 106, respectively, as explained above.

[0150] According to some examples, at block 412, method 400 includes evaluating the performance of the identified CCL 106 based on the received performance metrics. The MnS consumer 102 based on the received performance metrics may then evaluate the performance of the identified CCL 106 resulting in degraded performance in the desired SLS and may further decide whether to modify the one or more parameters of the identified CCL 106 or create a new CCL 106 for the desired SLS, as explained in subsequent paragraphs.

[0151] In an embodiment, the MnS consumer 102 may update the identified CCL 106 by sending a modifyMOIAttributes request message for that identified CCL 106, or by sending changeMOIs request message to the MnS producer 104, for the identified CCL 106. Accordingly, the MnS producer 104 may send modifyMOIAttributes Response or changeMOIs response message to MnS consumer 102 for the updated attributes of the selected CCL 106.

[0152] In an embodiment, the MnS consumer 102 may choose to create a new CCL 106 for the desired assurance goal / SLS by sending a createMOI Request message to the MnS producer 104. Accordingly, the MnS producer 104 may provide a createMOI () Response message for the newly created CCL 106 to the MnS consumer 102.

[0153] The order in which the various operations of the methods are described is not intended to be construed as a limitation, and any number of the method described blocks can be combined in any order to implement the method. Additionally, individual blocks may be deleted from the methods without departing from the spirit and scope of the subject matter described herein. Furthermore, the methods can be implemented in any suitable hardware, software, firmware, or combination thereof. It may be noted here that the subject matter of some or all embodiments described with reference to Figures 1-3 may be relevant for method 400 and the same is not repeated for the sake of brevity.

[0154] Figure 5 illustrates a method 500 of evaluating performance of a closed control loop (CCL), being performed by a Management service (MnS) producer 104 in a communication network, in accordance with an embodiment of the present disclosure. Although example method 500 depicts a particular sequence of operations, the sequence may be altered without departing from the scope of the present disclosure. For example, some of the operations depicted may be performed in parallel or in a different sequence that does not materially affect the function of method 500.

[0155] According to some examples, at block 502, method 500 includes receiving the first request, from the MnS consumer 102, to retrieve attributes of the plurality of CCLs associated with the MnS consumer 102, when the desired Service Level Specification (SLS) performance degrades. The 3rdGeneration Partnership Project (3GPP)　Technical Specifications (TS 32.156) defines semantic of the NRM implemented by the MnS Producer, such as the information object class (IOC), Attribute, Attribute properties etc. During the SLS assurance closed control loop operation phase, the MnS consumer 102 may request the MnS producer 104 to retrieve the attributes of the plurality of CCLs associated with the MnS consumer 102, as the task of the MnS producer 104 is to monitor the deployed CCLs performance and report the required attributes of the plurality of CCLs as and when asked by the MnS consumer 102.

[0156] According to some examples, at block 504, method 500 includes sending the retrieved attributes of the plurality of CCLs to the MnS consumer 102. The MnS producer 104 on receiving the request from the MnS consumer 102 may send the retrieved attributes of the plurality of CCLs to the MnS consumer 102.

[0157] According to some examples, at block 506, method 500 includes receiving the second request, from the MnS consumer 102, to obtain performance metrics of the selected CCL 106 among the plurality of CCLs. Upon receiving the retrieved attributes from the MnS producer 104, the MnS consumer 102 may further send the second request to the MnS producer 104 to obtain the performance metrics of the selected CCL 106 among the plurality of CCLs to further evaluate the performance of the CCL 106. In an embodiment of the present disclosure, the performance metrics may be TotalAssuranceGoalBreach, TimeBreachedGoalRecovery, and TotalCclConflicts_Filter. The subsequent paragraphs explain these performance metrics measured by the MnS producer 104.

[0158] In an embodiment, the number of occurrences of goal breach as indicated by the performance metric TotalAssuranceGoalBreach may provide measurement of a total number of occurrences of breach of an assurance goal, happened in the selected CCL 106, during an observation period. The total number of occurrences of goal breach may be measured by the MnS producer 104 by counting each incidence when the assurance goal is breached and incrementing a corresponding cumulative counter (CC) by one to get an integer value for each such occurrence within the observation period. It may be noted that the CC is reset to a well-defined value (usually "0") at the beginning of the measurement and further incremented by one when the assurance goal is breached.

[0159] In an embodiment, the time taken to meet the breached goal as indicated by the performance metric TimeBreachedGoalRecovery may provide measurement of time taken by the selected CCL 106 to meet the breached goal after re-activating the selected CCL 106. The time taken to meet the breached goal is measured by monitoring by the MnS producer 104, a first time stamp when an assurance goal is breached , monitoring, by the MnS producer 104, a second time stamp　when the assurance goal is met after re-activating the selected CCL 106, and calculating a difference by the MnS producer 104 between the first time stamp and the second time stamp. For example, the first time stamp may represent a time instance when an assurance goal is breached and may be captured in a discrete event registration (DER). Further the second time stamp　may represent a time instance when the assurance goal is met after re-activating the selected CCL 106 and may be captured in the DER. The obtained difference represents the time taken to meet the breached goal.

[0160] In an embodiment, the number of conflicts occurred by the CCL 106 as indicated by the performance metric TotalCclConflicts_Filter may provide measurement of a total number of conflicts that occur between the selected CCL and one or more CCLs other than the selected CCL during an observation period. The number of conflicts occurred by the CCL may be measured by the MnS producer 104 by counting each incidence when a conflict occurs between the selected CCL and the one or more CCLs and incrementing a corresponding cumulative counter (CC) by one to get an integer value for each such occurrence within the observation period. It may be noted that the CC is reset to a well-defined value (usually "0") at the beginning of the measurement and further incremented by one, when a conflict occurs between the selected CCL and the one or more CCLs other than the selected CCL.

[0161] In an embodiment, the number of conflicts occurred by the CCL 106 as indicated by the performance metric TotalCclConflicts_Filter may be at least implicit conflict or explicit conflict. The implicit conflict may represent an action conflict between the selected CCL and one or more CCLs other than selected CCL, and the explicit conflict may represent a conflict between the selected CCL and a new requested CCL.

[0162] In an embodiment of the present disclosure, the MnS Producer 104 may receive a modification request, from the MnS consumer 102, to modify one or more parameters of the identified CCL 106 or receive a creation request, from the MnS consumer 102, to create a new CCL 106 for the desired SLS, as described below in subsequent paragraphs.

[0163] In an embodiment, the MnS producer 104 may receive the modifyMOIAttributes request message for that identified CCL 106, or the MnS producer 104 may receive the changeMOIs request message, from the performance assurance (PA) / CCL MnS consumer 102, for the identified CCL 106. Accordingly, the MnS producer 104 may send a modifyMOIAttributes Response or changeMOIs response message to the MnS consumer 102 for the updated attributes of the selected CCL 106.

[0164] In an embodiment, the MnS producer 104 may receive a createMOI Request message from the PA / CCL MnS consumer 102 to create a new CCL 106 for the desired assurance goal / SLS. Accordingly, the MnS producer 104 may provide a createMOI () Response message for the newly created CCL 106 to the MnS consumer 102.

[0165] The order in which the various operations of the methods are described is not intended to be construed as a limitation, and any number of the method described blocks can be combined in any order to implement the method. Additionally, individual blocks may be deleted from the methods without departing from the spirit and scope of the subject matter described herein. Furthermore, the methods can be implemented in any suitable hardware, software, firmware, or combination thereof. It may be noted here that the subject matter of some or all embodiments described with reference to Figures 1-4 may be relevant for method 500 and the same is not repeated for the sake of brevity.

[0166] Figure 6 illustrates a sequence flow diagram 600 between the Management service (MnS) consumer 102 and the Management service (MnS) producer 104 to evaluate the performance of the CCL , in accordance with an embodiment of the present disclosure.

[0167] At step 601, the performance assurance (PA) CCL MnS consumer 102 notices that the desired SLS performance degrades. At step 602, the PA / CCL MnS consumer 102 sends a getMOIAttributeRequest message to the PA / CCL MnS producer 104 for getting information about all the attributes of the plurality of CCLs. At step 603, the PA / CCL MnS producer 104 provides the information of all the attributes of the plurality of CCLs to the MnS consumer 102 in getMOIAttributeResponse message. At step 604, the MnS consumer 102 may identify a CCL 106 among the plurality of CCLs responsible for maintaining the desired SLS performance based on the received attributes. At step 605, the PA / CCL MnS consumer 102 sends a createMOI (PerfMetricJob) request to the PA / CCL MnS producer 104 for obtaining the status of the performance metrics for the identified CCL 106. In an embodiment, the performance metrics may be TotalAssuranceGoalBreach, TimeBreachedGoalRecovery and TotalCclConflicts_Filter. At step 606, the PA / CCL MnS producer 104 provides the requested performance metric values via a createMOI () Response message to the PA / CCL MnS consumer 102. On receiving the performance metrics, the MnS consumer 102 may either modify the existing CCL 106 or create a new CCL 106 as per the desired assurance goal. At step 607, the PA / CCL MnS producer 104 may update the identified CCL 106 by sending a modifyMOIAttributes request message for that identified CCL 106, or by sending changeMOIs request message to the PA / CCL MnS producer 104, for the identified CCL 106. Accordingly, at step 608, the PA / CCL MnS producer 104 may send modifyMOIAttributes Response or changeMOIs response message to the PA / CCL MnS consumer 102 for the updated attributes of the selected CCL 106. At step 609, the PA / CCL MnS producer 104 may choose to create a new CCL 106 for the desired assurance goal / SLS by sending a createMOI Request message to the PA / CCL MnS producer 104. Accordingly, at step 610, the PA / CCL MnS producer 104 may provide a createMOI () Response message for the newly created CCL 106 to the PA / CCL MnS consumer 102.

[0168] The order in which the various operations of the above sequence are described is not intended to be construed as a limitation, and it may be noted here that the subject matter of some or all embodiments described with reference to Figures 1-5 may be relevant for the sequence flow diagram 600 and the same is not repeated for the sake of brevity.

[0169] The various operations of methods described above may be performed by any suitable means capable of performing the corresponding functions. The means may include various hardware and / or software component(s) including, but not limited to a circuit, an application specific integrated circuit (ASIC), or processor. Generally, where there are operations illustrated in Figures, those operations may be performed by any suitable corresponding counterpart means-plus-function components.

[0170] Furthermore, one or more computer-readable storage media may be utilized in implementing embodiments consistent with the present disclosure. A computer-readable storage medium refers to any type of physical memory on which information or data readable by a processor may be stored. Thus, a computer-readable storage medium may store instructions for execution by one or more processors, including instructions for causing the processor(s) to perform steps or stages consistent with the embodiments described herein. The term "computer-readable medium" should be understood to include tangible items and exclude carrier waves and transient signals, i.e., non-transitory. Examples include Random Access Memory (RAM), Read-Only Memory (ROM), volatile memory, nonvolatile memory, hard drives, Compact Disc (CD) ROMs, Digital Video Disc (DVDs), flash drives, disks, and any other known physical storage media.

[0171] Certain aspects may comprise a computer program for performing the operations presented herein. For example, such a computer program product may comprise a computer readable media having instructions stored (and / or encoded) thereon, the instructions being executable by one or more processors to perform the operations described herein. For certain aspects, the computer program product may include packaging material.

[0172] Various components, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require realization by different hardware units. Rather, as described above, various units may be combined in a hardware unit or provided by a collection of interoperative hardware units, including one or more processors as described above, in conjunction with suitable software and / or firmware.

[0173] The terms "an embodiment", "embodiment", "embodiments", "the embodiment", "the embodiments", "one or more embodiments", "some embodiments", and "one embodiment" mean "one or more (but not all) embodiments of the invention(s)" unless expressly specified otherwise.

[0174] The terms "including", "comprising", "having" and variations thereof mean "including but not limited to", unless expressly specified otherwise. The enumerated listing of items does not imply that any or all of the items are mutually exclusive, unless expressly specified otherwise. The terms "a", "an" and "the" mean "one or more", unless expressly specified otherwise.

[0175] A description of an embodiment with several components in communication with each other does not imply that all such components are required. On the contrary a variety of optional components are described to illustrate the wide variety of possible embodiments of the invention.When a single device or article is described herein, it will be readily apparent that more than one device / article (whether or not they cooperate) may be used in place of a single device / article. Similarly, where more than one device or article is described herein (whether or not they cooperate), it will be readily apparent that a single device / article may be used in place of the more than one device or article, or a different number of devices / articles may be used instead of the shown number of devices or programs. The functionality and / or the features of a device may be alternatively embodied by one or more other devices which are not explicitly described as having such functionality / features. Thus, other embodiments of the invention need not include the device itself.

[0176] Finally, the language used in the specification has been principally selected for readability and instructional purposes, and it may not have been selected to delineate or circumscribe the inventive subject matter. While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art.

Claims

1.A method of evaluating performance of a closed control loop (CCL), performed by a management service (MnS) consumer in a wireless communication system, the method comprising:in case that a service level specification (SLS) performance starts degrading for a first metric, obtaining, from a MnS producer, first information on attributes of all assurance closed control loops (ACCLs);identifying a CCL which contains the first metric among the ACCLs, based on the first information; andobtaining, from the MnS producer, second information on performance metrics of the identified CCL.2.The method of claim 1, wherein the second information includes third information on a total number of occurrences of breach of an assurance goal, as indicated by TotalAssuranceGoalBreach,wherein the third information provides measurement of the total number of occurrences of breach of the assurance goal, as defined in the identified CCL, during an observation period, andwherein the total number of occurrences of breach of the assurance goal is measured by counting each incidence, in case that the assurance goal is breached, and by incrementing a corresponding cumulative counter by one for each such occurrence within the observation period.3.The method of claim 1, wherein the second information includes fourth information on time taken by the identified CCL to meet a breached goal as indicated by TimeBreachedGoalRecovery,wherein the fourth information provides measurement of time taken by the identified CCL to meet the breached goal after re-activating the identified CCL,wherein the time taken by the identified CCL to meet the breached goal is indicated by a difference between a first time stamp and a second time stamp, andwherein the first time stamp represents a time instance in case that an assurance goal is breached, and the second time stamp　represents a time instance in case that the assurance goal is met after re-activating the identified CCL with required changes.　4.The method of claim 1, wherein the second information includes fifth information on a total number of conflicts occurred by the identified CCL as indicated by TotalCclConflicts_Filter,wherein the fifth information provides measurement of the total number of conflicts that occur between the identified CCL and one or more CCLs other than the identified CCL during an observation period, andwherein the total number of conflicts occurred by the identified CCL is measured by counting each incidence in case that a conflict occurs between the identified CCL and the one or more CCLs other than the identified CCL, and by incrementing a corresponding cumulative counter by one for each such occurrence within the observation period.5.The method of claim 4, wherein the conflict is at least one of: an implicit conflict and an explicit conflict, andwherein the implicit conflict represents an action conflict between the identified CCL and the one or more CCLs other than the identified CCL, and the explicit conflict represents a conflict between the identified CCL and a new requested CCL.6.The method of claim 1, further comprising:transmitting, to the MnS producer, a modification request to modify one or more parameters of the identified CCL for the SLS,receiving, from the MnS producer, a modification response containing information about modifications done for the identified CCL.7.The method of claim 1, further comprising:transmitting, to the MnS producer, a creation request to create a new CCL for the SLS; andreceiving, from the MnS producer, a creation response in response to the creation request, to create the new CCL for the SLS.8.A method of evaluating performance of a closed control loop (CCL), performed by a management service (MnS) producer in a wireless communication system, the method comprising:in case that a service level specification (SLS) performance starts degrading for a first metric, transmitting, to a MnS consumer, first information on attributes of all assurance closed control loops (ACCLs); andtransmitting, to the MnS consumer, second information on performance metrics of an identified CCL which contains the first metric among the ACCLs based on the first information.9.The method of claim 8, wherein the second information includes third information on a total number of occurrences of breach of an assurance goal, as indicated by TotalAssuranceGoalBreach,wherein the third information provides measurement of the total number of occurrences of breach of the assurance goal, as defined in the identified CCL, during an observation period, andwherein the total number of occurrences of breach of the assurance goal is measured by counting each incidence in case that the assurance goal is breached and incrementing a corresponding cumulative counter by one for each such occurrence within the observation period.10.The method of claim 8, wherein the second information includes fourth information on time taken by the identified CCL to meet a breached goal as indicated by TimeBreachedGoalRecovery,wherein the fourth information provides measurement of time taken by the identified CCL to meet the breached goal after re-activating the identified CCL,wherein the time by the identified CCL taken to meet the breached goal is measured by:monitoring a first time stamp in case that an assurance goal is breached;monitoring a second time stamp　in case that the assurance goal is met after re-activating the identified CCL; andcalculating a difference between the first time stamp and the second time stamp, wherein the difference represents the time taken to meet the breached goal.11.The method of claim 8, wherein the second information includes fifth information on a total number of conflicts occurred by the identified CCL as indicated by TotalCclConflicts_Filter,wherein the fifth information provides measurement of the total number of conflicts that occur between the identified CCL and one or more CCLs other than the identified CCL during an observation period, andwherein the total number of conflicts occurred by the identified CCL is measured by counting each incidence in case that a conflict occurs between the identified CCL and the one or more CCLs other than the identified CCL, and by incrementing a corresponding cumulative counter by one for each such occurrence within the observation period.12.The method of claim 11, wherein the conflict is at least one of: an implicit conflict and an explicit conflict, andwherein the implicit conflict represents an action conflict between the identified CCL and one or more CCLs other than the identified CCL, and the explicit conflict represents a conflict between the identified CCL and a new requested CCL.13.The method as claimed in claim 8, further comprising:receiving, from the MnS consumer, a modification request to modify one or more parameters of the identified CCL for the SLS, wherein upon receiving the modification request, a modification response is transmitted to the MnS consumer, containing information about modifications done for the identified CCL; orreceiving, from the MnS consumer, a creation request to create a new CCL for the desired SLS, wherein upon receiving the creation request, a creation response is transmitted to the MnS consumer, for the created new CCL for the desired SLS.14.A management service (MnS) consumer entity to evaluate performance of a closed control loop (CCL) in a wireless communication system, the MnS consumer entity comprising:a transceiver;at least one processor; anda memory coupled to the at least one processor,wherein the at least one processor coupled to the transceiver, and configured to:in case that a service level specification (SLS) performance starts degrading for a first metric obtain, from a MnS producer, first information on attributes of all assurance closed control loops (ACCLs);identify a CCL which contains the first metric among the ACCLs, based on the first information; andobtain, from the MnS producer, second information on performance metrics of the identified CCL.15.A management service (MnS) producer entity to evaluate performance of a closed control loop (CCL) in a wireless communication system, the MnS producer entity comprising:a transceiver;at least one processor; anda memory coupled to the at least one processor,wherein the at least one processor coupled to the transceiver, and configured to:in case that a service level specification (SLS) performance starts degrading for a first metric transmit, to a MnS consumer, first information of attributes of all assurance closed control loops (ACCLs); andtransmit, to the MnS consumer, second information on performance metrics of an identified CCL which contains the first metric among the ACCLs based on the first information.

Images