Method and apparatus for intent handler performance manager

Abstract

The disclosure relates to a 5G or 6G communication system for supporting a higher data transmission rate. A managed service (MnS) producer receives, from a MnS consumer, intent information, transmits, to the MnS consumer, intent report information associated with the intent information, and receives, from the MnS consumer, intent satisfaction information associated with the intent report information, wherein the intent satisfaction information indicates a level of satisfaction with an intent fulfilment.

Description

METHOD AND APPARATUS FOR INTENT HANDLER PERFORMANCE MANAGER

[0001] The subject matter of the disclosure relates to managing performance of an intent handler in a communications network.

[0002] 5G mobile communication technologies define broad frequency bands such that high transmission rates and new services are possible, and can be implemented not only in "Sub 6GHz" bands such as 3.5GHz, but also in "Above 6GHz" bands referred to as mmWave including 28GHz and 39GHz. In addition, it has been considered to implement 6G mobile communication technologies (referred to as Beyond 5G systems) in terahertz bands (for example, 95GHz to 3THz bands) in order to accomplish transmission rates fifty times faster than 5G mobile communication technologies and ultra-low latencies one-tenth of 5G mobile communication technologies.

[0003] At the beginning of the development of 5G mobile communication technologies, in order to support services and to satisfy performance requirements in connection with enhanced Mobile BroadBand (eMBB), Ultra Reliable Low Latency Communications (URLLC), and massive Machine-Type Communications (mMTC), there has been ongoing standardization regarding beamforming and massive MIMO for mitigating radio-wave path loss and increasing radio-wave transmission distances in mmWave, supporting numerologies (for example, operating multiple subcarrier spacings) for efficiently utilizing mmWave resources and dynamic operation of slot formats, initial access technologies for supporting multi-beam transmission and broadbands, definition and operation of BWP (BandWidth Part), new channel coding methods such as a LDPC (Low Density Parity Check) code for large amount of data transmission and a polar code for highly reliable transmission of control information, L2 pre-processing, and network slicing for providing a dedicated network specialized to a specific service.

[0004] Currently, there are ongoing discussions regarding improvement and performance enhancement of initial 5G mobile communication technologies in view of services to be supported by 5G mobile communication technologies, and there has been physical layer standardization regarding technologies such as V2X (Vehicle-to-everything) for aiding driving determination by autonomous vehicles based on information regarding positions and states of vehicles transmitted by the vehicles and for enhancing user convenience, NR-U (New Radio Unlicensed) aimed at system operations conforming to various regulation-related requirements in unlicensed bands, NR UE Power Saving, Non-Terrestrial Network (NTN) which is UE-satellite direct communication for providing coverage in an area in which communication with terrestrial networks is unavailable, and positioning.

[0005] Moreover, there has been ongoing standardization in air interface architecture / protocol regarding technologies such as Industrial Internet of Things (IIoT) for supporting new services through interworking and convergence with other industries, IAB (Integrated Access and Backhaul) for providing a node for network service area expansion by supporting a wireless backhaul link and an access link in an integrated manner, mobility enhancement including conditional handover and DAPS (Dual Active Protocol Stack) handover, and two-step random access for simplifying random access procedures (2-step RACH for NR). There also has been ongoing standardization in system architecture / service regarding a 5G baseline architecture (for example, service based architecture or service based interface) for combining Network Functions Virtualization (NFV) and Software-Defined Networking (SDN) technologies, and Mobile Edge Computing (MEC) for receiving services based on UE positions.

[0006] As 5G mobile communication systems are commercialized, connected devices that have been exponentially increasing will be connected to communication networks, and it is accordingly expected that enhanced functions and performances of 5G mobile communication systems and integrated operations of connected devices will be necessary. To this end, new research is scheduled in connection with eXtended Reality (XR) for efficiently supporting AR (Augmented Reality), VR (Virtual Reality), MR (Mixed Reality) and the like, 5G performance improvement and complexity reduction by utilizing Artificial Intelligence (AI) and Machine Learning (ML), AI service support, metaverse service support, and drone communication.

[0007] Furthermore, such development of 5G mobile communication systems will serve as a basis for developing not only new waveforms for providing coverage in terahertz bands of 6G mobile communication technologies, multi-antenna transmission technologies such as Full Dimensional MIMO (FD-MIMO), array antennas and large-scale antennas, metamaterial-based lenses and antennas for improving coverage of terahertz band signals, high-dimensional space multiplexing technology using OAM (Orbital Angular Momentum), and RIS (Reconfigurable Intelligent Surface), but also full-duplex technology for increasing frequency efficiency of 6G mobile communication technologies and improving system networks, AI-based communication technology for implementing system optimization by utilizing satellites and AI (Artificial Intelligence) from the design stage and internalizing end-to-end AI support functions, and next-generation distributed computing technology for implementing services at levels of complexity exceeding the limit of UE operation capability by utilizing ultra-high-performance communication and computing resources.

[0008] In a first aspect of the disclosure, provided herein is a method performed by a managed service (MnS) producer, the method comprising: receiving, from a MnS consumer, intent information; transmitting, to the MnS consumer, intent report information associated with the intent information; and receiving, from the MnS consumer, intent satisfaction information associated with the intent report information, wherein the intent satisfaction information indicates a level of satisfaction with an intent fulfilment.

[0009] In a second aspect of the disclosure, provided herein a managed service (MnS) producer comprising: a transceiver; and at least one processor coupled to the transceiver, configured to: receive, from a MnS consumer, intent information, transmit, to the MnS consumer, intent report information associated with the intent information, and receive, from the MnS consumer, intent satisfaction information associated with the intent report information, wherein the intent satisfaction information indicates a level of satisfaction with an intent fulfilment.

[0010] The subject matter of the disclosure is best described with reference to the accompanying figures, in which:

[0011] Figure 1 shows a functional architecture of state of art cognitive layer according to an embodiment of the disclosure;

[0012] Figure 2 shows a block diagram of an intent manager framework according to an embodiment of the disclosure;

[0013] Figure 3 shows a block diagram of a performance manager processor of intent handlers according to an embodiment of the disclosure;

[0014] Figure 4 shows a detailed block diagram of the proposed Intent Handler Performance Manager of Figure 3 according to an embodiment of the disclosure;

[0015] Figure 5 shows proposed changes in an example intent report model according to an embodiment of the disclosure;

[0016] Figure 6 shows an example sequence diagram for a score receiver according to an embodiment of the disclosure;

[0017] Figure 7 shows an intent reporting period according to an embodiment of the disclosure;

[0018] Figure 8 shows an example sequence diagram for the Model Performance Analyser of the intent handler performance manager of Figure 4 according to an embodiment of the disclosure;

[0019] Figure 9 shows an example sequence diagram for the Model Lifecycle Manager of the intent handler performance manager of Figure 4 according to an embodiment of the disclosure; and

[0020] Figure 10 shows a block diagram of a structure of the MnS consumer and the MnS producer according to an embodiment of the disclosure.

[0021] 6G communication systems are expected to provide hyper-connectivity as well as the high quality of experience for both consumers and industries. By combining AI / ML algorithms, the 6G networks will be able to collect and leverage a huge amount of data associated with billions of connected machines as well as human needs. 6G systems will improve and enable access to information, resources and social services without constraints of time or physical location. However, management of 6G networks will be very complex due to the heterogeneity of data. Therefore, there is a need to automatize the management.

[0022] Zero-Touch network management is envisaged for beyond 5G networks and currently being standardized in 3GPP SA5. Intent based automation is key to enable zero-touch network management and currently being standardized. However, the unique requirements for 6G have not been considered yet.

[0023] Intent is defined as the formal specification of all expectations including requirements, goals, and constraints given to a technical system. How to express an intent has been previously modelled.

[0024] In 3GPP TR 28.912 v1.1.0, how to report the intent is studied, and within that a new attribute "observationTime" is proposed. The "observationTime" represents the time required for the managed service (MnS) consumers to instruct the intent handlers how long they will need to observe the value of the specified expectation target(s) and take an action to fulfil them. At the end of each observation time, the intent handlers should report back to MnS consumers the current value of the observed expectation targets and the fulfilment status of their intent in an intent report.

[0025] Furthermore, a satisfactionFeedback attribute is proposed with a utility function, however the solution is not globally compatible to each domain. Furthermore, it requires MnS consumer to know how to calculate utility function for satisfaction which is not aligned with the 3GPP definition of Intent InformationObjectClass (IOC) (expectations including requirements, goals and constraints given to a 3GPP system). The feedback solution is not generated per IntentReport IOC and also does not serve to measure the performance of intent handlers per generated intent report at the end of the observationTime.

[0026] Figure 1 shows a functional architecture of state of art cognitive layer according to an embodiment of the disclosure, and figure 2 shows a block diagram of an intent manager framework according to an embodiment of the disclosure.

[0027] The cognitive layer of the prior art (figure 1) is then implemented in a prototype of prior art (figure 2), where the cognitive layer forms the intent manager framework given in Figure 2 and, together with cognitive agents (i.e. data grounding agent, proposal agent, prediction agent, evaluation agent, actuator agent), it enables the intent handlers. As shown in the prototype of the prior art (figure 2), the intent handler then processes that data and finds the best action to take in order to meet the expectation targets.

[0028] In the state-of-art intent handler, also known as intent manager framework in the prior art (figure 2), it does not include any entity and / or mechanism to evaluate how the intent handler performs.

[0029] In patent application numbered WO2022216192A1, proposed is a method and apparatus to manage the closed control loops, however they did not consider the lifecycle management of AI models associated with these closed control loops as well as they did not incorporate the performance analysis with the intent owner scores as well as intent reports.

[0030] It is an aim of the disclosure to address such problems and improve on the prior art.

[0031] Intent handlers are developed to serve the MnS consumers (intent owners) with algorithms and / or AI models. However, there is no mechanism and / or entity in the prior art that enables the performance management of intent handlers and its continuous development. The following requirements should be satisfied in order to enable performance management of intent handlers:

[0032] ㆍ Requirement 1: Intent handler should continuously monitor internal data generated by different entities and / or software agents, modules of intent handler in order to evaluate the intent handler performance.

[0033] ㆍ Requirement 2: Intent handlers should be equipped with mechanism to receive direct feedback from the intent owners (MnS consumers) for each IntentReport IOC, regarding intent handler performance.

[0034] ㆍ Requirement 3: Intent handlers should continuously improve its performance by evaluating current performance and taking action to improve performance whenever a degradation is discovered.

[0035] ㆍ Requirement 4: Intent handler should expose its performance analysis reports to the consumers in a transparent way.

[0036] Proposed is a method and apparatus to monitor and manage the performance of intent handlers, incorporating with MnS consumer feedback per each intent report regarding consistency, trustworthiness, etc. and identify the reason of the problem that caused the dissatisfaction, and / or take action in intent handlers whether updating and / or removing the models (i.e. AI models) leveraged by the cognitive agents of the intent handlers or updating resource allocation.

[0037] According to an embodiment of the disclosure, there is provided a method of managing a performance of an intent handler in a communications network, the intent handler including at least one machine learning model trained to generate an action for controlling the communications network based on an intent from a consumer, the method comprising: receiving the intent and an intent satisfaction score; determining, based on the intent satisfaction score, a performance of the intent handler in configuring the communications network to carry out the intent; determining an update to the at least one machine learning model of the intent handler based on the determined performance; and outputting a command to the intent handler to update the at least one machine learning model according to the update.

[0038] Advantageously the method provides a means to monitor and improve performance of the intent handler in performing the intent of the consumer.

[0039] In an embodiment of the disclosure, determining the update to the at least one machine learning model comprises at least one of: determining a change in status of the at least one machine learning model as active or idle; determining an update at the at least one machine learning model configuration and / or resource allocation; determining a re-deployment of and / or removal of the at least one machine learning model that is considered to improve the overall performance of the intent handler; determining that the at least one machine learning model needs retraining; and determining that a new machine learning model needs to be added to the at least one machine learning model.

[0040] In an embodiment of the disclosure, the method further comprises: creating a database to register the status of the at least one machine learning model; and managing the database to register changes in status of the at least one machine learning model.

[0041] In an embodiment of the disclosure, determining an update to the at least one machine learning model comprises: determining, by a model lifecycle manager, the update to the at least one machine learning model.

[0042] In an embodiment of the disclosure, the method further comprises: receiving knowledge from a knowledge base of the intent handler and a report from the intent handler, wherein determining, based on the intent satisfaction score, a performance of the intent handler in configuring the communications network to carry out the intent is also based on the received knowledge and the report.

[0043] In an embodiment of the disclosure, the knowledge comprises other intent reports and key performance indicator, KPI, measurements.

[0044] In an embodiment of the disclosure, the report is a report as specified in 3GPP TR 28.912 v18.0.1 modified to include the intent satisfaction score.

[0045] In an embodiment of the disclosure, the method further comprises: generating an analysis report and an expose describing the determined performance of the communications network.

[0046] In an embodiment of the disclosure, receiving the intent satisfaction score comprises: receiving, by a score receiver through representational state transfer, REST, application programming interfaces, APIs, to interface with a consumer which generated the intent, the intent satisfaction score.

[0047] In an embodiment of the disclosure, the method further comprises: preparing, by the score receiver, the intent satisfaction score in a template using at least one algorithm or at least one machine learning model.

[0048] In an embodiment of the disclosure, the intent satisfaction score is a score generated by a consumer which generated the intent to indicate a level of satisfaction regarding processing of the intent by the intent handler.

[0049] In an embodiment of the disclosure, the intent satisfaction score is an integer, and optionally wherein the intent integer is between 0 and 10 where 0 is unsatisfied and 10 is satisfied.

[0050] In an embodiment of the disclosure, wherein the intent satisfaction score is an array.

[0051] In an embodiment of the disclosure, the intent satisfaction score is an enumeration, ENUM.

[0052] In an embodiment of the disclosure, the ENUM includes constituent scores including: a trust score, a consistency score, an accuracy score, and a first contact resolution rate.

[0053] In an embodiment of the disclosure, each of the trust score, the consistency score, the accuracy score, and the first contact resolution rate is an integer or an array.

[0054] In an embodiment of the disclosure, the intent satisfaction score comprises an expectation satisfaction score, wherein optionally the expectation satisfaction score is a score generated by a consumer which generated the intent to indicate a level of satisfaction regarding processing of the intent expectation by the intent handler.

[0055] In an embodiment of the disclosure, the expectation satisfaction score is an integer, and optionally wherein the intent integer is between 0 and 10 where 0 is unsatisfied and 10 is satisfied.

[0056] In an embodiment of the disclosure, the expectation satisfaction score is an array.

[0057] In an embodiment of the disclosure, the expectation satisfaction score is an enumeration, ENUM.

[0058] In an embodiment of the disclosure, the ENUM includes constituent scores including: a trust score, a consistency score, an accuracy score, and a first contact resolution rate.

[0059] In an embodiment of the disclosure, each of the trust score, the consistency score, the accuracy score, and the first contact resolution rate is an integer or an array.

[0060] According to an embodiment of the disclosure, there is provided an intent handler performance manager for managing performance of an intent handler in a communications network, the performance manager including at least one machine learning model trained to generate an action for controlling the communications network based on an intent from a consumer, the intent handler performance manager comprising: a score receiver configured to receive the intent and an intent satisfaction score; a performance analyser configured to determine a performance of the communications network in carrying out the intent based on the intent satisfaction score; and a model lifecycle manager configured to: determine an update the at least one machine learning model of the intent handler based on the determined performance, and output a command to the intent handler to update the at least one machine learning model according to the update.

[0061] According to an embodiment of the disclosure, there is provided a score receiver of an intent handler, the score receiver comprising: an input comprising representational state transfer, REST, application programming interfaces, APIs, for receiving an intent satisfaction score from a consumer which created an intent associated with a communications network managed by the intent manager; a processor configured to prepare the intent satisfaction score in a template using an algorithm or a machine learning model; and an output configured to a performance analyser of the intent handler.

[0062] The state-of-art intent handlers do not include any mechanism to measure their performance and update and manage the lifecycle of the AI models based on this measurement. It also does not incorporate with the MnS consumers (intent owners) feedback whether they are satisfied with the report whether their intent is FULFILLED or NOTFULFILLED. Especially as the technology evolves to cyber-physical continuum, there is a requirement to enable automated life-cycle management of intent handlers based on dynamical changes in the network. To satisfy this requirement, the proposed method and apparatus have the following advantages compared to the state-of-art intent handlers:

[0063] ㆍ It enables continuous improvement of models deployed in the intent handler by analysing the intent reports and consumer scores;

[0064] ㆍ It enables the intent owners (MnS consumer) to send an intent satisfaction score which measures how much the MnS consumer is satisfied with an intent report;

[0065] ㆍ It enables an automated lifecycle management of models in an intent handler.

[0066] Each intent (20) registered to the intent handler (22) will be assigned a set of expectations where each expectation corresponds to a single KPI with a target value to achieve. A closed loop operation is assigned by the knowledge base (21) for each KPI aiming to achieve the target KPI. Each closed loop operation will consist of a number of agents, namely data grounding agent (23), proposal agent (24), prediction agent (25), evaluation agent (26) and actuator agent (27) as depicted in Figure 2. As the number of intents (20) grow in the intent handler (22), a high number of closed loop operations will be running where each closed operation will activate a high number of agents.

[0067] AI / ML will be an important part of 6G and it is envisioned that most of these agents will be designed as AI models producing the most accurate results when compared to simpler rule-based models. AI models such as decision trees, supervised learning and reinforcement learning models can be deployed in proposal agents (24), where AI model can learn from the environment (28) and propose accurate actions for a defined target KPI. Prediction agents (25) can use AI models for forecasting impact on all system KPIs for a proposed action from the proposal agent (24) and evaluation agents (26) can use AI algorithms like hierarchical deep reinforcement learning techniques to detect conflict between each proposed action and the predicted impact on each KPI to finally decide which action to execute.

[0068] In other words, the intent handler (22) includes at least one machine learning model trained to generate an action for controlling the communications network based on an intent from a consumer. The following features may be understood as forming part of a method of managing a performance of an intent handler communications network.

[0069] Accuracy of the models deployed for each of the agents is an important factor in achieving overall intent expectations. The model performance degrades over time due to various reasons listed below:

[0070] 1. Seasonal changes: AI models can be trained with network data from summer time, may not work well during winter time due to significant network traffic profile changes between seasons especially in certain geographical locations.

[0071] 2. Aging Data: Data used for AI model training can become very old, and new training data may be required as the network and user profile changes over time.

[0072] 3. Significant Network Changes (Partially): A significant network change in some parts of the network may result in reduced AI model performance in some local areas only. This will require partial re-training of the model using network data from parts of the network where the significant network changes are performed.

[0073] 4. Missing or Incomplete Data Feed: AI model accuracy is directly impacted when the input data feed is not accurate. Data feed accuracy can be impacted by any fault in the interface or within the environment where data is generated.

[0074] Proposed is a performance manager processor (or a performance manager) (30) for an intent handler (22) which interacts with intent owner (34) and the knowledge base of intent handler (22) for data collection and detecting any abnormal and / or undesired behaviour of intent handlers (22); with ML-ops tools (36) for managing the lifecycle of AI models maintaining the performance deployed within the intent handler (22); with resource managers (38) for ensuring adequate resources are allocated. Intent handler performance manager (30) continuously monitors the performance where it collects the new consumer satisfaction scores from the intent owner (consumer) (34) and generated intent reports (42) as depicted in Figure 3 and decides and / or executes action on resolving the performance issues detected.

[0075] In other words, the method of managing a performance of the intent handler (22) comprises receiving the intent (20) and an intent satisfaction score (40); determining, based on the intent satisfaction score (40), a performance of the intent handler (22) in configuring the communications network to carry out the intent (20); determining an update to the at least one machine learning model of the intent handler (22) based on the determined performance; and outputting a command to the intent handler (22) to update the at least one machine learning model according to the update.

[0076] With reference to Figure 4 the intent handler performance manager (30) includes: a score receiver (44), a score database (48), a performance analyser (46), a model lifecycle manager (43), a model registry (41), a model removal function (45), and a model deployment function (47).

[0077] The score receiver (44) is configured to receive the intent and an intent satisfaction score. The score receiver (44) comprising: an input comprising representational state transfer, REST, application programming interfaces, APIs, for receiving an intent satisfaction score from a consumer which created an intent associated with a communications network managed by the intent manager; a processor configured to prepare the intent satisfaction score (40) in a template using an algorithm or a machine learning model; and an output configured to a performance analyser (46) of the intent handler (22).

[0078] The performance analyser (46) is configured to determine a performance of the communications network in carrying out the intent based on the intent satisfaction score (40).

[0079] The model lifecycle manager (43) is configured to determine an update the at least one machine learning model of the intent handler based on the determined performance, and output a command to the intent handler to update the at least one machine learning model according to the update.

[0080] The score database (48) is a database for storing and receiving intent satisfaction scores (40).

[0081] Intent Satisfaction Score

[0082] Following the intent reporting model, specified in 3GPP TR 28.912 v18.0.1, one may extend the model to accommodate intent owner satisfaction score as shown in figure 5.

[0083] Intent satisfaction score (40) can be an IOC that can be used by the consumer to provide feedback on the fulfilment information received as part of the intent report (42). The consumer can request producer to create an instance of this IOC. The consumer's feedback will be using various attributes of this IOC.

[0084] In other words, the intent satisfaction score (40) may be a score generated by a consumer which generated the intent to indicate a level of satisfaction regarding processing of the intent by the intent handler.

[0085] The following attributes in Table 1 can be added in the IntentSatisfactionScore IOC:

[0086]

[0087] Table 1: Example table for IntentSatisfactionScore IOC attributes.

[0088] The intent satisfaction score (40) may comprise an expectation satisfaction score. The expectation satisfaction score may optionally be a score generated by a consumer which generated the intent to indicate a level of satisfaction regarding processing of the intent expectation by the intent handler.

[0089] The following attributes in Table 2 can be added in a Score <<data type>>.

[0090]

[0091] Table 2: Example Table for Score <<data type>> attributes.

[0092] The intent satisfaction score (40) and expectation satisfaction score each may be an integer, and optionally wherein the intent integer is between 0 and 10 where 0 is unsatisfied and 10 is satisfied. The intent satisfaction score (40) and expectation satisfaction score may each also be an array. The array be a plurality of integers between the lower and upper limits of the provided score range.

[0093] The following attributes in Table 3 can be added in a DetailedScore <<data type>>.

[0094]

[0095] Table 3: Example Table for the DetailedScore <<data type>> attributes.

[0096] The intent satisfaction score (40) and the expectation satisfaction score may each be an enumeration, ENUM. The ENUM may include constituent scores including: a trust score, a consistency score, an accuracy score, and a first contact resolution rate. Each of the trust score, the consistency score, the accuracy score, and the first contact resolution rate may be an integer or an array.

[0097] Implementation of intent satisfaction score (40) can be realized in different ways, either in a very abstract form which includes a single score representing the satisfaction of the intent owner (34) or more granular, specifying aspects of intent handler such as intent fulfilment, trust, consistency etc. In addition, the intent owner (34) can score their satisfaction per expectation.

[0098] In some examples, intent satisfaction score (40) can also be modelled as a JSON object. An example, but not limited to, is given as follows in Table 4:

[0099]

[0100] Table 4: Example intent satisfaction score (40).

[0101] IntentReportID may be an identification to be used for each intent report (42); for example but not limited to, the timestamp for each report can be used as IntentReportID.

[0102] Score Receiver and Score Databases

[0103] The score receiver (44) is an entity that is responsible for collecting intent satisfaction score (40) (an example given in Table 1) from the intent owner (consumer) (34) through REST APIs and interfaces towards the intent owner (34) and receives, prepares their score information into an intelligible template using algorithm(s) and / or AI module(s).

[0104] In this way, the intent satisfaction score (40) may be prepared, by the score receiver, in a template using at least one algorithm or at least one machine learning model.

[0105] In some examples, the score can be provided in an intent report (42) (an example given in Figure 5), and then delegated to the performance analyser (46).

[0106] In some examples, the score of the intent owner (34) can be delegated to the score receiver (44) in a natural language format for each intent report (42). In this case, the score receiver (44) may leverage Natural Language Processing (NLP) methods to understand the intent owner score and translates it through algorithm and / or AI module into the intelligible template.

[0107] An example sequence diagram for the score receiver is provided in Figure 6.

[0108] As shown in Figure 6, in some examples, the score receiver (44) may check the provided score, translate it into an intelligible template if needed and register it in the score database (48) in table(s). Each score can be registered as an object associated with a Consumer ID.

[0109] As a next step, the score receiver (44) sends the intent satisfaction score (40) for the submitted intent report (42), together with previous scores submitted by the same MnS consumer with Consumer ID to the performance analyser (46).

[0110] Performance Analyser

[0111] Performance analyser (46) is an entity which analyses the performance of the intent handler (22) with each IntentSatisfactionScore IOC as well as data from the knowledge base such as previous intent reports, KPI measurements etc. and detects the problem for the abnormal and / or undesired behaviour of intent handler (22).

[0112] In this way the method may further comprise: receiving knowledge from a knowledge base of the intent handler and a report from the intent handler, wherein determining, based on the intent satisfaction score (40), a performance of the intent handler in configuring the communications network to carry out the intent is also based on the received knowledge and the report.

[0113] In this way, the knowledge may comprise other intent reports and key performance indicator, KPI, measurements.

[0114] Based on the analysis, it keeps a performance score for any aspects of the intent handler (22) (i.e. AI models, resource allocation etc.). It generates a report for performance analysis which proposes action(s) for detected low performance issues and stores all analysis information in a database. In this way, the method may comprise generating an analysis report and an expose describing the determined performance of the communications network.

[0115] In some examples, the performance analyser (46) also exposes the analysis information (i.e. analysis reports) through an API for better observability of the intent handler (22).

[0116] The performance analyser (46) can leverage the following data, however data sources can be extended in order to improve the performance analysis:

[0117] ㆍ Intent Reports (42) generated for each intent (3GPP TR 28.912 v18.0.1) along with proposed Intent Satisfaction Score (40) assigned for each intent report (42) (i.e. IntentReport IOC) that is received from the score receiver (44) in an intelligible template (examples of Table 1 or Figure 5). The report may be a report as specified in 3GPP TR 28.912 v18.0.1 modified to include the intent satisfaction score (40).

[0118] ㆍ Intent reports (42) have key importance to understand the consumer satisfaction (or dissatisfaction) and identify the source of the problem. For example, if the intent is reported as fulfilled however the intent owner satisfaction score is low, the problem can be understood looking at the transparent intent reporting windows as stated in 3GPP TR 28.912. An example is given in Figure 7.

[0119] In the example provided in Figure 7, intents A (50) and C (52) are reported as fulfilled at the end of the reporting period, however, the intent was not fulfilled in the whole duration of the reporting period. As the red zones (54) increase in the reporting period, it is expected that the intent owner (34) may not be satisfied even though their intent is reported as fulfilled. In this case, for example, the intent owner (34) may not be satisfied with the consistency of the intent reports (42). The performance analyser (46) may consider the data in the knowledge base to understand the source of the problem further.

[0120] ㆍ Any relevant knowledge data registered in the knowledge base of the intent handler (22). An example would be any registered data associated with each expectation target, such as the prediction data registered by the prediction agent(s). In some examples, predictions may not be good (i.e. due to not considering other conflicting intents that are present). This can be identified by the model performance analyser (46) leveraging an algorithm and / or AI module by leveraging the following example input data:

[0121] o All registered intents and expectations in the system

[0122] o Proposed action(s)

[0123] o Measurements from the network (i.e. target expectation values)

[0124] o Any other prediction data (i.e. from other prediction agents)

[0125] o Available resources and consumed resources

[0126] o Relations of target expectation objects

[0127] ㆍ Deployed models information registered in the model registry (41). This information could be Model ID, Model scope, deployment date, deployment flavour (allocated resources) and any relevant information registered in the model registry (41).

[0128] The performance analyser (46) identifies the potential problem sources (i.e. not-accurate prediction, example performance degradation can be identified through comparing target KPI measurements obtained from the network with the predicted target KPIs from AI models) and reports to the model lifecycle manager (43) with their report. This report, as an example but not limited to, can be implemented as a JSON object and shared with the model lifecycle manager (43).

[0129] An example report is provided in Table 5 for a problem identified with a model configuration. In the report given in Table 5, the model related information is received from the model registry (41) (an example given in Table 6) and the proposed action is the outcome of the model performance analysis and indicates what is the proposed action to take in order to improve the performance of the intent handler (22). These actions could be:

[0130] ㆍ UPDATE Model: This action instructs updating the model by re-configuration, re-training or appropriate resource allocation for the model.

[0131] ㆍ REMOVE Model: This action instructs removing the model.

[0132]

[0133] Table 5: An example report generated by the Performance Analyser.

[0134] In another example, the problem can be identified for the allocated resources of the intent handler (22). In this case, the performance analyser (46) can generate a report to allocate more resources for the intent handler (22) and share this analysis with the relevant resource manager(s) (38) that are deployed in the system.

[0135] An example sequence diagram for the performance analyser (46) is given in Figure 8.

[0136] Model Registry

[0137] The model registry (41) is a database that includes table(s) register information for each model deployed in the intent handler. An example information table that is registered in a model registry (41) is given as follows:

[0138]

[0139] Table 6: An example table in model registry (41).

[0140] As per the example given in Table 6, the model registry (41) may include, but not limited to, the following information:

[0141] ㆍ Model ID: A given ID of the model, which could be an integer or a string;

[0142] ㆍ Model Status: It is the status of the model whether Active, Idle or Removed;

[0143] ㆍ Scope: The Scope of the Model (i.e. Prediction, proposal, evaluation etc.);

[0144] ㆍ Target KPI: The network KPI that the model is associated with (i.e. Latency, packet loss, throughput etc.);

[0145] ㆍ Target User or User Group: The IDs of the UEs or UE groups that the model is deployed for;

[0146] ㆍ Flavour: The deployment flavour of the model may be pre-defined as Small, Medium, Large which represents the required resources (i.e. memory, storage etc.);

[0147] ㆍ Deployment Date: The timestamp that represents the deployment of the model;

[0148] ㆍ Associated Agent ID: The ID of the agent that the model is associated with;

[0149] ㆍ Associated Agent Last Triggered Time: The timestamp that shows the last time the agent was triggered.

[0150] In this way, the model registry (41) may comprise creating a database to register the status of the at least one machine learning model; and managing the database to register changes in status of the at least one machine learning model.

[0151] Model Lifecycle Manager

[0152] The model lifecycle manager (43) is an entity which monitors the models whether they are working in ACTIVE status or remaining in IDLE status. In some examples, the model lifecycle manager (43) manages the lifecycle of models deployed for intent handler entities (i.e. cognitive agents) and decides an action (i.e. re-training, deployment and removal of model(s)). In some examples, the model lifecycle manager (43) registers their status in the model registry (41) as ACTIVE or IDLE. For example, if the agent associated with the model has been triggered in the last X amount of time, the model lifecycle manager (43) may decide that the model is ACTIVE. Otherwise, it may change the status as IDLE.

[0153] In other words, determining the update to the at least one machine learning model may comprise determining a change in status of the at least one machine learning model as active or idle, determining an update at the at least one machine learning model configuration and / or resource allocation; determining a re-deployment of and / or removal of the at least one machine learning model that is considered to improve the overall performance of the intent handler and determining that the at least one machine learning model needs retraining; determining that a new machine learning model needs to be added to the at least one machine learning model.

[0154] In some examples, it receives the analysis report from the model performance analyser (46) and then the model lifecycle manager (43) receives information from the model registry (41) related to the identified model in the analysis report. The model lifecycle manager (43) decides the following:

[0155] ㆍ UPDATE the Model: If the model lifecycle manager (43) receives the proposed action to update the model, it has two options on how to update the model:

[0156] o Sending a request to the relevant entities (i.e. ML ops (36)) for re-training and validation of the AI model;

[0157] o Sending a request to the relevant entities (i.e. resource managers (38)) for changing the AI model resource flavour

[0158] Upon confirmation from the relevant entity, the model lifecycle manager (43) sends a request to model deployment function (47) to deploy the updated model in the intent handler (22) and updates the AI model status in the registry (41).

[0159] ㆍ REMOVE the Model: If the model lifecycle manager (43 )receives the proposed action to remove the AI model, sends a request to model removal function (45) to remove the model in the intent handler (22) and updates the AI model status in the registry (41).

[0160] An example sequence diagram for the model lifecycle manager (43) is provided in Figure 9.

[0161] Model Removal Function

[0162] Model removal function (45) is a function that removes the model that was deployed in the intent handler (22). Upon successful removal, it sends a response to the model lifecycle manager (43) that the model is successfully removed.

[0163] Model Deployment Function

[0164] Model deployment function (47) is a function that deploys the model to the intent handler (22) with new configuration or resource flavour. Upon successful deployment, it sends a response to the model lifecycle manager (43) that the model is successfully deployed.

[0165] Aspects of the disclosures may be best understood with respect to the following clauses.

[0166] Clause 1. An InformationObjectClass (IOC) which may be called "Intent Satisfaction Score" and may be associated with each intent report (42) (i.e. IntentReport IOC), and may be used to measure the satisfaction of the intent owner (MnS consumer) with the intent report, regarding the processing of their intent by the intent handler.

[0167] Clause 2. An IOC according to clause 1 comprising modification of the intent report model specified in 3GPP TR 28.912 v18.0.1 to accommodate intent satisfaction score and be leveraged as a way to collect Intent Satisfaction Score through intent reporting.

[0168] Clause 3. A Score receiver entity which is responsible for collecting Intent Satisfaction Score from the intent owner (MnS consumer) through REST APIs and interfaces towards the intent owner, and it receives, and / or if needed it prepares their score information into an intelligible template using algorithm(s) and / or AI module(s).

[0169] Clause 4. An intent handler performance manager processor which evaluates the performance of the intent handlers, leveraging the algorithm(s) and / or AI module(s), and uses generated intent reports as well as collected "intent satisfaction score(s) for each intent report", and related knowledge stored in the knowledge base, and monitors and manages the intent handler performance, manages the lifecycle of models deployed in an intent handler, for example but not limited to, changing and registering status as Active or Idle, and / or updating the model (i.e. AI model) configuration and / or resource allocation, and / or facilitate re-deployment of and / or removal of the model that is considered to improve the overall performance of the intent handler.

[0170] Clause 5. An intent handler performance manager processor according to clause 4 comprising a an intent handler performance manager processor which evaluates the performance of the intent handlers, leveraging the algorithm(s) and / or AI module(s), and uses generated intent reports as well as collected "intent satisfaction score(s) for each intent report", and related knowledge stored in the knowledge base, and monitors and manages the intent handler performance, manages the lifecycle of models deployed in an intent handler, for example but not limited to, changing and registering status as Active or Idle, and / or updating the model (i.e. AI model) configuration and / or resource allocation, and / or facilitate re-deployment of and / or removal of the model that is considered to improve the overall performance of the intent handler.

[0171] Clause 6. An intent handler performance manager according to clause 4 comprising a model lifecycle manager which manages the lifecycle of models (i.e. AI models) leveraged by the intent handler entities (i.e. cognitive agents) and decides whether re-training, deployment and removal of models, and updates the status of the models in a database.

[0172] Clause 7. An intent handler performance manager according to clause 4 comprising creation of a database to register the status of models (i.e. AI models) in table(s) and relevant information.

[0173] Figure 10 shows a block diagram of a structure of the MnS consumer (1010) and the MnS producer (1040) according to an embodiment of the disclosure.

[0174] Referring to figure 10, the MnS consumer (1010) according to the disclosure may include a processor (1030), and a transceiver (1020). According to the above-described communication method of the MnS consumer (1010), the processor (1030), and the transceiver (1020) of the MnS consumer (1010) may operate. However, components of the MnS consumer (1010) are not limited to the above-described example. For example, the MnS consumer (1010) may include components that are more than or fewer than the above-described components. Moreover, the processor (1030), and the transceiver (1020) may be implemented in the form of a single chip. The processor (1030) may refer to one or more processors.

[0175] The transceiver (1020) may collectively refer to a receiver and a transmitter of the MnS consumer (1010), and transmit and receive a signal to and from the MnS producer (1040). The signal transmitted and received to and from the MnS producer (1040) may include intent information, intent report information, and intent satisfaction information. To this end, the transceiver (1020) may include an RF transmitter that up-converts and amplifies a frequency of a transmission signal and an RF receiver that low-noise-amplifies a received signal and down-converts a frequency. However, this is merely an example of the transceiver (1020), components of which are not limited to the RF transmitter and the RF receiver.

[0176] The transceiver (1020) may receive a signal through a radio channel and output the received signal to the processor (1030), and transmit a signal output from the processor (1030) through the radio channel.

[0177] The MnS consumer (1010) may further include a memory. The memory may store programs and data required for an operation of the MnS consumer (1010). The memory may also store control information or data included in a signal obtained by the MnS consumer (1010). The memory may include a storage medium, such as ROM, RAM, hard-disk, CD-ROM, DVD, and the like, or a combination thereof.

[0178] The processor (1030) may control a series of processes such that the MnS consumer (1010) operates according to the above-described embodiment of the disclosure. For example, the processor (1030) may control components of the MnS consumer (1010) to perform the method of managing barring according to an embodiment of the disclosure. For example, the transceiver (1020) may receive a data signal including a control signal, and the processor (1030) may determine a reception result for the data signal.

[0179] The MnS producer (1040) according to the disclosure may include a processor (1060), and a transceiver (1050). According to the above-described communication method of the MnS producer (1040), the processor (1060), and the transceiver (1050) of the MnS producer (1040) may operate. However, components of the MnS producer (1040) are not limited to the above-described example. For example, the MnS producer (1040) may include components that are more than or fewer than the above-described components. Moreover, the processor (1060), and the transceiver (1050) may be implemented in the form of a single chip. The processor (1060) may refer to one or more processors.

[0180] The transceiver (1050) may collectively refer to a receiver and a transmitter of the MnS producer (1040), and transmit and receive a signal to and from the MnS consumer (1010). The signal transmitted and received to and from the MnS consumer (1010) may include intent information, intent report information, and intent satisfaction information. To this end, the transceiver (1050) may include an RF transmitter that up-converts and amplifies a frequency of a transmission signal and an RF receiver that low-noise-amplifies a received signal and down-converts a frequency. However, this is merely an example of the transceiver (1050), components of which are not limited to the RF transmitter and the RF receiver.

[0181] The transceiver (1050) may receive a signal through a radio channel and output the received signal to the processor (1060), and transmit a signal output from the processor (1060) through the radio channel.

[0182] The MnS consumer (1010) may be an intent owner (34).

[0183] The MnS producer (1040) may further include a memory. The memory may store programs and data required for an operation of the MnS producer (1040). The memory may also store control information or data included in a signal obtained by the MnS producer (1040). The memory may include a storage medium, such as ROM, RAM, hard-disk, CD-ROM, DVD, and the like, or a combination thereof.

[0184] The processor (1060) may control a series of processes such that the MnS producer (1040) operates according to the above-described embodiment of the disclosure. For example, the processor (1060) may control components of the MnS producer (1040) to perform the method of managing barring according to an embodiment of the disclosure. For example, the transceiver (1050) may receive a data signal including a control signal, and the processor (1060) may determine a reception result for the data signal.

[0185] The MnS producer (1040) may be an intent handler (22), or an intent handler performance manager (30).

[0186] According to an embodiment of the disclosure, wherein the intent satisfaction information is an integer in a range.

[0187] According to an embodiment of the disclosure, wherein the intent information comprises intent expectation information including requirements and goals of the MnS consumer.

[0188] According to an embodiment of the disclosure, wherein the intent report information comprises fulfilment information of the intent information.

[0189] According to an embodiment of the disclosure, a method performed by a MnS producer comprises continuously monitoring the intent fulfilment.

[0190] According to an embodiment of the disclosure, a method performed by a MnS producer comprises analysing a performance of an intent satisfaction.

[0191] According to an embodiment of the disclosure, wherein the intent information comprises expectation target information to achieve.

[0192] According to an embodiment of the disclosure, a method performed by a MnS producer comprises determining an update to at least one machine learning model based on the intent satisfaction information.

[0193] According to an embodiment of the disclosure, wherein the at least one processor is further configured to: continuously monitor the intent fulfilment.

[0194] According to an embodiment of the disclosure, wherein the at least one processor is further configured to: analyse a performance of an intent satisfaction.

Claims

1.A method performed by a managed service (MnS) producer, the method comprising:receiving, from a MnS consumer, intent information;transmitting, to the MnS consumer, intent report information associated with the intent information; andreceiving, from the MnS consumer, intent satisfaction information associated with the intent report information,wherein the intent satisfaction information indicates a level of satisfaction with an intent fulfilment.2.The method of claim 1, wherein the intent satisfaction information is an integer in a range.3.The method of claim 1, wherein the intent information comprises intent expectation information including requirements and goals of the MnS consumer.4.The method of claim 1, wherein the intent report information comprises fulfilment information of the intent information.5.The method of claim 1, further comprising:continuously monitoring the intent fulfilment.6.The method of claim 5, further comprising:analysing a performance of an intent satisfaction.7.The method of claim 1, wherein the intent information comprises expectation target information to achieve.8.The method of claim 1, further comprising:determining an update to at least one machine learning model based on the intent satisfaction information.9.A managed service (MnS) producer comprising:a transceiver; andat least one processor coupled to the transceiver, configured to:receive, from a MnS consumer, intent information,transmit, to the MnS consumer, intent report information associated with the intent information, andreceive, from the MnS consumer, intent satisfaction information associated with the intent report information,wherein the intent satisfaction information indicates a level of satisfaction with an intent fulfilment.10.The MnS producer of claim 9, wherein the intent satisfaction information is an integer in a range.11.The MnS producer of claim 9, wherein the intent information comprises intent expectation information including requirements and goals of the MnS consumer.12.The MnS producer of claim 9, wherein the intent report information comprises fulfilment information of the intent information.13.The MnS producer of claim 9, wherein the at least one processor is further configured to:continuously monitor the intent fulfilment.14.The MnS producer of claim 9, wherein the at least one processor is further configured to:analyse a performance of an intent satisfaction.15.The MnS producer of claim 9, wherein the intent information comprises expectation target information to achieve.

Images