Method and apparatus for intent handler performance manager

Abstract

The intent handler includes at least one machine learning model trained to generate an action for controlling the communications network based on an intent from a consumer. An intent satisfaction scor

Description

FIELD

[01] The subject-matter of the present disclosure relates to managing performance of an intent handler in a communications network. BACKGROUND

[02] 6G communication systems are expected to provide hyper-connectivity as well as the high quality of experience for both consumers and industries. By combining AI / ML algorithms, the 6G networks will be able to collect and leverage a huge amount of data associated with billions of connected machines as well as human needs. 6G systems will improve and enable access to information, resources and social services without constraints of time or physical location. However, management of 6G networks will be very complex due to the heterogeneity of data. Therefore, there is a need to automatize the management.

[03] Zero-Touch network management is envisaged for beyond 5G networks and currently being standardized in 3GPP SA5. Intent based automation is key to enable zero-touch network management and currently being standardized. However, the unique requirements for 6G have not been considered yet.

[04] Intent is defined as the formal specification of all expectations including requirements, goals, and constraints given to a technical system. How to express an intent has been previously modelled.

[05] In 3GPP TR 28.912 v1.1.0, how to report the intent is studied, and within that a new attribute “observationTime” is proposed. The “observationTime” represents the time required for the managed service (MnS) consumers to instruct the intent handlers how long they will need to observe the value of the specified expectation target(s) and take an action to fulfil them. At the end of each observation time, the intent handlers should report back to MnS consumers the current value of the observed expectation targets and the fulfilment status of their intent in an intent report.

[06] Furthermore, a satisfactionFeedback attribute is proposed with a utility function, however the solution is not globally compatible to each domain. Furthermore, it requires MnS consumer to know how to calculate utility function for satisfaction which is not aligned with the 3GPP definition of Intent InformationObjectClass (IOC) (expectations including requirements, goals and constraints given to a 3GPP system). The feedback solution is not generated per IntentReport IOC and also does not serve to measure the performance of intent handlers per generated intent report at the end of the observationTime.

[07] Figure 1 shows a functional architecture of state of art cognitive layer and figure 2 shows a prior art intent manager framework.

[08] The cognitive layer of the prior art (figure 1) is then implemented in a prototype of prior art (figure 2), where the cognitive layer forms the intent manager framework given in Figure 2 and, together with cognitive agents (i.e. data grounding agent, proposal agent, prediction agent, evaluation agent, actuator agent), it enables the intent handlers. As shown in the prototype of the prior art (figure 2), the intent handler then processes that data and finds the best action to take in order to meet the expectation targets.

[09] In the state-of-art intent handler, also known as intent manager framework in the prior art (figure 2), it does not include any entity and / or mechanism to evaluate how the intent handler performs.

[10] In patent application numbered WO2022216192A1, proposed is a method and apparatus to manage the closed control loops, however they did not consider the lifecycle management of Al models associated with these closed control loops as well as they did not incorporate the performance analysis with the intent owner scores as well as intent reports.

[11] It is an aim of the present invention to address such problems and improve on the prior art. TECHNICAL PROBLEM

[12] Intent handlers are developed to serve the MnS consumers (intent owners) with algorithms and / or Al models. However, there is no mechanism and / or entity in the prior art that enables the performance management of intent handlers and its continuous development. The following requirements should be satisfied in order to enable performance management of intent handlers: • Requirement 1: Intent handler should continuously monitor internal data generated by different entities and / or software agents, modules of intent handler in order to evaluate the intent handler performance. • Requirement 2: Intent handlers should be equipped with mechanism to receive direct feedback from the intent owners (MnS consumers) for each IntentReport IOC, regarding intent handler performance. • Requirement 3: Intent handlers should continuously improve its performance by evaluating current performance and taking action to improve performance whenever a degradation is discovered. • Requirement 4: Intent handler should expose its performance analysis reports to the consumers in a transparent way. SUMMARY

[13] Proposed is a method and apparatus to monitor and manage the performance of intent handlers, incorporating with MnS consumer feedback per each intent report regarding consistency, trustworthiness, etc. and identify the reason of the problem that caused the dissatisfaction, and / or take action in intent handlers whether updating and / or removing the models (i.e. Al models) leveraged by the cognitive agents of the intent handlers or updating resource allocation.

[14] According to an aspect of the present disclosure, there is provided a method of managing a performance of an intent handler in a communications network, the intent handler including at least one machine learning model trained to generate an action for controlling the communications network based on an intent from a consumer, the method comprising: receiving the intent and an intent satisfaction score; determining, based on the intent satisfaction score, a performance of the intent handler in configuring the communications network to carry out the intent; determining an update to the at least one machine learning model of the intent handler based on the determined performance; and outputting a command to the intent handler to update the at least one machine learning model according to the update.

[15] Advantageously the method provides a means to monitor and improve performance of the intent handler in performing the intent of the consumer.

[16] In an embodiment, determining the update to the at least one machine learning model comprises at least one of: determining a change in status of the at least one machine learning model as active or idle; determining an update to the at least one machine learning model configuration and / or resource allocation; determining a redeployment of and / or removal of the at least one machine learning model that is considered to improve the overall performance of the intent handler; determining that the at least one machine learning model needs retraining; and determining that a new machine learning model needs to be added to the at least one machine learning model.

[17] In an embodiment, the method further comprises: creating a database to register the status of the at least one machine learning model; and managing the database to register changes in status of the at least one machine learning model.

[18] In an embodiment, determining an update to the at least one machine learning model comprises: determining, by a model lifecycle manager, the update to the at least one machine learning model.

[19] In an embodiment, the method further comprises: receiving knowledge from a knowledge base of the intent handler and a report from the intent handler, wherein determining, based on the intent satisfaction score, a performance of the intent handler in configuring the communications network to carry out the intent is also based on the received knowledge and the report.

[20] In an embodiment, the knowledge comprises other intent reports and key performance indicator, KPI, measurements.

[21] In an embodiment, the report is a report as specified in 3GPP TR 28.912 v18.0.1 modified to include the intent satisfaction score.

[22] In an embodiment, the method further comprises: generating an analysis report and an expose describing the determined performance of the communications network.

[23] In an embodiment, receiving the intent satisfaction score comprises: receiving, by a score receiver through representational state transfer, REST, application programming interfaces, APIs, to interface with a consumer which generated the intent, the intent satisfaction score.

[24] In an embodiment, the method further comprises: preparing, by the score receiver, the intent satisfaction score in a template using at least one algorithm or at least one machine learning model.

[25] In an embodiment, the intent satisfaction score is a score generated by a consumer which generated the intent to indicate a level of satisfaction regarding processing of the intent by the intent handler.

[26] In an embodiment, the intent satisfaction score is an integer, and optionally wherein the intent integer is between 0 and 10 where 0 is unsatisfied and 10 is satisfied.

[27] In an embodiment, wherein the intent satisfaction score is an array.

[28] In an embodiment, the intent satisfaction score is an enumeration, ENUM.

[29] In an embodiment, the ENIIM includes constituent scores including: a trust score, a consistency score, an accuracy score, and a first contact resolution rate.

[30] In an embodiment, each of the trust score, the consistency score, the accuracy score, and the first contact resolution rate is an integer or an array.

[31] In an embodiment, the intent satisfaction score comprises an expectation satisfaction score, wherein optionally the expectation satisfaction score is a score generated by a consumer which generated the intent to indicate a level of satisfaction regarding processing of the intent expectation by the intent handler.

[32] In an embodiment, the expectation satisfaction score is an integer, and optionally wherein the intent integer is between 0 and 10 where 0 is unsatisfied and 10 is satisfied.

[33] In an embodiment, the expectation satisfaction score is an array.

[34] In an embodiment, the expectation satisfaction score is an enumeration, ENUM.

[35] In an embodiment, the ENUM includes constituent scores including: a trust score, a consistency score, an accuracy score, and a first contact resolution rate.

[36] In an embodiment, each of the trust score, the consistency score, the accuracy score, and the first contact resolution rate is an integer or an array.

[37] According to an aspect of the present disclosure, there is provided an intent handler performance manager for managing performance of an intent handler in a communications network, the performance manager including at least one machine learning model trained to generate an action for controlling the communications network based on an intent from a consumer, the intent handler performance manager comprising: a score receiver configured to receive the intent and an intent satisfaction score; a performance analyser configured to determine a performance of the communications network in carrying out the intent based on the intent satisfaction score; and a model lifecycle manager configured to: determine an update the at least one machine learning model of the intent handler based on the determined performance, and output a command to the intent handler to update the at least one machine learning model according to the update.

[38] According to an aspect of the present disclosure, there is provided a score receiver of an intent handler, the score receiver comprising: an input comprising representational state transfer, REST, application programming interfaces, APIs, for receiving an intent satisfaction score from a consumer which created an intent associated with a communications network managed by the intent manager; a processor configured to prepare the intent satisfaction score in a template using an algorithm or a machine learning model; and an output configured to a performance analyser of the intent handler. ADVANTAGES

[39] The state-of-art intent handlers do not include any mechanism to measure their performance and update and manage the lifecycle of the Al models based on this measurement. It also does not incorporate with the MnS consumers (intent owners) feedback whether they are satisfied with the report whether their intent is FULFILLED or NOTFULFILLED. Especially as the technology evolves to cyber-physical continuum, there is a requirement to enable automated life-cycle management of intent handlers based on dynamical changes in the network. To satisfy this requirement, the proposed method and apparatus have the following advantages compared to the state-of-art intent handlers: • It enables continuous improvement of models deployed in the intent handler by analysing the intent reports and consumer scores; • It enables the intent owners (MnS consumer) to send an intent satisfaction score which measures how much the MnS consumer is satisfied with an intent report; • It enables an automated lifecycle management of models in an intent handler. BRIEF DESCRIPTION OF DRAWINGS

[40] The subject-matter of the present disclosure is best described with reference to the accompanying figures, in which:

[41] Figure 1 shows a functional architecture of state of art cognitive layer according to the prior art;

[42] Figure 2 shows a block diagram of an intent manager framework according to the prior art;

[43] Figure 3 shows a block diagram of a performance manager processor of intent handlers according to one or more embodiments;

[44] Figure 4 shows a detailed block diagram of the proposed Intent Handler Performance Manager of Figure 3;

[45] Figure 5 shows proposed changes in an example intent report model (TR 28.912 V18.0.1);

[46] Figure 6 shows an example sequence diagram for a score receiver;

[47] Figure 7 shows an intent reporting period provided in S5-232455;

[48] Figure 8 shows an example sequence diagram for the Model Performance Analyser of the intent handler performance manager of Figure 4; and

[49] Figure 9 shows an example sequence diagram for the Model Lifecycle Manager of the intent handler performance manager of Figure 4. DESCRIPTION OF EMBODIMENTS

[50] Each intent 20 registered to the intent handler 22 will be assigned a set of expectations where each expectation corresponds to a single KPI with a target value to achieve. A closed loop operation is assigned by the knowledge base 21 for each KPI aiming to achieve the target KPI. Each closed loop operation will consist of a number of agents, namely data grounding agent 23, proposal agent 24, prediction agent 25, evaluation agent 26 and actuator agent 27 as depicted in Figure 2. As the number of intents 20 grow in the intent handler 22, a high number of closed loop operations will be running where each closed operation will activate a high number of agents.

[51] AI / ML will be an important part of 6G and it is envisioned that most of these agents will be designed as Al models producing the most accurate results when compared to simpler rule-based models. Al models such as decision trees, supervised learning and reinforcement learning models can be deployed in proposal agents 24, where Al model can learn from the environment 28 and propose accurate actions for a defined target KPI. Prediction agents 25 can use Al models for forecasting impact on all system KPIs for a proposed action from the proposal agent 24 and evaluation agents 26 can use Al algorithms like hierarchical deep reinforcement learning techniques to detect conflict between each proposed action and the predicted impact on each KPI to finally decide which action to execute.

[52] In other words, the intent handler 22 includes at least one machine learning model trained to generate an action for controlling the communications network based on an intent from a consumer. The following features may be understood as forming part of a method of managing a performance of an intent handler communications network.

[53] Accuracy of the models deployed for each of the agents is an important factor in achieving overall intent expectations. The model performance degrades over time due to various reasons listed below: 1. Seasonal changes: Al models can be trained with network data from summer time, may not work well during winter time due to significant network traffic profile changes between seasons especially in certain geographical locations 2. Aging Data: Data used for Al model training can become very old, and new training data may be required as the network and user profile changes over time. 3. Significant Network Changes (Partially): A significant network change in some parts of the network may result in reduced Al model performance in some local areas only. This will require partial re-training of the model using network data from parts of the network where the significant network changes are performed. 4. Missing or Incomplete Data Feed: Al model accuracy is directly impacted when the input data feed is not accurate. Data feed accuracy can be impacted by any fault in the interface or within the environment where data is generated.

[54] Proposed is a performance manager processor (or a performance manager) 30 for an intent handler 22 which interacts with intent owner 34 and the knowledge base of intent handler 22 for data collection and detecting any abnormal and / or undesired behaviour of intent handlers 22; with ML-ops tools 36 for managing the lifecycle of Al models maintaining the performance deployed within the intent handler 22; with resource managers 38 for ensuring adequate resources are allocated. Intent handler performance manager 30 continuously monitors the performance where it collects the new consumer satisfaction scores from the intent owner (consumer) 34 and generated intent reports 42 as depicted in Figure 3 and decides and / or executes action on resolving the performance issues detected.

[55] In other words, the method of managing a performance of the intent handler 22 comprises receiving the intent 20 and an intent satisfaction score 40; determining, based on the intent satisfaction score 40, a performance of the intent handler 22 in configuring the communications network to carry out the intent 20; determining an update to the at least one machine learning model of the intent handler 22 based on the determined performance; and outputting a command to the intent handler 22 to update the at least one machine learning model according to the update.

[56] With reference to Figure 4 the intent handler performance manager 30 includes: a score receiver 44, a score database 48, a performance analyser 46, a model lifecycle manager 43, a model registry 41, a model removal function 45, and a model deployment function 47.

[57] The score receiver 44 is configured to receive the intent and an intent satisfaction score. The score receiver 44 comprising: an input comprising representational state transfer, REST, application programming interfaces, APIs, for receiving an intent satisfaction score from a consumer which created an intent associated with a communications network managed by the intent manager; a processor configured to prepare the intent satisfaction score 40 in a template using an algorithm or a machine learning model; and an output configured to a performance analyser 46 of the intent handler 22.

[58] The performance analyser 46 is configured to determine a performance of the communications network in carrying out the intent based on the intent satisfaction score 40.

[59] The model lifecycle manager 43 is configured to determine an update the at least one machine learning model of the intent handler based on the determined performance, and output a command to the intent handler to update the at least one machine learning model according to the update.

[60] The score database 48 is a database for storing and receiving intent satisfaction scores 40. Intent Satisfaction Score

[61] Following the intent reporting model, specified in 3GPP TR 28912 v18.0.1, one may extend the model to accommodate intent owner satisfaction score as shown in figure 5.

[62] Intent satisfaction score 40 can be an IOC that can be used by the consumer to provide feedback on the fulfilment information received as part of the intent report 42. The consumer can request producer to create an instance of this IOC. The consumer’s feedback will be using various attributes of this IOC.

[63] In other words, the intent satisfaction score 40 may be a score generated by a consumer which generated the intent to indicate a level of satisfaction regarding processing of the intent by the intent handler.

[64] The following attributes in Table 1 can be added in the IntentSatisfactionScore IOC: Attribute Explanation DataType Support Qualifier intentSatisfactionScore It represents an overall score from the MnS consumer, associated with the satisfaction of the intent report. Score M expectationSatisfactionScore It provides more granular scores for each expectation associated with the intent report Score M expectation Ref This provide indication on the set of expectation that have resulted in the lower score. Expectation O Table 1: Example table for IntentSatisfactionScore IOC attributes.

[65] The intent satisfaction score 40 may comprise an expectation satisfaction score. The expectation satisfaction score may optionally be a score generated by a consumer which generated the intent to indicate a level of satisfaction regarding processing of the 5 intent expectation by the intent handler. The following attributes in Table 2 can be added in a Score «data type». Attribute Explanation DataType Support Qualifier satisfactionscore It represents an overall score from the MnS consumer, associated with the satisfaction of the intent report. Integer M satisfaction ScoreRange It represents the lower and the upper limits of the provided score range Array M detailedScores It provides more granular scores for various attributes such as Trust Score, Consistency Score etc. DetailedScore O Table 2: Example Table for Score «data type» attributes.

[66] The intent satisfaction score 40 and expectation satisfaction score each may be an integer, and optionally wherein the intent integer is between 0 and 10 where 0 is 10 unsatisfied and 10 is satisfied. The intent satisfaction score 40 and expectation satisfaction score may each also be an array. The array be a plurality of integers between the lower and upper limits of the provided score range.

[67] The following attributes in Table 3 can be added in a DetailedScore «data type». Attribute Explanation DataType Support Qualifier scoreCategory It provides more granular scores for various attributes such as Trust Score, Consistency Score, Accuracy Score, First Contact Resolution Rate etc: ENUM O TRUST: This attribute indicates how much the MnS consumer trusts the intent handling by the MnS producer. Given the TRUST score is low, the MnS producer may opt for i.e. changing models in favour of methods, (i.e. Explainable Al) CONSISTENCY: This attributes indicates how much the intent fulfilment was consistent with the intent report, i.e. even though an expectation is reported as FULFILLED, there may be times when the intent is NOTFULFILLED. Given the CONSISTENCY score is low, the MnS producer may opt for i.e. updating the model targeting the intent KPI(s), i.e. re-training and re-validation of the models ACCURACY: This indicates whether the MnS consumer thinks that the intent report is accurate with measured KPIs and FULFILLED status. Given the ACCURACY score is low, the MnS producer may opt for i.e. updating the model targeting the intent KPI(s), i.e. re-training and revalidation of the models. FIRST CONTACT FULFILLMENT RATE: This indicates whether the MnS consumer is satisfied with the time between intent submission and first intent report when the intent is reported as FULFILLED. Given the FIRST CONTACT FULFILLMENT RATE score is low, the MnS producer may opt for i.e., updating the resource allocation for the related intent handler entities satisfactionscore It represents an overall score from the MnS consumer, associated with the satisfaction of the intent report. Integer M Table 3: Example Table for the DetailedScore «data type» attributes.

[68] The intent satisfaction score 40 and the expectation satisfaction score may each be an enumeration, ENIIM. The ENUM may include constituent scores including: a trust score, a consistency score, an accuracy score, and a first contact resolution rate. Each 5 of the trust score, the consistency score, the accuracy score, and the first contact resolution rate may be an integer or an array.

[69] Implementation of intent satisfaction score 40 can be realized in different ways, either in a very abstract form which includes a single score representing the satisfaction of the intent owner 34 or more granular, specifying aspects of intent handler such as 10 intent fulfilment, trust, consistency etc. In addition, the intent owner 34 can score their satisfaction per expectation.

[70] In some examples, intent satisfaction score 40 can also be modelled as a JSON object. An example, but not limited to, is given as follows in Table 4: { “intentReportID”: <intentReport ID>, “consumerlD”: <consumerlD>, “intentSatisfactionScore”: { “satisfactionScore”: <Satisfaction Score, i.e. 9> “satisfactionScoreRange”: <Satisfaction Score Range, i.e. [1... 10]> “detailedScores”: [{ “scorecategory”: “CONSISTENCY” “satisfactionScore”: 8 }, { “scorecategory”: “FIRST CONTACT FULFILLMENT RATE” “satisfactionScore”: 4 }] }, “expectationSatisfactionScore”: { “satisfactionScore”: <Satisfaction Score, i.e. 9> “satisfactionScoreRange”: <Satisfaction Score Range, i.e. [1... 10]> “detailedScores”: [ { “scoreCategory”: “FIRST CONTACT FULFILLMENT RATE" “satisfactionScore”: 4 }] }, “expectationRef’: <[a set of expectations] > J______________________________________________________ Table 4: Example intent satisfaction score 40.

[71] IntentReportID may be an identification to be used for each intent report 42; for 5 example but not limited to, the timestamp for each report can be used as IntentReportID. Score Receiver and Score Databases

[72] The score receiver 44 is an entity that is responsible for collecting intent satisfaction score 40 (an example given in Table 1) from the intent owner (consumer) 34 through REST APIs and interfaces towards the intent owner 34 and receives, prepares 10 their score information into an intelligible template using algorithm(s) and / or Al module(s).

[73] In this way, the intent satisfaction score 40 may be prepared, by the score receiver, in a template using at least one algorithm or at least one machine learning model.

[74] In some examples, the score can be provided in an intent report 42 (an example given in Figure 5), and then delegated to the performance analyser 46.

[75] In some examples, the score of the intent owner 34 can be delegated to the score receiver 44 in a natural language format for each intent report 42. In this case, the score receiver 44 may leverage Natural Language Processing (NLP) methods to understand the intent owner score and translates it through algorithm and / or Al module into the intelligible template (Error! Reference source not found. - 4 or Error! Reference source not found.).

[76] An example sequence diagram for the score receiver is provided in Figure 6.

[77] As shown in Figure 6, in some examples, the score receiver 44 may check the provided score, translate it into an intelligible template if needed and register it in the score database 48 in table(s). Each score can be registered as an object associated with a Consumer ID.

[78] As a next step, the score receiver 44 sends the intent satisfaction score 40 for the submitted intent report 42, together with previous scores submitted by the same MnS consumer with Consumer ID to the performance analyser 46. Performance Analyser

[79] Performance analyser 46 is an entity which analyses the performance of the intent handler 22 with each IntentSatisfactionScore IOC as well as data from the knowledge base such as previous intent reports, KPI measurements etc. and detects the problem for the abnormal and / or undesired behaviour of intent handler 22.

[80] In this way the method may further comprise: receiving knowledge from a knowledge base of the intent handler and a report from the intent handler, wherein determining, based on the intent satisfaction score 40, a performance of the intent handler in configuring the communications network to carry out the intent is also based on the received knowledge and the report.

[81] In this way, the knowledge may comprise other intent reports and key performance indicator, KPI, measurements.

[82] Based on the analysis, it keeps a performance score for any aspects of the intent handler 22 (i.e. Al models, resource allocation etc.). It generates a report for performance analysis which proposes action(s) for detected low performance issues and stores all analysis information in a database. In this way, the method may comprise generating an analysis report and an expose describing the determined performance of the communications network.

[83] In some examples, the performance analyser 46 also exposes the analysis information (i.e. analysis reports) through an API for better observability of the intent handler 22.

[84] The performance analyser 46 can leverage the following data, however data sources can be extended in order to improve the performance analysis: • Intent Reports 42 generated for each intent (3GPP TR28.912 v18.0.1) along with proposed Intent Satisfaction Score 40 assigned for each intent report 42 (i.e. IntentReport IOC) that is received from the score receiver 44 in an intelligible template (examples of Table 1 or Figure 5). The report may be a report as specified in 3GPP TR 28.912 v18.0.1 modified to include the intent satisfaction score 40. • Intent reports 42 have key importance to understand the consumer satisfaction (or dissatisfaction) and identify the source of the problem. For example, if the intent is reported as fulfilled however the intent owner satisfaction score is low, the problem can be understood looking at the transparent intent reporting windows as stated in 3GPP TR28.912. An example is given in Figure 7. In the example provided in Figure 7, intents A 50 and C 52 are reported as fulfilled at the end of the reporting period, however, the intent was not fulfilled in the whole duration of the reporting period. As the red zones 54 increase in the reporting period, it is expected that the intent owner 34 may not be satisfied even though their intent is reported as fulfilled. In this case, for example, the intent owner 34 may not be satisfied with the consistency of the intent reports 42. The performance analyser 46 may consider the data in the knowledge base to understand the source of the problem further. • Any relevant knowledge data registered in the knowledge base of the intent handler 22. An example would be any registered data associated with each expectation target, such as the prediction data registered by the prediction agent(s). In some examples, predictions may not be good (i.e. due to not considering other conflicting intents that are present). This can be identified by the model performance analyser 46 leveraging an algorithm and / or Al module by leveraging the following example input data: o All registered intents and expectations in the system o Proposed action(s) o Measurements from the network (i.e. target expectation values) o Any other prediction data (i.e. from other prediction agents) o Available resources and consumed resources o Relations of target expectation objects • Deployed models information registered in the model registry 41. This information could be Model ID, Model scope, deployment date, deployment flavour (allocated resources) and any relevant information registered in the model registry 41.

[85] The performance analyser 46 identifies the potential problem sources (i.e. not-accurate prediction, example performance degradation can be identified through comparing target KPI measurements obtained from the network with the predicted target KPIs from Al models) and reports to the model lifecycle manager 43 with their report. This report, as an example but not limited to, can be implemented as a JSON object and shared with the model lifecycle manager 43.

[86] An example report is provided in Table 5 for a problem identified with a model configuration. In the report given in Table 5, the model related information is received from the model registry 41 (an example given in Table 6) and the proposed action is the outcome of the model performance analysis and indicates what is the proposed action to take in order to improve the performance of the intent handler 22. These actions could be: • UPDATE Model: This action instructs updating the model by re-configuration, retraining or appropriate resource allocation for the model. • REMOVE Model: This action instructs removing the model. { “intentReportID”: <Report ID> “Timestamp”:<Timestamp> “consumerlD”: <ConsumerlD> “potentialPerformanceProblems”: “AI_Model” “potentialproblem Description”: [ {“scope”: “Prediction”, “modellD”: <Model ID> “targetKPI”: <Target KPI> “targetuser”: <Target User or User Group> “proposedAction”: <Proposed Action> }. {“scope”: “Evaluation”, “modellD”: <Model ID> “targetKPI”: <Target KPI> “targetuser”: <Target User or User Group> “proposedAction”: <Proposed Action> } “takenActions”: <Taken Actions> }_______________________________________________________________________ Table 5: An example report generated by the Performance Analyser.

[87] In another example, the problem can be identified for the allocated resources of the intent handler 22. In this case, the performance analyser 46 can generate a report to 5 allocate more resources for the intent handler 22 and share this analysis with the relevant resource manager(s) 36 that are deployed in the system.

[88] An example sequence diagram for the performance analyser 46 is given in Figure 8. Model Registry 10

[89] The model registry 41 is a database that includes table(s) register information for each model deployed in the intent handler. An example information table that is registered in a model registry 41 is given as follows: Wei 10 Status Scops Target KPI Target User or User Group Flavour Deployment time Associated Agent 10 Associated Agent Last Triggered Time 1 Adive Predion Latency URLLC Laras TrmeStampI Agentt TlmeStampt 2 Idle Proposal Packet URLLC F^sdii-m TimeStsme2 AgentJ TlmeStampz 3 Removed Evaiua&m TrwrgitM UEJ Small TimeStercdS Agent3 TirneStsmS N Adive Predidkm Packet ......ym............................................. MIOT Largs TlmeStempA Agent-1 Tirne$tamp4 Tate 6:An examte tate in modal 41

[90] As per the example given in Table 6, the model registry 41 may include, but not limited to, the following information: • Model ID: A given ID of the model, which could be an integer or a string • Model Status: It is the status of the model whether Active, Idle or Removed; • Scope: The Scope of the Model (i.e. Prediction, proposal, evaluation etc.); • Target KPI: The network KPI that the model is associated with (i.e. Latency, packet loss, throughput etc.); • Target User or User Group: The IDs of the UEs or UE groups that the model is deployed for; • Flavour: The deployment flavour of the model may be pre-defined as Small, Medium, Large which represents the required resources (i.e. memory, storage etc.); • Deployment Date: The timestamp that represents the deployment of the model. • Associated Agent ID: The ID of the agent that the model is associated with • Associated Agent Last Triggered Time: The timestamp that shows the last time the agent was triggered. In this way, the model registry 41 may comprise creating a database to register the status of the at least one machine learning model; and managing the database to register changes in status of the at least one machine learning model. Model Lifecycle Manager

[91] The model lifecycle manager 43 is an entity which monitors the models whether they are working in ACTIVE status or remaining in IDLE status. In some examples, the model lifecycle manager 43 manages the lifecycle of models deployed for intent handler entities (i.e. cognitive agents) and decides an action (i.e. re-training, deployment and removal of model(s)). In some examples, the model lifecycle manager 43 registers their status in the model registry 41 as ACTIVE or IDLE. For example, if the agent associated with the model has been triggered in the last X amount of time, the model lifecycle manager 43 may decide that the model is ACTIVE. Otherwise, it may change the status as IDLE.

[92] In other words, determining the update to the at least one machine learning model may comprise determining a change in status of the at least one machine learning model as active or idle, determining an update at the at least one machine learning model configuration and / or resource allocation; determining a re-deployment of and / or removal of the at least one machine learning model that is considered to improve the overall performance of the intent handler and determining that the at least one machine learning model needs retraining; determining that a new machine learning model needs to be added to the at least one machine learning model.

[93] In some examples, it receives the analysis report from the model performance analyser 46 and then the model lifecycle manager 43 receives information from the model registry 41 related to the identified model in the analysis report. The model lifecycle manager 43 decides the following: • UPDATE the Model: If the model lifecycle manager 43 receives the proposed action to update the model, it has two options on how to update the model: o Sending a request to the relevant entities (i.e. MLops 36) for re-training and validation of the Al model; o Sending a request to the relevant entities (i.e. resource managers 38) for changing the Al model resource flavour Upon confirmation from the relevant entity, the model lifecycle manager 43 sends a request to model deployment function 47 to deploy the updated model in the intent handler 22 and updates the Al model status in the registry 41. • REMOVE the Model: If the model lifecycle manager 43 receives the proposed action to remove the Al model, sends a request to model removal function 45 to remove the model in the intent handler 22 and updates the Al model status in the registry 41.

[94] An example sequence diagram for the model lifecycle manager 43 is provided in Figure 9. Model Removal Function

[95] Model removal function 45 is a function that removes the model that was deployed in the intent handler 22. Upon successful removal, it sends a response to the model lifecycle manager 43 that the model is successfully removed. Model Deployment Function

[96] Model deployment function 47 is a function that deploys the model to the intent handler 22 with new configuration or resource flavour. Upon successful deployment, it sends a response to the model lifecycle manager 43 that the model is successfully deployed.

[97] Aspects of the present disclosures may be best understood with respect to the following clauses.

[98] Clause 1. An InformationObjectClass (IOC) which may be called “Intent Satisfaction Score” and may be associated with each intent report 42 (i.e. IntentReport IOC), and may be used to measure the satisfaction of the intent owner (MnS consumer) with the intent report, regarding the processing of their intent by the intent handler.

[99] Clause 2. An IOC according to clause 1 comprising modification of the intent report model specified in 3GPP TR 28.912 v18.0.1 to accommodate intent satisfaction score and be leveraged as a way to collect Intent Satisfaction Score through intent reporting.

[100] Clause 3. A Score receiver entity which is responsible for collecting Intent Satisfaction Score from the intent owner (MnS consumer) through REST APIs and interfaces towards the intent owner, and it receives, and / or if needed it prepares their score information into an intelligible template using algorithm(s) and / or Al module(s).

[101] Clause 4. An intent handler performance manager processor which evaluates the performance of the intent handlers, leveraging the algorithm(s) and / or Al module(s), and uses generated intent reports as well as collected “intent satisfaction score(s) for each intent report”, and related knowledge stored in the knowledge base, and monitors and manages the intent handler performance, manages the lifecycle of models deployed in an intent handler, for example but not limited to, changing and registering status as Active or Idle, and / or updating the model (i.e. Al model) configuration and / or resource allocation, and / or facilitate re-deployment of and / or removal of the model that is considered to improve the overall performance of the intent handler.

[102] Clause 5. An intent handler performance manager processor according to clause 4 comprising a an intent handler performance manager processor which evaluates the performance of the intent handlers, leveraging the algorithm(s) and / or Al module(s), and uses generated intent reports as well as collected “intent satisfaction score(s) for each intent report”, and related knowledge stored in the knowledge base, and monitors and manages the intent handler performance, manages the lifecycle of models deployed in an intent handler, for example but not limited to, changing and registering status as Active or Idle, and / or updating the model (i.e. Al model) configuration and / or resource allocation, and / or facilitate re-deployment of and / or removal of the model that is considered to improve the overall performance of the intent handler.

[103] Clause 6. An intent handler performance manager according to clause 4 comprising a model lifecycle manager which manages the lifecycle of models (i.e. Al models) leveraged by the intent handler entities (i.e. cognitive agents) and decides whether re-training, deployment and removal of models, and updates the status of the 5 models in a database.

[104] Clause 7. An intent handler performance manager according to clause 4 comprising creation of a database to register the status of models (i.e. Al models) in table(s) and relevant information. 10

Claims

28 02 251. A method of managing a performance of an intent handler (22) in a communications network, the intent handler including at least one machine learning model trained to generate an action for controlling the communications network based on an intent from a consumer, the method comprising:receiving the intent and an intent satisfaction score (40);determining, based on the intent satisfaction score (40), a performance of the intent handler (22) in configuring the communications network to carry out the intent;determining an update to the at least one machine learning model of the intent handler (22) based on the determined performance; andoutputting a command to the intent handler (22) to update the at least one machine learning model according to the update.

2. The method of Claim 1, wherein determining the update to the at least one machine learning model comprises at least one of:determining a change in status of the at least one machine learning model as active or idle;determining an update to the at least one machine learning model configuration and / or resource allocation;determining a re-deployment of and / or removal of the at least one machine learning model that is considered to improve the overall performance of the intent handler (22);determining that the at least one machine learning model needs retraining; and determining that a new machine learning model needs to be added to the at least one machine learning model.

3. The method of Claim 2, further comprising:creating a database to register the status of the at least one machine learning model; andmanaging the database to register changes in status of the at least one machine learning model.

4. The method of any preceding claim, wherein determining an update to the at least one machine learning model comprises:28 02 25determining, by a model lifecycle manager (43), the update to the at least one machine learning model.

5. The method of any preceding claim, wherein the method further comprises:receiving knowledge from a knowledge base of the intent handler (22) and a report from the intent handler (22),wherein determining, based on the intent satisfaction score (40), a performance of the intent handler (22) in configuring the communications network to carry out the intent (20) is also based on the received knowledge and the report.

6. The method of Claim 5, wherein the knowledge comprises other intent reports and key performance indicator, KPI, measurements.

7. The method of Claim 5 or Claim 6, wherein the report is a report as specified in 3GPP TR 28.912 v18.0.1 modified to include the intent satisfaction score (40).

8. The method of any preceding claim, further comprising:generating an analysis report and an expose describing the determined performance of the communications network.

9. The method of any preceding claim, wherein receiving the intent satisfaction score (40) comprises:receiving, by a score receiver (44) through representational state transfer, REST, application programming interfaces, APIs, to interface with a consumer which generated the intent, the intent satisfaction score (40).

10. The method of Claim 9, further comprising:preparing, by the score receiver (44), the intent satisfaction score (40) in a template using at least one algorithm or at least one machine learning model.

11. The method of any preceding claim, wherein the intent satisfaction score (40) is a score generated by a consumer which generated the intent to indicate a level of satisfaction regarding processing of the intent by the intent handler (22).

12. The method of any preceding claim, wherein the intent satisfaction score (40) is an integer, and optionally wherein the intent integer is between 0 and 10 where 0 is unsatisfied and 10 is satisfied.28 02 2513. The method of any of claims 1 to 11, wherein the intent satisfaction score (40) is an array.

14. The method of any of claims 1 to 11, wherein the intent satisfaction score (40) is an enumeration, ENUM.

15. The method of Claim 14, wherein the ENUM includes constituent scores including: a trust score, a consistency score, an accuracy score, and a first contact resolution rate.

16. The method of Claim 15, wherein each of the trust score, the consistency score, the accuracy score, and the first contact resolution rate is an integer or an array.

17. The method of any preceding claim, wherein the intent satisfaction score (40) comprises an expectation satisfaction score, wherein optionally the expectation satisfaction score is a score generated by a consumer which generated the intent to indicate a level of satisfaction regarding processing of the intent expectation by the intent handler (22).

18. The method of Claim 17, wherein the expectation satisfaction score is an integer, and optionally wherein the intent integer is between 0 and 10 where 0 is unsatisfied and 10 is satisfied.

19. The method of Claim 17 or Claim 18, wherein the expectation satisfaction score is an array.

20. The method of any of Claims 17 to 19, wherein the expectation satisfaction score is an enumeration, ENUM.

21. The method of Claim 20, wherein the ENUM includes constituent scores including: a trust score, a consistency score, an accuracy score, and a first contact resolution rate.

22. The method of Claim 21, wherein each of the trust score, the consistency score, the accuracy score, and the first contact resolution rate is an integer or an array.

23. An intent handler performance manager (30) for managing performance of an intent handler (22) in a communications network, the performance manager (30) including28 02 25at least one machine learning model trained to generate an action for controlling the communications network based on an intent from a consumer, the intent handler performance manager (30) comprising:a score receiver (44) configured to receive the intent (20) and an intent satisfaction score (40);a performance analyser (46) configured to determine a performance of the communications network in carrying out the intent based on the intent satisfaction score (40); anda model lifecycle manager (43) configured to :determine an update the at least one machine learning model of the intent handler (22) based on the determined performance, andoutput a command to the intent handler (22) to update the at least one machine learning model according to the update.

24. The intent handler performance manager (30) of Claim 23, wherein the score receiver (44) comprises:an input comprising representational state transfer, REST, application programming interfaces, APIs, for receiving an intent satisfaction score (40) from a consumer which created an intent associated with a communications network managed by the intent manager;a processor configured to prepare the intent satisfaction score (40) in a template using an algorithm or a machine learning model; andan output configured to a performance analyser of the intent handler (22).

Images