Machine learning model performance degradation detection

Abstract

Various aspects of the present disclosure relate to a network entity, comprising: at least one memory; and at least one processor coupled with the at least one memory and configured to cause the network entity to: receive, from at least one application entity, information of a deviation of at least one metric associated with an ML-enabled operation, wherein the ML-enabled operation is associated within an ML model executable within an application layer; detect a degradation of the ML model, based at least in part on the deviation of the at least one metric associated with the ML-enabled operation; determine an action for the ML-enabled operation based at least in part on the detected degradation of the ML model; and adapt the ML-enabled operation based at least in part on the determined action.

Description

MACHINE LEARNING MODEL PERFORMANCE DEGRADATION DETECTIONTECHNICAL FIELD

[0001] The present disclosure relates to wireless communications, and more specifically to managing (e.g., detecting, reporting) degradation of artificial intelligence (AI) / machine learning (ML) (AI / ML) model performance.BACKGROUND

[0002] A wireless communications system may include one or multiple network communication devices, such as base stations, which may support wireless communications for one or multiple user communication devices, which may be otherwise known as user equipment (UE), or other suitable terminology. The wireless communications system may support wireless communications with one or multiple user communication devices by utilizing resources of the wireless communication system (e.g., time resources (e.g., symbols, slots, subframes, frames, or the like) or frequency resources (e.g., subcarriers, carriers, or the like). Additionally, the wireless communications system may support wireless communications across various radio access technologies including third generation (3G) radio access technology, fourth generation (4G) radio access technology, fifth generation (5G) radio access technology, among other suitable radio access technologies beyond 5G (e.g., sixth generation (6G)).SUMMARY

[0003] An article “a” before an element is unrestricted and understood to refer to “at least one” of those elements or “one or more” of those elements. The terms “a,” “at least one,” “one or more,” and “at least one of one or more” may be interchangeable. As used herein, including in the claims, “or” as used in a list of items (e.g., a list of items prefaced by a phrase such as “at least one of’ or “one or more of’ or “one or both of’) indicates an inclusive list such that, for example, a list of at least one of A, B, or C means A or B or C or AB or AC or BC or ABC (i.e., A and B and C). Also, as used herein, the phrase “based on” shall not be constmed as a reference to a closed set of conditions. For example, an example step that is described as “based on condition A” may be based on both a condition A and acondition B without departing from the scope of the present disclosure. In other words, as used herein, the phrase “based on” shall be construed in the same manner as the phrase “based at least in part on. Further, as used herein, including in the claims, a “set” may include one or more elements.

[0004] A method performed by a network entity is described. The method may include receiving, from at least one application entity, information of a deviation of at least one metric associated with an ML-enabled operation, wherein the ML-enabled operation is associated within an ML model executable within an application layer; detecting a degradation of the ML model, based at least in part on the deviation of the at least one metric associated with the ML-enabled operation; determining an action for the ML- enabled operation based at least in part on the detected degradation of the ML model; and adapting the ML-enabled operation based at least in part on the determined action.

[0005] A network entity is described. The network entity may include: at least one memory; and at least one processor coupled with the at least one memory and configured to cause the network entity to: receive, from at least one application entity, information of a deviation of at least one metric associated with an ML-enabled operation, wherein the ML- enabled operation is associated within an ML model executable within an application layer; detect a degradation of the ML model, based at least in part on the deviation of the at least one metric associated with the ML-enabled operation; determine an action for the ML- enabled operation based at least in part on the detected degradation of the ML model; and adapt the ML-enabled operation based at least in part on the determined action.

[0006] In some implementations of the method and network entity described herein, the ML model may be owned, provided, or deployed by the at least one application entity, wherein the at least one application entity may be a cloud or edge platform function.

[0007] In some implementations of the method and network entity described herein, the network entity may comprise an ML support function executable within an application enablement layer.

[0008] In some implementations of the method and network entity described herein, the action may include at least one or more of: training a different ML model for the ML-enabled operation; re-training the ML model; terminating the ML-enabled operation and initiating a different ML-enabled operation associated with a different ML model; and updating KPIs of the ML-enabled operation and continuing the ML-enabled operation.

[0009] Some implementations of the method and network entity described herein may further include operations or instructions for subscribing to the at least one application entity, wherein the information of the deviation of the at least one metric associated with the ML-enabled operation is received based at least in part on the network entity being subscribed to the at least one application entity.

[0010] In some implementations of the method and network entity described herein, the network entity may comprise be an AI / ML enablement layer (AIMLE) function. Some implementations of the method and network entity described herein may further include operations or instructions for the AIMLE function to output a command to an ML repository for a set of application entities associated with the ML model. In some implementations of the method and network entity described herein, the command may include a request for performance information of the ML model.

[0011] In some implementations of the method and network entity described herein, the performance information of the ML model may include one of more of historical data on prior degradation, a rating of the ML model, a type of ML operation configured for the ML model, or performance statistics of the ML model.

[0012] In some implementations of the method and network entity described herein, the ML-enabled operation may comprise one or more of an ML model training operation, an ML inference operation and a data management operation.

[0013] In some implementations of the method and network entity described herein, the network entity may comprise an edge enablement layer function.

[0014] In some implementations of the method and network entity described herein, the ML-enabled operation may comprise an analytics operation.

[0015] In some implementations of the method and network entity described herein, the analytics operation may comprise one or more of an edge analytics operation, a cloud analytics operation, and an AD AES analytics operation.

[0016] In some implementations of the method and network entity described herein, the at least one metric may comprise an accuracy level, a latency metric, a Quality of Experience (QoE) metric associated with the ML-enabled operation, a Quality of Service (QoS) metric associated with the ML-enabled operation, a Key Performance Indicator (KPI), a confidence level, an F-Score, a precision, a recall metric.

[0017] In some implementations of the method and network entity described herein, the degradation of the application layer ML model may be evaluated using at least one parameter comprising one or more of an Fl -score, a recall, a precision, or an accuracy.

[0018] Some implementations of the method and network entity described herein may further include operations or instructions for receiving a configuration for monitoring the performance of the ML model. In some implementations of the method and network entity described herein, the requirement comprises an identity or profile of the ML model, an ML- enabled operation, a time and area of interest, and wherein the information of the deviation of the at least one metric associated with the ML-enabled operation is received based at least in part on the received configuration for monitoring the performance of the ML model.

[0019] A processor for use in a network entity is described. The processor may include: at least one controller coupled with at least one memory and configured to cause the processor to: obtain, from at least one application entity, information of a deviation of at least one metric associated with an ML-enabled operation , wherein the ML-enabled operation is associated with an ML model executable within an application layer; detect a degradation of the ML model, based at least in part on the deviation of the at least one metric associated with the ML-enabled operation; determine an action for the ML-enabled operation based at least in part on the detected degradation of the ML model; and adapt the ML-enabled operation based at least in part on the determined action.

[0020] An application entity for wireless communication is described. The application entity may include: at least one memory; and at least one processor coupled with the at least one memory and configured to cause the application entity to: receive a request from a network entity to monitor performance of an ML model executable within an application layer; determine a deviation of at least one metric associated with an ML-enabled operation associated with the ML model; and send information for the deviation to the network entity.

[0021] The application entity may comprise an application client, a UE application enabler client, a VAL client, an edge application, a cloud application, or an edge enablement server.BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 illustrates an example of a wireless communications system in accordance with aspects of the present disclosure.

[0023] Figure 2 illustrates a functional model of AIMLE.

[0024] Figure 3 illustrates an example of a procedure to support AI / ML lifecycle management.

[0025] Figure 4 illustrates an example of a mechanism for monitoring the performance of an ML model in accordance with aspects of the present disclosure.

[0026] Figure 5 illustrates an example of a mechanism for monitoring the performance of an ML model by AIMLE in accordance with embodiments of the present disclosure.

[0027] Figure 6 illustrates an example of a mechanism for monitoring the performance of an ML model by AD AES in accordance with embodiments of the present disclosure.

[0028] Figure 7 illustrates an example of a network entity 700 in accordance with aspects of the present disclosure.

[0029] Figure 8 illustrates an example of a processor 800 in accordance with aspects of the present disclosure.

[0030] Figure 9 illustrates an example of an application entity 900 in accordance with aspects of the present disclosure.

[0031] Figure 10 illustrates a flowchart of a method performed by a network entity in accordance with aspects of the present disclosure.

[0032] Figure 11 illustrates a flowchart of a method performed by an application entity in accordance with aspects of the present disclosure.DETAILED DESCRIPTION

[0033] Artificial Intelligence (Al) and Machine Learning (ML) (described herein as AI / ML) may be integrated into wireless communication systems (also referred to as wireless networks) to enhance performance, improve network efficiency, and enable new capabilities (e.g., services, applications). In some cases, AI / ML technologies can be employed in these systems to analyze information, making intelligent decisions for operations (e.g., tasks, actions) such as resource allocation, network optimization, among other examples. In some other cases, AI / ML may also be integrated in these systems for supporting Al-enabled applications to seamlessly interact within these systems (e.g., a Radio Access Network (RAN), one or more network entities of a core network, a service or application enablement layer, etc.). As wireless communication systems evolve in 5G and beyond, AI / ML may continue to drive innovation and improve overall network intelligence.

[0034] Application Data Analytics Enabler Server (AD AES) may provide data-driven insights for enhancing performance within a Vertical Application Layer (VAL). By leveraging AD AES analytics, network operators can obtain, process, and analyze information, enabling informed decision-making for tasks such as traffic management, etc. The VAL may benefit from these analytics by dynamically adapting behavior (e.g., actions, tasks) based on network conditions, improving user experience and enabling seamless operation of vertical applications. In some cases, one or more application layer entities may support AI / ML models (e.g., techniques, methods) for AD AES analytics and VAL automation-related operations (e.g., actions, tasks). Currently, there are no effective techniques available for detecting a degradation in performance of an AI / ML model, as well as how to address the degradation (e.g., via a corrective action) if the AI / ML model is the cause for the degradation. Additionally, there is no technique for determining whether a new AI / ML model is required for a target AI / ML operation (e.g., an analytics task, a VAL- triggered AI / ML task, etc.). Because there are no current techniques (e.g., methods,procedures) related to how monitoring of performance of an AI / ML model may be executed, nor what impact the degradation of the performance of the AI / ML model may have on a target AI / ML operation, particularly when the AI / ML model may be deployed for more than one target AI / ML operation.

[0035] Aspects of the present disclosure relate to one or more techniques for a network entity, such as an AI / ML Enablement (AIMLE) server to detect a degradation associated with an AI / ML-related operation (e.g., analytics operation), and link to an AI / ML model degradation (e.g., including an expected or a predicted AI / ML model degradation) and perform an action to mitigate the cause of the degradation (e.g., via training a new AI / ML model or re-training a current AI / ML model). The one or more techniques described herein which address the shortcomings of current solutions for analytics and AI / ML model evaluation and correctness are related to the Core Network Network Data Analytics Function (NWDAF) and disregard whether AI / ML entities or analytics entities monitoring the performance are edge or UE applications. The one or more techniques described herein further address the translation capabilities between different types of degradation (e.g., operation versus model degradation), as well as aspects of prediction.

[0036] Aspects of the present disclosure are described in the context of a wireless communications system.

[0037] Figure 1 illustrates an example of a wireless communications system 100 in accordance with aspects of the present disclosure. The wireless communications system 100 may include one or more NE 102, one or more UE 104, and a core network (CN) 106. The wireless communications system 100 may support various radio access technologies. In some implementations, the wireless communications system 100 may be a 4G network, such as an LTE network or an LTE- Advanced (LIE- A) network. In some other implementations, the wireless communications system 100 may be a NR network, such as a 5G network, a 5G- Advanced (5G-A) network, or a 5G ultrawideband (5G-UWB) network. In other implementations, the wireless communications system 100 may be a combination of a 4G network and a 5G network, or other suitable radio access technology including Institute of Electrical and Electronics Engineers (IEEE) 802.11 (Wi-Fi), IEEE 802.16 (WiMAX), IEEE 802.20. The wireless communications system 100 may support radioaccess technologies beyond 5G, for example, 6G. Additionally, the wireless communications system 100 may support technologies, such as time division multiple access (TDMA), frequency division multiple access (FDMA), or code division multiple access (CDMA), etc.

[0038] The one or more NE 102 may be dispersed throughout a geographic region to form the wireless communications system 100. One or more of the NE 102 described herein may be or include or may be referred to as a network node, a base station, a network element, a network function, a network entity, a radio access network (RAN), a NodeB, an eNodeB (eNB), a next-generation NodeB (gNB), or other suitable terminology. An NE 102 and a UE 104 may communicate via a communication link, which may be a wireless or wired connection. For example, an NE 102 and a UE 104 may perform wireless communication (e.g., receive signaling, transmit signaling) over a Uu interface.

[0039] An NE 102 may provide a geographic coverage area for which the NE 102 may support services for one or more UEs 104 within the geographic coverage area. For example, an NE 102 and a UE 104 may support wireless communication of signals related to services (e.g., voice, video, packet data, messaging, broadcast, etc.) according to one or multiple radio access technologies. In some implementations, an NE 102 may be moveable, for example, a satellite associated with a non-terrestrial network (NTN). In some implementations, different geographic coverage areas 112 associated with the same or different radio access technologies may overlap, but the different geographic coverage areas may be associated with different NE 102.

[0040] The one or more UE 104 may be dispersed throughout a geographic region of the wireless communications system 100. A UE 104 may include or may be referred to as a remote unit, a mobile device, a wireless device, a remote device, a subscriber device, a transmitter device, a receiver device, or some other suitable terminology. In some implementations, the UE 104 may be referred to as a unit, a station, a terminal, or a client, among other examples. Additionally, or alternatively, the UE 104 may be referred to as an Internet-of- Things (loT) device, an Intemet-of-Everything (loE) device, or machine-type communication (MTC) device, among other examples.

[0041] A UE 104 may be able to support wireless communication directly with other UEs 104 over a communication link. For example, a UE 104 may support wireless communication directly with another UE 104 over a device-to-device (D2D) communication link. In some implementations, such as vehicle-to-vehicle (V2V) deployments, vehicle-to-everything (V2X) deployments, or cellular-V2X deployments, the communication link 114 may be referred to as a sidelink. For example, a UE 104 may support wireless communication directly with another UE 104 over a PC5 interface.

[0042] An NE 102 may support communications with the CN 106, or with another NE 102, or both. For example, an NE 102 may interface with other NE 102 or the CN 106 through one or more backhaul links (e.g., SI, N2, N2, or network interface). In some implementations, the NE 102 may communicate with each other directly. In some other implementations, the NE 102 may communicate with each other or indirectly (e.g., via the CN 106. In some implementations, one or more NE 102 may include subcomponents, such as an access network entity, which may be an example of an access node controller (ANC). An ANC may communicate with the one or more UEs 104 through one or more other access network transmission entities, which may be referred to as a radio heads, smart radio heads, or transmission-reception points (TRPs).

[0043] The CN 106 may support user authentication, access authorization, tracking, connectivity, and other access, routing, or mobility functions. The CN 106 may be an evolved packet core (EPC), or a 5G core (5GC), which may include a control plane entity that manages access and mobility (e.g., a mobility management entity (MME), an access and mobility management functions (AMF)) and a user plane entity that routes packets or interconnects to external networks (e.g., a serving gateway (S-GW), a Packet Data Network (PDN) gateway (P-GW), or a user plane function (UPF)). In some implementations, the control plane entity may manage non-access stratum (NAS) functions, such as mobility, authentication, and bearer management (e.g., data bearers, signal bearers, etc.) for the one or more UEs 104 served by the one or more NE 102 associated with the CN 106.

[0044] The CN 106 may communicate with a packet data network over one or more backhaul links (e.g., via an SI, N2, N2, or another network interface). The packet data network may include an application server. In some implementations, one or more UEs 104may communicate with the application server. A UE 104 may establish a session (e.g., a protocol data unit (PDU) session, or the like) with the CN 106 via an NE 102. The CN 106 may route traffic (e.g., control information, data, and the like) between the UE 104 and the application server using the established session (e.g., the established PDU session). The PDU session may be an example of a logical connection between the UE 104 and the CN 106 (e.g., one or more network functions of the CN 106).

[0045] In the wireless communications system 100, the NEs 102 and the UEs 104 may use resources of the wireless communications system 100 (e.g., time resources (e.g., symbols, slots, subframes, frames, or the like) or frequency resources (e.g., subcarriers, carriers)) to perform various operations (e.g., wireless communications). In some implementations, the NEs 102 and the UEs 104 may support different resource structures. For example, the NEs 102 and the UEs 104 may support different frame structures. In some implementations, such as in 4G, the NEs 102 and the UEs 104 may support a single frame structure. In some other implementations, such as in 5 G and among other suitable radio access technologies, the NEs 102 and the UEs 104 may support various frame structures (i.e., multiple frame structures). The NEs 102 and the UEs 104 may support various frame structures based on one or more numerologies.

[0046] One or more numerologies may be supported in the wireless communications system 100, and a numerology may include a subcarrier spacing and a cyclic prefix. A first numerology (e.g., / r=0) may be associated with a first subcarrier spacing (e.g., 15 kHz) and a normal cyclic prefix. In some implementations, the first numerology (e.g., / r=0) associated with the first subcarrier spacing (e.g., 15 kHz) may utilize one slot per subframe. A second numerology (e.g., / r=l) may be associated with a second subcarrier spacing (e.g., 30 kHz) and a normal cyclic prefix. A third numerology (e.g., / r=2) may be associated with a third subcarrier spacing (e.g., 60 kHz) and a normal cyclic prefix or an extended cyclic prefix. A fourth numerology (e.g., / r=3) may be associated with a fourth subcarrier spacing (e.g., 120 kHz) and a normal cyclic prefix. A fifth numerology (e.g., / r=4) may be associated with a fifth subcarrier spacing (e.g., 240 kHz) and a normal cyclic prefix.

[0047] A time interval of a resource (e.g., a communication resource) may be organized according to frames (also referred to as radio frames). Each frame may have a duration, forexample, a 10 millisecond (ms) duration. In some implementations, each frame may include multiple subframes. For example, each frame may include 10 subframes, and each subframe may have a duration, for example, a 1 ms duration. In some implementations, each frame may have the same duration. In some implementations, each subframe of a frame may have the same duration.

[0048] Additionally or alternatively, a time interval of a resource (e.g., a communication resource) may be organized according to slots. For example, a subframe may include a number (e.g., quantity) of slots. The number of slots in each subframe may also depend on the one or more numerologies supported in the wireless communications system 100. For instance, the first, second, third, fourth, and fifth numerologies (i.e., / r=0, jU=l , / r=2, jU=3, / r=4) associated with respective subcarrier spacings of 15 kHz, 30 kHz, 60 kHz, 120 kHz, and 240 kHz may utilize a single slot per subframe, two slots per subframe, four slots per subframe, eight slots per subframe, and 16 slots per subframe, respectively. Each slot may include a number (e.g., quantity) of symbols (e.g., OFDM symbols). In some implementations, the number (e.g., quantity) of slots for a subframe may depend on a numerology. For a normal cyclic prefix, a slot may include 14 symbols. For an extended cyclic prefix (e.g., applicable for 60 kHz subcarrier spacing), a slot may include 12 symbols. The relationship between the number of symbols per slot, the number of slots per subframe, and the number of slots per frame for a normal cyclic prefix and an extended cyclic prefix may depend on a numerology. It should be understood that reference to a first numerology (e.g., / r=0) associated with a first subcarrier spacing (e.g., 15 kHz) may be used interchangeably between subframes and slots.

[0049] In the wireless communications system 100, an electromagnetic (EM) spectrum may be split, based on frequency or wavelength, into various classes, frequency bands, frequency channels, etc. By way of example, the wireless communications system 100 may support one or multiple operating frequency bands, such as frequency range designations FR1 (410 MHz - 7.125 GHz), FR2 (24.25 GHz - 52.6 GHz), FR3 (7.125 GHz - 24.25 GHz), FR4 (52.6 GHz - 114.25 GHz), FR4a or FR4-1 (52.6 GHz - 71 GHz), and FR5 (114.25 GHz - 300 GHz). In some implementations, the NEs 102 and the UEs 104 may perform wireless communications over one or more of the operating frequency bands. Insome implementations, FR1 may be used by the NEs 102 and the UEs 104, among other equipment or devices for cellular communications traffic (e.g., control information, data). In some implementations, FR2 may be used by the NEs 102 and the UEs 104, among other equipment or devices for short-range, high data rate capabilities.

[0050] FR1 may be associated with one or multiple numerologies (e.g., at least three numerologies). For example, FR1 may be associated with a first numerology (e.g., / r=0), which includes 15 kHz subcarrier spacing; a second numerology (e.g., / r=l), which includes 30 kHz subcarrier spacing; and a third numerology (e.g., / r=2), which includes 60 kHz subcarrier spacing. FR2 may be associated with one or multiple numerologies (e.g., at least 2 numerologies). For example, FR2 may be associated with a third numerology (e.g., / r=2), which includes 60 kHz subcarrier spacing; and a fourth numerology (e.g., / r=3), which includes 120 kHz subcarrier spacing.

[0051] In 3 GPP the employment of Al has been discussed in different domains (RAN, core network, service or application enablement layer) as described below, either as a tool to optimize existing features and procedures by making them more intelligent, or for assisting Al-enabled apps to communicate via the mobile communications system.

[0052] Work has been carried out to investigate AI / ML support focusing particularly on AI / ML support in the NWDAF and in network assistance for AI / ML services.

[0053] Network analytics and AI / ML is deployed in the 5G core network (as described below with reference to Figure 1). This is achieved via the introduction of NWDAF considering the support of various analytics types that can be distinguished using different Analytics IDs, e.g., “UE Mobility”, “NF Load”, etc. Each NWDAF may support one or more Analytics IDs and may have the role of AI / ML inference (NWDAF Analytical Function (AnLF)) and / or AI / ML training (NWDAF Model Training Logical Function (MTLF)).

[0054] For Federated Learning (FL) use cases, federated learning has been defined amongst different NWDAF MTLFs where the ML model training is running in multiple local MTLFs.

[0055] Currently enhancements to 5G Core are specified for assisting the AI / ML operations in the application layer (between one or more AI / ML users and AI / ML server). The Network Exposure Function (NEF) may assist the AI / ML application server in scheduling available UE(s) to participate in the AI / ML operation (e.g. Federated Learning). Also, 5GC may assist the selection of UEs to serve as FL clients, by providing a list of target member UE(s), then subscribing to the NEF to be notified about the subset list of UE(s) (i.e. list of candidate UE(s)) that fulfil certain filtering criteria.

[0056] Consideration has been given to the correctness of NWDAF analytics. In particular, it is required that:An NWDAF containing MTLF with accuracy checking capability is able to provide or notify the ML model accuracy degradation to the consumers of such service.ML Model accuracy improvement can be achieved by comparing prediction using the current trained ML model and its corresponding ground truth data i.e. the corresponding true observed events.The MTLF is to reselect a new ML model or retrain the existing ML model that is provided to the AnLF when it determines ML model degradation by MTLF determining ML model degradation by collecting new test data (including input data, ground truth data and the corresponding inference) and testing the ML model accuracy. MTLF can compute accuracy by comparing the predictions and the corresponding ground truth data.

[0057] An AIMLE entity (which is a new Service Enabler Architecture Layer (SEAL) server) has been defined to support AI / ML services via a service enablement layer.

[0058] Figure 2 illustrates an on -network functional model of AIMLE. In the vertical application layer, a VAL client 202 communicates with the VAL server 204 over a VAL- UU reference point 206. VAL-UU 206 supports both unicast and multicast delivery modes. The AIMLE functional entities on the UE and the server are grouped into AIMLE client(s) 208 and AIMLE server(s) 210 respectively. The AIMLE includes a common set of services for comprehensive enablement of AIML functionality, including federated and distributed learning (e.g., FL client registration management, FL client discovery andselection), and reference points. The AIMLE services are offered to the vertical application layer (VAL). The AIMLE client 208 communicates with the AIMLE server(s) 210 over AIML-UU reference points 212. The AIMLE client 208 provides functionality to the VAL client(s) 202 over an AIML-C reference point 214. The VAL server(s) 204 communicate with the AIMLE server(s) 210 over AIML-S reference points 216. The AIMLE servers 210 communicate with the underlying 3 GPP network systems 218 using the respective 3 GPP interfaces specified by the 3 GPP network system. An AIML-E reference point 220 enables interactions between two AIMLE servers (e.g. central and edge AIMLE servers). The AIMLE server 210 interacts with an ML repository 222which serves as repository for the ML model and ML participants over AIML-R.

[0059] Lurthermore, in the SEAL layer, the AD AES is an analytics entity at enablement layer for supporting over the top analytics e.g. for server or app session performance, edge load, slice load etc. In the context of AIMLE, the AD AES may be seen as consumer of AIMLE services for supporting ML-enabled analytics. In particular, the AIMLE may be requested to train an ML model to be used for a given Analytics ID.

[0060] Previous consideration has been given as to how to support transfer learning at application enablement layers.

[0061] Figure 3 illustrates an example of a procedure to support AI / ML lifecycle management for ML model retraining and update (e.g., for transfer learning) if requested by the AIMLE consumer when model performance degradation is observed. In this solution, one necessary step was the detection of the ML model. This involves an AI / ML Model and Data Repository 302, data producers 304, an AI / ML Enablement Server 306 and AI / ML Enablement Consumer 308.

[0062] In this procedure, the AIML enablement consumer 308, which could be a VAL server or AD AES, detects the ML model performance degradation and triggers the model update (re-training). However, no consideration has been given as to how to detect a model performance degradation and how to support the correct action to avoid issues in the situation where it is the fault of the model itself, so a new model is required for the target ML task (e.g. analytics task or VAL-triggered ML task). There is therefore a requirement tospecify how the monitoring of model performance happens and what is the impact with respect to the target ML task.

[0063] There is a need to monitor ML model performance (e.g. accuracy) at the application layer assuming 3rdparty owned models and detect a possible downgrade of performance. It would also be helpful to avoid ML model performance degradation, for example by predictive detection.

[0064] Figure 4 illustrates an example of a mechanism for monitoring the performance of an ML model in accordance with aspects of the present disclosure.

[0065] At step 401, a consumer 426 of a model performance monitoring service (e.g., aVAL, an AD AES, an AIMLE) may output (e.g., transmit) a subscription request to a producer 424 of the monitoring service (e.g., an AD AES or an AIMLE server) for receiving monitoring events (e.g., predictive events or actual events) related to attributes of ML model performance at an application layer. The performance monitoring service may be considered to be an ML support function and may include one or both of the consumer 426 and the producer 424. The subscription may include the monitored events as well as the performance attributes to monitor, the service and area of interest, the ML task or analytics identifier for which the subscription is applicable and the ML method / type for which the model is going to be used. The application layer ML model is owned, provided or deployed by an application entity, wherein the application entity can be in certain embodiments a cloud or edge platform function. The consumer 426 and producer 424 may together operate as a network entity as disclosed herein.

[0066] At step 402, the producer 424 of the monitoring service may authorize the request and output (e.g., transmit) a subscription response to the consumer 426.

[0067] At step 403, the producer 424 of the monitoring service may request from an ML repository 420 to receive a list of ML participant identifiers / addresses that are associated with a target ML model. Such participants can be AIMLE or VAL entities that are training the ML model or support ML model inference. For the case of FL process, multiple ML participants may be the entities that locally train the model at the server or UE side. This request also includes ML model information related to the performance (e.g.,whether there is historical data on prior degradation, the rating of the ML model if available, the ML operation for which this model is used and its performance stats (e.g. accuracy, precision, etc.) and whether the model is used for a given ADAE analytics ID. The ML model participants may be application entities as described herein.

[0068] At step 404 the ML repository 420 may fetch the list of ML participants for the requested ML model, together with ML model information which is relevant to the subscription. Examples of such information may include information related to prior evaluation / rating of the ML model and historical data on the previous model degradations. Such information may include prior flags that the ML model has been degraded.

[0069] At step 405 the ML repository may output the list of ML model participants and relevant information to the producer 424 of the ML model monitoring service.

[0070] At step 406 the producer 424 of the monitoring service may start monitoring the ML model performance. This may include discovering the application entity or entities 422 training the model (e.g. FL clients in FL process) and requesting them to provide reports on the status of the ML model training. This may be either on-time or regular reporting or based on events based on the reporting configuration. The application entities may include UEs, application clients, UE application enabler clients, VAL clients, edge applications, cloud applications, and edge enablement servers.

[0071] At step 407 the producer 424 of the monitoring service may identify possible mismatches of the expected performance vs the achieved performance upon receiving the reports from the application entities acting as ML participants 422 (e.g., FL clients). This may include expected deviation of at least one metric. The producer 424 determines a degradation of the ML model’s performance given a target ML task (or tasks). This can be predicted or actual degradation of performance, where the cause can be determined as an issue of the ML model itself or the input data. Such cause may be either provided in step 406 from the ML participants 422, or may be identified after verifying this via multiple reports on the correctness of the ML model. The metric may include, for example, one or more of an accuracy metric, a latency metric, a Quality of Experience (QoE) metric associated with the ML-enabled operation, a Quality of Service (QoS) metric associated with the ML-enabled operation, a Key Performance Indicator (KPI), a confidence level, aprecision, a recall metric, and an F-score (which is a mechanism for comparing the performance of different ML models on the same task).

[0072] The degradation may also be evaluated on the basis of an Fl -score (which calculates overall prediction correctness), recall of earlier behavior, precision, and / or accuracy.

[0073] For example, if all FL clients provide good accuracy of the model, whereas one of the FL clients has low accuracy, then the issue lies with the data, or the data producer, or the FL client.

[0074] At step 408 the producer 424 of the monitoring service may then notify the consumer 426 that the ML model is degraded and notify the ML repository 420 for the record so that this model is not used in future tasks.

[0075] At step 409 the consumer of the monitoring service then determines an action which may include selection of another model or re-training of the model for the given ML task.

[0076] At step 410 the consumer of the monitoring service enforces the determined action, and interacts with the corresponding ML participants 422 for fetching and training a new model or re-training the existing one.

[0077] Figure 5 illustrates an example of a mechanism for monitoring the performance of an ML model in accordance with an embodiment of the present disclosure. In the embodiment of Figure 5, the producer of the monitoring service may be an AIMLE server 524 and the consumer of the monitoring service may be a VAL server 526 or any other entity which is expected to select or use the trained model. The AIMLE Server 524 and VAL server 526 can together be considered as a network entity as described herein.

[0078] At step 501 the VAL server 526 may output an ML model monitoring subscription request to the AIMLE server 524. The subscription request may include some or all of the following parameters: an ML model ID or profile, an ML operation ID or type (e.g. Model training), a VAL server ID, an area of interest and time of validity, a monitoring reporting configuration (thresholds for triggering a monitoring event, e.g.minimum accuracy, delay, whether the reporting is one time or periodical or event-based), a KPI for the ML model operation, security information, one or more policies for triggering an action based on a monitoring event (e.g. if degradation is detected, train a new model).

[0079] At step 502 The AIMLE server 524 may authorize the subscription request.

[0080] At step 503 the AIMLE server 524 may output an ML model monitoring subscription response to the VAL server 526. The response is a positive or negative acknowledgement of the subscription and may include a subscription ID in the case of success and a cause of failure in the case of failure (e.g. failure to authorize).

[0081] At step 504 the AIMLE server 524 may output a request to the ML repository 520 for a list of ML members / participants 522 which are mapped to operations involving the target ML model. The ML members may be selected or candidate members (AIMLE clients or VAL clients) which are expected or currently performing an operation with the target model (for example an FL client training a model is an ML member). The ML members may be application entities as described herein. This request may also include a requirement for ML model information, such as whether the ML model was degraded in the past or the evaluation or rating of the model based on previous ML model operations.

[0082] At step 505 the ML repository 520 may fetch the requested information including degradation stats and / or evaluation of the model.

[0083] At step 506 the ML repository 520 may output the requested ML model member list and information for the target ML model ID / profile to the AIMLE server 524.

[0084] At step 507 the AIMLE server 524, based on the received information, may output a monitoring or subscription request to the AIMLE client or clients 522 which are involved in the ML operation for the given ML model to start monitoring the performance of the ML model operation.

[0085] At step 508 the AIMLE client 522 (or VAL client in the situation where the ML member is a VAL client behind the AIMLE client) may start monitoring the ML operation performance. This may involve the continuous checking of the performance (e.g. accuracy, latency) of the ML operation (e.g. training, inference) and the triggering criteria forproviding a notification / trigger event to the AIMLE server 524 identifying a deviation in terms of ML operation performance. Such identification may be a detection of the ML operation or model degradation or event providing a mismatch in the ML operation performance with a possible cause (e.g. data producer, ML model, UE / channel conditions).

[0086] At step 509 the AIMLE client 522, after identifying a possible degradation of ML operation performance or mismatch, may output a response or notification (depending on whether it is request-response or subscribe-notify model) to the AIMLE server 524 indicating a degradation / mismatch at the target performance for the ML operation. This message may include the monitoring report comprising the expected or actual performance metric mismatch / deviation (e.g. accuracy, delay) for the ML operation, the ML model ID / profile, the VAL service / server ID and address, the possible cause of downgrade (model itself, data producer, UE capability change, radio conditions) and in case of prediction a time horizon and area of applicability and confidence level for the predicted parameter.

[0087] At step 510 the AIMLE server 524, based on the received monitoring notification / response from the one or more AIMLE clients 522, may identify an expected or predicted ML model degradation given a cause which may be one or more of (1) an issue with the data producer as input to the model, (2) the ML model itself (for example if the model is pre-trained for another ML task), and (3) conditions related to mobility and access status for the respective VAL UEs which are ML members.

[0088] In (3) the ML model degradation may be due to the fact that the performance of the model is limited by poor channel conditions for the link between a UE and the network or capability changes or high UE mobility / frequent handovers.

[0089] This step may include translating the ML operation degradation identified by the AIMLE client(s) 522 to an ML model degradation which can be either expected, current or predicted.

[0090] At step 511 the AIMLE server 524 may send to the VAL server 526 a monitoring notification including the detected ML model degradation.

[0091] At step 512 the VAL server 526 may decide to perform an action or request the AIMLE server 524 to decide an action, or the AIMLE and VAL servers may jointly decide the action. Such action may be one or more of:1. the training of a new ML model for the given ML operation by the same or different ML members 522 (AIMLE / VAL clients or server entities);2. the re-training of the existing ML model by the same or different ML members (AIMLE / VAL clients or server entities);3. the termination of the existing ML operation and initiating a new ML model operation; and4. the update of the existing ML operation’s KPIs and continuing the existing operation.

[0092] At step 513 the AIMLE server 524 may trigger an update to the ML model or operation based on the action in step 512.

[0093] Figure 6 illustrates an example of a mechanism for monitoring the performance of an ML model in accordance with an embodiment of the present disclosure. In the embodiment of Figure 6, analytics performance monitoring is provided by an ADAE Server (AD AES) 624, and an AIMLE server (or VAL server) 626 detects an expected or predicted model degradation performance based on ML-enabled analytics performance feedback.The AIMLE Server 626 may be a network entity as described herein. Such ADAE analytics may include edge performance or VAL server or session performance analytics which utilise ML methods (training by AIMLE) for predicting end-to-end performance.

[0094] There is a precondition (shown as step 600) that the ADAE server has an ongoing analytics task (e.g. VAL server performance analytics) and already utilizes AIMLE service to receive the trained ML model output to derive analytics.

[0095] At step 601 the AIMLE server 626 may output an ADAE analytics performance monitoring subscription request to the AD AES 624. The subscription request may include some or all of the following parameters: an analytics ID or type, a VAL server ID, an area of interest and time of validity, a monitoring reporting configuration (thresholds for triggering a monitoring event, e.g. minimum accuracy, confidence level, whether the reporting is one time or periodical or event-based), a KPI for the analytics service, securityinformation, one or more policies for triggering an action based on a monitoring event (e.g. if analytics degradation is detected, re-start the analytics process).

[0096] At step 602 the AD AES 624 may authorizes the subscription request.

[0097] At step 603 the ADAEA 624 may output an ADAE analytics performance monitoring subscription response to the AIMLE server 626. The response is a positive or negative acknowledgement of the subscription and includes a subscription ID in the case of success and a cause in the case of failure (e.g. failure to authorize).

[0098] At step 604 the AD AES 624 may discover application entities 622, based on the analytics ID of ADAE clients (ADAEC) or other entities (e.g. edge AD AES) 622 which are undertaking an analytics task for the given analytics ID (e.g. some local prediction at UE side or AD AES side).

[0099] At step 605 the AD AES 624 may send a request / subscription to the discovered entities 622 (ADAEC(s) or edge AD AES) indicating a monitoring requirement for the analytics task. This request may include some or all of the analytics ID, the requestor ID, the event trigger criteria for providing a monitoring report (e.g. reaching a threshold), the target KPIs to be monitored (e.g. accuracy), the area and time of interest, whether this is a predicted parameter or actual parameter to be monitored. Such request can be in form of an analytics correctness feedback request either in real time or when the analytics task finishes.

[0100] At step 606 the discovered ADAEC(s) and / or edge AD AES 622 may start monitoring the performance of the analytics task and in particular the attributes which are requested (accuracy, delay, evaluation vs ground truth data). This may involve the continuous checking of the performance (accuracy, latency) of the analytics service for providing a notification / trigger event to the AIMLE server 626 identifying a deviation in terms of ADAE analytics performance. Such identification may be a detection of a mismatch in the analytics performance with a possible cause (e.g. data producer, ML model, UE / channel conditions).

[0101] At step 607 the discovered ADAEC(s) and / or edge AD AES 622, based on the trigger criteria, send to the AD AES 624 a monitoring notification / response indicating anexpected or predicted degradation of the performance of the analytics task. Such message may also include the possible cause identified in step 606 (e.g. data producer, ML model, UE / channel conditions).

[0102] At step 608 the AD AES 624 may identify or predict an ADAE service degradation or performance (against the KPIs for that analytics ID) and send a response or notification (depending on whether it is request-response or subscribe-notify model) to the AIMLE server 626) responsible for detecting a degradation / mismatch at the target performance for the analytics task. This message may include a monitoring report comprising some or all of the expected or actual performance metric mismatch or deviation (e.g. accuracy, delay) for the analytics task, the ML model ID used, the VAL service / server ID and address, the possible cause of downgrade (model itself, data producer, UE capability change, radio conditions) and in case of prediction a time horizon and area of applicability and confidence level for the predicted parameter.

[0103] At step 609 the AIMLE server 626, based on the received monitoring notification / response from the one or more ADAE clients 622, may identify an expected or predicted ML model degradation given a cause which may be one or more of (1) an issue with the data producer as input to the model, (2) the ML model itself (for example if the model is pre-trained for another ML task), and (3) conditions related to mobility and access status for the respective VAL UEs which are ML members.

[0104] In (3) the ML model degradation may be due to the fact that the performance of the model is limited by poor channel conditions for the link between a UE and the network or capability changes or high UE mobility / frequent handovers.

[0105] This step may include translating the ADAE analytics degradation identified by the ADEA client(s) 622 to an ML model degradation which can be either expected, current or predicted.

[0106] At step 610 actions equivalent to steps 511 to 513 may be carried out, where the ML operation is equivalent to the AD AES analytics task.

[0107] Figure 7 illustrates an example of a network entity (NE) 700 in accordance with aspects of the present disclosure. The NE 700 may operate as an ML support function andmay include one or more model monitoring producers and consumers 424,426 such as an AIMLE server 524, 626, VAL server 526, 626, and AD AES 624. The NE 700 may include a processor 702, a memory 704, a controller 706, and a transceiver 708. The processor 702, the memory 704, the controller 706, or the transceiver 708, or various combinations thereof or various components thereof may be examples of means for performing various aspects of the present disclosure as described herein. These components may be coupled (e.g., operatively, communicatively, functionally, electronically, electrically) via one or more interfaces.

[0108] The processor 702, the memory 704, the controller 706, or the transceiver 708, or various combinations or components thereof may be implemented in hardware (e.g., circuitry). The hardware may include a processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), or other programmable logic device, or any combination thereof configured as or otherwise supporting a means for performing the functions described in the present disclosure.

[0109] The processor 702 may include an intelligent hardware device (e.g., a general- purpose processor, a DSP, a CPU, an ASIC, an FPGA, or any combination thereof). In some implementations, the processor 702 may be configured to operate the memory 704. In some other implementations, the memory 704 may be integrated into the processor 702. The processor 702 may be configured to execute computer-readable instructions stored in the memory 704 to cause the NE 700 to perform various functions of the present disclosure.

[0110] The memory 704 may include volatile or non-volatile memory. The memory 704 may store computer-readable, computer-executable code including instructions when executed by the processor 702 cause the NE 700 to perform various functions described herein. The code may be stored in a non-transitory computer-readable medium such the memory 704 or another type of memory. Computer-readable media includes both non- transitory computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A non-transitory storage medium may be any available medium that may be accessed by a general-purpose or special-purpose computer.

[0111] In some implementations, the processor 702 and the memory 704 coupled with the processor 702 may be configured to cause the NE 700 to perform one or more of the functions described herein (e.g., executing, by the processor 702, instructions stored in the memory 704). For example, the processor 702 may support wireless communication at the NE 700 in accordance with examples as disclosed herein. The NE 700 may be configured to support a means for receiving, from at least one application entity, information of a deviation of at least one metric associated with an ML-enabled operation, wherein the ML- enabled operation is associated with an ML model executable within an application layer; detecting a degradation of the ML model, based at least in part on the of the at least one metric associated with the ML-enabled operation deviation; determining an action for the ML-enabled operation based at least in part on the detected degradation of the ML model; and adapting the ML-enabled operation based at least in part on the determined action.

[0112] The controller 706 may manage input and output signals for the NE 700. The controller 706 may also manage peripherals not integrated into the NE 700. In some implementations, the controller 706 may utilize an operating system such as iOS®, ANDROID®, WINDOWS®, or other operating systems. In some implementations, the controller 706 may be implemented as part of the processor 702.

[0113] In some implementations, the NE 700 may include at least one transceiver 708. In some other implementations, the NE 700 may have more than one transceiver 708. The transceiver 708 may represent a wireless transceiver. The transceiver 708 may include one or more receiver chains 710, one or more transmitter chains 712, or a combination thereof.

[0114] A receiver chain 710 may be configured to receive signals (e.g., control information, data, packets) over a wireless medium. For example, the receiver chain 710 may include one or more antennas for receive the signal over the air or wireless medium. The receiver chain 710 may include at least one amplifier (e.g., a low-noise amplifier (LNA)) configured to amplify the received signal. The receiver chain 710 may include at least one demodulator configured to demodulate the receive signal and obtain the transmitted data by reversing the modulation technique applied during transmission of the signal. The receiver chain 710 may include at least one decoder for decoding the processing the demodulated signal to receive the transmitted data.

[0115] A transmitter chain 712 may be configured to generate and transmit signals (e.g., control information, data, packets). The transmitter chain 712 may include at least one modulator for modulating data onto a carrier signal, preparing the signal for transmission over a wireless medium. The at least one modulator may be configured to support one or more techniques such as amplitude modulation (AM), frequency modulation (FM), or digital modulation schemes like phase-shift keying (PSK) or quadrature amplitude modulation (QAM). The transmitter chain 712 may also include at least one power amplifier configured to amplify the modulated signal to an appropriate power level suitable for transmission over the wireless medium. The transmitter chain 712 may also include one or more antennas for transmitting the amplified signal into the air or wireless medium.

[0116] Figure 8 illustrates an example of a processor 800 in accordance with aspects of the present disclosure. The processor 800 may be an example of a processor configured to perform various operations in accordance with examples as described herein. The processor 800 may include a controller 802 configured to perform various operations in accordance with examples as described herein. The processor 800 may optionally include at least one memory 804, which may be, for example, an L1 / L2 / L3 cache. Additionally, or alternatively, the processor 800 may optionally include one or more arithmetic-logic units (ALUs) 806. One or more of these components may be in electronic communication or otherwise coupled (e.g., operatively, communicatively, functionally, electronically, electrically) via one or more interfaces (e.g., buses).

[0117] The processor 800 may be a processor chipset and include a protocol stack (e.g., a software stack) executed by the processor chipset to perform various operations (e.g., receiving, obtaining, retrieving, transmitting, outputting, forwarding, storing, determining, identifying, accessing, writing, reading) in accordance with examples as described herein. The processor chipset may include one or more cores, one or more caches (e.g., memory local to or included in the processor chipset (e.g., the processor 800) or other memory (e.g., random access memory (RAM), read-only memory (ROM), dynamic RAM (DRAM), synchronous dynamic RAM (SDRAM), static RAM (SRAM), ferroelectric RAM (FeRAM), magnetic RAM (MRAM), resistive RAM (RRAM), flash memory, phase change memory (PCM), and others).

[0118] The controller 802 may be configured to manage and coordinate various operations (e.g., signaling, receiving, obtaining, retrieving, transmitting, outputting, forwarding, storing, determining, identifying, accessing, writing, reading) of the processor 800 to cause the processor 800 to support various operations in accordance with examples as described herein. For example, the controller 802 may operate as a control unit of the processor 800, generating control signals that manage the operation of various components of the processor 800. These control signals include enabling or disabling functional units, selecting data paths, initiating memory access, and coordinating timing of operations.

[0119] The controller 802 may be configured to fetch (e.g., obtain, retrieve, receive) instructions from the memory 804 and determine subsequent instruction(s) to be executed to cause the processor 800 to support various operations in accordance with examples as described herein. The controller 802 may be configured to track memory address of instructions associated with the memory 804. The controller 802 may be configured to decode instructions to determine the operation to be performed and the operands involved. For example, the controller 802 may be configured to interpret the instruction and determine control signals to be output to other components of the processor 800 to cause the processor 800 to support various operations in accordance with examples as described herein. Additionally, or alternatively, the controller 802 may be configured to manage flow of data within the processor 800. The controller 802 may be configured to control transfer of data between registers, arithmetic logic units (ALUs), and other functional units of the processor 800.

[0120] The memory 804 may include one or more caches (e.g., memory local to or included in the processor 800 or other memory, such RAM, ROM, DRAM, SDRAM, SRAM, MRAM, flash memory, etc. In some implementations, the memory 804 may reside within or on a processor chipset (e.g., local to the processor 800). In some other implementations, the memory 804 may reside external to the processor chipset (e.g., remote to the processor 800).

[0121] The memory 804 may store computer-readable, computer-executable code including instructions that, when executed by the processor 800, cause the processor 800 to perform various functions described herein. The code may be stored in a non-transitorycomputer-readable medium such as system memory or another type of memory. The controller 802 and / or the processor 800 may be configured to execute computer-readable instructions stored in the memory 804 to cause the processor 800 to perform various functions. For example, the processor 800 and / or the controller 802 may be coupled with or to the memory 804, the processor 800, the controller 802, and the memory 804 may be configured to perform various functions described herein. In some examples, the processor 800 may include multiple processors and the memory 804 may include multiple memories. One or more of the multiple processors may be coupled with one or more of the multiple memories, which may, individually or collectively, be configured to perform various functions herein.

[0122] The one or more ALUs 806 may be configured to support various operations in accordance with examples as described herein. In some implementations, the one or more ALUs 806 may reside within or on a processor chipset (e.g., the processor 800). In some other implementations, the one or more ALUs 806 may reside external to the processor chipset (e.g., the processor 800). One or more ALUs 806 may perform one or more computations such as addition, subtraction, multiplication, and division on data. For example, one or more ALUs 806 may receive input operands and an operation code, which determines an operation to be executed. One or more ALUs 806 be configured with a variety of logical and arithmetic circuits, including adders, subtractors, shifters, and logic gates, to process and manipulate the data according to the operation. Additionally, or alternatively, the one or more ALUs 806 may support logical operations such as AND, OR, exclusive-OR (XOR), not-OR (NOR), and not- AND (NAND), enabling the one or more ALUs 806 to handle conditional operations, comparisons, and bitwise operations.

[0123] The processor 800 may support wireless communication in accordance with examples as disclosed herein. The processor 800 may be configured to or operable to support a means for obtaining, from at least one application entity, information of a deviation of at least one metric associated with an ML-enabled operation , wherein the ML- enabled operation is associated with an ML model executable within an application layer; detecting a degradation of the ML model, based at least in part on the deviation of the at least one metric associated with the ML-enabled operation; determining an action for theML-enabled operation based at least in part on the detected degradation of the ML model; and adapting the ML-enabled operation based at least in part on the determined action.

[0124] Figure 9 illustrates an example of an application entity 900 in accordance with aspects of the present disclosure. The application entity may be an ML participant 422, 522, 622 and may comprise one or more of a UE, an application client, a UE application enabler client, a VAL client, an edge application, a cloud application, and an edge enablement server. The application entity 900 may include a processor 902, a memory 904, a controller 906, and a transceiver 908. The processor 902, the memory 904, the controller 906, or the transceiver 908, or various combinations thereof or various components thereof may be examples of means for performing various aspects of the present disclosure as described herein. These components may be coupled (e.g., operatively, communicatively, functionally, electronically, electrically) via one or more interfaces.

[0125] The processor 902, the memory 904, the controller 906, or the transceiver 908, or various combinations or components thereof may be implemented in hardware (e.g., circuitry). The hardware may include a processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), or other programmable logic device, or any combination thereof configured as or otherwise supporting a means for performing the functions described in the present disclosure.

[0126] The processor 902 may include an intelligent hardware device (e.g., a general- purpose processor, a DSP, a CPU, an ASIC, an FPGA, or any combination thereof). In some implementations, the processor 902 may be configured to operate the memory 904. In some other implementations, the memory 904 may be integrated into the processor 902. The processor 902 may be configured to execute computer-readable instructions stored in the memory 904 to cause the application entity 900 to perform various functions of the present disclosure.

[0127] The memory 904 may include volatile or non-volatile memory. The memory 904 may store computer-readable, computer-executable code including instructions when executed by the processor 902 cause the application entity 900 to perform various functions described herein. The code may be stored in a non-transitory computer-readable medium such the memory 904 or another type of memory. Computer-readable media includes bothnon-transitory computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A non-transitory storage medium may be any available medium that may be accessed by a general-purpose or special-purpose computer.

[0128] In some implementations, the processor 902 and the memory 904 coupled with the processor 902 may be configured to cause the application entity 900 to perform one or more of the functions described herein (e.g., executing, by the processor 902, instructions stored in the memory 904). For example, the processor 902 may support wireless communication at the application entity 900 in accordance with examples as disclosed herein. The application entity 900 may be configured to support a means for receiving a request from a network entity to monitor performance of an ML model executable within an application layer; determining a deviation of at least one metric associated with an ML- enabled operation associated with the ML model; and sending information for the deviation to the network entity.

[0129] The controller 906 may manage input and output signals for the application entity 900. The controller 906 may also manage peripherals not integrated into the application entity 900. In some implementations, the controller 906 may utilize an operating system such as iOS®, ANDROID®, WINDOWS®, or other operating systems. In some implementations, the controller 906 may be implemented as part of the processor 902.

[0130] In some implementations, the application entity 900 may include at least one transceiver 908. In some other implementations, the application entity 900 may have more than one transceiver 908. The transceiver 908 may represent a wireless transceiver. The transceiver 908 may include one or more receiver chains 910, one or more transmitter chains 912, or a combination thereof.

[0131] A receiver chain 910 may be configured to receive signals (e.g., control information, data, packets) over a wireless medium. For example, the receiver chain 910 may include one or more antennas for receive the signal over the air or wireless medium. The receiver chain 910 may include at least one amplifier (e.g., a low-noise amplifier (LNA)) configured to amplify the received signal. The receiver chain 910 may include atleast one demodulator configured to demodulate the receive signal and obtain the transmitted data by reversing the modulation technique applied during transmission of the signal. The receiver chain 910 may include at least one decoder for decoding the processing the demodulated signal to receive the transmitted data.

[0132] A transmitter chain 912 may be configured to generate and transmit signals (e.g., control information, data, packets). The transmitter chain 912 may include at least one modulator for modulating data onto a carrier signal, preparing the signal for transmission over a wireless medium. The at least one modulator may be configured to support one or more techniques such as amplitude modulation (AM), frequency modulation (FM), or digital modulation schemes like phase-shift keying (PSK) or quadrature amplitude modulation (QAM). The transmitter chain 912 may also include at least one power amplifier configured to amplify the modulated signal to an appropriate power level suitable for transmission over the wireless medium. The transmitter chain 912 may also include one or more antennas for transmitting the amplified signal into the air or wireless medium.

[0133] Figure 10 illustrates a flowchart of a method in accordance with aspects of the present disclosure. The operations of the method may be implemented by a network entity as described herein. In some implementations, the network entity may execute a set of instructions to control the function elements of the UE to perform the described functions.

[0134] At 1002, the method may include receiving, from at least one application entity, information of a deviation of at least one metric associated with an ML-enabled operation, wherein the ML-enabled operation is associated with an ML model executable within an application layer. The operations of 1002 may be performed in accordance with examples as described herein. In some implementations, aspects of the operations of 1002 may be performed by a network entity as described with reference to Figure 7.

[0135] At 1004. the method may include detecting a degradation of the ML model, based at least in part on the of the at least one metric associated with the ML-enabled operation. The operations of 1004 may be performed in accordance with examples as described herein. In some implementations, aspects of the operations of 1004 may be performed by a network entity as described with reference to Figure 7.

[0136] At 1006, the method may include determining an action for the ML-enabled operation based at least in part on the detected degradation of the ML model. The operations of 1006 may be performed in accordance with examples as described herein. In some implementations, aspects of the operations of 1006 may be performed by a network entity as described with reference to Figure 7.

[0137] At 1008, the method may include adapting the ML-enabled operation based at least in part on the determined action.. The operations of 1008 may be performed in accordance with examples as described herein. In some implementations, aspects of the operations of 1008 may be performed by a network entity as described with reference to Figure 7.

[0138] It should be noted that the method described herein describes a possible implementation, and that the operations and the steps may be rearranged or otherwise modified and that other implementations are possible.

[0139] Figure 11 illustrates a flowchart of a method in accordance with aspects of the present disclosure. The operations of the method may be implemented by an application entity as described herein. In some implementations, the application entity may execute a set of instructions to control the function elements of the application entity to perform the described functions.

[0140] At 1102, the method may include receiving a request from a network entity to monitor performance of an ML model executable within an application layer. The operations of 1102 may be performed in accordance with examples as described herein. In some implementations, aspects of the operations of 1102 may be performed by an application entity as described with reference to Figure 9.

[0141] At 1104, the method may include determining a deviation of at least one metric associated with an ML-enabled operation associated with the ML model. The operations of 1104 may be performed in accordance with examples as described herein. In some implementations, aspects of the operations of 1104 may be performed by an application entity as described with reference to Figure 9.

[0142] At 1106, the method may include sending information for the deviation to the network entity. The operations of 1106 may be performed in accordance with examples as described herein. In some implementations, aspects of the operations of 1106 may be performed an application entity as described with reference to Figure 9.

[0143] It should be noted that the method described herein describes a possible implementation, and that the operations and the steps may be rearranged or otherwise modified and that other implementations are possible.

[0144] The description herein is provided to enable a person having ordinary skill in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to a person having ordinary skill in the art, and the generic principles defined herein may be applied to other variations without departing from the scope of the disclosure. Thus, the disclosure is not limited to the examples and designs described herein but is to be accorded the broadest scope consistent with the principles and novel features disclosed herein.

Claims

CLAIMS:

1. A network entity, comprising: at least one memory; and at least one processor coupled with the at least one memory and configured to cause the network entity to: receive, from at least one application entity, information of a deviation of at least one metric associated with a machine learning (ML)- enabled operation, wherein the ML-enabled operation is associated with an ML model executable within an application layer; detect a degradation of the ML model based at least in part on the deviation of the at least one metric associated with the ML-enabled operation; determine an action for the ML-enabled operation based at least in part on the detected degradation of the ML model; and adapt the ML-enabled operation based at least in part on the determined action.

2. The network entity of claim 1 , further comprising a ML-support function executable within an application enablement layer.

3. The network entity of claim 1 or 2, wherein the action includes at least one of: training a different ML model for the ML-enabled operation; re-training the ML model; terminating the ML-enabled operation and initiating a different ML-enabled operation associated with a different ML model; and updating KPIs of the ML-enabled operation and resume the ML-enabled operation.

4. The network entity of claim 1, 2 or 3, wherein the processor is further configured to cause the network entity to:subscribe to the at least one application entity, wherein the information of the deviation of the at least one metric associated with the ML-enabled operation is received based at least in part on the network entity being subscribed to the at least one application entity.

5. The network entity of any of claims 1 to 4, wherein the network entity comprises an artificial intelligence (AI) / ML (AI / ML) enablement layer (AIMLE) function, and wherein the processor is further configured to cause the AIMLE function to: output a command to an ML repository for a set of application entities associated with the ML model.

6. The network entity of claim 5, wherein the command includes a request for performance information of the ML model.

7. The network entity of claim 6, wherein the performance information of the ML model includes one of more of historical data on prior degradation, a rating of the ML model, a type of ML operation configured for the ML model, or performance statistics of the ML model.

8. The network entity of claim 5, 6 or 7, wherein the ML-enabled operation comprises one or more of an ML model training operation, an ML inference operation, or a data management operation.

9. The network entity of any of claims 1 to 4, comprising an edge enablement layer function.

10. The network entity of claim 9, wherein the ML-enabled operation comprises an analytics operation.

11. The network entity of claim 10, wherein the analytics operation comprises one or more of an edge analytics operation, a cloud analytics operation, or an Application Data Analytics Enabler Server (AD AES) analytics operation.

12. The network entity of any of the preceding claims, wherein the at least one metric comprises one or more of an accuracy level, a latency metric, a Quality ofExperience (QoE) metric associated with the ML-enabled operation, a Quality of Service (QoS) metric associated with the ML-enabled operation, a Key Performance Indicator (KPI), a confidence level, an F-Score, a precision, or a recall metric.

13. The network entity of any of the preceding claims, wherein the degradation is evaluated using at least one parameter comprising one or more of an Fl - score, recall, precision, or accuracy.

14. The network entity of any of the preceding claims, wherein the processor is further configured to cause the network entity to: receive a configuration for monitoring the performance of the ML model, wherein the requirement comprises an identity or profile of the ML model, an ML-enabled operation, a time and area of interest, and wherein the information of the deviation of the at least one metric associated with the ML-enabled operation is received based at least in part on the received configuration for monitoring the performance of the ML model.

15. A processor for use in a network entity, comprising: at least one controller coupled with at least one memory and configured to cause the processor to: obtain, from at least one application entity, information of a deviation of at least one metric associated with an ML-enabled operation, wherein the ML-enabled operation is associated with an ML model executable within an application layer; detect a degradation of the ML model, based at least in part on the deviation of the at least one metric associated with the ML-enabled operation; determine an action for the ML-enabled operation based at least in part on the detected degradation of the ML model; and adapt the ML-enabled operation based at least in part on the determined action..

16. An application entity for wireless communication, comprising:at least one memory; and at least one processor coupled with the at least one memory and configured to cause the application entity to: receive a request from a network entity to monitor performance of an ML model executable within an application layer; determine a deviation of at least one metric associated with an ML-enabled operation associated with the ML model; and send information for the deviation to the network entity.

17. The application entity of claim 16, wherein the performance metric comprises one or more of an Fl -score, a recall, a precision, and an accuracy.

18. The application entity of claim 16 or 17, wherein the application entity comprises an application client, a UE application enabler client, a VAL client, an edge application, a cloud application, or an edge enablement server.

19. A method of operating a network entity, comprising: receiving, from at least one application entity, information of a deviation of at least one metric associated with an ML-enabled operation, wherein the ML-enabled operation is associated with an ML model executable within an application layer; detecting a degradation of the ML model, based at least in part on the of the at least one metric associated with the ML-enabled operation deviation; determining an action for the ML-enabled operation based at least in part on the detected degradation of the ML model; and adapting the ML-enabled operation based at least in part on the determined action..

20. The method of claim 19, wherein the at least one application entity comprises one or more of an application client, a UE application enabler client, a VAL client, an edge application, a cloud application, and an edge enablement server.

Images