ML Model Performance monitoring host switching

Abstract

A method for performance monitoring of an artificial intelligence (AI) or machine learning (ML) model. A first AI / ML model is used by a user equipment (UE). The UE then receives from a network node a configuration to start monitoring the performance of the used first AI / ML model 300, 301, wherein the request includes a performance monitoring criteria associated with the used model and a triggering criteria to indicate triggering of switching of AI / ML model performance monitoring from the UE to a first apparatus, wherein the UE continues to use the first AI / ML model and a triggering criteria to indicate triggering of switching of AI / ML model performance monitoring from the first apparatus to the UE. The UE determines whether to start at least part of the configured AI / ML model performance monitoring 302 and determines whether at least one of the triggering criteria is fulfilled 303. In response to at least one of the triggering criteria being fulfilled the UE transmits a request to the first apparatus to switch at least part of the model performance monitoring while continuing to use the determined AI / ML model 304, 307b.

Description

FIELD

[0001] Various example embodiments relate generally to wireless networks and, more particularly, for machine learning (ML) performance monitoring host switching. BACKGROUND

[0002] Machine learning (ML) model or functionality performance monitoring can be performed either at a user equipment (UE) or the network. Performing model or functionality performance monitoring at the UE may useful when the UE has to perform an action based on model inference or functionality outputs and ML models or functionalities are performing a UE centric function (e.g., beam management).

[0003] Cases may exist, however, where UE side monitoring may not be as reliable as network side monitoring. SUMMARY

[0004] In an aspect of the present disclosure, a method performed by a user equipment (UE) connected to a radio access network (RAN) for performance monitoring of an artificial intelligence (AI) or machine learning (ML), Al ML, model includes using, by the UE, a first AI / ML model, receiving, from a network node, a configuration to start monitoring the performance of the used first AI / ML model, wherein the request includes a performance monitoring criteria associated with the used model and a triggering criteria to indicate triggering of switching of AI / ML model performance monitoring from the UE to a first apparatus, wherein the UE continues to use the first AI / ML model and a triggering criteria to indicate triggering of switching of AI / ML model performance monitoring from the first apparatus to the UE, determining, by the UE, whether to start at least part of the configured AI / ML model performance monitoring, determining, by the UE, whether at least one of the triggering criteria is fulfilled, and in response to at least one of the triggering criteria being fulfilled, transmitting, by the UE, a request to the first apparatus, to switch at least part of the configured AI / ML model performance monitoring from the UE to the first apparatus while continuing to use the determined AI / ML model.

[0005] In an aspect of the method, upon receiving an acknowledgement from the network node, suspending, by the UE, performing model or functionality performance monitoring.

[0006] In an aspect of the method, the method further includes continuing, by the UE, using a model associated with the model or functionality performance monitoring.

[0007] In an aspect of the method, upon receiving a negative acknowledgement from the network node, switching, by the UE, to a fall back functionality.

[0008] In an aspect of the method, the AI / ML model is an ML AI / ML functionality and the AI / ML model performance monitoring is an AI / ML functionality performance monitoring.

[0009] In an aspect of the method, a triggering criteria to discontinue AI / ML model performance monitoring includes the UE having a lack of resources to perform monitoring and lifecycle management (LCM) decisions.

[0010] In an aspect of the method, a triggering criteria to discontinue AI / ML model performance monitoring includes the UE having a lack of options for model switching.

[0011] In an aspect of the method, a triggering criteria to discontinue AI / ML model performance monitoring includes an accuracy of an AI / ML model performance for beam selection being below a non-ML model performance for beam selection.

[0012] In an aspect of the method, a triggering criteria to discontinue AI / ML model performance monitoring includes downlink channel conditions falling below a threshold.

[0013] In an aspect of the method, a triggering criteria to discontinue AI / ML model performance monitoring includes an ML model performance for estimating location being different from a global navigation satellite system (GNSS) location.

[0014] In an aspect of the method, a triggering criteria to discontinue AI / ML model performance monitoring includes a UE being unable to activate an AI / ML model.

[0015] In an aspect of the method, the method further includes determining, by the UE, whether to restart configured AI / ML model performance monitoring.

[0016] In an aspect of the method, the method further includes transmitting, by the UE, a ready message to the network node, the ready message indicating an ability of the UE to restart configured AI / ML model performance monitoring.

[0017] In an aspect of the method, upon receiving an acknowledgement from the network node, restarting, by the UE, configured AI / ML model performance monitoring.

[0018] In an aspect of the method, AI / ML model performance monitoring is discontinued by the network node.

[0019] In an aspect of the present disclosure, a UE includes at least one processor, and at least one memory storing instructions which, when executed by the at least one processor, cause the UE at least to perform any of the foregoing methods.

[0020] In an aspect of the present disclosure, a processor-readable medium storing instructions which, when executed by at least one processor of an apparatus, cause the apparatus at least to perform any of the foregoing methods.

[0021] According to some aspects, there is provided the subject matter of the independent claims. Some further aspects are defined in the dependent claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Some example embodiments will now be described with reference to the accompanying drawings.

[0023] FIG. 1 is a diagram of an example embodiment of wireless networking between a network system and a user equipment (UE), according to one illustrated aspect of the disclosure;

[0024] FIG. 2 is a diagram of example components of a network system, according to one illustrated aspect of the disclosure;

[0025] FIG. 3 is a diagram of an example embodiment of signals and operations among network (NW) node and a UE for a UE triggered model monitoring switching, according to one illustrated aspect of the disclosure;

[0026] FIG. 4 is a diagram of an example embodiment of signals and operations among a NW node and a UE for a NW triggered model monitoring switching, according to one illustrated aspect of the disclosure; and

[0027] FIG. 5 is a diagram of an example embodiment of components of a UE or of a network apparatus, according to one illustrated aspect of the present disclosure. DETAILED DESCRIPTION

[0028] In the following description, certain specific details are set forth in order to provide a thorough understanding of disclosed aspects. However, one skilled in the relevant art will recognize that aspects may be practiced without one or more of these specific details or with other methods, components, materials, etc. In other instances, well-known structures associated with transmitters, receivers, or transceivers have not been shown or described in detail to avoid unnecessarily obscuring descriptions of the aspects.

[0029] Reference throughout this specification to “one aspect” or “an aspect” means that a particular feature, structure, or characteristic described in connection with the aspect is included in at least one aspect. Thus, the appearances of the phrases “in one aspect” or “in an aspect” in various places throughout this specification are not necessarily all referring to the same aspect. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more aspects.

[0030] Embodiments described in the present disclosure may be implemented in wireless networking apparatuses, such as, without limitation, apparatuses utilizing Worldwide Interoperability for Microwave Access (WiMAX), Global System for Mobile communications (GSM, 2G), GSM EDGE radio access Network (GERAN), General Packet Radio Service (GRPS), Universal Mobile Telecommunication System (UMTS, 3G) based on basic wideband-code division multiple access (W-CDMA), high-speed packet access (HSPA), Long Term Evolution (LTE), LTE- Advanced, enhanced LTE (eLTE), 5G New Radio (5G NR), 5G Advance, 6G (and beyond) and 802.1 lax (Wi-Fi 6), among other wireless networking systems. The terra ‘eLTE’ here denotes the LTE evolution that connects to a 5G core. LTE is also known as evolved UMTS terrestrial radio access (EUTRA) or as evolved UMTS terrestrial radio access network (EUTRAN).

[0031] The present disclosure may use the term “serving network device” to refer to a network node or network device (or a portion thereof) that services a UE. As used herein, the terms “transmit to,” “receive from,” and “cooperate with,” (and their variations) include communications that may or may not involve communications through one or more intermediate devices or nodes. The term “acquire” (and its variations) includes acquiring in the first instance or reacquiring after the first instance. The term “connection” may mean a physical connection or a logical connection.

[0032] The present disclosure uses 5G NR as an example of a wireless network and may use smartphones and / or extended reality headsets as an example of UEs. It is intended and shall be understood that such examples are merely illustrative, and the present disclosure is applicable to other wireless networks and user equipment.

[0033] FIG. 1 is a diagram depicting an example of wireless networking between a network system 100 and a user equipment (UE) 150. The network system 100 may include one or more network nodes 120, one or more servers 110, and / or one or more network equipment 130 (e.g., test equipment). The network nodes 120 will be described in more detail below. As used herein, the term “network apparatus” may refer to any component of the network system 100, such as the server 110, the network node 120, the network equipment 130, any component(s) of the foregoing, and / or any other component(s) of the network system 100. Examples of network apparatuses include, without limitation, apparatuses implementing aspects of 5G NR, among others. The present disclosure describes embodiments related to 5G NR and embodiments that involve aspects defined by 3rd Generation Partnership Project (3GPP). However, it is contemplated that embodiments relating to other wireless networking technologies are encompassed within the scope of the present disclosure.

[0034] The following description provides further details of examples of network nodes. In a 5G NR network, a gNodeB (also known as gNB) may include, e.g., a node that provides new radio (NR) user plane and control plane protocol terminations towards the UE and that is connected via a NG interface to the 5G core (5GC), e.g., according to 3GPP TS 38.300 V16.6.0 (2021-06) section 3.2, which is hereby incorporated by reference herein.

[0035] A gNB supports various protocol layers, e.g., Layer 1 (LI) - physical layer, Layer 2 (L2), and Layer 3 (L3).

[0036] The layer 2 (L2) of NR is split into the following sublayers: Medium Access Control (MAC), Radio Link Control (RLC), Packet Data Convergence Protocol (PDCP) and Service Data Adaptation Protocol (SDAP), where, e.g.: o The physical layer offers to the MAC sublayer transport channels; o The MAC sublayer offers to the RLC sublayer logical channels; o The RLC sublayer offers to the PDCP sublayer 'RLC channels; o The PDCP sublayer offers to the SDAP sublayer radio bearers; o The SDAP sublayer offers to 5GC quality of service (QoS) flows; o Control channels include broadcast control channel (BCCH) and physical control channel (PCCH).

[0037] Layer 3 (L3) includes, e.g., radio resource control (RRC), e.g., according to 3GPP TS 38.300 V16.6.0 (2021-06) section 6, which is hereby incorporated by reference herein.

[0038] A gNB central unit (gNB-CU) includes, e.g., a logical node hosting, e.g., radio resource control (RRC), service data adaptation protocol (SDAP), and packet data convergence protocol (PDCP) protocols of the gNB or RRC and PDCP protocols of the en-gNB, that controls the operation of one or more gNB distributed units (gNB-DUs). The gNB-CU terminates the Fl interface connected with the gNB-DU. A gNB-CU may also be referred to herein as a CU, a central unit, a centralized unit, or a control unit.

[0039] A gNB Distributed Unit (gNB-DU) includes, e.g., a logical node hosting, e.g., radio link control (RLC), media access control (MAC), and physical (PHY) layers of the gNB or en-gNB, and its operation is partly controlled by the gNB-CU. One gNB-DU supports one or multiple cells. One cell is supported by only one gNB-DU. The gNB-DU terminates the Fl interface connected with the gNB-CU. A gNB-DU may also be referred to herein as DU or a distributed unit.

[0040] As used herein, the term “network node” may refer to any of a gNB, a gNB-CU, or a gNB-DU, or any combination of them. A RAN (radio access network) node or network node such as, e.g., a gNB, gNB-CU, or gNB-DU, or parts thereof, may be implemented using, e.g., an apparatus with at least one processor and / or at least one memory with processor-readable instructions (“program”) configured to support and / or provision and / or process CU and / or DU related functionality and / or features, and / or at least one protocol (sub-)layer of a RAN (radio access network), e.g., layer 2 and / or layer 3. Different functional splits between the central and distributed unit are possible. An example of such an apparatus and components will be described in connection with FIG. 5 below.

[0041] The gNB-CU and gNB-DU parts may, e.g., be co-located or physically separated. The gNB-DU may even be split further, e.g., into two parts, e.g., one including processing equipment and one including an antenna. A central unit (CU) may also be called baseband unit / radio equipment controller / cloud-RAN / virtual-RAN (BBU / REC / C-RAN / V-RAN), open-RAN (O-RAN), or part thereof. A distributed unit (DU) may also be called remote radio head / remote radio unit / radio equipment / radio unit (RRH / RRU / RE / RU), or part thereof. Hereinafter, in various example embodiments of the present disclosure, a network node, which supports at least one of central unit functionality or a layer 3 protocol of a radio access network, may be, e.g., a gNB-CU. Similarly, a network node, which supports at least one of distributed unit functionality or a layer 2 protocol of the radio access network, may be, e.g., a gNB-DU.

[0042] A gNB-CU may support one or multiple gNB-DUs. A gNB-DU may support one or multiple cells and, thus, could support a serving cell for a user equipment (UE) or support a candidate cell for handover, dual connectivity, and / or earner aggregation, among other procedures.

[0043] The user equipment (UE.) 150 may be or include a wireless or mobile device, an apparatus with a radio interface to interact with a RAN (radio access network), a smartphone, an in-vehicle apparatus, an loT device, or a M2M device, among other types of user equipment. Such UE 150 may include: at least one processor; and at least one memory including program code; where the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus at least to perform certain operations, such as, e.g., RRC connection to the RAN. An example of components of a UE will be described m connection with FIG. 5. In embodiments, the UE 150 may be configured to generate a message (e.g., including a cell ID) to be transmitted via radio towards a RAN (e.g., to reach and communicate with a serving cell). In embodiments, the UE 150 may generate and transmit and receive RRC messages containing one or more RRC PDUs (packet data units). Persons skilled m the art will understand RRC protocol as well as other procedures a UE may perform.

[0044] With continuing reference to FIG. 1, in the example of a 5G NR network, the network system 100 provides one or more cells, which define a coverage area of the network system 100. As described above, the network system 100 may include a gNB of a 5G NR network or may include any other apparatus configured to control radio communication and manage radio resources within a cell. As used herein, the term “resource” may refer to radio resources, such as a resource block (RB), a physical resource block (PRB), a radio frame, a subframe, a time slot, a sub-band, a frequency region, a sub-carrier, a beam, etc. In embodiments, the network node 120 may be called a base station.

[0045] FIG. 1 provides an example and is merely illustrative of a network system 100 and a UE 150. Persons skilled in the art will understand that the network system 100 includes components not illustrated in FIG. 1 and will understand that other user equipment may be in communication with the network system 100.

[0046] FIG. 2 is a block diagram of example components of the network system 100 of FIG. 1. A 5G NR network may be described as an example of the network system 100, and it is intended that aspects of the following description shall be applicable to other types of network systems, as well. The network system may operate in accordance with the signals and connections shown in FIG. 1 such that the UE 150 is m communication with the network system 100 through the radio access network 225. Additionally, the network system may be divided into user plane components and functions and control plane components and functions, as shown and described herein. Unless indicated otherwise, the terms “component”, “function”, and “service” may be used interchangeably herein, and they may refer to and be implemented by instructions executed by one or more processors.

[0047] Example functions of the components are described below. The example functions are merely illustrative, and it shall be understood that additional operations and functions may be performed by the components described herein. Additionally, the connections between components may be virtual connections over service-based interfaces such that any component may communicate with any other component. In this manner, any component may act as a service “producer,” for any other component that is a service “consumer,” to provide services for network functions.

[0048] For example, a core network 210 is described in the control plane of the network system. The core network 210 may include an authentication server function (AUSF) 211, an access and mobility function (AMF) 212, and a session management function (SMF) 213. The core network 210 may also include a network slice selection function (NSSF) 214, a network exposure function (NEF) 215, a network repository function (NRF) 216, and a unified data management function (UDM) 217, which may include a uniform data repository (UDR) 224.

[0049] Additional components and functions of the core network 210 may include an application function 218, policy control function (PCF) 219, network data analytics function (NWDAF) 220, analytics data repository function (ADRF) 221, management data analytics function (MDAF) 222, and operations and management function (0 AM) 223.

[0050] The user plane includes the UE- 150, a radio access network (RAN) 225, a user plane function (UPF) 226, and a data network (DN) 227. The RAN 225 may include one or more components described in connection with FIG. 1, such as one or more network nodes. However, the RAN 225 may not be limited to such components. The UPF 226 provides connection for data being transmitted over the RAN 225. The DN 226 identifies sendees from service providers, Internet access, and third party services, for example.

[0051] The AMF 212 processes connection and mobility' tasks. The AUSF 211 receives authentication requests from the AMF 212 and interacts with UDM 217 to authenticate and validate network responses for determination of successful authentication. The SMF 213 conducts packet data unit (PDU) session management, as well as manages session context with the UPF 226.

[0052] The NSSF 214 may select a network slicing instance (NSI) and determine the allowed network slice selection assistance information (NSSAI). This selection and determination is utilized to set the AMF 212 to provide service to the UE 150. The NEF 215 secures access to network services for third parties to create specialized network services. The NRF 216 acts as a repository to store network functions to allow the functions to register with and discover each other.

[0053] The UDM 217 generates authentication vectors for use by the AUSF 211 and ADM. 212 and provides user identification handling. The UDM 217 may be connected to the UDR 224 which stores data associated with authentication, applications, or the like. The AF 218 provides application services to a user (e.g., streaming services, etc.). ThePCF 219 provides policy control functionality. For example, the PCF 219 may assist in network slicing and mobility management, as well as provide quality of service (QoS) and charging functionality.

[0054] The NAWAF 220 collects data (e g., from the UE 150 and the network system) to perform network analytics and provide insight to functions that utilize the analytics in the providing of services. The ADRF 221 allows the storage, retrieval, and removal of data and analytics by consumers. The MDAF 222 provides additional data analytics services for network functions. The OAM 223 provides provisioning and management processing functions to manage elements in or connected to the network (e.g., UE 150, network nodes, etc.).

[0055] FIG. 2 is merely an example of components of a network system, and variations are contemplated to be within the scope of the present disclosure. In embodiments, the network system may include other components not illustrated in FIG. 2. In embodiments, the network system may not include every component illustrated in FIG. 2. In embodiments, the components and connections may be implemented with different connections than those illustrated in FIG. 2. Such and other embodiments are contemplated to be within the scope of the present disclosure.

[0056] Although further detail will be provided below, described herein is a method for a UE to inform a network node of its intention to suspend, discontinue, or stop artificial intelligence (AI) / machine learning (ML) model or functionality performance monitoring or for the network to signal the UE to suspend its ML model or functionality performance management, wherein a functionality is enabled by an ML model and wherein ML model or functionality performance monitoring can be used synonymously with ML performance monitoring. In various embodiments, the network performs model performance monitoring for some time and UE can request to start model performance monitoring when it is ready and starts model performance monitoring again based on network allowing it to perform such decisions. In various embodiments described herein, ML model or functionality may mean AI / ML model or functionality, and the terms are used interchangeably herein.

[0057] As described above, lifecycle management (LCM) for ML model or functionality performance monitoring can be performed either at UE or the network (e.g., network node). Performing model or functionality performance monitoring at the UE may useful when the UE has to perform an action based on model inference or functionality outputs and ML models or functionalities are performing a UE centric function (e.g., beam management).

[0058] However, as mentioned above, cases may exist when the UE may not be as reliable as the network in ML model or functionality performance monitoring. For example, the UE may lack resources (power, computational resources) for performance monitoring and reporting monitoring results to the network, if configured. On top of model performance monitoring of the active models, the model or functionality management framework includes offline performance monitoring of the inactive models, which can consume some energy. If a UE is running low on battery, for example, it may not be feasible to spend extra energy on ML model performance monitoring tasks. In various embodiments, the UE has the option to completely abandon use of an ML model or functionality and switch to a fallback functionality, but UE may have the alternative option to stop performing ML model or functionality performance monitoring and still keep using ML model or functionality.

[0059] In various embodiments, the UE may not be able to identify best model or functionality, e.g., the best performing model or functionality, to use or to switch to, and expects the network to be in a better position to perform LCM temporarily.

[0060] When model or functionality performance monitoring is performed at the UE, the UE is responsible for LCM decisions. However, the UE informs the network about its decisions / actions in some form. The network may analyze these decisions regularly and may observe discrepancies on the LCM decisions made by the UE. The network may not have confidence in UE side model or functionality performance monitoring due to differences in its expected performance and the UE’s ML model or functionality performance monitoring results.

[0061] In various embodiments, for beam management, for example, a network node (e.g., gNodeB) could be better suited to monitor beam management models and / or functionalities when many UEs are configured to use beam management models and / or functionalities because it could use the outputs of many UEs to perform the performance monitoring, resulting in a model or functionality performance monitoring accuracy and performance improvement. Additionally, when a UE is configured with many component carriers (CC) in a carrier aggregation (CA) configuration or a high multiple-in, multiple-out (MIMO) rank, (e.g., the number of MIMO layers), it could be a burden to measure on the full set of beams transmitted by the gNodeB gNB for the purpose of performance monitoring. The beam management enhancement provided by UE-side AI / ML models enables the UE to measure on a subset of the full set of beams transmitted by the gNodeB for beam selection. Therefore, if the gNB performs monitoring using another mechanism, (e.g., analysis of throughput or MCS selection performance), the UE would be enabled to support a larger number of CCs.

[0062] In various embodiments, for positioning enhancement, for example, monitoring the performance of a UE-side positioning model or functionality on the UE could be feasible when a large number of global navigation satellite system (GNSS) satellites are available to the UE or a legacy gNB-based, e.g., enhanced cell ID positioning (E-CID) and / or LMF-based, (e.g., new radio (NR) time difference of arrival (TDOA)), for calculating an accurate positioning estimate. In case the UE is not able to estimate an accurate position for the purpose of model or functionality performance monitoring of its UE-side positioning model or functionality, then network assistance might be utilized.

[0063] Accordingly, in various embodiments, the network may take control of model or functionality performance monitoring and perform monitoring decisions when UE has been configured to take these decisions for an ML functionality.

[0064] In various embodiments described below, the UE triggers a change in the entity performing model / functionality performance monitoring. In other various embodiments, the network (e.g., network node) triggers model / functionality performance monitoring decision making change.

[0065] The UE may trigger a change, or switch, when the UE does not have enough resources to perform the extra task of ML model / functionality performance monitoring, or it is unable to activate correct models among the options it has. The network may trigger the change / switch when network believes that the UE’s LCM decision making is not correct / reliable, and the network desires to take control of LCM decisions.

[0066] As used herein, a communication with a radio access network (RAN) may refer to and mean a communication with a portion of a RAN, such as with a network node (e.g., a DU and / or a CU), or another portion of a RAN. As used herein, a communication with a core network may refer to and mean a communication with one or more services / applications of the core network, such as AMF or another service of a core network.

[0067] As used herein, the terms “first” and “second”, or the like, may refer to a first or second instance of a message being transmitted / received by a component (e.g., UE, apparatus, etc.), or a first or second component in a sequence of described components. As such, the terms are used in a non-limiting manner, and can refer to any message, operation, device, component, or the like.

[0068] In accordance with the brief description, FIG. 3 is a diagram of an example embodiment of signals and operations among network (NW) node and a UE for a UE triggered model monitoring switching, according to one illustrated aspect of the disclosure. In various embodiments, the components depicted in FIG. 3 may correspond to similar components described above in FIGS. 1 and 2. In various embodiments, for example, the network node may include a gNB. It will be understood that a described signal may have associated operations and a described operation may have associated signals.

[0069] At operation 300, the network node transmits a message to configure the UE to perform model or functionality performance monitoring and the UE receives the message to configure the UE to perform model or functionality performance monitoring. In various embodiments, the UE has been configured to make LCM decisions for an ML functionality which include model activation / deactivation, model seiection / switching, etc. In various embodiments, the message to configure the UE to perform model or functionality performance monitoring includes LCM decision triggers as well as decision reporting configurations (such as when and how to report information related to the LCM decisions).

[0070] At operation 301, the network node transmits a message to configure the UE with triggers to stop (e.g., discontinue) and start model or functionality performance monitoring and the UE receives the message to configure the UE with triggers to stop and start model or functionality performance monitoring. The UE is therefore configured with triggers to start and stop LCM decisions. In various embodiments, these triggers are set of conditions that UE needs to meet to perform LCM, and may be described in further detail below. In various embodiments, the trigger to start and stop LCM decision making may include that a performance degradation exceeds a threshold. Although it may not be clear that poor performance is due to a wrong model or performance monitoring and decisions based thereon, the network node may take further actions to resolve it.

[0071] At operation 302, the UE performs LCM decisions based on the received configurations and measurements. Depending on the configuration settings, it may transmit information to the network node (e.g., gNB / LMF).

[0072] At operation 303, due to one of the trigger conditions being met (e.g., lack of resources to perform monitoring and LCM decisions or lack of options for model switching), the UE decides to stop making LCM decisions.

[0073] At operation 304, the UE transmits a UE model or functionality performance monitoring stop message to the network node and the network node receives the UE model or functionality monitoring stop message. In various embodiments, the UE requests the network to take control of LCM decision making. In various embodiments, the UE sends a reason code indicating the reason for the request. The request could be provided as an indication in a MAC CE associated with an index provided in the RRC Configuration related to the monitoring configuration, including where the monitoring takes place, as UE Assistance Information (UAI) containing a similar index to that which was previously described as well as a reason code, or as part of an RRC Reconfiguration Complete message upon configuration with an AI / ML functionality or upon handover.

[0074] At operation 305, the network node evaluates the possibility of taking control of LCM decisions. In various embodiments, for some ML models or functionalities, the network may not prefer to take this control, (e.g. if NW has already accepted to perform LCM for too many UEs and using an ML model or functionality if not very critical for the UE). Additionally, the UE is capable of making the measurements and then transmitting a simple indication of the model’s performance. Otherwise, the UE would spend air interface resources on transmitting the measurements that would only be used for monitoring.

[0075] At operation 306, the network node transmits an ack or NACK to the UE and the UE receives the ACK or NACK. In various embodiments, if the network is not able to take control over LCM decisions and monitoring, it sends a NACK message to the UE and the UE is configured to switch to a non-ML functionality. If the network accepts the UE’s request, the network starts ML model or functionality performance monitoring and prepares for LCM decision making, the UE is sent an ACK message.

[0076] If, at operation 306, the UE receives a NACK, at operation 307a, the UE discontinues performance monitoring and making LCM decisions, and switches to a non-ML functionality. As model or functionality performance management is a critical part of LCM, the UE is not allowed to run ML model or functionality without model or functionality performance monitoring. The UE has to deactivate ML functionality in this case.

[0077] If, at operation 306, the UE receives an ACK, at operation 307b, the UE discontinues monitoring and making LCM decisions, but continues ML functionality. At operation 307c, the network node starts network side model / functionality performance monitoring.

[0078] After some time or based on a configured trigger condition, at operation 308, the UE might evaluate based on the configuration set by the network, that it is ready to perform performance monitoring and LCM decisions again. As an example of a configured trigger, the UE might evaluate model or functionality performance and make one or more LCM decisions. If these decisions are the same as made by the NW monitoring, the UE might trigger a process to request to start LCM monitoring again.

[0079] Accordingly, at operation 309, the UE transmits a UE model or functionality management ready message to the network node and the network node receives the UE model or functionality management ready message. For example, the UE sends an indication message to let the network know of the UE’s intention to start model monitoring and LCM decision making. In various embodiments, the UE transmits information regarding its readiness for model performance monitoring (e.g., improvement in its battery, availability of more models for better matching of models to inputs, etc.).

[0080] At operation 310, the network node transmits an ACK or NACK to the UE and the UE receives the ACK or NACK. That is, the network node acknowledges the UE’s request. If, at operation 310, the transmission is a NACK, the UE continues operation 307a and the network node continues operation 307c.

[0081] If, at operation 310, the transmission is an ACK, then at operation 311a, the network node discontinues model or functionality performance monitoring and at operation 311b, the UE starts ML model or functionality performance monitoring.

[0082] The operations of FIG. 3 are merely illustrative, and variations are contemplated to be within the scope of the present disclosure. In embodiments, the operations may include other operations not illustrated in FIG. 3. In embodiments, the operations may not include every operation illustrated in FIG. 3. In embodiments, the operations may be implemented in a different order than that illustrated in FIG. 3. Such and other embodiments are contemplated to be within the scope of the present disclosure. Persons of skill in the art will appreciate that, although various example components are described as perform various functions, other components may perform those functions described in FIG. 3.

[0083] FIG. 4 is a diagram of an example embodiment of signals and operations among a NW node and a UE for a NW triggered model monitoring switching, according to one illustrated aspect of the disclosure. In various embodiments, the components depicted in FIG. 3 may correspond to similar components described above in FIGS. 1 and 2. In various embodiments, for example, the network node may include a gNB. It will be understood that a described signal may have associated operations and a described operation may have associated signals.

[0084] At operation 400, the network node transmits a message to configure the UE to perform model or functionality performance monitoring and the UE receives the message to configure the UE to perform model or functionality performance monitoring. In various embodiments, the UE has been configured to make LCM decisions for an ML functionality which include model activation / deactivation, model selection / switching etc. In various embodiments, the message to configure the UE to perform model or functionality performance monitoring includes LCM decision triggers as well as decision reporting configurations (such as when and how to report information related to the LCM decisions).

[0085] At operation 401, the network node transmits a message to configure the UE with triggers to restart model or functionality performance monitoring and the UE receives the message to configure the UE with triggers to restart model or functionality performance monitoring after stopping it based on network indication. The UE is therefore configured with triggers to restart LCM decisions. In various embodiments, these triggers are set of conditions that UE needs to meet to perform LCM, and may be described in further detail below. In various embodiments, the trigger to start and stop LCM decision making may include that a performance degradation is below expectation. Although it may not be clear that poor performance is due to a wrong model or performance monitoring, the network node may take further actions to resolve it.

[0086] At operation 402, the UE performs LCM decisions based on the received configurations and measurements. Depending on the configuration settings, it may transmit information to the network node (e.g., gNB / LMF).

[0087] At operation 403, The network node evaluates that the UE’s LCM decisions are not meeting performance requirements and it has low confidence in UE’s LCM decisions. In various embodiments, the evaluation could be triggered by quality of LCM decisions of other UEs under the same conditions. For instance, the UE is reporting performance for a functionality which is not consistent with performance of other UEs using the same ML functionality under same conditions. This could be caused due to measurement errors for performance monitoring. Such a condition triggers a procedure to switch LCM decision making from the UE to the NW. In various embodiments, the network node may be receiving reports from the UE.

[0088] At operation 404, the network node transmits a UE model or functionality performance monitoring stop message to the network node and the network node receives the UE model or functionality monitoring stop message. In various embodiments, the network node sends an indication message to UE for switching the LCM decision. In various embodiments, the network node transmits parameters / code to specify reasons for the indication. The request could be provided as an indication in a MAC CE associated with an index provided in the RRC Configuration related to the monitoring configuration, including where the monitoring takes place, or as part of an RRC Reconfiguration message upon configuration with an AI / ML functionality or upon handover.

[0089] At operation 405, the UE transmits an ACK to the network node and the network node receives the ACK.

[0090] At operation 406a the UE discontinues monitoring and making LCM decisions, but continues ML functionality. At operation 406b, the network node starts network side model / functionality performance monitoring.

[0091] After some time or based on a configured trigger condition, at operation 407, the UE might evaluate based on the configuration set by the network, that it is ready to perform performance monitoring and LCM decisions again. An example of a configured trigger, the UE might evaluate functionality performance and make one or more LCM decisions. If these decisions are the same as made by the NW monitoring, the UE might trigger a process to request to start LCM monitoring again.

[0092] Accordingly, at operation 408, the UE transmits a UE model or functionality management ready message to the network node and the network node receives the UE model or functionality management ready message. For example, the UE sends an indication message to let the network know of the UE’s intention to start model monitoring and LCM decision making. In various embodiments, the UE transmits information regarding its readiness for model performance monitoring (e.g., improvement in its battery, availability of more models for better matching of models to inputs, etc.).

[0093] At operation 409, the network node transmits an ACK or NACK to the UE and the UE receives the ACK or NACK. That is, the network node acknowledges the UE’s request. If, at operation 409, the transmission is a NACK, the UE continues operation 306a and the network node continues operation 306b.

[0094] If, at operation 409, the transmission is an ACK, then at operation 410a, the network node discontinues model or functionality performance monitoring and at operation 410b, the UE starts ML model or functionality performance monitoring.

[0095] The operations of FIG. 4 are merely illustrative, and variations are contemplated to be within the scope of the present disclosure. In embodiments, the operations may include other operations not illustrated in FIG. 4. In embodiments, the operations may not include every operation illustrated m FIG. 4. In embodiments, the operations may be implemented in a different order than that illustrated in FIG. 4. Such and other embodiments are contemplated to be within the scope of the present disclosure. Persons of skill in the art will appreciate that, although various example components are described as perform various functions, other components may perform those functions described in FIG. 4.

[0096] As mentioned above, in various embodiments, triggers may be utilized to determine switching of model or functionality performance monitoring entity (e.g., UE or network node). In various embodiments, the UE and NW triggered LCM control switching mechanisms may rely on the configuration of thresholds which cause the UE or NW to request a switch of the entity which performs monitoring.

[0097] For example, with respect to beam management, the UE measures NSetBBeams, where N SetB Beams <N SetA Beams, and where SetB is a subset of SetA. The ML model, which enables the ML-enabled beam management functionality, takes as input measurements made on SetB beams, and outputs a configured number of best beams, \ Best_B which consider the full set of SetA beams, of which at least some were not measured.

[0098] In various embodiments, to monitor the performance of a beam management model, the UE may occasionally, or as configured by the network, measure not only SetB, but also the entire setA. Using methods for best beam determination, the UE may determine that beams other than those predicted from measurements on SetB are consistently determined. Conversely, the network, if responsible for performance monitoring, could request that the UE transmits the measurements on the SetA beams.

[0099] During performance monitoring instances, the UE or network determines that based on the performance of the UE, and from the configured triggering conditions, that despite the difference between the best beams determined through performance monitoring and the best beams determined by the ML-enabled beam management method, that the performance matches expectations based on the following first and second triggering conditions.

[00100] In various embodiments, the ML model or functionality output may be compared to a non-ML output. For example, the non-ML output may indicate an order of beams quality in order of beams A, B, C, D, E, F, G, H, whereas the ML output indicates an order of beams quality in order of beams E, F, G. H, A, B, C, D. In various embodiments, a trigger may be an example threshold that \ikMikams »on\n must be present in the ML-enabled method output, NBestBeams,ML.

[00101] In various embodiments, the trigger may be based on determination of RSRP or RSRQ measurements. For example, the Monitoring Output: beams, in order of RS measurements, (e.g., SSB RSRP or RSRQ on SetA).

[00102] In various embodiments, ML-enabled methods for beam management may function differently than expected from a legacy algorithmic approach. For instance, the beam predictions produced by the ML-enabled method might not immediately make sense given the legacy calculations or measurements on the SetA beams. For example, the training of an ML-enabled method could include implicit information about beam adjacency, and because beams that are too close together could be difficult for the UE to differentiate, certain combinations of best beams are actually detrimental to UE performance, despite each of these ideally best beams would be high performing on their own, (e.g., if a UE only receives DL transmission from a single beam at a time).

[00103] Accordingly, in various embodiments, additional triggering conditions may be evaluated after the first triggering conditions have been met, indicating that there is at least a mismatch between the performance monitoring output and the ML-enabled method output.

[00104] For example, in various embodiments, an additional trigger may include an order of modulation scheme (MCS) used for downlink reception by the UE or uplink reception by the NW, where a potential trigger may include that MCS >ThresholdMcs, where the specific threshold could depend on other factors, (e.g., the configured MCS table, the average synchronization signal block (SSB) RSRP and / or RSRQ, or channel state information (CSI) reports).

[00105] In various embodiments, an additional trigger may include downlink channel conditions (e.g., RSRP, RSRQ, Channel state information).

[00106] In various embodiments, an additional trigger may include low bit error rate (BER) or number of negative acknowledgements (NACKs) are being produced by the UE or NW, where a potential trigger may include BER <ThresholdBER (%), or NACK <ThresholdNACK (%).

[00107] In various embodiments, an additional trigger may include a high number of simultaneous beams are being scheduled toward the UE, where a potential trigger may include Ndlbearns >ThresholdNDLBearns-

[00108] Further, any combination of the above criteria may be used to determine performance. In various embodiments, the criteria could also be reversed to trigger on performance lower than a threshold.

[00109] The network node (e.g., gNB) may provide the UE with different trigger profiles in advance to quickly switch the UE, possibly through the use of a MAC CE, to change its triggers based on the gNB’s perception of the UE’s performance.

[00110] In various embodiments, the triggering conditions described above could be configured on the UE. A gNB includes this information and could evaluate any of them on its own, or in combination with one another, without reporting the results to the UE. Additionally, the gNB could use measurements transmitted by the UE, or signals measured by the gNB, to determine the expected performance of the UE, including, for example, reports of SSB RSRP or RSRQ, and / or measurements on sounding reference signals (SRS), which are uplink transmissions of reference signals to the gNB.

[00111] In various embodiments, an overall criteria for requesting to stop performance monitoring by the UE or NW may include the UE or NW detects high performance, but the ML model or functionality performance monitoring indicates poor ML-enabled method performance.

[00112] With regard to the monitoring of positioning performance monitoring, a trigger may include the UE estimating its position using a GNSS positioning method. The trigger for switching monitoring back to the gNB may include the following: during a monitoring occasion, the UE estimates its position using a GNSS positioning method, and the UE compares the GNSS estimate to the ML-enabled method prediction, and the GNSS and ML-enabled method outputs are different by more than a threshold, which could be provided, (e.g., in meters).

[00113] For example, the ML model or functionality performance monitoring, (e.g., based on pre-provisioned samples tagged with ground truth (GT) for the area), indicates that the model is performing accurately to nearest 50cm. However, the UE, based on the difference between GNSS and its output determines that the accuracy threshold has not been met, and thus, the ML model or functionality performance monitoring is not functioning properly. Accordingly, once the UE indicates to the gNB that its performance monitoring mechanism is not performing accurately, the gNB may take over monitoring with its own performance monitoring mechanism, which, in various embodiments, may utilize a legacy positioning approach.

[00114] The following describes operations from the perspective of UE. From such a perspective, a method may include using, by the UE, a first AI / ML model, receiving, from a network node, a configuration to start monitoring the performance of the used first AI / ML model, wherein the request includes a performance monitoring criteria associated with the used model and a triggering criteria to indicate triggering of switching of AI / ML model performance monitoring from the UE to a first apparatus, wherein the UE continues to use the first AI / ML model and a triggering criteria to indicate triggering of switching of AI / ML model performance monitoring from the first apparatus to the UE, determining, by the UE, whether to start at least part of the configured AI / ML model performance monitoring, determining, by the UE, whether at least one of the triggering criteria is fulfilled, and in response to at least one of the triggering criteria being fulfilled, transmitting, by the UE, a request to the first apparatus, to switch at least part of the configured AI / ML model performance monitoring from the UE to the first apparatus while continuing to use the determined AI / ML model.

[00115] Referring now to FIG. 5, there is shown a block diagram of example components of a UE or a network apparatus (e.g., of a RAN or a core network). The apparatus includes an electronic storage 510, a processor 520, a network interface 540, and a memory 550. The various components may be communicatively coupled with each other. The processor 520 may be and may include any type of processor, such as a single-core central processing unit (CPU), a multi-core CPU, a microprocessor, a digital signal processor (DSP), a System-on-Chip (SoC), or any other type of processor. The memory 550 may be a volatile type of memory, e.g., RAM, or a non-volatile type of memory, e.g., NAND flash memory. The memory 550 includes processor-readable instructions that are executable by the processor 520 to cause the apparatus to perform various operations, including those mentioned herein, such as the operations described in FIGS. 3A-4B.

[00116] The electronic storage 510 may be and include any type of electronic storage used for storing data, such as hard disk drive, solid state drive, optical disc, and / or other non-transitory computer-readable mediums, among other types of electronic storage. The electronic storage 510 stores processor-readable instructions for causing or configured for causing the apparatus to perform its operations and also stores data associated with such operations, such as storing data relating to 5G NR standards, among other data. The network interface 540 may implement wireless networking technologies such as 5G NR and / or other wireless networking technologies.

[00117] The components shown in FIG. 5 are merely examples, and persons skilled in the art will understand that an apparatus includes other components not illustrated and may include multiples of any of the illustrated components. Such and other embodiments are contemplated to be within the scope of the present disclosure. For example, a transmitter and a receiver may be included as components for transmitting and receiving signals.

[00118] Further embodiments of the present disclosure include the following examples.

[00119] Example 1.1. A UE, comprising: means for using, by the UE, a first Al ML model; means for receiving, from a network node, a configuration to start monitoring the performance of the used first AI / ML model, wherein the request includes a performance monitoring criteria associated with the used model and a triggering criteria to indicate triggering of switching of AI / ML model performance monitoring from the UE to a first apparatus, wherein the UE continues to use the first AI / ML model and a triggering criteria to indicate triggering of switching of AI / ML model performance monitoring from the first apparatus to the UE; means for determining, by the UE, whether to start at least part of the configured AI / ML model performance monitoring; means for determining, by the UE, whether at least one of the triggering criteria is fulfilled; and means for, in response to at least one of the triggering criteria being fulfilled, transmitting, by the UE, a request to the first apparatus, to switch at least part of the configured AI / ML model performance monitoring from the UE to the first apparatus while continuing to use the determined AI / ML model.

[00120] Example 1.2. The UE of example 1.1, wherein upon receiving an acknowledgement from the network node, suspending, by the UE, performing model or functionality performance monitoring.

[00121] Example 1.3. The UE of example 1.2, further comprising means for continuing, by the UE, using a model associated with the model or functionality performance monitoring.

[00122] Example 1.4. The UE of example 1.1, wherein upon receiving a negative acknowledgement from the network node, switching, by the UE, to a fall back functionality.

[00123] Example 1.5. The UE of example 1.1, wherein the AI / ML model is an ML AI / ML functionality and the AI / ML model performance monitoring is an AI / ML functionality performance monitoring.

[00124] Example 1.6. A UE as in any one of examples 1.1 to 1.5, wherein a triggering criteria to discontinue AI / ML model performance monitoring includes the UE having a lack of resources to perform monitoring and lifecycle management (LCM) decisions.

[00125] Example 1.7. A UE as in any one of examples 1.1 to 1.5, wherein a triggering criteria to discontinue AI / ML model performance monitoring includes the UE having a lack of options for model switching.

[00126] Example 1.8. A UE as in any one of examples 1.1 to 1.5, wherein a triggering criteria to discontinue AI / ML model performance monitoring includes an accuracy of an AI / ML model performance for beam selection being below a non-ML model performance for beam selection.

[00127] Example 1.9. A UE as in any one of examples 1.1 to 1.5, wherein a triggering criteria to discontinue AI / ML model performance monitoring includes downlink channel conditions falling below a threshold.

[00128] Example 1.10. A UE as in any one of examples 1.1 to 1.5, wherein a triggering criteria to discontinue AI / ML model performance monitoring includes an ML model performance for estimating location being different from a global navigation satellite system (GNSS) location.

[00129] Example 1.11. A UE as in any one of examples 1.1 to 1.5, wherein a triggering criteria to discontinue AI / ML model performance monitoring includes a UE being unable to activate an AI / ML model.

[00130] Example 1.12. A UE as in any one of examples 1.1 to 1.11, further comprising means for determining, by the UE, whether to restart configured AI / ML model performance monitoring.

[00131] Example 1.13. The UE of example 1.12, further comprising means for transmitting, by the UE, a ready message to the network node, the ready message indicating an ability of the UE to restart configured AI / ML model performance monitoring.

[00132] Example 1.14. The UE of example 1.13, wherein upon receiving an acknowledgement from the network node, restarting, by the UE, configured AI / ML model performance monitoring.

[00133] Example 1.15. The UE of example 1.13, wherein AI / ML model performance monitoring is discontinued by the network node.

[00134] The embodiments and aspects disclosed herein are examples of the present disclosure and may be embodied in various forms. For instance, although certain embodiments herein are described as separate embodiments, each of the embodiments herein may be combined with one or more of the other embodiments herein. Specific structural and functional details disclosed herein are not to be interpreted as limiting, but as a basis for the claims and as a representative basis for teaching one skilled in the art to variously employ the present disclosure in virtually any appropriately detailed structure. Like reference numerals may refer to similar or identical elements throughout the description of the figures.

[00135] The phrases “in an aspect,” “in aspects,” “in various aspects,” “in some aspects,” or “in other aspects” may each refer to one or more of the same or different aspects in accordance with this present disclosure. The phrase “a plurality of’ may refer to two or more.

[00136] In various embodiments, the terms “first message” and “second message”, as well as any subsequent messages may refer to any messages that are transmitted or received in an order and are not necessarily limited to any particular message.

[00137] The phrases “in an embodiment,” “in embodiments,” “m various embodiments,” “in some embodiments,” or “in other embodiments” may each refer to one or more of the same or different embodiments in accordance with the present disclosure. A phrase in the form “A or B” means “(A), (B), or (A and B).” A phrase in the form “at least one of A, B, or C” means “(A); (B); (C); (A and B); (A and C); (B and C); or (A, B, and C).”

[00138] Any of the herein described methods, programs, algorithms or codes may be converted to, or expressed in, a programming language or computer program. The terms “programming language” and “computer program,” as used herein, each include any language used to specify instructions to a computer, and include (but is not limited to) the following languages and their derivatives: Assembler, Basic, Batch files, BCPL, C, C+, C++, Delphi, Fortran, Java, JavaScript, machine code, operating system command languages, Pascal, Perl, PL1, Python, scripting languages, Visual Basic, metalanguages which themselves specify programs, and all first, second, third, fourth, fifth, or further generation computer languages. Also included are database and other data schemas, and any other meta-languages. No distinction is made between languages which are interpreted, compiled, or use both compiled and interpreted approaches. No distinction is made between compiled and source versions of a program. Thus, reference to a program, where the programming language could exist in more than one state (such as source, compiled, object, or linked) is a reference to any and all such states. Reference to a program may encompass the actual instructions and / or the intent of those instructions.

[00139] While aspects of the present disclosure have been shown in the drawings, it is not intended that the present disclosure be limited thereto, as it is intended that the present disclosure be as broad in scope as the art will allow and that the specification be read likewise. Therefore, the above description should not be construed as limiting, but merely as exemplifications of particular aspects. Those skilled in the art will envision other modifications within the scope and spirit of the claims appended hereto.

Claims

1. A method performed by a user equipment (UE) connected to a radio access network (RAN) for performance monitoring of an artificial intelligence (AI) or machine learning (ML), AI / ML, model, the method comprising:using, by the UE, a first AI / ML model;receiving, from a network node, a configuration to start monitoring the performance of the used first AI / ML model, wherein the request includes a performance monitoring criteria associated with the used model and a triggering criteria to indicate triggering of switching of AI / ML model performance monitoring from the UE to a first apparatus, wherein the UE continues to use the first AI / ML model and a triggering criteria to indicate triggering of switching of AI / ML model performance monitoring from the first apparatus to the UE;determining, by the UE, whether to start at least part of the configured AI / ML model performance monitoring;determining, by the UE, whether at least one of the triggering criteria is fulfilled; andin response to at least one of the triggering criteria being fulfilled, transmitting, by the UE, a request to the first apparatus, to switch at least part of the configured AI / ML model performance monitoring from the UE to the first apparatus while continuing to use the determined AI / ML model.

2. The method of claim 1, wherein upon receiving an acknowledgement from the network node, suspending, by the UE, performing model or functionality performance monitoring.

3. The method of claim 2, further comprising continuing, by the UE, using a model associated with the model or functionality performance monitoring.

4. The method of claim 1, wherein upon receiving a negative acknowledgement from the network node, switching, by the UE, to a fall back functionality.

5. The method of claim 1, wherein the AI / ML model is an ML AI / ML functionality and the AI / ML model performance monitoring is an AI / ML functionality performance monitoring.

6. A method as in any one of claims 1 to 5, wherein a triggering criteria to discontinue AI / ML model performance monitoring includes the UE having a lack of resources to perform monitoring and lifecycle management (LCM) decisions.

7. A method as in any one of claims 1 to 5, wherein a triggering criteria to discontinue AI / ML model performance monitoring includes the UE having a lack of options for model switching.

8. A method as in any one of claims 1 to 5, wherein a triggering criteria to discontinue AI / ML model performance monitoring includes an accuracy of an AI / ML model performance for beam selection being below a non-ML model performance for beam selection.

9. A method as in any one of claims 1 to 5, wherein a triggering criteria to discontinue AI / ML model performance monitoring includes downlink channel conditions falling below a threshold.

10. A method as in any one of claims 1 to 5, wherein a triggering criteria to discontinue AI / ML model performance monitoring includes an ML model performance for estimating location being different from a global navigation satellite system (GNSS) location.

11. A method as in any one of claims 1 to 5, wherein a triggering criteria to discontinue AI / ML model performance monitoring includes a UE being unable to activate an AI / ML model.

12. A method as in one of claims 1 to 11, further comprising determining, by the UE, whether to restart configured AI / ML model performance monitoring.

13. The method of claim 12, further comprising transmitting, by the UE, a ready message to the network node, the ready message indicating an ability of the UE to restart configured AI / ML model performance monitoring.

14. The method of claim 13, wherein upon receiving an acknowledgement from the network node, restarting, by the UE, configured AI / ML model performance monitoring.

15. The method of claim 13, wherein AI / ML model performance monitoring is discontinued by the network node.

16. A user equipment (UE), comprising:at least one processor; andat least one memory storing instructions which, when executed by the at least one processor, causes the apparatus at least to perform:using, by the UE, a first artificial intelligence / machine learning (AI / ML) model;receiving, from a network node, a configuration to start monitoring the performance of the used first AI / ML model and a triggering criteria to indicate triggering of switching of AI / ML model performance monitoring from the first apparatus to the UE, wherein the request includes a performance monitoring criteria associated with the used model and a triggering criteria to indicate triggering of switching of AI / ML model performance monitoring from the UE to a first apparatus, wherein the UE continues to use the first AI / ML model;determining, by the UE, whether to start at least part of the configured AI / ML model performance monitoring;determining, by the UE, whether at least one of the triggering criteria is fulfilled; andin response to at least one of the triggering criteria being fulfilled, transmitting, by the UE, a request to the first apparatus, to switch at least part of the configured AI / ML model performance monitoring from the UE to the first apparatus while continuing to use the determined AI / ML model.

17. A user equipment (UE), comprising:at least one processor; andat least one memory storing instructions which, when executed by the at least one processor, cause the apparatus at least to perform a method as in any of claims 1-14.

18. A processor-readable medium storing instructions which, when executed by at least one processor of an apparatus, cause the apparatus at least to perform a method as in any one of claims 1-14.

Images