Event driven reports for UE side performance monitoring of artificial intelligence based beam management using a UE side model

Abstract

An apparatus to process configuration information for monitoring of artificial intelligence (AI) or machine learning (ML) (AI / ML) based beam management, process measurements of a first set of beams using at least part of the configuration information to obtain a first set of beam measurements, compare the first set of beam measurements to predicted beam results obtained by inputting a second set of beam measurements into one or more AI / ML models used for the AI / ML based beam management, determine an event has occurred based on the configuration information and the comparing of the first set of beam measurements to the predicted beam results, wherein the event is indicative of one or more errors of the one or more AI / ML models and generate an event driven report based on the event, the event driven report comprising information relating to the performance of the one or more AI / ML models.

Description

Event Driven Reports for UE Side Performance Monitoring of Artificial Intelligence Based Beam Management Using a UE Side ModelInventors: Huaning Niu, Chunxuan Ye, Dawei Zhang, Haitong Sun, Oghenekome Oteri, Wei Zeng and Weidong YangPriority / Incorporation By Reference

[0001] This application claims priority to U.S. Provisional Patent Application Serial No. 63 / 644,575 filed on May 9, 2024, and entitled "Event Driven Reports for UE Side Performance Monitoring of Artificial Intelligence Based Beam Management Using a UE Side Model," the entirety of which is incorporated by reference herein.Technical Field

[0002] This application relates generally to wireless communication systems, including apparatuses, systems, and methods using artificial intelligence (Al) and / or machine learning (ML) models for beam management in wireless communication systems.Background

[0003] 5G New Radio (NR) has introduced many radio access network (RAN) and core network (CN) enhancements, as well as an enhanced security architecture. Artificial intelligence (Al) and / or machine learning (ML) processes, e.g., deep learning neural networks, may be used to facilitate and optimize certain decisions in one or more network functionalities (e.g., in the RAN or CN) . For example, the use cases for AI / ML for the air interface include channel state information (CSI) feedback enhancement (e.g., overhead reduction, improved accuracy, prediction) ; beam management (e.g., beam prediction in time, and / or spatial domain for overhead and latency reduction, beamselection accuracy improvement) ; and positioning accuracy enhancements. Additionally, the AI / ML services may be used by applications at the UE, the RAN, or external to the UE / RAN (e.g., Al-as-a-Service (AlaaS) .

[0004] In any of these use cases, one or multiple UEs served by the RAN, or the RAN itself (e.g. , a RAN node such as a gNB) , may function as an Al agent that trains all or part of the AI / ML model (s) . For example, a UE may train the model based on, e.g. , data collected by the UE (e.g. , radio-related measurements, application-related measurements, sensor input, etc. ) .

[0005] In many scenarios, it is crucial to ensure that the trained AI / ML models meet a minimum required quality and may be trusted by the clients (e.g. , network functions, UEs and / or external applications) using the AI / ML services. For example, where AI / ML models are used for beam management, it may be beneficial to do performance monitoring to determine when the AI / ML models should be discontinued or changed due to poor performance .Summary

[0006] Some example embodiments are related to an apparatus having processing circuitry coupled to memory, the processing circuitry configured to process, based on signaling received from a base station, configuration information for performing monitoring of artificial intelligence (Al) or machine learning (ML) (AI / ML) based beam management, process measurements performed for a first set of beams using at least part of the configuration information to obtain a first set of beam measurements, compare the first set of beam measurements to predicted beam results, the predicted beam results obtained byinputting a second set of beam measurements into one or more AI / ML models used for the AI / ML based beam management, determine an event has occurred based on the configuration information and the comparing of the first set of beam measurements to the predicted beam results, wherein the event is indicative of one or more errors of the one or more AI / ML models and generate, for transmission to the base station, an event driven report based on the event, the event driven report comprising information relating to the performance of the one or more AI / ML models.

[0007] Other example embodiments are related to an apparatus having processing circuitry coupled to memory, the processing circuitry configured to generate, for transmission to a user equipment (UE) , configuration information to perform monitoring of artificial intelligence (Al) or machine learning (ML) (AI / ML) based beam management, the configuration information including information for the UE to measure a first set of beams to obtain a first set of beam measurements and for the UE to determine if there are errors in one or more AI / ML models used for the AI / ML based beam management, process, based on signals received from the UE, an event driven report, the event driven report comprising event information indicative of one or more errors of the one or more AI / ML models, the event information generated by the UE based on comparing the first set of beam measurements to predicted beam results obtained by inputting a second set of beam measurements into the one or more AI / ML models and determine an action to take based on the event driven report, the action including de-activation of the one or more AI / ML models and / or switching to a different AI / ML model.Brief Description of the Drawings

[0008] Fig. 1 shows a network arrangement according to various example embodiments.

[0009] Fig. 2 shows an example UE according to various example embodiments.

[0010] Fig. 3 illustrates an example base station according to various example embodiments.

[0011] Fig. 4 is an example flow diagram illustrating an example method for UE side performance monitoring of Al based beam management according to various example embodiments.

[0012] Fig. 5 shows an example of a window-based evaluation method for determining an event for an event driven report to be sent to the network according to various example embodiments.

[0013] Fig. 6 shows an example of a counter-based evaluation method for determining an event for an event driven report to be sent to the network according to various example embodiments.

[0014] Figs. 7A and 7B show an example of a counter and timer-based evaluation method for determining an event for an event driven report to be sent to the network according to various example embodiments.Detailed Description

[0015] The example embodiments may be further understood with reference to the following description and the related appended drawings, wherein like elements are provided with the same reference numerals. The example embodiments describeapparatuses, systems, and methods for ensuring that artificial intelligence (Al) and / or machine learning (ML) models used for beam management in a network are trustworthy with regard to quality. The example embodiments provide signaling and reporting mechanisms for determining and reporting events indicating that UE side performance monitoring of the AI / ML based beam management has found errors with the AI / ML models such that the AI / ML models should be discontinued or changed.

[0016] In some example embodiments, an apparatus (such as the UE 110) may be configured to receive configuration information from the network for performing monitoring of AI / ML based beam management that uses AI / ML models. The UE may determine if the AI / ML models are performing acceptably by comparing actual beam measurements to the predicted beam results from the AI / ML models. The UE may measure a first set of beams using at least part of the configuration information to obtain a first set of beam measurements. The first set of beam measurements may be compared to predicted beam results, which are obtained by inputting a second set of beam measurements into one or more AI / ML models. The UE determines an event has occurred based on the configuration information and the comparison of the first set of beam measurements to the predicted beam results, wherein the comparison includes making a decision on whether each predicted beam result is accurate or correct based on the actual beam measurements. If an event occurs indicating one or more errors of the AI / ML models, the UE may send an event driven report to the network. The event driven report may include information relating to the performance of the one or more AI / ML models so that the network may take action based on the event driven report.

[0017] The example aspects are described with regard to a UE . However, the use of a UE is provided for illustrative purposes. The example aspects may be utilized with any electronic component that may establish a connection with a network and is configured with the hardware, software, and / or firmware to exchange information and data with the network. Therefore, the UE as described herein is used to represent any electronic component that is capable of accessing a wireless network and performing AI / ML training or inferencing operations.

[0018] The example aspects are described with regard to the network being a 5G New Radio (NR) network and a base station being a next generation Node B (gNB) . However, the use of the 5G NR network and the gNB are provided for illustrative purposes. The example aspects may apply to any type of network that utilizes similar functionalities. For example, some AI / ML operations may be RAT-independent .

[0019] The example embodiments are further described with regard to artificial intelligence (Al) and / or machine learning (ML) based operations. Any number of different AI / ML models may be used, depending on UE and network implementation. For example, in some embodiments, advanced AI / ML techniques (e.g., a deep learning neural network (NN) ) may be used while in other embodiments simpler AI / ML techniques (e.g. , a decision tree) may be used. Further, the various types of models may use different types of data for training the model, including, e.g. , radiorelated measurements, application-related measurements or sensor data. Thus, reference to any particular AI / ML-based model is provided for illustrative purposes. The example aspects described herein may apply to any type of AI / ML-based modeling that uses a training phase and an inference phase that may beexecuted at a UE, a RAN (e.g. , a network node such as a base station) , and / or a network-side function or entity (e.g., a core network element such as a location management function (LMF) for providing UE positioning services; an application server; etc. ) .

[0020] In some embodiments, the Al agent may be a user equipment (UE) in the 5G New Radio (NR) radio access network (RAN) while in other embodiments, the Al agent is a node of the RAN (e.g., a gNB) or a network-side entity, e.g., the core network, RAN or an application server. The techniques described herein may be used regardless of whether the Al agent is a UE, the RAN, or a network-side node and regardless of whether the Al manager is a UE, the RAN, or a network-side node. The methods by which the UE, RAN or network-side node are enabled with Al agent or Al manager functionalities are varied and may depend on any combination of preconfigured functionalities, RAN conf iguration / indication, CN entity conf iguration / indication, UE indication, etc.

[0021] Thus, although some techniques are described with respect to a UE being enabled for Al agent functionalities and a RAN (or network-side) node being enabled for Al manager functionalities, any one of the aforementioned entities may serve as the Al manager (e.g. , providing one or more types of metrics, assistance information, etc. ) or as the Al agent (e.g. , training the model, evaluating the metrics, and reporting the trained model) . Additionally, in some scenarios, the Al agent and the Al manager may both be network-side nodes or functionalities (e.g., the Al agent is a base station and the Al manager is a core network entity) or may both be UEs (e.g. , the Al agent is a first UE and the Al manager is a second UE connected to the first UE via a sidelink) .

[0022] Fig. 1 shows an example network arrangement 100 according to various example embodiments. The example network arrangement 100 includes a user equipment (UE) 110. The UE may be any type of electronic component that is configured to communicate via a network, e.g., mobile phones, tablet computers, smartphones, phablets, embedded devices, wearable devices, Cat-M devices, Cat-Mi devices, MTC devices, eMTC devices, other types of Internet of Things (loT) devices, etc. An actual network arrangement may include any number of UEs being used by any number of users. Thus, the example of a single UE 110 is merely provided for illustrative purposes.

[0023] The UE 110 may communicate directly with one or more networks. In the example of the network arrangement 100, the networks with which the UE 110 may wirelessly communicate are a 5G NR radio access network (5G NR-RAN) 120, an LTE radio access network (LTE-RAN) 122 and a wireless local access network (WLAN) 124. Therefore, the UE 110 may include a 5G NR chipset to communicate with the 5G NR-RAN 120, an LTE chipset to communicate with the LTE-RAN 122 and an ISM chipset to communicate with the WLAN 124. However, the UE 110 may also communicate with other types of networks (e.g. , legacy cellular networks) and the UE 110 may also communicate with networks over a wired connection. With regard to the example aspects, the UE 110 may establish a connection with the 5G NR-RAN 120.

[0024] The 5G NR-RAN 120 and the LTE-RAN 122 may be portions of cellular networks that may be deployed by cellular providers (e.g., Verizon, AT&T, T-Mobile, etc. ) . These networks 120, 122 may include, for example, cells or base stations (Node Bs, eNodeBs, HeNBs, eNBS, gNBs, gNodeBs, macrocells, microcells,small cells, femtocells, etc.) that are configured to send and receive traffic from UEs that are equipped with the appropriate cellular chip set. The WLAN 124 may include any type of wireless local area network (WiFi, Hot Spot, IEEE 802. llx networks, etc . ) .

[0025] The UE 110 may connect to the 5G NR-RAN via at least one of the next generation nodeB (gNB) 120A and / or the gNB 120B. Reference to two gNBs 120A, 120B is merely for illustrative purposes. The example aspects may apply to any appropriate number of gNBs.

[0026] In addition to the networks 120, 122 and 124 the network arrangement 100 also includes a cellular core network 130, the Internet 140, an IP Multimedia Subsystem (IMS) 150, and a network services backbone 160. The cellular core network 130, e.g., the 5GC for the 5G NR network, may be considered to be the interconnected set of components that manages the operation and traffic of the cellular network. The cellular core network 130 also manages the traffic that flows between the cellular network and the Internet 140. The core network 130 may include, e.g., a location management function (LMF) to support location determinations for a UE .

[0027] The IMS 150 may be generally described as an architecture for delivering multimedia services to the UE 110 using the IP protocol. The IMS 150 may communicate with the cellular core network 130 and the Internet 140 to provide the multimedia services to the UE 110. The network services backbone 160 is in communication either directly or indirectly with the Internet 140 and the cellular core network 130. The network services backbone 160 may be generally described as aset of components (e.g. , servers, network storage arrangements, etc. ) that implement a suite of services that may be used to extend the functionalities of the UE 110 in communication with the various networks .

[0028] Fig. 2 shows an example UE 110 according to various example embodiments. The UE 110 will be described with regard to the network arrangement 100 of Fig. 1. The UE 110 may represent any electronic device and may include a processor 205, a memory arrangement 210, a display device 215, an input / output (I / O) device 220, a transceiver 225, and other components 230. The other components 230 may include, for example, an audio input device, an audio output device, a battery that provides a limited power supply, a data acquisition device, ports to electrically connect the UE 110 to other electronic devices, sensors to detect conditions of the UE 110, etc. Additionally, the UE 110 may be configured to access an SNPN.

[0029] The processor 205 may be configured to execute a plurality of engines for the UE 110. For example, the engines may include an AI / ML engine 235 for performing various operations related to training an AI / ML model (as an Al agent) or facilitating the training and generation of a trained AI / ML model via one or more remote Al agents (as an Al manager) . In some embodiments, when the UE 110 is the Al agent, the AI / ML engine 235 may assess a trustworthiness of an AI / ML model trained (or to be trained) by the UE 110.

[0030] The engines may also include a Beam ManagementMonitoring Engine 240 for monitoring the performance of the AI / ML models used in beam management. The Beam Management Monitoring Engine 240 may perform measurements of sets of beamsand may compare the measurements with the inference results of the AI / ML models. Based on configuration information relating to a definition of an event where a report needs to be sent to the network, the Beam Management Monitoring Engine 240 may determine whether the predictions of the AI / ML model are correct or not. Depending on the results of the comparison (i.e., whether the predictions of the Al model is correct) , then the Beam Management Monitoring Engine 240 may trigger the transmission of an event driven report) of the model performance to the network (gNB) , wherein the report includes the performance of the Al model. These operations will be described in greater detail below .

[0031] The above referenced engines being applications (e.g. , one or more programs) executed by the processor 205 is only example. The functionality associated with the engines may also be represented as a separate incorporated component of the UE 110 or may be a modular component coupled to the UE 110, e.g., an integrated circuit with or without firmware. For example, the integrated circuit may include input circuitry to receive signals and processing circuitry to process the signals and other information. The engines may also be embodied as one application or separate applications. In addition, in some UEs, the functionality described for the processor 205 is split among two or more processors such as a baseband processor and an applications processor. The example aspects may be implemented in any of these or other configurations of a UE .

[0032] The memory arrangement 210 may be a hardware component configured to store data related to operations performed by the UE 110. The display device 215 may be a hardware component configured to show data to a user while theI / O device 220 may be a hardware component that enables the user to enter inputs. The display device 215 and the I / O device 220 may be separate components or integrated together such as a touchscreen .

[0001] The transceiver 225 may be a hardware component configured to establish a connection with the 5G-NR RAN 120, the LTE RAN 122 etc. Accordingly, the transceiver 225 may operate on a variety of different frequencies or channels (e.g. , set of consecutive frequencies) . The transceiver 225 includes circuitry configured to transmit and / or receive signals (e.g. , control signals, data signals) . Such signals may be encoded with information implementing any one of the methods described herein. The processor 205 may be operably coupled to the transceiver 225 and configured to receive from and / or transmit signals to the transceiver 225. The processor 205 may be configured to encode, decode and / or process signals (e.g. , signaling from a base station of a network) for implementing any one of the methods described herein.

[0033] The example network base station, in this case gNB 120A, may represent a serving cell for the UE 110. The gNB 120A may represent any access node of the 5G NR network through which the UE 110 may establish a connection and manage network operations. The gNB 120A may include a processor, a memory arrangement, an input / output (I / O) device, a transceiver, and other components. The other components may include, for example, an audio input device, an audio output device, a battery, a data acquisition device, ports to electrically connect the gNB 120A to other electronic devices, etc. The functionality associated with the processor of the gNB 120A may also be represented as a separate incorporated component of thegNB 120A or may be a modular component coupled to the gNB 120A, e.g., an integrated circuit with or without firmware. For example, the integrated circuit may include input circuitry to receive signals and processing circuitry to process the signals and other information. In addition, in some gNBs, the functionality described for the processor is split among a plurality of processors (e.g., a baseband processor, an applications processor, etc.) . The example aspects may be implemented in any of these or other configurations of a gNB.

[0034] The memory may be a hardware component configured to store data related to operations performed by the UEs 110, 112. The I / O device may be a hardware component or ports that enable a user to interact with the gNB 120A. The transceiver may be a hardware component configured to exchange data with the UE 110 and any other UE in the network arrangement 100. The transceiver may operate on a variety of different frequencies or channels (e.g., set of consecutive frequencies) . Therefore, the transceiver may include one or more components (e.g., radios) to enable the data exchange with the various networks and UEs.

[0035] Fig. 3 shows an example base station 300 according to various example embodiments. The base station 300 may represent the gNB 120A, the gNB 120B or any other access node through which the UE 110 may establish a connection and manage network operations.

[0036] The base station 300 may include a processor 305, a memory arrangement 310, an input / output (I / O) device 315, a transceiver 320, and other components 325. The other components 325 may include, for example, an audio input device, an audio output device, a battery, a data acquisitiondevice, ports to electrically connect the base station 300 to other electronic devices and / or power sources, etc.

[0037] The processor 305 may be configured to execute a plurality of engines for the base station 300. For example, the engines may include an AI / ML engine 330 for performing various operations related to training an AI / ML model (as an Al agent) or facilitating the training and generation of a trained AI / ML model via one or more remote Al agents (as an Al manager) . In some embodiments, when the base station (e.g. , gNB 120A) is the Al agent, the AI / ML engine 330 may assess a trustworthiness of an AI / ML model trained (or to be trained) by the base station (gNB 120A) .

[0038] The engines may also include Configuration Engine335. The Configuration Engine 330 may include logic and / or circuitry that will send configuration information to the UE for performance monitoring to enable UE measurement of multiple beam sets. The configuration information may include information for configuring events, thresholds, UE report (s) , and container (s) for event driven report (s) . The Configuration Engine 330 may also send the UE a transmission for performance monitoring that may include Synchronization Signal Block (SSB) and / or Channel State Information-Reference Signals (CSI-RS) for measuring multiple beam sets.

[0039] Though engines 330 and 335 are shown as separate engines, in some embodiments, engines 330 and 335 may be combined into a single engine. Each of these example operations will be described in further detail below.

[0040] The memory arrangement 310 may be a hardware component configured to store data related to operations performed by the base station 300. The I / O device 315 may be a hardware component or ports that enable a user to interact with the base station 300.

[0041] The transceiver 320 may be a hardware component configured to exchange data with the UE 110 and any other UE in the network arrangement 100. The transceiver 320 may operate on a variety of different frequencies or channels (e.g., set of consecutive frequencies) . The transceiver 320 includes circuitry configured to transmit and / or receive signals (e.g., control signals, data signals) . Such signals may be encoded with information implementing any one of the methods described herein. The processor 305 may be operably coupled to the transceiver 320 and configured to receive from and / or transmit signals to the transceiver 320. The processor 305 may be configured to encode, decode and / or process signals (e.g., signaling from a UE) for implementing any one of the methods described herein.

[0042] Artificial Intelligence (Al) and Machine Learning (ML) is envisioned to be an integral part of Beyond 5G (B5G) (Rel-18 and beyond) , as well as 6G. In particular, AI / ML may play a role for the optimization of network functionalities. AI / ML models trained by the Al agent (s) in the network may be used to facilitate certain decision makings in one or more network functionalities (e.g., in RAN or Core Network) , including but not limited to: beam management; positioning, resource allocation; network management (operation and management (0AM) ) ; route election; energy saving; and load Balancing. In addition, in Al-as-a-Service (AlaaS) , the AI / ML services may be consumedby applications initiated at either the user or network side. The trained AI / ML model may be provided by any Al agent reachable in the network, including the UE . In various use cases, one or more UEs in a network may function as Al agents who may train at least a part of AI / ML models based on, e.g., data collected locally by each UE (e.g., radio-related or application-related measurements, sensor input, etc.) .

[0043] When the AI / ML model is trained by the UE for provision by the network as services to be consumed by some functions externally instantiated (e.g., on the network side or in an application server) , the UE may report / transf er the trained models to the network.

[0044] In many scenarios, the trained AI / ML models should meet a minimum required quality and may be trusted by the clients (e.g., network functions, UEs and / or external applications) using the AI / ML services. For critical applications (e.g., autonomous driving) , a poor quality AI / ML model may have disastrous effects.

[0045] Previously, in Release 18, discussions were had for BM-Case 1 and BM-Case 2 with a UE-side AI / ML model regarding performance monitoring. This included discussions regarding conf iguration / signaling from the gNB to the UE for measurement and / or reporting, the UE calculating performance metric (s) and either reporting it to the network or reporting an event to the network based on the performance metric (s) . However, the definition of an event and the performance metric (s) used to identify an event were held for future study. Some agreement was reached for model selection, activation, deactivation,switching, and fallback at least for UE sided models and two- sided models .

[0046] Release 19 has proposed AI / ML for beam management, in which DL Tx beam prediction for both UE-sided model and networksided model is envisioned. This may encompass one or more of the following: spatial-domain DL Tx beam prediction for Set A of beams based on measurement results of Set B of beams ("BM-Case 1") ; temporal DL Tx beam prediction for Set A of beams based on the historic measurement results of Set B of beams ("BM-Case 2") ; specifying necessary signaling / mechanism ( s ) to facilitate Life Cycle Management (LCM) operations specific to the Beam Management use cases, if any; and enabling method (s) to ensure consistency between training and inference regarding NW-side additional conditions (if identified) for inference at the UE .

[0047] Support for a UE-sided model, at least for BM-Case 1, includes the following options for a set of beams, where the set of beams is Set A (i.e. , the beams for UE prediction) :• Option 1: Beam information on predicted Top K beam(s) among a set of beams• Option 2: Beam information on predicted Top K beam(s) among a set of beams and RSRP of predicted Top K beam(s) among a set of beams, at least K=1 and more• Option 3: Beam information on predicted Top K beam(s) among a set of beams and probability information of predicted Top K beam(s) among a set of beams, where the probability information is the probability of the beam to be the Top 1 or Top K beam• Option 4: Beam information on predicted Top K beam(s) among a set of beams, Reference Signal Received Power (RSRP) of predicted Top K beam(s) among a set of beams, and confidence information of the RSRP.

[0048] In order to support the UE-sided AI / ML models for BM-Case 1 and BM-Case 2 , a definition of an event of a failure of the Al model is needed, as well as methods for how the UE may prepare and send an event driven report for performance monitoring of Al based beam management on the UE side . It is proposed herein to define an event driven report for the UE side model with UE side performance monitoring that includes a definition of layer 1 ( LI ) measurement and prediction failures for di f ferent events .

[0049] According to various example embodiments described herein, signaling and reporting mechanisms are disclosed for determining and reporting events indicating that UE side performance monitoring of the AI / ML based beam management has found errors with the AI / ML models such that the AI / ML models should be discontinued or changed . The UE may determine i f the AI / ML models are performing acceptably by comparing actual beam measurements to the predicted beam results from the AI / ML models . The UE determines an event has occurred based on the configuration information and the comparison of the first set of beam measurements to the predicted beam results , wherein the comparison includes making a decision on whether each predicted beam result is correct or accurate based on the actual beam measurements . I f an event occurs indicating one or more errors of the AI / ML models , the UE may send an event driven report to the network so that the network may take action based on the event driven report .

[0050] With respect to the examples and figures described herein, reference may be made to Al models and Al based beammanagement, but the proposed solutions and methods disclosed herein apply equally to ML models and ML based beam management.

[0051] In some example embodiments, in a first event where an LI failure (also referred to as an error indication or an indication failure) is deemed to have occurred, the predicted beam(s) are not part of the best beam sets with performance monitoring RS. This applies to the case where the UE side Al model output has predicted top K beams among a set of beams. In other example embodiments, in a second event, the predicted beam(s) 's Ll-RSRP and a measured set A' s Ll-RSRP difference is greater than a threshold. This applies to the case where the UE side Al model output has predicted top K beams among a set of beams and a RSRP of the predicted Top K beam(s) among a set of beams. In further example embodiments, in a third event, predicted beam(s) with a probability greater than a threshold are not part of the best beam sets with performance monitoring RS in set A. This applies to the case of predicted Top K beam(s) among a set of beams and probability information of predicted Top K beam(s) among a set of beams. In additional example embodiments, in a fourth event, the predicted beam(s) 's Ll-RSRP and a measured set A' s Ll-RSRP difference is greater than a threshold and / or a confidence is greater than a threshold. This applies to the case of beam information on predicted Top K beam(s) among a set of beams, RSRP of predicted Top K beam(s) among a set of beams, and confidence information of the RSRP.

[0052] For all of the events, certain procedures may also be used to define the event to mitigate short term measurement error. These procedures may include averaging in an evaluation window, using a counter and / or timer to count errors, and / orusing a Ll-RSRP filtering procedure. These procedures make the determining and reporting of events more stable.

[0053] UE side monitoring according to the proposed methods for an event driven report will now be discussed with reference to Fig. 4. Fig. 4 is an example flow diagram illustrating an example method for UE side performance monitoring of Al based beam management according to various example embodiments.

[0054] With the UE side model, the UE has all the measurements used to determine the Al model performance. In the method 400 of Fig. 4, the network (e.g., a gNB or base station) will send configuration information to the UE for performance monitoring to enable UE measurement of beams in set A and set B(410) . The configuration information may include information for configuring events, thresholds, UE report (s) , and container (s) for event driven report (s) . The gNB or base station may also send the UE a transmission for performance monitoring that may include Synchronization Signal Block (SSB) and / or Channel State Information-Reference Signals (CSI-RS) for measuring the beams in Set A and / or Set B (420) . In some example embodiments, this information may include CSI-RS information for Set A and either SSB or CSI-RS for Set B. The UE may monitor performance by performing measurement of the set A beams and comparing the measurements with the inference results using the set B measurements (430) . In some example embodiments, the UE may perform the set A measurements and the set B measurements independently. The Set A measurements may be the true or actual measurements, and the Set B measurements may be the input to the Al model. The UE may then compare the results using the definition of an event previously set forth to see whether for this particular measurement whether the predictions of the Almodel are correct or not. Depending on the results of the comparison (whether the predictions of the Al model is correct) , then the UE may transmit a report (event driven report) of the model performance to the network (gNB) , wherein the report includes the performance of the Al model (440) . Based on the event driven report, the network may decide to take further action, including de-activation of Ai modeling and / or switching Al models ( 450 ) .

[0055] Now that the basic method has been discussed, different cases or scenarios will be addressed for determining when an event occurs that indicates that there is an error in the AI model. For example, in a first case, when the AI output is the top-K predicted beams, an LI measurement error may be defined according to the following events.

[0056] In a first event, the predicted beam(s) are not part of the best beam sets with performance monitoring RS, and an LI error may be deemed to have occurred. This applies to the case where the UE side AI model output predicted top K beams among a set of beams. In some example embodiments, when K=l, each LI error indication may be defined as when the predicted beam is not the best beam based on the set A beam measurement using the configured CSI-RS resource for monitoring. That is, if the AI model has predicted beam 2 to be the best beam and the beam measurements performed by the UE on set A of beams shows that beam 2 is not the best beam, then an LI error is indicated. In other example embodiments, when K>1, each LI error indication may be defined as one or more beams not being part of the top K beams. For example, in a first alternative, if K=2 and one predicted beam or both predicted beams are not the top K beam in measured set A, one error case happens (if predicted best beamsare beam 1 and 2 , and best measured beam in set A for performance monitoring is beam 2 and beam 3 , one error happens ) . In a second alternative , an error is defined as only when all top K predicted beams are not part of the top measured set A beams for monitoring ( i f predicted best beams are beam 1 and 2 , an error is deemed to occur only i f neither beam 1 nor beam 2 are the best measured beam in set A for performance monitoring) . In some example embodiments , which of these di f ferent alternatives are used is part of the configuration information sent by the network to the UE .

[0057] Still referring to the first case , where the Al output is the top-K predicted beams , a window-based procedure may be used to define the event to mitigate short term measurement error and to make the determining and reporting of events more stable . Due to estimation error, particularly in a case with low Signal to Noise Ratio ( SNR) , some averaging of LI indications may be used to define the triggering event . In a first example embodiment , an evaluation window is defined . The evaluation window may be a moving average window . There may be multiple LI indications within an evaluation window, with an LI error indication being defined as previously discussed . In this example embodiment , the number of LI indication failures are examined over the evolution window, and i f the number of LI indication failures is greater than a threshold, then an event is triggered, and an event driven report may be sent to the network from the UE .

[0058] Fig . 5 shows an example of a window-based evaluation method for determining an event for an event driven report to be sent to the network according to various example embodiments . Fig . 5 shows periodic RS configuration for performancemonitoring. Semipersistent (SP) and aperiodic (ap)-RS configurations may also be used.

[0059] Fig. 5 shows the Set A RS transmission for performance monitoring and the Set A RS transmission for performance monitoring (with inference based on the Al model) . Set B RS may be twenty milliseconds (20 ms) apart in one embodiment, while Set A RS may be 80 ms apart. As discussed above, the evaluation window may be a moving average window in one embodiment. The bottom line of Fig. 5 shows the result of the UE's comparison of the set A measurements with the set B predictions, with a check indicating a correct prediction and an X indicating a failure. Assuming the threshold for an event is more incorrect predictions in the window than correct, in the first evaluation window Wl, there are two LI error indications and two correct predictions, so no event is triggered. However, in the second evaluation window W2, there are three LI error indications and only one correct prediction, so an event is triggered, and an event driven report may be sent from the UE to the network.

[0060] Still referring to the first case, where the Al output is the top-K predicted beams, a counter-based procedure may be used to define the event to mitigate short term measurement error and to make the determining and reporting of events more stable. Fig. 6 shows an example of a counter-based evaluation method for determining an event for an event driven report to be sent to the network according to various example embodiments. In this example embodiment, a counter may be used to count the number of LI errors. The counter may be increased by one when there is LI indication failure, with an LI error indication being defined as previously discussed (see second CSI-RS in Fig. 6) . The counter may be reset to zero when the UE determinesthere is a correct prediction ( see third CSI-RS in Fig . 6 ) . An event occurs when the counter reaches a preconfigured threshold . That is , when the counter meets the preconfigured threshold ( in Fig . 6 , assume the threshold is three errors ) , the Al based solution is deemed to be a failure and an event driven report is sent to the network by the UE . The counter-based solution requires periodically transmitted CS I-RS for performance monitoring purposes , and if the number of LI indication failures is equal to or greater than a threshold ( depending on the network configuration information) , then an event is triggered .

[0061] In a further variant , a procedure that combines the counter of Fig . 6 together with a timer may be used to define the event to mitigate short term measurement error and to make the determining and reporting of events more stable . Figs . 7A and 7B show an example of a counter and timer-based evaluation method for determining an event for an event driven report to be sent to the network according to various example embodiments . In the counter and timer-based solution, a counter is used like in Fig, 6 . However, the counter is reset upon an expiry of a timer ( see Fig . 7A) . As seen in Fig . 7A, the timer is started / restarted when an LI Al indication is received . The counter may be increased by 1 when there is LI indication failure . However, unless a threshold number of LI indication failures occurs before the timer expires , the timer will reset and no event is triggered . An event may be triggered when consistent Al failures are detected as determined by the counter reaching a ( configured) threshold before the expiration of the timer ( see Figure 7B ) .

[0062] The above description is for BM-Case 1 , but the same definition for errors or failures of the Al model (when Aloutput is top-K predicted beam) may be used for BM-case 2, where multiple time slots (N4) were predicted in the future. BM-case 1 deals with one time instance or slot, while BM-Case 2 uses past beams to predict future beams, such as looking at the previous eight beams to predict the future two beams in one example. In a first embodiment for BM-Case 2, the LI failure indication may be defined for each predicted time slot. An error event is counted for N4, or multiple N4, slots. In a second alternative, an LI failure indication may be defined for N4 predicted slots.

[0063] The basic method discussed in Fig. 4 above may be applied to additional cases or scenarios for determining when an event occurs that indicates that there is an error in the Al model. For example, a second case deals with when the Al output is the top-K predicted beams and Ll-RSRP. That is, in this second case, an event is deemed to have occurred when a difference between the predicted beam(s) 's Ll-RSRP (as determined by set B measurements) and the measured set A' s Ll- RSRP is greater than a threshold. This applies to the case where the UE side Al model output predicts top K beams among a set of beams and the RSRP of predicted Top K beam(s) among a set of beams. For BM-case 1, and when K=l, each LI error indication occurs when the measured Ll-RSRP of the best beam minus the predicted Ll-RSRP is greater than a preconfigured threshold.

[0064] In BM-Case 1, and where K>1, each LI error indication may be defined as one or more of the following:Alternative 1: (measured Ll-RSRP of the best beam minus the predicted Ll-RSRP of best beam) > threshold;Alternative 2: sum (measured Ll-RSRP of top-K beams minus the predicted Ll-RSRP of top K beams) > threshold (i.e., add together the differences between the measured Ll-RSRPs and the predicted Ll-RSSPs for each of the top K beams and see if the sum exceeds a threshold) ;Alternative 3: max (measured Ll-RSRP of top-K beams minus the predicted Ll-RSRP of top K beams) > threshold (i.e., see if the largest of the differences between the measured Ll-RSRPs and the predicted Ll-RSSPs for the top K beams exceeds a threshold) ; and Alternative 4: min (measured Ll-RSRP of top-K beams - the predicted Ll-RSRP of top K beams) > threshold (i.e., see if the smallest of the differences between the measured Ll-RSRPs and the predicted Ll-RSSPs for the top K beams exceeds a threshold.

[0065] For BM case 2, the above LI error indication for BM- Case 1 may be extended to the time dimension. In a first embodiment, an LI indication is generated per predicted time slot. In a second embodiment, an LI indication is generated similar to the K>1 case for BM-Case 1 per N4 predicted time slots .

[0066] For the second case, where the Al output is the top-K predicted beam and Ll-RSRP averaging, averaging of LI indications may be used to further define the triggering event, as previously discussed. That is, similar evaluation window and counter / timer approaches as shown in Figs. 5, 6, &A, and 7B may be used. On top of the approaches, additional filtering of Ll- RSRP may be defined. In one embodiment, a threshold comparing a filtered Ll-RSRP to a network configured threshold may be used. For example, from TS 38.331, 5.5.3.2, a Layer 3 filter is defined as:

[0067] A similar equation may be used for a Ll-RSRP filter for monitoring event. The filter coefficient alpha in the equation may be configured by the network in one embodiment.

[0068] The basic method discussed in Fig. 4 above may be applied to additional cases or scenarios for determining when an event occurs that indicates that there is an error in the Al model. For example, a third case deals with when the Al output is the top-K predicted beams and probability information, where the probability information is the probability of the beam to be the Top 1 or Top K beam.

[0069] In this scenario, an event may be defined as the situation where a predicted beam(s) with a probability greater than a threshold is / are not part of the best beam sets with performance monitoring RS in set A (the UE's actual measurements of the beams in Set A) . This applies to the case where there are predicted Top K beam(s) among a set of beams and there is probability information of the predicted Top K beam(s) among a set of beams. For example, where K=2, beams 1 and 2 may be predicted as the top beams, with beam 1 having an 80% probability of being the best beam and beam 2 having a 20% probability as being the best beam. For BM-case 1, if K=l, an LI error indication is defined as when the predicted beam is not the best beam based on set A beam measurements using the configured CSI-RS resource for monitoring, and / or the predicted beam probability is lower than a threshold.

[0070] For BM-Case 1 and where K>1, each LI error indication is defined as one or more beams not being part of the top K beams, and / or (1) the lowest (minimum) predicted beam probability of the top K beams is lower than a preconfigured threshold; and / or (2) the highest (maximum) predicted beam probability of the top K beams is lower than a preconfigured threshold. In one embodiment, the network may configure whether to use (1) and / or (2) above, as well as threshold values.

[0071] For BM case 2, the above LI error indication for BM- Case 1 may be extended to the time dimension. In a first embodiment, an LI indication is generated per predicted time slot. In a second embodiment, an LI indication is generated similar to the K>1 case for BM-Case 1 per N4 predicted time slots .

[0072] For the third case, where the Al output is the top-K predicted beams and a probability, averaging of LI indications may be used to further define the triggering event, as previously discussed. That is, similar evaluation window and counter / timer approaches as shown in Figs. 5, 6, &A, and 7B may be used.

[0073] The basic method discussed in Fig. 4 above may be applied to additional cases or scenarios for determining when an event occurs that indicates that there is an error in the Al model. For example, a fourth case deals with the scenario when the Al output is the top-K predicted beams, Ll-RSRPs, and a confidence indicator or confidence level. The confidence indicator is a value that indicates the amount of confidence in the Ll-RSRPs.

[0074] In this scenario, an event may be defined as the situation where the difference between the predicted beam(s) 's Ll-RSRP (the measurements of Set B) and the measured set A' s Ll- RSRP is greater than a threshold and / or the confidence indicator is less than a threshold. This applies to the case of having beam information on predicted Top K beam(s) among a set of beams, RSRPs of predicted Top K beam(s) among a set of beams, and confidence information of the RSRPs.

[0075] For BM case 1 and case 2, the same methods discussed above with respect to the second case (when the Al output is the top-K predicted beams and Ll-RSRP) may be applied here, with the only difference being the addition of the confidence information. Similarly, the evaluation window, the counter / timer , and / or the filtering of Ll-RSRP may be used in this scenario as well.Examples

[0076] In a first example, a method, comprising processing, based on signaling received from a base station, configuration information for performing monitoring of artificial intelligence (Al) or machine learning (ML) (AI / ML) based beam management, processing measurements performed for a first set of beams using at least part of the configuration information to obtain a first set of beam measurements, comparing the first set of beam measurements to predicted beam results, the predicted beam results obtained by inputting a second set of beam measurements into one or more AI / ML models used for the AI / ML based beam management, determining an event has occurred based on the configuration information and the comparing of the first set of beam measurements to the predicted beam results, wherein the event is indicative of one or more errors of the one or moreAI / ML models and generating, for transmission to the base station, an event driven report based on the event, the event driven report comprising information relating to the performance of the one or more AI / ML models.

[0077] In a second example, the method of the first example, wherein the configuration information comprises Synchronization Signal Block (SSB) information and / or Channel State Information-Reference Signals (CSI-RS) .

[0078] In a third example, the method of the first example, wherein the first set of beam measurements indicates a best beam of the first set of beams and the predicted beam results include a predicted best beam, and wherein the processing circuitry is configured to determine an event has occurred when the predicted best beam is not the best beam as indicated by the first set of beam measurements.

[0079] In a fourth example, the method of the first example, wherein the first set of beam measurements indicates a top K number of beams of the first set of beams and the predicted beam results include top K predicted best beams, where K is a positive integer greater than one; and wherein the processing circuitry is configured to determine an event has occurred when one or more of the top K predicted best beams are not the top K number of beams in the first set of beams.

[0080] In a fifth example, the method of the first example, wherein the first set of beam measurements indicates a top K number of beams of the first set of beams and the predicted beam results include top K predicted best beams, where K is a positive integer greater than one; and wherein the processingcircuitry is configured to determine an event has occurred when none of the top K predicted best beams are part of the top K number of beams in the first set of beams.

[0081] In a sixth example, the method of the first example, further comprising determining an event has occurred when comparing the first set of beam measurements to the predicted beam results indicates one or more errors in the predicted beam results.

[0082] In a seventh example, the method of the first example, further comprising determining an event has occurred when a number of error indications found when comparing the first set of beam measurements to the predicted beam results exceeds a predetermined threshold value during an evaluation window .

[0083] In an eighth example, the method of the first example, further comprising determining an event has occurred when a number of error indications found when comparing the first set of beam measurements to the predicted beam results exceeds a predetermined threshold value of a counter, wherein the counter is increased by one for each error indication and is reset to zero when the comparing of the first set of beam measurements to the predicted beam results indicates a correct prediction .

[0084] In a ninth example, the method of the first example, further comprising determining an event has occurred only when a number of error indications found when comparing the first set of beam measurements to the predicted beam results exceeds a predetermined threshold value of a counter before a timerassociated with the counter expires, wherein the counter is increased by one for each error indication and the counter is reset to zero when the comparing of the first set of beam measurements to the predicted beam results indicates a correct prediction or when the timer expires, and wherein the timer is started or restarted upon receiving an indication of an error.

[0085] In a tenth example, the method of the first example, wherein the predicted beam results comprise predictions for multiple future time slots, and wherein the processing circuitry is configured to compare the first set of beam measurements to the predicted beam results for each predicted time slot of the multiple future time slots to determine one or more errors in the predicted beam results, wherein the one or more errors in the predicted time slot is counted for all of the multiple future time slots.

[0086] In an eleventh example, the method of the first example, wherein the predicted beam results comprise predictions for multiple future time slots, and wherein the processing circuitry is configured to compare the first set of beam measurements to the predicted beam results for the multiple future time slots to determine one or more errors in the predicted beam results for the multiple future time slots.

[0087] In a twelfth example, the method of the first example, wherein the first set of beam measurements indicates a best beam of the first set of beams and comprises a measured Reference Signal Received Power (RSRP) of the best beam, the predicted beam results include a predicted RSRP of a predicted best beam, the method further comprising determining an event has occurred when a difference between the measured RSRP ofthe best beam and the predicted RSRP of the predicted best beam is greater than a preconfigured threshold.

[0088] In a thirteenth example, the method of the first example, wherein the first set of beam measurements indicates a top K number of beams of the first set of beams and comprises measured Reference Signal Received Powers (RSRPs) of the top K beams of the first set of beams, where K is a positive integer greater than one, the predicted beam results include top K predicted best beams and predicted RSRPs for each of the top K predicted best beams, the method further comprising determining an event has occurred when a difference between the measured RSRP of a best beam of the first set of beams and the predicted RSRP of a predicted best beam of the top K predicted best beams is greater than a preconfigured threshold .

[0089] In a fourteenth example, the method of the first example, wherein the first set of beam measurements indicates a top K number of beams of the first set of beams and comprises measured Reference Signal Received Powers (RSRPs) of the top K beams of the first set of beams, where K is a positive integer greater than one, the predicted beam results include top K predicted best beams and predicted RSRPs for each of the top K predicted best beams the method further comprising determining an event has occurred when a difference between the measured RSRP and the predicted RSRP is calculated for each of the top K beams to obtain a plurality of differences and a sum of the plurality of differences is greater than a preconfigured threshold.

[0090] In a fifteenth example, the method of the first example, wherein the first set of beam measurements indicates a top K number of beams of the first set of beams and comprises measured Reference Signal Received Powers (RSRPs) of the top K beams, where K is a positive integer greater than one, the predicted beam results include top K predicted best beams and predicted RSRPs for each of the top K predicted best beams and the method further comprising determining an event has occurred when a difference between the measured RSRP and the predicted RSSP is calculated for each of the top K beams to obtain a plurality of differences and a largest one of the plurality of differences is greater than a preconfigured threshold .

[0091] In a sixteenth example, the method of the first example, wherein the first set of beam measurements indicates a top K number of beams of the first set of beams and comprises measured Reference Signal Received Powers (RSRPs) of the top K beams, where K is a positive integer greater than one, the predicted beam results include top K predicted best beams and predicted RSRPs for each of the top K predicted best beams and the method further comprising determining an event has occurred when a difference between the measured RSRP and the predicted RSSP is calculated for each of the top K beams to obtain a plurality of differences and a smallest one of the plurality of differences is greater than a preconfigured threshold .

[0092] In a seventeenth example, the method of the thirteenth example, wherein the processing circuitry is configured to determine an event has occurred based on additional filtering of the RSRPs, where a threshold comparing a filtered RSRP to anetwork configured threshold is used, where a filter for theRSRP is defined aswherein alpha is preconfigured in the configuration information.

[0093] In an eighteenth example, the method of the first example, wherein the first set of beam measurements indicates a best beam of the first set of beams, the predicted beam results include a predicted best beam and a probability information, wherein the probability information includes a probability of the predicted best beam to be the best beam; and the method further comprising determining an event has occurred when the predicted best beam is not the best beam based on the first set of beam measurements, and / or the probability of the predicted best beam is lower than a preconfigured threshold.

[0094] In a nineteenth example, the method of the first example, wherein the first set of beam measurements indicates a top K number of beams of the first set of beams, where K is a positive integer greater than one, the predicted beam results include top K predicted best beams and probability information for each of the top K predicted best beams, wherein the probability information includes a probability of each of the top K predicted best beams to be in the top K number of best beams and the method further comprising determining an event has occurred when a lowest probability of each of the probabilities of the top K predicted best beams to be in the top K number of best beams is lower than a preconfigured threshold.

[0095] In a twentieth example, the method of the first example, wherein the first set of beam measurements indicates a top K number of beams of the first set of beams, where K is a positive integer greater than one, the predicted beam results include top K predicted best beams and probability information for each of the top K predicted best beams, wherein the probability information includes a probability of each of the top K predicted best beams to be in the top K number of best beams and the method further comprising determining an event has occurred when a highest probability of each of the probabilities of the top K predicted best beams to be in the top K number of best beams is lower than a preconfigured threshold.

[0096] In a twenty first example, the method of the first example, wherein the first set of beam measurements indicates a best beam or beams of the first set of beams and comprises a measured Reference Signal Received Power (RSRP) of the best beam or beams, the predicted beam results include a predicted RSRP of a predicted best beam or beams and confidence information, wherein the confidence information includes a value that indicates an amount of confidence in the predicted RSRP and the method further comprising determining an event has occurred when a difference between the measured RSRP of the best beam and the predicted RSRP of the predicted best beam is greater than a preconfigured threshold and / or when the value that indicates the amount of confidence in the predicted RSRP is less than a preconfigured threshold.

[0097] In a twenty second example, a processor configured to perform any of the methods of the first through twenty first examples .

[0098] In a twenty third example, a user equipment (UE) configured to perform any of the methods of the first through twenty first examples.

[0099] In a twenty fourth example, a method, comprising generating, for transmission to a user equipment (UE) , configuration information to perform monitoring of artificial intelligence (Al) or machine learning (ML) (AI / ML) based beam management, the configuration information including information for the UE to measure a first set of beams to obtain a first set of beam measurements and for the UE to determine if there are errors in one or more AI / ML models used for the AI / ML based beam management, processing, based on signals received from the UE, an event driven report, the event driven report comprising event information indicative of one or more errors of the one or more AI / ML models, the event information generated by the UE based on comparing the first set of beam measurements to predicted beam results obtained by inputting a second set of beam measurements into the one or more AI / ML models and determining an action to take based on the event driven report, the action including de-activation of the one or more AI / ML models and / or switching to a different Al / ML model .

[0100] In a twenty fifth example, a processor configured to perform the method of the twenty fourth example.

[0101] In a twenty sixth example, a baser station configured to perform the method of the twenty fourth example.

[0102] Those skilled in the art will understand that the above-described example embodiments may be implemented in any suitable software or hardware configuration or combination thereof . An example hardware platform for implementing the example embodiments may include , for example, an Intel x86 based platform with compatible operating system, a Windows OS , a Mac platform and MAC OS , a mobile device having an operating system such as iOS , Android, etc . The example embodiments of the above described method may be embodied as a program containing lines of code stored on a non-transitory computer readable storage medium that , when compiled, may be executed on a processor or microprocessor .

[0103] Although this application described various embodiments each having di fferent features in various combinations , those skilled in the art will understand that any of the features of one embodiment may be combined with the features of the other embodiments in any manner not speci fically disclaimed or which is not functionally or logically inconsistent with the operation of the device or the stated functions of the disclosed embodiments .

[0104] It is well understood that the use of personally identi fiable information should follow privacy policies and practices that are generally recogni zed as meeting or exceeding industry or governmental requirements for maintaining the privacy of users . In particular, personally identifiable information data should be managed and handled so as to minimi ze risks of unintentional or unauthori zed access or use , and the nature of authori zed use should be clearly indicated to users .

[0105] It will be apparent to those skilled in the art that various modi fications may be made in the present disclosure , without departing from the spirit or the scope of the disclosure . Thus , it is intended that the present disclosure cover modifications and variations of this disclosure provided they come within the scope of the appended claims and their equivalent .

Claims

What is Claimed:

1. An apparatus comprising processing circuitry coupled to memory, the processing circuitry configured to: process, based on signaling received from a base station, configuration information for performing monitoring of artificial intelligence (Al) or machine learning (ML) (AI / ML) based beam management; process measurements performed for a first set of beams using at least part of the configuration information to obtain a first set of beam measurements; compare the first set of beam measurements to predicted beam results, the predicted beam results obtained by inputting a second set of beam measurements into one or more AI / ML models used for the AI / ML based beam management; determine an event has occurred based on the configuration information and comparing the first set of beam measurements to the predicted beam results, wherein the event is indicative of one or more errors of the one or more AI / ML models; and generate, for transmission to the base station, an event driven report based on the event, the event driven report comprising information relating to the performance of the one or more AI / ML models.

2. The apparatus of claim 1, wherein the configuration information comprises Synchronization Signal Block (SSB) information and / or Channel State Information-Reference Signals (CSI-RS) .

3. The apparatus of claim 1, wherein the first set of beam measurements indicates a best beam of the first set of beams and the predicted beam results include a predicted best beam,and wherein the processing circuitry is configured to determine the event has occurred when the predicted best beam is not the best beam as indicated by the first set of beam measurements .

4. The apparatus of claim 1, wherein the first set of beam measurements indicates a top K number of beams of the first set of beams and the predicted beam results include top K predicted best beams, where K is a positive integer greater than one; and wherein the processing circuitry is configured to determine the event has occurred when one or more of the top K predicted best beams are not the top K number of beams in the first set of beams.

5. The apparatus of claim 1, wherein the first set of beam measurements indicates a top K number of beams of the first set of beams and the predicted beam results include top K predicted best beams, where K is a positive integer greater than one; and wherein the processing circuitry is configured to determine the event has occurred when none of the top K predicted best beams are part of the top K number of beams in the first set of beams.

6. The apparatus of claim 1, wherein the processing circuitry is configured to determine the event has occurred when comparing the first set of beam measurements to the predicted beam results indicates one or more errors in the predicted beam results.

7. The apparatus of claim 1, wherein the processing circuitry is configured to determine the event has occurred when a number of error indications found when comparing thefirst set of beam measurements to the predicted beam results exceeds a predetermined threshold value during an evaluation window .

8. The apparatus of claim 1, wherein the processing circuitry is configured to determine the event has occurred when a number of error indications found when comparing the first set of beam measurements to the predicted beam results exceeds a predetermined threshold value of a counter, wherein the counter is increased by one for each error indication and is reset to zero when the comparing of the first set of beam measurements to the predicted beam results indicates a correct prediction .

9. The apparatus of claim 1, wherein the processing circuitry is configured to determine the event has occurred only when a number of error indications found when comparing the first set of beam measurements to the predicted beam results exceeds a predetermined threshold value of a counter before a timer associated with the counter expires, wherein the counter is increased by one for each error indication and the counter is reset to zero when the comparing of the first set of beam measurements to the predicted beam results indicates a correct prediction or when the timer expires, and wherein the timer is started or restarted upon receiving an indication of an error.

10. The apparatus of claim 1, wherein the predicted beam results comprise predictions for multiple future time slots, and wherein the processing circuitry is configured to compare the first set of beam measurements to the predicted beam results for each predicted time slot of the multiple futuretime slots to determine one or more errors in the predicted beam results, wherein the one or more errors in the predicted time slot is counted for all of the multiple future time slots .

11. The apparatus of claim 1, wherein the predicted beam results comprise predictions for multiple future time slots, and wherein the processing circuitry is configured to compare the first set of beam measurements to the predicted beam results for the multiple future time slots to determine one or more errors in the predicted beam results for the multiple future time slots.

12. The apparatus of claim 1, wherein: the first set of beam measurements indicates a best beam of the first set of beams and comprises a measured Reference Signal Received Power (RSRP) of the best beam; the predicted beam results include a predicted RSRP of a predicted best beam; and the processing circuitry is configured to determine the event has occurred when a difference between the measured RSRP of the best beam and the predicted RSRP of the predicted best beam is greater than a preconfigured threshold.

13. The apparatus of claim 1, wherein: the first set of beam measurements indicates a top K number of beams of the first set of beams and comprises measured Reference Signal Received Powers (RSRPs) of the top K beams of the first set of beams, where K is a positive integer greater than one;the predicted beam results include top K predicted best beams and predicted RSRPs for each of the top K predicted best beams; and the processing circuitry is configured to determine the event has occurred when a difference between the measured RSRP of a best beam of the first set of beams and the predicted RSRP of a predicted best beam of the top K predicted best beams is greater than a preconfigured threshold.

14. The apparatus of claim 1, wherein: the first set of beam measurements indicates a top K number of beams of the first set of beams and comprises measured Reference Signal Received Powers (RSRPs) of the top K beams of the first set of beams, where K is a positive integer greater than one; the predicted beam results include top K predicted best beams and predicted RSRPs for each of the top K predicted best beams; and the processing circuitry is configured to determine the event has occurred when a difference between the measured RSRP and the predicted RSRP is calculated for each of the top K beams to obtain a plurality of differences and a sum of the plurality of differences is greater than a preconfigured threshold .

15. The apparatus of claim 1, wherein: the first set of beam measurements indicates a top K number of beams of the first set of beams and comprises measured Reference Signal Received Powers (RSRPs) of the top K beams, where K is a positive integer greater than one;the predicted beam results include top K predicted best beams and predicted RSRPs for each of the top K predicted best beams; and the processing circuitry is configured to determine the event has occurred when a difference between the measured RSRP and the predicted RSSP is calculated for each of the top K beams to obtain a plurality of differences and a largest one of the plurality of differences is greater than a preconfigured threshold.

16. The apparatus of claim 1, wherein: the first set of beam measurements indicates a top K number of beams of the first set of beams and comprises measured Reference Signal Received Powers (RSRPs) of the top K beams, where K is a positive integer greater than one; the predicted beam results include top K predicted best beams and predicted RSRPs for each of the top K predicted best beams; and the processing circuitry is configured to determine the event has occurred when a difference between the measured RSRP and the predicted RSSP is calculated for each of the top K beams to obtain a plurality of differences and a smallest one of the plurality of differences is greater than a preconfigured threshold.

17. The apparatus of claim 1, wherein: the first set of beam measurements indicates a best beam of the first set of beams; the predicted beam results include a predicted best beam and a probability information, wherein the probability information includes a probability of the predicted best beam to be the best beam; andthe processing circuitry is configured to determine the event has occurred when the predicted best beam is not the best beam based on the first set of beam measurements, and / or the probability of the predicted best beam is lower than a preconfigured threshold.

18. The apparatus of claim 1, wherein: the first set of beam measurements indicates a top K number of beams of the first set of beams, where K is a positive integer greater than one; the predicted beam results include top K predicted best beams and probability information for each of the top K predicted best beams, wherein the probability information includes a probability of each of the top K predicted best beams to be in the top K number of best beams; and the processing circuitry is configured to determine the event has occurred when a lowest probability of each of the probabilities of the top K predicted best beams to be in the top K number of best beams is lower than a preconfigured threshold .

19. The apparatus of claim 1, wherein: the first set of beam measurements indicates a top K number of beams of the first set of beams, where K is a positive integer greater than one; the predicted beam results include top K predicted best beams and probability information for each of the top K predicted best beams, wherein the probability information includes a probability of each of the top K predicted best beams to be in the top K number of best beams; and the processing circuitry is configured to determine the event has occurred when a highest probability of each of theprobabilities of the top K predicted best beams to be in the top K number of best beams is lower than a preconfigured threshold .

20. An apparatus comprising processing circuitry coupled to memory, the processing circuitry configured to: generate, for transmission to a user equipment (UE) , configuration information to perform monitoring of artificial intelligence (Al) or machine learning (ML) (AI / ML) based beam management, the configuration information including information for the UE to measure a first set of beams to obtain a first set of beam measurements and for the UE to determine if there are errors in one or more AI / ML models used for the AI / ML based beam management; process, based on signals received from the UE, an event driven report, the event driven report comprising event information indicative of one or more errors of the one or more AI / ML models, the event information generated by the UE based on comparing the first set of beam measurements to predicted beam results obtained by inputting a second set of beam measurements into the one or more AI / ML models; and determine an action to take based on the event driven report, the action including de-activation of the one or more AI / ML models and / or switching to a different AI / ML model.

Images